Modelos de impacto de mercado para execução em backtests e estimativa de custos

# ML4T Library: Market Impact Models


# ML4T Library: Market Impact Models

**Docker image**: `ml4t`

The previous notebooks derived impact models by hand. A backtest needs them as code it can call
on every fill, and `ml4t.backtest.execution` provides four:

1. **NoImpact** charges nothing, which is what a backtest does by default and why backtests
   flatter strategies that trade a lot.
2. **LinearImpact** charges in proportion to the participation rate.
3. **SquareRootImpact** charges in proportion to volatility times the square root of
   participation, which is the shape `03_market_impact_calibration` argues for.
4. **PowerLawImpact** takes the exponent as a parameter, so it covers both of the above and
   everything between.

All four answer the same question - given an order of this size against this much volume, how far
does the price move against me - and all four return a **signed per-share price move**: positive
for a buy, negative for a sell, so that adding it to the decision price always makes the fill
worse. What separates them is the shape of the relationship and the parameters that scale it.

**Learning Objectives**
- Call any of the four models through one interface and read the sign convention correctly
- Separate what an impact model's exponent decides from what its coefficient decides, and see why
  two models' levels cannot be compared until their coefficients are put on the same footing
- Compose an impact model with a participation limit to produce the fill a backtest would record
- Substitute a measured volatility into a model whose coefficient you have not measured, and say
  what that does and does not establish
- Recognize that these models are stateless, and supply persistence across child orders yourself
  when you need it

**Book Reference:** Chapter 18, Section 18.4

**Prerequisites:** Read
[`03_market_impact_calibration`](03_market_impact_calibration.ipynb) for the economic meaning of
impact coefficients and
[`07_ml4t_volume_participation`](07_ml4t_volume_participation.ipynb) for quantity constraints per
bar.

## Imports & Setup

```python
"""ML4T Library: Market Impact Models - Comparing impact model APIs for backtesting."""

from datetime import timedelta

import numpy as np
import plotly.graph_objects as go
import polars as pl
from IPython.display import Markdown, display
from ml4t.backtest.execution.impact import (
    LinearImpact,
    MarketImpactModel,
    NoImpact,
    PowerLawImpact,
    SquareRootImpact,
)
from ml4t.backtest.execution.limits import VolumeParticipationLimit
from ml4t.backtest.execution.result import ExecutionResult

from data import load_etfs
from utils.reproducibility import set_global_seeds
from utils.style import COLORS, ml4t_palette, show_plotly_with_alt
```

```python
SEED = 42
UNIVERSE = ["SPY", "QQQ", "IWM", "EEM", "DBC"]
LOOKBACK_DAYS = 365
IMPACT_COEFFICIENT = 0.5
SCENARIO_VOLATILITY = 0.02
MAX_PARTICIPATION = 0.10
PARAMETERIZATION_PARTICIPATION = 0.05
PERSISTENCE_FRACTION = 0.50
```

```python
set_global_seeds(SEED)
```

What each setting decides:

- `UNIVERSE` and `LOOKBACK_DAYS` fix the five ETFs and the trailing window their prices, volumes
  and volatilities are measured over. The five span a range of volatility wide enough that the
  model's volatility term visibly changes the answer.
- `IMPACT_COEFFICIENT` is the library's own default for the square-root model. It is a stated
  figure and every impact level in this notebook scales directly with it.
- `SCENARIO_VOLATILITY` is the daily volatility used wherever a single round number is wanted
  instead of a measured one, so the scenario sections stay comparable with each other.
- `MAX_PARTICIPATION` caps how much of a bar's volume one order may take. Anything above it is
  filled partially and the remainder is carried, which is what a backtest must do rather than
  pretending an unlimited order filled at one price.
- `PARAMETERIZATION_PARTICIPATION` is the fixed rate at which the five ETFs are compared, so that
  the only thing differing between them is their measured volatility.
- `PERSISTENCE_FRACTION` is the share of each child order's impact that the notebook carries into
  the next child's reference price. The library models are stateless and do not do this; the
  value is the notebook's own assumption.

## Part 1: Understanding the Impact Model API

All impact models inherit from `MarketImpactModel` and implement a single method:

```python
def calculate(
    self,
    quantity: float,    # Order quantity (shares)
    price: float,       # Current market price
    volume: float,      # Bar volume (daily, minute, etc.)
    is_buy: bool,       # Trade direction
) -> float:            # Signed per-share price move
```

The return is a **signed per-share price move** in currency units, not basis points: positive for
buys and negative for sells, so adding it to the decision price always moves the fill against the
trader. Its absolute value is the per-share cost.

The `volume` argument deserves care, because it is where look-ahead enters a backtest most
easily. A bar's total volume is known only once the bar has closed, so sizing an order against it
uses information the order did not have. Use volume accumulated so far, a lagged or forecast
figure, or execute on the following bar. Every example below supplies volume as a stated scenario
input rather than reading it from a bar the order is supposedly trading inside.

These models are also **stateless**: each call sees one order and knows nothing about the ones
before it. That matters for a parent order worked in slices, because permanent impact - the part
of the move that does not revert - should be paid again by every later slice. Part 4 supplies
that persistence explicitly, outside the model.

```python
# Instantiate all four models with default parameters
models = {
    "NoImpact": NoImpact(),
    "LinearImpact": LinearImpact(coefficient=0.1),
    "SquareRootImpact": SquareRootImpact(
        coefficient=IMPACT_COEFFICIENT, volatility=SCENARIO_VOLATILITY
    ),
    "PowerLawImpact": PowerLawImpact(coefficient=0.1, exponent=0.5),
}

# Sample trade parameters
price = 100.0
volume = 1_000_000  # Daily volume
is_buy = True

# Impact (price units) at 1% / 5% / 10% participation, buying a $100 stock vs 1M ADV
impact_comparison = pl.DataFrame(
    [
        {
            "model": name,
            "impact_per_share_1pct_usd": model.calculate(volume * 0.01, price, volume, is_buy),
            "impact_per_share_5pct_usd": model.calculate(volume * 0.05, price, volume, is_buy),
            "impact_per_share_10pct_usd": model.calculate(volume * 0.10, price, volume, is_buy),
        }
        for name, model in models.items()
    ]
)
impact_comparison
```

**Finding**: Model levels are not comparable until their coefficients, volatility inputs, and
volume conventions are aligned. This table is an API and unit check; it shows how the illustrative
parameterizations differ rather than which model is most accurate.

## Part 2: Model Deep Dive

### LinearImpact

$$\text{Impact} = \text{coefficient} \times \frac{Q}{V} \times P$$

- Simple, intuitive model
The coefficient is the whole model. It is the per-share concession, as a fraction of price, that
an order equal to the market's entire volume would pay, and every smaller order pays that
fraction scaled down in proportion. Multiply the coefficient by ten thousand to read it as the
cost in basis points at full participation.

```python
# LinearImpact sensitivity analysis
linear = LinearImpact(coefficient=0.1)

participation_rates = np.linspace(0.001, 0.20, 100)
quantities = participation_rates * volume

linear_impacts = [linear.calculate(q, price, volume, is_buy) for q in quantities]
print(f"LinearImpact with coefficient {linear.coefficient}, buying a ${price:.0f} stock:")
for participation in (0.01, 0.10, 1.00):
    impact = linear.calculate(volume * participation, price, volume, is_buy)
    print(
        f"  {participation:>5.0%} of volume -> ${impact:.4f} per share, "
        f"{impact / price * 10_000:>6.0f} bps"
    )
```

**Reading the output**: The three lines are exactly proportional, which is the model. That makes
it easy to sanity-check - at full participation the cost in basis points is the coefficient times
ten thousand - and it is also its weakness. Proportionality means the last share of a large order
costs the same as the first, which contradicts what the square-root shape in
`03_market_impact_calibration` was fitted to.

### SquareRootImpact (Almgren-Chriss)

$$\text{Impact} = \text{coefficient} \times \sigma \times \sqrt{\frac{Q}{ADV}} \times P$$

- Common research-motivated baseline across asset classes
Two things scale this model. The `coefficient` plays the same role as the linear model's, and
`volatility` is the instrument's daily standard deviation of returns, which sets how much a given
amount of participation costs in a market that moves a lot against one that does not.
`adv_factor` converts the `volume` argument into an average daily volume when the bars are
shorter than a day: it is the number of bars in a session, so it stays at one for daily bars.

```python
# SquareRootImpact with different volatility regimes
sqrt_low_vol = SquareRootImpact(coefficient=IMPACT_COEFFICIENT, volatility=SCENARIO_VOLATILITY / 2)
sqrt_mid_vol = SquareRootImpact(coefficient=IMPACT_COEFFICIENT, volatility=SCENARIO_VOLATILITY)
sqrt_high_vol = SquareRootImpact(coefficient=IMPACT_COEFFICIENT, volatility=SCENARIO_VOLATILITY * 2)

sqrt_impacts_low = [sqrt_low_vol.calculate(q, price, volume, is_buy) for q in quantities]
sqrt_impacts_mid = [sqrt_mid_vol.calculate(q, price, volume, is_buy) for q in quantities]
sqrt_impacts_high = [sqrt_high_vol.calculate(q, price, volume, is_buy) for q in quantities]
sqrt_10pct_low = sqrt_low_vol.calculate(volume * 0.10, price, volume, is_buy)
sqrt_10pct_mid = sqrt_mid_vol.calculate(volume * 0.10, price, volume, is_buy)
sqrt_10pct_high = sqrt_high_vol.calculate(volume * 0.10, price, volume, is_buy)

print(f"SquareRootImpact with coefficient {IMPACT_COEFFICIENT}, at the same participation:")
for model, impact in (
    (sqrt_low_vol, sqrt_10pct_low),
    (sqrt_mid_vol, sqrt_10pct_mid),
    (sqrt_high_vol, sqrt_10pct_high),
):
    print(
        f"  volatility {model.volatility:.1%} a day: ${impact:.4f} per share, "
        f"{impact / price * 10_000:.1f} bps"
    )
```

**Reading the output**: Doubling the volatility doubles the cost, at every participation rate,
because volatility enters the model as a plain multiplier. That is what makes this model
responsive to market conditions in a way the linear one is not: the same order in the same
instrument costs more in a week when the price is moving, without anything about the order or
the instrument's liquidity having changed.

### PowerLawImpact

$$\text{Impact} = \text{coefficient} \times \left(\frac{Q}{V}\right)^{\text{exponent}} \times P$$

The exponent is what the other two models fix and this one exposes. Set it to one and the model
is the linear one; set it to a half and it is the square-root one. Any exponent between zero and
one makes the curve concave, so each additional unit of participation costs less than the last,
and an exponent above one makes it convex, so each costs more.
- `min_impact` is a minimum absolute per-share price move, not a fixed dollar trade cost

```python
# PowerLawImpact with different exponents
power_linear = PowerLawImpact(coefficient=0.1, exponent=1.0)  # Same as LinearImpact
power_sqrt = PowerLawImpact(coefficient=0.1, exponent=0.5)
power_concave = PowerLawImpact(coefficient=0.1, exponent=0.3)
power_mild_concave = PowerLawImpact(coefficient=0.1, exponent=0.8)

power_impacts_linear = [power_linear.calculate(q, price, volume, is_buy) for q in quantities]
power_impacts_sqrt = [power_sqrt.calculate(q, price, volume, is_buy) for q in quantities]
power_impacts_concave = [power_concave.calculate(q, price, volume, is_buy) for q in quantities]
power_impacts_mild_concave = [
    power_mild_concave.calculate(q, price, volume, is_buy) for q in quantities
]
power_10pct_linear = power_linear.calculate(volume * 0.10, price, volume, is_buy)
power_10pct_sqrt = power_sqrt.calculate(volume * 0.10, price, volume, is_buy)
power_10pct_concave = power_concave.calculate(volume * 0.10, price, volume, is_buy)

print(f"PowerLawImpact with coefficient {power_linear.coefficient}, at the same participation:")
for model, impact, shape in (
    (power_linear, power_10pct_linear, "linear"),
    (power_sqrt, power_10pct_sqrt, "square root"),
    (power_concave, power_10pct_concave, "strongly concave"),
):
    print(
        f"  exponent {model.exponent}: ${impact:.4f} per share, "
        f"{impact / price * 10_000:.1f} bps ({shape})"
    )
```

**Finding**: Every exponent between zero and one is concave; smaller exponents bend the curve
more strongly. For participation below one, a smaller exponent also raises the level when the
coefficient is held fixed, so curvature and coefficient must be chosen or estimated jointly.

## Part 3: Visual Comparison

The first chart compares the illustrative parameter sets used above. Its levels are not a model
ranking: `SquareRootImpact` includes volatility in its scale, while `PowerLawImpact` does not.

```python
fig = go.Figure()
comparison_specs = (
    ("Linear (c=0.1)", linear_impacts, COLORS["blue"], "solid"),
    ("Square root (c=0.5, σ=2%)", sqrt_impacts_mid, COLORS["amber"], "dash"),
    ("Power law (c=0.1, exp=0.5)", power_impacts_sqrt, COLORS["copper"], "dot"),
)
for label, impacts, color, dash in comparison_specs:
    fig.add_scatter(
        x=participation_rates * 100,
        y=[impact / price * 10000 for impact in impacts],
        mode="lines",
        name=label,
        line=dict(width=2.5, color=color, dash=dash),
    )
fig.update_layout(
    title="Signed buy impact against participation rate, by impact model",
    xaxis_title="Participation rate (% of volume)",
    yaxis_title="Signed buy impact (bps)",
    height=500,
    legend=dict(orientation="h", y=-0.23, x=0.5, xanchor="center"),
    margin=dict(b=105),
)
show_plotly_with_alt(
    fig,
    "Three impact curves against participation rate. The linear one is a straight line through "
    "the origin; the square-root and power-law ones share the same concave shape, rising fast "
    "near zero participation and flattening after it. The power-law curve is the highest of the "
    "three over almost the whole range, even though it is the square-root curve that shares its "
    "shape. The square-root curve starts above the linear one at the smallest participations and "
    "crosses below it early, staying the lowest of the three from there on.",
)
```

**Reading the chart**: Functional form alone does not rank cost severity. The power-law curve
sits above the square-root curve here even though the two have the identical shape, because one
of them multiplies by a volatility and the other does not, and their coefficients are on
different footings as a result. A reader who took the higher curve to be the more pessimistic
model would be reading the coefficients, not the models.

### Volatility Sets Scale

The coefficient is held at the library's default and only the volatility varies, so the three
curves differ by exactly the factor their volatilities differ by. This gets its own chart because
its vertical scale is not comparable with the power-law chart that follows.

```python
volatility_fig = go.Figure()
VOLATILITY_COLORS = ml4t_palette(3, categorical=True)
vol_specs = (
    ("σ=1%", sqrt_impacts_low, VOLATILITY_COLORS[0], "solid"),
    ("σ=2%", sqrt_impacts_mid, VOLATILITY_COLORS[1], "dash"),
    ("σ=4%", sqrt_impacts_high, VOLATILITY_COLORS[2], "dot"),
)
for label, impacts, color, dash in vol_specs:
    volatility_fig.add_scatter(
        x=participation_rates * 100,
        y=[impact / price * 10000 for impact in impacts],
        mode="lines",
        name=label,
        line=dict(width=2.5, color=color, dash=dash),
    )
volatility_fig.update_layout(
    title="Square-root impact against participation rate, by volatility",
    xaxis_title="Participation rate (% of volume)",
    yaxis_title="Signed buy impact (bps)",
    height=460,
    legend=dict(orientation="h", y=-0.23, x=0.5, xanchor="center"),
    margin=dict(b=100),
)
volatility_fig.update_yaxes(rangemode="tozero")
show_plotly_with_alt(
    volatility_fig,
    "Three square-root impact curves, one per volatility, identical in shape and stacked in "
    "proportion to the volatility that scales them.",
)
```

### The Exponent Sets Curvature

Now the coefficient is held and only the exponent varies. Every exponent shown is between zero
and one, so every curve is concave; the ones nearer to one are only mildly so, which is easy to
misread as convexity when a curve is close to straight.

```python
EXPONENT_COLORS = ml4t_palette(4, categorical=True)
exponent_specs = (
    ("exp=1.0 (linear)", power_impacts_linear, EXPONENT_COLORS[0], "solid"),
    ("exp=0.8", power_impacts_mild_concave, EXPONENT_COLORS[1], "dash"),
    ("exp=0.5", power_impacts_sqrt, EXPONENT_COLORS[2], "dot"),
    ("exp=0.3", power_impacts_concave, EXPONENT_COLORS[3], "dashdot"),
)
exponent_fig = go.Figure()
for label, impacts, color, dash in exponent_specs:
    exponent_fig.add_scatter(
        x=participation_rates * 100,
        y=[impact / price * 10000 for impact in impacts],
        mode="lines",
        name=label,
        line=dict(width=2.5, color=color, dash=dash),
    )
```

```python
exponent_fig.update_layout(
    title="Power-law impact against participation rate, by exponent",
    xaxis_title="Participation rate (% of volume)",
    yaxis_title="Signed buy impact (bps)",
    height=480,
    legend=dict(orientation="h", y=-0.23, x=0.5, xanchor="center"),
    margin=dict(b=105),
)
exponent_fig.update_yaxes(rangemode="tozero")
show_plotly_with_alt(
    exponent_fig,
    "Four power-law impact curves sharing a coefficient. The exponent-one curve is a straight "
    "line; smaller exponents bow further above it and rise more steeply near zero participation.",
)
```

**Finding**: Doubling volatility doubles square-root impact at every participation rate. Changing
a positive exponent below one changes concavity, but a coefficient comparison is still required
before one shape can be called more expensive than another.

## Part 4: Trade Sequence Simulation

A parent order is worked in equal child orders against a fixed daily volume. Because the models
have no memory, the notebook carries a stated fraction of each child's impact into the next
child's reference price itself - see `PERSISTENCE_FRACTION` above. That fraction is the
notebook's assumption and not something the library tracks, and it is applied identically to
every model so the comparison between them is unaffected by it.

```python
# Trade sequence simulation
total_shares = 100_000
price = 100.0
adv_shares = 1_000_000
n_child_orders = 10
shares_per_order = total_shares / n_child_orders

# Models to compare
simulation_models = {
    "NoImpact": NoImpact(),
    "LinearImpact": LinearImpact(coefficient=0.1),
    "SquareRootImpact": SquareRootImpact(
        coefficient=IMPACT_COEFFICIENT, volatility=SCENARIO_VOLATILITY
    ),
    "PowerLawImpact": PowerLawImpact(coefficient=0.1, exponent=0.6),
}
```

### Simulate Multi-Order Execution

```python
# Simulate execution
results = []
for model_name, model in simulation_models.items():
    exec_price = price
    total_cost = 0.0
    fill_prices = []

    for i in range(n_child_orders):
        # Each 10,000-share child is measured against the full 1,000,000-share ADV.
        impact = model.calculate(shares_per_order, exec_price, adv_shares, is_buy=True)

        # Fill at impacted price
        fill_price = exec_price + impact
        fill_prices.append(fill_price)

        # Cost for this fill
        order_cost = (fill_price - price) * shares_per_order
        total_cost += order_cost

        # Carry the stated scenario fraction into the next child order.
        exec_price = exec_price + impact * PERSISTENCE_FRACTION

        results.append(
            {
                "model": model_name,
                "order": i + 1,
                "fill_price": fill_price,
                "impact_per_share_usd": impact,
                "cumulative_cost": total_cost,
            }
        )

results_df = pl.DataFrame(results)
```

### First-Child Unit Oracle

The linear model is simple enough to check by hand, which makes it the right place to confirm
the units before trusting anything downstream. The first child's participation is its share count
divided by the daily volume, and the linear model's signed move must equal the coefficient times
that participation times the price. The cell below computes both sides and raises if they differ.

```python
first_linear = results_df.filter((pl.col("model") == "LinearImpact") & (pl.col("order") == 1)).row(
    0, named=True
)
first_child_participation = shares_per_order / adv_shares
first_child_impact_per_share = first_linear["impact_per_share_usd"]
expected_impact = LinearImpact().coefficient * first_child_participation * price
if not np.isclose(first_child_impact_per_share, expected_impact):
    raise ValueError("Library linear impact disagrees with coefficient x participation x price")
display(
    Markdown(
        f"**Unit check:** the first child is **{shares_per_order:,.0f} shares** against "
        f"**{adv_shares:,.0f}** of daily volume, a participation rate of "
        f"**{first_child_participation:.1%}**. The library returns "
        f"**${first_child_impact_per_share:.4f} per share**, which is "
        f"**{first_child_impact_per_share / price * 10_000:.1f} bps** of the "
        f"**${price:.0f}** reference price and matches the hand calculation exactly."
    )
)
```

### Sequence-Level Cost Summary

```python
# Sequence cost summary: 100,000 shares @ $100
sequence_summary_rows = []
for model_name in simulation_models.keys():
    model_data = results_df.filter(pl.col("model") == model_name)
    final_cost = model_data["cumulative_cost"][-1]
    sequence_summary_rows.append(
        {
            "model": model_name,
            "vwap_usd": price + final_cost / total_shares,
            "total_cost_usd": final_cost,
            "cost_bps": final_cost / (price * total_shares) * 10000,
        }
    )
sequence_summary = pl.DataFrame(sequence_summary_rows)
sequence_summary
```

**Finding**: The sequence summary holds the parent order, child schedule, and assumed persistence
fixed. Differences therefore come from the illustrative model parameterizations, not from the
library maintaining cumulative state.

```python
# Visualize execution paths
fig = go.Figure()
sequence_colors = dict(
    zip(
        ["NoImpact", "LinearImpact", "SquareRootImpact", "PowerLawImpact"],
        ml4t_palette(4, categorical=True),
        strict=True,
    )
)
sequence_dashes = {
    "NoImpact": "solid",
    "LinearImpact": "solid",
    "SquareRootImpact": "dash",
    "PowerLawImpact": "solid",
}
sequence_markers = {
    "NoImpact": (7, "circle"),
    "LinearImpact": (9, "circle"),
    "SquareRootImpact": (6, "diamond"),
    "PowerLawImpact": (7, "circle"),
}
for model_name in simulation_models:
    model_data = results_df.filter(pl.col("model") == model_name)
    marker_size, marker_symbol = sequence_markers[model_name]
    label = "SqRt = Linear (tie)" if model_name == "SquareRootImpact" else model_name
    fig.add_scatter(
        x=model_data["order"].to_list(),
        y=model_data["fill_price"].to_list(),
        mode="lines+markers",
        name=label,
        line=dict(width=2.5, color=sequence_colors[model_name], dash=sequence_dashes[model_name]),
        marker=dict(size=marker_size, symbol=marker_symbol),
    )

fig.add_hline(
    y=price,
    line_dash="dash",
    line_color=COLORS["neutral"],
    annotation_text="Decision price",
)

fig.update_layout(
    title="Fill price by child order, four impact models",
    xaxis_title="Child order",
    yaxis_title="Fill price (USD)",
    height=520,
    legend=dict(orientation="h", y=-0.23, x=0.5, xanchor="center"),
    margin=dict(b=105),
)
show_plotly_with_alt(
    fig,
    "Fill price against child-order number for four models, over a dashed horizontal line at "
    "the decision price. The no-impact path sits on that line for every child. The other three "
    "already start above it on the first child and climb steadily from there, the power-law path "
    "far above the other two, whose linear and square-root paths lie exactly on top of each "
    "other for the whole sequence.",
)
```

**Reading the chart**: Carrying half of each slice's impact into the next slice's reference price
is what makes the fills climb. Without it every child would fill at the same price, because the
models have no memory of the ones before.

The linear and square-root paths lie on top of each other, and that is worth pausing on rather
than passing over. Every child order here is the same fraction of volume, and at that particular
fraction the two parameterizations happen to charge the same amount: the linear coefficient times
the participation rate equals the square-root coefficient times the volatility times the square
root of the same rate. They agree at that one participation rate and nowhere else - the matching
used
deliberately in `05_almgren_chriss_optimal_execution` to compare two shapes without a level
difference confusing the comparison. A schedule that varied its participation would pull them
apart immediately.

## Part 5: A Real Universe to Price Orders Against

Every number so far has been a round scenario. The rest of the notebook prices orders against
five real ETFs - their traded prices, their volumes, and the volatility measured from their
returns - so that the participation rates and the resulting costs are ones a reader could
actually face.

The five are fixed by name and the window is the trailing year of the dataset. That is a
retrospective teaching sample, not a universe anyone selected in advance, and nothing in this
notebook is held out from anything.

```python
etf_data = load_etfs()

max_date = etf_data["timestamp"].max()
if max_date is None:
    raise ValueError("ETF dataset is empty; cannot parameterize impact models.")
min_date = max_date - timedelta(days=LOOKBACK_DAYS)
etf_filtered = etf_data.filter(
    (pl.col("symbol").is_in(UNIVERSE)) & (pl.col("timestamp") >= min_date)
)
missing_symbols = sorted(set(UNIVERSE) - set(etf_filtered["symbol"].unique().to_list()))
if missing_symbols:
    raise ValueError(f"Missing required ETF symbols: {missing_symbols}")
```

### Compute Descriptive Inputs in Polars

Three quantities per symbol drive everything that follows: the average traded price, the average
daily volume that a participation rate is a fraction of, and the standard deviation of daily
returns, which is the volatility the square-root model scales by. The high-low range is reported
alongside as a second view of how much each fund moves; it is not a spread and is not passed to
any model.

```python
etf_stats = (
    etf_filtered.sort(["symbol", "timestamp"])
    .with_columns(daily_return=pl.col("close").pct_change().over("symbol"))
    .group_by("symbol")
    .agg(
        observations=pl.len(),
        avg_price=pl.col("close").mean(),
        avg_volume=pl.col("volume").mean(),
        daily_volatility=pl.col("daily_return").std(),
        avg_high_low_range_bps=((pl.col("high") / pl.col("low") - 1) * 10_000).mean(),
    )
    .with_columns(annualized_volatility=pl.col("daily_volatility") * np.sqrt(252))
    .sort("symbol")
)
etf_stats
```

```python
display(
    Markdown(
        f"The sample covers **{etf_filtered['symbol'].n_unique()} ETFs** from "
        f"**{etf_filtered['timestamp'].min():%Y-%m-%d}** through "
        f"**{etf_filtered['timestamp'].max():%Y-%m-%d}**, and their daily volatilities span "
        f"**{etf_stats['daily_volatility'].min():.2%}** to "
        f"**{etf_stats['daily_volatility'].max():.2%}**."
    )
)
```

## Part 6: Backtest Integration Pattern

`FillExecutor` consumes the limit's `ExecutionResult` internally, uses its fillable quantity,
applies the signed impact once to the base price, applies slippage, and emits a `Fill` whose
`price` includes those adjustments. The limit result's `adjusted_price` and `impact_cost` remain
placeholders. The helper below exposes the intermediate limit-plus-impact arithmetic in an
`ExecutionResult`-shaped record for teaching; it is not the production executor's return value.

### Expose Limit-plus-Impact Arithmetic

```python
def compose_limit_and_impact(
    quantity: float,
    side: str,
    decision_price: float,
    volume: float,
    impact_model: MarketImpactModel,
    max_participation: float,
) -> ExecutionResult:
    """Expose one limit-plus-impact calculation in an inspection record."""
    if side not in {"buy", "sell"}:
        raise ValueError("side must be 'buy' or 'sell'")
    if volume <= 0:
        raise ValueError("volume must be positive")

    is_buy = side == "buy"
    limit_result = VolumeParticipationLimit(max_participation=max_participation).calculate(
        abs(quantity), volume, decision_price
    )
    fillable = limit_result.fillable_quantity
    remaining = limit_result.remaining_quantity

    impact = impact_model.calculate(
        quantity=fillable,
        price=decision_price,
        volume=volume,
        is_buy=is_buy,
    )
    adjusted_price = decision_price + impact
    return ExecutionResult(
        fillable_quantity=fillable,
        remaining_quantity=remaining,
        adjusted_price=adjusted_price,
        impact_cost=abs(impact) * fillable,
        participation_rate=limit_result.participation_rate,
    )
```

### An Order Book Built from the Real Panel

One order per ETF, alternating side, each sized as a stated multiple of that fund's own average
daily volume. Sizing by participation rather than by share count is what makes the five orders
comparable across funds whose volumes differ by orders of magnitude, and it puts two of them
above the participation cap on purpose so the partial-fill path is exercised.

```python
order_participations = [0.05, 0.10, 0.15, 0.08, 0.20]
trades_data = [
    (
        row["symbol"],
        row["avg_volume"] * participation,
        "buy" if index % 2 == 0 else "sell",
        row["avg_price"],
        row["avg_volume"],
    )
    for index, (row, participation) in enumerate(
        zip(etf_stats.iter_rows(named=True), order_participations, strict=True)
    )
]
pl.DataFrame(
    trades_data,
    schema=["symbol", "order_shares", "side", "decision_price", "daily_volume"],
    orient="row",
)
```

### Compare No-Impact vs Square-Root Impact

```python
integration_rows = []
for model_name, impact_model in [
    ("NoImpact", NoImpact()),
    (
        "SquareRootImpact",
        SquareRootImpact(coefficient=IMPACT_COEFFICIENT, volatility=SCENARIO_VOLATILITY),
    ),
]:
    for symbol, qty, side, decision_price, volume in trades_data:
        result = compose_limit_and_impact(
            qty, side, decision_price, volume, impact_model, MAX_PARTICIPATION
        )
        integration_rows.append(
            {
                "model": model_name,
                "symbol": symbol,
                "side": side,
                "decision_price": decision_price,
                "fillable_qty": result.fillable_quantity,
                "remaining_qty": result.remaining_quantity,
                "is_partial": result.is_partial,
                "adjusted_price": result.adjusted_price,
                "participation_rate": result.participation_rate,
                "total_impact_cost_usd": result.impact_cost,
            }
        )

integration_results = pl.DataFrame(integration_rows)
integration_results
```

### Verify Partial Fills and Adverse Direction

```python
square_root_results = integration_results.filter(pl.col("model") == "SquareRootImpact")
partial_count = square_root_results.filter(pl.col("is_partial")).height
sell_direction_ok = (
    square_root_results.filter(pl.col("side") == "sell")
    .select((pl.col("adjusted_price") < pl.col("decision_price")).all())
    .item()
)
if not sell_direction_ok:
    raise ValueError("A sell order's adjusted price must fall below its decision price")
display(
    Markdown(
        f"**Reading the table:** The {MAX_PARTICIPATION:.0%} cap leaves **{partial_count} of "
        f"{square_root_results.height} orders partially filled**, with the unfilled remainder "
        "carried in `remaining_qty` for a later bar. Under `NoImpact` the adjusted price equals "
        "the decision price on every row; under the square-root model it rises for buys and falls "
        "for sells, which is the sign convention working as intended."
    )
)
```

### Hold the Coefficient Fixed and Substitute Realized Volatility

```python
volatility_parameterized_models = {}

for row in etf_stats.iter_rows(named=True):
    symbol = row["symbol"]
    volatility_parameterized_models[symbol] = SquareRootImpact(
        coefficient=IMPACT_COEFFICIENT,
        volatility=row["daily_volatility"],
        adv_factor=1.0,
    )

parameterized_impact_rows = []
for symbol, model in volatility_parameterized_models.items():
    stats = etf_stats.filter(pl.col("symbol") == symbol).row(0, named=True)
    asset_price = stats["avg_price"]
    asset_volume = stats["avg_volume"]
    quantity = asset_volume * PARAMETERIZATION_PARTICIPATION
    impact = model.calculate(quantity, asset_price, asset_volume, is_buy=True)
    parameterized_impact_rows.append(
        {
            "symbol": symbol,
            "daily_volatility": stats["daily_volatility"],
            "impact_per_share_usd": impact,
            "impact_bps": impact / asset_price * 10_000,
        }
    )
parameterized_impact = pl.DataFrame(parameterized_impact_rows).sort("impact_bps")
```

### Model-Implied Impact Mirrors the Volatility Input

The check below recomputes the same quantity from the closed form rather than from the library
call, and raises if the two disagree. Reproducing a result a second way is what turns a library
call into something a reader can trust; the notebook is otherwise taking the library's word for
its own arithmetic.

```python
expected_bps = (
    IMPACT_COEFFICIENT
    * parameterized_impact["daily_volatility"]
    * np.sqrt(PARAMETERIZATION_PARTICIPATION)
    * 10_000
)
if not np.allclose(parameterized_impact["impact_bps"].to_numpy(), expected_bps.to_numpy()):
    raise ValueError("Library impact disagrees with the closed-form square-root expression")

impact_fig = go.Figure()
impact_fig.add_bar(
    x=parameterized_impact["impact_bps"].to_list(),
    y=parameterized_impact["symbol"].to_list(),
    orientation="h",
    marker_color=COLORS["blue"],
    text=[f"{value:.1f}" for value in parameterized_impact["impact_bps"]],
    textposition="outside",
)
impact_fig.update_layout(
    title=(
        "Model-implied buy impact per ETF at a fixed participation rate"
        "<br><sup>Trailing 365 days; 5% ADV; illustrative library-default c=0.5</sup>"
    ),
    xaxis_title="Model-implied buy impact (bps)",
    yaxis_title="ETF",
    height=450,
    margin=dict(r=55),
)
impact_fig.update_xaxes(range=[0, parameterized_impact["impact_bps"].max() * 1.18])
show_plotly_with_alt(
    impact_fig,
    "A horizontal bar per ETF of model-implied impact in basis points, sorted ascending and "
    "labelled with its value. The range across the five funds is roughly threefold.",
)
```



```python
lowest_impact = parameterized_impact.row(0, named=True)
highest_impact = parameterized_impact.row(-1, named=True)
display(
    Markdown(
        f"**Reading the chart:** With the coefficient held at {IMPACT_COEFFICIENT:.1f} and "
        f"participation fixed at {PARAMETERIZATION_PARTICIPATION:.0%}, modeled impact runs from "
        f"**{lowest_impact['impact_bps']:.1f} bps** for {lowest_impact['symbol']} to "
        f"**{highest_impact['impact_bps']:.1f} bps** for {highest_impact['symbol']}. Price and "
        "volume cancel at a fixed participation rate, so the ordering is the ordering of the five "
        "volatilities and nothing else. That is a property of the model, not a finding about "
        "these funds: it would hold whatever coefficient was chosen."
    )
)
```

## Part 7: Cross-Model Impact at Increasing Order Sizes

The reference price and volume are round numbers rather than one of the ETFs, because the point
of the table is the spread between the four models at a common set of participation rates, and a
round price makes the basis-point arithmetic checkable by eye. The four columns are four
assumptions, priced identically; the differences between them are what a reader should stress
test before choosing one for a backtest.

```python
# Summary comparison table
summary_data = []
reference_price = 100.0
reference_volume = 1_000_000.0

test_scenarios = [
    ("Small trade (1% ADV)", 0.01),
    ("Medium trade (5% ADV)", 0.05),
    ("Large trade (10% ADV)", 0.10),
    ("Very large (20% ADV)", 0.20),
]

test_models = {
    "NoImpact": NoImpact(),
    "Linear (0.1)": LinearImpact(coefficient=0.1),
    "SqRt (2% vol)": SquareRootImpact(
        coefficient=IMPACT_COEFFICIENT, volatility=SCENARIO_VOLATILITY
    ),
    "Power (0.6)": PowerLawImpact(coefficient=0.1, exponent=0.6),
}

for scenario, participation in test_scenarios:
    row = {"Scenario": scenario}
    qty = reference_volume * participation

    for model_name, model in test_models.items():
        impact = model.calculate(qty, reference_price, reference_volume, is_buy=True)
        impact_bps = impact / reference_price * 10_000
        row[model_name] = f"{impact_bps:.1f} bps"

    summary_data.append(row)

summary_df = pl.DataFrame(summary_data)
summary_df
```

**Finding**: The cross-model table keeps price, volume, and participation identical. Its level
differences therefore reflect model assumptions and parameter conventions, which should be
stress-tested rather than mistaken for observed execution costs.

## Key Takeaways

1. **Add a signed impact to the decision price exactly once.** The models return a positive move
   for buys and a negative one for sells, so the arithmetic is the same on both sides and a sign
   error shows up as a fill that is better than the decision price. Check that direction on a
   sell order before trusting any cost number built on it.

2. **An exponent and a coefficient are not independent, so a level comparison between two models
   means nothing until they are matched somewhere.** Two of the curves in Part 3 have the same
   square-root shape and sit at different levels purely because their coefficients follow
   different conventions. Pick a participation rate, match the models there, and compare what is
   left.

3. **Size orders as a share of volume, not as a share count.** Participation is what every impact
   model consumes and what a volume cap acts on, and it is the only quantity comparable across
   instruments whose volumes differ by orders of magnitude.

4. **A backtest must carry the unfilled remainder.** Capping participation means some orders fill
   partially, and a backtest that quietly fills the whole order anyway has assumed away the
   constraint it just imposed.

5. **Substituting a measured volatility into a model whose coefficient you assumed is not
   calibration.** The ETF section changes one input and holds the other fixed, so what it shows
   is the model's own volatility sensitivity. Estimating the coefficient needs execution records,
   as `03_market_impact_calibration` sets out.

6. **These models have no memory, so persistence is the caller's job.** Each call sees one order.
   A parent order worked in slices needs the permanent part of each slice's impact carried into
   the next slice's reference price, and this notebook does that itself rather than expecting the
   model to.

7. **Reproduce a library result a second way before building on it.** Part 5 recomputes the
   square-root impact from the closed form and raises if the two disagree, which is what makes
   the rest of the section evidence rather than trust.

### Known limitations

- The impact coefficients throughout are stated, including the library defaults. Every cost level
  in the notebook scales directly with them and none of them was fitted to executions.
- Impact is charged against average daily volume, so the intraday variation that
  `03_market_impact_calibration` measured is absent: the same order at the same participation
  costs the same at noon as at the open.
- The persistence assumption in Part 4 is a flat fraction applied uniformly. Real permanent
  impact depends on how much information the trading reveals, which varies by order and by name.
- The five ETFs are a fixed retrospective sample over the trailing year, chosen for a spread of
  volatilities rather than sampled from anything.
- `ExecutionResult.adjusted_price` and `impact_cost` are populated by the notebook's own helper.
  The production `FillExecutor` applies impact to the base price and emits a `Fill`; the fields
  on the limit's own result are placeholders.

**Next**: `07_ml4t_volume_participation` takes the participation cap used here and shows what it
does to a parent order across many bars; `10_gross_vs_net_performance` puts these models into a
full backtest.

**Book**: Chapter 18, Section 18.4.
![notebook output](figures/p1_1.png)
![notebook output](figures/p1_2.png)
![notebook output](figures/p1_3.png)
![notebook output](figures/p1_4.png)
![notebook output](figures/p1_5.png)