Viabilidade de dados para uma estratégia de ações do S&P 500 informada por opções

# S&P 500 Equity + Option Analytics: Feasibility Analysis


# S&P 500 Equity + Option Analytics: Feasibility Analysis

Before building a trading strategy it is worth asking whether the data can support one at all.
This notebook does that and nothing else: it fits no model and makes no forecast.

The strategy being checked is described in `config/setup.yaml`. Once a week it ranks S&P 500
constituents by a number read off their listed options, and buys the shares at the top of that
ranking. It holds shares, never options: the option market supplies the opinion, and the equity
market is where the position is taken. That file says which stocks it may hold, on which day of
the week it decides, what it assumes a trade costs, and how the history is divided between
designing the strategy and testing it. This notebook checks each of those assumptions against the
data and reports what it finds.

## Learning objectives

By the end of this notebook you will be able to:

- Read an implied volatility off an option surface summary, and find out what a dataset does
  with a calculation that failed before computing anything from the column
- Describe a universe of several hundred stocks by the property the strategy's cost model is
  sensitive to, rather than by counting its members
- Explain why the number of stocks carrying an option quote swings on a monthly cycle, from the
  expiration calendar rather than from anything the strategy does
- See why a charge quoted per share and a charge quoted as a percentage are different assumptions
  about friction, not two units for the same one
- Measure a move between the two prices a position is actually opened and closed at, then read
  off one chart what fraction of those moves are larger than the cost of trading them
- Measure how much of a ranking across stocks is left by the next decision date, computing the
  correlation between two orderings rather than inside a single stock's history
- Check that a walk-forward split fits the history available and leaves the test period unread

## Book reference

Chapter 6, Sections 6.2-6.6. This notebook reads the daily implied volatility surface summary,
daily share prices and `config/setup.yaml`, and writes nothing.

## Prerequisites

None beyond what the sections below define. A reader who has not traded an option will find the
instrument, its price and implied volatility explained where they are first used.

```python
"""S&P 500 Equity + Option Analytics - Feasibility Analysis."""

import datetime as dt

import matplotlib.dates as mdates
import matplotlib.pyplot as plt
import numpy as np
import polars as pl
import yaml
from IPython.display import display
from ml4t.diagnostic.splitters.calendar import TradingCalendar

from case_studies.utils.feasibility import (
    cross_sectional_persistence,
    exceedance_curve,
    fold_timeline,
)
from case_studies.utils.warning_policy import apply_notebook_warning_policy
from data import load_sp500_daily_bars, load_sp500_options_surface
from utils.cv_splits import generate_cv_splits
from utils.paths import get_case_study_dir
from utils.style import COLORS, FIGSIZE, add_message_title, show_with_alt

apply_notebook_warning_policy()
```

```python
CASE_STUDY_ID = "sp500_equity_option_analytics"
START_DATE = "2017-01-01"
END_DATE = "2021-12-31"
```

## Configuration

Everything the strategy assumes is declared in `config/setup.yaml`, and this notebook reads those
values rather than repeating them, so the two can never disagree. Four groups of settings matter
here, and each one decides something the sections below test.

**How the history is divided.** The sample runs from the start of 2017 to the end of 2021. The
final year is the *holdout*: a stretch of history that is not looked at while the strategy is
being designed, so that when it is finally evaluated there, the result is not a rehearsal of
choices already tuned on the same data. Everything computed in this notebook uses the earlier
part, called the development period, and `evaluation.holdout_start` is where the line falls.

**What the strategy ranks, and what it holds.** The ranking is built from one column of the
option surface summary, the thirty-day at-the-money implied volatility that Section A defines,
and `features.surface` records the contracts it is selected from. A stock is available to rank on
a date only where that number exists for it, which is not every date. From the ranking the
strategy holds the leading names, at most `backtest.sweep.top_k_grid` of them, so that count is
the floor the universe has to clear on a decision date and Section B.2 counts against it.

**What a trade is assumed to cost.** `costs.model` declares a percentage regime: one round-trip
charge in basis points, the same for every stock whatever it costs per share. A basis point is
one hundredth of one percent. Alongside it the sweep carries a per-share regime, where the charge
is a commission plus a half-spread quoted in cents and the percentage follows from the share
price. Section B.3 measures how far apart the two are across this universe.

**What is being predicted, and how it is evaluated.** `labels.primary` names the forward share
return the strategy is trying to rank stocks by, and `labels.horizons` gives every holding period
the design considers. Because a position opened on one decision date is still open on the next,
`labels.buffer` is the separation a walk-forward split has to leave between training and
validation, and Section D reads it.

```python
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
SETUP = yaml.safe_load((CASE_DIR / "config" / "setup.yaml").read_text())

HOLDOUT_START = str(SETUP["evaluation"]["holdout_start"])
HOLDOUT_END = str(SETUP["evaluation"]["holdout_end"])
PRIMARY_LABEL = SETUP["labels"]["primary"]
LABEL_BUFFER = SETUP["labels"]["buffer"]
BUFFER_SESSIONS = int(LABEL_BUFFER.rstrip("D"))
HORIZONS = sorted({int(h.rstrip("D")) for h in SETUP["labels"]["horizons"].values()})
PRIMARY_HORIZON = int(SETUP["labels"]["horizons"][PRIMARY_LABEL].rstrip("D"))
BREADTH_FLOOR = max(SETUP["backtest"]["sweep"]["top_k_grid"][PRIMARY_LABEL])
COST_BPS = SETUP["costs"]["round_trip_cost_bps"]
PER_SHARE = SETUP["costs"]["per_share"]
HALF_SPREADS = SETUP["backtest"]["sweep"]["cost_grid_half_spread_usd"]
ATM_IV = "iv_30_atm"
# Every series below is taken inside one security, as 02_labels and 03_financial_features do.
SECURITY = "sec_id"
DTE_LOW, DTE_HIGH = SETUP["features"]["surface"]["dte_buckets"]["30d"]
IV_LAG = int(SETUP["decision"]["iv_feature_lag"].split("_")[0])
IV_STALE = SETUP["features"]["windows"]["iv_forward_fill"]
CALENDAR = TradingCalendar(SETUP["evaluation"]["calendar"])
SESSIONS = CALENDAR.trading_days_between(START_DATE, HOLDOUT_START)
# Persistence is read in decision dates rather than sessions, so the ordering is compared against
# the next decision the strategy takes. Two months of them is long enough to see it flatten.
PERSISTENCE_WEEKS = 8

print(f"Sample: {START_DATE} to {END_DATE}")
print(f"  Development period, used everywhere below:  {START_DATE} to {HOLDOUT_START}")
print(f"  Holdout, not read by this notebook:         {HOLDOUT_START} to {HOLDOUT_END}")
print(
    f"Ranked on: {ATM_IV}, the at-the-money implied volatility of contracts maturing in "
    f"{DTE_LOW} to {DTE_HIGH} days, read {IV_LAG} session late and carried at most {IV_STALE}"
)
print(
    f"Universe: {SETUP['universe']['n_assets']} constituents under "
    f"{SETUP['universe']['eligibility_rule']}, of which the strategy holds up to {BREADTH_FLOOR}"
)
print(
    f"Assumed cost: {SETUP['costs']['model']} regime, {COST_BPS} bps a round trip, against a "
    f"per-share companion of {PER_SHARE} commission plus a half-spread from "
    f"{min(HALF_SPREADS)} to {max(HALF_SPREADS)} dollars"
)
print(
    f"Prediction: {PRIMARY_LABEL}, over horizons of {HORIZONS} sessions, decided at the "
    f"{SETUP['decision']['snapshot']} snapshot and executed at the "
    f"{SETUP['decision']['execution_delay']}"
)
print(
    f"Walk-forward: {SETUP['evaluation']['n_splits']} folds on the "
    f"{SETUP['evaluation']['calendar']} calendar, with {BUFFER_SESSIONS} sessions of separation "
    f"between training and validation"
)
```

## A. Orientation

### What the option market is being asked

An option is a contract on a stock. A **call** gives its holder the right to buy the stock at a
fixed price, the **strike**, up to a fixed date, the **expiration**; a **put** gives the right to
sell it on the same terms. The holder pays for that right, and the amount paid is the
**premium**. A contract whose strike sits nearest the current share price is **at the money**.

What a contract is worth depends above all on how far the market expects the stock to travel
before it expires. So the traded premium implies a movement, which is quoted the way
statisticians quote one: as an annualized standard deviation of returns. That number is the
**implied volatility**. It is not a price the exchange publishes; it is solved for, by asking
what volatility would make a pricing model return the premium the contract actually trades at.
Solving can fail, on a contract that has barely traded or whose quotes are crossed, and the
vendor records the failure rather than dropping the row. Section B.1 is where that is handled.

### The one number this strategy reads

A stock has hundreds of listed contracts at any moment, at many strikes and expirations, so a
strategy that wants one number per stock per day has to say which. `features.surface` says: take
the contracts maturing inside the declared day range, take the call and the put nearest the
money, and average their implied volatilities. `materialize_options.py` does that once for the
whole market and writes the daily surface summary this notebook loads. A stock therefore has a
value on a date only if it had a listed expiration inside that maturity window, which is a
property of the option listing calendar rather than of the stock, and Section B.2 is about what
that does to the universe.

### What the strategy does with it

It ranks the stocks by that number and buys the shares of the ones at the top, in equal weights,
holding until the next weekly decision. It never holds an option. Two consequences run through
everything below. Every cost is an ordinary equity trading cost, charged on the share price, not
an option cost charged on a premium. And there is nothing to hedge: a share position's exposure
to the share price is the position, not a side effect of it.

### The three questions this notebook asks

1. **Does the universe exist when the strategy acts?** Decisions are taken once a week, and a
   stock can be ranked only where the option market gave it a value, so enough stocks have to
   carry one on each of those dates to fill the book.
2. **Is a typical move in the share price worth more than it costs to trade?** The declared
   percentage cost charges every stock alike; a per-share charge does not, and the universe runs
   from a few dollars a share to well over a thousand.
3. **Is there enough history to evaluate this honestly?** Enough to split into training and
   validation periods more than once, with the holdout left untouched, and with a gap wide enough
   that a position opened in training does not resolve inside validation.

## B. Universe and cost feasibility

### B.1 Load the data and look at the universe

Two files are loaded: the surface summary, one row per stock and session, and daily share prices.

One property of the implied volatility decides how it may be used. Where the solve described in
Section A did not converge, the vendor records the failure as a negative number rather than as an
empty field, and a difference or ratio taken across tenors inherits it. A negative annualized
standard deviation is not a quantity, and it is worse than a missing one: a drop-missing leaves
it in place, and in a ranking it sorts that stock to the bottom on the strength of a solver
failure rather than a cheap option. `load_sp500_options_surface` returns those placeholders as
missing values, along with every surface measure derived from one, so that this notebook and
every later stage see the same thing. The count of stock-sessions with no value is reported
below; the assertion states the invariant the loader is there to give.

The other assertion is that the summary holds at most one row per stock and session, since a
second would double that stock's weight in every average taken across the panel.

The entity every series below is taken inside is the **security**, `sec_id`, not the ticker. A
ticker is reassigned after a merger or a spin-off, and it is also changed while the security
behind it stays the same, so a window bounded by the ticker either carries a dead company's
implied volatility into its successor or cuts a live company's history in half at a rename.
`02_labels` and `03_financial_features` both bound every window by `sec_id` for that reason, and
this notebook measures what they will build. The two counts printed below are what the
distinction costs: the tickers, and the securities behind them.

```python
surface = load_sp500_options_surface(start_date=START_DATE, end_date=END_DATE)
bars = load_sp500_daily_bars(start_date=START_DATE, end_date=END_DATE)
development = pl.col("timestamp") < pl.lit(HOLDOUT_START).str.to_date()
quotes = bars.filter(development).select(
    "timestamp", "symbol", "sec_id", "open", "close", "adj_factor"
)
summary = surface.filter(development)

assert not summary.select(pl.struct("symbol", "timestamp").is_duplicated().any()).item(), (
    "a stock carries more than one surface row on a session"
)
assert summary.select((pl.col(ATM_IV).drop_nulls() > 0).all()).item(), (
    "an implied volatility that is present is not positive"
)

solved = summary.select("timestamp", "symbol", ATM_IV).drop_nulls(ATM_IV)
panel = solved.join(quotes, ["timestamp", "symbol"]).sort([SECURITY, "timestamp"])
print(
    f"{panel['symbol'].n_unique()} tickers standing for {panel[SECURITY].n_unique()} securities, "
    f"{len(panel):,} stock-sessions over "
    f"{panel['timestamp'].n_unique():,} sessions, {panel['timestamp'].min()} to "
    f"{panel['timestamp'].max()}\n"
    f"{summary[ATM_IV].null_count():,} of {len(summary):,} surface rows carry no implied "
    f"volatility, whether unquoted or unsolved"
)
```

Six hundred tickers are a list, not a description. The property to sort them by here is the share
price, because Section B.3's subject is that a charge quoted in cents per share and a charge
quoted as a percentage of the trade rank this universe differently, and the price is what
separates them. The table below cuts the universe at fifty, one hundred and two hundred and fifty
dollars a share and reports what each group looks like.

Three things to read off it. The price range across the whole universe spans nearly three orders
of magnitude, which is what makes a fixed charge per share a very different assumption at one end
than at the other. Implied volatility is not ordered by price: the cheapest group and the dearest
group are the two most volatile, so a cost model built on the price is not quietly selecting on
how much the stock moves. And no group is quotable on more than about half the sessions in the
period, which is the listing cycle Section B.2 takes up.

```python
PRICE_CUTS = [50, 100, 250]
band = pl.col("price")
label = pl.when(band < PRICE_CUTS[0]).then(pl.lit(f"under {PRICE_CUTS[0]}"))
for low, high in zip(PRICE_CUTS, PRICE_CUTS[1:], strict=False):
    label = label.when(band < high).then(pl.lit(f"{low} to {high}"))
label = label.otherwise(pl.lit(f"{PRICE_CUTS[-1]} and over"))

n_sessions = quotes["timestamp"].n_unique()
by_security = panel.group_by(SECURITY).agg(
    pl.col("close").median().alias("price"),
    (pl.col(ATM_IV) * 100).median().alias("implied_vol"),
    pl.len().alias("stock_sessions"),
)
groups = (
    by_security.with_columns(label.alias("share_price"))
    .group_by("share_price")
    .agg(
        pl.len().alias("stocks"),
        pl.col("price").median().round(0).alias("median_price"),
        pl.col("implied_vol").median().round(1).alias("median_implied_vol_pct"),
        (pl.col("stock_sessions").median() / n_sessions * 100).round(0).alias("pct_of_sessions"),
        pl.col("stock_sessions").sum().alias("stock_sessions"),
    )
    .sort("median_price")
)
with pl.Config(tbl_rows=groups.height, tbl_cols=groups.width):
    display(groups)
print(
    f"Share price across the universe {by_security['price'].min():.2f} to "
    f"{by_security['price'].max():.2f}, over {n_sessions:,} sessions"
)
```

### B.2 How many stocks can be ranked when the strategy decides

`setup.yaml::decision.cadence` decides at the last close of each week, so that is where the
universe has to exist. A stock counts on that date under the delay the design assumes: the
surface summary describes the close it is stamped at and is not read until the next session, and
`03_financial_features` carries a value forward over a short gap, so the count below applies that
lag and that tolerance first. Both are declared in sessions, and the summary skips the sessions a
stock had no surface point on, so the panel is placed on the trading calendar before either is
applied. On a row-counted lag, a month-old value reads as one session late.

The count swings, and the reason is the option listing calendar rather than anything about the
stocks. Every optionable US stock has **monthly** expirations, one in the third week of each
month, on the third Friday or on the session before it when that Friday is a market holiday.
Liquid names additionally have **weekly** expirations on most other Fridays. A stock enters the
summary on a session only if one of its expirations falls inside the maturity window, so on the
sessions when a monthly expiration is inside that window the whole universe qualifies, and on
the rest only the names carrying weeklies do. The figure separates the decision dates on that
basis, asking the question of the session the value was stamped on rather than of the decision
date, because those are one session apart and the answer differs between them.

The maturity window is eleven calendar days wide and a monthly expiration arrives once a month,
so at most one can be inside the window at a time, which the assertion states.

```python
sessions = bars["timestamp"].unique().sort().to_list()
last_session = max(s for s in sessions if s < dt.date.fromisoformat(HOLDOUT_START))


def _monthly_expirations(first: dt.date, last: dt.date) -> set[dt.date]:
    """Third-week expirations, moved back to the previous session on a market holiday."""
    trading = set(sessions)
    dates, day = set(), first
    while day <= last:
        if day.weekday() == 4 and 15 <= day.day <= 21:
            expiry = day
            while expiry not in trading and expiry <= last_session:
                expiry -= dt.timedelta(days=1)
            dates.add(expiry)
        day += dt.timedelta(days=1)
    return dates


monthlies = _monthly_expirations(
    dt.date.fromisoformat(START_DATE), last_session + dt.timedelta(days=DTE_HIGH)
)
in_window = {
    day
    for day in sessions
    if any(day + dt.timedelta(days=k) in monthlies for k in range(DTE_LOW, DTE_HIGH + 1))
}
# The value read at a decision was stamped IV_LAG sessions earlier, so the listing question is
# about that session rather than about the decision date, and the flag is carried forward with it.
listings = (
    pl.DataFrame({"timestamp": sessions})
    .sort("timestamp")
    .with_columns(pl.col("timestamp").is_in(in_window).shift(IV_LAG).alias("monthly_in_window"))
    .drop_nulls("monthly_in_window")
)
assert len({(d.year, d.month) for d in monthlies}) == len(monthlies), (
    "two monthly expirations fall in the same month"
)
```

The summary carries a row only where a stock had a surface point, so it is not a complete grid:
a stock's rows skip whole weeks. Both `iv_feature_lag` and `iv_forward_fill` are declared in
sessions, so the panel is placed on the session grid before either is applied. Shifting the rows
as they come would read "one session late" off a value up to a month old, and would spend the
five-session tolerance on five rows that can span far more than five sessions.

The grid is the sessions each **security** traded, taken from the share bars, which is the grid
`03_financial_features::on_session_grid` reindexes the surface onto. Building it that way is what
keeps the lag and the fill from crossing a ticker changing hands: they are applied inside
`sec_id`, so a dead security's last quoted volatility is not carried into whichever company
picked the ticker up.

```python
grid = quotes.select("timestamp", "symbol", SECURITY).join(
    summary.select("timestamp", "symbol", ATM_IV), on=["timestamp", "symbol"], how="left"
)
rankable = (
    grid.sort([SECURITY, "timestamp"])
    .with_columns(pl.col(ATM_IV).shift(IV_LAG).forward_fill(limit=IV_STALE).over(SECURITY))
    .drop_nulls(ATM_IV)
)
decisions = rankable.group_by(pl.col("timestamp").dt.truncate("1w")).agg(
    pl.col("timestamp").max().alias("decision_date")
)
entries = rankable.join(decisions, left_on="timestamp", right_on="decision_date").join(
    listings, on="timestamp"
)
# Whether a monthly expiration is in the maturity window is a property of the date rather than of
# the stock, so the two groups are two sets of decision dates, not two parts of one count.
breadth = (
    entries.group_by("timestamp")
    .agg(
        pl.col(SECURITY).n_unique().alias("n_stocks"),
        pl.col("monthly_in_window").first(),
    )
    .sort("timestamp")
)

fig, ax = plt.subplots(figsize=FIGSIZE["single"])
ax.plot(breadth["timestamp"], breadth["n_stocks"], color=COLORS["silver_muted"], lw=0.6, zorder=1)
for monthly, color, name in (
    (False, COLORS["blue"], "only weekly expirations in the window"),
    (True, COLORS["amber"], "a monthly expiration in the window"),
):
    group = breadth.filter(pl.col("monthly_in_window") == monthly)
    ax.scatter(group["timestamp"], group["n_stocks"], s=9, color=color, label=name, zorder=2)
ax.axhline(
    BREADTH_FLOOR, color=COLORS["copper"], ls="--", lw=1.5, label="stocks the book has to fill"
)
ax.set_ylim(0, SETUP["universe"]["n_assets"])
ax.set_ylabel("Stocks that can be ranked")
ax.xaxis.set_major_locator(mdates.YearLocator())
ax.xaxis.set_major_formatter(mdates.DateFormatter("%Y"))
ax.legend(frameon=False, fontsize=8, ncol=2, loc="upper center", bbox_to_anchor=(0.5, -0.14))
add_message_title(
    ax,
    "The universe doubles on the dates a monthly expiration is in reach",
    subtitle="Stocks with a lagged implied volatility and a share price, at each weekly decision",
)
show_with_alt(
    fig,
    "Scatter of the number of stocks that can be ranked at each weekly decision date from 2017 "
    "to the end of 2020, on an axis running to the declared universe of 633. The points fall "
    "into two separate bands with nothing between them. Dates with a monthly expiration inside "
    "the maturity window sit in a flat band just above five hundred stocks for the whole sample. "
    "Dates carrying only weekly expirations sit in a lower band that starts near 210 in 2017, "
    "steps up to about 260 early in 2018 and drifts back to about 250 by the end of 2020. A thin "
    "grey line joins the points in date order and runs vertically between the two bands wherever "
    "consecutive decision dates fall on opposite sides of the monthly cycle. The dashed rule "
    "marking the twenty positions the book has to fill lies along the bottom of the chart, far "
    "below every point in either band.",
)
```

Breadth never approaches the floor, so it is not what limits this strategy. What the swing
changes is *which* stocks can be ranked. On the narrow dates the cross-section is about half the
universe and the half that disappears is the less liquid half, so a statistic averaged over
decision dates is an average over two different universes, and the carry-forward tolerance does
not close the gap because the narrow stretches run longer than it does.

### B.3 What a trade costs, and what a move is worth

`setup.yaml::costs.model` declares a percentage regime: one round trip in basis points, charged
alike on every stock. The companion the sweep keeps is a per-share regime, where the charge is a
commission plus a half-spread quoted in cents, so the percentage it works out to depends on the
share price, and a cent is a far larger fraction of a cheap share than of an expensive one.

The curve below states the per-share regime in the percentage regime's units, one point per
stock, ordered by share price. The band spans the declared half-spread grid from its cheapest
rung, which charges commission alone, to its dearest. The flat line is what the percentage regime
charges the same stock. Nominal quoted prices are used, not prices adjusted for splits and
dividends: a commission is charged on the price a share actually traded at.

```python
prices = (
    quotes.join(panel.select(SECURITY).unique(), on=SECURITY)
    .with_columns(
        (pl.col("open") * pl.col("adj_factor")).alias("adj_open"),
        (pl.col("close") * pl.col("adj_factor")).alias("adjusted"),
    )
    .sort([SECURITY, "timestamp"])
)
cost = (
    prices.group_by(SECURITY)
    .agg(pl.col("close").median().alias("price"))
    .with_columns(
        (2 * (min(HALF_SPREADS) + PER_SHARE) / pl.col("price") * 1e4).alias("cheapest"),
        (2 * (max(HALF_SPREADS) + PER_SHARE) / pl.col("price") * 1e4).alias("dearest"),
    )
    .sort("price", descending=True)
)

fig, ax = plt.subplots(figsize=FIGSIZE["single"])
rank = np.arange(len(cost))
ax.fill_between(rank, cost["cheapest"], cost["dearest"], color=COLORS["blue"], alpha=0.25)
ax.plot(rank, cost["cheapest"], color=COLORS["blue"], lw=1.2, label="commission only")
ax.plot(rank, cost["dearest"], color=COLORS["amber"], lw=1.2, label="widest declared half-spread")
ax.axhline(COST_BPS, color=COLORS["copper"], ls="--", lw=1.5, label="declared percentage cost")
ax.set_yscale("log")
ax.set_xlabel("Stocks, ordered from the highest share price to the lowest")
ax.set_ylabel("Round-trip cost (bps of the share price, log scale)")
ax.legend(frameon=False, fontsize=8, loc="upper left")
add_message_title(
    ax,
    "A per-share charge costs low-priced stocks orders of magnitude more",
    subtitle="Declared commission and half-spread over each stock's median quoted price",
)
show_with_alt(
    fig,
    "Two rising curves on a logarithmic vertical axis, one stock per horizontal position, "
    "ordered from the highest share price on the left to the lowest on the right, with the band "
    "between them shaded. The lower curve is the commission-only cost: it starts near four "
    "hundredths of a basis point, climbs steeply over the first few stocks, then rises gently "
    "across the middle of the universe and turns up sharply among the last twenty or so, ending "
    "around thirty. The upper curve is the widest declared half-spread and has the same shape "
    "about a factor of thirty higher, starting near one basis point and ending just above six "
    "hundred. The dashed horizontal rule at the declared thirteen basis points is crossed by the "
    "upper curve about a sixth of the way along and by the lower curve only among the last few "
    "stocks, so the shaded band straddles the rule across most of the universe.",
)
```

The band crosses the flat line rather than sitting to one side of it, so no single figure in
cents reproduces the percentage regime across a universe this wide in price. That is why the
percentage regime is the headline and the per-share regime is carried as a companion: the two are
different assumptions about what friction is, and Chapter 18 sweeps both rather than choosing.

Against that cost sits the move the strategy would have been holding through. It is measured
between the two prices a position is actually opened and closed at, which `decision` declares and
`02_labels` builds every label on: in at the adjusted open of the session after the decision,
out at the adjusted close $h$ sessions later. The decision close is one session earlier than the
first price the strategy can reach, so a move quoted from it contains the overnight gap between
reading the data and acting on it, which is a move no position was open for.

Three things bound what is counted. Prices are adjusted for splits and dividends, so a corporate
action does not enter as a move. Every window is taken inside one security identity, so a ticker
changing hands between two companies does not either. And the window has to be complete on the
market's own session grid rather than on the security's rows, because a name that stops trading
for a fortnight has consecutive rows spanning a hole; $h$ rows would then not be $h$ sessions.
The windows are measured over the stock-sessions the strategy could have ranked, which is the
population the question is about: a move in a stock the strategy could not have held that day is
not an opportunity it passed up. The last few development sessions carry no window, because
theirs would resolve inside the holdout.

The chart is an **exceedance curve**, and it reads from the right: for each magnitude on the
horizontal axis it gives the fraction of moves at least that large. Where it crosses the cost
line is the fraction of moves bigger than what it costs to trade them. One thing it is not: it
ignores direction entirely, so it is the distribution of how far the share travels, not the
return the strategy would earn. Whether the ranking picks the stocks that travel upward is the
question Chapter 7 onward asks.

```python
# The session grid is the market's, so `h` counts trading sessions rather than surviving rows.
grid_sessions = pl.DataFrame({"timestamp": sessions}).sort("timestamp").with_row_index("session")
held = prices.join(grid_sessions, on="timestamp").sort([SECURITY, "session"])
entry = pl.col("adj_open").shift(-1).over(SECURITY)
returns = (
    held.with_columns(
        *[
            pl.when(
                (pl.col("session").shift(-h).over(SECURITY) - pl.col("session") == h) & (entry > 0)
            )
            .then((pl.col("adjusted").shift(-h).over(SECURITY) / entry - 1).abs() * 1e4)
            .alias(f"h{h}")
            for h in HORIZONS
        ]
    )
    .join(rankable.select("timestamp", SECURITY), ["timestamp", SECURITY])
    .sort([SECURITY, "session"])
)

fig, ax = plt.subplots(figsize=FIGSIZE["single"])
for h, color in zip(HORIZONS, (COLORS["blue"], COLORS["amber"]), strict=True):
    magnitude, fraction = exceedance_curve(returns[f"h{h}"].drop_nulls().to_numpy())
    ax.plot(magnitude, fraction, color=color, lw=1.6, label=f"{h}-session move")
ax.axvline(COST_BPS, color=COLORS["copper"], ls="--", lw=1.5, label="declared round trip")
ax.set_xscale("log")
ax.set_xlim(1, 20_000)
ax.set_xlabel("Absolute move (bps, log scale)")
ax.set_ylabel("Fraction of moves at least this large")
ax.legend(frameon=False, fontsize=8, loc="lower left")
add_message_title(
    ax,
    "Moves at both horizons clear the declared round trip almost always",
    subtitle="Exceedance of absolute returns on the sessions that can be ranked",
)
show_with_alt(
    fig,
    "Two exceedance curves on a logarithmic horizontal axis of absolute move size in basis "
    "points, the axis running from one to twenty thousand. Each gives the fraction of moves at "
    "least as large as the magnitude beneath it, so both start flat at one on the left, fall "
    "away through the hundreds, and reach zero a little past a thousand. The ten-session curve "
    "lies above the five-session one throughout, meaning larger moves at every magnitude. The "
    "dashed vertical rule at the declared thirteen basis point round trip sits far to the left "
    "of where either curve begins to fall, so both are still close to one where they cross it.",
)
```

### B.4 How much of the ranking is left at the next decision

Re-ranking every week is worth the trading it causes only if what the data says on one decision
date still says something on the next. `mapping.entry_logic` acts on a stock's *place* in the
cross-section rather than on its implied volatility as a level, and the two are not the same
thing: every stock's volatility can rise together while the ordering among them is untouched.
So the quantity to measure is the ordering.

The measurement builds the ranking on one decision date, builds it again some number of decision
dates later, and correlates the two over the stocks present on both. Plotted against that number
of weeks, it shows how much of an ordering is still in place and how quickly the rest is
replaced.

It is computed this way, rather than by correlating each stock's own history against itself, for
a reason Section B.2 supplies. A correlation inside a stock needs that stock quoted without
interruption, and here more than two thirds of the universe is missing from the summary on the
narrow dates, so that form would answer the question for the always-quoting minority and discard
everyone the listing cycle actually affects. Comparing two orderings uses whichever stocks are
present on both dates and excludes nobody permanently. The shaded region shows how much the
result varies from one pair of dates to another, and the band around zero shows how large a
correlation could plausibly get between two unrelated orderings of this many stocks.

```python
persistence = cross_sectional_persistence(
    entries,
    time_col="timestamp",
    entity_col=SECURITY,
    value_col=ATM_IV,
    max_lags=PERSISTENCE_WEEKS,
)

fig, ax = plt.subplots(figsize=FIGSIZE["single"])
ax.axhspan(
    -persistence["band"][0],
    persistence["band"][0],
    color=COLORS["copper"],
    alpha=0.3,
    zorder=1,
    label="range expected from unrelated orderings",
)
ax.fill_between(
    persistence["lag"],
    persistence["rho_p10"],
    persistence["rho_p90"],
    color=COLORS["blue"],
    alpha=0.15,
    zorder=2,
    label="10th to 90th percentile across pairs of dates",
)
ax.plot(
    persistence["lag"],
    persistence["rho"],
    color=COLORS["blue"],
    lw=1.6,
    marker="o",
    ms=4,
    zorder=3,
    label="average across pairs of dates",
)
ax.set_ylim(0, 1.02)
ax.set_xticks(persistence["lag"].to_list())
ax.set_xlabel("Decision dates between the two rankings")
ax.set_ylabel("Correlation between the two orderings")
ax.legend(frameon=False, fontsize=8, ncol=2, loc="upper center", bbox_to_anchor=(0.5, -0.16))
add_message_title(
    ax,
    "Most of the ranking is still there a week later",
    subtitle="Correlation of the implied volatility ordering between decision dates",
)
show_with_alt(
    fig,
    "Line with markers showing the correlation between two implied volatility orderings against "
    "the number of decision dates separating them, from one to eight, on a vertical axis from "
    "zero to one. It starts a little above nine tenths at one week, falls fastest over the first "
    "four weeks, and then flattens near three quarters for the rest of the axis. A shaded band "
    "gives the tenth to ninetieth percentile across pairs of dates; it is barely visible at one "
    "week and widens with the lag, spanning roughly 0.68 to 0.83 by eight. A second shaded band "
    "along the bottom of the axis, reaching about 0.11, is the range expected between unrelated "
    "orderings of this many stocks, and the curve stays far above it at every lag.",
)
```

### B.5 Move size against cost

Three numbers summarise what B.3 and B.4 drew. The first is the median absolute move over the
primary horizon, as a multiple of the declared round trip, which says how much larger a typical
move is than a typical cost. The second is the fraction of moves larger than that round trip,
which is where the exceedance curve crosses the cost line. The third is how much of the ordering
is left one decision date later.

None of them says the strategy earns anything. A move counts the same whether the share rises or
falls, and a long position is helped by one and hurt by the other. What they rule out is the case
where the design fails immediately, because a typical move is smaller than the cost of taking the
position.

```python
primary = f"h{PRIMARY_HORIZON}"
median_move, above_cost = (
    returns.drop_nulls(primary)
    .select(pl.col(primary).median(), (pl.col(primary) > COST_BPS).mean().alias("share"))
    .row(0)
)
one_week = persistence.filter(pl.col("lag") == 1)["rho"][0]
print(
    f"Median {PRIMARY_HORIZON}-session move {median_move:.0f} bps, {median_move / COST_BPS:.0f}x "
    f"the declared {COST_BPS} bps round trip, and {above_cost:.3f} of moves are larger than it\n"
    f"Priced per share instead, the same round trip runs from {cost['cheapest'].min():.2f} bps on "
    f"the most expensive stock to {cost['dearest'].max():.0f} bps on the cheapest\n"
    f"The ranking correlates {one_week:.2f} with itself one decision date later, "
    f"{persistence['rho'][-1]:.2f} after {PERSISTENCE_WEEKS}"
)
```

The median absolute five-session move is 209 bps, sixteen times the declared 13 bps round trip,
and 0.965 of moves exceed it. Priced per share instead, the same round trip runs from 0.04 bps on
the most expensive stock to 619 bps on the cheapest. The implied volatility ordering correlates
0.92 with itself one decision date later and 0.75 after eight.

## C. Design decisions

The sections above are evidence. This section is where that evidence meets the choices recorded
in `setup.yaml`, and says what each one rests on.

### C.1 When to decide, and when to trade

`setup.yaml::decision.snapshot` reads the data at the Friday close and
`decision.execution_delay` fills at the following Monday open. Unlike the option-selling case
study alongside this one, this strategy trades shares, and shares have an opening auction, so the
first price a decision taken at Friday's close can reach is Monday's open rather than another
close. `decision.iv_feature_lag` is the other half of the same argument: a surface summary
describes the close it is stamped at and is not published before it, so the ranking is built from
the previous session's value and cannot read a number that did not exist when the decision was
taken.

Section B.4 supports acting weekly rather than daily. Most of the ordering is still there a week
later, so a daily rebalance would pay the spread again for a cross-section that has barely moved;
but enough of it is gone within a few weeks that a monthly rebalance would be acting on a stale
ranking.

### C.2 What would send this design back

A feasibility study is only useful if some result would have stopped it. Three would, and each is
measured where its evidence exists rather than here.

The one this notebook could have produced is a cost failure: if a typical move in the share price
were no larger than what a round trip costs, no ranking would repair that. Section B.5 is that
measurement, and Chapter 18 repeats it against the trades a backtest actually places, sweeping
both cost regimes rather than assuming one.

The other two are outcomes of the strategy rather than properties of the data, and neither can be
settled at this stage. Chapter 7 asks whether the ranking has any relationship at all to the
returns that follow it.

The question specific to this case study is whether the option market's opinion is worth more
than the share price history that is free, and the stages ahead do not settle it. From Chapter 11
on, every model is fitted on the option-derived and the price-derived columns together, so what
those stages report is how far each column carries a model that was given both, never what the
same model would have done on the price columns alone. Separating the two takes a second fit with
the option columns withheld, and no notebook here runs one;
[`13_model_analysis`](13_model_analysis.ipynb), which compares the model families on this data,
records the same limit against its own importance ranking.

### C.3 What the strategy does with the ranking

`setup.yaml::mapping.class` holds the top of the ranking long and in equal weights. Long-only
keeps borrow and locate frictions out of an example whose subject is the signal, and equal weight
avoids stacking a second optimization on top of the one being tested. Chapter 17 sweeps the
allocators against that baseline.

## D. Walk-forward structure

### D.1 How much an evaluation has to spend

A panel of stock-sessions looks large, but a strategy that decides once a week does not get to
treat every row as an independent opportunity. What it spends is decision dates, and Section B.4
is why even those are worth less than their count suggests: consecutive weeks rank the same
stocks in nearly the same order, so they are not independent draws.

```python
print(
    f"Trading sessions {SESSIONS} | decision dates {len(breadth):,} | stocks per decision date "
    f"{breadth['n_stocks'].min()} to {breadth['n_stocks'].max()}, median "
    f"{breadth['n_stocks'].median():.0f}\n"
    f"Below the {BREADTH_FLOOR} the book needs on "
    f"{breadth.filter(pl.col('n_stocks') < BREADTH_FLOOR).height} of {len(breadth)} decision dates"
)
```

### D.2 The folds

A model is fitted on one stretch of history and evaluated on the stretch that follows it, then
the pair moves forward and the process repeats. Each fit-then-evaluate pair is a **fold**, and
evaluating this way is called **walk-forward**, because the split always runs in the direction
time does.

One detail decides whether the evaluation is honest. A position opened near the end of a training
block is labelled with a price from after that block ends, so validating on the session
immediately following training would score the model on an outcome it had partly seen. The fix is
to leave a gap between the two at least as wide as the outcome takes to arrive, and that gap is
called **purging**. Its width comes from `labels.buffer`, which `generate_cv_splits` counts in
trading sessions on the calendar `evaluation.calendar` names.

What the splitter is handed is the sessions this panel carries: the dates on which some stock had
both a solved implied volatility and a share price. That is the timeline the later stages build
their folds from as well, because they read the label file, which is written on the same dates
and stops a horizon short of the sample's end since a forward return needs sessions after it. The
splitter numbers folds chronologically, so fold 0 is the earlier one and fold 1 ends against the
holdout. The figure draws them earliest-first and labels each with that number, so the labels
count up alongside the dates; every later stage prints the same ones.
It draws the boundaries the splitter returned rather than recomputing them.

The purge gap is narrow next to training blocks measured in years, so counting it off the session
timeline is the only way to confirm it is as wide as the buffer declares. The three assertions
check that, the fold count, and that no validation window reaches the holdout.

```python
splits = generate_cv_splits(
    panel.select("timestamp"),
    case_study_id=CASE_STUDY_ID,
    label_buffer=LABEL_BUFFER,
    outcome_horizon=SETUP["labels"]["horizons"][PRIMARY_LABEL],
    date_col="timestamp",
)

grid = np.sort(panel["timestamp"].unique().to_numpy())
purge_gaps = {
    int(((grid > np.datetime64(s["train_end"])) & (grid < np.datetime64(s["val_start"]))).sum())
    for s in splits
}
last_val = max(split["val_end"] for split in splits)
assert len(splits) == SETUP["evaluation"]["n_splits"], "fold count differs from setup.yaml"
assert last_val < np.datetime64(HOLDOUT_START), "a fold reaches into the holdout"
assert purge_gaps == {BUFFER_SESSIONS}, "a purge gap is not the declared label buffer"
print(
    f"{len(splits)} folds | purge gap {min(purge_gaps)} sessions at every boundary, from "
    f"labels.buffer {LABEL_BUFFER} | last validation ends {last_val.date()}, before the holdout "
    f"opens on {HOLDOUT_START}"
)

fig, ax = plt.subplots(figsize=FIGSIZE["single"])
fold_timeline(ax, splits, holdout=(HOLDOUT_START, HOLDOUT_END))
ax.set_xlabel("Session")
add_message_title(
    ax,
    "Both folds roll forward and end before the holdout opens",
    subtitle="Boundaries as generate_cv_splits returned them; the purge gap separates each pair",
)
show_with_alt(
    fig,
    "Timeline with one row per walk-forward fold, drawn earliest first and labelled with the "
    "fold number the splitter assigned, so the labels count down. Each row carries a two-year "
    "training bar followed by a one-year validation bar. Fold 1 trains from the start of 2017 "
    "to the end of 2018 and validates through 2019; fold 0 trains from the start of 2018 to the "
    "end of 2019 and validates through 2020, each shifted one year later than the row above it. "
    "The ten sessions of purging between each pair are too narrow to separate the bars at this "
    "scale. A shaded holdout region covers 2021 and neither fold's bars reach it.",
)
```

## E. What this notebook hands on

Nothing. Membership follows the option listings themselves, as Section B.2 showed, so there is no
eligibility table for a later stage to filter on, and `03_financial_features` rebuilds the same
universe by calling the same two loaders over the same window.

## F. What the evidence says about each setting

One row per setting: the evidence behind it, and the condition under which a reader working on
their own data would choose differently.

| Setting | Evidence | Choose differently when |
|---|---|---|
| `universe.n_assets` | B.2, stocks that can be ranked at each decision date | the count on a decision date falls towards the number of positions the book has to fill |
| `decision.cadence` | B.3 the move distribution, B.4 how long the ordering lasts | the ordering is replaced inside one week, or moves stop clearing the round trip |
| `costs.model` | B.3, the per-share band against the flat percentage line | a measured per-stock spread arrives, which is what would make the per-share regime the headline rather than the companion |
| `evaluation.n_splits` | D.1 decision dates, D.2 the fold boundaries | the folds no longer fit the development period ahead of the holdout |

```python
thin = breadth.filter(pl.col("n_stocks") < BREADTH_FLOOR)
print(
    f"universe.n_assets {SETUP['universe']['n_assets']}, {panel[SECURITY].n_unique()} securities "
    f"quoted a solved implied volatility, {breadth['n_stocks'].min()} to "
    f"{breadth['n_stocks'].max()} of them per decision date, below the {BREADTH_FLOOR} the book "
    f"needs on {thin.height} of {len(breadth)} dates\n"
    f"decision.cadence {SETUP['decision']['cadence']} | decision.execution_delay "
    f"{SETUP['decision']['execution_delay']} | labels.primary {PRIMARY_LABEL} | labels.buffer "
    f"{LABEL_BUFFER}\n"
    f"costs.model {SETUP['costs']['model']} at {COST_BPS} bps; the same round trip charged per "
    f"share spans {cost['cheapest'].min():.2f} to {cost['dearest'].max():.0f} bps across the "
    f"universe\n"
    f"evaluation.n_splits {SETUP['evaluation']['n_splits']}, generated {len(splits)}, last "
    f"validation ends {last_val.date()}, holdout untouched"
)
```

599 securities, trading under 609 tickers, quote a solved implied volatility over the development
period, and between 213 and 503 of them can be ranked on a decision date, against a declared
universe of 633 and a book of 20 that no date comes close to failing to fill. The declared 13 bps
round trip is the same charge as anything from 0.04 to 619 bps if it is levied per share instead.
Two folds are generated, the last validation ending 2020-12-23, ten sessions of purging at every
boundary.

## Key takeaways

1. **A placeholder is not a measurement.** Where a vendor writes a sentinel value for a
   calculation that failed, a ranking will happily sort on it, because it is a number and a
   drop-missing leaves it in place; find out what a file does with failure before computing
   anything from the column, and normalise it at the loader so every reader gets one answer.
2. **Count the universe on the dates the strategy acts.** Where membership follows a listing
   calendar rather than a liquidity rule, the count cycles for reasons the strategy does not
   control, and an average over all sessions hides it.
3. **Express a cost in the unit it is actually charged in.** A charge per share and a charge as a
   percentage of the trade are different assumptions about friction, and they rank a universe
   spanning three orders of magnitude in price very differently.
4. **Measure the persistence of the thing the strategy acts on.** For a strategy that ranks, that
   is the ordering across entities and not the level inside one, and correlating two orderings
   keeps the entities that come and go, which correlating a single entity's history cannot.
5. **Adjust prices for splits and dividends, and stay inside one security identity**, so that a
   corporate action does not enter the return distribution as a move.

### Known limitations

- Four years of development period, of which the ordering's slow decay makes fewer independent
  observations than the count of decision dates suggests.
- Cost here is the declared commission and half-spread. Market impact needs a position size to be
  expressed against and enters at the cost stage.
- The moves in Section B.3 are unsigned and unconditional. Nothing here says the ranking selects
  the stocks that move upward, which is what Chapter 7 onward measures.
- The implied volatility is read from contracts inside a maturity window rather than at a fixed
  tenor, so a change of expiration moves the series alongside the volatility it is meant to
  measure.

**Next**: labels at the declared horizons, built on this development period.
![notebook output](figures/p1_1.png)
![notebook output](figures/p1_2.png)
![notebook output](figures/p1_3.png)
![notebook output](figures/p1_4.png)
![notebook output](figures/p1_5.png)