הערכת אסטרטגיית S&P 500 שנבחרה בשנת החזקה בצד

# S&P 500 Equity+Options: Holdout Backtest


# S&P 500 Equity+Options: Holdout Backtest

[`18_holdout_predictions`](18_holdout_predictions.ipynb) refit the selected
configuration and predicted 2021. This notebook runs the selected *strategy*
on those predictions - the same allocator, the same concentration, the same
risk overlay, the same costs - and reports what the portfolio would have
earned. It is the first number in this case study that was not produced on
data the selection could see.

**Learning objectives**

1. Hold a strategy specification fixed across two windows and change only the
   predictions it consumes.
2. Read a holdout result against the validation figure that selected it, and
   know what the gap between them does and does not establish.
3. Recognise which parts of a degradation are attributable to selection and
   which to the period.

**Book reference:** Chapter 20, Section 20.3.

**Prerequisites:** [`18_holdout_predictions`](18_holdout_predictions.ipynb).
Signals form after Friday's close and execute at the next available open. The
current-constituent universe retains survivorship bias, and it does so more
consequentially here than anywhere else in the case study: a 2021 window
evaluated on today's membership excludes every company that left the index,
so this number is optimistic by an amount this case study does not measure.

```python
"""S&P 500 Equity+Options: run the selected strategy on the holdout predictions."""

import json
import sqlite3

import matplotlib.pyplot as plt
import polars as pl

from case_studies.research import Study, open_selection_field
from case_studies.research.holdout import build_holdout_training_spec
from case_studies.utils.backtest_loaders import (
    get_backtest_config,
    load_backtest_prices_for,
    warmup_periods_for,
)
from case_studies.utils.backtest_presets import (
    clone_backtest_spec,
    ensure_backtest_spec,
    strategy_view,
)
from case_studies.utils.backtest_runner import run_backtest
from case_studies.utils.notebook_contracts import prediction_members_in_force
from case_studies.utils.registry import (
    backtest_hash_from_parts,
    model_source,
    read_predictions,
    resolve_best_backtest_runs,
)
from case_studies.utils.registry.specs import training_hash_from_spec
from case_studies.utils.strategy_analysis import resolve_solvent_carrier
from case_studies.utils.uncertainty import load_daily_returns_with_timestamp
from utils.paths import get_case_study_dir
from utils.style import COLORS, FIGSIZE, add_message_title, show_with_alt
```

```python
CASE_STUDY_ID = "sp500_equity_option_analytics"
LABEL = ""
MAX_SYMBOLS = 0
```

### What is asked for, and what it resolves to

The parameters above are the request; the values this notebook runs on are resolved here under
different names, so a resolved value cannot overwrite the request that produced it. An injected
parameter wins; otherwise the case study's own declaration does.

```python
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)
REGISTRY_DB = CASE_DIR / "run_log" / "registry.db"
bt_config = get_backtest_config(CASE_STUDY_ID)
# The label this stage runs under is a property of what the selection chose, so it is resolved
# below rather than here. `labels.primary` was the winner's label only by coincidence: the field
# spans every declared label, and the stages after the selection - price windows, schedule
# thinning, the return contract - have to be keyed to the label that won.
REQUESTED_LABEL = LABEL
print(f"Case study: {CASE_STUDY_ID}")
```

```python
_study = Study.at(CASE_DIR, case_study=CASE_STUDY_ID, entry_point="19_holdout_backtest")
CURRENT_MEMBERS, _population_notes = prediction_members_in_force(_study)
for _note in _population_notes:
    print(_note)
```

## 1. The strategy, read out of the frozen candidate set

[`16_risk_management`](16_risk_management.ipynb) freezes the field it ranked
over, and both this notebook and
[`18_holdout_predictions`](18_holdout_predictions.ipynb) read the selection out
of that set rather than re-deriving it. The set is immutable, so the two
resolve the same configuration without either being told what it is; handing
it over in a parameter or a file would add a way for them to disagree.

A holdout run that could be pointed at a strategy of the reader's choosing
would be a way to search the holdout. This one cannot be.

```python
CANDIDATE_SET_NAME = f"{CASE_STUDY_ID}:holdout-candidates"
# The frozen set where it exists, and the same construction applied live where it does not.
# 16_risk_management writes it by opening the study, which canonical regeneration refuses
# wherever the generated directories are not symlinks - a reader's clean clone and the test
# fixtures both - so the set is in the published run log and absent everywhere else. Reading it
# is the stronger path: it is immutable, so it cannot follow an upstream change. Re-deriving is
# the same rule applied live, and cannot notice that something moved. Which one ran is printed.
#
# Both paths go through `open_selection_field`, which is also what 16 freezes with. They used to
# be separate copies and they disagreed: the freeze spanned every declared label and this
# fallback spanned one, so which configuration a reader selected depended on whether their
# registry held a `candidate_sets` table.
FIELD = open_selection_field(
    _study,
    case_study=CASE_STUDY_ID,
    name=CANDIDATE_SET_NAME,
    prediction_hashes=CURRENT_MEMBERS,
    resolve_best_backtest_runs=resolve_best_backtest_runs,
)
CANDIDATES = FIELD.candidate_set
FIELD_HASHES = list(FIELD.members)
FIELD_NAME = f"frozen candidate set {CANDIDATES.hash}" if CANDIDATES is not None else "live ranking"
SELECTION_SOURCE = FIELD.source
# The field says which backtests may be chosen from; the resolver says which one is chosen,
# and it is handed the field rather than the whole registry. `SelectionField.selected` is
# `CandidateSet.best_validation_sharpe`, which ranks the stored Sharpe column with a hash
# tie-break and applies nothing else - no re-ranking onto the timestamps every candidate
# prices, no label or universe restriction, no refusal of a run whose equity reached zero.
# This case study's field holds a conformal allocator that sits out its warm-up and books it
# as returns of exactly zero, so the two rankings read two different samples: they name the
# same backtest on this registry and value it at 2.6087 stored against 2.6329 over the shared
# sessions. `20_strategy_analysis` resolves the same way, so all three notebooks describe one
# configuration.
CARRIER = resolve_solvent_carrier(CASE_STUDY_ID, admitted=frozenset(FIELD_HASHES))
SELECTED = _study.results.open(CARRIER["val_backtest_hash"])
print(
    f"{FIELD_NAME}: {len(FIELD_HASHES)} members, resolver picks {SELECTED.hash}, "
    f"stored-Sharpe pick {FIELD.selected.hash}"
)

# The label the stages after the selection run under is the winner's, not the case study's
# primary. An injected LABEL is a request to run a different one, and it has to agree with what
# was selected or the holdout backtest would be keyed to a contract the selection does not name.
HOLDOUT_LABEL = CARRIER["label"]
if REQUESTED_LABEL and REQUESTED_LABEL != HOLDOUT_LABEL:
    raise RuntimeError(
        f"LABEL={REQUESTED_LABEL!r} was requested but the selection carried forward is "
        f"{SELECTED.hash} on {HOLDOUT_LABEL!r}. Running the holdout backtest under another "
        "label's contract would report a different strategy from the one selected."
    )
print(f"Label carried by the selection: {HOLDOUT_LABEL}")
print(f"Selection read from the {SELECTION_SOURCE}")

_why = SELECTED.completeness()
if _why is not None:
    raise RuntimeError(
        f"the selected validation backtest {SELECTED.hash} is incomplete: {_why}. "
        f"It was chosen from the {SELECTION_SOURCE}."
    )
CARRIER_SPEC = SELECTED.spec()
CARRIER_STRATEGY = strategy_view(CARRIER_SPEC)
carrier_record = SELECTED.registry_record()
CARRIER_PREDICTION_HASH = carrier_record["prediction_hash"]

with sqlite3.connect(REGISTRY_DB) as db:
    _carrier_source = db.execute(
        "SELECT t.family, t.config_name FROM prediction_sets p "
        "JOIN training_runs t USING(training_hash) WHERE p.prediction_hash = ?",
        (CARRIER_PREDICTION_HASH,),
    ).fetchone()
print(
    f"{FIELD_NAME} with {len(FIELD_HASHES)} members selects "
    f"{model_source(*_carrier_source)} with "
    f"{(CARRIER_STRATEGY.get('allocation') or {}).get('method', 'equal_weight')} allocation, "
    f"top-{(CARRIER_STRATEGY.get('signal') or {}).get('top_k')}, "
    + (
        f"risk overlay {(CARRIER_STRATEGY.get('risk') or {}).get('name')}"
        if (CARRIER_STRATEGY.get("risk") or {}).get("name")
        else "no risk overlay"
    )
)
```

## 2. Which holdout prediction set belongs to this configuration

The holdout fit is a different training run from the validation fit by
construction - a different interval is a different computation - so the two
cannot be matched on a training hash, and matching on a family and
configuration name would accept a fit of the same model over the wrong window.

Rather than compare fields and hope the list is complete, this derives the
identity the selected configuration *should* have on the holdout, by the same
`build_holdout_training_spec` [`18_holdout_predictions`](18_holdout_predictions.ipynb)
fits, and requires a registered prediction under exactly that hash. The
derivation is deterministic and costs a dataset read rather than a fit, so the
two notebooks agree by construction: every boundary, the label buffer, the
feature floor, the fold identifier and the request metadata are all inside the
hash, and any of them differing makes it a different identity that this will
not accept.

```python
with sqlite3.connect(REGISTRY_DB) as db:
    carrier_training_hash = db.execute(
        "SELECT training_hash FROM prediction_sets WHERE prediction_hash = ?",
        (CARRIER_PREDICTION_HASH,),
    ).fetchone()[0]
    VALIDATION_SPEC = json.loads(
        db.execute(
            "SELECT spec_json FROM training_runs WHERE training_hash = ?",
            (carrier_training_hash,),
        ).fetchone()[0]
    )
    carrier_checkpoint = db.execute(
        "SELECT checkpoint_kind, checkpoint_value FROM prediction_sets WHERE prediction_hash = ?",
        (CARRIER_PREDICTION_HASH,),
    ).fetchone()
    holdout_rows = db.execute(
        "SELECT p.prediction_hash, p.training_hash, p.checkpoint_kind, p.checkpoint_value "
        "FROM prediction_sets p WHERE p.split = 'holdout'"
    ).fetchall()
```

```python
OBSERVATIONS = (
    pl.read_parquet(_study.root / "labels" / f"{VALIDATION_SPEC['label']}.parquet")
    .get_column("timestamp")
    .unique()
    .sort()
    .to_list()
)
EXPECTED_HOLDOUT_SPEC = build_holdout_training_spec(
    _study, VALIDATION_SPEC, timeline=OBSERVATIONS, case_study=CASE_STUDY_ID
)
EXPECTED_HOLDOUT_TRAINING = training_hash_from_spec(EXPECTED_HOLDOUT_SPEC)
_expected_fold = EXPECTED_HOLDOUT_SPEC["computation"]["cv"]["folds"][0]
print(
    f"The selected configuration's holdout fit is {EXPECTED_HOLDOUT_TRAINING}: "
    f"train {str(_expected_fold['train_start'])[:10]} to {str(_expected_fold['train_end'])[:10]}, "
    f"evaluate {str(_expected_fold['val_start'])[:10]} to {str(_expected_fold['val_end'])[:10]}"
)

matches = [row for row in holdout_rows if row[1] == EXPECTED_HOLDOUT_TRAINING]
if not matches:
    raise RuntimeError(
        f"no holdout prediction is registered under {EXPECTED_HOLDOUT_TRAINING}, the identity "
        f"this configuration's configuration derives. Run 18_holdout_predictions. The registry holds "
        + (
            ", ".join(f"{row[0]} (training {row[1]})" for row in holdout_rows)
            if holdout_rows
            else "no holdout predictions at all"
        )
    )
if len(matches) > 1:
    raise RuntimeError(
        f"{len(matches)} holdout predictions share training {EXPECTED_HOLDOUT_TRAINING}: "
        + ", ".join(row[0] for row in matches)
        + ". A holdout fit publishes exactly the selected checkpoint, so this registry carries "
        "rows from an older rule."
    )

HOLDOUT_PREDICTION_HASH, HOLDOUT_TRAINING_HASH, holdout_kind, holdout_value = matches[0]
if (holdout_kind, holdout_value) != carrier_checkpoint:
    raise RuntimeError(
        f"the holdout prediction is at {holdout_kind}={holdout_value} but the selection was made "
        f"on {carrier_checkpoint[0]}={carrier_checkpoint[1]}"
    )
print(
    f"Holdout prediction {HOLDOUT_PREDICTION_HASH} from training {HOLDOUT_TRAINING_HASH}, "
    f"the validation configuration {carrier_training_hash} refitted over the holdout interval "
    f"at {holdout_kind}={holdout_value}"
)
```

The two windows the two fits saw, side by side. The gap between the training
end and the holdout open is one label buffer, which is what stops the last
training label's five-day outcome from resolving inside the window being
judged.

```python
with sqlite3.connect(REGISTRY_DB) as db:
    holdout_spec = json.loads(
        db.execute(
            "SELECT spec_json FROM training_runs WHERE training_hash = ?",
            (HOLDOUT_TRAINING_HASH,),
        ).fetchone()[0]
    )
holdout_fold = holdout_spec["computation"]["cv"]["folds"][0]
validation_folds = VALIDATION_SPEC["computation"]["cv"]["folds"]

pl.DataFrame(
    [
        {
            "fit": "validation",
            "folds": len(validation_folds),
            "train from": min(str(f["train_start"])[:10] for f in validation_folds),
            "train to": max(str(f["train_end"])[:10] for f in validation_folds),
            "evaluated": f"{min(str(f['val_start'])[:10] for f in validation_folds)}"
            f" to {max(str(f['val_end'])[:10] for f in validation_folds)}",
        },
        {
            "fit": "holdout",
            "folds": 1,
            "train from": str(holdout_fold["train_start"])[:10],
            "train to": str(holdout_fold["train_end"])[:10],
            "evaluated": f"{str(holdout_fold['val_start'])[:10]}"
            f" to {str(holdout_fold['val_end'])[:10]}",
        },
    ]
)
```

## 3. Run the selected strategy on the holdout window

The specification is the selected configuration's own, cloned and re-pointed at the holdout
predictions. Nothing about the strategy is re-derived here: re-deriving it
would let a change anywhere upstream alter what the holdout evaluates without
the change being visible as a different selection.

Prices are loaded for the holdout window with the allocator's warmup prefix.
The covariance estimator this configuration uses needs history before its first
rebalance, and without the prefix it would fall back to an imputed warmup on
exactly the dates the result is read from.

```python
prices = load_backtest_prices_for(
    CASE_STUDY_ID,
    HOLDOUT_LABEL,
    split="holdout",
    warmup_periods=warmup_periods_for(CASE_STUDY_ID),
    max_symbols=MAX_SYMBOLS,
)
print(
    f"Price support: {len(prices):,} rows across {prices['symbol'].n_unique()} symbols, "
    f"{prices['timestamp'].min()} to {prices['timestamp'].max()}"
)
predictions = read_predictions(CASE_STUDY_ID, HOLDOUT_PREDICTION_HASH)
print(
    f"Predictions: {predictions.height:,} rows over {predictions['timestamp'].n_unique()} sessions"
)
```

```python
holdout_strategy_spec = clone_backtest_spec(
    ensure_backtest_spec(
        CASE_STUDY_ID,
        bt_config,
        CARRIER_SPEC,
        prices=prices,
        prediction_hash=HOLDOUT_PREDICTION_HASH,
        initial_cash=bt_config.initial_cash,
    )
)
HOLDOUT_BACKTEST_HASH = backtest_hash_from_parts(HOLDOUT_PREDICTION_HASH, holdout_strategy_spec)

# The whole backtest specification, not the strategy block. `strategy_view` returns signal,
# allocation and risk and stops there, so a check built on it would pass a holdout run at
# different commissions, slippage, fill timing or stop behaviour - and holding those fixed is
# most of what makes the holdout number comparable to the validation one. What legitimately
# differs between the two runs is the predictions consumed and the price panel sliced to, so
# only those are projected out.
BACKTEST_VARYING_PATHS = (
    ("_runtime_backtest_config",),
    ("input_identity",),
    ("backtest_config", "metadata", "prediction_hash"),
    ("backtest_config", "metadata", "preset_path"),
)


def _without(value, path: tuple[str, ...]):
    """``value`` with ``path`` removed, leaving every sibling in place."""
    if not isinstance(value, dict) or path[0] not in value:
        return value
    pruned = dict(value)
    if len(path) == 1:
        pruned.pop(path[0])
    else:
        pruned[path[0]] = _without(pruned[path[0]], path[1:])
    return pruned


def _comparable_backtest(spec: dict) -> dict:
    projected = spec
    for path in BACKTEST_VARYING_PATHS:
        projected = _without(projected, path)
    return projected


carried = _comparable_backtest(holdout_strategy_spec)
validation_carried = _comparable_backtest(CARRIER_SPEC)
if carried != validation_carried:
    moved = sorted(
        key
        for key in set(carried) | set(validation_carried)
        if carried.get(key) != validation_carried.get(key)
    )
    raise RuntimeError(f"the holdout strategy differs from the selected one in {moved}")
print(f"Holdout backtest identity {HOLDOUT_BACKTEST_HASH}")
```

```python
with sqlite3.connect(REGISTRY_DB) as db:
    already = db.execute(
        "SELECT stage FROM backtest_runs WHERE backtest_hash = ?", (HOLDOUT_BACKTEST_HASH,)
    ).fetchone()
if already is None:
    result = run_backtest(
        CASE_STUDY_ID,
        HOLDOUT_PREDICTION_HASH,
        holdout_strategy_spec,
        prices=prices,
        predictions=predictions,
        label=HOLDOUT_LABEL,
        register=True,
        initial_cash=bt_config.initial_cash,
        calendar=bt_config.calendar,
    )
    print(
        f"Ran: Sharpe={result.metrics['sharpe']:.3f}, "
        f"max drawdown={result.metrics['max_drawdown']:.1%}"
    )
else:
    print(f"Already registered under stage {already[0]!r}; read, not re-run")

with sqlite3.connect(REGISTRY_DB) as db:
    registered_stage = db.execute(
        "SELECT stage FROM backtest_runs WHERE backtest_hash = ?", (HOLDOUT_BACKTEST_HASH,)
    ).fetchone()
if registered_stage is None:
    raise RuntimeError("the holdout backtest did not register")
if registered_stage[0] != "holdout":
    raise RuntimeError(
        f"the holdout backtest registered under stage {registered_stage[0]!r}; the stage is "
        "inferred from the prediction's split, so this row is not reachable as holdout evidence"
    )
```

## 4. What the holdout says

The two rows below are the same strategy on two windows. Both are read from
`backtest_metrics`, so each is measured over the span its own run priced. The
validation row is therefore the registered run's own figure and not the number the
selection was decided on: the field here holds a conformal allocator that sits out
its warm-up and books it as returns of exactly zero, so the ranking re-measures every
candidate over the sessions they all price. That figure is printed above the table,
and on this registry the two differ - 2.6329 over the shared sessions against 2.6087
stored.

Either way the validation figure is optimistic by construction: it is the maximum of
a search, and the maximum of a search is a biased estimate of the thing searched over.
The holdout figure has no such bias from *this* case study's selection.

It carries a different one. The window is a single year, and 2021 was a
particular year - a broad advance in US equities - so a result this good in a
year like that is weaker evidence than the same result across a decade. One
window cannot separate a strategy that works from a strategy that suits the
window it was tested on, and nothing downstream of here can either.

```python
# The metric the configuration was actually selected on, stated before the table so the
# validation row below is read as what it is - the registered run over its own span.
print(
    f"selected on Sharpe={CARRIER['val_sharpe']:.4f} over the "
    f"{CARRIER['comparison_n_periods'] or 'full'} sessions every candidate prices"
)

with sqlite3.connect(REGISTRY_DB) as db:
    comparison = pl.read_database(
        """
        SELECT b.backtest_hash, b.stage, m.sharpe, m.total_return, m.max_drawdown,
               m.volatility, m.num_trades, m.n_periods,
               m.sharpe_ci95_lo, m.sharpe_ci95_hi
        FROM backtest_runs b JOIN backtest_metrics m USING(backtest_hash)
        WHERE b.backtest_hash IN (?, ?)
        """,
        connection=db,
        execute_options={"parameters": [HOLDOUT_BACKTEST_HASH, SELECTED.hash]},
    )
if comparison.height != 2:
    raise RuntimeError("one of the two backtests has no registered metrics")

comparison = comparison.with_columns(
    pl.when(pl.col("backtest_hash") == HOLDOUT_BACKTEST_HASH)
    .then(pl.lit("holdout (2021)"))
    .otherwise(pl.lit("validation (registered span)"))
    .alias("window")
).sort("window")
comparison.select(
    "window",
    "sharpe",
    "sharpe_ci95_lo",
    "sharpe_ci95_hi",
    "total_return",
    "max_drawdown",
    "volatility",
    "num_trades",
    "n_periods",
)
```

The Sharpe intervals are the part to read first, and what they support is
narrower than it looks.

Each interval describes uncertainty in *its own* estimate. The holdout's says
how much a single year of this strategy's weekly rebalances pins down its
Sharpe; where it spans zero, that year does not establish that the strategy
earns anything at all. The validation interval says the same for the
development window, and carries the additional problem that its point estimate
is the maximum of a search and is optimistic by an amount neither interval
measures.

What neither interval gives is a test of the *difference* between the two
windows. That would need an interval for the gap itself, and the two are
measured over disjoint periods on different numbers of observations, so it
cannot be read off by asking whether one point estimate falls inside the
other's interval. The cells below therefore report both intervals and the
arithmetic difference, and stop there. Attributing the difference - to
selection bias, to a different market, or to both - is not something one
holdout year can do, and no notebook downstream of this one can either.

```python
_holdout = comparison.filter(pl.col("window") == "holdout (2021)").row(0, named=True)
_validation = comparison.filter(pl.col("window") == "validation (registered span)").row(
    0, named=True
)
_spans_zero = _holdout["sharpe_ci95_lo"] <= 0.0 <= _holdout["sharpe_ci95_hi"]
print(
    f"Validation Sharpe {_validation['sharpe']:.3f} over the registered span "
    f"[{_validation['sharpe_ci95_lo']:.3f}, {_validation['sharpe_ci95_hi']:.3f}] "
    f"over {int(_validation['n_periods'])} sessions"
)
print(
    f"Holdout Sharpe {_holdout['sharpe']:.3f} "
    f"[{_holdout['sharpe_ci95_lo']:.3f}, {_holdout['sharpe_ci95_hi']:.3f}] "
    f"over {int(_holdout['n_periods'])} sessions"
)
print(f"Arithmetic difference {_holdout['sharpe'] - _validation['sharpe']:+.3f}, untested")
print(
    "The holdout interval "
    + (
        "spans zero, so this year does not establish that the strategy earns anything"
        if _spans_zero
        else "excludes zero, so this year places its Sharpe away from zero"
    )
)
```

The holdout equity path, drawn on its own axis. It is one year at weekly
rebalances, so the shape carries far less information than the same picture
over the validation window and is shown to make the drawdown legible rather
than to support a claim.

```python
holdout_returns = load_daily_returns_with_timestamp(CASE_STUDY_ID, HOLDOUT_BACKTEST_HASH)
if holdout_returns is None or holdout_returns.is_empty():
    raise RuntimeError("the holdout backtest registered no daily return series")

curve = holdout_returns.sort("timestamp").with_columns(
    ((1.0 + pl.col("ret")).cum_prod()).alias("equity")
)
curve = curve.with_columns((pl.col("equity") / pl.col("equity").cum_max() - 1.0).alias("drawdown"))

fig, (ax_equity, ax_dd) = plt.subplots(
    2,
    1,
    figsize=FIGSIZE["dual_v"],
    sharex=True,
    height_ratios=[2, 1],
    constrained_layout=True,
)
ax_equity.plot(curve["timestamp"], curve["equity"], color=COLORS["blue"], linewidth=1.6)
ax_equity.set_ylabel("Growth of 1")
ax_dd.fill_between(curve["timestamp"], curve["drawdown"], 0.0, color=COLORS["negative"], alpha=0.5)
ax_dd.set_ylabel("Drawdown")
add_message_title(
    ax_equity,
    "Selected strategy on the 2021 holdout: equity curve and drawdown",
    "Growth of 1 above, peak-to-trough drawdown below, daily",
)
show_with_alt(
    fig,
    "Two panels sharing a date axis over the holdout year: growth of one unit of capital above, "
    "and peak-to-trough drawdown filled below.",
)
```

## Key takeaways

1. The strategy run here is the selected configuration's own specification, cloned and
   re-pointed at the holdout predictions. Nothing about it was re-derived, so
   a change upstream would show up as a different selection rather than as a
   quietly different holdout.
2. The holdout prediction set is matched to the selected configuration by comparing the two
   training specifications field by field, because a holdout refit is a
   different training identity and cannot be matched on a hash.
3. A validation figure is the maximum of a search and is optimistic. A single
   holdout year is unbiased with respect to that search and is noisy, and the
   registered Sharpe interval is what says how noisy. Neither interval is a
   test of the difference between the two, which is why none is reported.
4. A drop from validation to holdout has at least two sufficient explanations
   - selection bias and a different market - and one window cannot separate
   them. Neither can any notebook downstream of this one.
5. Survivorship bias bites hardest here. The universe is today's S&P 500
   membership, so the 2021 window is evaluated on companies known to have
   stayed in the index.

**Next:** [`20_strategy_analysis`](20_strategy_analysis.ipynb) assembles what
the case study has established across all of its stages, and states which of
its claims this window supports.
![notebook output](figures/p1_1.png)