Prédictions hors échantillon sur options S&P 500

# S&P 500 Options: Holdout Predictions


# S&P 500 Options: Holdout Predictions

**Chapter 20 - Out-of-sample evaluation**

Every number in this case study so far was measured on the validation folds, and every
choice was made by looking at them: which model family, which configuration, how many
straddles to write each week, how to size them, what to charge for the option spread and
the hedge. A result selected that way cannot also be evidence that the selection was
sound - the ranking and the evidence would be the same measurement.

The holdout is the window nothing has been selected on. This notebook fits the selected
configuration on the history available before that window opens and writes its predictions
over it. [`17_holdout_backtest`](17_holdout_backtest.ipynb) turns those predictions into
short straddles and a return series, with the concentration, the allocator and the cost
assumption the case study settled on, and
[`18_strategy_analysis`](18_strategy_analysis.ipynb) reads both back.

**What this notebook is careful about**

A holdout prediction is not the validation model scored on a later window. Section 2 fits
again, over a training interval that ends before the window opens, and the new training
identity is what makes the refit visible rather than asserted: the identity covers the CV
interval, so a run that came back with the validation training hash would mean no refit
happened. The check is in section 3 and it raises.

The label makes the gap between training and the window unusually long here. `ret_to_expiry`
resolves when a contract expires, not a fixed number of sessions after the decision, so the
buffer is declared in `setup.yaml` as `labels.buffer` and is a full option cycle rather than
a formality.

**Prerequisites:** [`15_costs`](15_costs.ipynb), which is the last stage that selects.

**Scope:** one training run and one prediction set. No backtest, no straddles, no
selection, no comparison - those are 17 and 18.

```python
"""S&P 500 Options: Holdout Predictions."""

import polars as pl

from case_studies.research import open_study
from case_studies.research.holdout import build_holdout_training_spec
from case_studies.research.models import reconstruct_locked_model_request
from case_studies.utils.registry import training_hash_from_spec
from case_studies.utils.registry.maintenance import delete_prediction_generation
from case_studies.utils.strategy_analysis import (
    holdout_generations_to_retire,
    registered_holdout_generations,
    resolve_solvent_carrier,
)
from utils.paths import get_case_study_dir
```

```python
CASE_STUDY_ID = "sp500_options"
EXECUTION_TIER = "canonical"
WORKSPACE: str = ""
# Replace a holdout evaluation this window already carries, rather than adding a second one.
# Off by default, because the default should be the methodology: one configuration is measured
# on the holdout, and the number means what it says only while that stays true. It is a
# parameter and not a prohibition because a holdout does get re-derived - a corrected input
# upstream moves every training identity, and the evaluation that was right for the old
# geometry is then an evaluation of a configuration this case study no longer selects.
# Refusing that outright would mean a correction could never be carried through to the end.
REPLACE_HOLDOUT = False
```

```python
study = open_study(CASE_STUDY_ID, execution_tier=EXECUTION_TIER, workspace=WORKSPACE or None)
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)


def _delete_holdout_generation(case_dir, prediction_hash):
    """Remove one holdout prediction set and everything registered against it.

    Reached only when ``REPLACE_HOLDOUT`` says a generation is superseded. The rows go rather
    than being marked, because a superseded holdout evaluation that is still readable is still
    a number someone can quote, and the point of replacing it is that it should not be one.

    The cascade lives in `case_studies/utils/registry/maintenance.py` and derives the child
    tables from the schema, because the version of this that listed them by hand missed
    `cohort_metrics.leader_hash` - which with foreign keys enabled aborts the delete rather
    than orphaning a row.
    """
    deleted = delete_prediction_generation(case_dir / "run_log" / "registry.db", prediction_hash)
    for table, n in sorted(deleted.items()):
        print(f"  deleted {n:>3} from {table}")
```

## 1. Which configuration the holdout runs

The holdout runs the configuration the case study reports, resolved through the same
`resolve_solvent_carrier` [`15_costs`](15_costs.ipynb) prices. Resolving it again here
rather than passing it along is deliberate: the two notebooks must agree by construction,
and a hash written down in one and read in the other agrees only until the sweep is rebuilt.

Nothing about the holdout enters this choice. The selected configuration is the cross-stage
validation rank-1, and it was fixed before this notebook ran.

**Its validation Sharpe is negative.** That is this case study's result, not a reason to look for
a different configuration: writing straddles on a signal this weak does not pay for the option
spread and the hedge, and the holdout is being used to see whether that reading holds on a window
nothing was chosen on. Picking a configuration for its sign would be the selection the holdout
exists to make honest.

```python
carrier = resolve_solvent_carrier(CASE_STUDY_ID)
print(
    f"Selected configuration: {carrier['val_backtest_hash']}  stage={carrier['val_stage']}  "
    f"family={carrier['family']}  config={carrier['config_name']}  "
    f"label={carrier['label']}"
)
print(
    f"  validation Sharpe {carrier['val_sharpe']:.3f}, max drawdown {carrier['max_drawdown']:.3f}"
)
print(f"  fitted by training run {carrier['training_hash']}")
```

The checkpoint is part of the configuration. Where a family publishes a prediction set per
checkpoint on a declared schedule, the selected configuration's prediction set names one of them,
and refitting without it would produce a model at the end of training rather than the one that was
ranked. A family with no checkpoint dimension - which is where this case study's configuration
sits - stores NULL in both columns and carries that NULL through unchanged.

```python
validation_prediction = study.results.open(carrier["val_prediction_hash"])
prediction_record = validation_prediction.registry_record()
CHECKPOINT_KIND = prediction_record["checkpoint_kind"]
CHECKPOINT_VALUE = prediction_record["checkpoint_value"]
print(f"Checkpoint: {CHECKPOINT_KIND}={CHECKPOINT_VALUE}")
```

## 2. The window, and the model that is allowed to see it

The holdout window is not a choice made here. It is `evaluation.holdout_start` and
`evaluation.holdout_end` from this case study's own `setup.yaml`, read through the same
`canonical_window` the fold derivation and the backtest slice both go through, so the three
cannot disagree.

The training interval is everything available before that window, bounded above by the label
buffer. **For this case study the buffer is doing real work.** `ret_to_expiry` is realised
at expiration rather than a fixed horizon after the decision, so a row dated `t` records an
outcome that is not known until its contract expires - and a training set running to the day
the window opens would be fitted on labels that resolve inside it. The buffer is therefore an
option cycle's worth, and the derivation refuses to default it: a zero gap here would be a
leak.

Everything else about the configuration is carried across unchanged, and the fields that
cannot be - the eligibility manifest, and any parameter this family resolves from a fold's
own training rows - are recomputed against the holdout fold. Carrying those forward would fit
a model keyed to the validation folds and call it a retrain.

```python
observation_timeline = (
    pl.read_parquet(study.root / "labels" / f"{carrier['label']}.parquet")
    .get_column("timestamp")
    .unique()
    .sort()
    .to_list()
)
validation_spec = study.results.open(carrier["training_hash"]).spec()
holdout_spec = build_holdout_training_spec(
    study,
    validation_spec,
    timeline=observation_timeline,
    case_study=CASE_STUDY_ID,
)

fold = holdout_spec["computation"]["cv"]["folds"][0]
print(f"Holdout fold {fold['fold']}")
print(f"  trains  {fold['train_start']} -> {fold['train_end']}")
print(f"  predicts {fold['val_start']} -> {fold['val_end']}")
print(f"  label buffer: {holdout_spec['computation']['cv']['request']['label_buffer']}")

# The validation folds are what the buffer is measured against, and the last of them ends
# before the holdout opens. Printing both is what lets a reader check the gap rather than take
# it on the derivation's word.
validation_folds = validation_spec["computation"]["cv"]["folds"]
latest_validation_end = max(str(entry["val_end"]) for entry in validation_folds)
print(f"Validation folds: {len(validation_folds)}, latest evaluation end {latest_validation_end}")
print(f"Holdout training ends {fold['train_end']}, holdout opens {fold['val_start']}")
```

## 3. Fit, and register the predictions

`reconstruct_locked_model_request` builds the request from the spec above. Its name comes
from a locked holdout path this case study does not use; it takes a training specification
and a checkpoint, not a lock, and it is used here because it is the one call that refuses a
request that is not exactly the spec it was handed - the training identity, the checkpoint
schedule, the feature lineage and the runtime parameters are all checked before anything is
fitted.

The training identity below is new. It has to be: it covers the CV interval, and the holdout
fold is not one of the validation folds. A run that came back with the validation training
hash would mean the refit did not happen, so that is checked rather than assumed. **That is
not a hypothetical here.** Two holdout prediction sets were registered against this case
study under the validation training identity, by a driver that refits the model and then
registers the result under the identity it started from; their split said `holdout` and
their model had been fitted on folds ending inside the window. They were removed, and the
check below is what makes their shape impossible to reintroduce.

**The window carries one configuration at a time.** The check below is on the selected
configuration rather than on the notebook, and it has three outcomes. With the selected
configuration unchanged this is an idempotent replay: the derivation is deterministic and the
training identity covers it, so the same identity comes back and the fit is served from the
registry, which is why re-running the notebook is free and safe. With the selected configuration
changed it stops and names both configurations. With `REPLACE_HOLDOUT` set it replaces the
earlier generation instead of standing beside it, so the registry never holds two refits of the
same window.

What the replacement does not do is undo having observed the earlier result, and that is the
part worth understanding rather than enforcing. A holdout number is out-of-sample because
nothing about the model was chosen after seeing it. Evaluate a second configuration on the
same window and the second number is no longer that, however the registry is arranged: the
selection may have been informed by the first. Deleting rows removes the evidence, not the
knowledge.

So the honest use of this switch is narrow: an input upstream was corrected, every training
identity moved with it, and the evaluation being replaced belongs to a configuration this
case study no longer selects. That is maintenance, and it has to be possible or a correction
could never reach the end of the pipeline. Reaching for it because the first holdout
disappointed is the thing that makes an out-of-sample claim false, and no parameter default
can tell those two apart - the researcher can, and it is their call.

```python
holdout_training_hash = training_hash_from_spec(holdout_spec)
this_generation = (holdout_training_hash, (CHECKPOINT_KIND, CHECKPOINT_VALUE))
retire = holdout_generations_to_retire(CASE_DIR, this_generation=this_generation)
# A row whose training run records no CV split cannot be shown either way, and deleting on
# that would discard a result nothing has established is wrong. It stops the run instead.
if retire.unattributable:
    raise RuntimeError(
        "the holdout window carries prediction sets whose training runs record no CV split, "
        "so whether they were refitted for the holdout cannot be established: "
        + ", ".join(
            f"{row['prediction_hash']} (training {row['training_hash']})"
            for row in retire.unattributable
        )
        + ". Establish what produced them before registering another evaluation on the same "
        "window; this notebook will not delete a row it cannot show is not a holdout result."
    )
# A row whose training run declares a non-holdout CV may not be reported as a holdout
# result, and it is also not something to delete unattended: `generate_holdout` refits on a
# holdout fold and then registers the predictions under the VALIDATION training identity, so
# this record covers both a validation-fitted model published over the window and a real
# refit filed under the wrong identity. Nothing owned this before - the filter here was
# `row["refitted"]`, which made exactly these rows invisible to the refusal and to
# everything after it.
if retire.not_out_of_sample and not REPLACE_HOLDOUT:
    raise RuntimeError(
        "the holdout window carries prediction sets whose training runs declare a CV split "
        "other than the holdout: "
        + ", ".join(
            f"{row['prediction_hash']} ({row['config_name']}, training {row['training_hash']})"
            for row in retire.not_out_of_sample
        )
        + ". Each is either a validation-fitted model published over the window, which is "
        "not an out-of-sample result, or a refit registered under its validation training "
        "identity, which the retired `20_strategy_synthesis/holdout.py::generate_holdout` "
        "wrote until it was deleted on 2026-09-12 - and "
        "the registry cannot tell those apart. Establish which, then set REPLACE_HOLDOUT="
        "True to remove it, or leave it and resolve the identity instead."
    )
for row in retire.not_out_of_sample:
    print(
        f"REMOVING {row['prediction_hash']} ({row['config_name']}, training "
        f"{row['training_hash']}): its training run declares a non-holdout CV, so it is not "
        "reportable as a holdout evaluation under the identity it carries"
    )
    _delete_holdout_generation(CASE_DIR, row["prediction_hash"])
superseded = list(retire.superseded)
if superseded and not REPLACE_HOLDOUT:
    raise RuntimeError(
        "the holdout window already carries a refit of a different configuration: "
        + ", ".join(
            f"{row['prediction_hash']} ({row['config_name']}, training {row['training_hash']})"
            for row in superseded
        )
        + f". This run would evaluate {carrier['config_name']} (training "
        f"{holdout_training_hash}, checkpoint {CHECKPOINT_KIND}={CHECKPOINT_VALUE}) on the "
        "same window, which is a second configuration measured on a period this case study "
        "reports as unseen. Set REPLACE_HOLDOUT=True to replace the earlier generation - "
        "correct when an upstream fix moved every training identity and the evaluation being "
        "replaced belongs to a configuration no longer selected - or leave the selection "
        "where it was."
    )
for row in superseded:
    print(f"REPLACING holdout generation {row['prediction_hash']} ({row['config_name']})")
    _delete_holdout_generation(CASE_DIR, row["prediction_hash"])
```

```python
request = reconstruct_locked_model_request(
    study,
    holdout_spec,
    checkpoint_kind=CHECKPOINT_KIND,
    checkpoint_value=CHECKPOINT_VALUE,
)
model_run = request.run()
holdout_prediction = model_run.predictions[0]

if model_run.training.hash == carrier["training_hash"]:
    raise RuntimeError(
        "the holdout refit produced the validation training identity "
        f"{carrier['training_hash']}, which means it did not refit"
    )
print(f"Holdout training run:  {model_run.training.hash}")
print(f"Holdout prediction set: {holdout_prediction.hash}")
```

What the prediction set covers, read back from the registry rather than from the request.
The two agree only if the fit published what it declared. The unit here is a contract on a
date, not a name on a date: one underlying carries many strikes and expirations at once, so
the row count is far larger than the session count and the symbol count is underlyings
rather than tradeable instruments.

```python
record = holdout_prediction.registry_record()
predictions = holdout_prediction.load()
print(
    f"split={record['split']}  checkpoint={record['checkpoint_kind']}={record['checkpoint_value']}"
)
print(f"rows={predictions.height:,}  sessions={predictions['timestamp'].n_unique():,}")
print(
    f"  {predictions['timestamp'].min()} -> {predictions['timestamp'].max()}, "
    f"{predictions['symbol'].n_unique():,} underlyings"
)
```

Every holdout prediction set the registry holds, and whether the model behind it was
refitted for the window. All of them are listed rather than one silently preferred, because
the registry is immutable and a reader looking at it later will see whatever is there. A row
marked VALIDATION-FITTED is not an out-of-sample result whatever its numbers say.

```python
for row in registered_holdout_generations(CASE_DIR):
    note = (
        "refitted for the holdout" if row["refitted"] else "VALIDATION-FITTED - not out of sample"
    )
    print(
        f"  {row['prediction_hash']}  training={row['training_hash']}  {row['config_name']}  {note}"
    )
```

## What this notebook establishes, and what it does not

It establishes one thing: a prediction set over the holdout window, produced by the
configuration this case study selected, fitted on data that ends a full option cycle before
the window opens. That is a precondition for an out-of-sample claim, not the claim itself.
Nothing here says whether the predictions are any good - they have not been scored, turned
into straddles, or traded.

It does not make the holdout a fresh test in the strict sense. The configuration reached this
notebook through a selection made on the validation folds, and this window is being used once
per configuration that gets here. What it does remove is the specific circularity of scoring
a validation-fitted model on the period meant to judge it - which, for this case study, is
not an abstract risk: it had already happened once.

Re-running this notebook is free: the same configuration re-derives the same training identity and
the fit is served from the registry. Evaluating a DIFFERENT configuration is not, and is
refused here. If a later pass finds the selection was wrong, that is a question for the
registry's lifecycle, which records that a second look was taken - not something to settle by
deleting rows until the registry agrees.

**Next:** [`17_holdout_backtest`](17_holdout_backtest.ipynb).