Ausgewähltes Futures-Modell für die Holdout-Evaluation neu trainieren

# CME Futures: Holdout Predictions


# CME Futures: Holdout Predictions

**Chapter 20 - Out-of-sample evaluation**

Every number in this case study so far was measured on the validation periods, and every
choice was made by looking at them: which model family, which configuration, how the
positions are sized, whether a risk overlay earns its place. A result selected that way
cannot also be evidence that the selection was sound - the ranking and the evidence would
be the same measurement.

The holdout is the window nothing has been selected on: 2024-01-01 to 2025-12-31. This
notebook fits the selected configuration on the history available before that window
opens and writes its predictions over it.
[`18_holdout_backtest`](18_holdout_backtest.ipynb) turns those predictions into a return
series with the sizing this case study settled on, and
[`19_strategy_analysis`](19_strategy_analysis.ipynb) reads both back.

**What this notebook is careful about**

A holdout prediction is not the validation model scored on a later window. Those two
differ in what the model saw while fitting, and only the first is out of sample. Section
3 fits again, and the new training identity is what makes the refit visible rather than
asserted - a run that came back carrying the validation training hash would mean no refit
happened.

There is a second thing this case study had to fix before the refit could mean anything.
`04_model_based_features` fits ARIMA and a hidden Markov model per period, so a period
the file never wrote carries no forecast and no regime probability. The holdout period
was being appended to the fold spec without those features being generated for it, which
left a model reading two of nine features over the window it is judged on. The artifact
now carries a holdout vintage: coefficients estimated at the last session before the
window opens, rolled forward across it frozen.

**Prerequisites:** [`16_costs`](16_costs.ipynb), which fixes the configuration the
holdout runs.

**Scope:** one training run and one prediction set. No backtest, no selection, no
comparison - those are 18 and 19.

```python
"""CME Futures: Holdout Predictions."""

import sqlite3

import polars as pl

from case_studies.research import open_study
from case_studies.research.holdout import build_holdout_training_spec
from case_studies.research.models import reconstruct_locked_model_request
from case_studies.utils.registry import training_hash_from_spec
from case_studies.utils.strategy_analysis import (
    resolve_solvent_carrier,
    training_run_fitted_for_the_holdout,
)
from case_studies.utils.warning_policy import apply_notebook_warning_policy
from utils.paths import get_case_study_dir

apply_notebook_warning_policy()
```

```python
CASE_STUDY_ID = "cme_futures"
EXECUTION_TIER = "canonical"
WORKSPACE: str = ""
# Whether a holdout generation for a DIFFERENT configuration may be superseded by this run.
# Off by default: see section 3.
#
# The flag exists because the holdout is not a one-shot resource, which is a ruling and not
# an oversight. What the rule against consulting the holdout forbids is SELECTING on it: the
# configuration evaluated here is chosen by validation backtest Sharpe, and no holdout number
# feeds back into that choice. It says nothing about how many times the evaluation may be
# computed, and a wrong result is deleted and re-run rather than left standing because it was
# observed. Reading the rule as a physical constraint is what produced a lock layer around
# this window, and it is being removed. The guard here is against something narrower and real:
# two generations readable at once, so nobody downstream has to choose between them and nobody
# can quote whichever number they prefer.
```

```python
study = open_study(CASE_STUDY_ID, execution_tier=EXECUTION_TIER, workspace=WORKSPACE or None)
CASE_DIR = get_case_study_dir(CASE_STUDY_ID)


def _registered_holdout_generations(case_dir):
    """Every holdout prediction set in the registry, and whether its model was refitted.

    `refitted` is read from the training run's own CV rather than from the prediction
    set's split: the split says where the predictions land, and a model fitted on the
    validation folds can publish predictions over the holdout window. That is the defect
    the generation below replaces, and it is why this is the same predicate the canonical
    lineage resolver applies.
    """
    with sqlite3.connect(str(case_dir / "run_log" / "registry.db")) as conn:
        rows = conn.execute(
            """
            SELECT p.prediction_hash, p.training_hash, p.checkpoint_kind, p.checkpoint_value,
                   t.config_name, t.spec_json
            FROM prediction_sets p
            JOIN training_runs t ON t.training_hash = p.training_hash
            WHERE p.split = 'holdout'
            ORDER BY p.prediction_hash
            """
        ).fetchall()
    return [
        {
            "prediction_hash": prediction_hash,
            "training_hash": training_hash,
            # The checkpoint is part of the configuration, not a detail of it: one training
            # run publishes one prediction set per declared checkpoint, and moving the
            # selection from one checkpoint to another is a different configuration
            # evaluated on the same window. Identity on the training hash alone would see
            # that as the same generation and let both stand.
            "checkpoint": (checkpoint_kind, checkpoint_value),
            "config_name": config_name,
            "refitted": training_run_fitted_for_the_holdout(training_spec_json),
        }
        for (
            prediction_hash,
            training_hash,
            checkpoint_kind,
            checkpoint_value,
            config_name,
            training_spec_json,
        ) in rows
    ]
```

## 1. Which configuration the holdout runs

The holdout runs the configuration the case study reports, resolved through the same
`resolve_solvent_carrier` [`16_costs`](16_costs.ipynb) uses. Resolving it again here
rather than passing it along is deliberate: the two notebooks must agree by construction,
and a hash written down in one and read in the other agrees only until the sweep is
rebuilt.

Nothing about the holdout enters this choice. The selected configuration is the validation
rank-1, and it was fixed before this notebook ran.

```python
carrier = resolve_solvent_carrier(CASE_STUDY_ID)
print(
    f"Selected configuration: {carrier['val_backtest_hash']}  stage={carrier['val_stage']}  "
    f"family={carrier['family']}  config={carrier['config_name']}  "
    f"label={carrier['label']}"
)
print(
    f"  validation Sharpe {carrier['val_sharpe']:.3f}, max drawdown {carrier['max_drawdown']:.3f}"
)
print(f"  fitted by training run {carrier['training_hash']}")
```

The checkpoint is part of the configuration, and it is read from the selected configuration's own
prediction set rather than assumed. Every family this case study fits publishes at the
end of training, so the value here is `final` and carries no iteration - but reading it
is what keeps that true rather than asserted: a family that later publishes on a
schedule would name one of its checkpoints here, and refitting without it would produce
the model at the end of training instead of the one that was ranked.

```python
validation_prediction = study.results.open(carrier["val_prediction_hash"])
prediction_record = validation_prediction.registry_record()
CHECKPOINT_KIND = prediction_record["checkpoint_kind"]
CHECKPOINT_VALUE = prediction_record["checkpoint_value"]
print(f"Checkpoint: {CHECKPOINT_KIND}={CHECKPOINT_VALUE}")
```

## 2. The window, and the model that is allowed to see it

The holdout window is not a choice made here. It is `evaluation.holdout_start` and
`evaluation.holdout_end` from the case study's own `setup.yaml`, read through the same
`canonical_window` the fold derivation and the backtest slice both go through, so the
three cannot disagree.

The training interval is everything available before that window, bounded above by a
label buffer. The buffer is what stops the last training label's outcome from resolving
inside the holdout: this case study dates each row by the session the decision is taken
on and the label looks forward from there, so a row dated inside a horizon's reach of
2024-01-01 has its outcome realised in the holdout. The buffer is therefore a horizon's
worth of sessions - 5 for `fwd_ret_5d`, 21 for `fwd_ret_21d` - and the derivation seals
on the widest of the declared labels rather than on the one being fitted, so no label
can reach past its own boundary. A zero gap would be a leak, not a conservative choice,
so the derivation refuses to default it.

Everything else about the configuration is carried across unchanged, and the fields that
cannot be - the eligibility manifest, and any parameter this family resolves from a
fold's own training rows - are recomputed against the holdout fold. Carrying those
forward would fit a model keyed to the validation folds and call it a retrain.

```python
observation_timeline = (
    pl.read_parquet(study.root / "labels" / f"{carrier['label']}.parquet")
    .get_column("timestamp")
    .unique()
    .sort()
    .to_list()
)
validation_spec = study.results.open(carrier["training_hash"]).spec()
holdout_spec = build_holdout_training_spec(
    study,
    validation_spec,
    timeline=observation_timeline,
    case_study=CASE_STUDY_ID,
)

fold = holdout_spec["computation"]["cv"]["folds"][0]
print(f"Holdout fold {fold['fold']}")
print(f"  trains  {fold['train_start']} -> {fold['train_end']}")
print(f"  predicts {fold['val_start']} -> {fold['val_end']}")
print(f"  label buffer: {holdout_spec['computation']['cv']['request']['label_buffer']}")

# The validation folds are what the buffer is measured against, and the last of them ends
# before the holdout opens. Printing both is what lets a reader check the gap rather than
# take it on the derivation's word.
validation_folds = validation_spec["computation"]["cv"]["folds"]
latest_validation_end = max(str(entry["val_end"]) for entry in validation_folds)
print(f"Validation folds: {len(validation_folds)}, latest evaluation end {latest_validation_end}")
print(f"Holdout training ends {fold['train_end']}, holdout opens {fold['val_start']}")
```

## 3. Fit, and register the predictions

`reconstruct_locked_model_request` builds the request from the spec above. Its name
comes from the locked holdout path this case study no longer uses; it takes a training
specification and a checkpoint, not a lock, and it is used here because it is the one
call that refuses a request that is not exactly the spec it was handed - the training
identity, the checkpoint schedule, the feature lineage and the runtime parameters are
all checked before anything is fitted.

The training identity below is new. It has to be: it covers the CV interval, and the
holdout fold is not one of the validation folds. A run that came back with the
validation training hash would mean the refit did not happen.

**The window carries one configuration at a time.** The holdout is re-runnable, and that
is not the same as free: every configuration evaluated on it is another look at a period
the case study reports as unseen, and two evaluated quietly would make that report false.

So the check below is on the selected configuration rather than on the notebook, and it has
exactly two outcomes. With the selected configuration unchanged this is an idempotent replay: the
derivation is deterministic and the training identity covers it, so the same identity comes back
and the fit is served from the registry. With the selected configuration changed it refuses,
names both configurations, and stops.

It refuses rather than offering a replacement switch, and the reason is that a replacement
would not be one. Deleting the earlier generation's rows does not undo having observed its
result: the selection that produced the new configuration may have been informed by the old
holdout number, and no deletion reaches that. A switch here would let the case study take a
second look at the window while leaving a registry that shows only one, which is the
specific thing that would make the out-of-sample claim false rather than merely weak.

```python
holdout_training_hash = training_hash_from_spec(holdout_spec)
this_generation = (holdout_training_hash, (CHECKPOINT_KIND, CHECKPOINT_VALUE))
superseded = [
    row
    for row in _registered_holdout_generations(CASE_DIR)
    if row["refitted"] and (row["training_hash"], row["checkpoint"]) != this_generation
]
if superseded:
    raise RuntimeError(
        "the holdout window already carries a refit of a different configuration: "
        + ", ".join(
            f"{row['prediction_hash']} ({row['config_name']}, training {row['training_hash']})"
            for row in superseded
        )
        + f". This run would evaluate {carrier['config_name']} (training "
        f"{holdout_training_hash}, checkpoint {CHECKPOINT_KIND}={CHECKPOINT_VALUE}) on the "
        "same window, which would be a second configuration measured on a period this case "
        "study reports as unseen. This notebook has no way past that: deleting the earlier "
        "generation would not undo having observed it, and the selection bias it introduces "
        "is not removed by removing the rows. Either leave the selection where it was, or "
        "retire the earlier evaluation through the registry's own lifecycle, which records "
        "that a second look was taken."
    )
```

```python
request = reconstruct_locked_model_request(
    study,
    holdout_spec,
    checkpoint_kind=CHECKPOINT_KIND,
    checkpoint_value=CHECKPOINT_VALUE,
)
model_run = request.run()
holdout_prediction = model_run.predictions[0]

if model_run.training.hash == carrier["training_hash"]:
    raise RuntimeError(
        "the holdout refit produced the validation training identity "
        f"{carrier['training_hash']}, which means it did not refit"
    )
print(f"Holdout training run:  {model_run.training.hash}")
print(f"Holdout prediction set: {holdout_prediction.hash}")
```

What the prediction set covers, read back from the registry rather than from the
request. The two agree only if the fit published what it declared, and the count is the
one number a reader can check the window against: the holdout spans two years of
sessions, the strategy decides weekly, and the number of rows is those decision dates
times the products eligible on each.

```python
record = holdout_prediction.registry_record()
predictions = holdout_prediction.load()
# `13_backtest` records that reader-facing rows use `product` and the shared boundary
# converts to the engine's `symbol` key, so a prediction set can arrive carrying either.
# Named from the frame rather than assumed, so this line cannot quietly report nothing.
_ENTITY_COL = next(c for c in ("product", "symbol") if c in predictions.columns)
print(
    f"split={record['split']}  checkpoint={record['checkpoint_kind']}={record['checkpoint_value']}"
)
print(f"rows={predictions.height:,}  dates={predictions['timestamp'].n_unique()}")
print(
    f"  {predictions['timestamp'].min()} -> {predictions['timestamp'].max()}, "
    f"{predictions[_ENTITY_COL].n_unique():,} {_ENTITY_COL}s"
)
```

The registry now holds more than one holdout prediction set for this case study, and
only one of them was fitted on data that ends before the window. The other is the
defective generation this notebook replaces: it carries the validation training identity,
which is how it was found. Both are listed rather than one silently preferred, because
the registry is immutable and a reader looking at it later will see both.

```python
for row in _registered_holdout_generations(CASE_DIR):
    note = (
        "refitted for the holdout" if row["refitted"] else "VALIDATION-FITTED - not out of sample"
    )
    print(
        f"  {row['prediction_hash']}  training={row['training_hash']}  {row['config_name']}  {note}"
    )
```

## What this notebook establishes, and what it does not

It establishes one thing: a prediction set over the holdout window, produced by the
configuration this case study selected, fitted on data that ends before the window
opens. That is a precondition for an out-of-sample claim, not the claim itself. Nothing
here says whether the predictions are any good - they have not been scored, sized or
traded.

It does not make the holdout a fresh test in the strict sense. The configuration reached
this notebook through a selection made on the validation folds, and this window is being
used once per configuration that gets here. What it does remove is the specific
circularity of scoring a validation-fitted model on the period meant to judge it.

The holdout is re-runnable. If a later pass finds the selection was wrong, the answer is
to delete this generation and produce another, not to treat the first as spent.

**Next:** [`18_holdout_backtest`](18_holdout_backtest.ipynb).