Réajustement sur échantillon test d’une stratégie FX, sélectionnée par validation, sans sélection sur l’échantillon test
Notebook Machine Learning for Trading
Résumé
Ce notebook décrit la création de prévisions sur échantillon test pour une stratégie FX, après sélection de sa configuration à partir des résultats de validation. Il retient le backtest de validation au ratio de Sharpe le plus élevé parmi les candidats admis restés solvables, puis construit l’intervalle test à partir de la chronologie des observations de la cible sélectionnée. L’entraînement s’arrête avant l’échantillon test, avec une marge correspondant à l’horizon de la cible, afin que les résultats d’entraînement ne s’étendent pas à la période d’évaluation. La demande de modèle est reconstruite à partir des spécifications de validation ; la géométrie des plis change, mais la configuration et le point de contrôle sélectionnés sont conservés.
Le notebook vérifie que le réajustement possède une identité d’entraînement distincte, ne publie que le point de contrôle sélectionné et produit une prévision complète sur l’échantillon test avec la couverture attendue. Il valide également l’état persistant du modèle ajusté. Ces précautions établissent ce qui a été ajusté et l’intervalle prédit ; elles n’empêchent ni les réutilisations répétées de l’échantillon test ni la sélection des résultats à publier. Réexécuter plusieurs fois l’évaluation et retenir le meilleur résultat nuirait à son interprétation ; l’analyse ultérieure des performances est donc traitée séparément.
Idées clés
- Choisir la configuration FX à partir des résultats de validation avant de créer les prévisions sur l’échantillon test.
- Déduire les limites de l’échantillon test à partir de la grille d’observations de la cible sélectionnée et laisser une marge complète correspondant à l’horizon de la cible avant l’évaluation.
- Reconstruire l’ajustement sur l’échantillon test à partir des spécifications de validation en ne modifiant que la géométrie des plis.
- Ne publier que le point de contrôle sélectionné afin que les résultats de l’échantillon test ne créent pas un nouveau choix entre modèles.
- Tester plusieurs fois l’échantillon test et sélectionner les résultats publiés rend l’évaluation difficile à interpréter.
Étiquettes
Texte intégral
# Holdout Predictions - FX Pairs
# Holdout Predictions - FX Pairs
This notebook refits the one configuration validation selected, on the holdout interval, and
registers its predictions. It writes predictions and nothing else: the backtest is
`18_holdout_backtest`, and what any of it is worth is `19_strategy_analysis`.
Selection is not a parameter and is not made here. `resolve_solvent_carrier` reads the
highest-Sharpe registered validation backtest across the baseline, allocation and risk-overlay
stages, restricted to runs that stayed solvent, so this notebook cannot select a configuration
the validation stages did not already rank first.
The holdout is not a one-shot transaction and nothing here pre-registers it. There is no lock,
no seal and no gate: the whole rule is retrain the selected configuration on everything up to
the holdout window, predict, and backtest that same configuration on the result. Re-running is
therefore ordinary. A reader who runs it five hundred times and quotes the best number has
produced something uninterpretable, and that is a property of what they did rather than
something the software can prevent - the earlier design tried to, and bought
unfixability: a holdout found to be wrong after a bug fix could not be corrected, because the
lock was by construction the one artifact that could not be revised.
**Learning objectives**
- Derive a holdout interval from the panel's observation grid rather than the calendar.
- Reconstruct one training identity from an immutable validation specification.
- Register holdout predictions without giving them any influence over selection.
**Book reference**: Chapters 16-20
**Prerequisite**: `16_costs`. The selected configuration is resolved from the registered
validation backtests, so every stage that registers one must have run.
```python
"""Refit the validation-selected FX configuration on the holdout interval."""
import polars as pl
from case_studies.research import open_study
from case_studies.research.comparison import CandidateSet
from case_studies.research.holdout import build_holdout_training_spec
from case_studies.research.models import (
reconstruct_locked_model_request,
validate_locked_model_run,
)
from case_studies.utils.strategy_analysis import resolve_solvent_carrier
```
```python
CASE_STUDY_ID = "fx_pairs"
EXECUTION_TIER = "canonical"
WORKSPACE: str = ""
CANDIDATE_SET_NAME = "fx_pairs:holdout-candidates"
```
## Resolve the selection and the holdout interval it determines
The label the selection was made on decides which observation grid the holdout interval is
stepped back along, so it is read from the selected lineage rather than assumed. FX carries
three labels on one daily grid, which is exactly the coincidence that would let an assumption
here survive untested.
The training window ends a whole label buffer, counted in observations, before the holdout
opens, so the last training label's outcome cannot resolve inside the holdout. The buffer is
the widest this case study configures rather than the primary label's: a 21-day forward return
resolves three weeks after the session it is stamped on, and a 1-day buffer would leave three
weeks of holdout outcomes reachable from the training window.
```python
study = open_study(CASE_STUDY_ID, execution_tier=EXECUTION_TIER, workspace=WORKSPACE or None)
# The selected configuration is the highest-Sharpe registered VALIDATION backtest across the
# baseline, allocation and risk-overlay stages, restricted to runs that stayed solvent. It is
# resolved from the registry rather than named here, so this notebook cannot select a configuration
# that the validation stages did not rank first. `15_risk_management` froze the set the holdout is
# allowed to choose from, and that set is passed into the resolution rather than checked against its
# answer. The two are different tests: when a conformal candidate is in the field the resolver
# re-ranks every candidate on the timestamps they all share, so a row that was never admitted still
# decides how far that intersection reaches and therefore which admitted row wins. fx has 194
# conformal backtests, so this is a live path here rather than a hypothetical one. Checking
# membership afterwards would pass while the answer had already been changed by an ineligible row.
holdout_candidates = CandidateSet.one(study, name=CANDIDATE_SET_NAME)
carrier = resolve_solvent_carrier(CASE_STUDY_ID, admitted=frozenset(holdout_candidates.members))
print(
f"Frozen candidate set {holdout_candidates.hash}: "
f"{len(holdout_candidates.members)} members, "
f"raw-Sharpe pick {holdout_candidates.best_validation_sharpe().hash}"
)
validation_prediction = study.results.open(carrier["val_prediction_hash"])
prediction_record = validation_prediction.registry_record()
CHECKPOINT_KIND = prediction_record["checkpoint_kind"]
CHECKPOINT_VALUE = prediction_record["checkpoint_value"]
# The label the selected configuration was fitted on decides which observation grid the holdout
# interval is stepped back along, so it is read from the selected configuration rather than assumed.
# FX carries three labels on one daily grid, which is exactly the coincidence that would let an
# assumption here survive untested.
observation_timeline = (
pl.read_parquet(study.root / "labels" / f"{carrier['label']}.parquet")
.get_column("timestamp")
.unique()
.sort()
.to_list()
)
validation_spec = study.results.open(carrier["training_hash"]).spec()
holdout_spec = build_holdout_training_spec(
study,
validation_spec,
timeline=observation_timeline,
case_study=CASE_STUDY_ID,
)
holdout_fold = holdout_spec["computation"]["cv"]["folds"][0]
pl.DataFrame(
{
"field": [
"selected backtest",
"selected stage",
"validation Sharpe",
"family",
"configuration",
"label",
"checkpoint",
"validation training",
"validation prediction",
"holdout train window",
"holdout evaluation window",
],
"value": [
carrier["val_backtest_hash"],
str(carrier["val_stage"]),
f"{carrier['val_sharpe']:.4f}",
str(carrier["family"]),
str(carrier["config_name"]),
str(carrier["label"]),
f"{CHECKPOINT_KIND}={CHECKPOINT_VALUE}",
str(carrier["training_hash"]),
validation_prediction.hash,
f"{holdout_fold['train_start']} to {holdout_fold['train_end']}",
f"{holdout_fold['val_start']} to {holdout_fold['val_end']}",
],
}
)
```
## Refit the selected configuration on the holdout fold
The request is reconstructed from the immutable validation specification with only the fold
geometry re-keyed, so the holdout model differs from the validation model in what it was
fitted on and in nothing else. It publishes the selected checkpoint alone: a holdout refit
that published its whole checkpoint schedule would hand the next notebook a choice, and
choosing among holdout checkpoints is selection on the holdout under another name.
The one thing checked afterwards that a specification cannot state about itself is that this
is a refit at all. A holdout training identity equal to the validation one means the fold
re-keying changed nothing, and the model is a validation fit predicting forward over a later
window rather than a model trained up to it.
```python
request = reconstruct_locked_model_request(
study,
holdout_spec,
checkpoint_kind=CHECKPOINT_KIND,
checkpoint_value=CHECKPOINT_VALUE,
)
model_run = request.run()
if model_run.training.hash == carrier["training_hash"]:
raise RuntimeError(
f"the holdout refit produced the validation training identity "
f"{carrier['training_hash']}, so it did not refit"
)
if len(model_run.predictions) != 1:
raise RuntimeError(
f"the holdout refit published {len(model_run.predictions)} prediction sets; "
"only the selected checkpoint may be published"
)
prediction = model_run.predictions[0]
record = prediction.registry_record()
if record["split"] != "holdout":
raise RuntimeError(f"the holdout refit published a {record['split']!r} prediction")
if record["checkpoint_kind"] != CHECKPOINT_KIND or record["checkpoint_value"] != CHECKPOINT_VALUE:
raise RuntimeError(
f"the holdout prediction is at checkpoint {record['checkpoint_kind']}="
f"{record['checkpoint_value']}, not the carrier's {CHECKPOINT_KIND}={CHECKPOINT_VALUE}"
)
if not prediction.complete:
raise RuntimeError("the holdout prediction is incomplete")
# Completeness verifies the published prediction artifact, not that the persisted fitted state
# reproduces it. A cached run whose model state is missing or inconsistent passes the check
# above and fails only where someone tries to use the model again. The family's own validator
# reads that state and returns its digest, which is the one thing about this run that the
# specification cannot state about itself.
fitted_state_digest = validate_locked_model_run(request, model_run)
print(f"Holdout training run: {model_run.training.hash}")
print(f"Fitted-state digest: {fitted_state_digest}")
print(f"Holdout prediction set: {prediction.hash}")
```
## What the holdout refit covers
The coverage is printed rather than assumed: a holdout prediction set that silently covers a
shorter window than the fold declares would make every number downstream a measurement of a
different interval than the one this notebook says it measured.
```python
frame = prediction.load()
coverage = pl.DataFrame(
{
"field": ["prediction", "rows", "symbols", "sessions", "first session", "last session"],
"value": [
prediction.hash,
str(frame.height),
str(frame.get_column("symbol").n_unique()),
str(frame.get_column("timestamp").n_unique()),
str(frame.get_column("timestamp").min()),
str(frame.get_column("timestamp").max()),
],
}
)
coverage
```
## Key takeaways
- The configuration refitted here was selected on validation, from an immutable set, upstream.
- The holdout training window stops a full label buffer short of the holdout, counted in
observations rather than calendar days.
- Only the selected checkpoint is published, so no choice among holdout results remains to be
made downstream.Reproduit dans son intégralité avec attribution, conformément à la licence de la source. Licence: MIT
Ce résumé a été rédigé par l’agent de recherche de Stratmill à partir de la source originale ; il n’en est pas une copie.