A strategy agent can contaminate a test set without putting a single test row in a training job. If the agent sees the test score, changes a parameter, and runs again, that score has already influenced the strategy. The code may call the period “out of sample.” The research process has made it part of the search.
I’ve seen this happen through convenience: an agent runs a batch, reads a summary table, then proposes a small improvement. Nobody asked it to fit on the holdout. The feedback loop did the fitting anyway.
Mistake 1: Letting the agent peek after every run
The wreckage usually looks like a test Sharpe that rises in tidy little steps. The agent tries 30 variants, keeps the best, then tries 30 variants around that one. Each individual change looks reasonable. Together, they turn the test period into an optimization target.
Here’s a toy example. Suppose the strategy has no real edge and you test 100 independent variants on a holdout. If you select the highest Sharpe, it’s quite possible to find a flattering result by chance. The variants won’t be truly independent in most strategy searches, but that doesn’t rescue the process: repeated feedback still rewards settings that happen to fit the holdout’s quirks.
Put the holdout behind a result boundary. The agent can develop against training and validation periods; it cannot retrieve holdout metrics or charts. Keep the final evaluation in a separate job, run once after the research choices are frozen, and record the strategy version and data snapshot with its result.
Mistake 2: Calling every later period “fresh”
A date range isn’t fresh just because it comes after the training window. If the agent has seen a score from that range, or a human used its drawdown to choose a different feature, that range has become part of the feedback loop.
| Period | Allowed use | What changes its status |
|---|---|---|
| Training | Fit rules and parameters | Nothing; it is meant to be searched |
| Validation | Compare candidates and make research decisions | Repeated use makes it easier to overfit |
| Final holdout | One evaluation of the frozen candidate | Reading its result and changing the candidate |
| Future paper period | Observe implementation behavior prospectively | Using outcomes to tune creates a new research cycle |
Rolling walk-forward evaluation can help estimate how a fixed research process behaves across time. It doesn’t grant unlimited peeks at every fold. If an agent sees fold results and adapts its search, those folds are feedback too. Preserve a final untouched period for the decision you actually want to make.
Mistake 3: Freezing the code while changing the question
Sometimes the strategy code stays fixed but the researcher changes the evaluation after seeing the result: swap the benchmark, exclude a rough month, report a different universe, or focus on the metric that looks strongest. The test set has guided the story even if it didn’t change a parameter.
Wreckage appears as a polished report with one clean headline and no record of the other views that were tried. An equity strategy might look weak against a broad market index but strong against cash; a crypto strategy might shine on one venue after several venues were inspected. Those comparisons can be useful, but choosing among them after seeing the holdout is another selection step.
Keep a research ledger. For each run, save the hypothesis, code version, data snapshot, parameter search, evaluation windows, metrics requested, and whether the agent or researcher saw the result. A compact log is more useful than a perfect-sounding “out of sample” label.
Then make the final question explicit before evaluation: which candidate, which market universe, which benchmark, which costs, and which metrics? If the answer changes after the result arrives, treat the next run as research and reserve new data for the next final check.
What can I still learn from a contaminated holdout?
It isn’t useless. It can show how a strategy behaved during that period and help diagnose failure. Just stop presenting its score as an independent confirmation. Use it as development evidence, freeze the next candidate, and evaluate that candidate on a genuinely future paper period with the same accounting and execution assumptions.
In an agent-driven workflow, experiment count is easy to lose: one prompt can create dozens of runs before anyone notices. The protection is procedural. Limit what the agent can inspect, log every experiment, and keep one evaluation whose result cannot steer the candidate being judged.
← All posts
