On Monday, our AI research agent found a crypto futures strategy with a 4.1 Sharpe ratio. By Thursday, after we counted the experiments and stopped letting the agent tune against the same validation window, the result was 0.8. The strategy hadn't changed. Our estimate of what the search had accomplished had.
We ran this as a small research diary because “optimize carefully” is easy advice to give and hard to operationalize when an agent can write code, launch backtests, inspect the score, and try again in minutes. Here’s what broke, in order.
Monday: Give the agent a bounded question
We asked the agent to test a simple hourly trend idea on a liquid perpetual futures market: enter when a short moving average crossed a longer one, size to a fixed volatility target, and flatten on the reverse signal. We specified the instrument, dates, fee schedule, funding history, execution assumption, and the parameter ranges it could explore.
The historical sample ran from 2021 through 2025. We assigned 2021–2023 to development, 2024 to validation, and kept 2025 sealed as a final holdout. That split was written into the job configuration, so the agent could read development results and validation scores but couldn't retrieve holdout returns.
Our first mistake was allowing 300 parameter combinations in one job without logging the search budget as a research result. The agent returned the top performer: a 9-hour fast average, a 73-hour slow average, and a 0.8% volatility target. Its validation Sharpe was 4.1. We knew the agent had tried many settings. We hadn't yet made that fact impossible to forget.
Tuesday: The winner moved when we reran it
We reran the same job from a clean environment. The top setting changed, and its Sharpe fell to 2.7. A market-data cache had served an updated funding series in the first run, while the rerun used a pinned snapshot. Both jobs claimed to use the same date range; only one had a reproducible data version.
We stopped comparing scores and started recording the full experiment: code revision, data snapshot, fees, funding, execution settings, parameter values, and every trial the agent launched. That showed the 4.1 wasn't a mysterious market edge. It belonged to one setting in a search over 300 attempts on the same validation year.
Wednesday: Freeze the search before opening the holdout
We reran the experiment with a fixed data snapshot and a smaller, predeclared grid: 24 combinations. The agent could rank them on the 2021–2023 development period. We let it see 2024 once, for selection, then locked the chosen settings and the code.
The selected strategy scored 0.8 Sharpe on validation. On the sealed 2025 holdout, it scored 0.3 after taker fees, funding, and the same one-bar execution delay applied throughout the study. That was a useful answer. It wasn't evidence that the strategy would make money in a live account.
One aside from this week: we spent more time agreeing on what “one trial” meant than writing the moving-average code. We counted a changed parameter set as a trial, and also counted runs that changed the signal or sizing rule. A new data snapshot or execution assumption got a separate run record, because it could change the ranking just as much.
Thursday: What we’d skip next time
We would skip the open-ended first job. A research agent needs a search budget and a record of what it tried, especially when it can use each result to decide what to try next. The holdout also needs protection from informal peeking: once someone reads its result and changes the strategy, it has become another validation set.
For a small search, a practical experiment ledger can be plain and explicit:
| Record | Why it matters |
|---|---|
| Code and data versions | So a rerun means the same inputs and logic |
| Every attempted variant | So the winner is understood in light of the search |
| Development, validation, and holdout roles | So a final score isn't quietly reused for tuning |
| Fees, funding, and execution assumptions | So the reported result describes a testable simulation |
After the holdout is opened, return to development with a new research question and preserve the old result as historical evidence. Don’t keep polishing the same strategy against a period it has already seen.
The agent did its job: it searched quickly and found a configuration worth examining. Our job was to make the search legible, preserve an untouched test, and treat the score as a measurement with a history. The interesting result wasn't 4.1 or 0.3. It was seeing exactly how the first number got made.
← All posts
