I gave Claude Code ten years of USD/JPY daily bars and asked it to find a trading rule. It ran backtests through an API, as many as it wanted up to...
For further actions, you may consider blocking this person and/or reporting abuse
Can the agent start a fresh study and get another look at the same holdout? I'd want the one-look rule to survive a new session. Otherwise it's easy to do ten "one-shot" experiments and only write up the convincing one, even if every individual run follows the rules.
Good catch, it could. The count was kept per study, so a new study on the same data started from zero. I've fixed it. Looks are now counted across all your studies on the same data, and a new study shows how many times that hidden period has already been used.
Thanks for fixing it! Showing the previous uses is a nice touch. A fresh study can still be useful, but now it can't quietly pass off that period as unseen data.
Claude‑Code searches for trading strategies with hidden out‑of‑sample data. Uses deflated Sharpe against selection‑bias; strategy did not collapse but performance decayed; stresses statistical guardrails for AI‑driven back‑testing.
Thanks for reading!
Solid experimental design — most people skip the holdout entirely and call it a day.
Two things I'd flag. First, the 2022–2024 window isn't truly unseen. The intervention regime, the yield curve inversion, the BOJ yield-curve-control exit — all of that is baked into the model's pretraining corpus. It can "recognize" the regime without learning a causal structure, which inflates apparent generalization.
Second, the honesty test is stronger if you don't prompt for it. Does the agent surface the overfit on its own, or only when you ask "are you overfitting?" That distinction separates a model that's been RLHF'd to confess from one that actually tracks its own uncertainty.
Quick sanity check: run the identical prompt three times with temperature > 0. If the "discovered" strategy jumps from mean-reversion to breakout to carry depending on the seed, the agent isn't discovering — it's confabulating. Variance across seeds is a better honesty signal than any single run's narrative.
Thanks, both are fair points. You're right that 2022–2024 isn't unseen for the model itself. It's only hidden from the search. I'll say that in the post. I like the three-run test. I'll run the same prompt three times and post what it finds each time.
The hidden holdout plus the written prediction is the right shape, and the 13-of-30 stopping is the detail I'd trust most — an agent that stops because "further searching would add selection bias" is doing the thing humans usually don't.
Two things I'd want pinned down before reading the final number. First: does the deflated Sharpe count the runs it abandoned mid-thought? Trials that never produced a result still happened, and excluding them is the cheapest way to make DSR look better. Second: the prediction was written after seeing the visible period, so it can be hindsight-shaped — pre-registering the metric and the pass/fail line before the search starts would close that.
How does the API count N for the deflation — every call, or only the runs that returned a usable result?
Good question. Every run that finished is counted, including the losing ones. Two runs failed, but both were connection errors before any backtest ran, so there was nothing to count.
You're right about the prediction. I wrote it after the search. Next time I'll set the pass line before starting.
Stopping after 13 trials because the deflated Sharpe metric showed returns were already bounded by trial count is the standout part here. Most agent workflows let the model burn the whole search budget and end up tuning the exit rules to the visible sample without catching that parameter sensitivity like the 2 vs 3 vs 4 yen trail is just curve-fitting noise.
Thanks! Same, I didn't expect it to stop at 13. I was half waiting for it to come back with a 2.5 yen trail. I think seeing "chance" after every run is what turned it toward checking #3 instead of trying new stuff, but that's one run, so I can't really say. Next I want to run the same thing without the ledger and see if it just uses all 30.