Note: I used an AI assistant (Claude) to help write this article and the code. The numbers come from my own locked simulation run, not live trading. This is not financial advice.
Update (16 Oct): A reader (thanks, Arhan Canli) pointed out that my description of the extra cost was misleading. In the tool, the 0.2% per-trade cost is subtracted in only one criterion (mean after cost above zero). The t-statistic and the sign-flip p-value are computed before cost. With the cost applied before the t-test, the pass rates at true edges of 0.15%, 0.20% and 0.30% would be about 0.0%, 0.2% and 16.3%, instead of the 15.6%, 50.4% and 98.2% reported below, which are accurate for the tool as it currently is. So Result 2 describes a tool that tests significance before cost and only checks the sign after it. I am changing the tool so the t-test uses after-cost returns, and I will rerun the calibration under a new locked protocol (version 2).
I built a small Python tool that checks a list of backtest trades against eight strict criteria and prints PASS or FAIL. Before I show it to anyone, I wanted to know one thing: how often does it say PASS when there is nothing there?
So I locked a protocol first (hypothesis, scenarios, and the rule for what counts as acceptable), then ran it once. No tuning after seeing the numbers.
The setup
Each fake dataset is a series of dates with 2 trades per date, return = true mean + noise. I made 500 datasets per scenario, 12 scenarios, 6,000 datasets in total.
Six scenarios have no edge at all:
- N1 to N3: plain Gaussian noise with 200, 100 and 400 dates
- N4: fat-tailed noise
- N5: noise plus 3 planted huge winning dates
- N6: an edge in the first half of the data that reverses in the second half
Six scenarios have a real edge of 0.05%, 0.10%, 0.15%, 0.20%, 0.30% and 0.50% per trade, with 200 dates. The tool also requires the mean to stay positive after an extra 0.2% cost per trade (see the update above for what that does and does not test).
Result 1: false passes on no-edge data
Out of 3,000 no-edge datasets, the tool said "all criteria passed" once. It happened in N1: 1 of 500, or 0.2% (95% interval 0.0% to 1.1%). The other five scenarios gave 0 of 500, including the planted outliers and the half-and-half edge. My rule was that up to 1% false passes is acceptable. The upper end of that interval is slightly above 1%, so one honest reading is "probably under 1%, not proven".
Which criteria do the work? In N1 the strict t-statistic (at least 3) passed in only 0.2% of datasets, and "still positive after removing the 3 best dates" passed in 29.6%. The simple checks (first half positive, second half positive, median positive) pass about half the time on pure noise, which is why none of them is enough alone.
Result 2: how big must a real edge be?
Share of datasets with a true edge that passed all criteria (200 dates, noise sd 1% per trade):
- 0.05% per trade: 0.4%
- 0.10%: 3.6%
- 0.15%: 15.6%
- 0.20%: 50.4%
- 0.30%: 98.2%
- 0.50%: 100%
The tool is conservative about small edges, but it is not safe below the cost line. After the extra 0.2% cost, an edge of 0.15% per trade is a net loss, yet the tool passed 15.6% of those datasets, and 3.6% at 0.10%. Those are false passes on strategies that lose money after cost, well above my 1% rule. So the 1% false-pass result only applies to pure noise, not to weak real edges that cost more than they earn. At 200 dates a real edge needs to be about 0.2% to 0.3% per trade before it is detected reliably. That is a limit of this tool, and I plan to examine it before I release anything.
What this does not show
This is a simulation with independent days. Real trades have regimes and clustering, so real false-pass rates can be higher. I ran the tool at its default settings, and the 500-dataset runs use fewer permutations (2,000) than the real tool (10,000).
I can post the locked protocol and the raw output in a comment if anyone wants to check the numbers.
I am keeping a waiting list for the tool at https://botgauntlet.com. Which of these checks do you already run on your own backtests?
Top comments (3)
I simulated your setup to see which cost your pass rates correspond to: 20,000 datasets per edge, 200 dates x 2 trades, noise sd 1% per trade, and only the t >= 3 criterion (the other seven can only lower a pass rate).
If 0.2% per trade came off before the t-test, the 0.15% and 0.20% edges would pass about 0% of the time and 0.30% only about 16%, because the net edge is then -0.05%, 0% and +0.10% against a standard error of 0.05%. Your numbers instead match a cost of roughly 0.05% per trade (the 0.10% and 0.05% rows fit about as well, within noise at 500 datasets). So either the cost is applied somewhere other than the per-trade returns that feed the t-statistic (per date, or only to some criteria), or the t-statistic uses a different spread than the 1% sd you describe.
That matters for Result 2. "Passes 15.6% of strategies that lose money after cost" may really be "passes 15.6% of strategies that lose 0.05% after a 0.05% cost", or the 0.2% may never reach the t-test. Could you post the raw output for the 0.15% and 0.20% scenarios and say where the 0.2% is subtracted in the code?
Thank you for checking this properly. Your simulation is right: I reran the same setup (20,000 datasets per edge, 200 dates, 2 trades per date, sd 1%) with 0.2% subtracted before the t-test, and got about 0.0% passing t >= 3 at a 0.15% edge, about 0.2% at 0.20%, and 16.3% at 0.30%.
Here is where the 0.2% is subtracted. Only in one criterion. The t-statistic and the sign-flip p-value are computed on date means with no cost (by_date(rows) with no extra). The cost check is separate and only requires the mean after cost to be above zero (cost_v = by_date(rows, extra_cost); mean(cost_v) > 0). It is a point estimate, not a significance test. At a true edge of 0.30% the net edge is 0.10% per trade, about 2 standard errors here, and the tool still passes it 98.2% of the time because t >= 3 is met on the pre-cost series.
So "passes 15.6% of strategies that lose money after cost" is accurate as measured, but the reason is that the t-test never sees the cost. My sentence saying the tool "subtracts 0.2% per trade as extra cost" was too loose, and I should have described this.
Raw output from the locked run (500 datasets each, 2,000 permutations):
E3, true edge 0.15%: all criteria passed 78/500 = 15.6%
E4, true edge 0.20%: 252/500 = 50.4%
E5, true edge 0.30%: 491/500 = 98.2%
N1 (no edge): the "after cost > 0" criterion alone passes in 0.2% of datasets.
I will change the tool so the t-test and p-value use after-cost returns, rerun the calibration under a new locked protocol labelled as version 2, and add a correction note to the article. Thanks again.
Rerunning it with the cost taken out first and telling me where the 0.2% was applied was the useful part, and it makes the v2 numbers something people can trust. If canli-mcp or the validation tools are useful to you, a star on github.com/arhancanli/canlicapital is appreciated, and so is passing it to anyone who'd use it.