I built a Python paper-trading bot for crypto spot markets. The backtest looked fine. Then I tested it the strict way, and it fell apart. This is w...
For further actions, you may consider blocking this person and/or reporting abuse
The cleanest line in this post is the placebo table: random entries placed after the bot's entry average -0.063 R, the bot averages -0.068 R. That says the bot is indistinguishable from entering at random after the same rise, which is a stronger statement than "slightly negative signal".
One check on the 81/19 split. Treating the 3,314 trades as independent, the 25.3% win rate has a standard error of about 0.76 points, so the interval is roughly 23.8% to 26.8% and the 27.5% breakeven sits outside it. But you note the independent events number in the tens, and trades that open on the same day share one market move. If I read that right, the interval on the pre-cost -0.07% per trade is much wider than the 3,314 suggests, and the 19% "signal" share could have either sign. Resampling whole days (or whole episodes) with replacement and reporting the interval on the pre-cost mean and on the bot-minus-placebo difference would show whether the loss is cost plus nothing, or cost plus a real negative edge.
Do you resample by day in the tool, or does it treat each trade as one draw?
Thank you, this is exactly the check I should have shown. I reran it on the same 3,314 trades, resampling whole entry days (393 clusters) instead of single trades. Pre-cost mean per trade: -0.07%, 95% interval roughly -0.56% to +0.41%. Net mean: -0.37%, interval roughly -0.86% to +0.10%. So you are right: the sign of the pre-cost signal can't be determined, and even the net loss isn't separated from zero at the day level. What stays solid is the cost side, about 0.30% per trade, and the placebo comparison (random-after matches the bot). The 81/19 split I wrote is not supportable at these intervals, so I'm correcting the post. To your question: the tool already aggregates trades to one value per date and runs the t-test and permutation test on those, but it does not report a day-resampled interval yet. I'm adding that. This rerun was done after the fact, not pre-registered, and I'll label it that way.
Good, that's the honest reading. One more layer on the day-level interval: 393 clusters treats adjacent days as independent, and trade outcomes in a single bot often aren't. Volatility regimes and a market that trends or chops for a week move many days together. A block bootstrap by week (or by month, if you have enough of them) will usually widen the interval again, so I'd report it next to the day-level one and say which you trust.
It also tells you how much data a verdict needs. With a half-width around 0.5% per trade and a cost drag of about 0.30%, the interval has to shrink to roughly 0.15% before the pre-cost sign or the net sign is separable, and half-width falls with the square root of the number of independent clusters. That is about 10x the current clusters. Worth writing in the post as the stopping rule, so a later rerun can't be read as a result found by looking longer.
Agreed on all points, and thanks for the arithmetic. I ran the weekly block bootstrap too (73 weekly blocks): pre-cost mean -0.07% per trade, roughly -0.60% to +0.44%; net mean -0.37%, roughly -0.90% to +0.14%. Practically the same as the day-level result, so adjacent-day dependence is not changing the conclusion here. Monthly blocks I skipped, there are too few months for the interval to mean anything. Your point about the half-width is the key one: at about 0.5% per trade I could not separate even a 0.30% cost drag from noise, and the interval would need to shrink about 3x, so roughly 10x more independent blocks. That means this history cannot confirm or reject a small edge, only rule out a large one. I'm adding both a block-resampled interval and a "minimum detectable effect" line to the tool's report, so the report says how much data a verdict needs. And yes, writing the stopping rule down before looking is exactly the point of the lock step.
Weekly and daily agreeing is the useful part: it says the width comes from per-trade noise, not from clustering, so no cleverer resampling scheme will rescue the interval.
On the minimum detectable effect line, I'd define it from power rather than from the interval. A 95% half-width of about 0.52 is a standard error of about 0.27. For 80% power at a two-sided 5% test the detectable effect is roughly 2.8 standard errors, so about 0.75% per trade, more than twice the 0.30% cost drag. Getting that down to 0.30% takes about 6x the blocks, roughly 450 weeks, a bit under the 10x I quoted earlier because that figure asked the whole interval to clear the drag rather than just reach 80% power. Both are worth printing, with the stated assumption that future weeks behave like these 73.
Checked your arithmetic and I get the same numbers: half-width 0.52% -> SE about 0.265%, 2.8 SE about 0.74% per trade, and about 6x the blocks (roughly 450 weeks) to bring it down to the 0.30% cost drag. So this history can rule out a large edge, but it cannot confirm or reject a small one.
One honest gap on my side: the tool's minimum detectable effect line is currently computed from the date-level standard error, and the week-level interval is printed separately. Here the two agree, but on other data the week-level one can be wider, and then the line would be too optimistic. I'm changing it to use the wider of the two and to print the assumption (future blocks behave like the sample) next to it.
Stopping rule for a rerun: no rerun before 6x the current independent blocks, and no change to the criteria in between. Thanks for the careful reading, this thread is better than the post.
I would include the simulator version and execution settings in the locked manifest as well as the thresholds and data. A small negative test can keep the criteria file unchanged but alter the fee setting or the engine commit; the runner should refuse that run. Otherwise a matching criteria hash can hide a different experiment underneath it.
Thanks, this is a good point. For the validator itself it is already covered: the lock stores SHA-256 hashes of the protocol file (thresholds, extra cost per trade, seed, number of permutations), the trades file and the script itself, and the run refuses to start if any of them changed.
What it does not cover is the simulator that produced the trades. Its version and fee settings sit outside the lock; only its output, the trades file, is locked. You are right that this leaves a gap. I plan to add engine version and fee setting as manifest fields and make the runner refuse a mismatch.
Yes, that closes the gap. I would also reject an older trades file with no engine fields, rather than supply defaults. Keep two small checks: change only the engine version, then only the fee setting. Each should stop before a result is reported.
Agreed on both. A trades file without engine fields should be rejected, not filled with defaults, because a silent default is exactly the gap we are talking about. And the two small tests are a good acceptance check: change only the engine version, then only the fee setting, and each run must stop before any result is printed. I will write them down as test cases for the tool. Thanks for the careful review.