The easiest way to publish a false positive in time-series work, with a worked example from my own project.
The result that looked publishable
A walk-forward test, 913 causal anchors, direction accuracy at a 30-minute horizon.
Benchmark note that matters: the thing to beat is not 50%, it is the majority class. If 51% of moves were up, "always guess up" scores 51%. Scoring against 50% is how a slight directional bias gets mistaken for signal.
Result: 55.2% against a 51.1% majority class. +4.1 pp on n=912, about 2.4 standard errors.
Why it was not real
Anchors were spaced about 7 minutes apart. The horizon was 30 minutes.
A forecast at 10:00 covering to 10:30, and one at 10:07 covering to 10:37, share 23 of their 30 minutes of outcome. Those are not independent observations. n=912 was a much smaller number of distinct market episodes, counted repeatedly.
Standard error scales with the square root of n. Inflate n artificially and you shrink your error bars artificially.
Re-scored on non-overlapping anchors
Overlapping: n=912, majority 51.1%, model 55.2%, edge +4.1 pp, 2.4 s.e.
Non-overlapping: n=163, majority 54.0%, model 59.5%, edge +5.5 pp, 1.4 s.e.
The edge grew. The significance nearly halved. The inflation factor was about 2.4x, which is the square root of 912/163.
The trap is that the effect size grows. If it had collapsed to zero the result would be obviously dead. Instead it improves, which tempts you into "the real effect is stronger, I just need more data." A larger estimate on a much smaller sample is what noise looks like.
The check: compare anchor spacing to forecast horizon. Spacing smaller than horizon means overlap. Either re-score on non-overlapping anchors or space them a full horizon apart from the start.
This applies to any sliding-window evaluation where the window is wider than the stride. Sensor data, load forecasting, anomaly detection, medical monitoring.
https://michaelhairetis.medium.com/how-overlapping-windows-manufactured-my-best-result-a85e076865b6
Top comments (1)
Same thing bit me with trades instead of windows. My partner's futures fills: 88 trades in the holdout, often a dozen on the same day. Scored trade by trade, a fixed bracket beat her own exits and the count looked healthy. But trades on the same morning share the same tape, so 88 was really closer to "a handful of days, counted many times."
What I did was resample whole days instead of single trades. The bracket came out ahead in 66% of resamples. That's barely better than a coin flip, a much weaker claim than the per-trade number suggested, and the one I actually published.
For October I wrote the rule down before the data exists: one day is one observation, sign test on daily differences. Your 7-minute vs 30-minute check is the cleaner general version of the same question: what's the real unit of independence here?