I built a ghost detector for backtest edges. It was 99% confident and 80% right.

Every strategy in my archive looked alive once. A few still do. The rest are ghosts: equity curves that fell apart when the data changed, parameters that were tuned on the same bars they were judged on, an edge that existed only inside one sample.

I wanted a detector that answers one question — given only what you could have known in-sample, how likely is this edge to survive? — and, more importantly, a detector honest about its own confidence. The second part turned out to be the hard one.

What I Built

ghostscore is a small CLI. You give it a strategy's in-sample statistics; it returns a probability that the edge survives out of sample. It is trained entirely on synthetic markets where the answer is known, then applied to real configs from my archive.

$ python3 -m ghostscore.cli real --n 900 --seed 7
spearman(ghost_score, OOS Sharpe) = 0.512   p = 0.0045   n = 29
top-5 mean OOS Sharpe 0.482 · bottom-5 mean OOS Sharpe 0.220

On 29 real configs — MACD variants, Donchian, Bollinger, RSI, time-series momentum, AO, each long/short with and without an EMA200 gate — the ranking correlates with realised out-of-sample Sharpe at 0.51. Not a crystal ball. Enough to be useful as a filter.

The most instructive row is the one that looks best in-sample:

Donchian (20) has the highest in-sample Sharpe of the entire set and a survival probability of 0.101. That is the ghost the tool is for.

Demo

A real terminal recording, not a mock-up. This clip runs with --n 300 to keep it short, so its holdout line (AUC 0.860, ECE 0.178) differs slightly from the numbers in this post, which come from --n 900. Regenerate either with scripts/record_demo.py or ghostscore.cli real.

Code

github.com/rytjy/ghostscore — MIT, Python, no API keys, CPU only.

How I Built It

A generator where the truth is known

Three market regimes:

• noise — iid returns, nothing to exploit but costs;
• trend — positively autocorrelated returns, momentum pays;
• revert — negatively autocorrelated, momentum bleeds.

Returns are Student-t (fat tails) with occasional negative jumps, because Gaussian innovations put every real config outside the training support — more on that below. Half the strategy population uses an EMA200 gate, matching the archive and producing the long flat stretches that real PnL series actually contain.

A strategy is sign(trailing k-bar return), filled at the close, costing 1/2/5 bps per edge. In-sample length is sampled from 900–1,900 bars so the model sees the length of the real configs (~1,094).

Labels that are not one noisy draw

This was my first failure. Using a single out-of-sample window as the label gave AUC 0.489 — chance. A 480-bar OOS Sharpe has a standard error near 1.0; the label was mostly noise. The correlation between in-sample and out-of-sample Sharpe was −0.02.

The fix: average OOS Sharpe over three independent continuation paths and strengthen the regimes. Same data, same features, back to 0.85.

Features that survive scale changes

Eleven in-sample features, all scale-free: Sharpe, skew, kurtosis, hit rate, turnover, max drawdown, profit factor, t-stat, lag-1 autocorrelation, Sharpe standard error, bar count. Absolute mean and volatility are deliberately excluded: synthetic markets run at 1% daily vol, crypto runs at 3–5%, and a tree model reads that difference as a regime.

One report, one named backend

TabPFN is the intended classifier — a tabular foundation model, few-shot, CPU, no training loop. A gradient-boosting fallback keeps the demo runnable. Every report prints backend, so a number can never be attributed to the wrong model.

The calibration curve is the honest part

Synthetic holdout, n=270:

AUC 0.850 · Brier 0.180 · ECE 0.141 · base rate 0.400

When it says "almost certainly real" it is right 80% of the time, not 99%. When it says "almost certainly a ghost" it is right 14% of the time, not 1%. Ranking and calibration are different abilities; this tool is good at the first and overconfident at the second. That gap is the finding.

The second failure: every real score was 0.000

The first application to the archive returned 0.000 for all 29 configs. Two causes:

1. max_dd divided by a peak that can sit near zero, producing values like −4.8e11;
2. Gaussian innovations (kurtosis ≈ 3) against real crypto kurtosis of 10–240.

Fixes: multiplicative drawdown bounded to [−1, 0], Student-t innovations with negative jumps, and the EMA200 gate in half the synthetic population. After that the scores spread from 0.003 to 0.865 and the rank correlation appeared.

I kept both failures in the repository history. A detector that was chance, then zero, then useful is a more honest artefact than one that arrived useful.

Why Does Open Innovation Matter?

The data never leaves. The generator, the classifier, the configs — all local, CPU only, no API key. A tool that audits my strategies should not require uploading them to someone else's service.

The generator is auditable. Every label comes from code anyone can read, not from a vendor's "AI score". You can disagree with my regimes and rerun the experiment.

The failure modes are visible. A closed product would show the AUC and not the calibration table. Open code makes the overconfidence checkable.

The Spirit of It

I keep a folder of dead configs. Some were killed by costs, some by a regime that ended, one by a drawdown I did not see because my accounting was wrong. This project is an ofrenda for them — not a memorial, a post-mortem. The detector's job is to say, before the money is spent, which of the living are already ghosts.

The Donchian row is why I trust it more than its AUC. It had the best in-sample Sharpe I have ever produced, and the tool looked at it and said: probably a ghost.

About the Model, Honestly

No partner category is claimed. The TabPFN path is implemented in the repository, but its install failed on this machine (repeated read timeouts from the package index), and every number in this post comes from the gradient-boosting fallback — the reports print backend=sklearn-hgb for that reason. Claiming a partner category on a backend I did not run would be exactly the kind of in-sample story this tool is built to catch.

All numbers are reproducible from the repository: python3 -m ghostscore.cli real --n 900 --seed 7 and python3 scripts/plot_calibration.py results/holdout.json. Written with an AI assistant; every figure comes from code in the repo.