the method
six ways a backtest lies.
none of them feel like lying.
Every one of these is an ordinary, defensible-looking step taken by a competent researcher in good faith. That is what makes them dangerous: none of them announce themselves, and each one alone is survivable. Scientia's position is that a result which has not been checked against all six is not a result, it is a hypothesis with a chart attached.
candidates evaluated
630
failed at least one gate
628
99.5%
gates, all mandatory
6
no partial credit
epochs run
1
lookahead
the strategy knew something it could not have known
how it happens
A signal computed on the closing price and filled at that same close. A fundamental restated three quarters later, read as if it had been published on the day. An index membership list downloaded today and applied to 2019. None of these look like cheating in a notebook — they look like a join.
what the crucible does
The replay walks the tape forward one day at a time. A rebalance at t reads the close at t and at t−skip−lookback and nothing else, and positions set on day t are marked to market on day t+1 — never the reverse. There is no join to get wrong, because there is no join: there is an index that only moves in one direction.
survivorship
you only tested on the ones that made it
how it happens
The universe is built from what is listed today. Everything that went to zero, got delisted, halted permanently or was quietly wound down is missing — and those are precisely the names your strategy would have been holding.
what the crucible does
The tape carries 6 products Coinbase has since delisted — WLUNA above all, which went to zero in May 2022. The universe on any day is whatever actually had a bar that day, so a collapsing name is held all the way down and marked out at the last price it printed. Every specimen is also re-run on a universe rebuilt from what is still listed today; the median difference is 8 points of return, and that difference is the gate.
overfitting
you tried four hundred things and reported one
how it happens
Search a large enough space and something in it will have a beautiful Sharpe by luck alone. The number is real; the inference from it is not. Reporting the winner without reporting the search is the most common lie in the field, and usually an honest one.
what the crucible does
The grid is exhaustive, so the trial count is not a claim: it is 630, and it is charged against every in-sample Sharpe on the board. PBO is estimated by combinatorially symmetric cross-validation over the whole grid and comes out at 0.60 — on three fifths of the splits, the configuration that won in sample ranked below the median out of it.
fill fantasy
you assumed the mid, instantly, in full
how it happens
The backtest fills the whole order at the printed price the moment the signal fires. Real orders join a queue, get partially filled, cross a spread, and move the price they are trying to take.
what the crucible does
Every rebalance pays half the spread, the venue's taker fee, and square-root market impact measured against that name's own dollar volume on that day. A larger book pays more for the same signal, which is why capacity is a number here and not an afterthought. The Crucible page lets you drag those inputs and watch the edge go.
regime luck
one bull run flatters everything
how it happens
A strategy fitted across a period containing a single enormous trend has learned that trend, not a mechanism. Overlapping labels leak information across the train/test boundary and hide it.
what the crucible does
Train, embargo, test — and then a further 90 days held back from selection entirely. The embargo is wide enough to contain the longest lookback in the grid, so a signal straddling the boundary cannot carry information across it. Every number on the leaderboard is from the window after the gap.
cost blindness
the friction was left out because it was boring
how it happens
No fees. No funding. No borrow. No slippage. Each omission is individually defensible and collectively fatal: they are paid every single trade, and a high-turnover strategy pays them thousands of times a year.
what the crucible does
Costs are charged into the curve on the day they occur, not netted off at the end where they round away. Each specimen is also re-run with every friction switched off, so the gap between gross and net is measured rather than estimated — 3 of 10 are positive before costs and negative after them.
all six, or nothing
a result that has not cleared these is a hypothesis.
Failing one gate is disqualifying. There is no weighted score, no overall grade, and no way to trade a strong number on one axis against a failure on another — because in production the failure is what happens to you.