How backtesting works

The barrier model

Every strategy ends in a Stop / target exit. When the entry opens a position, the engine draws two horizontal barriers around the entry price and watches which one price touches first:

• A stop below entry, placed stop_atr × ATR away.
• A target above entry, placed target_atr × ATR away.
• A horizon — after horizon bars with neither hit, the trade is closed at market.

This is the triple-barrier method: each trade has exactly three possible outcomes (target, stop, or time-out), so a backtest reduces to a clean sequence of win/loss/scratch labels. Distances scale with ATR so the same rule behaves consistently across calm and volatile regimes.

entry @ 100.0,  ATR = 2.0
  target = 100 + 2.0×2.0 = 104.0   ┐
  entry  ──────────────────────────┤  whichever price touches first
  stop   = 100 − 1.5×2.0 =  97.0   ┘  decides the outcome
  ...if neither in 36 bars → exit at market (a "scratch")

Costs are charged on every trade

Slippage and commission are real and they kill marginal edges. The engine subtracts cost_atr × ATR from every trade. That is why the headline number is expectancy net of costs — a strategy that’s profitable gross but not net has no edge you can actually trade.

DEV vs. locked HOLDOUT

Here is the single most important design decision in the Lab. The historical data is split into two slices the moment you run:

• DEV — the development slice. Iterate here freely: tweak params, swap bricks, re-run as much as you like.
• HOLDOUT — a locked, out-of-sample slice. You never tune against it. It exists to answer one question: does the edge survive on data you didn’t fit to?

The results drawer shows your stats on three slices so the comparison is unavoidable:

           n      win     rr      expectancy (net)
all      1081   42.9%   1.16    -0.108
dev       718   42.3%   1.16    -0.126   ← what you tuned on
holdout   363   44.1%   1.16    -0.075   ← what actually matters

verdict:  NO EDGE net of costs

A healthy strategy shows a positive expectancy that holds up from Dev into Holdout. A big positive Dev number that collapses on Holdout is the classic signature of overfitting — you fit the noise, not the signal.

Why a positive backtest can still be noise

Imagine flipping 1,000 coins ten times each. A handful will land heads 9 or 10 times — not because they’re special coins, but because you flipped a thousand of them. Strategy search is exactly this. Every param you nudge, every brick you swap, is another coin. Try enough variants and one will look brilliant on Dev by chance alone.

The overfitting guards

On top of the holdout, the Lab reports two statistics built specifically to catch selection-driven luck. They turn “trust me, it’s good” into a number.

PBO — Probability of Backtest Overfitting

PBO estimates the probability that the configuration that looked best in-sample will actually underperform out-of-sample. The method (combinatorially symmetric cross-validation) repeatedly splits history into in-sample and out-of-sample halves, picks the in-sample winner each time, and checks how often that “winner” lands in the bottom half out-of-sample.

• PBO near 0% — the in-sample best tends to stay good out-of-sample. Encouraging.
• PBO near 50% or above — your selection process is a coin flip. The “best” strategy is just the luckiest one; expect it to disappoint live.

Deflated Sharpe Ratio

A raw Sharpe ratio overstates skill for two reasons: you tried many strategies (so the maximum is inflated), and returns aren’t perfectly normal (fat tails and skew flatter the number). The Deflated Sharpe Ratio (DSR) corrects for both. It asks: given how many variants were tried and the shape of these returns, what’s the probability the true Sharpe is actually above zero?

• It deflates the observed Sharpe by the number of trials — more attempts, bigger haircut.
• It adjusts for skew and kurtosis, since lumpy, fat-tailed P&L is less trustworthy than smooth P&L.
• A DSR comfortably above zero (high confidence) is real evidence; a DSR near zero means the impressive Sharpe is probably an artifact of searching.

The verdict

The headline verdict is the engine folding all of this — holdout expectancy net of costs, plus the overfitting guards — into one honest, plain-English string, for example:

"NO EDGE net of costs"      → flat/negative once costs are paid; do not trade.
"edge confined to DEV"      → looked good in-sample, didn't survive holdout.
"edge holds out-of-sample"  → survived holdout and the guards. Promising — not a guarantee.

Even the best verdict is a hypothesis that has not yet been falsified, not a promise of profit. Markets regime-shift and edges decay. Treat a green verdict as permission to forward-test small and keep watching — never as a green light to size up.

A sane workflow

1. Form a hypothesis before you build — a reason the edge should exist.
2. Build it, iterate on Dev only.
3. Keep your number of attempts honest — every tweak counts against PBO and DSR.
4. Look at Holdout once, to decide go / no-go.
5. Check PBO and DSR. If they’re weak, walk away — don’t “fix” the holdout.
6. If it survives, forward-test small in The Trader before risking real size.

Stuck on any of this? The AI assistant can explain PBO, the Deflated Sharpe Ratio, and the Dev/Holdout split in context — it’s briefed on exactly the model described here.