Backtesting, Calibration and Honest Model Scores

A backtest is a controlled historical question

A backtest applies a defined rule to historical data and records what would have happened under explicit assumptions. It does not recreate every spread, queue position, halt, fee or decision a person would have faced. Its value comes from consistency and transparency, not from producing an attractive chart.

The symbol universe, dates, entry timing, exit rules, corporate-action treatment and unavailable-data policy must be fixed before results are interpreted. Changing them after seeing outcomes turns research into selection.

Preventing information leakage

Every feature must be available at the simulated publication time. A final daily close cannot be used for a decision supposedly made before that close, and a swing cannot be treated as confirmed before the later candles required to confirm it existed. Revised fundamentals and constituent lists can leak future knowledge in subtler ways.

Walk-forward testing trains or selects parameters on an earlier window and evaluates them on a later untouched window. Repeating that process across time gives a more realistic view of stability than fitting once to the entire history.

Score versus probability

A heuristic score ranks agreement among configured features. It becomes a probability only after calibration against independent outcomes with enough coverage. Calibration asks whether events assigned a similar probability occur at approximately that frequency, not whether the average return was positive.

Small samples require wide uncertainty. Showing 80% after four historical examples suggests precision that the evidence cannot support. Trading.Red labels heuristic conviction separately from historical checks and should display unavailable when a calibrated estimate is not defensible.

Point-in-time publication records

A trustworthy archive saves the ticker, direction, observed price, target, stop, model version, input timestamp and publication time. Closed scenarios retain the price at which their rule closed rather than being recalculated from today's price. This prevents historical results from drifting as live data changes.

Failures belong in the archive alongside successes. Removing weak trades, replacing yesterday's candidate or ranking only surviving symbols creates survivorship and outcome bias.

Metrics that reveal different weaknesses

Win rate alone ignores payoff size. Average return ignores path and tail loss. Expectancy combines win frequency and payoff but still depends on execution assumptions. Drawdown, coverage, turnover, calibration error and results by regime provide complementary checks. No single metric establishes that future performance will match the past.