Evidence Deflator
GitHub
Evidence checks for AI trading

How much of an AI trading result actually counts?

AI agents now write, backtest and rank strategies in minutes. Two things almost nobody reports: what the model may already have seen, and how many attempts the winner was picked from. Evidence Deflator turns both into numbers.

See the October leaderboard → Check a backtest
Liquidity Arena 2026 Track A, Phase I · each dot is one team’s t-statisticFull board →
0 / 41teams reach even t = 2
3.26bar for the best of 41
1.63closest: Kinance, rank 11 by return
~5 mobefore a winner could be named, at a true Sharpe of 5
The method

Three numbers, before any performance figure

Each is plain arithmetic on inputs a project can state in one line. None needs the strategy’s code.

Read the method →
01 · OPEN-BOOK MONTHS

What the model may have seen

Months of the test window on or before the model’s training cutoff. For those months, we cannot rule out that it already knows what happened.

02 · CLEAN MONTHS NEEDED

How long the result must run

A Sharpe of 1 needs about 48 clean months to reach t = 2; a Sharpe of 2 needs 12. The quadratic is the whole story.

03 · BAR AFTER SELECTION

How many tries it beat

The best of N attempts must clear a higher bar. Best of 41: t ≥ 3.26. Best of 23,376: t ≥ 4.76.

Worked example

A 2020–2025 backtest, model cutoff 2024-10

66 months in the window. 58 cannot be ruled out as seen. 8 are clean, against the 48 a Sharpe-1 claim needs. Verdict: underpowered — it rejects nothing, and it shows nothing.

58 open-book8 cleanneeds 48
python tools/window.py --cutoff 2024-10 --start 2020-01 --end 2025-06
Latest

Published checks

Every number recomputed from public sources, dated, with the raw data kept.

All audits →
2026-10-13Leaderboard No. 1Liquidity Arena and HKUDS AI-Trader, rescored by evidence instead of return.Leaderboard2026-10-12Audit 005 · Nof1's "proto scaling law for financial intelligence"Cannot assess: the claim is public, the evidence is not.Audit2026-10-12Audit 004 · TradingAgents run_backtestData-level look-ahead is handled carefully; weight-level look-ahead is not checked. A 2024–2026 run with the default model is 88% open-book.Audit2026-10-12Audit 003 · Liquidity Arena 2026 Track A, Phase I (41 teams)No team reaches t = 2; the highest is 1.63. The format is closed-book; the window is too short.Audit2026-10-12Audit 002 · MetisAI's Liquidity Arena resultLiterally true, but the comparison set was narrowed: Sharpe 4th of 41, drawdown 2nd worst, t ≈ 1.2 against a bar of 3.26.Audit2026-10-0517 published LLM trading papersOf the 9 with a documented cutoff, none reaches 48 clean months.Paper audit
Who it is for

One check, three places it belongs

Agent frameworks

One call between “backtest” and “deploy”: flag runs whose window sits before the model’s cutoff, and charge the result for every variant tried.

Python, CLI and MCP →

Competition operators

Show a t-statistic next to return, and the date a winner could first be named. Carry records across phases so evidence accumulates.

See the leaderboard →

Allocators and reviewers

Send a deck, a backtest or a public claim. Get a one-page report: what is open-book, what is underpowered, what is missing.

Request an audit →

Use it

Pure Python, no keys, no network

The engine is the standard library. Fetching data is your job; the value is a repeatable, comparable answer.

API reference →
from evidence_deflator import effective_window

w = effective_window(cutoff="2024-10", start="2020-01",
                     end="2025-06", trials=41)
print(w.open_book_months, w.effective_months, w.months_required, w.verdict)
# 58 8 128 underpowered