Audits

One page per claim. How much of the evidence counts?

Each audit takes a public AI trading claim and asks for three inputs: the model’s knowledge cutoff, the test window, and how many attempts the result was picked from. Where a project does not publish them, the audit says so.

Request an audit →Free, public, usually within 48 hours.

001 · 2026-10-12 Catalyst (AI trading agents for retail)Cannot assess: none of the three inputs is public. The only published number is trading volume, not performance. Cannot assess 002 · 2026-10-12 MetisAI's Liquidity Arena resultLiterally true, but the comparison set was narrowed: Sharpe 4th of 41, drawdown 2nd worst, t ≈ 1.2 against a bar of 3.26. Narrowed comparison 003 · 2026-10-12 Liquidity Arena 2026 Track A, Phase I (41 teams)No team reaches t = 2; the highest is 1.63. The format is closed-book; the window is too short. 0 of 41 reach t = 2 004 · 2026-10-12 TradingAgents run_backtestData-level look-ahead is handled carefully; weight-level look-ahead is not checked. A 2024–2026 run with the default model is 88% open-book. 88% open-book 005 · 2026-10-12 Nof1's "proto scaling law for financial intelligence"Cannot assess: the claim is public, the evidence is not. Cannot assess Case audit · 2026-10-06A “22% chance of a nuclear detonation” on PolymarketFive discounts on one prediction-market price: about 16 effective wallets, and a contract that also counts tests.Wrong question Paper audit · 2026-10-0517 published LLM trading papersOf the 9 with a documented cutoff, none reaches 48 clean months.0 of 9 reach 48 months

Rules every audit follows

Sources are public and fetched on the stated date; numbers are recomputed, not copied. “Cannot be ruled out as seen” is the strongest claim made about a model’s training data, never “contaminated”. Findings state what the numbers can and cannot support, with no judgement of intent. Short is not the same as wrong: an underpowered result has not been shown to fail, only not shown to work.