Your agent ran N backtests. What is the best one worth?

An agent that searches for a strategy reports its best result. The number of tries behind it is the one fact the report leaves out. In an agent loop it is also the one fact that can be counted from the log, so the correction for picking the best no longer depends on what anyone remembers.

Load a real example CICC’s factor loop: 16,939 candidates tested, best single factor Sharpe 2.74 over 1.51 years. Audit 014.

How the number is worked out

t = Sharpe × √years. A result picked blind needs t ≥ 2. The best of N tries needs the t that luck alone would reach less than 2.3% of the time across all N, so the bar rises with N: about 3.4 for 69 tries, 4.7 for 16,939. Tries that resemble each other are fewer independent chances than the count says, so this bar is the strict end; the result is still only as honest as the count. The bar after selection.

Count the tries instead of asking for them

• Claude Code plugin. A hook sees every backtest command the agent runs, keeps the count for the session, and tells the agent in context what the best result now has to clear. /plugin marketplace add Beltran12138/evidence-deflator, then /plugin install evidence-deflator@evidence-deflator.
• Any agent’s log. tools/trace_audit.py reads a Claude Code or Codex session log and counts tries under each definition you give it (script runs, hypotheses, verdict lines). The counts often differ by an order of magnitude, and it reports the spread rather than one number. It also checks whether the held-out data was opened before the plan was written down.
• Where the count came from. The Python package keeps the provenance next to the number: a count a person declared, one derived from a grid, or one read from a log. Only the last can be checked.

Two more inputs: window and training cutoff

A backtest run by a language model over months before the model’s training cutoff cannot rule out that the model has read about what happened. Only months after the cutoff are clean. Check a backtest splits the window for you.

For agents that pick traders

An agent about to copy a trader or a vault can ask first: the hosted MCP tool hyperliquid_wallet(address, picked_from) and a JSON endpoint return the same verdict as the wallet check, with a clears_bar field for a fixed rule. API and MCP.

To use it as a gate, the Python package turns a verdict into allow or deny: python -m hl_wallet 0xADDRESS --picked-from N --gate exits 0 only for a record that clears the bar for how it was picked, and denies a record too short to score, so “cannot tell” means a limit of zero.

Register before it happens

A record found by looking back was picked. Register an agent’s wallet and only what it does afterwards counts, timestamped by GitHub and listed whatever it shows. Forward registry.

What this does not do

It does not judge whether a strategy makes sense, run your backtest, or ask a language model to grade anything: every verdict is arithmetic on numbers you can check. Over a short window, t measures consistency, not the loss that has not happened yet.