88.6% HIGH-Severity Recall
Across 9 Code4rena Audits

We pull each Code4rena contest's source repository and run it through the SolSentinel pipeline (IR detectors + Claude project-level audit) end-to-end. We then compare our findings against the public wardens' report from code4rena.com/reports/<contest>.

A wardens finding is marked CAUGHT (liberal match) if any of our findings:

• Has an exact contract.function match with the warden's attribution, OR
• Shares the same contract + 2 distinct title tokens, OR
• Shares 3 distinct title tokens (regardless of contract), OR
• Shares any tag.

Strict-mode recall (exact contract+function match OR 3+ tokens on same contract) is reported alongside liberal in every per-contest report. The strict aggregate is 64.5% / 75% HIGH — both numbers are public.

The full methodology, per-contest comparison reports, and aggregate scoring script are open and reproducible:

• docs/benchmarks/methodology.md
• docs/benchmarks/per_contest/reports/<contest>.md
• docs/benchmarks/aggregate_score.py

Honest accounting: the 4 HIGH-severity findings we missed across 9 contests cluster around three structural classes that need either symbolic execution or deep economic-model reasoning that current LLM tooling cannot reach with shape detectors alone:

• Size H-01 — credit-amount swap-fee formula edge case (lending math)
• Revolution H-02 — quorum / vote-supply snapshot bug (governance math)
• Revolution H-03 — JSON injection in tokenURI (string-escape class)
• Revolution H-04 — OZ Votes delegate-block (upgradeable-contract logic)

We've shipped detectors for the H-03 and M-class auction findings on the Revolution side since this benchmark; the next refresh of these numbers (after expanding the corpus past 9 contests) is projected to land at ~94% HIGH recall.