How a crown is earned
A model is state of the art in a category on a date when it leads the approved primary evidence beyond the margin of error. The verdict is always model × category × date. Reasoning-effort settings are configurations of one model. A harness does not make the base model SOTA.
Verdict states
Three automatic safeguards
- A crown change needs two consecutive snapshots of a primary source, or two independent primary sources. The first holder of an empty category does not wait.
- If primary sources disagree on the leader, the verdict is low-confidence co-SOTA. The engine does not pick a side.
- Badges never say NOT SOTA. They show the best rank, such as #14 · Coding.
Self-reported vendor numbers are trust 0.5 and cannot be the only basis for a crown. SWE-bench Verified and METR time horizons are display-only. Benchmark versions are not mixed. A slice such as open weights or ≤32B needs at least five eligible models before a Yes can be the headline. A meaningful slice Yes is never published as a bare No.
Sources in this run
- LMArena leaderboard dataset — CC BY 4.0
- Epoch AI benchmarking hub — CC BY 4.0
- OpenRouter models — OpenRouter API terms
- Epoch AI notable models — CC BY 4.0
- Artificial Analysis — Artificial Analysis Data Platform Terms (derived verdicts only)
- MTEB leaderboard — MTEB project (MIT); official leaderboard API
- Open ASR Leaderboard — Hugging Face Open ASR leaderboard results
- Terminal-Bench Harbor Hub — Official Terminal-Bench leaderboard via the public Harbor read API
- Scale Labs leaderboards — Published Scale Labs leaderboard
Not in the numbers
- livebench kept LiveBench/version LiveBench-2024-11-25 (1 versions seen).
- Artificial Analysis was skipped because ARTIFICIAL_ANALYSIS_API_KEY is unset. Intelligence, coding, agentic, and speech-arena scores are not used. Text-to-speech has no crown until that key is set. Raw scores are never written to the public catalog.
LMArena’s leaderboard dataset and Epoch AI’s benchmark tables are used under CC BY 4.0. MTEB is an MIT project and the official leaderboard is stored as published. Terminal-Bench 4.0 comes from the public Harbor board. SWE-bench Pro V2 HARD is the Scale Labs public board. Open ASR word-error rates come from the Hugging Face leaderboard export. Artificial Analysis raw scores are not redistributed: pages and the API show the derived verdict and a link. Based on data from Artificial Analysis. isitsota is not endorsed by Artificial Analysis.