SOTA glossary

Definitions for the verdict states, generated from methodology v1.0.

• SOTA — Under methodology v1.0, SOTA means the model leads the approved primary evidence in a category. A later crown needs 2 consecutive snapshots or 2 independent primary sources.
• co-SOTA — Under methodology v1.0, co-SOTA means more than one model shares the lead, including when primary sources conflict. Those models share position 1.
• near-SOTA — Under methodology v1.0, near-SOTA means rank 1–3 or within 4 percentile points of the leader, without holding the crown.
• was SOTA — Under methodology v1.0, was SOTA means the model held the crown and a later confirmed change closed that interval.
• pending — Under methodology v1.0, pending lasts 7 days after release and only for a crown contender: rank 1–3, or within 4 percentile points of the leader. The public line is Leading, crown pending confirmation.
• style control — Style control is version style-control on 6 boards, including LMArena Text (style-controlled, overall). The metric is Arena score.
• Elo/Bradley-Terry — LMArena publishes Arena score on 11 boards. That score is the preference rating these pages cite for Elo and Bradley-Terry style comparisons. Methodology v1.0.
• pass@1 — pass@1 is the metric on SWE-bench Pro V2 HARD. higher is better. Ceiling 100. Methodology v1.0.