AI benchmarks
The leaderboards frontier labs actually race against — coding, reasoning, math, general knowledge, and human preference. Scores are snapshots; check the source for the live leaderboard.
Francois Chollet's Abstraction and Reasoning Corpus. Grid-based visual puzzles designed to be trivial for humans but hard for machines that memorised the internet. The most publicised general-intelligence benchmark of 2024-25.
23 tasks from BIG-Bench that were hard for models at the time. Multi-step reasoning across NLP, math, and symbolic tasks.
Graduate-level Google-proof Q&A — 198 physics/chem/bio questions written by domain PhDs and verified to be non-Google-answerable. Expert baseline is ~65%.
Grade-school math word problems. 8500 human-generated arithmetic reasoning tasks. Now saturated at the frontier — replaced by MATH-500 and AIME for hard-math evaluation.
OpenAI's 164-problem Python coding benchmark. Model must complete a function given its docstring. Nearly saturated at the frontier — kept for historical continuity.
Blind pairwise human preference voting on real-world chat prompts. Rankings computed via Bradley-Terry. The dominant industry preference signal for chat quality.
Massive Multitask Language Understanding — 57 subjects across humanities, sciences, and professional exams. Now saturated at the frontier; MMLU-Pro is the current standard.
Successor to MMLU with harder questions and 10 answer choices per question (vs. 4). Actively used for frontier-model differentiation.
Multi-turn benchmark with 80 open-ended questions across 8 categories, judged by GPT-4. A quick chat-quality proxy before Arena Elo became dominant.
Real-world software-engineering benchmark — the model is given a GitHub issue and the repo, and must produce a patch that passes hidden test cases. The Verified subset filters to 500 hand-verified issues. The default benchmark for coding agents.