How does OlympicArena evaluate model performance?

It uses fine-grained process-level evaluation to score the reasoning path, not just the final answer. This covers 8 logical-reasoning types and 5 visual-reasoning types, distinguishing it from generic multiple-choice suites.

Does OlympicArena detect data contamination?

Yes, it runs instance-level leakage detection to help users determine how much of a model's score is due to contamination rather than genuine capability.

Is OlympicArena a hosted service or API?

No, it is a free, open-source research benchmark without a hosted API or commercial tier. Users must bring their own inference stack to run evaluations, with the dataset on Hugging Face and code on GitHub.

OlympicArena

Olympiad-level multi-discipline benchmark for stress-testing reasoning in LLMs and multimodal models.

Free· Free, open-source research benchmarkEvaluationGPT-4o, Claude-3.5-Sonnet, Doubao-Pro-32k, DeepSeek-Coder-V2, Qwen2-72B-Instruct7.0 / 10

Visit website →

In short

OlympicArena evaluates frontier LLMs and multimodal models using 11,163 hard, multi-step problems from 62 academic competitions. It is best for researchers needing process-level reasoning scores and contamination detection across seven STEM disciplines.

Best for

Pick OlympicArena if you are benchmarking frontier LLMs or multimodal models on hard STEM reasoning and want process-level scoring with contamination checks.

Skip if

Skip it if you need a turnkey eval SaaS, a writing/coding-only benchmark, or a hosted API to run evaluations for you.

OlympicArena is an open evaluation benchmark built by GAIR (Shanghai Jiao Tong University, Shanghai AI Lab) that throws 11,163 problems drawn from 62 Olympic-tier academic competitions at language and multimodal models. The coverage spans mathematics, physics, chemistry, biology, geography, astronomy, and computer science, with problems available in both English and Chinese and with 13 distinct answer types to keep models from gaming a single output format.

What separates it from generic MMLU-style suites is the focus on genuinely hard, multi-step reasoning at Olympiad difficulty plus a fine-grained process-level evaluation: it scores not just the final answer but the reasoning path, covering 8 logical-reasoning types and 5 visual-reasoning types. It also runs instance-level leakage detection so you can see how much of a model's score is contamination rather than capability. It's aimed at researchers and serious model-builders comparing frontier LLMs and VLMs, not casual users.

The dataset lives on Hugging Face, code is on GitHub, and there's a public leaderboard you can submit to. Everything is free for research use; there's no hosted API or commercial tier, so you bring your own inference stack to run evaluations.

Editor's take

One of the more honest hard-reasoning benchmarks: Olympiad problems, bilingual coverage, and explicit leakage detection. The process-level evaluation is the real differentiator versus MMLU-style multiple choice. Use it alongside, not instead of, task-specific evals.

— The AI Tool Bible editorial team

Pros

✅ Olympiad-level difficulty pushes past saturated benchmarks like MMLU
✅ Covers seven STEM disciplines plus multimodal and bilingual EN/ZH problems
✅ Process-level scoring evaluates reasoning steps, not just final answers
✅ Built-in leakage detection helps separate capability from contamination
✅ Fully open: dataset on Hugging Face, code on GitHub, public leaderboard

Cons

⚠️ Research benchmark, not a hosted product or SaaS
⚠️ No managed API or runner; you supply the inference infrastructure
⚠️ Heavy STEM focus means limited signal for writing or creative tasks

Use cases

llm-evaluationmultimodal-evalreasoning-benchmarkleaderboard-submissioncontamination-detection

Frequently asked

What types of problems does OlympicArena include?: The benchmark contains 11,163 problems drawn from 62 Olympic-tier competitions spanning mathematics, physics, chemistry, biology, geography, astronomy, and computer science. Problems are available in English and Chinese with 13 distinct answer types.
How does OlympicArena evaluate model performance?: It uses fine-grained process-level evaluation to score the reasoning path, not just the final answer. This covers 8 logical-reasoning types and 5 visual-reasoning types, distinguishing it from generic multiple-choice suites.
Does OlympicArena detect data contamination?: Yes, it runs instance-level leakage detection to help users determine how much of a model's score is due to contamination rather than genuine capability.
Is OlympicArena a hosted service or API?: No, it is a free, open-source research benchmark without a hosted API or commercial tier. Users must bring their own inference stack to run evaluations, with the dataset on Hugging Face and code on GitHub.