MathEval

MathEval is a comprehensive benchmark that measures how well large language models actually do math, spanning 22+ datasets and roughly 30,000 problems from elementary arithmetic through college-entrance exams (Gaokao 2023/2024), competition math (AIME, AMC), MATH, GSM8K, MMLU math subsets, and advanced higher-mathematics topics in both English and Chinese. Submissions are run through a standardized pipeline that uses GPT-4 (or a distilled DeepSeek-LLM-7B verifier) for answer extraction and grading, so models are compared against a consistent rubric rather than brittle regex matchers.

It is aimed at LLM researchers, model builders, and infra teams who want a defensible third-party number on math reasoning instead of cherry-picking GSM8K. The pipeline supports HuggingFace checkpoints, API-based models, and custom open-source models, with both zero-shot and few-shot regimes. The project is open source (Python, math-eval org on GitHub) and the leaderboard at matheval.ai is free to read; running an evaluation is done by submitting through the site because end-to-end runs are compute-heavy.

Annually refreshed Gaokao problems are a deliberate hedge against benchmark contamination, which is one of the better reasons to use MathEval over a single static dataset. The trade-off is that there is no turnkey hosted API, evaluations are queued rather than instant, and the project leans academic in tone and cadence.