MixEval

Dynamic LLM benchmark that mixes web queries with existing datasets to mirror Chatbot Arena rankings at a fraction of the cost.

Free· Free and open sourceEvaluation

Visit website →

Best for

Pick MixEval if you need a cheap, leaderboard-correlated offline benchmark to rank LLM checkpoints during pretraining or fine-tuning.

Skip if

Skip it if you want a managed eval-as-a-service product with dashboards, RBAC, or a hosted API rather than a research repo.

MixEval is an open-source evaluation benchmark for large language models that combines real-world user queries scraped from the web with curated existing benchmark datasets. The authors report a 0.96 model ranking correlation with Chatbot Arena while costing roughly 6% of what a full MMLU evaluation does, making it one of the more practical offline proxies for human-preference leaderboards. A harder variant, MixEval-Hard, sharpens the gap between frontier models, and the suite uses ground-truth grading rather than an LLM judge to avoid known judge biases.

It's built by researchers from the National University of Singapore, CMU, and the Allen Institute for AI, and was accepted at NeurIPS 2024. The benchmark is refreshed periodically (around 85% unique queries per version) to limit data contamination as models train on yesterday's eval sets. Everything - code, data, and the evaluation harness - lives on GitHub and Hugging Face under a permissive license, so teams can run it locally on their own checkpoints.

MixEval-X, released later, extends the methodology beyond text into multi-modal evaluation. There is no hosted product, no API, and no paid tier; this is a research artifact you self-host as part of a model-evaluation pipeline.

Editor's take

MixEval is the kind of benchmark you actually want to wire into CI: cheap to run, well-correlated with human preference, and refreshed often enough to stay honest. The lack of a hosted product is a feature for serious ML teams and a non-starter for anyone hoping to click a button. Pair it with your own judge-based evals for full coverage.

— The AI Tool Bible editorial team