AI eval dataset builder
Editorial picks for "llm eval datasets".
32 tools
Braintrust
FeaturedBraintrust is an AI observability platform for tracing production, running evals, and catching regressions before they reach users.
LangSmith
LangChain's eval + observability platform.
Weights & Biases
The ML experiment tracker, now with LLM eval features.
Arize AI
Arize AX is an AI engineering platform for improving AI agents and applications.
Giskard
Continuous AI red teaming platform that detects vulnerabilities and safeguards AI agents.
Great Expectations
Open source framework for testing, validating, and documenting data quality across modern data pipelines, workflows, and teams.
Humanloop
Prompt management + evals for collaborative AI teams.
LiveBench
Contamination-free LLM benchmark that refreshes its questions monthly to keep frontier models honest.
Athina AI
Collaborative LLM evaluation and observability platform for teams shipping AI features to production.
Berkeley Function-Calling Leaderboard
Open benchmark from UC Berkeley that ranks LLMs on real-world tool-use and function-calling accuracy.
OpenAI Evals
TruLens
Open source library for evaluating and tracing AI agents.
CAMEL-AI
Open-source Python framework for building multi-agent systems and synthetic data pipelines.
PromptHub
A community-driven platform for discovering, managing, versioning, testing, and deploying prompts.
Patronus
Automated LLM evaluation for hallucinations, safety, and quality.
MathEval
Inspect AI
Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute and Meridian Labs.
Kiln AI
Open-source workbench for building, evaluating, and fine-tuning AI agents across 190+ models.
LLMEval
Research initiative from Fudan NLP Lab building evaluation frameworks for LLMs across academic disciplines and medical AI.
Promptfoo
Open-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.
Weco AI
Weco AI optimizes code against measurable metrics such as speed, accuracy, and cost.
AlpacaEval
Automatic LLM evaluator and leaderboard that benchmarks instruction-following with length-controlled win rates.
Maxim AI
End-to-end evaluation, simulation, and observability platform for shipping production-grade AI agents.
SEAL Leaderboard
Private, expert-graded leaderboards from Scale AI that rank frontier LLMs on domains contaminated public benchmarks can no longer measure.
VisualWebArena
Open benchmark for evaluating multimodal web agents on realistic visual browsing tasks.
OlympicArena
Agenta
Open-source LLMOps platform for prompt engineering, evaluation, and observability in one workspace.
CompassRank
Public leaderboard from the OpenCompass project ranking open and closed LLMs across 100+ benchmarks.
InfiBench
MixEval
Parea AI
LLM evaluation, observability, and prompt management platform for teams shipping production AI apps.
ModelBias
100 AI models, 100 simple questions, asked 3 times each, with 30,000 answers to explore.