Skip to main content
📖 The AI Tool Bible

Best AI tools for evals datasets

48 tools in the Evaluation category, filtered to evals datasets.

All Evaluation →
BR

Braintrust

Featured
Evaluation · Platform (any LLM)
8.9

Braintrust is an AI observability platform for tracing production, running evals, and catching regressions before they reach users.

Freemium· Starter plan is $0 / month; Pro plan is $249 / monthtracing productionrunning evals
LA

LangSmith

Evaluation · Platform (any LLM)
8.7

LangChain's eval + observability platform.

Freemium· Developer: $0 · Plus: $39 · Enterprise: Custom pricingLLM tracingevals
WB

Weights & Biases

Evaluation · Platform (any LLM)
8.4

The ML experiment tracker, now with LLM eval features.

Freemium· Free: $0/mo · Pro: Starts at $60/month · Enterprise: Custom plans · Personal: $0/mo · Advanced Enterprise: Custom planML experimentsLLM eval
AA

Arize AI

Evaluation · Multi-model
8.2

Arize AX is an AI engineering platform for improving AI agents and applications.

Freemium· Free tier · from $50/moChatbotsRAG systems
GI

Giskard

Evaluation · Multi-model
8.2

Continuous AI red teaming platform that detects vulnerabilities and safeguards AI agents.

FreemiumPrompt injection testingContinuous AI agent testing
GE

Great Expectations

Evaluation
8.2

Open source framework for testing, validating, and documenting data quality across modern data pipelines, workflows, and teams.

Freemiumdata quality testing and validationnotifying teams of critical failures
HU

Humanloop

Evaluation · Platform (any LLM)
8.2

Prompt management + evals for collaborative AI teams.

Paid· From $200/mo teamprompt managementteam collab
LI

LiveBench

Evaluation · Multi-model
8.2

Contamination-free LLM benchmark that refreshes its questions monthly to keep frontier models honest.

Free· Free and open source; self-hosted evaluation runnerllm-benchmarkingmodel-selection
AA

Athina AI

Evaluation · Multi-model
8.1

Collaborative LLM evaluation and observability platform for teams shipping AI features to production.

Freemium· Starter free (10k logs/mo); Pro & Enterprise customllm-evaluationprompt-management
BF

Berkeley Function-Calling Leaderboard

Evaluation · Multi-model
8.1

Open benchmark from UC Berkeley that ranks LLMs on real-world tool-use and function-calling accuracy.

Free· Free and open source; you pay only for inference when reproducing runs.function-calling evaltool-use benchmarking
HO

HoneyHive

Evaluation · Multi-model
8.1

OpenTelemetry-native observability and evaluation platform for LLM agents in production.

Freemium· Free tier available; paid/enterprise tiers via salesagent-observabilityllm-evaluation
ML

MLflow

Evaluation · Multi-model
8.1

Open-source platform for tracking, evaluating, and deploying ML models and LLM applications.

Free· Free and open source (Apache 2.0); managed offering via Databricksllm-evaluationexperiment-tracking
OE

OpenAI Evals

Evaluation · OpenAI GPT models (extensible)
8.1

FreeEvaluating LLMs and LLM systemsTesting different dimensions of OpenAI models
TR

TruLens

Evaluation · Multi-model (LLM-as-judge)
8.1

Open source library for evaluating and tracing AI agents.

FreeAI agent evaluation and tracingAgent tool selection and plan adherence
WB

W&B Weave

Evaluation · Multi-model
8.1

Production observability, tracing, and evaluation for LLM and agent systems from the Weights & Biases stack.

Freemium· Free tier available; paid and enterprise plans via W&Bllm-tracingagent-observability
PR

PromptHub

Writing · Multi-model (OpenAI, Anthropic, Google, Meta, Mistral, Bedrock, Azure)
8.0

A community-driven platform for discovering, managing, versioning, testing, and deploying prompts.

Freemium· Free plan; Pro plan is $12/month; Team plan is $20/user/monthPrompt management and versioningPrompt evaluation
LS

LLM Stats

Evaluation · Multi-model
7.9

Free
PA

Patronus

Evaluation · Platform (any LLM)
7.8

Automated LLM evaluation for hallucinations, safety, and quality.

Paid· Individual: Free · Base: $25 · Enterprise: Contact us for Pricinghallucination detectionsafety
LA

Langfuse

Evaluation · Model-agnostic
7.3

Open-source AI engineering platform for building, monitoring, and improving LLM applications.

Freemium· Hobby plan is Free; Core plan is $29 per month; Pro plan is $199 per month; Enterprise plan is $2499 per monthAI agent tracing and evaluationPrompt management
MA

MathEval

Evaluation · GPT-4 grader / DeepSeek-LLM-7B verifier
7.3

FreeEvaluating mathematical capabilities of large language modelsHorizontal comparison of mathematical capabilities between large models
OP

Opik

Evaluation · Multi-model
7.3

End-to-end AI observability platform for testing, shipping, and improving agentic systems.

Freemium· Free tier · from $19/moAI agent tracing and observabilityLLM evaluation
IA

Inspect AI

Evaluation · Multi-model
7.2

Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute and Meridian Labs.

Free· Free and open source (MIT-style license); you pay only for underlying model API usage.Coding and reasoning evaluationAgent evaluation
KA

Kiln AI

Evaluation · Multi-model
7.2

Open-source workbench for building, evaluating, and fine-tuning AI agents across 190+ models.

Freemium· Free Individual tier; Team (request access); Enterprise (custom)llm-evaluationfine-tuning
LL

LLMEval

Evaluation · Multi-model
7.2

Research initiative from Fudan NLP Lab building evaluation frameworks for LLMs across academic disciplines and medical AI.

FreeLLM evaluationEvaluation methodology research
PR

Promptfoo

Evaluation · Multi-model
7.2

Open-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.

Freemium· Free tierllm-evalsred-teaming
WA

Weco AI

Evaluation · Multi-model (LLM + AIDE tree search)
7.2

Weco AI optimizes code against measurable metrics such as speed, accuracy, and cost.

Freemiumfraud-detectionprompt-engineering
AL

AlpacaEval

Evaluation · GPT-4 Preview (Nov 2024) as annotator
7.1

Automatic LLM evaluator and leaderboard that benchmarks instruction-following with length-controlled win rates.

Free· Free and open-source; pay only for the underlying OpenAI annotator API callsllm-benchmarkinginstruction-following eval
MA

Maxim AI

Evaluation · Multi-model
7.1

End-to-end evaluation, simulation, and observability platform for shipping production-grade AI agents.

Freemium· Developer: Free · Professional: $29 /seat /month · Business: $49 /seat /month · Enterprise: Customagent-evaluationllm-observability
PF

Prompt Foundry

Evaluation · OpenAI + Anthropic (multi-model)
7.1

Prompt management and side-by-side LLM evaluation for OpenAI and Anthropic models.

Freemium· Free tier (10 prompts, 500 evals/mo); Pro $15/user/mo; Enterprise customprompt-managementmodel-comparison
RF

Respan (formerly Keywords AI)

Evaluation · Multi-model (500+ via gateway)
7.1

The AI router with built-in observability and automated evals.

Freemium· Free plan at $0; Team plan at $199 per month; Enterprise plan with custom pricingModel routingAgent tracing and monitoring
SL

SEAL Leaderboard

Evaluation · Multi-model (GPT, Claude, Gemini, Llama, etc.)
7.1

Private, expert-graded leaderboards from Scale AI that rank frontier LLMs on domains contaminated public benchmarks can no longer measure.

Free· Free to view; paid custom evals via Scale enterprise salesmodel-selectionbenchmark-tracking
VI

VisualWebArena

Evaluation · Model-agnostic (GPT-4V, Gemini, Claude, open VLMs)
7.1

Open benchmark for evaluating multimodal web agents on realistic visual browsing tasks.

Free· Free and open source (MIT-style research release)multimodal-agent-evalweb-browsing-benchmark
LL

llmfit

Evaluation · Multi-model
7.0

A terminal tool that right-sizes LLM models to your system's RAM, CPU, and GPU.

FreeDetermining which open-source LLMs the hardware can runEstimating SSD capacity for keeping runnable models
OL

OlympicArena

Evaluation · GPT-4o, Claude-3.5-Sonnet, Doubao-Pro-32k, DeepSeek-Coder-V2, Qwen2-72B-Instruct
7.0

Free· Free, open-source research benchmarkprocess-level evaluationleakage detection
PH

Phoenix

Evaluation · Multi-model
7.0

Open-source AI observability and evaluation platform for tracing and improving AI agents.

Freemium· Free tier · from $50/moTracing and improving AI agentsBuilding evaluations to score outputs
AG

Agenta

Evaluation · Multi-model
6.9

Open-source LLMOps platform for prompt engineering, evaluation, and observability in one workspace.

Freemium· from $29/moprompt-engineeringllm-evaluation
CO

CompassRank

Evaluation · Multi-model
6.9

Public leaderboard from the OpenCompass project ranking open and closed LLMs across 100+ benchmarks.

Free· Free leaderboard; OpenCompass toolkit is Apache 2.0 open sourcellm-benchmarkingmodel-selection
IN

InfiBench

Evaluation
6.9

Free· Free and open source (CC BY-SA 4.0)Evaluating code LLMs on real-world questionsSystematic evaluation of code LLMs
MI

MixEval

Evaluation · GPT-3.5-Turbo-0125, GPT-4o-2024-05-13, Claude 3.5 Sonnet, MixEval, MixEval-Hard
6.9

FreeModel rankingEvaluating various model sizes
AA

Arena AI

Evaluation · Multi-model
6.8

Head-to-head LLM battle arena with a public leaderboard for ranking AI models.

Free· Free to use; no public paid tier listedllm-benchmarkingmodel-comparison
AA

Artificial Analysis

Evaluation · Multi-model
6.8

Independent benchmarking platform comparing AI models and inference providers across intelligence, speed, and cost.

Freemium· from $417/momodel-benchmarkingprovider-comparison
PA

Parea AI

Evaluation · Multi-model
6.8

LLM evaluation, observability, and prompt management platform for teams shipping production AI apps.

Freemium· Free (2 seats, 3k logs/mo); Team $150/mo; Enterprise customllm-evaluationprompt-management
AW

AI World Bakeoff

Evaluation · Ten frontier coding models (specific list not published on landing page; includes at least one Claude Opus generation referenced as 'Opus 5')

Ten AI coding models build three identical explorable 3D worlds in a single autonomous shot.

Free
LM

Lagotto Meter

Evaluation

Measure the gap between what your site claims and what an AI agent actually finds

Free· Free (currently in public beta)AI-visibility audit of a landing pageFact-vs-claim gap analysis for marketing copy
LA

LangWatch

Evaluation · Model-agnostic; supports OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Vertex AI, and any OpenTelemetry-instrumented LLM

Agent observability and evaluation platform for AI platform teams.

Freemium· Developer: €0 free forever; Growth: €29 / core-seat / monthRed teaming and adversarial simulationPrompt version control
LG

LLM GPU Checker (KO)

Evaluation · Catalog covers open models on Hugging Face (Llama, Qwen, Mistral, Gemma, etc.)

Match LLMs to GPUs and plan multi-model AI stacks by VRAM, bandwidth and precision.

Free· Free (open web tool hosted on GitHub Pages).GPU sizing for self-hosted LLMsMulti-GPU RAG stack planning
MO

ModelBias

Evaluation · 100 models across Anthropic, OpenAI, Google, DeepSeek, Meta, xAI, Mistral, Qwen and others (via OpenRouter)

100 AI models, 100 simple questions, asked 3 times each, with 30,000 answers to explore.

FreeComparing hidden defaults, associations, and biases of AI modelsUsing the dataset in one's own project or research
MO

ModelFuzz

Evaluation · Model-agnostic; works with any OpenAI-compatible endpoint (Qwen 2.5 used in official examples).

ModelFuzz secures LLM agents against prompt injection by scanning for vulnerabilities and blocking unsafe tool calls at the execution layer.

Freemium· Free / open-source (MIT) via pip. Hosted dashboard with centralized policies, audit logs and continuous scanning coming soon via waitlist (pricing not yet public).Red-teaming OpenAI-compatible endpointsBlocking unsafe tool calls