Skip to main content
📖 The AI Tool Bible

Best AI tools for llm observability

42 tools in the Evaluation category, filtered to llm observability.

All Evaluation →
BR

Braintrust

Featured
Evaluation · Platform (any LLM)
8.9

Braintrust is an AI observability platform for tracing production, running evals, and catching regressions before they reach users.

Freemium· Starter plan is $0 / month; Pro plan is $249 / monthtracing productionrunning evals
LA

LangSmith

Evaluation · Platform (any LLM)
8.7

LangChain's eval + observability platform.

Freemium· Developer: $0 · Plus: $39 · Enterprise: Custom pricingLLM tracingevals
WB

Weights & Biases

Evaluation · Platform (any LLM)
8.4

The ML experiment tracker, now with LLM eval features.

Freemium· Free: $0/mo · Pro: Starts at $60/month · Enterprise: Custom plans · Personal: $0/mo · Advanced Enterprise: Custom planML experimentsLLM eval
HE

Helicone

Evaluation · Platform (any LLM)
8.3

AI Gateway and LLM Observability platform

Freemium· Hobby plan is Free; Pro plan is $79 per month; Team plan is $799 per monthCost tracking for LLM modelsMonitoring local and cloud LLM interactions
AG

AgentOps

Agents · Multi-model
8.2

Trace, debug, and deploy reliable AI agents.

Freemium· Basic plan: $0 per month; Pro plan: starts at $40 per monthAgent observabilityDebugging
AA

Arize AI

Evaluation · Multi-model
8.2

Arize AX is an AI engineering platform for improving AI agents and applications.

Freemium· Free tier · from $50/moChatbotsRAG systems
GI

Giskard

Evaluation · Multi-model
8.2

Continuous AI red teaming platform that detects vulnerabilities and safeguards AI agents.

FreemiumPrompt injection testingContinuous AI agent testing
HU

Humanloop

Evaluation · Platform (any LLM)
8.2

Prompt management + evals for collaborative AI teams.

Paid· From $200/mo teamprompt managementteam collab
AA

Athina AI

Evaluation · Multi-model
8.1

Collaborative LLM evaluation and observability platform for teams shipping AI features to production.

Freemium· Starter free (10k logs/mo); Pro & Enterprise customllm-evaluationprompt-management
HO

HoneyHive

Evaluation · Multi-model
8.1

OpenTelemetry-native observability and evaluation platform for LLM agents in production.

Freemium· Free tier available; paid/enterprise tiers via salesagent-observabilityllm-evaluation
ML

MLflow

Evaluation · Multi-model
8.1

Open-source platform for tracking, evaluating, and deploying ML models and LLM applications.

Free· Free and open source (Apache 2.0); managed offering via Databricksllm-evaluationexperiment-tracking
OE

OpenAI Evals

Evaluation · OpenAI GPT models (extensible)
8.1

FreeEvaluating LLMs and LLM systemsTesting different dimensions of OpenAI models
TR

TruLens

Evaluation · Multi-model (LLM-as-judge)
8.1

Open source library for evaluating and tracing AI agents.

FreeAI agent evaluation and tracingAgent tool selection and plan adherence
WB

W&B Weave

Evaluation · Multi-model
8.1

Production observability, tracing, and evaluation for LLM and agent systems from the Weights & Biases stack.

Freemium· Free tier available; paid and enterprise plans via W&Bllm-tracingagent-observability
PR

PromptHub

Writing · Multi-model (OpenAI, Anthropic, Google, Meta, Mistral, Bedrock, Azure)
8.0

A community-driven platform for discovering, managing, versioning, testing, and deploying prompts.

Freemium· Free plan; Pro plan is $12/month; Team plan is $20/user/monthPrompt management and versioningPrompt evaluation
LS

LLM Stats

Evaluation · Multi-model
7.9

Free
PR

PromptLayer

Evaluation · Platform (any LLM)
7.9

Prompt management platform for AI teams to version prompts, run LLM evals, and monitor agents in production.

Freemium· Free: $0 per month; Pro: $49 per month; Team: $500 per monthPrompt managementAgent observability
PA

Patronus

Evaluation · Platform (any LLM)
7.8

Automated LLM evaluation for hallucinations, safety, and quality.

Paid· Individual: Free · Base: $25 · Enterprise: Contact us for Pricinghallucination detectionsafety
LA

Langfuse

Evaluation · Model-agnostic
7.3

Open-source AI engineering platform for building, monitoring, and improving LLM applications.

Freemium· Hobby plan is Free; Core plan is $29 per month; Pro plan is $199 per month; Enterprise plan is $2499 per monthAI agent tracing and evaluationPrompt management
OP

Opik

Evaluation · Multi-model
7.3

End-to-end AI observability platform for testing, shipping, and improving agentic systems.

Freemium· Free tier · from $19/moAI agent tracing and observabilityLLM evaluation
IA

Inspect AI

Evaluation · Multi-model
7.2

Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute and Meridian Labs.

Free· Free and open source (MIT-style license); you pay only for underlying model API usage.Coding and reasoning evaluationAgent evaluation
KA

Kiln AI

Evaluation · Multi-model
7.2

Open-source workbench for building, evaluating, and fine-tuning AI agents across 190+ models.

Freemium· Free Individual tier; Team (request access); Enterprise (custom)llm-evaluationfine-tuning
PR

Promptfoo

Evaluation · Multi-model
7.2

Open-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.

Freemium· Free tierllm-evalsred-teaming
WA

Weco AI

Evaluation · Multi-model (LLM + AIDE tree search)
7.2

Weco AI optimizes code against measurable metrics such as speed, accuracy, and cost.

Freemiumfraud-detectionprompt-engineering
AL

AlpacaEval

Evaluation · GPT-4 Preview (Nov 2024) as annotator
7.1

Automatic LLM evaluator and leaderboard that benchmarks instruction-following with length-controlled win rates.

Free· Free and open-source; pay only for the underlying OpenAI annotator API callsllm-benchmarkinginstruction-following eval
AR

Arthur

Evaluation · Multi-model
7.1

Arthur unites security, finance, and the teams shipping AI on one control plane for AI you can trust.

Freemium· Free plan at $0 per month; Premium plan at $60 per monthEnforcing guardrails in LLM applicationsBuilding and governing agentic applications
FA

Fiddler AI

Evaluation · Fiddler Centor (proprietary evaluators)
7.1

The AI Control Plane for enterprises providing visibility, context, and control through evaluation, monitoring, enforcement, governance, and cost efficiency.

Enterprise· Free plan; Developer plan at $0.002 per traceMonitoring and evaluationsHallucination and toxicity protection
MA

Maxim AI

Evaluation · Multi-model
7.1

End-to-end evaluation, simulation, and observability platform for shipping production-grade AI agents.

Freemium· Developer: Free · Professional: $29 /seat /month · Business: $49 /seat /month · Enterprise: Customagent-evaluationllm-observability
RF

Respan (formerly Keywords AI)

Evaluation · Multi-model (500+ via gateway)
7.1

The AI router with built-in observability and automated evals.

Freemium· Free plan at $0; Team plan at $199 per month; Enterprise plan with custom pricingModel routingAgent tracing and monitoring
PH

Phoenix

Evaluation · Multi-model
7.0

Open-source AI observability and evaluation platform for tracing and improving AI agents.

Freemium· Free tier · from $50/moTracing and improving AI agentsBuilding evaluations to score outputs
SU

Superwise

Evaluation · Multi-model
7.0

Agentic management platform for runtime guardrails, policy enforcement, and observability across LLM agents.

Freemium· Free tier · from $9/mollm-guardrailsai-governance
AG

Agenta

Evaluation · Multi-model
6.9

Open-source LLMOps platform for prompt engineering, evaluation, and observability in one workspace.

Freemium· from $29/moprompt-engineeringllm-evaluation
CO

CompassRank

Evaluation · Multi-model
6.9

Public leaderboard from the OpenCompass project ranking open and closed LLMs across 100+ benchmarks.

Free· Free leaderboard; OpenCompass toolkit is Apache 2.0 open sourcellm-benchmarkingmodel-selection
IZ

Izlo

Agents · Model-agnostic
6.9

Izlo provides prompt control for teams to unlock prompts trapped in code, documents, and spreadsheets.

Paid· Solo: $20 per month; Pro: $25 per user/month; Enterprise: $39 per user/monthprompt managementversion control
AA

Arena AI

Evaluation · Multi-model
6.8

Head-to-head LLM battle arena with a public leaderboard for ranking AI models.

Free· Free to use; no public paid tier listedllm-benchmarkingmodel-comparison
CT

Cleanlab TLM

Evaluation · Multi-model (wraps any LLM)
6.8

Cleanlab TLM scores the trustworthiness of responses from any LLM in real-time.

FreemiumAgentsChatbots
PA

Parea AI

Evaluation · Multi-model
6.8

LLM evaluation, observability, and prompt management platform for teams shipping production AI apps.

Freemium· Free (2 seats, 3k logs/mo); Team $150/mo; Enterprise customllm-evaluationprompt-management
AM

AI Meter

Evaluation

Local usage meter that turns AI coding-agent tokens into estimated electricity and water consumption.

Free· Free for individuals and companies; open source under a public GitHub repo.Tracking daily token usage across Claude Code and CursorEstimating electricity draw of an AI-assisted coding session
CL

ClickHouse

RAG

ClickHouse is an open, composable platform for data-intensive workloads including real-time analytics, transactions, observability, security, and AI engineering.

FreemiumReal-time analyticsObservability for SREs and agents
HA

Habibi

Evaluation · GPT-4o, Perplexity Sonar, Gemini (bring-your-own API keys)

Self-hosted generative engine optimization for tracking brand mentions across ChatGPT, Perplexity, and Gemini

Freemium· Opsily Server: €40AI answer engine visibility trackingChatGPT brand mention monitoring
LA

Lakera

Evaluation · Proprietary in-house classifiers; model-agnostic (works in front of GPT-4o, Claude, Gemini, Llama, and custom LLMs)

An AI-native security platform designed to accelerate GenAI initiatives for enterprise teams.

FreemiumPreventing prompt injections, data leakage, and jailbreaksShadow AI discovery across apps and browsers
LA

LangWatch

Evaluation · Model-agnostic; supports OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Vertex AI, and any OpenTelemetry-instrumented LLM

Agent observability and evaluation platform for AI platform teams.

Freemium· Developer: €0 free forever; Growth: €29 / core-seat / monthRed teaming and adversarial simulationPrompt version control