Skip to main content
📖 The AI Tool Bible

LLM observability

Editorial picks for "best llm observability".

48 tools

All Evaluation →
BR

Braintrust

Featured
Evaluation · Platform (any LLM)
8.9

Braintrust is an AI observability platform for tracing production, running evals, and catching regressions before they reach users.

Freemium· Starter plan is $0 / month; Pro plan is $249 / monthtracing productionrunning evals
LA

LangSmith

Evaluation · Platform (any LLM)
8.7

LangChain's eval + observability platform.

Freemium· Developer: $0 · Plus: $39 · Enterprise: Custom pricingLLM tracingevals
WB

Weights & Biases

Evaluation · Platform (any LLM)
8.4

The ML experiment tracker, now with LLM eval features.

Freemium· Free: $0/mo · Pro: Starts at $60/month · Enterprise: Custom plans · Personal: $0/mo · Advanced Enterprise: Custom planML experimentsLLM eval
HE

Helicone

Evaluation · Platform (any LLM)
8.3

AI Gateway and LLM Observability platform

Freemium· Hobby plan is Free; Pro plan is $79 per month; Team plan is $799 per monthCost tracking for LLM modelsMonitoring local and cloud LLM interactions
AG

AgentOps

Agents · Multi-model
8.2

Trace, debug, and deploy reliable AI agents.

Freemium· Basic plan: $0 per month; Pro plan: starts at $40 per monthAgent observabilityDebugging
AA

Arize AI

Evaluation · Multi-model
8.2

Arize AX is an AI engineering platform for improving AI agents and applications.

Freemium· Free tier · from $50/moChatbotsRAG systems
GI

Giskard

Evaluation · Multi-model
8.2

Continuous AI red teaming platform that detects vulnerabilities and safeguards AI agents.

FreemiumPrompt injection testingContinuous AI agent testing
HU

Humanloop

Evaluation · Platform (any LLM)
8.2

Prompt management + evals for collaborative AI teams.

Paid· From $200/mo teamprompt managementteam collab
AA

Athina AI

Evaluation · Multi-model
8.1

Collaborative LLM evaluation and observability platform for teams shipping AI features to production.

Freemium· Starter free (10k logs/mo); Pro & Enterprise customllm-evaluationprompt-management
HO

HoneyHive

Evaluation · Multi-model
8.1

OpenTelemetry-native observability and evaluation platform for LLM agents in production.

Freemium· Free tier available; paid/enterprise tiers via salesagent-observabilityllm-evaluation
ML

MLflow

Evaluation · Multi-model
8.1

Open-source platform for tracking, evaluating, and deploying ML models and LLM applications.

Free· Free and open source (Apache 2.0); managed offering via Databricksllm-evaluationexperiment-tracking
OE

OpenAI Evals

Evaluation · OpenAI GPT models (extensible)
8.1

FreeEvaluating LLMs and LLM systemsTesting different dimensions of OpenAI models
TR

TruLens

Evaluation · Multi-model (LLM-as-judge)
8.1

Open source library for evaluating and tracing AI agents.

FreeAI agent evaluation and tracingAgent tool selection and plan adherence
WB

W&B Weave

Evaluation · Multi-model
8.1

Production observability, tracing, and evaluation for LLM and agent systems from the Weights & Biases stack.

Freemium· Free tier available; paid and enterprise plans via W&Bllm-tracingagent-observability
PR

PromptHub

Writing · Multi-model (OpenAI, Anthropic, Google, Meta, Mistral, Bedrock, Azure)
8.0

A community-driven platform for discovering, managing, versioning, testing, and deploying prompts.

Freemium· Free plan; Pro plan is $12/month; Team plan is $20/user/monthPrompt management and versioningPrompt evaluation
PR

PromptLayer

Evaluation · Platform (any LLM)
7.9

Prompt management platform for AI teams to version prompts, run LLM evals, and monitor agents in production.

Freemium· Free: $0 per month; Pro: $49 per month; Team: $500 per monthPrompt managementAgent observability
PA

Patronus

Evaluation · Platform (any LLM)
7.8

Automated LLM evaluation for hallucinations, safety, and quality.

Paid· Individual: Free · Base: $25 · Enterprise: Contact us for Pricinghallucination detectionsafety
KA

Kong AI Gateway

Agents · Multi-model
7.3

Kong AI Gateway governs LLM, MCP, and agent-to-agent traffic with a single gateway.

Paid· Plus plan from $25/month plus usage, billed monthlyLLM, MCP, and A2A traffic governancePII sanitization and data leakage prevention
LA

Langfuse

Evaluation · Model-agnostic
7.3

Open-source AI engineering platform for building, monitoring, and improving LLM applications.

Freemium· Hobby plan is Free; Core plan is $29 per month; Pro plan is $199 per month; Enterprise plan is $2499 per monthAI agent tracing and evaluationPrompt management
OP

Opik

Evaluation · Multi-model
7.3

End-to-end AI observability platform for testing, shipping, and improving agentic systems.

Freemium· Free tier · from $19/moAI agent tracing and observabilityLLM evaluation
SE

Seldon

Agents · Multi-model (bring your own)
7.3

Kubernetes-native MLOps framework for real-time inference, observability, and compliance at scale.

Freemiumreal-time inferenceA/B testing and canary deployments
IA

Inspect AI

Evaluation · Multi-model
7.2

Inspect is a framework for frontier AI evaluations developed by the UK AI Security Institute and Meridian Labs.

Free· Free and open source (MIT-style license); you pay only for underlying model API usage.Coding and reasoning evaluationAgent evaluation
PL

Plano

Agents · Multi-model
7.2

AI-native proxy and data plane for agents handling routing, orchestration, traces, guardrails, and model routing.

Free· Open source; commercial pricing not disclosedAgent orchestrationLLM routing
PA

Portkey AI Gateway

Agents · Multi-model (1,600+ LLMs)
7.2

Freemium· Developer plan is Free Forever; Production plan is $49/monthModel routing and failoverSemantic and simple caching
PR

Promptfoo

Evaluation · Multi-model
7.2

Open-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.

Freemium· Free tierllm-evalsred-teaming
WA

Wallaroo.AI

Agents · Multi-model
7.2

Production AI inference platform for deploying and monitoring models across cloud, on-prem, and edge.

Enterprise· Starter: $500 · Wallaroo Community Edition: Free · Ampere Community Edition: Freemodel-deploymentmlops
AR

Arthur

Evaluation · Multi-model
7.1

Arthur unites security, finance, and the teams shipping AI on one control plane for AI you can trust.

Freemium· Free plan at $0 per month; Premium plan at $60 per monthEnforcing guardrails in LLM applicationsBuilding and governing agentic applications
FA

Fiddler AI

Evaluation · Fiddler Centor (proprietary evaluators)
7.1

The AI Control Plane for enterprises providing visibility, context, and control through evaluation, monitoring, enforcement, governance, and cost efficiency.

Enterprise· Free plan; Developer plan at $0.002 per traceMonitoring and evaluationsHallucination and toxicity protection
MA

Maxim AI

Evaluation · Multi-model
7.1

End-to-end evaluation, simulation, and observability platform for shipping production-grade AI agents.

Freemium· Developer: Free · Professional: $29 /seat /month · Business: $49 /seat /month · Enterprise: Customagent-evaluationllm-observability
PO

Portkey

Agents · Multi-model
7.1

Portkey is a production platform for AI teams offering an AI Gateway, Observability, Guardrails, Governance, and Prompt Management.

Freemium· Developer plan is Free Forever; Production plan is $49/monthAI GatewayObservability
PA

Puzzlet AI

Agents · Multi-model
7.1

Git-native prompt management and observability platform for teams shipping LLM applications.

Freemiumprompt-managementllm-observability
RF

Respan (formerly Keywords AI)

Evaluation · Multi-model (500+ via gateway)
7.1

The AI router with built-in observability and automated evals.

Freemium· Free plan at $0; Team plan at $199 per month; Enterprise plan with custom pricingModel routingAgent tracing and monitoring
PH

Phoenix

Evaluation · Multi-model
7.0

Open-source AI observability and evaluation platform for tracing and improving AI agents.

Freemium· Free tier · from $50/moTracing and improving AI agentsBuilding evaluations to score outputs
SU

Superwise

Evaluation · Multi-model
7.0

Agentic management platform for runtime guardrails, policy enforcement, and observability across LLM agents.

Freemium· Free tier · from $9/mollm-guardrailsai-governance
SY

SystemPrompt

Agents · Multi-model
7.0

SystemPrompt is a self-hosted control plane that sits between AI clients, model providers, and tools to manage access and audit.

Free· Evaluation and non-production use are freeModel access managementTool call governance
AG

Agenta

Evaluation · Multi-model
6.9

Open-source LLMOps platform for prompt engineering, evaluation, and observability in one workspace.

Freemium· from $29/moprompt-engineeringllm-evaluation
GA

Guild AI

Agents · Multi-model (bring your own)
6.9

Guild is a neutral control plane for AI agents designed to work across all models and vendors.

EnterpriseDeploying tracing and evaluationsScoping credentials and enforcing policy
TR

TrueFoundry

Agents · Multi-model
6.9

Enterprise control plane for deploying, governing, and scaling agentic AI on your own infrastructure.

Enterprise· Developer: $0 · Pro*: $499 · Pro Plus: $2999 · Enterprise: Customagent-deploymentllm-serving
CT

Cleanlab TLM

Evaluation · Multi-model (wraps any LLM)
6.8

Cleanlab TLM scores the trustworthiness of responses from any LLM in real-time.

FreemiumAgentsChatbots
PA

Parea AI

Evaluation · Multi-model
6.8

LLM evaluation, observability, and prompt management platform for teams shipping production AI apps.

Freemium· Free (2 seats, 3k logs/mo); Team $150/mo; Enterprise customllm-evaluationprompt-management
AM

AI Meter

Evaluation

Local usage meter that turns AI coding-agent tokens into estimated electricity and water consumption.

Free· Free for individuals and companies; open source under a public GitHub repo.Tracking daily token usage across Claude Code and CursorEstimating electricity draw of an AI-assisted coding session
CL

ClickHouse

RAG

ClickHouse is an open, composable platform for data-intensive workloads including real-time analytics, transactions, observability, security, and AI engineering.

FreemiumReal-time analyticsObservability for SREs and agents
HA

Habibi

Evaluation · GPT-4o, Perplexity Sonar, Gemini (bring-your-own API keys)

Self-hosted generative engine optimization for tracking brand mentions across ChatGPT, Perplexity, and Gemini

Freemium· Opsily Server: €40AI answer engine visibility trackingChatGPT brand mention monitoring
HY

Hydra

Agents · Multi-provider: routes across Claude, GPT, Gemini Flash, OpenRouter-hosted models, and local Qwen via Ollama / LM Studio

Free· Open Source plan is $0.00 with unlimited seats and requests, including every feature and commandCost-optimized task routingOn-device audit ledger for AI usage
LM

Lagotto Meter

Evaluation

Measure the gap between what your site claims and what an AI agent actually finds

Free· Free (currently in public beta)AI-visibility audit of a landing pageFact-vs-claim gap analysis for marketing copy
LA

Lakera

Evaluation · Proprietary in-house classifiers; model-agnostic (works in front of GPT-4o, Claude, Gemini, Llama, and custom LLMs)

An AI-native security platform designed to accelerate GenAI initiatives for enterprise teams.

FreemiumPreventing prompt injections, data leakage, and jailbreaksShadow AI discovery across apps and browsers
LA

LangWatch

Evaluation · Model-agnostic; supports OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Vertex AI, and any OpenTelemetry-instrumented LLM

Agent observability and evaluation platform for AI platform teams.

Freemium· Developer: €0 free forever; Growth: €29 / core-seat / monthRed teaming and adversarial simulationPrompt version control
TO

TokenPath

RAG · model-agnostic (works with any LLM output; uses in-house attribution model)

TokenPath is an API that grounds generated tokens in their source documents for output from any model.

Freemium· Small requests are free; Usage beyond the free tier costs $1 per million tokens, pay as you goChatting with pages, PDFs, or YouTube videosCross-language attribution for contracts