LangWatch

Simulation-based testing, evaluation, and observability for LLM agents

Freemium· Developer: Free forever (50k events/mo, 14-day retention, 2 users) / Growth: EUR 29 per core-seat/mo (200k events, then EUR 5 per 100k) / Enterprise: custom (hybrid, self-hosted, on-prem, SSO/RBAC, SLAs)EvaluationModel-agnostic; supports OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Vertex AI, and any OpenTelemetry-instrumented LLM

Visit website →

Best for

Engineering teams building multi-step LLM agents in production who need tracing, scenario-based simulation, and CI-gated evals in one open-source-friendly platform they can self-host if compliance requires it.

Skip if

Solo builders shipping a single-prompt wrapper or teams that only need lightweight trace logging; the simulation and prompt-management surface will be overkill and the seat pricing hard to justify.

LangWatch is an open-source (Apache 2) observability and evaluation platform aimed at teams shipping production LLM agents. It combines three things that usually live in separate tools: end-to-end tracing of every agent step, token and cost via OpenTelemetry; a simulation harness that runs multi-turn text or voice scenarios against your agent so regressions are caught before deploy; and an LLM-as-a-judge evaluation layer that scores outputs against custom rubrics, safety checks, and factuality tests. The platform also ships a prompt registry with versioning and A/B testing, red-teaming for jailbreak and prompt-injection probes, and 'Langy', an AI-engineer assistant that turns plain-English requirements into tests and opens PRs against your repo. Instrumentation is framework-agnostic: Python, TypeScript and Go SDKs plus native hooks for LangGraph, CrewAI, Vercel AI SDK, LlamaIndex, DSPy, Claude Code, Codex, OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, and Vertex AI. Typical workflows are: instrument an agent, replay production traces as datasets, define scenarios with expected outcomes, run judges on every PR in CI, and watch cost, latency and quality drift on dashboards. The tool is used by Backbase, Deloitte, PagBank, and Visma for mission-critical agent rollouts, and offers cloud, hybrid, and self-hosted deployment for teams with data-residency requirements.

Editor's take

LangWatch is one of the few eval platforms that treats simulation as a first-class citizen instead of a bolt-on, and the OpenTelemetry-native tracing plus Apache 2 license make it a rare buy-vs-build tie-breaker. If you already run agents in production and are chasing regressions with print statements, this is the shortest path to a real evaluation loop.

— The AI Tool Bible editorial team

Pros

✅ Apache 2 open source with self-hosted and on-prem deployment options for regulated teams
✅ OpenTelemetry-native tracing works with virtually any framework or custom stack
✅ Combines observability, evaluation, simulation, and prompt management in one product instead of stitching four tools
✅ First-class multi-turn conversation simulation (text and voice), not just single-shot eval
✅ LLM-as-a-judge scoring can run on single outputs or entire conversations, including multimodal inputs
✅ SDKs in Python, TypeScript, and Go plus deep integrations with LangGraph, CrewAI, DSPy, Bedrock, Vertex, and Azure
✅ Generous free tier (50k events/mo) with no credit card, so teams can prove value before buying

Cons

⚠️ Feature surface is broad; smaller teams may find the UI heavier than a focused tracer like Langfuse or Phoenix
⚠️ LLM-judge evaluations add their own token cost that stacks on top of your agent's inference bill
⚠️ Growth plan bills per core-seat and per 100k events, which can escalate quickly on chatty production agents
⚠️ Self-hosting the full stack (Postgres, ClickHouse, workers) is non-trivial versus SaaS-only competitors
⚠️ Simulation quality depends heavily on how well you author scenarios; poorly written cases give false confidence

Use cases

LLM agent regression testing in CIRAG answer-quality evaluationVoice-agent conversation simulationPrompt versioning and A/B testingProduction trace observability and cost trackingRed-team and jailbreak probingMulti-turn chatbot evaluationGuardrail and safety scoringDataset creation from production tracesLLM-as-a-judge scorecards