Skip to main content
📖 The AI Tool Bible

LLM monitoring

Editorial picks for "best llm monitoring".

39 tools

All Evaluation
ScrapeGraphAI preview image
ScrapeGraphAI logo

ScrapeGraphAI

Agents · Multi-model (LLM, unspecified)
8.4

LLM-driven web scraping API that turns natural-language prompts into structured JSON.

Freemium· Free 500 credits; Starter $20/mo, Growth $100/mo, Pro $500/mo, Enterprise customweb-scrapingdata-extraction
Dataiku preview image
Dataiku logo

Dataiku

Agents · Multi-model (LLM Mesh: OpenAI, Anthropic, Bedrock, Vertex, OSS)
8.3

Enterprise AI platform unifying data, ML, LLMs, and agents under one governed workflow.

Enterprise· Basic: $10 · Pro: $20 · Enterprise: Contact salesenterprise-aiagent-orchestration
AgentOps preview image
AgentOps logo

AgentOps

Agents · Multi-model
8.2

Observability and debugging platform purpose-built for AI agents, with time-travel replay and cost tracking across 400+ LLMs.

Freemium· Free up to 5,000 events; Pro from $40/mo; Enterprise customagent-observabilityllm-tracing
Arize AI preview image
Arize AI logo

Arize AI

Evaluation · Multi-model
8.2

Enterprise observability and evaluation platform for LLM agents and generative AI applications.

Freemium· AX Free: Free · AX Pro: $50 · AX Enterprise: Customllm-observabilityagent-evaluation
Browser Use preview image
Browser Use logo

Browser Use

Agents · Multi-model (BYO LLM; Claude in hosted Box)
8.2

Open-source browser automation harness and cloud platform for LLM agents that drive real websites.

Freemium· PAYG: $0 · Dev: $29 · Business: $299 · Scaleup: $999web-automationscraping
Feast preview image
Feast logo

Feast

RAG
8.2

Open-source feature store that serves consistent features to ML training and online inference, with RAG vector search built in.

Free· Free, open source (Apache 2.0); self-hostedfeature-storerag-retrieval
Great Expectations preview image
Great Expectations logo

Great Expectations

Evaluation
8.2

Open-source data quality framework for validating the datasets that feed your ML and analytics pipelines.

Freemium· Developer: Free · Team: Custom · Enterprise: Contact Salesdata-validationpipeline-testing
Sim preview image
Sim logo

Sim

Agents · Multi-model (Claude, OpenAI, Google)
8.2

Open-source visual workspace for building, deploying, and monitoring AI agents.

Freemium· Free: $0 · Pro: $25 · Max: $100 · Enterprise: Customagent-workflowsinternal-automation
Athina AI preview image
Athina AI logo

Athina AI

Evaluation · Multi-model
8.1

Collaborative LLM evaluation and observability platform for teams shipping AI features to production.

Freemium· Starter free (10k logs/mo); Pro & Enterprise customllm-evaluationprompt-management
OpenMetadata preview image
OpenMetadata logo

OpenMetadata

Agents
7.4

Open-source metadata platform that gives AI agents a semantic context graph over your data stack.

Freemium· Free self-hosted OSS; Collate managed cloud has free tier + paid plansdata-catalogagent-context
OpenBB preview image
OpenBB logo

OpenBB

Agents · Multi-model (Agent Rita is model-agnostic)
7.3

Open-source financial workspace where analysts and AI agents share the same governed data.

Freemium· Community: Free · Lite: $100 · Pro: Custom pricing · Snowflake: $41.67financial-researchinvestment-workflows
Opik preview image
Opik logo

Opik

Evaluation · Multi-model
7.3

Open-source LLM observability and evaluation platform for debugging and monitoring AI agents in production.

Freemium· Free open-source self-host; free Cloud tier (no card); Enterprise contact salesllm-tracingagent-evaluation
Seldon preview image
Seldon logo

Seldon

Agents · Multi-model (bring your own)
7.3

Kubernetes-native MLOps platform for deploying and orchestrating ML and generative AI models in production.

Freemium· Basic: $10 · Pro: $20 · Enterprise: Contact salesmodel-servinginference-pipelines
Wallaroo.AI preview image
Wallaroo.AI logo

Wallaroo.AI

Agents · Multi-model
7.2

Production AI inference platform for deploying and monitoring models across cloud, on-prem, and edge.

Enterprise· Starter: $500 · Wallaroo Community Edition: Free · Ampere Community Edition: Freemodel-deploymentmlops
Arthur preview image
Arthur logo

Arthur

Evaluation · Multi-model
7.1

Open-source toolkit for testing, tracing, and monitoring production AI agents.

Freemium· Free: $0/mo · Premium: $60/mo · Enterprise: Customagent-evaluationprompt-management
Fiddler AI preview image
Fiddler AI logo

Fiddler AI

Evaluation · Fiddler Centor (proprietary evaluators)
7.1

Enterprise AI observability and guardrails platform for monitoring agents, LLMs, and ML models in production.

Enterprise· Free: Free · Developer: $0.002 per trace · Enterprise: Contact salesllm-observabilityagent-monitoring
Iguazio preview image
Iguazio logo

Iguazio

Agents · Multi-model
7.1

Enterprise MLOps and GenAI platform for taking models from notebook to production at scale.

Enterprise· Contact sales; free trial availablemlopsllm-fine-tuning
KNIME preview image
KNIME logo

KNIME

Agents · Multi-model (OpenAI, Anthropic, Gemini, Ollama)
7.1

Visual node-based data science platform with built-in connectors for OpenAI, Anthropic, Gemini, and local LLMs.

Freemium· Free: Free · Pro: $19 / month · Team: $99 / month · Business Hub: Contact salesdata-pipelinesllm-workflows
Maxim AI preview image
Maxim AI logo

Maxim AI

Evaluation · Multi-model
7.1

End-to-end evaluation, simulation, and observability platform for shipping production-grade AI agents.

Freemium· Developer: Free · Professional: $29 /seat /month · Business: $49 /seat /month · Enterprise: Customagent-evaluationllm-observability
Respan (formerly Keywords AI) preview image
Respan (formerly Keywords AI) logo

Respan (formerly Keywords AI)

Evaluation · Multi-model (500+ via gateway)
7.1

LLM engineering platform combining a multi-model gateway with tracing, evals, and prompt management.

Freemium· Free tier; paid plans (pricing not public); enterprise on requestllm-observabilityprompt-management
Tableau preview image
Tableau logo

Tableau

Agents · Salesforce Einstein (multi-model, incl. OpenAI via Einstein Trust Layer)
7.1

Salesforce-owned BI platform that bolted generative AI onto enterprise dashboards via Tableau Pulse and Tableau Agent.

Paid· Tableau Standard: $ 15 USD/User/Month (Billed annually) · Tableau Enterprise: $ 35 USD/User/Month (Billed annually) · Tableau Next: $ 40 USD/User/Month (Billed annually)business-intelligencenatural-language-queries
Bisheng preview image
Bisheng logo

Bisheng

Agents · Multi-model
7.0

Open-source enterprise AgentOps platform for building, orchestrating, and governing LLM agents at scale.

Freemium· Open-source community edition free; commercial enterprise edition via DataElement salesagent-orchestrationrag
Pagerly preview image
Pagerly logo

Pagerly

Agents
6.9

Slack-native on-call scheduling and incident response with AI-assisted rotations.

Freemium· Basic: $19 · Starter: $39 · Enterprise: Customon-call schedulingincident response
ReBillion.ai preview image
ReBillion.ai logo

ReBillion.ai

Agents
6.9

AI transaction coordinator that extracts dates, deadlines, and compliance rules from real estate contracts.

Paid· AI Toolkit + Customization: $199 · Human Assistant + AI + Customization: $499transaction-coordinationcontract-analysis
AI Meter preview image
AI Meter logo

AI Meter

Evaluation

Local usage meter that turns AI coding-agent tokens into estimated electricity and water consumption.

Free· Free for individuals and companies; open source under a public GitHub repo.Tracking daily token usage across Claude Code and CursorEstimating electricity draw of an AI-assisted coding session
AutotuneLLM preview image
AutotuneLLM logo

AutotuneLLM

Fine-tuning

An open-source optimization layer that sits between your app and Ollama to squeeze more performance out of local LLMs.

Free· Free and open source (MIT licensed).Local LLM inference on Apple SiliconReducing KV cache RAM for Ollama models
Browserbase preview image
Browserbase logo

Browserbase

Agents · Model-agnostic via Model Gateway (OpenAI, Anthropic, Google, others)

Headless browser infrastructure for AI agents

Freemium· Start for Free: Free · Get Developer: $20 · Get Startup: $99 · Contact Sales: CustomAI web-browsing agentsLLM-driven end-to-end testing
Caveman preview image
Caveman logo

Caveman

Agents · Model-agnostic proxy (Claude, GPT, Gemini, 30+ others); ships CaveGemma, a fine-tuned Gemma-4 released under MIT

The token-efficient stack for agent-native development

Freemium· Free: $0 · Indie: $29/mo · Team: $349/mo · Enterprise: CustomLLM cost optimizationMulti-provider AI gateway
Competiflow preview image
Competiflow logo

Competiflow

Agents

Structured competitor monitoring with AI severity scoring, delivered to your dashboard, Slack or agents via MCP.

Paid· Free trial (100 checks, no card) / Starter $19/mo (2,000 checks, 20 pages, daily) / Growth $49/mo (5,000 checks, 50 pages, every 6h + SEO analysis) / Pro $99/mo (10,000 checks, 100 pages, hourly + residential proxies)Competitor pricing-page monitoringBattlecard freshness automation
Decagon preview image
Decagon logo

Decagon

Agents

Enterprise AI concierge platform for voice, chat, and email support agents.

Enterprise· Custom enterprise pricing; not disclosed publicly. Sales-led with pilot engagements typical for CX/support agent platforms.AI customer support agentVoice support automation
Firecrawl MCP Server logo

Firecrawl MCP Server

MCP Servers

MCP server that gives Claude, Cursor, and other AI agents native web scrape, crawl, search, and extract tools via Firecrawl.

Freemium· Free: $0 USD · Team: $4 USD per user/month · Enterprise: $21 USD per user/monthLive documentation lookup inside Claude DesktopCompetitor pricing extraction during a Cursor coding session
Home Assistant MCP logo

Home Assistant MCP

MCP Servers

Open-source MCP server that lets any LLM control and observe a Home Assistant smart home in real time.

Free· Free and open source (Apache 2.0). Self-hosted; only cost is your own Home Assistant instance and the LLM client you connect to it.Conversational smart-home control from Claude Desktop or CursorLLM-driven Home Assistant automations
Lakera preview image
Lakera logo

Lakera

Evaluation · Proprietary in-house classifiers; model-agnostic (works in front of GPT-4o, Claude, Gemini, Llama, and custom LLMs)

Runtime security and guardrails for GenAI apps, agents, and RAG systems.

Freemium· Free community/developer tier at platform.lakera.ai; paid Enterprise plans (custom pricing, contact sales). No public price list.Prompt injection defense for chatbotsRAG guardrails against indirect injection
Neverbell preview image
Neverbell logo

Neverbell

Agents · Agent-framework agnostic (Claude Code, OpenClaw, Hermes and other compatible frameworks)

Market access skill that lets AI agents monitor and trade real markets

Freemium· Early access with $1,000 in test funds; production pricing tiers not yet disclosedAgent-executed equity and ETF tradesAutonomous crypto trading
Postgres MCP Pro logo

Postgres MCP Pro

MCP Servers · Model-agnostic (works with any MCP-capable LLM); optional OpenAI models for experimental LLM-based index tuning

Open-source Postgres MCP server with deterministic health checks, index tuning, and safe SQL execution.

Free· Free and open source (MIT license). No paid tiers.AI-assisted query optimization in Cursor or Claude DesktopAutomated index recommendations for slow workloads
Reddit MCP (adhikasp/mcp-reddit) logo

Reddit MCP (adhikasp/mcp-reddit)

MCP Servers

MCP server that lets Claude and other LLM clients fetch and analyse Reddit threads

Free· Free and open source (MIT). No paid tier. You run it locally; Reddit's public JSON endpoints are hit unauthenticated, subject to Reddit rate limits.Subreddit hot-thread summarisationCommunity sentiment monitoring
Stagehand preview image
Stagehand logo

Stagehand

Agents · Model-agnostic (OpenAI GPT-4o, Anthropic Claude, Google Gemini via Vercel AI SDK)

The SDK for browser agents

Freemium· SDK is free/open-source (MIT). Runs locally against any LLM API key. Optional Browserbase cloud runtime: Free $0/mo, Developer $20/mo, Startup $99/mo, Scale custom.AI web scrapingAutonomous browser agents
Tavily preview image
Tavily logo

Tavily

RAG · GPT-4

One secure API for real-time web access for AI agents

Freemium· Researcher: Free · Pay As You Go: $0.008 · Project: $01234567890123456789 · Enterprise: CustomRAG chatbot groundingAutonomous research agents
Weather MCP Server preview image
Weather MCP Server logo

Weather MCP Server

Agents

MCP server that gives Claude live weather, climate, terrain, and hazard data for any location on Earth.

Freemium· Free tier: 50 daily calls, standard polygon size. Paid plans: higher quotas and larger polygon analysis areas (contact / registration required for higher tiers).Site-specific fire risk assessmentFlood susceptibility screening