AI eval dataset builder

Eval, monitor, and improve AI products end-to-end.

LangChain's eval + observability platform.

The ML experiment tracker, now with LLM eval features.

Enterprise observability and evaluation platform for LLM agents and generative AI applications.

Continuous AI red teaming platform that stress-tests LLM agents for vulnerabilities before they hit production.

Open-source data quality framework for validating the datasets that feed your ML and analytics pipelines.

Prompt management + evals for collaborative AI teams.

Contamination-free LLM benchmark that refreshes its questions monthly to keep frontier models honest.

Collaborative LLM evaluation and observability platform for teams shipping AI features to production.

Open benchmark from UC Berkeley that ranks LLMs on real-world tool-use and function-calling accuracy.

OpenAI's open-source framework for benchmarking LLMs against a shared registry of evaluations.

Open-source evaluation and tracing framework for LLM apps and agents, built on OpenTelemetry.

Open-source Python framework for building multi-agent systems and synthetic data pipelines.

Git-style prompt management, testing, and deployment platform for teams running multiple LLMs in production.

Automated LLM evaluation for hallucinations, safety, and quality.

Holistic benchmark suite for evaluating mathematical reasoning in large language models.

Open-source LLM evaluation framework from the UK AI Security Institute with 200+ built-in benchmarks.

Open-source workbench for building, evaluating, and fine-tuning AI agents across 190+ models.

Open academic benchmark suite for stress-testing LLMs on contamination-resistant, domain-specific tasks.

Open-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.

Autoresearch engine that iteratively rewrites code to optimize against a numeric evaluation metric.

Automatic LLM evaluator and leaderboard that benchmarks instruction-following with length-controlled win rates.

End-to-end evaluation, simulation, and observability platform for shipping production-grade AI agents.

Private, expert-graded leaderboards from Scale AI that rank frontier LLMs on domains contaminated public benchmarks can no longer measure.

Open benchmark for evaluating multimodal web agents on realistic visual browsing tasks.

Olympiad-level multi-discipline benchmark for stress-testing reasoning in LLMs and multimodal models.

Open-source LLMOps platform for prompt engineering, evaluation, and observability in one workspace.

Public leaderboard from the OpenCompass project ranking open and closed LLMs across 100+ benchmarks.

Stack Overflow-derived benchmark for evaluating code LLMs on real-world programming questions.

Dynamic LLM benchmark that mixes web queries with existing datasets to mirror Chatbot Arena rankings at a fraction of the cost.

LLM evaluation, observability, and prompt management platform for teams shipping production AI apps.

100 models, 100 prompts, 30,000 answers — an interactive look at AI defaults