How does Inspect AI handle the execution of model-generated code?

It supports sandboxed tool execution via Docker, Kubernetes, or Modal. This allows users to safely run untrusted model-generated code during evaluations without risking the host system.

Is Inspect AI free to use, and what are the associated costs?

Inspect AI is free and open source under an MIT-style license. Users only pay for the underlying model API usage costs, as they must bring their own model API keys to run evaluations.

Which model providers are compatible with Inspect AI?

The framework supports more than 20 model providers, including OpenAI, Anthropic, Google, and HuggingFace. It also offers MCP tool support and external agent integration for building internal evals harnesses.

Inspect AI

Open-source LLM evaluation framework from the UK AI Security Institute with 200+ built-in benchmarks.

Free· Free and open source (MIT-style license); you pay only for underlying model API usage.EvaluationMulti-model7.2 / 10

Visit website →

In short

Inspect AI is an open-source Python framework for running rigorous, auditable evaluations of large language models. It is best suited for AI safety researchers and ML engineers who need a self-hosted, reproducible testing environment with 200+ built-in benchmarks.

Best for

Pick Inspect AI if you're an AI safety researcher or ML engineer who needs a rigorous, auditable, self-hosted framework for benchmarking and red-teaming LLMs.

Skip if

Skip it if you want a hosted, click-through eval dashboard or a lightweight prompt-testing tool for non-technical users.

Inspect AI is an open-source Python framework built by the UK AI Security Institute (AISI) and Meridian Labs for running rigorous evaluations of large language models. It ships with composable primitives — datasets, solvers, tools, scorers, and agents — plus a library of over 200 pre-built benchmark implementations covering coding, reasoning, multi-modal understanding, agentic tasks, and capture-the-flag security challenges.

What sets Inspect apart is its provenance and its focus on serious, reproducible model assessment rather than vibes-based leaderboards. It supports 20+ model providers (OpenAI, Anthropic, Google, HuggingFace, and more), a built-in ReAct agent with multi-agent primitives, and sandboxed tool execution via Docker, Kubernetes, or Modal for safely running untrusted model-generated code. It's aimed squarely at AI safety researchers, red teams, and model developers who need auditable eval pipelines.

The framework includes a web-based Inspect View for inspecting traces and a VS Code extension for authoring evals. Install with pip install inspect-ai; you bring your own model API keys and pay only the underlying provider costs. MCP tool support and external agent integration make it a credible backbone for building a full internal evals harness.

Editor's take

Inspect is quickly becoming the default serious-evals framework outside the big labs, and the AISI stewardship gives it credibility that most eval libraries lack. If you're building an internal model evaluation pipeline in 2026 and you're not paying for a hosted platform, this is the one to start with.

— The AI Tool Bible editorial team

Pros

✅ Backed by the UK AI Security Institute — serious pedigree for safety work
✅ 200+ pre-built evaluations ready to run out of the box
✅ Supports 20+ model providers plus sandboxed code execution
✅ Composable Python API with CLI, Inspect View UI, and VS Code extension
✅ Fully open source with no vendor lock-in

Cons

⚠️ Python-first — no low-code path for non-engineers
⚠️ Running large eval suites incurs real model API costs
⚠️ Steeper learning curve than hosted eval platforms

Use cases

llm-benchmarkingagent-evaluationsafety-testingcapture-the-flagcustom-evals

Frequently asked

Who developed Inspect AI and what is its primary purpose?: Inspect AI was built by the UK AI Security Institute (AISI) and Meridian Labs. It is designed for running rigorous evaluations of large language models, focusing on serious, reproducible model assessment rather than casual leaderboards.
What types of benchmarks and tasks does Inspect AI support?: The framework includes over 200 pre-built benchmark implementations covering coding, reasoning, multi-modal understanding, agentic tasks, and capture-the-flag security challenges. It also supports custom evaluations using composable primitives like datasets, solvers, and scorers.
How does Inspect AI handle the execution of model-generated code?: It supports sandboxed tool execution via Docker, Kubernetes, or Modal. This allows users to safely run untrusted model-generated code during evaluations without risking the host system.
Is Inspect AI free to use, and what are the associated costs?: Inspect AI is free and open source under an MIT-style license. Users only pay for the underlying model API usage costs, as they must bring their own model API keys to run evaluations.
Which model providers are compatible with Inspect AI?: The framework supports more than 20 model providers, including OpenAI, Anthropic, Google, and HuggingFace. It also offers MCP tool support and external agent integration for building internal evals harnesses.