Promptfoo

Open-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.

Freemium· Open-source free; Enterprise SaaS contact salesEvaluationMulti-model

Best for

Pick Promptfoo if you ship LLM features to production and want versioned evals, regression tests, and automated red-teaming in CI.

Skip if

Skip it if you just need a chat playground or a no-code prompt comparison tool with zero setup.

Promptfoo is an open-source toolkit for testing, evaluating, and red-teaming LLM applications. It started as a CLI for running prompt and model comparisons against test cases in YAML, then grew into a broader AI security platform with automated vulnerability discovery, guardrails, and CI/CD integration. You write tests once, point them at any provider (OpenAI, Anthropic, local models, your own API), and get diffable scorecards covering correctness, regressions, prompt injections, jailbreaks, PII leakage, and policy violations.

It's aimed at engineering teams who treat LLM features the way they treat any other production code: unit-tested, regression-tracked, and security-reviewed. The OSS core is genuinely free and self-hostable, while a paid enterprise SaaS adds team dashboards, continuous red-teaming, and centralized policy management. The company claims adoption at 156 Fortune 500s and contributors from OpenAI, Google, and Microsoft, which lines up with how often you see it referenced in serious LLMOps stacks.

Integrations cover GitHub, GitLab, Jenkins, IDE plugins, and an MCP proxy for inspecting agent traffic. It's model-agnostic by design, so you can use it as a neutral harness across providers rather than being locked into any one vendor's eval framework.

Editor's take

Promptfoo is the closest thing the LLM world has to a default testing framework, and the fact that the OSS version is actually usable on its own is rare in this category. If you're past the demo stage and shipping to real users, the security and regression coverage alone justify the setup time.

— The AI Tool Bible editorial team