Skip to main content
📖 The AI Tool Bible
Promptfoo preview image
Promptfoo logo

Promptfoo

Not yet source-checked

Open-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.

Machine-drafted from the vendor's site · not yet source-checkedProfile updated Link checked Pricing captured How we rate

Freemium· Free tierEvaluationMulti-model7.2 / 10

In short

Promptfoo is an open-source toolkit for testing, evaluating, and red-teaming LLM applications. It is best for engineering teams shipping LLM features to production who need versioned evals, regression tracking, and automated security checks in their CI/CD workflows.

Best for

Pick Promptfoo if you ship LLM features to production and want versioned evals, regression tests, and automated red-teaming in CI.

Skip if

Skip it if you just need a chat playground or a no-code prompt comparison tool with zero setup.

Promptfoo is an open-source toolkit for testing, evaluating, and red-teaming LLM applications. It started as a CLI for running prompt and model comparisons against test cases in YAML, then grew into a broader AI security platform with automated vulnerability discovery, guardrails, and CI/CD integration. You write tests once, point them at any provider (OpenAI, Anthropic, local models, your own API), and get diffable scorecards covering correctness, regressions, prompt injections, jailbreaks, PII leakage, and policy violations.

It's aimed at engineering teams who treat LLM features the way they treat any other production code: unit-tested, regression-tracked, and security-reviewed. The OSS core is genuinely free and self-hostable, while a paid enterprise SaaS adds team dashboards, continuous red-teaming, and centralized policy management. The company claims adoption at 156 Fortune 500s and contributors from OpenAI, Google, and Microsoft, which lines up with how often you see it referenced in serious LLMOps stacks.

Integrations cover GitHub, GitLab, Jenkins, IDE plugins, and an MCP proxy for inspecting agent traffic. It's model-agnostic by design, so you can use it as a neutral harness across providers rather than being locked into any one vendor's eval framework.

Editor's take

Promptfoo is the closest thing the LLM world has to a default testing framework, and the fact that the OSS version is actually usable on its own is rare in this category. If you're past the demo stage and shipping to real users, the security and regression coverage alone justify the setup time.

— The AI Tool Bible editorial team

Pros

  • ✅ Genuinely open source and self-hostable, not a fake-OSS funnel
  • ✅ Model-agnostic; works across OpenAI, Anthropic, local, custom APIs
  • ✅ Red-teaming covers prompt injection, jailbreaks, PII, policy violations
  • ✅ Clean CI integration with GitHub/GitLab/Jenkins for regression catching
  • ✅ Large community and Fortune-500 adoption signal staying power

Cons

  • ⚠️ YAML-heavy config has a learning curve for non-engineers
  • ⚠️ Enterprise pricing is opaque (contact sales only)
  • ⚠️ Red-team scans can be slow and token-expensive at scale

Pricing (captured )

TierMonthlyKey features
EnterpriseCustom
  • All Community features
  • Custom red teaming limits
  • Team sharing & collaboration
  • Continuous monitoring
On-PremiseCustom
  • All Enterprise features
  • Deployment on your own infrastructure
  • Complete data isolation
  • Dedicated runner

Prices change; we show the grid as captured on Nov 13, 2026. Check current pricing on the vendor's site →

Facts

Platforms
cli, api
API available
Yes
Free trial
Yes
Open source
Yes (license: NOASSERTION)
GitHub stars
23,011 (checked Nov 13, 2026)
Last GitHub push
Aug 28, 2026
First commit
May 2024
Archived on GitHub
Yes
Company
OpenAI
Model used
Multi-model
Vendor pages
Changelog
Docs pages indexed
333 pages we found in the docs
Hacker News
23 stories · top 384 points · last Mar 9, 2026 · checked Nov 19, 2026
Earliest Wayback capture
2023-05-20 (earliest archived capture of the site, not a launch date)

Where this tool sits

Use cases

llm-evalsred-teamingprompt-regressionrag-testingai-securityci-cd-guardrails

Recent updates from OpenAI

Frequently asked

What types of LLM applications can Promptfoo test?
Promptfoo is model-agnostic and works across providers like OpenAI, Anthropic, local models, and custom APIs. It supports testing LLM apps, prompts, and RAG pipelines.
Does Promptfoo support security testing?
Yes, it includes automated red-teaming capabilities that cover prompt injections, jailbreaks, PII leakage, and policy violations.
Is Promptfoo available for free?
The core toolkit is open-source, free, and self-hostable. A paid enterprise SaaS option is available for team dashboards, continuous red-teaming, and centralized policy management.
How does Promptfoo integrate with development workflows?
It integrates with GitHub, GitLab, Jenkins, IDE plugins, and an MCP proxy for inspecting agent traffic, allowing for CI/CD guardrails and regression catching.
Who is Promptfoo designed for?
It is aimed at engineering teams that treat LLM features like production code, requiring unit tests, regression tracking, and security reviews. It is not intended for no-code prompt comparison or chat playgrounds.

Compare Promptfoo with similar tools

All alternatives →

Explore related

Reviews