
Promptfoo
Not yet source-checkedOpen-source eval and red-teaming framework for LLM apps, prompts, and RAG pipelines.
Machine-drafted from the vendor's site · not yet source-checkedProfile updated Link checked Pricing captured How we rate
In short
Promptfoo is an open-source toolkit for testing, evaluating, and red-teaming LLM applications. It is best for engineering teams shipping LLM features to production who need versioned evals, regression tracking, and automated security checks in their CI/CD workflows.
Pick Promptfoo if you ship LLM features to production and want versioned evals, regression tests, and automated red-teaming in CI.
Skip it if you just need a chat playground or a no-code prompt comparison tool with zero setup.
Promptfoo is an open-source toolkit for testing, evaluating, and red-teaming LLM applications. It started as a CLI for running prompt and model comparisons against test cases in YAML, then grew into a broader AI security platform with automated vulnerability discovery, guardrails, and CI/CD integration. You write tests once, point them at any provider (OpenAI, Anthropic, local models, your own API), and get diffable scorecards covering correctness, regressions, prompt injections, jailbreaks, PII leakage, and policy violations.
It's aimed at engineering teams who treat LLM features the way they treat any other production code: unit-tested, regression-tracked, and security-reviewed. The OSS core is genuinely free and self-hostable, while a paid enterprise SaaS adds team dashboards, continuous red-teaming, and centralized policy management. The company claims adoption at 156 Fortune 500s and contributors from OpenAI, Google, and Microsoft, which lines up with how often you see it referenced in serious LLMOps stacks.
Integrations cover GitHub, GitLab, Jenkins, IDE plugins, and an MCP proxy for inspecting agent traffic. It's model-agnostic by design, so you can use it as a neutral harness across providers rather than being locked into any one vendor's eval framework.
Promptfoo is the closest thing the LLM world has to a default testing framework, and the fact that the OSS version is actually usable on its own is rare in this category. If you're past the demo stage and shipping to real users, the security and regression coverage alone justify the setup time.
— The AI Tool Bible editorial team
Pros
- ✅ Genuinely open source and self-hostable, not a fake-OSS funnel
- ✅ Model-agnostic; works across OpenAI, Anthropic, local, custom APIs
- ✅ Red-teaming covers prompt injection, jailbreaks, PII, policy violations
- ✅ Clean CI integration with GitHub/GitLab/Jenkins for regression catching
- ✅ Large community and Fortune-500 adoption signal staying power
Cons
- ⚠️ YAML-heavy config has a learning curve for non-engineers
- ⚠️ Enterprise pricing is opaque (contact sales only)
- ⚠️ Red-team scans can be slow and token-expensive at scale
Pricing (captured )
| Tier | Monthly | Annual | Key features |
|---|---|---|---|
| Community | Free | Free |
|
| Enterprise | Custom | Custom |
|
| On-Premise | Custom | Custom |
|
Prices change; we show the grid as captured on Oct 22, 2026. Check current pricing on the vendor's site →
Facts
- Platforms
- cli, api
- API available
- Yes
- Free trial
- Yes
- Open source
- Yes (license: NOASSERTION)
- GitHub stars
- 22,943 (checked Oct 29, 2026)
- Last GitHub push
- Aug 28, 2026
- First commit
- May 2024
- Archived on GitHub
- Yes
- Company
- OpenAI
- Model used
- Multi-model
- Vendor pages
- Changelog
- Docs pages indexed
- 333 pages we found in the docs
- Hacker News
- 23 stories · top 384 points · last Mar 9, 2026 · checked Oct 19, 2026
- Earliest Wayback capture
- 2023-05-20 (earliest archived capture of the site, not a launch date)
Where this tool sits
Use cases
Recent updates from OpenAI
Frequently asked
- What types of LLM applications can Promptfoo test?
- Promptfoo is model-agnostic and works across providers like OpenAI, Anthropic, local models, and custom APIs. It supports testing LLM apps, prompts, and RAG pipelines.
- Does Promptfoo support security testing?
- Yes, it includes automated red-teaming capabilities that cover prompt injections, jailbreaks, PII leakage, and policy violations.
- Is Promptfoo available for free?
- The core toolkit is open-source, free, and self-hostable. A paid enterprise SaaS option is available for team dashboards, continuous red-teaming, and centralized policy management.
- How does Promptfoo integrate with development workflows?
- It integrates with GitHub, GitLab, Jenkins, IDE plugins, and an MCP proxy for inspecting agent traffic, allowing for CI/CD guardrails and regression catching.
- Who is Promptfoo designed for?
- It is aimed at engineering teams that treat LLM features like production code, requiring unit tests, regression tracking, and security reviews. It is not intended for no-code prompt comparison or chat playgrounds.