Promptfoo - LLM Evaluation and Red Teaming

Open-source LLM evaluation and red teaming framework P Cloud Infrastructure

Basic Information

  • Company/Brand: Promptfoo (acquired by OpenAI in March 2026)
  • Founder: Ian Webster
  • Country/Region: USA
  • Official Website: https://www.promptfoo.dev/
  • GitHub: https://github.com/promptfoo/promptfoo
  • Type: Open-source LLM evaluation and red teaming framework
  • Founded: 2023
  • Status: Joined OpenAI in March 2026, remains open-source (MIT license)

Product Description

Promptfoo is a CLI and library for developers to evaluate and red team LLM applications. It covers 50+ vulnerability types, from prompt injection to jailbreaking, and can generate thousands of context-aware attacks tailored to applications. Used by OpenAI and Anthropic, it has over 300,000 developers and 127 Fortune 500 companies as users. Acquired by OpenAI in March 2026, it will continue to remain open-source.

Core Features

  • 50+ Vulnerability Types: Direct/indirect prompt injection, jailbreaking, PII leakage, tool misuse, toxic content
  • Adaptive Red Teaming: Smart AI agents generate context-specific attacks from the start
  • Agent Tracing: Tracing and debugging agents in complex workflows
  • Compliance Mapping: OWASP, NIST, MITRE ATLAS, EU AI Act compliance mapping
  • MCP Testing: Comprehensive MCP (Model Context Protocol) testing capabilities
  • Multi-round Testing: Supports multi-round testing via plugins and attack strategies
  • Web UI: Elegant web interface for displaying test results
  • CI/CD Integration: Command-line and CI/CD integration
  • Python Deep Integration: Full Python SDK support
  • YAML/JSON Configuration: Declarative configuration files

Business Model

  • Open-source (MIT): Fully free and open-source, remains so after OpenAI acquisition
  • Enterprise Edition: Enterprise features integrated into OpenAI Frontier platform
  • Community-driven: Active open-source community

Deployment Methods

  • Run directly via npx
  • Install via npm/yarn
  • Install via Python pip
  • Docker
  • CI/CD pipeline integration

Target Users

  • LLM application developers (300,000+)
  • Enterprise AI security teams (127 Fortune 500 companies)
  • AI red teamers
  • Teams requiring LLM evaluation
  • Organizations needing automated AI testing in CI/CD

Competitive Advantages

  • Adopted by 300,000+ developers and 127 Fortune 500 companies
  • Internally used by OpenAI and Anthropic
  • Covers 50+ vulnerability types
  • Adaptive red teaming (not static prompt lists)
  • MCP testing capabilities (industry-leading)
  • Comprehensive compliance framework mapping
  • Acquisition by OpenAI validates technical value

Comparison with Competitors

DimensionPromptfooGarakDeepTeam
MaintainerOpenAI (acquired)NVIDIAIndependent community
Vulnerability Types50+100+ attack vectors40+ vulnerability classes
UIWeb UICLICLI
Red Teaming ApproachAdaptive AIStatic + AdaptiveAttack strategies
Evaluation CapabilitiesBuilt-inLimitedLimited
MCP TestingSupportedNoneNone
Compliance MappingOWASP/NIST/MITRE/EUAVIDOWASP/NIST

Relationship with OpenClaw Ecosystem

Promptfoo is the go-to tool for LLM evaluation and security testing within the OpenClaw ecosystem. Its adaptive red teaming capabilities can automatically generate context-specific attacks against OpenClaw AI agents, helping to uncover potential security vulnerabilities. MCP testing capabilities are particularly important for OpenClaw's MCP integration scenarios. CI/CD integration enables automated security testing, ensuring every update is security-verified. OWASP, NIST, and other compliance mappings help OpenClaw meet various security compliance requirements.

External References

Learn more from these authoritative sources: