promptfoo
A CLI and library for evaluating, red-teaming, and comparing LLM prompts, agents, and RAGs with simple declarative configs.
What is promptfoo?
Promptfoo is a CLI and library for evaluating and red-teaming LLM applications. It helps developers test prompts, agents, and RAGs, compare different AI models, and identify security vulnerabilities through automated evaluations. The tool enables teams to ship secure, reliable AI applications by replacing trial-and-error with systematic testing.
AI developers, ML engineers, and product teams building LLM-powered applications who need to ensure prompt reliability, model performance, and application security.
Developers choose Promptfoo because it provides comprehensive LLM testing capabilities while keeping all evaluations local and private. Its battle-tested approach, flexible configuration, and integration with CI/CD pipelines make it a practical solution for production AI applications.
Overview
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Use Cases
Best For
- Automated testing of LLM prompts and responses
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
