A command-line tool for red-teaming and vulnerability scanning of large language models (LLMs).
Garak is a command-line vulnerability scanner designed specifically for large language models (LLMs). It systematically probes LLMs for security weaknesses including hallucination, data leakage, prompt injection, toxicity generation, and jailbreak vulnerabilities. The tool helps developers and security teams assess the robustness of their LLM deployments against various attack vectors.
AI security researchers, ML engineers deploying LLMs in production, and red teams responsible for assessing the security of generative AI systems. It's particularly valuable for organizations using commercial or open-source LLMs that need to understand model vulnerabilities.
Garak provides a comprehensive, open-source framework for LLM security testing with extensive probe coverage and support for numerous model providers. Unlike generic security tools, it's specifically designed for the unique attack surfaces of large language models with a modular architecture that allows for custom testing scenarios.
the LLM vulnerability scanner
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Supports a wide range of LLMs including Hugging Face, OpenAI, AWS Bedrock, Replicate, and custom REST endpoints, as detailed in the LLM support section.
Includes probes for DAN attacks, encoding-based injection, toxicity, data leakage, and many other vulnerabilities, with a modular design for custom extensions.
Generates detailed JSONL reports, hit logs, and debugging information, enabling comprehensive analysis and integration into pipelines.
Allows developers to create custom probes, detectors, and generators, as highlighted in the plugin development guide.
Lacks a graphical user interface, requiring terminal expertise and scripting for operation, which can be a barrier for non-technical users.
Requires environment setup with Conda, multiple API keys, and specific installations, as noted in the install instructions, adding overhead.
Focuses solely on vulnerability detection without providing tools or guidance to fix identified issues, leaving mitigation to users.