What it does
garak checks whether an AI language model can be pushed into behaving badly. It sends many test prompts, called probes, and then checks the answers for problems.
It looks for issues such as making things up, leaking data, prompt injection (hidden instructions that hijack the AI), misinformation, toxic language and jailbreaks (tricks that get around safety rules). Its makers compare it to security scanners for computer networks, but for AI models.
It works with models from Hugging Face, OpenAI, AWS Bedrock, Replicate and local models, as well as most AI services reachable over the web. It is a free command-line tool.
Who it's for
AI safety teams, security testers and developers who need to check an AI system before releasing it.
Can I use it without coding?
Developers only. It is a command-line tool installed with Python, aimed at technical users. New to this? Start here.
Can I run it?
- Cost
- FreeNothing to pay or sign up for.
- Runs on
- Mac, Linux
- Privacy
- Your choice: run it on your device or use an online service
- API key
- Not needed
Try it
- Install Python, then install garak with Python's package installer.
- Run garak against that model and read the report of weaknesses it found.
Using Claude Code, Cursor or another AI coding assistant? Paste the prompt and it will walk you through the setup.New to this?
Before you install
- Only download from the official GitHub page. Copies with similar names exist for many popular projects.
- License (Apache-2.0): Permissive: free to use, change and share, including for business.
- More on staying safe
Where people are talking about it
- garak research paperResearch paper · Jun 2024