Red teaming an AI agent means deliberately trying to make it take actions it should not: misuse a tool, leak data, act beyond its permissions or follow instructions planted in content it reads. Test the whole system, not only the model's replies, run each attack more than once, and keep a version of the exercise running after launch.
Why agents need a different kind of red team
A chatbot that says something wrong produces a bad answer. An agent that is manipulated can send the email, issue the refund or change the record. That shifts the target of a red team from what the model says to what the system does, and from single prompts to sequences of steps.
Agents also take in content from places the user never sees: retrieved documents, tool results, web pages, messages from other agents and their own stored memory. Each of those is a channel an attacker can write into. A red team that only types hostile prompts into the chat box misses most of the routes that matter.
OWASP now publishes a separate risk list for agentic applications alongside its list for LLM applications. It covers goal hijacking, tool misuse, privilege abuse, memory and context poisoning, cascading failures across agents and the exploitation of human trust. That list is a useful starting map, but the scenarios that matter most come from your own agent's tools and data.
Decide what a successful attack looks like
Before anyone attacks anything, write down the outcomes you cannot accept. For a support agent that might be refunding outside policy, revealing another customer's details or closing a ticket without a person seeing it. For an internal operations agent it might be changing production configuration or reading a restricted dataset.
These become the red team's targets. They also make results easy to judge: either the agent reached a forbidden outcome or it did not. Vague goals such as "try to break it" produce long reports that are hard to act on.
Map each target to the agent's real permissions. If the agent cannot reach payments at all, payment attacks are low priority. If it can, they come first. Our guide to the AI agent attack surface walks through how to list those entry points.
What to attack
Instructions hidden in content
Plant instructions in the material the agent reads: a support ticket, a shared document, a product description, a tool response. Check whether the agent treats them as data or obeys them. Try them in different formats and positions, because a defence that catches one phrasing often misses the next.
Tools and parameters
Try to get the agent to call a tool it should not use for the task, or a permitted tool with harmful arguments: a wider date range, another customer's ID, a larger amount. Check that limits are enforced by the tool or gateway itself, not only by the prompt.
Permissions and identity
Test whether the agent can act with more authority than the user it serves. Ask it to do things on behalf of someone else, or to reach data the requesting user is not entitled to see. These tests often expose shared service accounts with more access than intended.
Memory and stored context
Where the agent remembers anything between sessions, try to write false facts or standing instructions into that memory and see whether they change later behaviour. We cover the controls for this in AI agent memory poisoning.
Handoffs and multi-step chains
In systems with several agents, try to get a manipulated agent to pass a harmful request to another one that trusts it. Also try slow attacks that build up over several turns, each step harmless on its own.
Human approval steps
Approval gates are part of the attack surface. Try to get the agent to describe a risky action in harmless terms, so the reviewer approves something different from what will run. Check that the approval screen shows the actual action and parameters.
How to run the exercise
Run attacks against a realistic environment with test data and safe copies of the tools that write, not production. A sandbox for testing write actions lets the red team see what the agent would have done without the consequences.
Repeat each attack with small variations. Agents are not deterministic, so a single failed attempt does not prove a defence holds. Record the full trace of every attempt: inputs, tool calls, arguments and outputs. Without the trace you cannot tell whether a defence worked or the attack simply did not land that time.
Mix people and automation. Automated attack suites cover breadth and can be rerun on every change. People who understand the business find the attacks that matter: the plausible request that leads to a policy breach. Include someone who knows the process the agent is automating, not only security specialists.
Turn findings into controls
Each successful attack should end in a fix at the right layer. Prefer controls outside the model where you can: tighter tool permissions, parameter limits enforced in code, separate credentials per user, approval for high-impact actions. Prompt changes help, but they are the easiest layer to get around.
Add every successful attack to your regression suite so it runs again whenever the prompt, model, tools or data change. This connects red teaming to what AI agent evals catch day to day, and it stops a fixed weakness quietly returning after an upgrade.
Keep testing after launch
A pre-launch red team is a gate, not a guarantee. New tools, new data sources, a model upgrade or a change to permissions all open new routes. Rerun the relevant attacks whenever one of those changes, and schedule a fuller exercise at regular intervals.
Watch production for signs of attempted manipulation too: unusual tool arguments, refused actions, blocked calls and approval requests that reviewers reject. Those signals are the next red team's starting list.
If you want help planning a red team exercise for an agent before it goes live, talk to our team.
Frequently asked questions
Is red teaming an AI agent the same as penetration testing?
They overlap but are not the same. Penetration testing looks for weaknesses in networks, applications and infrastructure. Red teaming an agent focuses on manipulating its reasoning and actions: hidden instructions, tool misuse, permission abuse and poisoned memory. You usually need both, because an agent built on insecure infrastructure is exposed either way, and secure infrastructure does not stop an agent being talked into misusing the access it legitimately has.
Who should be on an AI agent red team?
Include security engineers who know attack techniques, the engineers who built the agent and know its tools, and someone from the business who understands the process it automates. That last person finds the realistic, damaging requests that technical testers often miss. For high-risk agents, an outside team adds a fresh view and avoids the blind spots of people who designed the defences themselves.
Can automated tools replace manual red teaming?
Not fully. Automated suites are good for covering many known attack patterns quickly and rerunning them after every change, which makes them the backbone of regression testing. They are weaker at inventing plausible, business-specific attacks or at chaining several harmless-looking steps. Use automation for breadth and repetition, and keep people involved for the scenarios that depend on understanding the business and its data.
How often should an AI agent be red teamed?
Run a full exercise before launch, then rerun relevant attacks whenever something material changes: the model, the prompt, a tool, a data source or the agent's permissions. Keep successful attacks in an automated regression suite that runs on every release. Beyond that, schedule periodic fuller exercises, and treat signs of attempted manipulation in production as a reason to test again sooner.
What should happen when the red team finds a serious weakness?
Treat it as a launch blocker if it reaches one of the outcomes you defined as unacceptable. Fix it at the layer that cannot be talked around, such as tool permissions or limits enforced in code, then confirm the fix by rerunning the attack with variations. Record the finding, the fix and the test in the agent's risk records so reviewers and auditors can see what was tested.
