AI red teaming for enterprise agents: why one pre-launch test isn't enough

TL;DR

• AI red teaming is adversarial testing: you attack your own AI system on purpose to find where it fails before someone else does. For an enterprise agent, that means far more than probing the model for a jailbreak. The agent can call tools, carry memory between sessions, and act with inherited permissions, and each of those is its own attack surface.
• A one-time, pre-launch red-team is a snapshot of a system that keeps moving. Models get swapped, prompts get edited, new tools get connected, and the memory fills with real data. As of September 2026, 66% of enterprises run pre-deployment red-teaming for public-facing agents, but pre-deployment is where most stop, and that's the gap.
• The frameworks and tools are maturing fast: the OWASP Top 10 for LLM Applications, the NIST AI Risk Management Framework's call for adversarial testing, and open scanners like garak, PyRIT, and Promptfoo. Use them, and map findings to the OWASP risks so nothing gets hand-waved.
• A red-team finding is only worth as much as your ability to enforce the fix. That's an architecture question: can you scope the permission, gate the action, trace the reasoning, and roll it back? Computer, by DevRev handles that through Safe Actions and Trusted Answers.

What is AI red teaming?

AI red teaming is the practice of deliberately attacking your own AI system, the way an adversary would, to surface security and safety failures before they reach production. Borrowed from cybersecurity, it means simulating prompt injection, jailbreaks, data leakage, and misuse against the live system and grading how it holds up. The goal isn't a passing score; it's a list of the specific ways your system breaks.

For a plain chatbot, that mostly means testing the model: can a crafted prompt make it say something it shouldn't? For an enterprise agent, the definition has to stretch, because an agent does things a model alone can't.

Red-teaming an agent is not red-teaming a model

Here's the mistake worth naming up front. Most red-teaming guidance, and most of the tooling, targets the language model: feed it adversarial prompts, measure the jailbreak rate, log the toxic outputs. That work matters, and it maps to the top of the OWASP Top 10 for LLM Applications (LLM01 Prompt Injection). But an agent isn't a model with a chat window. It reads from and writes to your systems, it invokes tools, and it carries context forward. The moment an agent can take an action, the interesting attacks move past the prompt.

Three surfaces beyond the prompt don't exist for a bare model, and they're where the real damage lives:

Read the agentic risk literature and the same theme recurs: prompt injection, agent privilege escalation, data poisoning, and tool abuse are the classes that separate agent security from model security. NIST's AI Risk Management Framework and its Generative AI Profile call for red-teaming against exactly these, and the OWASP GenAI Security Project's 2026 edition now maps risks alongside agentic-application coverage. If your red-team only exercises the prompt, you have tested the smallest part of the system.

Why one pre-launch test isn't enough

Even a thorough red-team has a shelf life, because the thing you tested doesn't stay still.

An enterprise agent in production is a moving target. The underlying model gets upgraded. Someone edits the system prompt to fix a tone issue. A new connector gets wired in, adding a tool the last red-team never saw. And the memory, empty on day one, fills with real customer data that a poisoning attack can now target. The clean report you signed off in Q1 describes a system that no longer exists by Q3.

This is where the numbers get pointed. In a Forrester 2026 panel cited in enterprise adoption research, agents without automated evaluation had a 47% rollback rate over the prior year; agents with full eval coverage had a 9% rollback rate. Microsoft's AI Red Team, for its part, has run more than 200 red-team operations on AI systems since 2018, and its own lessons from testing 100 generative AI products point to adversarial testing as continuous work, not a launch gate. The pattern is consistent: teams that test once ship confidently and roll back often; teams that test continuously catch the regression before a customer does.

Continuous red-teaming is the answer, and it's a discipline, not a product you buy once:

• [ ] Run the adversarial suite on every material change, not just before launch: model swap, prompt edit, new tool, new data source.
• [ ] Cover all four surfaces, model, tools, memory, and permissions, not just the prompt.
• [ ] Map every finding to a framework risk (OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS) so it can be tracked and closed, not just noted.
• [ ] Automate what you can with scanners like garak, PyRIT, or Promptfoo, and keep a human red-team for the creative, multi-step attacks tools miss.
• [ ] Re-test after every fix to confirm you closed the hole and didn't open a new one.
• [ ] Feed findings back into your evaluation set so yesterday's exploit becomes today's regression test.

None of this replaces the security program around the agent; it complements it. Red-teaming finds the holes. For the defensive posture that keeps them closed, see AI agent security. For the discipline of validating behavior before and after release, testing AI agents in production and testing autonomous AI systems safely go deeper on the how.

From findings to guarantees

A red-team is only as useful as your ability to act on what it finds. This is the part most tooling can't help with, and it's where the conversation quietly shifts from testing to architecture. Suppose your red-team proves the agent can be talked into an unauthorized refund. What do you actually do with that? You can't out-prompt every future attacker. What you can do is make the dangerous action impossible to complete unsupervised, so the exploit stops mattering.

That's the design principle behind how Computer, by DevRev is built. Consequential actions pause for human-in-the-loop approval, so an agent tricked into a refund still can't issue it without a person signing off. The agent inherits the exact permissions of the person it acts for and can't escalate past them, which closes the privilege-escalation surface at the source rather than testing for it forever. Every action lands in an immutable audit trail that names the agent, the initiating user, the tool, and the outcome, so a red-team finding becomes a specific, traceable line rather than a vague worry. And any action can be reversed, so a bad call is a rollback, not an incident. These are the Safe Actions that turn a finding into a guarantee, and you can watch them hold in this short walkthrough.

The other half is what the agent is allowed to believe. A red-team that poisons memory or slips in stale context is attacking the agent's answers, not just its actions. Trusted Answers is the counter: responses are grounded in your real business data with sources cited, and the agent reasons only over data the acting user is authorized to see, enforced at the data layer rather than filtered after the fact.

This is also where red-teaming stops being a separate exercise and becomes part of the build. Before anything ships, Agent Studio's Playground and bulk evaluation let teams run adversarial and regression suites against the agent, scored on evaluators like faithfulness, tool-calling accuracy, and instruction adherence, with one-click rollback to the last good version. The findings become evals, the evals gate the build, and the session traces show exactly what the agent did when an attack landed. Access enforcement underneath it is covered in AI access control.

Frequently asked questions

What is AI red teaming in simple terms?

It's ethical hacking for AI. A red team deliberately attacks your own AI system, using prompt injection, jailbreaks, data-leakage attempts, and tool misuse, to find where it fails before a real adversary does. For agents, it extends past the model to the agent's tools, memory, and permissions. The output is a specific list of weaknesses to fix.

How is red teaming an AI agent different from red teaming an LLM?

Red-teaming an LLM tests whether input can bend the model's output, mainly jailbreaks and prompt injection. Red-teaming an agent adds the surfaces a model doesn't have: tool calls it can be steered into, memory that can be poisoned across sessions, and inherited permissions that can be escalated. An agent takes actions, so its failures cost more than a bad sentence.

What tools are used for AI red teaming?

Common open-source options include NVIDIA's garak (an LLM vulnerability scanner), Microsoft's PyRIT, and Promptfoo for configuration-driven adversarial testing. Teams map findings to frameworks like the OWASP Top 10 for LLM Applications, the NIST AI Risk Management Framework, and MITRE ATLAS. Automated scanners catch known patterns; a human red-team is still needed for creative, multi-step attacks.

Is one red-teaming exercise before launch enough?

No. An agent changes after launch, new models, edited prompts, added tools, and accumulating memory, and each change can reopen a closed hole or open a new one. Red-teaming has to run continuously, triggered on every material change, with findings fed back into the evaluation suite so past exploits become permanent regression tests.

How does red teaming relate to compliance frameworks?

Frameworks increasingly expect it. The NIST AI Risk Management Framework's MEASURE function calls for adversarial testing, and the OWASP Top 10 for LLM Applications gives a shared vocabulary for classifying findings. Mapping each red-team result to a named risk turns adversarial testing from an ad-hoc exercise into an auditable control you can show a regulator or a customer's security team.

Where to start

If you're only testing the prompt today, the first move is to widen the aperture: run your next adversarial pass against the tools, the memory, and the permissions too, map each finding to an OWASP risk, and put the whole suite on a trigger so it fires on every model swap, prompt edit, and new connector. That gets you continuous coverage of how your agent breaks.

The harder question is what you can do about what you find, and that's worth pressure-testing against your own stack: when a red-team proves the agent can be pushed into an action it shouldn't take, can you enforce the fix in the architecture, or only warn about it? Bring a real red-team finding to a walkthrough and watch the approval gate hold, the permission stay scoped, and the action roll back. That's the difference between a finding you file and a guarantee you can stand behind.