Stop red-teaming the model. Red-team the application.
Most AI red teaming tests a model. Your risk doesn’t live in the model. It lives in your application.
A generic jailbreak suite fires thousands of adversarial prompts at a chat window and grades the replies. Useful, but it has no idea that one of your tools can reach an internal service, that a RAG source holds sensitive records, or that an agent has a role-specific permission boundary.
Same model, different blast radius
The same model can be perfectly safe in one product and dangerous in another — purely because of the tools, data, and permissions wired around it. A model that politely refuses to write malware can still be the component that leaks your customer table, if the application hands it a database tool and a poisoned document in the same context window.
Grading the model’s replies tells you about the model. Your incident report will be about the system.
The hard problem isn’t generating prompts
Adversarial prompts are cheap now; that part is commoditized. The hard problem in red-teaming AI applications is twofold:
- Knowing which prompts are worth generating for this system. That takes context — what tools exist, what data is reachable, where the permission boundaries sit. Context tells you where to aim.
- Proving the exploit actually fires through the running application, not just in theory. A finding that says “the model said something spicy” is noise. A finding that says “this input, through this tool call, read that record” is evidence. Runtime tells you whether it fires.
Generic prompting gives you neither. It aims everywhere and proves nothing.
Better tests are better news
The encouraging part: application-level red teaming isn’t just more accurate — it’s more useful at every step. Findings arrive pre-triaged, because they’re tied to a real path through your system, with a real blast radius attached. Fix priority stops being a debate about hypotheticals. Security teams drown in less noise; engineering teams get reports they can act on the same day.
And it changes the conversation with the business. “Our model scored well on a jailbreak benchmark” reassures nobody. “We attacked our deployed system the way an attacker would, and here’s what held and what we fixed” — that’s assurance you can stand behind, and increasingly, that regulators and customers will ask for.
The field is maturing from benchmarking models to testing systems, the same way appsec matured from scanning code snippets to testing running applications. Teams that make that shift now aren’t just avoiding tomorrow’s incident — they’re building the confidence that lets them ship agents faster than everyone still grading chat replies.
If you’re red-teaming AI, test the thing you actually deployed. It’s not the same test — and that’s exactly the opportunity.