CONTROLLED AGENCY| Issue 14: The Future of AI Defense Isn’t Preventing Compromise. It’s Surviving It.

Why containment, not prevention, is becoming the defining principle of AI security

There’s a line I keep hearing from people I genuinely respect, and I’ve come to think it’s half right in a way that quietly costs us the most important idea in AI security. It comes up in Slack threads, on conference panels, in the hallway conversations that are usually more honest than the talks. It goes something like this:

“You can’t really defend AI systems; stop pretending you can. The mature move is to threat-model your exposure, accept that something will get through, and put your energy into solid incident response.”

And every time I hear it, I nod. Because most of it is right.

Prevention as a fantasy is correct. Anyone still selling you a guardrail that “stops prompt injection” is selling you comfort, not security. A model that cannot reliably tell instructions from data cannot be patched into doing so. It’s architectural, not a bug you close. So yes, if you plan to prevent every manipulation of an AI system, your plan is a wish.

The assume-breach humility is correct too. It’s the hard-won lesson of two decades of security, and pretending AI is exempt would be arrogance.

Threat modeling isn’t optional. Incident response isn’t optional. If you can’t detect and respond, you’re not in the game.

So I’m not here to argue against any of that. I’m here because I think the framing smuggles in something false, and I only noticed because I used to believe it myself.

We’re using the word “defend” too loosely

Here’s the first problem, and it’s hiding in the sentence itself.

Watch what happens when different people say “we can’t defend AI.” The AppSec person means one thing. The ML safety researcher means another. The GRC lead means a third. Nobody’s defining the word, so everyone’s arguing past each other and mistaking it for agreement.

If “defend” means prevent every prompt injection, then no, we probably can’t. But if it means reduce the damage a successful prompt injection can cause, then we defend distributed systems every day, and have for decades. Security has never been about making compromise impossible; it’s been about making compromise survivable.

So the sentence “we can’t defend AI” is doing something sneaky. It takes the narrowest, most impossible definition of defense, proves that one can’t be met, and then quietly retires the whole idea of defending. That’s the move I want to pull apart.

The binary that isn’t real

Coming from offensive security, I lived in the “you can’t defend, so respond” worldview. You get in eventually. The interesting work is detection and response. Prevention is for people who haven’t been on the attacking side. I brought that instinct straight into AI security, and for a while it felt right.

Here’s the thing that broke it for me.

The position quietly collapses two very different things into one. It treats the choice as: prevention (impossible) versus incident response (realistic). Pick the realistic one.

But there’s a whole layer sitting between those two that the framing walks right past. It’s not prevention. It’s not incident response. It’s CONTAINMENT.

And containment isn’t just another security layer bolted on. It changes what success even means.

Prevention measures success by how many attacks never happen. Incident response measures success by how quickly you recover after one does. Containment measures success by how little an attacker can accomplish after they’ve already succeeded. Those are three fundamentally different objectives, and the argument I keep hearing only ever names the first and the third. It skips the middle one entirely, and the middle one is where almost all the leverage is.

Why this suddenly matters, when it didn’t before

Now, a fair objection: containment is not new. Least privilege, sandboxing, RBAC, network segmentation, capability systems. We’ve had these ideas for decades. So why write about it now like it’s a discovery?

Because AI changes the economics of the whole thing, and it does it through one specific shift most people haven’t fully sat with.

Traditional software only executes code you wrote. You review it, you ship it, and barring a vulnerability, it does what it was written to do. The attacker’s job is to smuggle in code you didn’t write.

An agent is different. An agent executes plans it generates, in real time, in response to inputs you don’t control. You didn’t write the sequence of actions it’s about to take. It decided them, and that means the classic attack sentence changes shape.

It used to be: “an attacker executes malicious code.”

Now it’s: “your own trusted system decides to execute a sequence of individually legitimate actions, on behalf of someone who manipulated it.”

Read that twice, because it’s the whole thing. No single action is malicious. Each one is within the agent’s rights. The compromise isn’t a foreign payload sitting in memory. It’s your legitimate system, using its legitimate authority, toward an end someone else chose. There’s nothing to quarantine.

The malware is the agent doing its job.

That’s why containment stops being one option among many and becomes the primary move. The model is no longer the attack surface that matters most. Its authority is.

The reframe that actually helps

Which turns the whole question around.

For a while I described the goal as “make the manipulated agent unable to do much.” That’s close, but it’s soft. Here’s the sharper version.

Assume the manipulation succeeds. Don’t argue with it, don’t try to prevent it, grant it. Now ask a different question entirely: what authority survives the compromise?

Because that’s the only question you actually control, you cannot reliably stop the agent from being talked into something. You can decide, in advance, what it’s able to do once it has been.

An agent scoped to exactly the authority its task requires cannot turn one manipulated interaction into full-environment compromise, because the permissions to do so were never on the token. An agent whose credentials can be revoked mid-session doesn’t keep acting on stolen authority while your IR team is still spinning up. An agent whose blast radius was bounded at design time turns what would have been a catastrophe into a logged, contained non-event.

None of that is prevention, cos the injection still lands, the agent still gets manipulated, and you’ve conceded all of that, honestly. What you’ve changed is the size of the incident. You’ve made it small enough that your incident response is almost boring, which, if you’ve done real security work, you’ll recognize as exactly what winning feels like.

This is the layer the “we can only threat-model and respond” crowd keeps stepping over. And I get why. It’s less dramatic than prevention and less familiar than incident response. It lives in the unglamorous decisions nobody schedules time to make: scope this agent’s permissions, make its identity revocable, bound its delegation, limit what one compromised interaction can reach—boring architecture, but the most valuable kind.

Here’s the sentence I keep coming back to. AI security is becoming less about deciding whether the model can be manipulated, and more about deciding what a manipulated model is still allowed to do. Almost everything I’ve learned in the last two years supports that one line.

Why “just respond” fails harder than people think

Let me ground the whole argument in the thing that actually moved me, because it wasn’t a theory. It was a number.

I’d assumed detect-and-respond was a solid fallback. Then I measured it against multi-turn agent attacks, the kind that escalate across a conversation instead of arriving in one payload. Per-request monitoring, the thing most teams lean on, caught zero percent of them before the critical turn. Not “fewer than we’d like.” Zero, structurally, because every individual turn was benign and the harm lived in the accumulation.

Sit with what that does to the “we’ll just respond” position. If your detection fires after the critical turn, your incident response begins after the agent has already acted with the authority it holds. For an autonomous agent wired to real tools and real data, “after it acted” can mean after it moved money, exfiltrated records, or triggered a downstream action you can’t take back. Detect-and-respond, applied to accumulating agent attacks, often means detecting the fire after the building is gone.

That’s the trap. Incident response assumes there’s a meaningful gap between compromise and consequence, a window you can act inside. For a scoped agent, that window exists, because there’s a limit on what the compromise can reach. For an over-authorized agent, the compromise and the consequence are the same moment. There’s no window to respond into.

So containment isn’t a nice complement to incident response, it’s the thing that makes incident response possible at all.

Where I might be wrong

This is also why, over the last two years, I’ve gradually shifted my own work away from asking “can we build a better detector?” toward asking “what authority should an agent retain after compromise?” The former improves models. The latter changes architectures. If you’ve followed this series, that shift is the throughline running under all of it, the trajectory work, the identity work, the enforcement work. They’re all the same question wearing different clothes.

Now here’s the part I actually want you to push back on.

Some of you are already forming the objection: containment is just prevention wearing a different hat. You’re preventing the consequence instead of the manipulation, but it’s still prevention, and it’ll fail the same way, because a determined attacker will chain through whatever authority you did grant.

That’s a fair hit, and I don’t have a complete answer. Containment isn’t perfect either. Scoping reduces blast radius; it doesn’t zero it. So maybe the honest position is three layers, not two: you can’t prevent the manipulation, you can contain most of what it’s able to do, and you still need incident response for what containment misses.

But that’s a very different sentence than “you can’t defend AI, so just threat-model and respond.” It puts the highest-leverage work back in the middle, where I think it belongs.

So tell me where this breaks. Is containment just prevention I’m refusing to call prevention? Are there agent architectures where you genuinely can’t scope the authority down without breaking the thing that makes the agent useful? Those are the two objections I can’t fully answer, and I’d rather hear them than pretend they’re not there.

Or maybe the real question underneath all of this is simpler and more uncomfortable: have we been using the word defense to mean the wrong thing all along?

AI security will probably never eliminate compromise. It doesn’t have to. It only has to make compromise small enough that the organization survives it. That’s a very different definition of defense than the one we started with, and I think it’s the one we’ll eventually converge on.

I’ve changed my mind on this once already, from the offensive-security reflex to where I am now. I’d rather change it again than defend a position just because it’s mine.

Where do you land?