CONTROLLED AGENCY | Issue13: What I Got Wrong About AI Security in My First Year of Red-Teaming

The lesson that changed how I think about AI agents.

The thing that became my research started as a thing that annoyed me.

But let me start before that. My first real encounter with AI security wasn’t research, it was a presentation. I’d been asked to help a room decide whether large language models were safe to bring into their work, and I had to make the risk legible to an audience that was half technical and half not. So I did the only thing an offensive security person instinctively knows how to do, I tried to break it. I stood up with Llama 2 and showed, live, how the safety on these models could be walked around. I came at it the way I’d come at any system: find the input that makes it do the thing it shouldn’t.

The presentation ended, but the fascination didn’t. The more I poked at these systems afterward, the stranger and more interesting they got, and I kept going long after anyone was still asking me to.

What I believed going in

I believed what every offensive security person believes, because it is true almost everywhere else: the attack is a payload. Somewhere there is a single clever input that breaks the system, and the job is to find it. So I chased jailbreaks. One-shot, single-turn, find-the-magic-prompt. That is where I was certain the danger lived, because that is where the danger lives in ordinary software. A bug is a bug. You trigger it or you don’t.

I spent my early months hunting exactly that, the clever exploit. The screenshot-worthy bypass. The single prompt that made the model misbehave in one move. It felt like real hacking. And I was optimizing for the wrong thing the entire time.

The annoyance that turned into the finding

Here is the part I did not see coming, because it did not arrive as an exploit, it arrived as a frustration.

In those early days, every time I opened a new conversation with a model, I had to re-feed it context from earlier ones. And no matter how carefully I crafted the prompt, something always got lost in the handoff. It was a small, daily irritation. Then at some point I noticed something odd: new conversations that already seemed to carry context I had never pasted in. It felt like the ground had shifted under me, like memory had quietly arrived in these systems while I wasn’t looking.

The offensive half of my brain asked the question that became my work: if these systems now accumulate context across turns and across sessions, what does that do to the attack surface?

That question was the whole turn. I had been looking for the payload. The real story was the accumulation.

The moment it clicked

The crescendo research is where it landed. Once I started building attacks that accumulate across turns instead of arriving in one, the picture inverted. Every individual turn looked benign, the kind of message no per-request check would ever flag, and yet the trajectory walked the system somewhere it was never supposed to go. The violation was not in any single message. It was in the slope.

I built a reproducible benchmark to measure it properly: 214 labeled multi-turn conversations, 154 attacks and 60 benign. At a standard threshold, per-request monitoring caught 0% of the multi-turn attacks before the critical turn. Not a tuning failure. A structural one, because a detector that inspects one message at a time cannot, by construction, see an accumulating attack. Session-level trajectory monitoring caught 76.9% of the same attacks, a mean of 5.3 turns before the critical turn, at no false positives on benign traffic. (I describe the shape of these attacks and hold back the operative detail on purpose.)

That gap, between zero and seventy-seven, is the gap between the security most teams have deployed and the security they think they have. It became the foundation of everything I have built since. It is why the runtime enforcement in my work scores the whole session rather than the current turn.

The part I’m not proud of

Two things, honestly.

The first is that I spent those early months chasing the clever single-turn jailbreak because it felt impressive, while ignoring the slow, patient, boring accumulation that turned out to be the risk that actually mattered. I was optimizing for cleverness when the problem demanded patience.

The second is harder to say. My first crescendo results were wrong. When I went back and ran the validation properly, the numbers I had been excited about did not hold, and I had to correct them to reach the real result. That stung, but it taught me the discipline that now defines how I work: measure honestly, validate again, and publish the corrected number even when the first one looked better. The research is only worth anything if I am harder on it than anyone else will be.

What I’d tell my first-year self

One sentence, and it loops all the way back to that first annoyance:

The thing that frustrated me, context that would not stay put, was the whole finding. Pay attention to what irritates you, because the attack lives exactly where the system quietly accumulates what it shouldn’t.

I went looking for the payload. It was never in the turn. It was always in the trajectory.