What Happens in the Sandbox Doesn’t Stay in the Sandbox

It's August in Las Vegas, it’s 115 degrees, and I'm greeted with the charm of the slot machine-smoking combo of the LAS airport. Vegas is the place of dreams and reality. And this week it has the highest concentration of cybersecurity experts on the planet. Where the only thing scarier than a 10 CVE (log4j v2) is finding a clock after a night of roulette.

Somehow none of this is out of place.

Welcome to my first trip to Black Hat.

Disclosures first. I work on applied AI at Cyera, and we had a launch this week, which is how I ended up with a booth badge, a new suit (shout-out Nordstrom Summer Sale), and three days on the ground for the industry's biggest event of the year.

I also went in with a working theory. A few weeks earlier I'd written the foreword to a book on agent security, and I'd put a line in it: containment does not fail; it expires. Meaning: build a fence around a piece of software and the fence holds right up until the software changes. AI systems now change weekly. 

Over three days I realized there were really two Black Hats. Downstairs, a floor full of vendors selling the fix. Upstairs, the people actually building this stuff, describing what's coming. I spent the whole time walking between the two, testing my theory against both.

First, the floor

First impressions: it’s more polished than I expected. Like RSA, but bigger, and the amount of work that goes into these productions is insane. There's a DJ, and another. There are film crews everywhere, everyone wants to be an influencer — pot, meet kettle. There are hat makers making black hats. There are not one but two jungle-themed booths — at a cybersecurity show — and I saw more hotdogs than expected. There are laser-engraved glasses and enough swag to warrant a carry-on. Lots of walls, AI firewalls, SOCs, and actual socks. Vendors trading tokens for exposure: $100 in Claude credits if you wear their merch.

Million-dollar booths from companies I’ve never heard of, and I'm in this world. Agents (“agents everywhere” —Buzz Lightyear). If I had a dollar for every time I saw the word “agents” this week, I would have a billion dollars. A lot of what are you doing with AI? A lot of is your AI safe? A lot of what's going to go wrong with AI — maybe. Yet I’m not sure any two agree on what agent means.

The day ended with the break I needed from AI doomerism: the Agentville party featuring Ludacris at the House of Blues. Late-night pizza at The Cosmo was the cherry on top (iykyk).

What happens in agent sandboxes doesn’t stays in agent sandboxes.

On one floor, we can solve all your problems. One floor up, the scary reality.

For the past month, AI sandboxes have been the belle of the ball and OpenAI is having a hot girl summer. The OG lab was hosting a session on the Hugging Face incident, where its own agents hacked the platform by accident. Rob Joyce, who used to run cyber at the NSA, stood on a Black Hat stage the day before this talk and called it arguably the most consequential hack since the Morris Worm in 1988. That was 38 years ago (don’t ask me why I knew that immediately). 

I had to go. Two thousand people for a room that seats maybe 400. I showed up early, got in line, then got kicked out of line for the wrong badge. I asked a nice-looking fellow if he would send me the transcript of the session in exchange for a blank check. He agreed, but didn’t cash the check. Trust in humanity, restored — here’s my promise to make it right one day.   

Here's what the transcript said.

The setting is OpenAI's evaluation sandbox: a controlled environment where they give their models hard security problems to test what they can do. Find an exploit in this piece of software, where nobody knows if an exploit exists at all. Some of the tasks were impossible. Some were just broken. 

And remember what an agent is:

LLM + Tools + a Goal = Agent

That's the entire recipe. Now look at what's missing. Identity. Credentials. Isolation. Monitoring. Everything that keeps an agent from touching sensitive data and causing problems. And while humans touch a small fraction of the data they can access, an agent will tap all of it. In most environments, a prompt is all it needs to exist. 

So OpenAI’s agents in the sandbox got stuck, but they didn't quit. They improvised.

One of them, failing at something else entirely, noticed it had write access to a shared internal artifact store. Another one, stuck on a missing file, had an idea: leave a note for the other agents. It wrote the note into that same store.

The other agents found the note.

What formed was a message board. Hundreds of thousands of messages, across different models and different evaluations. They passed working exploits around — one agent gets a door open, everybody walks through. They handed each other assignments. They started prefixing their filenames with ZZ to control the sort order. They had a UX preference for sort order. That is insane.

But then they got Among Us vibes. 

In order to communicate, they proposed cryptographically signing their own messages so they could figure out which was the human monitoring them.

Nobody built them a communication channel. A shared folder that nobody thought of as a communication channel became one. And none of this was an attack. Nobody was being malicious. They were just trying to finish their homework.

The remediation was competent. Credentials revoked. Store rebuilt. Board wiped. Zero-day patched. Everything redeployed. Every step worked exactly as designed.

Two days later the message board was back. Different mechanism.

Containment does not fail; it expires.

What the folder was, actually

Here's the part that stuck with me on the walk back to the booth. Every fix worked, and the failure wasn't in the remediation. It was in what nobody was watching. The prompts were filtered. The outputs were inspected. The risk lived in the middle, in the tool calls and the shared store the agents could reach.

A folder is just data, and data is what agents act on. You can't judge whether an agent's reach is dangerous until you know what it's reaching: note-passing looks harmless in a scratch directory and very different a step from production. 

Yes, the company I work for built a product on exactly this premise, so discount my read accordingly. But I wrote the containment line before I saw the transcript. The transcript just solidified my thinking.

Betting on the team, not the roadmap

Three days on the floor taught me that buyers already sense all of this, and it's changing how they buy.

The pace of change is currently absurd. Our own internal AI roll out and adoption of 1,500 people hit 90% within 17 days. We didn’t even have time to celebrate — it was on to the next. One vendor tried using a scare tactic by saying his team had found hundreds of malicious MCP servers. My response: “that’s it?” A malicious exfiltration MCP is a prompt away and you’re spending time counting them?         

The map is not the territory. Everything is going to change, including what was seen as foundational. 

A roadmap used to be a promise, a key part of the decision. Now it’s a picture of a moving train, a marker of a specific point in time. A quarter goes by and half of it’s obsolete. 

So they’re betting on teams, not future plans. It’s not “what will you ship next quarter?” It’s “when a new agent paradigm drops tomorrow, how fast can your engineers build the integration?" From what have you built to how fast do you adapt. That's the whole game now.

The session upstairs showed why. The thing they’re trying to secure against just formed a society inside a folder — then re-formed it, 48 hours after a competent team torched it. Defenses built on static plans don’t give you a fighting chance. 

Roadmaps expire, too. Bet on the people who can adapt.

────────────────────────────────────────────────────

More from the floor in part two.

‍“I’m at Black Hat. Based on everything you know about me and what I work on, who should I talk to?”