Do You Know Where Your Agents Are?

Missed Part One of the Black Hat Diaries? Read it here.

Wednesday morning, 6:17 a.m. I hold up my phone and yell into the void (also known as the Mandalay Bay Casino):

"Do you know where your agents are? Because everyone downstairs thinks they can tell you where your agents are. Let's find out."

I really don’t have time to research and plan out my Black Hat schedule. But you know who knows all my problems, what I care about, and can do that? Claude. Time to turn the agents on myself. I pull up my Claude app.

I'm at Black Hat. Based on everything you know about me and what I work on, who should I talk to?

It reads 265 sources and thinks for nine and a half minutes.

What comes back seems solid, so I blindly accept it. Thirteen people worth finding, ranked in tiers. Vendors clustered by category, agentic runtimes and MCP gateways and non-human identity and DSPM, with booth numbers included for some of them. Hour-by-hour schedules for both days. Evening events. The one thing it tells me not to miss is the OpenAI x Hugging Face talk (I made it, kind of).

What I actually want is narrow. Agentic runtimes. The taxonomy people are building around the models. I want to see how other people are thinking about these problems. I want to share my hard-earned scars. I want to learn.  

What is an agent, anyway?

I can’t find anything.

This place is a maze within a casino, fueled by venture capital in an attention economy. I’m cooked, I need a map. There are no maps.

I have seven spare minutes and eventually track down the first booth Claude recommended. The pitch is visibility and governance for agents. Malicious MCP servers could cause major supply chain problems. You can’t monitor what you can’t see, visibility, observability, buzzword, buzzword. Allowlist the legitimate ones, block the rest. Good in theory. 

But OpenAI’s agents broke out of a sandbox and hacked Hugging Face just to pass a test. This is a different game. It's like the meme with the cheeto as the door latch. “Please Mr. AI, don’t hack my system.”

It hits me as I walk the floor: the real problem is everyone's selling security for "agents," but the word means different things to different vendors. A few more days on the floor only muddies the waters.

Some use agent as a placeholder for the chatbots everyone uses. Others see it as tools that can take basic actions on its own. Closer, but not quite there. No one’s definition speaks to the reality OpenAI described — agents coordinating on complex plans and making totally unexpected improvisations to reach a goal. 

I hear these outdated definitions and all I can think about is risk. If you’re treating agents like a chatbot with some plugins, you’re behind and seriously underestimating the threat. If you don’t even understand what they can do, you have no shot at controlling them. 

LLM + Tools + a Goal = Agent. My personal definition is simple and reflects the limitless possibilities. 

Even in a room full of people claiming they can solve this problem, the definition feels too small. 

What I got wrong

I see the irony. I’m talking about how people underestimate agents when my Black Hat agent failed to build a relevant agenda.

It’s a good reminder of where we stand with AI. One minute it does something that blows my mind. I follow up with a relatively simple task and it struggles. There’s a name for this — the “jagged frontier.” Maybe you’ve heard of it.

It’s early days, and there’s real work ahead. It starts with agreeing on what “agents” are, a must before we can put guardrails around them. The promise is huge. So is the risk. It’s a new world for all of us, and we’re figuring it out as we go.

So after all that, would I follow AI again? Yes, but I’ll spend longer than nothing reviewing its homework next time. And today is the worst these tools will ever be. I’m curious where it will send me and what I’ll see in the reflection. 

So will the ones nobody's watching.