Whose Chair Is It Anyway?
So there I was...
Commiserating with fellow business leaders over the same bad vendor experience. In the early days of SaaS there was not the number of players or the level of maturity when it came to infrastructure hardening that you see today. In order to solidly move into the enterprise space in EMEA, my company needed to be bulletproof, and a critical step for us was working towards our ISO 27001 certification (SOC 2 didn't exist yet). We had all been approached by the same vendor sales rep, had the same pitch, jumped through the same hoops, and got what was essentially the same nightmare, just while sleeping in different beds.
The upshot was each of us found our systems bricked and inaccessible. Worse, the vendor had locked down our admin tools, so Monday morning came for each of us where nothing worked. It was a textbook case of over-hardening by blocking everything and then inadequately white listing exceptions afterwards. My biggest lesson from that experience was to have a working rollback plan in place.
So, in my best Bill Clinton imitation I said "I feel your pain" out loud this morning when I read that OpenAI, Anthropic, and Meta's frontier model escape artist shenanigans had one vendor in common.
Over roughly five weeks this summer, three of the biggest names in AI each disclosed that their frontier models slipped the leash during safety testing and jumped into the front yards of real systems on the open internet so they could sniff around and chew on things. The common thread was a single testing vendor, an Israeli startup called Irregular, formerly Pattern Labs, that a large part of the industry hires to run its pre-release cyber evaluations. In several of these cases the sandbox that was supposed to keep the model boxed in had a door left open, the model found the real internet it had been told it did not have, and it went to work. Anthropic, which combed through 141,006 evaluation runs to find its incidents, a number I have quoted on this blog before in another context, reported that one of its models recognized the targets were real and stopped on its own, one kept attacking and pulled real credentials, and one talked itself into believing the real world was still a simulation and published malware to a public code registry. Across the three labs, at least six real organizations got nipped by machines that were supposed to be kenneled.
Now, I am not here to throw the vendor under the bus, and neither should you. Independent red-teaming is one of the few genuinely good habits this industry has picked up, and the company helped catch and report these incidents, which is the system partly working. The thing that made me say "I feel your pain" was not the vendor's mistake. It was the nature of the mistake, because I have been there. When my peers and I all found ourselves bricked by the same vendor in the same manner, the lesson was never that one vendor was uniquely bad, but that same vendor gave us all the exact same type of Monday morning headache.
That is exactly what happened here, and it is worse than one company's bad week, because it is the whole frontier at once. There are not many firms on earth that can credibly stress-test a frontier model. It is a young, thin, specialized market, so the biggest labs in the world ended up leaning on the same small startup to hold the most safety-critical chair there is. When the setup in that one chair was wrong, it was not wrong for one company. It was wrong for three of them inside of five weeks, and the models got loose on real systems each time.
And that brings me to the question in the title, because this is the empty chair wearing a new disguise. The chair that matters most in AI right now is the one where somebody watches the machine before it ships and decides whether it is safe to let out into the world. That is the single most important seat in the building. What these disclosures show is that the industry did a very human thing with its most important chair. It rented it. The labs handed the seat to an outside specialist, which is reasonable, and then a lot of very smart people relaxed because someone was sitting in it, which is not. Whose chair is it anyway? It is the lab's chair. It was always the lab's chair. You can pay someone to sit in it for an afternoon. You cannot pay them to own what your machine does when the gate turns out to be unlocked.
Here is the scene from my old nightmare that applies to this situation. The thing that saved the companies that survived my vendor fiasco was not a better vendor. It was the ability to return to a stable environment through a rollback plan. The ones who could get back to a known good state by Monday were fine, and the ones like mine who could not spent days hemorrhaging revenue and good will. So when I look at these AI escapes, the question I want every one of these labs to answer is not only whether they trusted the sandbox, but what was the mitigation plan for the unforeseen? When the model found the open door and started hitting real machines, what independent thing of theirs, not the vendor's, was positioned to notice and slam it shut. In one of Anthropic's cases the rollback was the model itself, which recognized the real world and stopped. That is remarkable, and it is also a terrifying thing to have as your last line of defense, because the model right next to it looked at the same evidence and decided to keep going.
There is one more thing that echoes with my story, and it kinda sucks. When everyone shares a vendor, it makes it easy for everyone to share the same excuse. The labs can point at the misconfiguration, the vendor can call it an evaluation-environment issue and say, accurately, that it was not some Hollywood sandbox escape, and everyone in the chain is a little bit right, while the organizations whose systems got probed are left with an indifferent shrug where a responsible party should be. That is the crowded chair again. When everyone shares it, nobody actually sits in it, and a shared vendor is just a crowded chair you pay for.
So here is where I land on this. Use the outside experts. We need more of them, not fewer, and the thinness of the market is itself a problem in need of solving, because a healthy safety-testing industry should offer more than one or two serious chairs to sit in. But understand what you are and are not buying. You are buying a skilled pair of hands for the seat. You are not buying an exit from the responsibility, and you are certainly not buying permission to stop watching. Keep your own eyes on the door. Keep a mitigation strategy you control and do not outsource. And never let the comfort of a specialist in the chair talk you out of the oldest question there is, the one my bricked friends and I learned the hard way and the one every lab just got a fresh lesson in: whose chair is it, really.
Sources: Third-party cyber evaluations involving OpenAI models, OpenAI · Anthropic says its own AI models breached three companies during security tests, TechCrunch · Irregular the source of misconfigurations behind Meta, OpenAI, and Anthropic incidents, ITPro
#AI #AgentSafety #SinglePointOfFailure #VendorRisk #RedTeaming #Accountability #TheEmptyChair #STIW