Hold My Beer
So there I was...
Breaking the thing we had just built. In the early days of social enterprise, everything was highly experimental, including organizing principles for permissions and access. Hashtags seemed like a viable option, right up until the point I demonstrated I could lock out almost everyone on our platform from their own content without their permission. The "should we build it this way?" debates were truly epic, and the underlying challenges always came down to social engineering, unintended consequences, and the limits of our red team imagination. My crew would tell me they "idiot proofed" something, and then I would say "hold my beer" and assume the role of an idiot they could never imagine.
The hard part was taking the conceptual work we were doing and making it enterprise-ready as we sprinted along the fastest path to revenue. We had to land that first big client (and we eventually did). So, the weekend after we released our friends and family beta, I sandboxed an instance and successfully made it completely unusable from the lowest permission level. I had no plan, I just thought to myself, "I wonder if they thought of this?" and entered keystrokes. Monday came and we quickly shifted gears. It is a lesson I still carry with me, and I get daily reminders as I see organizations fail to grasp the capabilities of AI agents. And not just their own agents, but also the agents deployed by others who say ho ho ho and slide down the chimney into their living rooms while they sleep.
That instinct, the one that makes you poke your own creation to find where it breaks, is the entire job of oversight. EY just measured how many organizations have actually wired it into their AI. Oh, lordy. In September they released a survey of 202 senior AI decision-makers, the board members and C-suite and VPs at US public companies north of a billion in revenue, in other words, the people who own this. What they found is a sizeable chasm between what these companies have written down and what is actually happening on the floor.
Start with the paperwork, because it looks terrific. Ninety-eight percent of them have a formal AI governance policy. Yippee! Then forty-seven percent admit they have thrown that policy out the window when a deployment felt urgent. SMH. Ninety-one percent are already running agentic AI, our friends that take actions on their own. Of those, forty-nine percent have not updated their governance for what agents actually do, and eighty-five percent say at least some of their systems are out there taking actions with no human watching in real time. Yep, not a soul looking. In most of these companies, machines are doing things right now that no person will see until after they are done.
And here is our chimney. Twenty-six percent of them cannot detect unauthorized AI agents operating inside their own walls. Not agents they deployed and lost track of (which is its own whole problem), but agents they never invited, running on their systems, invisible to the people nominally in charge. Their single biggest fear, named by eighty-one percent, is a third-party AI-enabled cyberattack, which is to say they are most afraid of precisely the thing a quarter of them admit they will never see coming. Bad Santa is already in the living room, and he's spotted the cookies.
It would be one thing if they had the people to fix this and simply had not gotten around to it. They don't. Sixty-nine percent say they lack the in-house expertise to even evolve their governance controls, and nearly two-thirds say the same about implementing or designing them. The gap is not laziness, but rather a shortage of the exact judgment required at a time the machines are fast enough that you need it most. You cannot put a competent human in a chair you have no one qualified to sit in.
None of this is theoretical. Eighty-nine percent of these organizations hit an AI-related risk in the past year, and thirty-six percent have already taken a materially negative hit, data loss, financial damage, operational disruption, and harm to the brand. More than a third are not bracing for the day after. They are already living in it.
So let me share with you the one number in this report I think is most important. Among the companies that actually ran a formal AI assurance review, sixty-four percent had to significantly change at least a quarter of their AI systems. Twenty-nine percent had to pause at least a quarter. Twenty-five percent had to pull the plug on at least a quarter outright. When somebody finally sat down and played the idiot nobody imagined, a quarter of the machines turned out to be doing something that had to be stopped. Every one of those paused systems is an awkward Monday morning dev meeting.
I will give EY props for this report, keeping in mind they sell AI assurance services. So a survey concluding that you need more assurance certainly bolsters the call to action. Fair enough. But their own technology risk leader said that the biggest agentic AI risk is that human oversight has not evolved, and the numbers back him up. This is the empty chair with a clear cost associated with it and not simply a theoretical problem. The chair where somebody pokes the system before the stranger does, asks the dumb question, wonders out loud whether anyone thought of this. Most of these companies have left it empty, and a third of them already have the scar to show for it.
I learned my version of this years ago on a quiet Austin weekend with no plan and a lot of mischievous energy. I broke the thing we built before a real idiot could. It cost me a weekend and prepared us for our first big client. You can pay for the oversight now, in the boring hours when nothing is on fire, or you can pay for it later when everything is. The machines are faster than they have ever been, they are taking actions nobody is watching, and some of them came down a chimney you did not know you had. Somebody needs to be sitting up by the fire. Hold my beer.
#AI #AgenticAI #AIGovernance #Oversight #RedTeam #ShadowAI #Accountability #TheEmptyChair #STIW