The Competent Chair
So there I was...
Running a code blue. It was the early 90's, and as a former Air Force medic, it turned out I was the only person working in a small Cupertino urgent care clinic who was Advanced Cardiac Life Support certified. Not the RNs, not the physicians, no one. The doc who owned and operated the clinic was a veteran Navy doctor who ran a field hospital during the Vietnam war, so when I showed up looking for a job after Desert Storm he hired me on the spot. I had more training, practical knowledge, and experience than anyone in the clinic save my boss. He was a really good guy to work for and gave me a lot of leeway because he knew I could fill a critical skills gap.
My first day on the job I decided to check the crash cart. Years of habit. I was horrified to discover that many of the meds and all of the IV bags had expired, and the decrepit Lifepak 5 sitting on top of it would not charge beyond 20 joules. The chief nurse said not to worry about it, she had never had any critical cardiac cases walk in. I told her the very fact that I discovered the crash cart was a mess guaranteed we'd get one, and so over the course of a couple shifts I got the cart up to snuff, and a new Lifepak 6 I convinced them to buy arrived a couple weeks later.
And as predicted, within a month an elderly man on antiarrhythmics showed up at the clinic confused, short of breath, and very fatigued. We hooked him up to our shiny new Lifepak 6 which revealed he had normal sinus rhythm, but bradycardic (slow heart rate), and a BP check that he was hypotensive. After I started an IV on him, an RN stayed in the room while I went to prep another patient with a laceration for sutures.
"Code blue!" A few minutes later I heard the nurse scream from the exam room, and our two docs on duty and I ran back to the room where she was frantically trying to figure out what to do. The patient was non-responsive and the monitor displayed asystole - flat line. Not good. The senior of the two docs yelled for 1MG of epinephrine as he lowered the bed flat to begin chest compressions. I looked at the BP monitor and it still showed the same low blood pressure as when he came in. It looked like the patient had not crashed out.
"Hold!," I yelled. "Check for a loose lead." Sure enough, one of the sticky patches holding a chest lead had fallen off. I adhered a new patch and attached the lead to it, and sure enough the line on the monitor popped back to life with a slow NSR. The patient had fainted, but after a quick sternum rub he opened his eyes and asked what was going on. How did I know to check the lead? Because I had been trained for this exact scenario. It was actually a part of the pass/fail megacode that earned me my ACLS certification and where others in my class failed.
While this wasn't actually the code blue I predicted, everything about that day revealed the consequences of inadequate training and certification and the potential for catastrophe from bad decisions. And so it is not hard to imagine how much worse this kind of situation can be when the lack of competence appears in the form of the military using a hallucinating AI to build a targeting package.
A story on CNN pulled me right back to that exam room. According to their reporting, in the spring of this year at the start of the fighting with Iran, an intelligence analyst attached to a special operations unit wrote a report with the help of an AI chatbot. It claimed a Chinese ship in the Middle East was carrying components of a nuclear weapons program. Warplanes were already in the air and a boarding operation was being organized when senior officials went back to the report, scrutinized the sourcing underneath it, and found the whole thing to be "entirely false." Another source called it something that "almost started a war." To this day nobody knows what that ship was actually carrying. A former senior official stated that the military's own AI tools are basically "copies of the commercial stuff." The same kind of machine that turned Kentucky into "Venecky" on a student handout almost started a shooting war with a nuclear power.
Two rooms, decades apart, the same problem appearing on a monitor. In my exam room the Lifepak 6 showed a flatline and that an old man's heart had stopped, and we were about to follow a specific lifesaving algorithm to bring him back. In the Middle East the AI said nuclear weapons on a boat. In both rooms a machine reported a catastrophe that was not happening, and in both rooms the people who trusted the readout were seconds from acting on it. The nurse was already screaming. The senior doc already had the bed flat and epinephrine called. The jets were already up. A false reading and an irreversible act came down, in each case, to someone willing to say a single word.
Hold. That word has been the core of my thesis for the life of this blog, and it asks a lot of us. I have written a great deal about the empty chair, the seat nobody fills, and the harm that follows. This story is about something more specific and more demanding, because the chair in my exam room was not empty. It was occupied by a seasoned RN who panicked and read the flatline without ruling out an issue with the machine. A highly educated GP read it the same way. An occupied chair with the wrong person in it does not merely fail to help. It lends its own authority to the machine's mistake. What stopped that code was competence, not the simple presence of a warm body in a chair. It was a particular piece of training, the ACLS megacode, the exact loose-lead scenario that washed people out of my certification class, sitting in my head at the moment it was needed.
So the lesson of that day was never that a human was present. Five of us were present. The lesson was that competence is the thing that makes a chair worth anything at all. And competence showed itself not as a hunch but as a cross-check. I did not override the monitor because I had a feeling. I looked at the other instrument. The man's blood pressure reading had not changed from the moment he walked in, and a heart that has truly stopped does not maintain blood pressure. The military's AI did not survive contact with a truly experienced set of eyes. Those senior officers refused to take the confident output at face value, they went and checked the sourcing beneath it, and the argument failed. Competence is not distrust of machines so much as the discipline to validate a machine's claim against experience and training before acting on it.
This is why the military I served in was built on deliberate friction. Two-man rules, verification, rules of engagement, a chain of people who each have to sign off before action is authorized. None of that friction exists because anyone loves extra work. It exists because someone learned, usually the hard way, that a single confident source, human or machine, is not usually sufficient.
And I cannot skip the other half of that day, because it started weeks before the code. The reason I was able to say hold was not luck. It was a crash cart I checked on my first shift when everyone told me not to bother, and found expired drugs and a defibrillator that was worthless. Competence in the moment is downstream of preparation long before the moment. That clinic had gone years without a cardiac case and had quietly decided the readiness did not matter, right up until the morning it did. We are making the same bet now, wiring confident machines into targeting and triage and intelligence and hoping the crash cart never gets called, when readiness is the one thing you cannot conjure after the alarm has already sounded.
In this example the competent chair was filled and a war did not start. But look at how thin the margin was. The catch came after the planes were airborne and not before the report was believed. A system set in motion by an AI hallucination did not pause until a particular person chose, under real time pressure, to distrust the output. We do not get to assume that person is always in the room. We were lucky that he was this time.
So this is the empty chair's harder cousin, the competent chair, and it is the one I want to focus on for today. It is not enough to put a human back in the loop if the human in the loop is the panicking nurse or the trusting analyst. The seat has to be filled by someone trained and experienced enough to recognize when a confident readout is not what it seems and is willing to yell "Hold!" Every one of us is going to spend the rest of our working lives sitting beside machines that will, now and then, tell us with total confidence something akin to the heart has stopped or the ship has a nuke. A machine that does not know it is wrong and will not hesitate to continue. Somebody in the room has to be competent enough to know when the machine is wrong and say something to stop the wheels already in motion. My question is whether we are training for the next time, or just praying that the right person happens to be on shift.
Source: US military had close call after using AI for false intelligence report, Zachary Cohen, CNN
#AI #Hallucination #HumanInTheLoop #NationalSecurity #Competence #Training #TheEmptyChair #STIW