The Chair Nobody Sat In

A note before we begin: this piece touches on real harm, including suicide. Support resources are at the end.

So there I was...

Looking at alarming lab results. The physician covering the ER that night had dismissed the physical signs: jaundice, edema, itching, and confusion. In all fairness, his specialty was far removed from the emergency medicine he was asked to practice for one shift so our overworked ER docs could get a break. And an 18 year-old dependent daughter of a Master Sergeant and college student was not the usual suspect for what we would soon discover. I was at least able to get a liver panel added to the routine labs after I drew her blood. The results came back after she was discharged before seeing what we did not want to see: elevated ALT, AST, ALP, Bilirubin; low Albumin. It was liver failure.

It was the late 1980's, so all of our processes were manual. Need care at an Air Force hospital? Go to medical records, show your ID, and then hand carry your file to your appointment. That resulted in a lot of files piling up, waiting for providers to add their notes and lab slips to be added. I knew that in a handoff from one shift to the next, something could get missed, and this was one of those cases where I could not let that happen. I took the lab slip with me to the hospital commander's office and gave him my usual report of the night's activities in the ER, and handed him the lab slip as I discussed the young lady who was now driving home with her parents.

The next few hours were a blur, but it included getting her back to the ER and then transporting her code 3 (lights and siren) to University of Nebraska Medical Center. A few days later she had a liver transplant, and on Christmas Eve she gave me a huge hug from her hospital bed when I came to check on her.

In the days that followed we identified a process gap in the formal review of what transpired that night, triggered by her return to the ER and emergency transfer to a civilian facility. The fix would prevent a patient from being discharged under similar circumstances. Looking back now, I think any of my peers would have done the same thing, I just happened to be on that night to catch it. But the chance that something like that could have been missed sits with me still to this day. And it drives me in always asking the empty chair question: what happens when there is no one there to catch it?

Last week I had an amazing conversation with a bioethicist, and as we discussed the ethics of technocare she reminded me of a story where there was an empty chair in almost every step of what was an epic government failure that literally destroyed lives. She was talking about Robodebt.

If you are in the United States, you may have never heard of it, so here is the Reader's Digest version. For about four years, the Australian government ran an automated scheme to claw back money it claimed welfare recipients had been overpaid. The mechanism it used is called income averaging. It takes a person's total earnings for a year, runs a bi-monthly average as if income never varies, and generates a debt notice based on very imperfect information. You owe us thousands, prove us wrong. Hundreds of thousands of people got those letters and much of the debt was pure invention. The method was actually unlawful, it did not comply with Australia's own social security law, and a Royal Commission would later call it a massive systemic failure. It has been tied to real devastation, including suicides. That is why the report itself opens with crisis line numbers. It is that damn bad.

Now here is why my bioethicist friend set it down next to the empty chair, and why I have not been able to put it down since.

Go back to my liver panel situation for a second and there's a reason I opened with that story. That night, the only thing standing between an 18 year old and a discharge that could have killed her was someone whose entire purpose was to save lives, who looked at the circumstances and felt something was off. I just simply refused to let it go. One person, in one chair, catching the thing the imperfect process would have missed. Robodebt is what happens when you take a person that cares deeply about outcomes out of the equation. And in this case, not just from one chair, but from basically every chair that mattered. At the point where someone should have asked whether averaging was even legal, empty chair. At the point where someone should have checked whether a debt was real before the letter went out, empty chair. At the point where the Ombudsman, the tribunal, the legal advisors, the information commissioner should have hit the brakes, chair after chair after chair sat cold and empty. The whole design was a handoff with no one at any station with the foresight to catch it.

And this is what makes this such a cautionary tale in the age of AI. The technology was stupid. I mean that technically. There was no AI here, no model, no neural anything. Income averaging is arithmetic a sixth grader could do at the front of the class on a chalkboard. On its own it was not dangerous, it became so when it negatively impacted a human life with an untrue "you are in debt to your government" letter. The government investigation that followed found that every human who could have owned that decision stepped out of the chair. The failure, top to bottom, was human absence dressed up as automation.

I have written a version of this all month, but Robodebt really encapsulates my thesis. It is The Straw Man at national scale, because the savings number came first and the unlawful method was reverse engineered to hit it. It is A Liar in the Chair, because when the lack of legal foundation began to surface, the response was, in the Commission's own words, dishonesty and collusion to keep it hidden. It is Lead From the Front stood on its head, because the Commissioner's sharpest line is that culture is set from the top down, and the top set exactly the wrong one. And it is the empty chair, not as a metaphor I reach for to make a point, but as a finding of fact.

What I find fascinating after reading through the entire report is in the recommendations, because they could have been lifted straight out of the Framework for Ethical Decision Making (FEDM). Put a human decision maker back in the loop who is not removed from the consequences. Tell people, out loud, when a decision about their life is being automated. Give them a real path to challenge it. Make accountability attached to a named person, not a system. That is the empty chair, filled. A Royal Commission spent ten months and read nearly a million documents and arrived at the thing I preach every day: somebody has to sit in the chair.

Robodebt used the dumbest possible tech stack and still cut a path of destruction through the lives of hundreds of thousands of Aussies. Today, we are handing decisions exactly like that (benefits, claims, hiring, credit, care, etc.) to systems that are far from dumb, can act on their own, and at a speed and scale we can barely imagine. If we carry the empty chair paradigm into our understanding of the current state, the least of our worries will be a scandal. We will get Robodebt on steroids.

So here is my question today, from a guy who once carried consequences printed on a lab slip into his commander's office because there was absolutely no other choice.

Every automated decision you are about to trust, every workflow you are about to let run on its own, has a chair assigned to it. The only question that matters is whether there is a human sitting in it who can be harmed by being wrong, who can see the whole picture, and who is empowered to say stop. This is the realm and responsibility of humans, not machines. So look at the systems you are standing up right now and find where to place the chair. Then ask who should be sitting in it, and what happens when there is no one?

Source: Report of the Royal Commission into the Robodebt Scheme, 2023

A DocAustin story, carrying the OLÉ mark · olemcs.com #STIW

#AI #Ethics #AutomatedDecisionMaking #Accountability #Governance #TheEmptyChair #STIW