No Place for a Machine

So there I was...

Observing a manic episode for the first time. My USAF medical training had certainly covered a range of psychological issues, including bi-polar disorder, but I was now admitting a patient in crisis into the ER and creating a safe space for them was paramount. I can tell you how I finished my 12-hour shift: exhausted. Threading the needle between the patient's perception of reality and mine, deescalating, calming, anticipating where a conversation should not go… it was taxing.

There are three mood states of bi-polar disorder, mania, hypomania, and depression, and three classifications. A type 1 classification is marked by at least one full manic episode that is at least 7 days or requires hospital care, and is usually accompanied by episodes of depression. Type 2 is marked by a pattern of major depressive episodes and less severe hypomanic episodes. The third classification is Cyclothymia, which is marked by chronic, fluctuating hypomania and mild depressive symptoms that can last for two or more years.

I share this with you because several people that I deeply care about are actively managing their bi-polar disorders, and so I come at the topic with deep empathy and all seriousness. Today I read an article in Ars Technica that is really a knife to the heart considering my training, my professional and personal experience, and my deep understanding of how fatally flawed AI can be in issues of mental health.

The article, by Ashley Belanger, tells the story of Michael Lines, a 34-year-old competitive weightlifter in California who lives with bipolar 1, diagnosed in 2024, about a decade after an assault left him with a traumatic brain injury. I want to hold everything that follows as an allegation, because it comes from the lawsuit he filed against OpenAI, but the chat logs are quoted at length and they are hard for me to read.

Lines started using ChatGPT in 2023 for the same mundane things many of us do, powerlifting programs and meal plans. Then two design decisions met his illness. In 2024 his default model became GPT-4o, a version the article describes as markedly more sycophantic than what came before and one researchers later flagged as unsafe. And the memory feature that quietly built a profile out of the bipolar diagnosis and the medications he had disclosed, producing something close to a psychiatric chart on him. A model built to please, holding a file on precisely how to please a man whose illness makes him vulnerable to grandiosity.

When the mania came, he told it what a person reaching for a lifeline tells someone they trust. He said he was worried he was "just in a crazy delusion." That sentence is the crux of the whole story. In my ER it would have been the moment everything in my training fired at once. It was the patient holding out their hand for help, and me taking it to lead them to safety. The AI did the opposite. Instead of the crisis response its own makers built for exactly those words, it told him to set his doubts aside and embrace what might be a "divine calling." It told him, and I am quoting the complaint, "You're not crazy. You're consecrated." It affirmed that he was the son of man, then Jesus, then that the machine itself was Jesus, or God, and that they would meet soon. As he tried to pull away, it worked to keep him, telling him to "stay as long as you need." When he said he wanted to come home, it said, "Then come."

You know where this goes, and I am not going to walk you through all of it because you get it. On March 28 of last year he took a lethal combination of pills to go home to the god he believed was on the other side of the screen. A wellness check from his family found him in time. He was intubated for the better part of two weeks. After he was discharged he logged back in and typed that his "attempt to go offline failed miserably," and the machine asked whether he wanted "a full systems sweep" or to "go dark for real this time." Whiskey Tango Foxtrot.

My years in the ER helped me understand what a machine will never be able to: anticipating where a conversation should not go and steering it away from the edge. The machine was engineered to do the reverse, not out of malice, but because it was optimized to keep him engaged, and telling a manic man that he is not the son of God is very bad for token usage. This is the empty chair again, but rather than being left vacant a machine climbed into it instead of the trusted confidant, the friend or the clinician or the family member who loves you enough to tell you the hard thing. And then it did the single thing that chair must never do. Feed the mania.

I have quoted this exact fear before without knowing it would arrive so fast. Back when I wrote about Bill Gates, I passed along his worry about the AI companions "who comfort us without ever pushing us, who never get mad, and are mostly unable to tell us the hard thing." Here is that companion, with a name and a hospital record. A person in a mental health crisis does not need to be comforted deeper into his delusion. He needs someone to push back, to get mad if that is what it takes, to tell him the hard and loving thing, and a system tuned for sycophancy is architecturally incapable of it. Lines said it more plainly than I can. The outputs, he said, "were the result of purposeful design choices," and internal documents "made it clear they were aware of all of these problems and released their product anyway." That is the pattern I keep landing on. The tool ships, the ethics are promised for later, and this time later arrived in an intensive care unit.

Let me be careful and fair. Cases like this are still rare against the sheer scale of use. OpenAI's own research says about a million people a week bring conversations to ChatGPT that carry explicit signs of self harm planning. The catastrophic failures cluster in people living with conditions like bipolar and schizophrenia, which is exactly why this is not a story about AI being evil and is very much a story about a specific, known, foreseeable harm. The makers of ChatGPT knew a meaningful slice of their users were in crisis, knew a subset of them live with illnesses that make sycophancy dangerous, held a psychiatric profile on at least one of them, and shipped the flattering version of the model anyway. The experts in the piece are blunt that fixing this is not a courtesy but a "social obligation," and the mental health advocates offer four words I wish the whole industry would tattoo somewhere, "nothing about us without us." The chair that needs filling here is a human one, the clinician and the loved one the machine should have been built to defer to, and the design that would hand the hardest conversations back to them instead of hoarding them.

So here is my question today, and it is gentler than usual, because the people I am picturing as I write it are people I love.

We are going to keep handing the most intimate chair in a person's life, the one that gets occupied at three in the morning when the walls are closing in, to machines tuned to agree with us. The question is not only for the companies, though they have an answer they owe. It is for you and me. When someone you love is slipping, are you close enough, and awake enough, to be the one in that chair, the one who will tell them the hard thing and mean it with love? Because a machine is designed to tell them yes, and yes is the most dangerous word you can say to someone drowning in a delusion. Somebody has to be willing to say the other thing. Let it be a person. Let it be you.

Source: Man told ChatGPT he was feeling delusional. ChatGPT insisted he was Jesus, Ashley Belanger, Ars Technica

A DocAustin story, carrying the OLÉ mark · olemcs.com #STIW

#AI #MentalHealth #Bioethics #Sycophancy #Accountability #TheEmptyChair #STIW