At the Merging of Rivers
So there I was...
Facing my first big ethical dilemma in AI. Now, this was during the first wave of AI (they are in order: predictive, generative, and agentic) over a decade ago. Compute was finally powerful enough to run the algorithms developed in the 1970's and do some really interesting things in predictive modeling.
I left a significant Silicon Valley big data company and joined an early Austin AI start-up. We landed a global manufacturing client and were tasked with modeling consumer behaviors for their finance side of the house. If our predictions worked, the potential lift was well into the tens of millions. Our biggest perceived barrier was getting access to the data. Yes, there was a time when that was a problem.
After months of back and forth, we finally got some sample data for model training... and quickly realized it was absolute crap. I mean, there was nothing you could do with it. Our data scientists said they could clean it up, but without knowing what actually moved the needle I saw no way that it could work.
So, I prepared to tell the client the truth and defy our team and find another project that could leverage clean data (and risk a significant revenue loss). The alternative would be to let the team burn a bunch of cycles on the customer's dime and prolong the agony and inevitable disappointing outcome. My path was super clear.
But I never got the chance. Organizational politics preempted my regularly scheduled program, and I found myself departing for what became much better opportunities and impact elsewhere. The client outcome, I learned later, was as I predicted. And it was at this moment something struck me.
I used to live near a town in Germany called Koblenz. The name means "at the merging of rivers," and I found myself living at the confluence of ethics (my academic passion) and AI (my technology passion)... es war mein neues Koblenz. And now that we are well into the third (agentic) wave of AI, I find yet another new confluence of AI and ethics: AI telling you what is ethical.
I just came across a paper published in March of this year that addresses the willingness of humans to accept the delivery of ethical advice by AI. And I have some very strong feelings about why this is the worst idea ever.
The paper comes out of Wharton, and it is careful, honest work, which is exactly why it deserves to be taken seriously and argued with. The researchers took real dilemmas from The Ethicist column in the New York Times and compared GPT-4's advice to that of the columnist himself, Kwame Anthony Appiah, a philosophy professor whose actual job this is. Readers rated the machine's advice just as useful. Given a blind choice, 57 percent preferred it. Even a panel of pastors, a rabbi, and academic ethicists leaned toward the machine. Then the bigger experiment: ask people in the abstract who should give them ethical advice, and 73 percent say a human. Show them the advice with the sources labeled, and the machine's share climbs to 47 percent. Hide the source entirely, and it wins outright at 54.
Sit with that progression, because it is the whole story. The advice never changed. Only the knowledge of where it came from did. People discount moral guidance the moment they learn a machine wrote it, and the paper, to its credit, says the quiet part out loud: perhaps organizations could present the AI's recommendation first and reveal its origin later, to help trust along. The authors immediately flag the ethical delicacy of that suggestion, and they should, because I wrote about this exact architecture last week. A plan that works best when someone is kept from knowing what they are dealing with is not a trust strategy. It is the thing my whole shelf of stories keeps warning about, dressed in better methodology.
But set the disclosure question aside, because my stronger feeling is about the destination itself. Notice what the study measured: whether the advice sounded useful. It did. Of course it did. These models are the most fluent advice-sounding machines ever built, trained on centuries of our books and sermons and philosophy, and they will hand you a balanced, warm, well-reasoned answer on demand. What the study could not measure is the part of ethical advice that has ever made it worth taking. Appiah signs his answers. A pastor sits in the same pews as the people she counsels. My old mentor who told me hard truths had scars from the times the truth cost him. When I was ready to walk into that client and tell them their data could not deliver what we promised, the advice I needed was not eloquent. It needed to come from somebody who understood what it would cost me, because it would have cost them too. The model has no client to lose. No reputation, no conscience keeping it up at night, no seat at the table where the consequences land. The paper concedes this in its own conclusion: AI lacks consciousness, personal experience, and material interests. That is not a footnote. That is the whole disqualification. Advice without stakes is not counsel. It is content.
And there is a compounding problem. Judgment, as I keep saying in these pages, is conditioning. You build it by working the hard call yourself, by sitting with who gets affected, by being wrong and carrying it. An organization that pipes its moral questions to a model gets back answers that sound like wisdom, minus the workout that produces any. The muscle atrophies exactly as fast as the convenience grows. And unlike my crap-data client, you may never get the clean failure that tells you something is wrong, because the advice will always sound fine. That is what makes it dangerous. Bad data announces itself eventually. Smooth moral outsourcing never does.
Koblenz sits where the Rhine meets the Mosel, and the Germans built a monument at the point where the waters join, because a confluence is a powerful thing. It is also where the flooding happens. Two rivers are running together right now, our oldest questions and our newest machines, and I have lived at that corner my whole career. The machines can carry information to the meeting point. They can even help us think. But somebody with a name, a face, and something to lose has to decide.
So, two questions this morning, and I mean them both.
When you face your next hard call, will you want an answer that sounds right, or an advisor who has something at stake in telling you the truth?
Und wissen Sie den Unterschied? Do you know the difference?
#AI #AIEthics #EthicalDecisionMaking #Trust #Leadership #MoralJudgment #STIW