I’m Gerd Dani, President of FreeAstroScience. My chair hums softly on tile as I roll into a clinic corridor, the air cool and a little antiseptic. Three ideas float in the chatter around AI today. Let the model decide. Human oversight is theatre. Efficiency is justice. I believed none of them. Then a small scene snapped everything into focus.
A nurse glanced at a dashboard. A red score blinked like a siren, harsh and steady. “The system says low priority,” she murmured, fingers tapping a plastic mouse that felt too warm. The doctor frowned, then waved me through anyway. That tiny gesture held more sense than the blinking score. My aha arrived like a clean breath: “human in the loop” isn’t a guarantee—it’s often a talisman, a comfort word we rub when control is thin. Scholars warn about this drift from “the human who supposedly controls” to “the algorithm that supposedly guarantees.” Treating performance as permission turns tools into oracles, and that’s a problem you can hear in every anxious beep of a waiting room monitor .
The Talisman Trap: When We Outsource Legitimacy
The story goes like this. We add a person “in the loop,” then act as if everything’s safe. The person becomes symbolic, like a lucky charm on a keyring, smooth and pointless under the thumb. In parallel, the model’s output starts to carry authority—“it works, so it must be right.” That’s the sleight of hand. A crisp printout glows under white lights, and suddenly efficacy masquerades as ethic .
I call this the hum of false certainty. It sounds like a fan that never stops, soothing and loud enough to drown doubt. But doubt is exactly what keeps institutions honest. The fix isn’t chanting “oversight” louder. It’s rebuilding how decisions earn their right to stand.
What “Human-Centred” Actually Means
“Human-centred” isn’t nostalgia. It’s not the smell of paper files or the warmth of a handshake, though both matter. It means judging systems by lived effects, protecting relationships, and deliberating ends before you optimise means. In health, education, and justice—spaces thick with breath, touch, and tension—this is non-negotiable. Performance can help, but people carry responsibility. This is the moral geometry the AI world keeps forgetting .
When Metrics Become Morals
A metric is a flashlight. Wave it carelessly and it blinds. Let it define “good,” and you’ll miss the quiet things—the tremor in a voice, the grit of fatigue, the long silence after bad news. When indicators become targets, they start replacing our goals. That’s how “faster triage” can mean colder care, even if dashboards look sharp and clean. Researchers have been clear: algorithms steer; they don’t understand. Numbers capture the countable, not the meaningful texture of a life .
Here’s a simple map you can run with. It fits on a page, and it breathes.
| Domain | Common Metric | What It Misses | Human Check | Practical Signal |
|---|---|---|---|---|
| Healthcare triage | Risk score | Context, language barriers, fear | Offer a short interview | Shaky voice, shallow breath |
| Education | Completion rate | Mastery depth, curiosity, stress | Viva or portfolio check | Eyes brighten at a tough question |
| Justice | Recidivism probability | Support networks, stigma costs | Community testimony | Tone softens when plans are concrete |
Start From Vulnerability, Not From Averages
I live with wheels under me. I think about ramps before doors. That’s design starting at the edge, not the centre. Public systems should do the same: co-design with the most fragile, keep non-digital channels alive, and build real appeal paths when the model gets it wrong. You can feel the difference like cool shade on hot skin. It dignifies everyone and even reduces costly rework over time .
Grey Boxes, Not Black Magic
Full transparency can be theatre, too. Dumping weights and code smells like hot circuitry and tells most people nothing. There’s a smarter middle path: grey boxes. Bake domain rules and safety limits into the model, then learn the rest from data. Return reasons people can act on—key variables, error margins, boundary conditions—and name who fixes mistakes. That’s governance you can touch, like the firm click of a brake engaging on a slope .
So, what exactly does that look like in math you can deploy tomorrow? It’s simpler than it sounds.
Probability from a model:
Decision with guardrails and human pause:
Where utility respects people, not just speed:
In words you can feel on the tongue: act when the score is high and safety constraints hold; otherwise pause. Measure benefit, subtract costs and emotional load, then reward justified reversals. That pause is the cool sip of water systems usually deny.
One Number, One Promise
Here’s my compact deal with any team. One auditable reason per automated decision. One top factor, one error range, one named owner for appeal. It smells like fresh air after rain because it clears the fog. This aligns with the practical guidance on reasons, margins, and responsibility you’ll find in serious governance work—less mystique, more grip .
From Aura To Accountability
AI will keep getting slick. The screens will glow brighter, the edges smoother to the touch. But the question stays coarse and human: who is served, and who is left colder in the waiting room? The answer isn’t an amulet. It’s public reasoning, contested ends, and designs that begin with those most exposed. That’s how we turn computation from a loud promise into a steady instrument.
This piece was written for you by Gerd of FreeAstroScience, where we explain complex science in plain language, with your life in mind. I want your systems to stand up in daylight—the kind that warms your face through a window and shows dust you can finally wipe away. Let’s build for that light, not for the glow of dashboards.
And if you ask me whether AI is our talisman, I’ll smile at the soft whirr of my wheels and say no. It’s a tool—powerful, yes—but only as good as the ends we declare and the reasons we can defend .
Post a Comment