Public Sphere

AI learns from what was. It cannot help but extend yesterday's world into tomorrow.

Prejudice, old power structures, blind spots. Whatever sits in the training data comes back as the answer.

Language models learn from text. From a great deal of text, written by people over the past few decades, carrying everything those people thought about the world. That text is the training material. Whatever shows up in it often counts as likely to the model. What is likely counts as normal. And what is normal is what you get back as the answer.

This isn’t a cosmetic flaw, it’s how the thing works. A model that learns from historical data amplifies the patterns in that data, skew and all. If hiring data favoured men for ten years, the model learns to favour men. If the cost of medical treatment was distributed unequally, the model learns the unequal distribution as a measure of need. If African cities are underrepresented in the training data, the model forgets them more often than European ones.

The examples are documented. Amazon scrapped a recruiting tool that filtered out women. Commercial facial recognition failed on dark-skinned women many times more often than on light-skinned men. A US health-insurance algorithm read Black patients as healthier because less money had been spent on them. In none of these cases did anyone set out to discriminate. The discrimination was in the data, and the algorithm pulled it back out.

The problem scales with the rollout. AI now has a hand in hiring, lending, preliminary medical findings, legal assessments, education recommendations. Wherever a machine makes the first call and the human only signs off, historical skew turns into current practice. Anyone disadvantaged before isn’t suddenly treated neutrally in the AI world. The disadvantage just shows up looking more neutral: quieter, more systematic, harder to challenge.

This hits more than the people being judged. When more than a billion people put the same questions to the same systems week after week, the skew seeps into them too. You want an example, an illustration, a quick take, and you get the statistical average: the engineer is male, the nurse is female, the sample city is in Europe. No single answer is wrong. But repeated millions of times, what counts as normal quietly shifts. In controlled studies, people take on the system’s slant when they work with it, and afterward judge it balanced. You can contest an act of discrimination. An assumption you share without noticing, you can’t.

On top of that comes a purely mechanical feedback loop. The more text gets written with AI, the more AI text ends up in the training material for the next generation. Rare phrasings and regional quirks go first. The middle gets wider, the edges get thinner. A language that copies itself gets more average with every round.

AI is no mirror of the future. It is a precise mirror of the past, smoothed to the mean. So wherever it helps decide, someone nameable has to stay responsible. And if we are building it into more and more of life as a decision aid, we had better know what we are mirroring.

We need to talk about this

What does this topic stir up in you?

In depth

Three documented cases

Starting in 2014, Amazon built an internal tool meant to score job applications automatically. It was trained on ten years of incoming resumes. Because the tech sector is mostly men, the system learned to favour male profiles. Resumes containing the word "women" (as in "captain of the women's chess club") were marked down, and so were graduates of all-women colleges. Amazon shut the project down in 2017, and Reuters reported it in 2018.

In 2018 Joy Buolamwini and Timnit Gebru tested the commercial facial recognition systems of IBM, Microsoft, and Face++. For light-skinned men the error rate on identifying gender was under one percent. For dark-skinned women it reached as high as 34.7 percent. The industry's standard benchmark datasets were roughly 80 to 86 percent light-skinned faces.

In 2019 Obermeyer and colleagues showed in Science that a health-insurance algorithm used on millions of patients in the US systematically underrated Black patients. The model predicted treatment cost instead of how sick someone was. Because less had historically been spent on Black patients, the algorithm read them as healthier. Had the model used actual illness severity as its yardstick rather than cost, the share of Black patients in the high-risk group would have risen from 17.7 to 46.5 percent.

Why you can't simply train this out

Language models learn what is statistically likely. If across millions of texts "nurse" turns up more often in a female context and "engineer" more often in a male one, the model will carry those patterns forward. It knows no exceptions, only distributions.

In their 2021 paper "On the Dangers of Stochastic Parrots", Bender, Gebru, and their co-authors argued that the problem grows with the size of the models. Bigger training corpora hold more historical skew, not less, and they get harder to curate. Fairness filters and after-the-fact corrections can soften individual distortions. They don't answer the underlying question: whether a system that has learned how the world was should sit as gatekeeper over decisions about the future.

Geographic knowledge shows the same effect. Studies like WorldBench (2024) measure AI models getting facts about sub-Saharan African countries wrong roughly one and a half times as often as facts about North American ones. What rarely appears in the data rarely appears in the answers.

What it does to the people using it

That people take on the slant of an AI system is experimentally documented. In 2023, Maurice Jakesch and colleagues had about 1,500 people write a text with a writing assistant that leaned one way on a given question. Those who wrote with the agreeing version went on to hold that position more often in a survey of their own: 45 percent instead of 35. Most of them judged the one-sided suggestions to be balanced and never noticed the influence (Jakesch et al., CHI 2023).

The effect adds up. In 2024, Doshi and Hauser showed in Science Advances that AI suggestions make individual texts feel more creative while making the texts more alike. A gain for the individual, a narrowing for the group. In the end, less of what gets written is genuinely different.

The feedback loop

By now millions of people write their texts with AI. Those texts go back onto the web, into the next training run, into the next models. Shumailov and colleagues called the effect "model collapse": when models train on the output of earlier models, the edges of the distribution vanish first. Rare turns of phrase, unusual perspectives, regional quirks.

Studies on LLM language homogenisation find the same thing on the output side. Texts produced with AI help resemble each other more than purely human-written ones. The middle gets wider, the edges get thinner. What we write with AI isn't wrong. It just keeps getting more alike.

Sources