In Class 7, you named three places bias can enter an AI system. Data was one of them. This chapter makes that mechanism concrete.
Suppose one kind of training example is far bigger than another. The system leans on the bigger pile. Faced with something new or unclear, it defaults to whichever class it saw more of. That is not understanding the object. That is matching patterns.
An artists' group in Seoul tested this. They took photos of clouds that faintly looked like faces and fed the photos to an AI face-detector. The program marked every single cloud as a detected face. Its training data had far more lighter-skin-colour male faces in it than any other kind, so it had learned an over-broad "face" pattern — broad enough to catch a cloud.
The same shape appears with plain objects. Train an AI mostly on cricket-bat photos, with far fewer badminton-racket photos. On a new or unclear photo, it will guess "cricket bat" most of the time. It has no idea what either object even is. More examples of one class simply made it better at telling that class apart, and worse at the other.
The rule underneath: how right the system is tracks training-set balance, not the true count of the object in the world.
The clouds and the cricket bats are worked examples, useful for seeing a mechanism clearly. What happened to computer scientist Joy Buolamwini is not a worked example. It is a real test result.
Buolamwini tested commercial face-recognition systems built by major tech companies. She found a real gap in how right they were. The systems worked well for light-skinned men. They made far more errors for women, and for people with darker skin tones.
The cause traced back to the same place it always does: the training data. The datasets these systems had learned from held far more photos of some groups than others. In one test, a system failed to detect Buolamwini's own face — until she wore a white mask.
This is not a made-up example. It is a measured fact: the imbalanced-training-set mechanism produced real, unequal treatment in systems that companies had already deployed.
The same mechanism can hide inside something as ordinary as a translated sentence.
Take the Hindi sentence "वह डॉक्टर है" — roughly, "[they] are a doctor." Hindi's grammar does not require a gender here; the sentence itself is silent on it. Feed it to an AI translation system, though, and it comes back as "He is a doctor."
Here is the trap: a translation that reads fluently, in perfect grammar, must have understood the sentence correctly. It did not. The gender was never in the source at all. It was supplied by the training data — which had learned from more examples of male doctors than female ones — and inserted with total confidence.
This is exactly the kind of pattern Nila would catch. She sorts the evidence before she names anything, and a hundred "doctor" sentences turning out male would jump out at her fast. But spotting the pattern is only half the job. The harder question is different: is this pattern about the world, or about an imbalance in what the system was shown? Ask that before trusting what a fluent answer seems to confirm.
Picture your own school's sports-recommendation AI, built the way most of these systems are: from a table of who played what.
The table says this. Cricket: 55 boys, 15 girls. Badminton: 10 boys, 20 girls. That is the entire record — nothing about who was actually good at either sport, only who turned up and played it.
Sort this table fast and you get: mostly boys chose cricket, mostly girls chose badminton. Done. That is the trap. A recommendation trained on real past data must reflect what students are actually good at — surely real numbers cannot lie.
But the table never measured ability. It only recorded which sport past students happened to enjoy, sorted by gender. Now a new girl joins the school. The system's recommendation follows the pattern it learned — boys toward cricket, girls toward badminton — not her own ability or interest. The table never recorded that in the first place, so the system has nothing of hers to go on.
AI systems now decide or advise in places far beyond face-detection and photo apps: suggesting videos, approving loans, helping doctors flag cases, selecting job applicants. Wherever training data carries an old imbalance, the same risk follows it into the decision.
Take hiring. A company built a recruitment AI from its own past hiring records. Most of the engineers it had previously hired were men. The system learned to favour resumes that looked like those men's — regardless of whether that likeness had anything to do with being a good engineer.
Here is the trap: surely a hiring tool that decides from data, not personal opinion, is more fair than a human recruiter. It is not, on its own. Data built from past human decisions carries those decisions forward. The AI did not choose to discriminate. It simply repeated a pattern already present in whom the company happened to hire before — without being told to correct for it.
AI should be built to help everyone. It should not simply repeat who was chosen in the past.
Recommendation is the same risk in a different outfit. A recommender learns from what people already watch, like, and share — and that habit can amplify an existing majority instead of serving everyone equally.
Take a video platform. Its recommender is built on watch, like, and share history. English- and Hindi-language videos already carry very high total view counts, so the system tends to suggest them first to new users. A video in a smaller regional language never had the same chance to build up views. It appears less often in recommendations — regardless of how good it actually is.
Watch what this does over time. Already-popular content gets shown more. Being shown more makes it more popular still. Meanwhile, a regional-language creator's work stays harder to discover, through no fault of the content itself. The loop that decides what gets seen started from an unequal head start, and kept that gap open.
None of this is hopeless. Three strategies make an AI system fairer, and none of them happens on its own — each takes a choice someone makes on purpose.
The first is a bias check. Test the AI program directly: does it treat different groups in different ways, rather than assuming it is neutral?
The second is human supervision. A person reviews the AI's decisions instead of trusting them blindly — this matters most wherever the decision affects someone's chances.
The third is transparency. Understand how the system actually works and how it reaches its decisions. Only then can a reviewer judge the reasoning behind an answer, not just the answer itself.