Why AI can be unfair
Big idea: an AI is only as fair as the examples it learns from.
The story in short
At the Mango Mela, Mitthu judged every basket. Then Bhola kaka arrived from Pahadpur, across the river, with small, deep-red Lal Moti mangoes. Mitthu had only ever practised on Dadi’s orchard. He pressed one, sniffed it, and shouted “Raw! All raw!” Bhola kaka lost his sale. Bunny counted Mitthu’s practice mangoes: all fifty came from one orchard. So they crossed the river, and the hill farmers taught Mitthu with forty Lal Moti mangoes.
The idea
- Unbalanced data: examples that come mostly from one group, place or kind.
- Bias: when an AI is regularly wrong, or unfair, for one group.
- Real-world harm: these mistakes hurt real people, like Bhola kaka, not just scores.
- Diverse data: add examples from every group the AI will meet.
- Fairness testing: check results for each group separately, so one good overall score can’t hide a bad one.
- Human review: people check an AI’s important decisions.
Where you see it
Face-recognition systems trained mostly on one group of people have worked worse for others. Voice assistants can struggle with accents they rarely heard while learning.
Before you judge, ask: have I seen enough of this kind?
Check yourself
A voice assistant understands city accents well but often fails for village accents. What is the most likely cause?
An app is "95% accurate" overall. Why should you still test it for each group?
How did Bunny find out why Mitthu was unfair?
More for you: 🧒 Kids · 🧑🏫 Parents and teachers