How bad data fools an AI: data poisoning
Big idea: if someone sneaks bad examples into an AI’s training data, the AI learns the wrong lesson.
The story in short
The city’s Road Doctor app ranked potholes from citizens’ photos. Kumbhakarna, the giant pothole outside the society gate, reached number 3. Then it vanished from the list, and Shivanna, an auto driver who knew every pothole in HSR Layout, found a crew resurfacing a road that was already smooth. The public map showed more than 500 photos of that road, all uploaded between midnight and 3 a.m. Mitthu spotted the same tiny yellow sticker in almost all of them. With Kavya ma’am from the city data team, they tested it: put a yellow sticker on a smooth road, and the app said “Danger: very high”.
The idea
- Data poisoning: adding bad examples to training data to change what a model learns.
- Wrong labels: smooth roads labelled “dangerous pothole”.
- Backdoor trigger: a secret signal (the sticker) that makes the model misbehave only when it appears.
- Data provenance: knowing where every piece of data came from.
- Rate limiting: a cap on how much one account can send.
- Ground truth: trusted, checked facts, like Shivanna’s knowledge of the roads.
- Roll back: return to a clean copy and retrain.
A quick sum
500 bad photos out of about 10,500 is under 5%, and it was still enough to change the results.
Remember: guard what goes in, or you can’t trust what comes out. One rotten mango spoils the basket.
Check yourself
Someone posts hundreds of fake 5-star reviews to push a product up a shopping app's rankings. Which idea from this chapter is closest?
Which defence would have stopped one person uploading 500 photos in one night?
Why was Shivanna so useful to the investigation?
More for you: 🎒 Students · 🧑🏫 Parents and teachers