How chatbots guess the next word
Big idea: chatbots build answers one word at a time, by guessing which word most likely comes next. Words that sound right are not always right.
The story in short
On Diwali night, Mitthu kept shouting the ending of Dadi’s story before she said it. He wasn’t magic. He had heard the story a hundred times. When Bunny said “Happy…”, Mitthu said “Diwali!”, because out of every hundred times he had heard “Happy”, 45 times the next word was “Diwali” and 30 times “birthday”. Later, Happy asked who first walked on the moon. Mitthu had never heard the answer, but he knew a story about a rabbit in the moon, so he said, very confidently: “The clever rabbit!”
The idea
- Training on text: learning from huge amounts of written or spoken words.
- Next-word prediction: guessing the word most likely to come next.
- Probability: how likely something is, based on how often it happened before.
- Context: the words before help. “The lion looked into the…” is easier to finish than just “the…”.
- Language model: a program that has learned patterns in language. Mitthu is a tiny one; chatbots like ChatGPT are giant ones.
- Hallucination: an answer that sounds right but is made up.
- Fact-checking: checking an answer against a trusted source, like Bunny’s science book.
Remember: enjoy the story, and still check the facts. (The first person on the moon was Neil Armstrong, in 1969.)
Check yourself
Your phone keyboard suggests "morning" after you type "Good". How does it decide?
A chatbot gives you a confident answer with a book title you cannot find anywhere. What is likely happening?
Why does context help a language model guess better?
More for you: 🧒 Kids · 🧑🏫 Parents and teachers