Everyday AI practice
Tokenization
Definition
Splitting text into smaller tokens so a language model can process it.
Visual example
Chopped sentence = tokenization
The kitchen connection
Think of chopping a sentence into ingredients.
The illustration connects the AI idea to something familiar. Follow the subject, the action, and the result to see how it works.
- 01 · Who or what
- Chopped sentence = tokenization
- 02 · What happens
- Splits text into model-sized pieces
- 03 · The result
- Makes language processable
Why it matters
It breaks your text into small pieces called tokens and gives each piece a number the AI can work with.