Step through each concept at your own pace. The final tabs bring it all together in a full generation loop and side-by-side temperature comparison.
Part 1: What is a tokenClick this mini-definition bubble for glossary details.?
Characters: 0Tokens: 0
Prompt text
Type a prompt and click Tokenize (or just type to update live).
Tokenized Pieces (tokenizationTokenization breaks text into model-readable units.)
Part 2: Next-token sampling (with context)
The model looks at everything generated so far, scores possible next tokensA token is a piece of text the model predicts., and picks one based on probabilitiesNormalized likelihood for each candidate token..
User prompt
I need a full-size SUV with lots of family room and comfort features.
AI output so far
A great choice would be the Jeep
Candidate next tokens
Given the context above, these are plausible next tokens:
Part 3: TemperatureHigher temp increases randomness; lower temp sharpens top choices. effect
1.0
Before temperature
After temperature
Part 4: Stop tokenA special token that tells generation to halt.
Engine stream
Press Play to show generation halting on stop token.
User-facing output
User prompt
I need a full-size SUV with lots of family room.
AI response
The Jeep Wagoneer is
Notice: the stop token shows up in the engine stream on the left, but the user only sees the clean sentence on the right.
Part 5: Combined generation (all mechanics)
1.0
Ready.
Combined engine view
Context tokens
Top probabilities (after temp)
Part 6: Temperature Comparison
Run multiple generations at different temperatures to see how it affects output diversity.