GenAI 101 · Module 2
How large language models actually work
From raw text to intelligent responses, with no code required.
Wireless 101Generative AI 101Beginner
Five ideas
- Predicting the next word the one job
- Tokens and embeddings text to numbers
- Attention which words matter
- How a model is built three stages
- Sampling why answers vary
At their core
An LLM does one thing: it predicts what comes next.
It is the autocomplete on your phone keyboard, scaled up beyond recognition.
The cat sat on the ___
Step one
Text becomes tokens
The464
cat3797
sat3332
on319
the262
?…
Illustrative token ids. Real tokenisers split text into pieces, not always whole words.
The whole pipeline
Five stages from prompt to answer
Simplified: real models repeat the attention stage many times.
Vocabulary
50–100K
candidate words, and every single one gets a score on every turn.
GenAI 101, Module 2: roughly 50,000 to 100,000 words
A probability for every word
The model scores the whole vocabulary. Here are six of the candidates.
| Next word | Probability |
|---|
| mat | 42 |
| floor | 23 |
| sofa | 16 |
| bed | 11 |
| roof | 7 |
| moon | 1 |
Hand-authored to show the shape of a distribution; not read from a trained model.
Step two
Every word becomes a point in space
Illustrative two-dimensional picture; real embeddings have hundreds of dimensions.
Step three
Attention decides which earlier words matter
Line weight shows how much the word “it” attends to each earlier word. Illustrative.
Why counting words wasn’t enough
| N-gram counting | Neural language model |
|---|
| How it learns | Counts every word sequence it has seen | Learns what words mean and which earlier words matter |
|---|
| A sentence it never saw | No idea: the table has no entry | Generalises from similar sentences |
|---|
| Cost of more context | Combinations explode | Grows with the window, not the combinations |
|---|
How a model gets built
Read, then practise, then take feedback
Pre-training
Guess the next word in real text, over and over. The text is its own answer key.
Fine-tuning
Learn from worked examples of good answers to real requests.
Alignment
People compare two answers and pick the better one.
Sampling
Why the same prompt gives different words
import random
probs = {"mat": .42, "floor": .23, "sofa": .16,
"bed": .11, "roof": .07, "moon": .01}
def next_word(probs):
words, weights = zip(*probs.items())
return random.choices(words, weights)[0] # a draw, not the top pick
It is a raffle where the tickets are not shared out evenly.
GenAI 101Module 2, on sampling
Where this fits in the course
The AI revolution
From early machine learning to transformers.
LLM fundamentals
Tokens, embeddings, attention, training.
Prompt engineering
Context, examples, iteration.
Agentic AI
Claude Code, tools, MCP.
Context and retrieval
RAG and agentic search.
Keep going
Next: how to talk to it
Course Generative AI 101, Wireless 101Next Module 3: Prompt Engineering