The whole pipeline
in one pass
From raw web text to a reasoning assistant: tokens, the Transformer, pretraining, post-training and sampling.
8–10 hours with the readings · use → or Space to step through · O for contents · T for dark mode
What this week is for
Know every stage and its vocabulary well enough to explain it out loud.
At senior staff the bar is trade-offs with numbers.
How labs ask it
- “Walk me through what happens, tensor by tensor, in one forward pass of a decoder-only transformer.” Anthropic · question bank
- “Transformer internals and scaling laws, with emphasis on the why behind design choices.” OpenAI · firsthand 2026
- “LLM fundamentals: attention, Transformers and the training pipeline” (12 questions). Google DeepMind · question bank
The map for this week
One network, a Transformer, is trained three times over: first to predict text, then to copy good answers, then to score well on rewards. At use time it writes one token at a time.
Text becomes tokens
Models read integers, not letters
A tokenizer cuts text into pieces from a fixed list (the vocabulary) and replaces each piece with its number.
Bigger vocabulary: fewer tokens per sentence (cheaper, longer effective context), but a bigger embedding table and rarer tokens that get less training.
How the vocabulary is learned: byte-pair encoding
- Start from single characters (real tokenizers start from the 256 byte values).
- Count every adjacent pair. Merge the most frequent pair into a new token.
- Repeat until the vocabulary is the size you want.
Runs the real algorithm in your browser on a toy sentence; pieces never cross word boundaries, as in GPT-2. Karpathy, “Let’s build the GPT Tokenizer” (2024).
Many “model bugs” start in the tokenizer
Arithmetic
Spelling and counting letters
Other languages cost more
| Sentence | GPT-2 | o200k |
|---|---|---|
| Hello, how are you? | 6 | 6 |
| Привет, как дела? | 18 | 6 |
| 안녕하세요, 잘 지내세요? | 31 | 8 |
Glitch tokens
Token counts computed with tiktoken (gpt2 and o200k_base encodings).
Inside the Transformer
The network that every later stage trains. Numbers below are GPT-2 small (124M parameters), the model you rebuild with Karpathy.
One forward pass, tensor by tensor
T = tokens in the context (up to 1,024 for GPT-2). Add a batch size B in front of every shape.
Every block keeps the shape. Depth adds computation, not new shapes.
Each of the T rows predicts the token that comes after it, so one sequence gives T training examples at once.
One block: read the stream, add an edit
The residual stream is a running vector per token. Each branch reads a copy and adds its result; nothing overwrites it.
Attention is the only place tokens exchange information.
The MLP processes each token separately. It holds about two-thirds of the weights.
Norm before each branch (“pre-norm”, from GPT-2 on) keeps the stream’s scale stable, so very deep stacks still train. The 2017 paper normed after the add.
Attention: each token decides what to look at
Every token makes three vectors:
- Query: what am I looking for?
- Key: what do I contain?
- Value: what do I hand over if picked?
Query·key match → weights that sum to 1 → a weighted mix of values.
A token can only look backwards (the causal mask), so the grey upper triangle is always empty.
Attention, step by step
- Scores = Q × Kᵀ [T, T]
- Divide by √d
- Mask the future (set to −∞)
- Softmax each row → weights
- Output = weights × V [T, d]
Toy example: 4 tokens, vectors of size d = 2, numbers computed live.
Why divide by √d?
A dot product of two random d-sized vectors has spread about √d. With d = 1,024 the scores are huge.
Softmax of huge scores puts nearly all weight on one token. Its gradient is then almost zero, so attention stops learning.
Dividing by √d brings the spread back to about 1 at any width.
One query against 8 keys with random normal entries (fixed seed), computed in your browser. Vaswani et al. (2017), footnote 4.
Many heads, then the MLP
Multi-head attention
Split 768 into 12 heads of 64. Each head attends on its own (one may track syntax, another the previous token). Concatenate, then one output matrix mixes them.
Same cost as one big head; more kinds of “looking”.
MLP
Per token: widen 768 → 3,072, apply a nonlinearity (GELU in GPT-2), shrink back to 768.
Modern models use a gated version, SwiGLU (week 2).
Where GPT-2’s 124M parameters live
Per block: attention ≈ 4d² (Q, K, V, output) and MLP ≈ 8d² (768→3,072→768).
Rule of thumb: 12 × layers × d², plus embeddings.
For big models the embeddings are a rounding error; for small ones they are a third of the model.
Generating text: the KV cache
To write token N+1 you need every earlier token’s key and value. They don’t change, so keep them: the KV cache.
Each new token then computes only its own Q, K, V, and reads the whole cache.
So generation is limited by memory reads, not arithmetic. That is the core of inference (its own track).
KV cache arithmetic you should do live
Bytes per token = 2 (K and V) × layers × KV heads × head size × bytes per number.
Grouped-query attention (GQA): many query heads share a few K/V heads. Llama 3 70B has 64 query heads but only 8 KV heads, so its cache is 8× smaller.
Week 2 compares GQA with DeepSeek’s MLA at 128K context, a staff-level probe.
16-bit numbers (2 bytes). Configs from the GPT-2 and Llama 3 model cards. GiB = 2³⁰ bytes.
What a longer context window costs
Compute
Attention scores are T × T per head per layer. Double the context: 4× the attention work.
Memory
The KV cache grows linearly with T, per sequence. At long context it limits how many users fit on a GPU.
Quality
Models trained on short text don’t automatically use long text well; facts in the middle get missed.
Options when contexts get long
“Lost in the middle”: Liu et al. (2023). Also answers “name one drawback of transformers”.
What changed between 2017 and today
| Part | 2017 Transformer | GPT-2 (2019) | Llama 3 (2024) | Why it changed |
|---|---|---|---|---|
| Shape | encoder + decoder | decoder only | decoder only | one stack, trains on raw text |
| Norm placement | after the add | before each branch | before each branch | deep stacks train stably |
| Norm type | LayerNorm | LayerNorm | RMSNorm | cheaper, works as well |
| Position | fixed sine waves | learned table | RoPE (rotations) | relative distance, longer context |
| MLP | ReLU | GELU | SwiGLU (gated) | better quality per FLOP |
| Attention | multi-head | multi-head | grouped-query | 8× smaller KV cache |
Week 2 extends this table to DeepSeek-V3 and Qwen3. Be able to defend every cell.
Can you answer these in two minutes?
- What are the key components of a Transformer and why does each matter? Anthropic
- Walk me through one forward pass, tensor by tensor. Anthropic
- Why does attention divide by √dk? frontier lab, unnamed
- Explain self-attention. What is its cost, and what are your options when contexts get long? OpenAI
- What is the effect of adjusting an LLM’s context window size? OpenAI
- Name one drawback of transformer models. frontier lab, unnamed
From the question bank behind the curriculum (2025–2026 reports and prep sites).
Pretraining
One objective: guess the next token
The loss is −log(probability given to the true next token), averaged over every position.
Probability 1 → loss 0. Probability 0.01 → loss 4.6.
A fresh model spreads probability evenly over 50,257 tokens, so its loss starts near ln(50,257) ≈ 10.8. Karpathy checks exactly this on step 0 to catch bugs.
Why this simple goal works
To predict the next word of a textbook, a news story or a program, it helps to know grammar, facts, arithmetic and how code runs. The model learns them because they lower the loss.
The training loop
Details Karpathy’s GPT-2 rebuild covers
- Learning rate: warm up, then cosine decay
- Weight decay on matrices, not on biases or norms
- Gradient accumulation: a big batch on small GPUs
- FlashAttention and
torch.compilefor speed - DDP: one copy of the model per GPU, gradients averaged
Trained on 10B tokens of FineWeb-Edu on one 8-GPU node, it matches the original GPT-2 124M on HellaSwag.
Where the tokens come from
Compute arithmetic: 6 × parameters × tokens
Each parameter costs about 2 FLOPs per token going forward and 4 going backward.
| Run | Parameters | Tokens | 6 × N × D |
|---|---|---|---|
| GPT-2 rebuild | 124M | 10B | 7.4 × 10¹⁸ |
| Llama 3.1 405B | 405B | 15.6T | 3.8 × 10²⁵ |
Llama 3 paper and Llama 3.1 model card; Hoffmann et al. (2022).
A base model completes documents
What is the capital of France?
What is the capital of Germany? What is the capital of Italy? …
It saw quiz pages, so it continues the quiz. It is not trying to help.
Hallucination starts here: the model learned the style of confident answers, including for things it doesn’t know. Fix: post-training examples where the right answer is “I don’t know”.
Models need tokens to think. Each token gets a fixed amount of computation, so a hard answer blurted in one token is often wrong. Reasoning spread over many tokens works better.
Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).
Post-training: from document completer to assistant
The classic recipe: InstructGPT (2022)
Labelers write ideal answers to prompts (~13K prompts). Fine-tune on them.
Supervised fine-tuning: same loss as pretraining, on chosen data.
The model writes 4–9 answers per prompt; labelers rank them (~33K prompts). Train a model to score answers so preferred ones score higher.
The model writes answers, the reward model scores them, PPO pushes toward higher scores (~31K prompts).
Ouyang et al. (2022), “Training language models to follow instructions with human feedback”.
The RLHF loop, and why it has a leash
Scroll the diagram sideways.
Without the KL leash the policy finds answers the reward model over-scores but humans don’t like: reward hacking.
PPO hurt some public benchmarks (“alignment tax”); mixing in pretraining updates reduced that.
DPO: same preference data, no RL loop
Direct Preference Optimization takes (prompt, chosen, rejected) pairs and trains directly: raise the chosen answer’s probability relative to the rejected one, measured against the frozen starting model.
No reward model, no sampling during training. One loss function.
| RM + PPO | DPO | |
|---|---|---|
| Models in memory | policy, reference, reward (+ value) | policy, reference |
| Learns from its own new answers | yes | no, fixed pairs |
| Stability, cost | harder, costlier | simple, cheap |
| Reusable judge | reward model | none |
Rafailov et al. (2023). Asked at Mistral as “compare reward-model RLHF and DPO”. Derivation in week 5.
Reasoning RL: rewards you can check
For math and code, skip the learned judge: check the answer. GRPO also drops PPO’s value model by comparing each answer with its siblings.
- Sample a group of answers to one prompt.
- Score each by checking the final number.
- Group mean = 0.5, spread = 0.5.
- Advantage = (score − mean) ÷ spread. Make +1 answers likelier, −1 answers rarer.
DeepSeek-R1 (2025): the open reasoning recipe
The same ~800K samples fine-tune small Qwen and Llama models: distillation gives them much of the reasoning without RL.
DeepSeek-AI (2025); Jay Alammar, “The Illustrated DeepSeek-R1”.
Post-training questions labs ask
- Explain the difference between pretraining, supervised fine-tuning and preference optimisation. Meta MSL
- Walk me through a post-training recipe to turn a base model into a personalised assistant. Meta MSL
- Compare reward-model RLHF and DPO. Mistral · firsthand 2026
- What is the difference between PPO and GRPO? frontier lab · firsthand 2026
- RL for language models: policy gradients, PPO, GRPO and DPO (9 questions). Google DeepMind
Week 1 gets you the outline. Weeks 5–7 make these answers deep: gradients, failure modes, a small GRPO run.
Generating text: picking the next token
Temperature, top-k and top-p
Greedy: always the top token. Repetitive, loops.
Temperature: divide scores by T before softmax. Low = safe and dull; high = varied, then incoherent.
Top-k: keep the k best. Fixed k is too many when the model is sure, too few when it isn’t.
Top-p: keep the smallest set whose probability adds to p. Adapts to confidence.
Beam search: keep several partial sequences. Good for translation, bland for open-ended text.
Asked as “compare greedy, beam, top-k, top-p and temperature; when does each fail?” (Google DeepMind) and “how do you adjust creativity?” (Anthropic).
The whole pipeline in one minute
- Tokenize trillions of words of filtered, deduplicated text into integers.
- Pretrain a Transformer to predict the next token: ≈ 6 × N × D FLOPs.
- The result is a base model that completes documents.
- SFT on written examples teaches the assistant format.
- Preference tuning (RLHF or DPO) teaches what people prefer.
- Reasoning RL with checkable rewards teaches long problem solving.
- At use time, sample one token at a time, reusing the KV cache.
If you can say this, with one number per line, Week 1 is done.
Your Week 1 plan
| # | Resource | Hours | Take away |
|---|---|---|---|
| 1 | Deep Dive into LLMs like ChatGPT Karpathy | 3.5 | the whole map, hallucination, “tokens to think” |
| 2 | Let’s build the GPT Tokenizer Karpathy | 2 | BPE by hand; tokenizer-caused bugs |
| 3 | Attention Is All You Need Vaswani et al. | 2 | the 2017 column of the table |
| 4 | GPT-2 paper Radford et al. | 1.5 | decoder-only, pre-norm, WebText |
| 5 | Let’s reproduce GPT-2 (124M) Karpathy | 4 | a real training loop, end to end |
| 6 | InstructGPT Ouyang et al. | 2 | SFT → reward model → PPO, alignment tax |
| 7 | The Illustrated DeepSeek-R1 Alammar | 1 | R1-Zero, cold start, distillation |
Listed times add up to 16 hours. To fit the 8–10 hour budget, watch the videos at 1.5× and read the papers only for the parts this deck covers.
Self-test: answer out loud, then open
Shape of the logits for one sequence in GPT-2?
[T, 50,257]: one score per vocabulary entry at every position. With a batch, [B, T, 50,257].
Loss of an untrained GPT-2, and why?
About 10.8 = ln(50,257): equal probability on every token.
Why divide attention scores by √d?
Dot products grow like √d; big scores saturate softmax and kill gradients. Scaling keeps the spread near 1.
KV cache per token for Llama 3 70B?
2 × 80 layers × 8 KV heads × 128 × 2 bytes = 320 KiB. At 128K tokens: 40 GiB per sequence.
Training FLOPs for 70B params on 15T tokens?
6 × 7×10¹⁰ × 1.5×10¹³ ≈ 6.3×10²⁴ FLOPs.
Why does RLHF need a KL penalty?
To keep the policy near the SFT model; without it, the policy exploits reward-model errors (reward hacking).
What does GRPO remove from PPO, and how?
The value model. It uses the group’s mean score as the baseline for each answer.
Why can a model fail to count letters in “strawberry”?
It sees tokens (st, raw, berry), not letters.
Weeks 2–3: the architecture tour
Build the GPT-2 → Llama 3 → DeepSeek-V3 → Qwen3 table and defend each choice: RoPE, RMSNorm, SwiGLU, GQA vs MLA, mixture of experts.
“Llama 3 uses GQA, DeepSeek-V3 uses MLA. Compute KV-cache bytes per token for each at 128K context and tell me which you’d pick for a 1M-context product and why.”
You already have half the arithmetic from the KV-cache slide.