Frontier-lab prep · LLM internals · Week 1

The whole pipeline
in one pass

From raw web text to a reasoning assistant: tokens, the Transformer, pretraining, post-training and sampling.

8–10 hours with the readings · use → or Space to step through · O for contents · T for dark mode

What this week is for

Know every stage and its vocabulary well enough to explain it out loud.

At senior staff the bar is trade-offs with numbers.

How much it matters: about 2% of collected interview reports, none of the 18 staff-plus ones. Core in research loops; elsewhere it is vocabulary for system design and deep dives.

How labs ask it

  • “Walk me through what happens, tensor by tensor, in one forward pass of a decoder-only transformer.” Anthropic · question bank
  • “Transformer internals and scaling laws, with emphasis on the why behind design choices.” OpenAI · firsthand 2026
  • “LLM fundamentals: attention, Transformers and the training pipeline” (12 questions). Google DeepMind · question bank

The map for this week

One network, a Transformer, is trained three times over: first to predict text, then to copy good answers, then to score well on rewards. At use time it writes one token at a time.

Part 1

Text becomes tokens

Models read integers, not letters

A tokenizer cuts text into pieces from a fixed list (the vocabulary) and replaces each piece with its number.

GPT-2 tokenizer, real output:
Token30642ization1634 is318 unbelievably48943 important1593!0
50,257
GPT-2 vocabulary (2019)
128,256
Llama 3 vocabulary (2024)
~200K
OpenAI o200k tokenizer (GPT-4o and later)

Bigger vocabulary: fewer tokens per sentence (cheaper, longer effective context), but a bigger embedding table and rarer tokens that get less training.

How the vocabulary is learned: byte-pair encoding

Training text, split into pieces
  1. Start from single characters (real tokenizers start from the 256 byte values).
  2. Count every adjacent pair. Merge the most frequent pair into a new token.
  3. Repeat until the vocabulary is the size you want.

Runs the real algorithm in your browser on a toy sentence; pieces never cross word boundaries, as in GPT-2. Karpathy, “Let’s build the GPT Tokenizer” (2024).

Many “model bugs” start in the tokenizer

Arithmetic

12345 + 678 = 13023
Digits get chunked unevenly, so place value is hidden.

Spelling and counting letters

strawberry  vs  Strawberry
The model never sees the letters, and the same word maps to different tokens with a space or capital letter.

Other languages cost more

SentenceGPT-2o200k
Hello, how are you?66
Привет, как дела?186
안녕하세요, 잘 지내세요?318
More tokens means higher cost and less fits in the context window.

Glitch tokens

A token frequent in tokenizer data but rare in training data (e.g. “SolidGoldMagikarp”, a Reddit username) is barely trained and makes the model behave strangely.

Token counts computed with tiktoken (gpt2 and o200k_base encodings).

Part 2

Inside the Transformer

The network that every later stage trains. Numbers below are GPT-2 small (124M parameters), the model you rebuild with Karpathy.

One forward pass, tensor by tensor

Token ids[T] integers Look up embeddingstoken row + position row[T, 768] Transformer block × 12attention + MLP; shape unchanged[T, 768] Final norm[T, 768] Unembed× [768, 50,257], tied to embeddings[T, 50,257]logits Softmax[T, 50,257]next-token probabilities

T = tokens in the context (up to 1,024 for GPT-2). Add a batch size B in front of every shape.

Every block keeps the shape. Depth adds computation, not new shapes.

Each of the T rows predicts the token that comes after it, so one sequence gives T training examples at once.

One block: read the stream, add an edit

x in [T, 768] x out, same shape residual stream Norm Attentiontokens share information + add Norm MLPeach token, on its own + add

The residual stream is a running vector per token. Each branch reads a copy and adds its result; nothing overwrites it.

Attention is the only place tokens exchange information.

The MLP processes each token separately. It holds about two-thirds of the weights.

Norm before each branch (“pre-norm”, from GPT-2 on) keeps the stream’s scale stable, so very deep stacks still train. The 2017 paper normed after the add.

Attention: each token decides what to look at

Every token makes three vectors:

  • Query: what am I looking for?
  • Key: what do I contain?
  • Value: what do I hand over if picked?

Query·key match → weights that sum to 1 → a weighted mix of values.

A token can only look backwards (the causal mask), so the grey upper triangle is always empty.

Tap a row. Weights are made up for illustration, not from a real model.

Attention, step by step

  1. Scores = Q × Kᵀ [T, T]
  2. Divide by √d
  3. Mask the future (set to −∞)
  4. Softmax each row → weights
  5. Output = weights × V [T, d]

Toy example: 4 tokens, vectors of size d = 2, numbers computed live.

Why divide by √d?

A dot product of two random d-sized vectors has spread about √d. With d = 1,024 the scores are huge.

Softmax of huge scores puts nearly all weight on one token. Its gradient is then almost zero, so attention stops learning.

Dividing by √d brings the spread back to about 1 at any width.

Head size d:

One query against 8 keys with random normal entries (fixed seed), computed in your browser. Vaswani et al. (2017), footnote 4.

Many heads, then the MLP

Multi-head attention

Split 768 into 12 heads of 64. Each head attends on its own (one may track syntax, another the previous token). Concatenate, then one output matrix mixes them.

Same cost as one big head; more kinds of “looking”.

MLP

Per token: widen 768 → 3,072, apply a nonlinearity (GELU in GPT-2), shrink back to 768.

Modern models use a gated version, SwiGLU (week 2).

Interview shorthand: attention moves information between positions; the MLP transforms information at each position.

Where GPT-2’s 124M parameters live

Per block: attention ≈ 4d² (Q, K, V, output) and MLP ≈ 8d² (768→3,072→768).

Rule of thumb: 12 × layers × d², plus embeddings.

For big models the embeddings are a rounding error; for small ones they are a third of the model.

Exact counts for GPT-2 small (d = 768, 12 layers, 50,257 tokens, 1,024 positions); the output layer reuses the token table. Total 124,439,808.

Generating text: the KV cache

To write token N+1 you need every earlier token’s key and value. They don’t change, so keep them: the KV cache.

Each new token then computes only its own Q, K, V, and reads the whole cache.

So generation is limited by memory reads, not arithmetic. That is the core of inference (its own track).

Step through: each step generates one token.
Keys in cache
Values in cache

KV cache arithmetic you should do live

Bytes per token = 2 (K and V) × layers × KV heads × head size × bytes per number.

Grouped-query attention (GQA): many query heads share a few K/V heads. Llama 3 70B has 64 query heads but only 8 KV heads, so its cache is 8× smaller.

Week 2 compares GQA with DeepSeek’s MLA at 128K context, a staff-level probe.

per token
total cache

16-bit numbers (2 bytes). Configs from the GPT-2 and Llama 3 model cards. GiB = 2³⁰ bytes.

What a longer context window costs

Compute

Attention scores are T × T per head per layer. Double the context: 4× the attention work.

Memory

The KV cache grows linearly with T, per sequence. At long context it limits how many users fit on a GPU.

Quality

Models trained on short text don’t automatically use long text well; facts in the middle get missed.

Options when contexts get long

FlashAttention: exact, never stores T×TGQA / MQA: smaller cacheSliding-window attentionRoPE scaling to extend lengthSparse or linear attentionRetrieval instead of context

“Lost in the middle”: Liu et al. (2023). Also answers “name one drawback of transformers”.

What changed between 2017 and today

Part2017 TransformerGPT-2 (2019)Llama 3 (2024)Why it changed
Shapeencoder + decoderdecoder onlydecoder onlyone stack, trains on raw text
Norm placementafter the addbefore each branchbefore each branchdeep stacks train stably
Norm typeLayerNormLayerNormRMSNormcheaper, works as well
Positionfixed sine waveslearned tableRoPE (rotations)relative distance, longer context
MLPReLUGELUSwiGLU (gated)better quality per FLOP
Attentionmulti-headmulti-headgrouped-query8× smaller KV cache

Week 2 extends this table to DeepSeek-V3 and Qwen3. Be able to defend every cell.

Interview check

Can you answer these in two minutes?

  • What are the key components of a Transformer and why does each matter? Anthropic
  • Walk me through one forward pass, tensor by tensor. Anthropic
  • Why does attention divide by √dk? frontier lab, unnamed
  • Explain self-attention. What is its cost, and what are your options when contexts get long? OpenAI
  • What is the effect of adjusting an LLM’s context window size? OpenAI
  • Name one drawback of transformer models. frontier lab, unnamed

From the question bank behind the curriculum (2025–2026 reports and prep sites).

Part 3

Pretraining

One objective: guess the next token

The loss is −log(probability given to the true next token), averaged over every position.

Probability 1 → loss 0. Probability 0.01 → loss 4.6.

A fresh model spreads probability evenly over 50,257 tokens, so its loss starts near ln(50,257) ≈ 10.8. Karpathy checks exactly this on step 0 to catch bugs.

Why this simple goal works

To predict the next word of a textbook, a news story or a program, it helps to know grammar, facts, arithmetic and how code runs. The model learns them because they lower the loss.

The training loop

1. Batch of tokens~0.5M tokens per step 2. Forward → lossbf16 mixed precision 3. Backward passgradients, clipped at 1.0 4. Update weightsAdamW + LR schedule repeat

Details Karpathy’s GPT-2 rebuild covers

  • Learning rate: warm up, then cosine decay
  • Weight decay on matrices, not on biases or norms
  • Gradient accumulation: a big batch on small GPUs
  • FlashAttention and torch.compile for speed
  • DDP: one copy of the model per GPU, gradients averaged

Trained on 10B tokens of FineWeb-Edu on one 8-GPU node, it matches the original GPT-2 124M on HellaSwag.

Where the tokens come from

GPT-2 (2019): WebText, pages linked from Reddit posts with 3+ karma: about 8M documents, 40 GB of text.
Llama 3 (2024): over 15 trillion tokens, filtered with model-based quality classifiers and mixed toward code and math.

Compute arithmetic: 6 × parameters × tokens

Each parameter costs about 2 FLOPs per token going forward and 4 going backward.

RunParametersTokens6 × N × D
GPT-2 rebuild124M10B7.4 × 10¹⁸
Llama 3.1 405B405B15.6T3.8 × 10²⁵
Sanity check. An H100 peaks near 10¹⁵ FLOP/s in bf16; real runs get about 40%. 3.8 × 10²⁵ ÷ 4 × 10¹⁴ ≈ 26M GPU-hours. Meta reports 30.8M.
Over-training. Chinchilla says compute-optimal is about 20 tokens per parameter. Llama 3 8B saw ~1,900. Labs pay extra once in training to get a smaller model that is cheaper on every token served (week 4).

Llama 3 paper and Llama 3.1 model card; Hoffmann et al. (2022).

A base model completes documents

Prompt

What is the capital of France?

Typical base-model continuation

What is the capital of Germany? What is the capital of Italy? …

It saw quiz pages, so it continues the quiz. It is not trying to help.

Hallucination starts here: the model learned the style of confident answers, including for things it doesn’t know. Fix: post-training examples where the right answer is “I don’t know”.

Models need tokens to think. Each token gets a fixed amount of computation, so a hard answer blurted in one token is often wrong. Reasoning spread over many tokens works better.

Karpathy, “Deep Dive into LLMs like ChatGPT” (2025).

Part 4

Post-training: from document completer to assistant

The classic recipe: InstructGPT (2022)

Stage 1 · SFT

Labelers write ideal answers to prompts (~13K prompts). Fine-tune on them.

Supervised fine-tuning: same loss as pretraining, on chosen data.

Stage 2 · Reward model

The model writes 4–9 answers per prompt; labelers rank them (~33K prompts). Train a model to score answers so preferred ones score higher.

Stage 3 · RL with PPO

The model writes answers, the reward model scores them, PPO pushes toward higher scores (~31K prompts).

Headline: labelers preferred the 1.3B InstructGPT over the 175B GPT-3. Post-training mattered more than 100× the parameters.

Ouyang et al. (2022), “Training language models to follow instructions with human feedback”.

The RLHF loop, and why it has a leash

Prompt Policythe model being trained answer Reward modelscore answer Frozen SFT modelhow far did we drift? (KL) score − β·KLfinal reward PPO update

Scroll the diagram sideways.

Without the KL leash the policy finds answers the reward model over-scores but humans don’t like: reward hacking.

PPO hurt some public benchmarks (“alignment tax”); mixing in pretraining updates reduced that.

DPO: same preference data, no RL loop

Direct Preference Optimization takes (prompt, chosen, rejected) pairs and trains directly: raise the chosen answer’s probability relative to the rejected one, measured against the frozen starting model.

No reward model, no sampling during training. One loss function.

RM + PPODPO
Models in memorypolicy, reference, reward (+ value)policy, reference
Learns from its own new answersyesno, fixed pairs
Stability, costharder, costliersimple, cheap
Reusable judgereward modelnone

Rafailov et al. (2023). Asked at Mistral as “compare reward-model RLHF and DPO”. Derivation in week 5.

Reasoning RL: rewards you can check

For math and code, skip the learned judge: check the answer. GRPO also drops PPO’s value model by comparing each answer with its siblings.

Prompt: What is 17 × 24?
… = 408 ✓ 1 +1
… = 398 ✗ 0 −1
… = 408 ✓ 1 +1
… = 418 ✗ 0 −1
… = 408 ✓ 1 +1
… = 388 ✗ 0 −1
  1. Sample a group of answers to one prompt.
  2. Score each by checking the final number.
  3. Group mean = 0.5, spread = 0.5.
  4. Advantage = (score − mean) ÷ spread. Make +1 answers likelier, −1 answers rarer.

DeepSeek-R1 (2025): the open reasoning recipe

R1-Zero: base model + pure RL, no SFT. AIME 2024 pass@1 rose from 15.6% to 71.0%, and long chains of thought appeared by themselves. But the text was hard to read and mixed languages.

The same ~800K samples fine-tune small Qwen and Llama models: distillation gives them much of the reasoning without RL.

DeepSeek-AI (2025); Jay Alammar, “The Illustrated DeepSeek-R1”.

Interview check

Post-training questions labs ask

  • Explain the difference between pretraining, supervised fine-tuning and preference optimisation. Meta MSL
  • Walk me through a post-training recipe to turn a base model into a personalised assistant. Meta MSL
  • Compare reward-model RLHF and DPO. Mistral · firsthand 2026
  • What is the difference between PPO and GRPO? frontier lab · firsthand 2026
  • RL for language models: policy gradients, PPO, GRPO and DPO (9 questions). Google DeepMind

Week 1 gets you the outline. Weeks 5–7 make these answers deep: gradients, failure modes, a small GRPO run.

Part 5

Generating text: picking the next token

Temperature, top-k and top-p

“The best part of visiting Paris is the ___” (made-up scores)

Greedy: always the top token. Repetitive, loops.

Temperature: divide scores by T before softmax. Low = safe and dull; high = varied, then incoherent.

Top-k: keep the k best. Fixed k is too many when the model is sure, too few when it isn’t.

Top-p: keep the smallest set whose probability adds to p. Adapts to confidence.

Beam search: keep several partial sequences. Good for translation, bland for open-ended text.

Asked as “compare greedy, beam, top-k, top-p and temperature; when does each fail?” (Google DeepMind) and “how do you adjust creativity?” (Anthropic).

The whole pipeline in one minute

  1. Tokenize trillions of words of filtered, deduplicated text into integers.
  2. Pretrain a Transformer to predict the next token: ≈ 6 × N × D FLOPs.
  3. The result is a base model that completes documents.
  4. SFT on written examples teaches the assistant format.
  5. Preference tuning (RLHF or DPO) teaches what people prefer.
  6. Reasoning RL with checkable rewards teaches long problem solving.
  7. At use time, sample one token at a time, reusing the KV cache.

If you can say this, with one number per line, Week 1 is done.

Your Week 1 plan

#ResourceHoursTake away
1Deep Dive into LLMs like ChatGPT Karpathy3.5the whole map, hallucination, “tokens to think”
2Let’s build the GPT Tokenizer Karpathy2BPE by hand; tokenizer-caused bugs
3Attention Is All You Need Vaswani et al.2the 2017 column of the table
4GPT-2 paper Radford et al.1.5decoder-only, pre-norm, WebText
5Let’s reproduce GPT-2 (124M) Karpathy4a real training loop, end to end
6InstructGPT Ouyang et al.2SFT → reward model → PPO, alignment tax
7The Illustrated DeepSeek-R1 Alammar1R1-Zero, cold start, distillation

Listed times add up to 16 hours. To fit the 8–10 hour budget, watch the videos at 1.5× and read the papers only for the parts this deck covers.

Self-test: answer out loud, then open

Shape of the logits for one sequence in GPT-2?

[T, 50,257]: one score per vocabulary entry at every position. With a batch, [B, T, 50,257].

Loss of an untrained GPT-2, and why?

About 10.8 = ln(50,257): equal probability on every token.

Why divide attention scores by √d?

Dot products grow like √d; big scores saturate softmax and kill gradients. Scaling keeps the spread near 1.

KV cache per token for Llama 3 70B?

2 × 80 layers × 8 KV heads × 128 × 2 bytes = 320 KiB. At 128K tokens: 40 GiB per sequence.

Training FLOPs for 70B params on 15T tokens?

6 × 7×10¹⁰ × 1.5×10¹³ ≈ 6.3×10²⁴ FLOPs.

Why does RLHF need a KL penalty?

To keep the policy near the SFT model; without it, the policy exploits reward-model errors (reward hacking).

What does GRPO remove from PPO, and how?

The value model. It uses the group’s mean score as the baseline for each answer.

Why can a model fail to count letters in “strawberry”?

It sees tokens (st, raw, berry), not letters.

Next

Weeks 2–3: the architecture tour

Build the GPT-2 → Llama 3 → DeepSeek-V3 → Qwen3 table and defend each choice: RoPE, RMSNorm, SwiGLU, GQA vs MLA, mixture of experts.

Staff-level probe to aim for

“Llama 3 uses GQA, DeepSeek-V3 uses MLA. Compute KV-cache bytes per token for each at 128K context and tell me which you’d pick for a 1M-context product and why.”

You already have half the arithmetic from the KV-cache slide.