Autoregressive Models

One token at a time

The autoregressive approach to generation is radically simple: model the joint probability of a sequence as a product of conditional probabilities, then generate one element at a time, each conditioned on everything before it.

This is the chain rule of probability — not an approximation, but an identity. Any joint distribution can be decomposed this way. The modeling choice is how to parameterize each conditional $p(x_t | x_{

Next-token prediction — the only loss

This is cross-entropy loss at every position, averaged over the sequence. The model sees a batch of text, predicts the next token at every position in parallel (thanks to causal masking — see Chapter 02), and receives a gradient for every prediction. No labels needed beyond the text itself — the next token IS the label. This is self-supervised learning.

GPT — decoder-only transformers

GPT is a stack of transformer decoder blocks with causal attention masking (covered in detail in Chapter 02). The architecture was described by Radford et al. (2018/2019) and has remained remarkably stable: token embeddings + positional encoding → $N$ transformer blocks (each: LN → causal self-attention → residual → LN → MLP → residual) → final LN → linear projection to vocabulary → softmax.

The GPT family progression demonstrates emergent capabilities at scale:

• GPT-1 (2018): 117M parameters. Fine-tuned for each downstream task.
• GPT-2 (2019): 1.5B parameters. Zero-shot task performance emerges.
• GPT-3 (2020): 175B parameters. Few-shot in-context learning works.
• GPT-4 (2023): estimated >1T parameters (MoE). Reasoning, multimodal input.

Each scale-up brought qualitative capability improvements that were not predicted by smaller models.

Decoding strategies — how to pick the next token

The model outputs a probability distribution over the vocabulary. How you sample from it dramatically affects output quality:

Top-p (nucleus) sampling is the adaptive winner: it keeps the smallest set of tokens whose cumulative probability exceeds $p$. When the model is confident, only 2–3 tokens pass; when uncertain, hundreds pass. This is why it produces better text than fixed top-k.

In-context learning — examples in the prompt

In-context learning is the ability to learn new tasks from examples in the prompt — without updating any parameters. Give GPT-3 three English-to-French translation examples, and it translates a fourth sentence correctly despite never being fine-tuned on translation.

Three regimes: zero-shot (task description only), few-shot (2–5 examples + query), and chain-of-thought (add "let me think step by step" to elicit intermediate reasoning). CoT dramatically improves math and logic performance — the model uses its own generated text as a working memory scratchpad.

RLHF — from prediction to helpfulness

A base language model predicts the most likely continuation — but likely text isn't always helpful, truthful, or safe. RLHF bridges this gap in three steps:

Scaling laws — bigger is predictably better

Kaplan et al. (2020) and Hoffmann et al. (2022, "Chinchilla") showed that loss follows clean power laws in parameters $N$ and data $D$:

If you know performance at one scale, you can predict it at 10× or 100× scale. Chinchilla's key finding: optimal training uses roughly equal compute on parameters and data — most existing models were undertrained (too many parameters, not enough data). This shifted the field toward training smaller models on more data (LLaMA, Mistral).

KV cache and inference tricks

Autoregressive generation is inherently sequential — each token requires a forward pass. The KV cache avoids recomputing keys and values for all previous tokens at every step: compute $K_t, V_t$ only for the new token, append to the cache, attend using the full cached $K_{1:t}, V_{1:t}$. This reduces per-step computation from $O(t \cdot d)$ to $O(d)$ for the projection.

Other key efficiency techniques:

• Grouped-Query Attention (GQA): share K/V heads across multiple Q heads to reduce KV cache size by $G$×. Used in LLaMA-2, Mistral.
• Speculative decoding: a small draft model proposes $K$ tokens; the large model verifies them in parallel. Produces the exact same distribution as the large model alone, just 2–3× faster.
• Quantization: INT4 weights (GPTQ, AWQ) reduce a 7B model from 14GB to 3.5GB with negligible quality loss.
• Flash Attention: IO-aware exact attention that reduces memory from $O(T^2)$ to $O(T)$. Standard everywhere.

Building MiniGPT from scratch

Character-level tokenization and BPE

The notebook starts with a character-level tokenizer (vocabulary of ~65 characters), then demonstrates BPE (Byte Pair Encoding) to show how subword tokenization works — iteratively merging the most frequent adjacent pairs. This is the algorithm behind GPT-2's tokenizer (50,257 tokens) and all modern tokenizers.

Training on Shakespeare

A tiny 4-layer, 4-head transformer (~200K parameters) is trained on Shakespeare's complete works. The loss curve shows the characteristic pattern: rapid initial descent (learning character frequencies and common words), then gradual refinement (learning longer-range patterns, dialogue structure). Perplexity drops from ~65 (nearly random among characters) to ~5 (the model has strong predictions most of the time).

Temperature effects

Generated samples at different temperatures are strikingly different: $\tau = 0.5$ produces conservative, repetitive text with common words; $\tau = 1.0$ produces varied, sometimes surprising text; $\tau = 1.5$ produces gibberish. The sweet spot ($\tau \approx 0.8$) captures Shakespeare's vocabulary and rhythm without degenerating.

Attention map analysis

Visualizing the attention maps for a sample prompt reveals head specialization: some heads show a strong diagonal pattern (attending to the previous token — positional), while others show broader, content-dependent patterns (attending to semantically related tokens earlier in the sequence). The average attention distance per head quantifies this: positional heads have low average distance; content heads have high average distance.

KV cache benchmarking

The notebook implements a KV-cached version of MiniGPT and benchmarks it against the naive version. For 400-token generation, the cached version is ~3× faster — and the speedup grows with sequence length because the naive version recomputes all attention at every step.

See how sampling changes the distribution

A simulated next-token distribution over 12 "tokens." Adjust temperature, top-k, and top-p to see how the probability bars change in real time. Notice: low temperature concentrates mass on the top token; top-k zeroes out the tail; top-p adaptively adjusts the cutoff based on cumulative probability.

Takeaways & exercises

1. Autoregressive = chain rule of probability. The factorization $p(x) = \prod p(x_t | x_{
2. Next-token prediction is the only objective — but to predict well, the model must learn everything about language.
3. Decoding strategy matters as much as the model. Temperature, top-k, and top-p dramatically change output quality.
4. In-context learning is an emergent ability — it appears at sufficient scale and enables few-shot task performance without fine-tuning.
5. RLHF bridges prediction and helpfulness — same weights, different behavior.
6. Scaling laws make outcomes predictable — loss follows power laws in parameters, data, and compute.
7. The KV cache is essential — without it, autoregressive generation is quadratically slow.

Further reading

• Radford et al. (2018/2019) — GPT and GPT-2.
• Brown et al. (2020) — GPT-3. In-context learning at scale.
• Ouyang et al. (2022) — InstructGPT / RLHF.
• Kaplan et al. (2020) — Scaling Laws for Neural Language Models.
• Hoffmann et al. (2022) — Chinchilla: compute-optimal training.
• Holtzman et al. (2020) — Nucleus Sampling (top-p).
• Leviathan et al. (2023) — Speculative Decoding.
• P. Kulkarni — Module 07 notebook.