A First-Principles Tutorial · Chapter 01

Foundations for
Generative AI

The math and building blocks that recur in every generative model — from dot products to residual connections.

By Prashant Kulkarni · with help from Claude Opus CC BY-NC 4.0

1 Motivation

Why these foundations matter

Every generative AI paper you will ever read assumes you know roughly thirty things. Not thirty thousand — thirty. The same thirty concepts appear in transformers, diffusion models, GANs, VAEs, flow models, and mixture-of-experts architectures, recombined in different ways. If you know these thirty things cold, a new paper drops from "impenetrable" to "oh, that's just attention with a different masking strategy."

This chapter is that list. It is not a textbook — it is a practitioner's reference, designed to build intuition fast and to answer the question every concept raises: why does this show up in generative AI?

The thirty things fall into five groups:

Linear algebra — vectors, matrices, the shape rule, norms, dot products.
Probability — distributions, KL divergence, cross-entropy, the Gaussian.
Calculus & optimization — gradients, the chain rule (= backpropagation), optimizers.
Activation functions — ReLU, GELU, sigmoid, and why nonlinearity is non-negotiable.
Architecture building blocks — embeddings, softmax, layer norm, residual connections, attention.

If you already know some of these, skim the sections you're confident about and slow down at the ones you're not. Every section is self-contained.

2 Linear algebra

Vectors, matrices, and the shape rule

A vector is an ordered list of numbers: $\mathbf{x} \in \mathbb{R}^n$. In deep learning, a vector is a token embedding, a hidden state, an output logit vector. When someone says "a 768-dimensional representation," they mean a vector in $\mathbb{R}^{768}$.

A matrix $A \in \mathbb{R}^{m \times n}$ is a 2D grid of numbers. In deep learning, matrices are weight matrices in linear layers, attention score matrices, batch-of-embeddings stacks.

The shape rule — the one thing to internalize

Matrix multiplication $C = AB$ requires the inner dimensions to match:

The inner dimensions (both p) must be equal — they "cancel out" in the multiplication, leaving the outer dimensions (m × n) as the shape of the result. This single rule governs every linear layer, every attention computation, and every projection in deep learning. Original SVG

A linear layer is just a matrix multiply plus a bias: $\mathbf{y} = W\mathbf{x} + \mathbf{b}$. For a batch of inputs $X \in \mathbb{R}^{B \times d_{\text{in}}}$, the output is $Y = XW^T + \mathbf{b} \in \mathbb{R}^{B \times d_{\text{out}}}$. (Note: PyTorch's nn.Linear stores weights as $(d_{\text{out}} \times d_{\text{in}})$ and transposes internally.)

Norms — measuring size

The $L^2$ norm $\|\mathbf{x}\|_2 = \sqrt{\sum_i x_i^2}$ measures the "length" of a vector. It appears everywhere: cosine similarity divides by norms, layer normalization divides by standard deviation (a norm variant), gradient clipping checks the gradient norm, weight decay penalizes $\|W\|_2^2$.

Broadcasting — the silent bug factory

When PyTorch operates on tensors of different shapes, it broadcasts: dimensions are compared from the right; a dimension of size 1 stretches to match. This is how a bias vector $\mathbf{b} \in \mathbb{R}^d$ gets added to every row of a batch $X \in \mathbb{R}^{B \times d}$. Getting broadcasting wrong is one of the most common sources of silent bugs — tensors multiply in unexpected ways without raising an error.

3 The dot product

The operation that runs everything

\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^n a_i\, b_i = \|\mathbf{a}\|\,\|\mathbf{b}\|\,\cos\theta

The dot product measures similarity. If two vectors point in the same direction, the dot product is large and positive. If they are orthogonal (unrelated), it is zero. If they point in opposite directions, it is large and negative.

Why should you care? Because the dot product is the atomic operation of deep learning:

Attention scores in transformers are dot products between query and key vectors: $\text{score} = \mathbf{q} \cdot \mathbf{k}$.
Cosine similarity (CLIP, retrieval, contrastive learning) is a normalized dot product.
A linear layer $\mathbf{y} = W\mathbf{x}$ computes one dot product per output dimension — each row of $W$ dotted with $\mathbf{x}$.
Logits at a language model's output are dot products between the hidden state and each vocabulary embedding.

Scaling in attention

In attention, dot products are divided by $\sqrt{d_k}$. Why? When $d_k$ is large, the raw dot products grow large in magnitude, pushing softmax into flat or peaked regions where gradients vanish. Dividing by $\sqrt{d_k}$ keeps the variance of the scores near 1 — a variance-preserving trick.

4 Probability

Distributions, expectation, and Bayes

Generative models are, at their core, models of probability distributions. A language model learns $P(\text{next token} \mid \text{context})$. A diffusion model implicitly learns $P(\text{image})$. A VAE parameterizes a distribution over latent variables. You cannot understand generative AI without understanding probability.

A discrete distribution assigns a probability to each outcome: $P(X = x_i) = p_i$, where all $p_i \geq 0$ and $\sum_i p_i = 1$. The softmax output of a language model is a discrete distribution over the vocabulary.

A continuous distribution is described by a probability density function $p(x)$ where $p(x) \geq 0$ and $\int p(x)\,dx = 1$. The latent space of a VAE or diffusion model uses continuous distributions — typically Gaussians.

The Gaussian — the distribution that rules everything

\mathcal{N}(x;\, \mu,\, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

Why the Gaussian is everywhere in generative AI: the forward process in diffusion adds Gaussian noise; the reverse process predicts Gaussian parameters; VAEs use Gaussian encoders and priors; the reparameterization trick samples from a Gaussian; weight initialization uses Gaussian draws; batch/layer normalization maps activations toward a Gaussian. If you understand one distribution deeply, make it this one.

Expectation

The expectation $\mathbb{E}[f(X)]$ is the average value of $f$ under the distribution of $X$. Every loss function in deep learning is an expected value: "the average prediction error over the data distribution." When we write $\mathcal{L} = \mathbb{E}_{x \sim p_\text{data}}[\ell(x, \theta)]$, we mean "average the per-sample loss over the data."

5 Information theory

KL divergence & cross-entropy

Two distributions — one "true," one "approximate." How different are they? This question is the heart of generative modeling, and KL divergence is the standard answer.

D_\text{KL}(P \| Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}

KL divergence is always $\geq 0$ and equals zero only when $P = Q$. It is not symmetric: $D_\text{KL}(P \| Q) \neq D_\text{KL}(Q \| P)$.

Where KL divergence appears:

VAE training: the ELBO contains $D_\text{KL}(q_\phi(z|x) \| p(z))$ — penalizing the encoder for straying from the prior.
Knowledge distillation: matching a student model's distribution to a teacher's.
RLHF: a KL penalty keeps the fine-tuned policy close to the base model.

Cross-entropy — the loss function you use every day

H(P, Q) = -\sum_x P(x) \log Q(x) = H(P) + D_\text{KL}(P \| Q)

Cross-entropy equals the entropy of $P$ plus the KL divergence from $P$ to $Q$. Since $H(P)$ is constant with respect to the model, minimizing cross-entropy is equivalent to minimizing KL divergence. This is why cross-entropy loss trains language models — it is secretly pushing the model distribution $Q$ toward the data distribution $P$.

6 Gradients

The chain rule is backpropagation

The gradient $\nabla_\theta \mathcal{L}$ tells you: for each parameter $\theta$, which direction would increase the loss, and by how much? To minimize, move in the opposite direction.

A neural network is a composition of functions: $\mathcal{L} = f_n(f_{n-1}(\cdots f_1(\mathbf{x})\cdots))$. The chain rule computes the gradient of the composition:

\frac{\partial \mathcal{L}}{\partial \theta_1} = \frac{\partial f_n}{\partial f_{n-1}} \cdot \frac{\partial f_{n-1}}{\partial f_{n-2}} \cdots \frac{\partial f_2}{\partial f_1} \cdot \frac{\partial f_1}{\partial \theta_1}

This chain of multiplications, executed from right to left (output to input), is backpropagation. There is no separate algorithm — backprop is just the chain rule applied systematically to a computational graph.

A two-layer network as a computational graph. The forward pass (solid arrows) flows left to right, computing predictions. The backward pass (dashed arrows) flows right to left, computing gradients via repeated application of the chain rule. Each backward arrow multiplies by one local derivative. Original SVG

Vanishing & exploding gradients

The chain rule is a product of many terms. If each term is slightly less than 1, the product shrinks exponentially — gradients vanish, and early layers stop learning. If each term is slightly greater than 1, the product grows exponentially — gradients explode, and training diverges. After 50 layers with a per-layer scale of 0.5, the gradient is $0.5^{50} \approx 10^{-15}$.

Solutions that appear throughout generative AI: residual connections (provide a gradient highway that bypasses the vanishing product), layer normalization (stabilizes activation scale), careful initialization (Xavier/Kaiming, preserves variance across layers), and gradient clipping (caps gradient norms to prevent explosion).

7 Optimizers

From SGD to AdamW

Once you have gradients, you need a rule for updating parameters. The progression from basic SGD to AdamW is a chain of "fix one problem, discover the next":

The optimizer progression. SGD oscillates in narrow valleys; momentum smooths the path; Adam gives each parameter its own adaptive learning rate (using running averages of gradients and squared gradients); AdamW decouples weight decay from the adaptive step, fixing a subtle interaction that hurts generalization. Original SVG

The Adam update rule uses first and second moment estimates of the gradient:

\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

where $\hat{m}_t$ is the bias-corrected running mean of gradients (momentum) and $\hat{v}_t$ is the bias-corrected running mean of squared gradients (per-parameter scaling). Default hyperparameters: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$.

For transformers, the standard recipe is AdamW with linear warmup + cosine decay: start with a tiny learning rate, linearly ramp up over the first few thousand steps, then cosine-decay toward zero.

8 Activations

Why nonlinearity is non-negotiable

Without activation functions, stacking $n$ linear layers gives you $W_n W_{n-1} \cdots W_1 \mathbf{x} = W_\text{combined}\,\mathbf{x}$ — still just one linear transformation. The activation function is what gives deep networks the ability to model nonlinear relationships.

ReLU is the default for most architectures; GELU (a smooth approximation of ReLU) is standard in transformers; sigmoid squashes to (0,1) and appears in gating mechanisms; SiLU/Swish is common in diffusion model U-Nets. Note: the activation function curves shown here are intentionally schematic — see your framework docs for exact shapes. Original SVG

9 Building blocks

The five components that appear everywhere

Embeddings

An embedding maps discrete tokens to continuous vectors: $\text{Embedding}(i) = W_e[i, :]$ where $W_e \in \mathbb{R}^{V \times d}$. It is a lookup table — mathematically equivalent to multiplying by a one-hot vector, but much cheaper. Every language model starts here. CLIP aligns image and text embeddings in a shared space. VQ-VAE codebooks are embeddings.

Softmax

Converts raw scores (logits) into a probability distribution: $\text{softmax}(z_i) = e^{z_i} / \sum_j e^{z_j}$. All outputs are positive and sum to 1. Temperature $\tau$ controls sharpness: $\text{softmax}(z_i / \tau)$ with $\tau \to 0$ approaches argmax (deterministic), $\tau = 1$ is standard, $\tau \to \infty$ approaches uniform (maximum randomness). Temperature is the inference-time knob for controlling LLM creativity.

Layer normalization

Normalizes activations across the feature dimension for each sample: $\text{LayerNorm}(\mathbf{x}) = \gamma \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$, where $\mu$ and $\sigma^2$ are computed over the feature dimension. Unlike batch norm (which computes statistics across the batch), layer norm works with variable-length sequences and doesn't depend on batch size. It appears in every transformer block.

Residual connections

The single most important architectural trick for deep networks: $\mathbf{y} = f(\mathbf{x}) + \mathbf{x}$. Instead of learning a full transformation, the network learns the residual — how much to change the input. The gradient of $\mathbf{y}$ with respect to $\mathbf{x}$ includes an identity term $I$, which ensures gradients flow even when $\partial f / \partial \mathbf{x}$ is small. This is why 100-layer transformers can train; without residual connections, gradients would vanish after 10–20 layers.

A standard pre-norm transformer block. The input $h$ flows through layer norm → attention → add (with residual skip) → layer norm → MLP → add (with residual skip) → output $h'$. The dashed gold lines are the residual connections — they guarantee gradient flow even through 100+ layers. Original SVG · architecture after Vaswani et al. (2017)

10 Preview

Attention — a taste

Attention is covered in depth in Chapter 02. Here is the minimal version to set up intuition.

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Three matrices — Queries ("what am I looking for?"), Keys ("what do I contain?"), Values ("what information do I provide?"). $QK^T$ computes pairwise similarity (dot products!) between all positions. Softmax turns similarities into weights. The weighted sum of Values is the output. The $\sqrt{d_k}$ scaling prevents large dot products from saturating the softmax (recall §3).

Multi-head attention runs this in parallel $h$ times with different learned projections, then concatenates. Each head can attend to different aspects — one head might track syntax, another might track co-reference, a third might track sentiment. Full treatment in Chapter 02.

11 Field notes

Residual connections save gradient flow

The companion notebook runs an experiment that makes the vanishing gradient problem viscerally concrete. Two 20-layer networks are trained to fit a sine wave: one with residual connections, one without.

Source

Experiments from the Module 01 notebook in the learn-generative-ai repo.

Gradient magnitudes across layers

After one forward+backward pass, measure the average gradient magnitude $\|\nabla_{W_l}\mathcal{L}\|$ at each layer $l$. Without residual connections, the gradient at layer 1 (earliest) is orders of magnitude smaller than at layer 20 (nearest to the loss). The network is effectively "blind" in its first few layers. With residual connections, gradients are roughly uniform across all layers — every layer gets a useful training signal.

Training dynamics

Over 3000 training steps, the plain network's loss plateaus early — it cannot learn the sine wave because its early layers are frozen by vanishing gradients. The residual network's loss drops steadily and fits the function cleanly. Same architecture, same optimizer, same data — the only difference is the $+ \mathbf{x}$ skip connection. That single addition makes a 20-layer network trainable.

Why this matters beyond foundations

Every transformer block, every U-Net level, and every ResNet stage uses residual connections for exactly this reason. Understanding why — that the skip connection adds an identity term to the gradient product, preventing exponential decay — is the single most important architectural intuition you can carry into the rest of this tutorial series.

12 Playground

Watch gradient descent find a minimum

A ball on a 2D loss surface. Drag the starting point, change the learning rate, toggle momentum — and watch how the optimizer navigates valleys and saddle points. The contour lines show equal-loss curves; darker means lower loss.

      
      LR: 0.030
       momentum

click to place start · press Run

⁂ ⁂ ⁂

13 Closing

Takeaways & exercises

The six things to carry forward

The dot product measures similarity — and it is the atomic operation of attention, linear layers, cosine similarity, and logit computation.
The shape rule $(m \times \mathbf{p}) \cdot (\mathbf{p} \times n) \to (m \times n)$ governs every matrix multiply. If you can track shapes through a forward pass, you can read any architecture.
Cross-entropy loss minimizes KL divergence — training a language model on next-token prediction is literally pushing the model distribution toward the data distribution.
Backpropagation is the chain rule — a product of local Jacobians. Vanishing gradients happen when that product shrinks; exploding gradients happen when it grows.
Residual connections add an identity term to the gradient — preventing the vanishing product. This is why deep networks are trainable at all.
AdamW with warmup + cosine decay is the default optimizer recipe for transformers. Knowing why (adaptive per-parameter LR + decoupled weight decay + stable early training) helps you debug when training goes wrong.

Exercises

1. Verify the shape rule: if $Q \in \mathbb{R}^{B \times T \times d_k}$ and $K \in \mathbb{R}^{B \times T \times d_k}$, what is the shape of $QK^T$? What does each dimension represent?

2. Compute $\text{softmax}([2.0, 1.0, 0.1])$ by hand. Then compute it with temperature $\tau = 0.5$ and $\tau = 2.0$. What changes?

3. Open the notebook. Compare gradient magnitudes in a 20-layer network with and without residual connections. At which layer does the plain network's gradient drop below $10^{-6}$?

4. In the playground above, set the learning rate to 0.001 (very low) and run. Then set it to 0.1 (very high). Describe what happens in each case and why.

5. Derive: for a residual block $y = f(x) + x$, show that $\partial y / \partial x = \partial f / \partial x + I$. Explain why this prevents vanishing gradients even when $\|\partial f / \partial x\| \ll 1$.