Generative Adversarial
Networks

How do you teach a network to imagine without ever telling it what's right?

Suppose I gave you a hundred thousand photographs of human faces and asked you to write a program that produces brand-new, plausible faces — faces that don't exist, but that look like they could. The straightforward answer is: model the probability distribution over faces, then sample from it. The trouble is that "the probability distribution over faces" is a function over millions of pixels, with structure on every scale from eyelashes to lighting, and we have no good way to write it down.

For most of the history of machine learning, generative modeling meant likelihood-based generative modeling — assume a probabilistic model $p_\theta(x)$, fit its parameters by maximum likelihood, then sample. Variational autoencoders, autoregressive models, normalizing flows: all of these compute, or at least bound, an explicit probability for every datapoint. They are honest about what they are: explicit density estimators that happen to know how to draw.

But likelihood is a strange yardstick for image quality. A model can assign reasonable likelihood to blurry, half-formed faces and lose almost no log-loss in the process. The gradient from a likelihood objective doesn't really know that a slightly off eye matters more than a slightly off background. Likelihood treats every pixel as equally important; perception does not.

This is the central trick of the Generative Adversarial Network. Instead of fitting a density, we set up a two-player game: a generator network that tries to produce realistic samples, and a discriminator network that tries to tell real from fake. We train them together, and we let them coevolve. The discriminator pulls itself up by its bootstraps as the generator improves. The generator pulls itself up by trying to fool an ever-smarter critic. If the game converges, we end up with a generator whose samples are indistinguishable from the data — and we never had to write down a single likelihood.

A GAN is a forger and a detective sharing an apartment. The forger gets better because the detective gets smarter. The detective gets smarter because the forger gets better. Eventually, neither of them can hear themselves think.

The original GAN paper, by Ian Goodfellow and collaborators in 2014, was four pages of math and a handful of fuzzy MNIST samples. It took about three years for the trick to reliably produce photorealistic faces, another two for it to handle complex scenes, and another five for it to be partly displaced by diffusion models. But the central idea — train a generative model by having an adversary score its samples — turns out to be one of the most generative ideas in deep learning. It seeded image-to-image translation, super-resolution, voice cloning, neural rendering, simulation-based inference, robust training, and a sprawling literature on stable two-player optimization. We are going to walk through that idea slowly, building it from a single intuition into a real architecture, and then into the modern variants that actually work.

One network imagines, another judges

Let's give the two players names and jobs.

The generator never sees a real image. Read that sentence again — it's the most counterintuitive thing about GAN training, and it's also the source of every bit of training instability we'll encounter later. The generator only learns through the discriminator's gradient. If the discriminator says "this fake looks more real today than it did yesterday," the generator updates in that direction. The data itself is invisible to the generator; the discriminator is the only window through which any signal about "real-ness" can reach the generator's parameters.

Both networks are just ordinary feedforward (or convolutional) neural networks. There is nothing architecturally exotic about them. What makes a GAN a GAN is the objective: they are trained against each other on a single shared loss function, with one trying to maximize it and the other trying to minimize it. That structure — a single loss, two adversaries — is the entire idea. Everything else in this chapter is consequence.

Writing down the game and solving it

Let's make the picture above formal. The discriminator $D$ wants to assign high $D(x)$ to real data and low $D(G(z))$ to fakes. That's a binary classification problem, and the natural loss is binary cross-entropy. Goodfellow's original paper packages both networks' objectives into a single value function:

$D$ wants to maximize $V$ — push $D(x) \to 1$ on reals and $D(G(z)) \to 0$ on fakes. $G$ wants to minimize $V$ — push $D(G(z)) \to 1$. The full GAN training problem is the saddle point

The asymmetry of $\min$ and $\max$ is the source of every GAN headache to follow. We are not minimizing a function. We are looking for an equilibrium of a game.

Step 1: solving the inner maximum analytically

Hold $G$ fixed and ask: what is the best discriminator? Write $p_g(x)$ for the distribution induced by pushing $z \sim p_z$ through $G$. Then for any $x$, the integrand of $V$ becomes a function of the scalar $D(x)$:

This is a one-dimensional concave problem in $D(x)$. Differentiate, set to zero:

Solve for $D(x)$:

This is a beautiful result and worth pausing on. The optimal discriminator is computing — at every point $x$ — the relative density of real data versus fake data. It returns $\tfrac{1}{2}$ when the two distributions match and tilts toward 1 or 0 wherever one dominates.

Step 2: substituting back to find the generator's problem

Plug $D^*$ back into $V$. After some careful algebra (add and subtract $\log 2$ terms, recognize the result as KL divergences), you arrive at:

where $\operatorname{JSD}$ is the Jensen–Shannon divergence — a symmetric, smoothed cousin of KL divergence that is bounded between $0$ and $\log 2$. The minimum of $C(G)$ is at $-\log 4$, achieved exactly when $p_g = p_{\text{data}}$.

When the generator wins the game perfectly, its sample distribution equals the data distribution. The minimax solution is "match the data."

So the GAN game has the right global optimum: at the saddle point, the generator's distribution is exactly the data distribution and the discriminator is the constant $\tfrac{1}{2}$ — it can do no better than guessing.

Step 3: the non-saturating fix

There's a practical problem with the original objective the moment you start training. At the start of training, $G$ is bad — its samples are obvious noise. So $D$ quickly learns to assign $D(G(z)) \approx 0$. Now consider the generator's gradient. $G$ is trying to minimize

The derivative of $\log(1 - D)$ with respect to $D$ is $-\tfrac{1}{1 - D}$. When $D \approx 0$, that derivative is roughly $-1$ — small. The generator gets almost no gradient at exactly the moment it most needs one. We say the loss has saturated.

The fix, also from Goodfellow's original paper, is a tiny but consequential reformulation. Instead of having $G$ minimize $\log(1 - D(G(z)))$, have it maximize $\log D(G(z))$:

At the same equilibrium, this loss has the same minimum, but its gradient at $D \approx 0$ is huge — $-\tfrac{1}{D}$ is enormous when $D$ is near zero. The generator gets a strong signal early and a weaker one late, which is exactly what you want from a curriculum. This is the non-saturating generator loss, and essentially every GAN since 2014 uses some version of it.

SATURATING                       NON-SATURATING
G minimizes log(1 - D(G(z)))     G maximizes log D(G(z))
gradient ∝ -1/(1-D)              gradient ∝ 1/D
shrinks as D → 0  ✗              grows as D → 0  ✓

The trouble with chasing a moving target

If GANs only had to solve a minimization problem, we'd be done. Stochastic gradient descent on a smooth-ish loss landscape works. But a GAN is not minimizing anything. It is searching for a saddle point — a point where one set of variables (the discriminator) is at a maximum and another set (the generator) is at a minimum. Saddle points behave very differently from minima under gradient descent. Let's catalog what goes wrong.

The toy example: gradient descent on $xy$

Consider the simplest possible minimax problem:

The unique saddle point is $(0, 0)$. The gradients are $\partial_x(xy) = y$ and $\partial_y(xy) = x$. Run alternating gradient descent–ascent with step $\eta$:

This is rotation. The norm $x^2 + y^2$ doesn't decrease — it actually grows slightly with naive Euler updates. The trajectory orbits the saddle point indefinitely. No matter how small you make $\eta$, you don't converge; you spiral.

Real GAN losses are much messier than $xy$, but the rotational character of the gradient field is the same. In a small neighborhood of the equilibrium, the dynamics look like rotation, and unless you do something careful — momentum tricks, regularization, two-time-scale updates — you don't actually arrive at the equilibrium. You just orbit it.

Three failure modes you will see in real training

1. Oscillation. Loss curves that look like they're dancing instead of descending. The generator and discriminator take turns being on top. Sometimes you get useful samples; sometimes the next epoch destroys them.
2. Vanishing gradients. The discriminator becomes too good. $D(G(z)) \approx 0$ for all fakes. With the original saturating loss, the generator stops moving. (This is the bug the non-saturating loss was invented to patch — but variants of the same problem reappear in different forms.)
3. Mode collapse. The generator finds one output that fools the discriminator and just produces it forever. The samples are sharp but identical. The variety of the data distribution has collapsed into a single point or a small set of points.

Where mode collapse comes from, mechanistically

Mode collapse is the most distinctive GAN failure, and it deserves a paragraph of its own. The generator's loss only depends on the discriminator's output. Suppose for some particular $G(z^*)$, the discriminator is fooled — $D(G(z^*)) \approx 1$. The generator has no gradient pushing it toward diversity; the loss only cares about whether each sample fools the discriminator, not whether different samples differ from each other. So the easiest way for $G$ to do well is to produce $G(z^*)$ for almost every $z$.

Eventually $D$ catches on and starts assigning low probability to that one mode, but in the meantime $G$ has flushed away its diversity. The dance continues: $G$ jumps to a new mode, $D$ chases, $G$ jumps again. Over time the generator may visit many modes, but at any instant it is producing a tiny slice of the data distribution. There is no force in the original GAN objective that prevents this.

Almost every later GAN paper is, in some sense, an answer to this single problem: how do we keep the generator honest about diversity while still letting it specialize? The answers — minibatch discrimination, gradient penalties, Wasserstein distances, spectral normalization, two-time-scale updates — will occupy us for the rest of the chapter.

From four pages of math to photorealism

The GAN literature exploded between 2014 and 2020 — at the peak there were a couple of new GAN variants per week, each with a clever acronym. Looking back, almost all of the productive work falls into a few clean threads, and the picture is much clearer now than it was while it was happening.

• 2014

  Goodfellow et al., "Generative Adversarial Nets"

  Four pages, the minimax objective, the existence proof, fuzzy MNIST samples and an early CIFAR-10 attempt. The paper is short because the idea is short. Almost everything else was discovered later.

• 2015

  DCGAN — Radford, Metz, Chintala

  The first GAN people could reliably train. Strided convolutions instead of pooling, batch norm everywhere, ReLU/LeakyReLU activations, no fully connected layers in the generator's body. A grab bag of architectural choices that made the loss surface tractable. Suddenly GANs produced recognizable bedrooms and faces.

• 2016

  Improved Techniques (Salimans et al.) and InfoGAN

  Feature matching, minibatch discrimination, virtual batch norm, label smoothing, historical averaging — most of the heuristics that everyone cargo-cults to this day come from Salimans et al. InfoGAN added a mutual-information term that disentangled latent codes — the first GAN that knew which dimension controlled "rotation" and which controlled "thickness."

• 2017

  WGAN and WGAN-GP — the Wasserstein turn

  Arjovsky, Chintala, Bottou observed that JSD-based GANs have zero gradient when supports of $p_{\text{data}}$ and $p_g$ don't overlap (which is generic for low-dimensional manifolds in high-dimensional space). They replaced JSD with the Wasserstein-1 distance, which has gradients everywhere. WGAN-GP (Gulrajani et al., a few months later) replaced WGAN's brittle weight clipping with a gradient penalty, and gave the field its first GAN that mostly just worked.

• 2017

  Progressive Growing of GANs — Karras et al.

  Train at 4×4, then 8×8, then 16×16, etc., gradually fading in new layers. Suddenly we had 1024×1024 celebrity faces. The same group later turned this into StyleGAN.

• 2017

  pix2pix and CycleGAN

  Image-to-image translation. pix2pix needed paired data (sketch ↔ photo); CycleGAN didn't, using a cycle-consistency loss instead. Horses became zebras, summer became winter, satellites became maps. A reminder that GANs aren't only for "draw me a face from noise" — they're a general tool for learning distributions over images.

• 2018

  Spectral Normalization (Miyato et al.) and SAGAN

  A clean, parameter-free way to enforce a Lipschitz constraint on the discriminator: divide each weight matrix by its spectral norm at every step. Suddenly WGAN-GP-quality results without the gradient penalty's overhead. Self-Attention GAN added attention layers and won on ImageNet at the time.

• 2018

  BigGAN — Brock, Donahue, Simonyan

  "What if we just made the model big?" 256×256 ImageNet samples that were genuinely good. Big batches, big models, careful regularization, the truncation trick. Showed that GANs scale, given enough compute and patience.

• 2019

  StyleGAN and StyleGAN2 — Karras et al.

  The most influential generator architecture of the era. A mapping network from $z$ to a "style" space $w$, AdaIN-based style injection at every resolution, noise injection for stochastic detail. StyleGAN2 fixed water-droplet artifacts with weight demodulation. The "this person does not exist" generation.

• 2021

  StyleGAN3 — alias-free generation

  Karras et al. noticed that StyleGAN's outputs had texture stuck to absolute pixel coordinates rather than to the underlying object — they'd discovered alias artifacts. StyleGAN3 reformulated the generator as a continuous signal-processing pipeline, fixing it. The technical deepest of the StyleGAN papers.

• 2021–22

  The diffusion turn

  Diffusion models eclipsed GANs for unconditional image generation around the time of DALL·E 2 and Imagen. GANs are sharper but less diverse; diffusion is slower but covers the distribution more honestly. The community moved.

• 2023

  GigaGAN — Kang et al.

  A reminder that GANs aren't dead. A 1B-parameter GAN with text conditioning that produces 512×512 images at competitive quality, in a single forward pass — orders of magnitude faster than diffusion. The story now is: GANs win on speed and on certain conditional tasks; diffusion wins on coverage and ease of training.

• 2024+

  Distillation and hybrids

  Modern systems often distill a diffusion model into a GAN-like one-step generator (e.g. consistency models, ADD, SD-Turbo). The generator's adversarial loss has become a tool used inside diffusion training rather than a competing paradigm. The two ideas have merged in practice.

From a vector of noise to an image

The generator is, mechanically, a function $G: \mathbb{R}^d \to \mathbb{R}^{H \times W \times 3}$. You hand it a small random vector and it hands back a picture. Behind that simple type signature, there are real architectural choices, and most GAN progress between 2015 and 2021 was about getting the generator's architecture right.

DCGAN: the first generator that worked

The original GAN's generator was a multilayer perceptron, which is fine for MNIST but doesn't scale. The DCGAN paper (Radford, Metz, Chintala 2015) made image generation tractable with a small set of architectural rules:

• Replace pooling layers with strided convolutions (in $D$) and fractionally-strided / transposed convolutions (in $G$).
• Use batch normalization in both $G$ and $D$, except in the very first layer of $G$ and the very last layer of $D$.
• Remove fully-connected hidden layers — go straight from latent vector to a small spatial feature map, then upsample.
• ReLU in $G$ (Tanh on the output); LeakyReLU in $D$.

The shape of the DCGAN generator is something like: take a 100-dimensional latent $z$, project it into a $4 \times 4 \times 1024$ tensor via a learned linear layer, then run a stack of transposed convolutions that double the spatial size and halve the channel count at each step. End with a Tanh nonlinearity to clip outputs to $[-1, 1]$.

   z (100,) ─►  fc + reshape  ─►  4×4×1024
                                     │
                                     ▼  ConvTranspose2d ↑2
                                  8×8×512
                                     │
                                     ▼  ConvTranspose2d ↑2
                                 16×16×256
                                     │
                                     ▼  ConvTranspose2d ↑2
                                 32×32×128
                                     │
                                     ▼  ConvTranspose2d ↑2 (Tanh)
                                 64×64×3

This is the canonical "generator pyramid." It's easy to draw, easy to implement, and it works on $64 \times 64$ images with a few hours of training on a single GPU. Beyond $128 \times 128$ it starts to struggle, and the next two architectural ideas — progressive growing and StyleGAN — are essentially answers to that struggle.

Progressive growing

Karras et al. observed that training a GAN at high resolution from scratch is hopeless: the discriminator instantly wins because real $1024 \times 1024$ images are nothing like the noise the generator produces. Their fix: start small. Train the GAN at $4 \times 4$ until it's stable. Then add a layer to $G$ and a corresponding layer to $D$, doubling resolution. Slowly fade the new layer in over thousands of steps so that no jarring distribution shift hits either network. Then double again. By the time you reach $1024 \times 1024$, the discriminator has been trained on every resolution along the way, and the generator's coarse structure is locked in before fine details even appear.

This is curriculum learning for a generator, and it broke the high-resolution barrier.

StyleGAN: the style-based generator

The StyleGAN family (StyleGAN, StyleGAN2, StyleGAN3) introduced one of the most important architectural ideas in generative modeling: separate the latent code's content from its style influence. The key innovations are:

1. The mapping network. Instead of feeding $z$ directly into the convolutional generator, pass it through a small MLP that produces an intermediate vector $w$. This $w$ space turns out to be much better disentangled than $z$ — directions in $w$ correspond more cleanly to interpretable factors like pose, hair length, lighting.
2. The synthesis network starts from a learned constant input — yes, a single constant feature map, not the latent itself — and is sculpted by $w$ at every layer.
3. AdaIN injection. At each resolution, $w$ is projected to a per-channel scale and bias, and then used to modulate a normalized feature map: $\text{AdaIN}(x, w) = \sigma_w \cdot \text{normalize}(x) + \mu_w$. Style is injected as channel-wise affine transformations.
4. Per-pixel noise. A small amount of independent Gaussian noise is added at each layer, allowing the network to express stochastic detail (individual hairs, freckles) without consuming latent capacity.

Why does this work so well? Empirically, the mapping network gives the latent space the room it needs to "untangle." A vanilla generator has to use $z$ for two jobs at once — coarse layout and fine style — and these tasks fight for the same dimensions. By routing $z$ through an MLP first and only injecting it as channel-wise modulations, StyleGAN allows the synthesis network to focus on spatial structure while the style code carries appearance information. Style mixing — feeding two different $w$ vectors at different layers — produces controllable hybrids: keep the pose from one face, the hairstyle from another.

What the discriminator actually learns

The discriminator gets less attention than the generator in the popular literature — generators produce the visible artifacts, after all — but the discriminator is doing more interesting work. It's not a classifier in the usual sense. It's a learned, ever-improving distance metric between distributions, and the generator only learns through its judgment.

Discriminator as density-ratio estimator

We saw in §3 that the optimal discriminator computes $D^*(x) = \tfrac{p_{\text{data}}}{p_{\text{data}} + p_g}$. Equivalently, its logit is $\log\tfrac{p_{\text{data}}}{p_g}$. So the discriminator is implicitly estimating the ratio of two densities — a problem that classical statistics studied for decades under the name density-ratio estimation. It turns out that estimating a ratio of densities, by training a classifier to tell samples apart, is a beautifully sample-efficient trick: you never need to compute either density on its own.

This connects GANs to a much wider literature. Noise Contrastive Estimation, contrastive predictive coding, and modern self-supervised methods (SimCLR, MoCo, CLIP's contrastive head) are all density-ratio estimators in disguise. A GAN's discriminator is a particularly aggressive instance: the "noise distribution" isn't fixed — it's the generator's current output, and it adapts.

Discriminator as learned perceptual loss

From the generator's vantage point, $-\log D(G(z))$ is just a loss function. But it is a very particular kind of loss function: a function from images to scalars, computed by a neural network. It is, in effect, a learned perceptual loss. The discriminator's gradient teaches the generator which features to attend to, and the answer changes over training.

Early in training, the discriminator picks up trivial cues — checkerboard artifacts from transposed convolutions, color statistics, low-frequency blur. The generator fixes those. The discriminator moves on to more subtle cues: edge sharpness, texture coherence, the geometry of an eye. The generator chases. The discriminator finds new things to complain about. This is why GAN samples are sharp where likelihood-trained models tend to be soft: the discriminator's gradient is uniquely good at punishing perceptually obvious flaws.

The Lipschitz question

The discriminator's gradient is exactly the signal the generator learns from. If those gradients can blow up — if a tiny change in $G(z)$ can produce a huge change in $D(G(z))$ — then the generator's training signal becomes wildly noisy. Conversely, if the gradients vanish, the generator stops moving. We want $D$ to be well-conditioned: its gradient with respect to its input should be of moderate, controlled magnitude everywhere.

Formally, we want $D$ to be Lipschitz-continuous: there exists a constant $K$ such that $|D(x) - D(y)| \leq K \|x - y\|$ for all $x, y$. Most modern GAN training is, in some sense, an answer to the question "how do I keep the discriminator Lipschitz without crippling its capacity?" The two main answers — gradient penalty (WGAN-GP) and spectral normalization — get full treatment in §11.

Recognizing what's going wrong from the loss curves

If you train enough GANs, you develop an intuition for failure modes — the visual signature each leaves in the loss curves and sample grids. Let's catalog the four most common ones, what they look like, and why they happen.

Mode collapse

What it looks like: sample diversity drops to near zero. Every generated image is the same image, or one of three or four images. The discriminator's loss may look fine. The FID score is terrible. A sample grid shows a wall of identical faces.

What it is: the generator has found a small set of outputs that the current discriminator can't tell from data, and the gradient signal pushes it to keep producing those. There is no force in the vanilla GAN objective that rewards diversity per se; the generator is judged sample-by-sample, not as a distribution.

What helps: minibatch discrimination (let $D$ see multiple generated samples at once and detect that they're identical), unrolled GANs (let $G$ "look ahead" at how $D$ will respond to its update), feature matching, packed GANs, and switching to a Wasserstein objective which gives diversity-aware gradients.

Vanishing discriminator gradient

What it looks like: generator loss flatlines very early. Sample quality stops improving around epoch 5 and never recovers. Discriminator output is saturated — almost always 1 on real, 0 on fake.

What it is: the discriminator has become so confident that the generator's gradient (under the saturating loss) is essentially zero. The non-saturating loss patches the simplest version of this, but more subtle versions remain — for instance, when JSD between $p_{\text{data}}$ and $p_g$ is at its maximum because the supports don't overlap, the gradient genuinely is zero everywhere, not just at the saturation points of the sigmoid.

What helps: Wasserstein-style losses (which give meaningful gradients even when distributions don't overlap), instance noise (add Gaussian noise to both real and fake before $D$ sees them, so their supports overlap), dropout in $D$, more conservative $D$ updates.

Oscillation and divergence

What it looks like: loss curves swing wildly. Sample quality is good at epoch 23, terrible at epoch 24, OK at epoch 27. Sometimes losses diverge entirely — generator loss goes to $\infty$ or NaN.

What it is: the rotational dynamics from §4. Without dampening, alternating gradient steps orbit the equilibrium, and at high learning rates they can spiral outward instead of in.

What helps: two-time-scale update rule (TTUR — different learning rates for $G$ and $D$, usually $D$ faster), Adam with appropriate $\beta$s, smaller learning rate, EMA (exponential moving average) of generator weights for evaluation, gradient penalty, spectral normalization.

Cycle starvation

What it looks like: the generator's losses look fine, but the samples are stuck reproducing a small region of the data manifold — say, only producing front-facing faces, never profiles. Subsets of the modes are missing rather than the entire output collapsing.

What it is: a softer version of mode collapse, often caused by an under-capacity generator or a training regime that visits some regions of latent space disproportionately. The optimal $D$ for the current $G$ is happy with what $G$ produces; the gradient toward unexplored modes is too weak to overcome the safety of staying put.

What helps: diversifying losses (e.g. minibatch standard deviation as a feature in $D$, mode-seeking regularization), oversampling rare classes, conditional GANs that explicitly require the generator to produce a specified class.

Six losses, one game

The original GAN loss was a minimax with binary cross-entropy. Almost every "new GAN" paper between 2015 and 2019 was, at heart, a new loss function. Each was an answer to a specific failure mode of the previous one. Walking through the chain shows you the shape of the field.

9.1 Original (saturating) BCE

The version Goodfellow wrote down. Beautifully clean theory, terrible at the start of training because of saturation. We've already discussed it at length in §3.

9.2 Non-saturating BCE

Same fixed point, much better gradients early in training. Still has the issue that when $p_{\text{data}}$ and $p_g$ have non-overlapping support — a routine occurrence in high-dimensional image space — the JSD has gradient zero. The non-saturating fix doesn't help in that regime.

9.3 Least-squares GAN (LSGAN)

Mao et al. 2017 noticed that BCE saturates in part because the sigmoid in $D$ flattens out far from the decision boundary. Their fix: drop the sigmoid and use squared error. The discriminator outputs an unbounded scalar, and the losses become

The discriminator now has gradients far from the boundary too. LSGAN trains more stably than vanilla GANs and produces sharper images on many tasks. It still has no theoretical answer to the support-mismatch problem, but it's a clean engineering improvement.

9.4 Wasserstein GAN (WGAN)

The big theoretical leap. Arjovsky, Chintala, Bottou (2017) argued that JSD is the wrong distance for image distributions because it doesn't degrade gracefully when supports don't overlap. They proposed using the Wasserstein-1 distance (also called Earth Mover's distance):

where $\Pi$ is the set of joint distributions with the right marginals. The Wasserstein distance has a wonderful property: it varies smoothly with the parameters of $p_g$ even when supports don't overlap. The infimum form is intractable, but Kantorovich–Rubinstein duality gives us

where the supremum is over 1-Lipschitz functions. If we let our discriminator $D$ play the role of $f$ — and somehow constrain it to be 1-Lipschitz — then training $D$ to maximize $\mathbb{E}_{p_{\text{data}}}[D(x)] - \mathbb{E}_{p_g}[D(G(z))]$ approximates the Wasserstein distance. The generator then minimizes the same quantity:

The output of $D$ is no longer a probability — it's a critic score, an unbounded real number. Training is famously more stable than BCE-based GANs. Crucially, the discriminator's loss now correlates with sample quality, so you can use it as a stopping criterion.

WGAN's only weakness was its enforcement of the Lipschitz constraint. The original paper used weight clipping: after each update, clamp every weight in $D$ to $[-c, c]$ for some small $c$. This works but is brittle — too small and $D$ underfits, too large and the constraint is violated. The hyperparameter is uncomfortably load-bearing.

9.5 WGAN-GP: the gradient penalty

Gulrajani et al. (2017) replaced weight clipping with a smarter constraint. A 1-Lipschitz function has gradient norm at most 1 everywhere. Why not just regularize $D$'s gradient norm directly? They added a soft penalty term:

where $\hat{x}$ is sampled along the line segment between a real $x$ and a fake $G(z)$, and $\lambda$ is typically 10. This says: "wherever the discriminator is being asked to discriminate, its gradient should have magnitude near 1." It's a pointwise rather than parameter-wise enforcement of the Lipschitz constraint, and it works much better than weight clipping.

WGAN-GP became the workhorse loss for GAN research for several years — most reliable, fewest hyperparameters, decent quality. Its main disadvantage is that the gradient penalty doubles the cost of each $D$ step (you need a second backward pass to compute the gradient norm).

9.6 Hinge loss

Used in SAGAN, BigGAN, StyleGAN's discriminator, and more. The hinge form is simple:

The hinge loss only penalizes the discriminator on examples that are within the margin (or wrong) — confident correct examples contribute zero. This focuses the discriminator's gradient on hard cases, which is exactly what you want from an SVM-flavored objective. Combined with spectral normalization, hinge loss is the most popular setup in modern GAN papers.

9.7 R1 regularization

Mescheder, Geiger, Nowozin (2018) provided one of the cleaner regularizers in this family. They observed that you only need the gradient penalty at real data points to stabilize training. R1 adds

to the discriminator's loss. This is what StyleGAN2 uses. Cheap, theoretically motivated, empirically stabilizing.

From imagining to following instructions

Up to this point, our generator has taken a noise vector and produced a sample drawn from some distribution close to the data — but we haven't told it what to draw. For most useful applications we want control: "draw a face that is smiling," "translate this sketch to a photo," "give me a horse, not a zebra." This is the territory of conditional generation.

cGAN: the simplest extension

Mirza and Osindero (2014) introduced the conditional GAN by adding a label $y$ to both networks. The generator becomes $G(z, y)$ — produce a sample of class $y$ from noise $z$. The discriminator becomes $D(x, y)$ — judge whether $x$ is a real example of class $y$. The minimax objective stays nearly identical:

The label can be a class index (one-hot embedded), a text caption (embedded), a sketch (concatenated as channels), or any other side information. The same theoretical guarantees as the unconditional GAN apply, conditioned on $y$.

AC-GAN: classification as auxiliary task

Auxiliary Classifier GAN (Odena et al. 2017) splits the discriminator into two heads: one that judges real-vs-fake, and one that predicts the class label. The generator is rewarded both for fooling the realness head and for being correctly classified by the class head. This decouples conditioning from realness and tends to produce more class-faithful samples.

pix2pix: image-to-image with paired data

Isola, Zhu, Zhou, Efros (2017) made the conditional GAN stunningly visual by treating one image as the conditioning input and another as the target. Sketches → photos. Maps → satellite views. Day → night. Their generator was a U-Net (encoder–decoder with skip connections); the discriminator was a "PatchGAN" that judged $70 \times 70$ patches independently rather than the whole image. The total loss combined a conditional GAN term with an L1 pixel loss:

The L1 term encourages the generator to be close to the target on average — this kills hallucination. The cGAN term insists the result be perceptually plausible — this kills the blur that pure L1 produces. The combination is more than the sum: pix2pix samples are simultaneously faithful and sharp, which neither loss alone can produce.

CycleGAN: image-to-image without pairs

Zhu, Park, Isola, Efros (2017) tackled the harder problem: what if you have unpaired source and target collections — a pile of horse photos and a pile of zebra photos, but no horse-zebra pairs? You can't compute an L1 loss because you don't know what each horse should become.

Their solution is conceptually beautiful: train two generators, $G: A \to B$ and $F: B \to A$, plus two discriminators $D_A, D_B$. The standard cGAN loss makes each generator produce samples that look like the target domain. The new ingredient is the cycle consistency loss: applying $G$ then $F$ should bring you back to where you started, and vice versa.

The cycle loss says: the translation must be invertible, which forces $G$ and $F$ to preserve content — it can't drop information that $F$ would need to reconstruct $x$. The result is a generator that changes style (horse texture → zebra stripes) while preserving content (the same pose, the same scene).

Text-to-image: the bridge to today

Conditioning generators on natural-language descriptions started early — Reed et al. (2016) trained a GAN on text-image pairs from the Caltech bird dataset and got recognizable bird sketches from sentences like "a small yellow bird with a black cap." The path from there to modern text-to-image runs through StackGAN, AttnGAN, and ultimately gets eclipsed by diffusion-based systems (DALL·E 2, Imagen, Stable Diffusion) because diffusion handles diverse, multi-modal text conditioning more gracefully.

But text-conditioning didn't disappear from GANs — GigaGAN (2023) showed that with enough scale, a GAN can match diffusion on text-to-image while running orders of magnitude faster at inference. The architectural lesson generalizes: conditioning information is best injected at multiple scales (via cross-attention, via AdaIN-like modulations, via concatenation early), and the discriminator should also be conditioned.

The ten percent that changes outcomes

If you went paper by paper through the GAN literature you'd find hundreds of suggested tricks. A small handful actually move the needle in modern setups. Here are the ones I keep reaching for, with the intuition behind each.

Spectral normalization

Miyato et al. (2018). For each weight matrix $W$ in the discriminator, divide by its largest singular value $\sigma(W)$ before using it in the forward pass. Result: every linear layer is 1-Lipschitz, every activation is at most 1-Lipschitz, and so the whole discriminator is 1-Lipschitz. The largest singular value can be approximated cheaply with a single power iteration per step.

This gives you the Lipschitz property that WGAN-GP enforces with a gradient penalty, but for free: no second backward pass, no $\lambda$ to tune, drop it in and go. Spectral normalization in $D$ + hinge loss + non-saturating $G$ loss is an extremely robust default.

Two-time-scale update rule (TTUR)

Heusel et al. (2017). Use different learning rates for $G$ and $D$ — typically $D$ runs faster. A common setting is $\alpha_G = 10^{-4}$, $\alpha_D = 4 \times 10^{-4}$. The intuition: the discriminator is the slower mover in saddle-point dynamics, so giving it a faster learning rate helps the system actually converge instead of orbit. TTUR comes with a convergence proof under reasonable assumptions; in practice it's a free win on most setups.

Adam $\beta_1 = 0.0$ or $0.5$

Standard Adam uses $\beta_1 = 0.9$, which heavily smooths the gradient via momentum. Momentum is a great idea when you're descending into a basin. It is a terrible idea when you're trying to converge to a saddle point — momentum will carry you past the equilibrium and amplify the rotational dynamics. Setting $\beta_1$ low (0 to 0.5 is typical for GANs) damps this. A small fix that matters more than people expect.

EMA of generator weights for evaluation

Maintain an exponentially moving average of $G$'s weights with a decay around 0.999 or 0.9999. At evaluation time, use the EMA weights, not the raw ones. Karras et al. popularized this in StyleGAN, and it makes a measurable FID difference. The intuition: even a converged GAN is technically still oscillating around its equilibrium; averaging across recent steps lands you closer to the actual equilibrium.

Truncation trick

Brock et al. (BigGAN). At inference time, sample $z$ from a truncated normal — clip values that are far from the mean — to trade off diversity for quality. With truncation $\psi = 1.0$ you get the full diversity of the model; with $\psi = 0.5$ you get more conservative samples that are closer to the "average" face but more reliably high-quality. A free post-hoc knob with no retraining required.

Style mixing regularization

StyleGAN-specific. During training, occasionally use two different $w$ codes for different layers of the synthesis network. This prevents the generator from learning that adjacent style codes must be correlated, which forces a more disentangled style space. At inference, you can then mix styles cleanly across different abstractions.

Differentiable augmentation

Karras et al. (ADA, 2020). Apply data augmentation to both real and fake samples before showing them to $D$, with the augmentation being differentiable so that gradients can flow back through it to $G$. This dramatically reduces the data requirements — StyleGAN2-ADA can train on a few thousand images where StyleGAN2 needed tens of thousands. Essential for fine-tuning or training on limited datasets.

Path length regularization

StyleGAN2's other contribution. Penalize sudden changes in the generator's output as you walk through latent space:

where $y \sim \mathcal{N}(0, I)$ is a random image-space direction, $J_G$ is the Jacobian of $G$ with respect to $w$, and $a$ is the running average of $\|J_G^T y\|_2$. This forces directions in $w$-space to map to similar-magnitude directions in image space — making the latent space's geometry more uniform and editing more predictable.

Evaluating a model whose loss doesn't tell you anything

Here is an awkward fact: a GAN's training loss does not correlate well with sample quality. The generator and discriminator are at war; whichever side is "winning" at any moment shifts the loss. You can't use validation loss for early stopping. You can't compare two GANs by their loss curves. You need external metrics, and the field has spent a lot of effort building them.

Inception Score (IS)

Salimans et al. (2016). Run generated samples through a pretrained Inception classifier and look at two things: the per-sample class distribution should be peaky (the classifier should be confident — sharp images), and the marginal class distribution should be uniform (samples should cover all classes — diversity). Combine into:

Higher is better. Real ImageNet has IS around 250. The metric is a clever way to score a generator without needing the data distribution itself, but it has known failure modes: it's blind to within-class mode collapse, and it can be gamed by a generator that hits the Inception network's blind spots. Don't use it as your only metric.

Fréchet Inception Distance (FID)

Heusel et al. (2017). The current default. Pass both real and fake samples through the Inception network, take the activations from a chosen layer (typically the pool3 layer, 2048-dim features), and fit a multivariate Gaussian to each set. Then measure the distance between the two Gaussians:

Lower is better. FID captures both per-sample quality (matching means) and diversity (matching covariances). It correlates with human judgments far better than IS does, and it's symmetric in real and fake — a bias toward Inception-friendly features still exists, but it's much weaker than IS.

Kernel Inception Distance (KID)

A nonparametric alternative to FID. Same Inception features, but instead of fitting Gaussians, compute a kernel-based discrepancy (Maximum Mean Discrepancy with a polynomial kernel). KID is unbiased — it doesn't systematically improve as you give it more samples — which makes small-sample-size comparisons fairer. Some labs prefer KID for that reason; FID is still the dominant report metric.

Precision and Recall for distributions

Sajjadi et al. (2018), refined by Kynkäänniemi et al. (2019). FID conflates two things: are the samples good (precision) and does the generator cover the data distribution (recall). A mode-collapsed generator can have decent FID if its few modes are very high-quality. Precision/Recall metrics decompose this. They build a manifold approximation around real samples, around fake samples, and ask: what fraction of fakes lie in the real manifold (precision), and what fraction of reals lie in the fake manifold (recall)?

This decomposition is more diagnostic. Diffusion models tend to have high recall but moderate precision. GANs tend to have high precision but moderate recall (mode coverage is their weak point). Reporting both lets you see where your model wins and loses.

Perceptual Path Length (PPL)

Karras et al. (StyleGAN). Walk through the latent space along short paths and measure how much each tiny step changes the LPIPS perceptual distance of the output. A "smooth" generator has low PPL: small latent changes produce small perceptual changes. A "tangled" generator has high PPL: tiny latent steps can produce different-looking faces. Lower PPL correlates with cleaner editing, better disentanglement, and sometimes better downstream FID.

Human evaluation

Still the gold standard for sample quality, especially in conditional generation where automated metrics struggle. Set up two-alternative forced choice ("which of these two images is real?") with controlled stimulus exposure times; use enough raters to get tight confidence intervals. Time-limited human eval is the only metric where a generator that produces 1990s-quality CGI gets the score it deserves.

What they still win, what they have lost

If you started reading the generative modeling literature in 2024 and didn't know better, you might think GANs were a defunct paradigm. Most attention has moved to diffusion. Most new image-generation papers in flagship venues are diffusion-flavored. Most products consumers see are downstream of Stable Diffusion or its descendants. So: are GANs over?

No. They've changed shape. Here's the lay of the land in 2026.

What diffusion took from GANs

• Unconditional image generation at the largest scales. Diffusion models with classifier-free guidance produce more diverse samples, cover more modes, and are easier to scale. If you're starting fresh with a billion images and want to model their distribution, diffusion is the safer bet.
• Text-to-image at high quality. Diffusion handles multi-modal text conditioning more gracefully than any GAN architecture. The CFG trick — sample at a stronger conditioning weight than was used during training — has no clean GAN analog.
• Likelihood-y workloads. Diffusion models can be massaged into providing approximate likelihoods (via the score matching identity), which matters for compression, anomaly detection, and statistical inference. GANs have no likelihood at all.

What GANs still win

• Inference speed. A GAN is a single forward pass. A diffusion model needs anywhere from 4 to 1000 forward passes per sample. For real-time applications — interactive editing, video generation, embedded devices — GANs are still orders of magnitude faster, and that gap doesn't close.
• Image-to-image translation. CycleGAN-style unpaired translation is still the dominant approach in many practical pipelines (medical imaging, remote sensing, neural rendering). The cycle-consistency idea generalizes well to settings where diffusion's iterative refinement is overkill.
• Super-resolution. Real-ESRGAN and SRGAN-style architectures remain widely used because they're fast, predictable, and produce sharp output. Diffusion-based super-res is starting to catch up but is much more expensive.
• Audio generation. HiFi-GAN, MelGAN, and BigVGAN are the workhorses of fast neural vocoding. Diffusion vocoders exist but are slower; for streaming TTS the GAN is still on top.
• Latent editing and disentanglement. StyleGAN's $w$-space remains the cleanest editable latent space anyone has built. Inverting an image into $w$ and then walking in interpretable directions is a workflow diffusion has not really matched.

The hybrid story: GANs inside diffusion

The most consequential recent development isn't "GAN vs. diffusion" — it's "GAN inside diffusion." Multi-step diffusion models have been distilled into one-step (or few-step) generators using adversarial distillation: a teacher diffusion model produces the targets, and a student is trained with an adversarial loss to match them in a single pass. Stability AI's SD-Turbo, Adobe's models, and many of the fast-inference systems shipping today work this way. The GAN's adversarial machinery is now a tool used to compress diffusion models, not a competitor to them.

So the question "is X a GAN or a diffusion model?" is becoming less meaningful. The right question is: what's the training objective, and what's the inference cost?

Watching the dance on a 2-D dataset

The pathologies of §8 and the loss-function arithmetic of §9 become much more vivid when you can see them in action. Below is a tiny GAN, in the browser, training on a 2-D toy distribution. The "real" data is a mixture of eight Gaussians arranged on a ring (a classic mode-collapse stress test). The generator is a small MLP from a 2-D noise vector to a 2-D point; the discriminator is a small MLP from a 2-D point to a single scalar. Watch the green real samples and the orange generated samples as training progresses. Try changing the loss function and see how mode coverage changes.

Things to try: start with hinge loss and watch how all 8 modes light up over a few thousand steps — that's a healthy GAN. Switch to non-saturating BCE and you'll often see mode collapse, especially if you crank D's learning rate. Try $D$ updates per $G$ update = 5 — observe how diversity changes when the discriminator races ahead. Reset and try LSGAN; you'll find it more stable than BCE but still vulnerable to collapse on this dataset.

What you should walk away with

The big ideas, in one paragraph each

The forger-detective game. A GAN trains a generator and a discriminator against each other. The generator wants its samples to fool the discriminator; the discriminator wants to tell real from fake. The shared loss is a single function that one player minimizes and the other maximizes. The equilibrium of this game — when neither side can improve — is exactly the point where the generator's distribution matches the data.

The math justifies the trick. For a fixed generator, the optimal discriminator is the density-ratio function $D^* = p_{\text{data}} / (p_{\text{data}} + p_g)$. Substituting back, the generator is minimizing the Jensen–Shannon divergence to the data, and the global optimum is $p_g = p_{\text{data}}$. The non-saturating reformulation gives the generator stronger gradients early in training without changing the equilibrium.

The training is the hard part. Saddle-point dynamics rotate; alternating gradient descent doesn't converge naively. Mode collapse, vanishing gradients, and oscillation are intrinsic, not bugs. Most GAN engineering is about damping these dynamics: better losses (Wasserstein, hinge), Lipschitz constraints (spectral norm), separated learning rates (TTUR), weight averaging (EMA), gradient penalties.

Conditioning extends the framework cleanly. Add a label, sketch, or text embedding to both networks; you have a conditional GAN. Pair it with a reconstruction loss for paired data (pix2pix), or a cycle-consistency loss for unpaired data (CycleGAN), and you get image-to-image translation. The same minimax game scaffolds everything.

Evaluation requires external metrics. Loss curves don't tell you anything about sample quality. FID is the current standard; IS is older and weaker; precision/recall decompose quality and coverage; PPL measures latent smoothness. Always report multiple metrics, always specify the exact computation pipeline.

GANs are not over. Diffusion has displaced them for the largest-scale unconditional and text-to-image tasks. But GANs still own the speed-critical applications, image-to-image translation, audio vocoding, latent editing, and increasingly, the distillation step that makes diffusion models fast.

Glossary of essential terms

Generator $G$

Neural network that maps a latent noise vector $z$ to a sample in data space.

Discriminator $D$

Neural network that maps a sample to a scalar score representing how real it appears. In WGAN-style losses, sometimes called a "critic."

Latent space

The input space of $G$, usually a simple distribution like $\mathcal{N}(0, I)$ in $\mathbb{R}^{100}$.

Minimax objective

The shared loss function $V(D, G)$ that $D$ maximizes and $G$ minimizes.

Saddle point / Nash equilibrium

A point where no player can improve by unilaterally changing their parameters. The training target.

Mode collapse

The pathology where $G$ produces only a small number of distinct outputs.

Lipschitz continuity

The property that a function's output changes by at most $K$ times its input change. Used to constrain the discriminator in Wasserstein-style training.

Spectral normalization

Dividing each weight matrix by its largest singular value to enforce a Lipschitz bound on the discriminator.

Two-time-scale update (TTUR)

Using different learning rates for $G$ and $D$ to stabilize training.

FID

Fréchet Inception Distance: the Wasserstein-2 distance between Gaussian fits to Inception activations of real and fake samples. The standard quality metric.

StyleGAN

An influential generator architecture that splits latent into a mapping network producing $w$ and a synthesis network modulated by $w$ at every layer via AdaIN.

CycleGAN

An unpaired image-to-image translation method using two generators, two discriminators, and a cycle-consistency loss.

Exercises

1. Derivation. Starting from $C(G) = V(D^*, G)$, show that $C(G) = -\log 4 + 2 \cdot \operatorname{JSD}(p_{\text{data}} \| p_g)$. (Hint: expand $D^*$, factor out $\tfrac{1}{2}$, recognize KL divergences.)
2. Saturation. Compute $\partial \mathcal{L}_G / \partial D$ for both the saturating and non-saturating losses at $D = 0.01$. Verify the ratio is approximately $-(1-D)/D \approx -100$. Explain why this matters at the start of training.
3. Wasserstein. Two distributions are point masses at $0$ and at $\theta$ on the real line. Compute $\operatorname{KL}$, $\operatorname{JSD}$, and $W_1$ as functions of $\theta$. Plot all three. Argue from your plot why WGAN gives gradient signal in regimes where JSD-based losses don't.
4. Mode collapse experiment. Modify the playground above (or use any GAN library) to train on the 8-Gaussian ring with the saturating BCE loss and Adam $\beta_1 = 0.9$. Demonstrate mode collapse. Then change to the non-saturating loss with $\beta_1 = 0.5$ and demonstrate that mode coverage improves.
5. Spectral norm. Show that if every layer of $D$ is 1-Lipschitz, then $D$ as a whole is 1-Lipschitz. Why does this argument require LeakyReLU rather than ReLU? (Hint: think about gradients at zero.)
6. Conditional GAN design. Sketch a generator and discriminator architecture for a conditional GAN that produces a 256×256 face image given a text description. Decide where in $G$ the text embedding enters and why; decide whether $D$ is conditioned on text and why.
7. FID computation. Implement FID for two distributions of 2048-dim Gaussians (you can use random covariance matrices). Verify that FID is symmetric and that it equals zero when the two distributions are identical.
8. CycleGAN reasoning. Argue that the cycle-consistency loss alone, without the adversarial loss, would cause the two generators to learn the identity function. Argue conversely that the adversarial loss alone, without the cycle loss, would let the generators output anything plausible-looking in the target domain — including outputs unrelated to the input. Why are both necessary?

Further reading

• Goodfellow et al. (2014), "Generative Adversarial Nets." The original four-page paper. Read it slowly; the proof is elegant.
• Radford, Metz, Chintala (2015), "Unsupervised Representation Learning with Deep Convolutional GANs." DCGAN — the architectural recipe that made GANs trainable.
• Arjovsky, Chintala, Bottou (2017), "Wasserstein GAN." The clearest exposition of why JSD-based GANs fail and what to do about it.
• Gulrajani et al. (2017), "Improved Training of Wasserstein GANs." The gradient penalty that made WGAN practical.
• Karras et al. (2019), "A Style-Based Generator Architecture for Generative Adversarial Networks." StyleGAN; one of the most influential generator designs.
• Karras et al. (2020), "Analyzing and Improving the Image Quality of StyleGAN." StyleGAN2 with weight demodulation, path-length regularization, and other refinements.
• Sauer et al. (2023), "GigaGAN: Large-Scale GAN for Text-to-Image Synthesis." A reminder that GANs scale.
• Nichol & Dhariwal (2021), "Improved Denoising Diffusion Probabilistic Models." Useful for understanding why diffusion displaced GANs in some settings.
• Sajjadi et al. (2018), "Assessing Generative Models via Precision and Recall." For thinking carefully about evaluation.

If you're reading these in order, the cleanest path through the literature is: original GAN → DCGAN → WGAN → WGAN-GP → spectral norm → StyleGAN → StyleGAN2 → BigGAN → CycleGAN. That's about a dozen papers and they tell the whole story. Skip everything else on a first pass; come back when you have a specific reason.