Encoder-Decoder

Input a sequence, output a different one

Some tasks are fundamentally asymmetric: the input is a complete, known sequence and the output is a different sequence that must be generated token by token. Translation (English → French), summarization (article → abstract), speech-to-text (audio → words). The encoder-decoder architecture handles this by splitting the model into two halves connected by a bridge.

From bottleneck to attention

The original RNN seq2seq (Sutskever et al., 2014) compressed the entire input into a single fixed-size vector — an information bottleneck. For long sequences, critical details were lost.

Bahdanau attention (2015) solved this: instead of one context vector, the decoder attends to all encoder hidden states at each generation step. The attention weights $\alpha_{t,i}$ learned soft alignment — which input tokens are relevant for each output token. This was the conceptual precursor to the transformer's cross-attention mechanism.

Cross-attention — the bridge

In self-attention (Chapter 02), Q, K, V all come from the same sequence. In cross-attention, Q comes from the decoder but K and V come from the encoder:

For translation, cross-attention weights often trace a roughly diagonal pattern (monotonic alignment), but for summarization they can be highly non-monotonic — the decoder might jump across the input to gather scattered key facts.

The decoder block — three sub-layers

A decoder block in an encoder-decoder transformer has three sub-layers (vs two in a decoder-only model):

1. Causal self-attention — the decoder attends to its own previously generated tokens (masked, same as GPT).
2. Cross-attention — the decoder attends to the encoder's output (the bridge).
3. MLP — the standard feed-forward expansion/compression.

Each sub-layer has its own residual connection and layer norm: $h' = \text{LN}(\text{SubLayer}(h) + h)$. The encoder block is simpler — just self-attention (bidirectional, no mask) + MLP.

Text-to-text — one format for everything

T5 (Raffel et al., 2020) unified all NLP as text-to-text: every task is "given this text, produce that text." Translation: "translate English to French: The cat" → "Le chat". Summarization: "summarize: [article]" → "[summary]". Classification: "sentiment: I loved it" → "positive".

One model, one loss, one pipeline. T5 was pre-trained with span corruption: random text spans are replaced with sentinel tokens, and the model fills them in. This denoising objective teaches both understanding (encoder) and generation (decoder).

BART — denoising seq2seq

BART (Lewis et al., 2019) uses a more aggressive denoising objective: tokens deleted, shuffled, masked, or replaced. The encoder sees the corrupted text; the decoder reconstructs the original. This teaches the model to handle many types of noise, making it especially strong for generation tasks like summarization and dialogue.

Vision encoder-decoders — Whisper, captioning

• Whisper (Radford et al., 2023): encoder processes audio spectrogram features; decoder generates text transcription. An encoder-decoder that bridges modalities.
• Image captioning: a vision encoder (ViT/CLIP) produces image representations; a text decoder generates captions, using cross-attention to attend to image regions.
• Pix2Seq: frames object detection as seq2seq — the encoder sees the image, the decoder outputs bounding box coordinates as token sequences.

When to use encoder-decoder vs decoder-only

Since 2022, the trend has been strongly toward decoder-only (GPT-4, Claude, LLaMA, Gemini). But encoder-decoder has structural advantages when input and output are fundamentally different:

• Translation: bidirectional encoding of source + causal generation in target.
• Speech recognition: audio encoder + text decoder (Whisper).
• Summarization: full-document understanding + condensed generation.
• Image captioning: vision encoder + text decoder.

Decoder-only models handle these by concatenating input and output, but encoder-decoder can be more parameter-efficient because the two halves specialize. The encoder doesn't need causal masking overhead; the decoder doesn't need to process the full input sequence at every layer.

Number-to-words translation

Cross-attention reveals alignment

The star visualization: the cross-attention heatmap between encoder positions (input digits) and decoder positions (output words). For a well-trained model, the heatmap shows a clean diagonal pattern — the decoder attends to digit "1" when generating "one," to "2" when generating "two," and so on. The model has learned monotonic left-to-right alignment from data alone, without being told that digits map to words in order.

Head specialization

Different cross-attention heads specialize: some show sharp diagonal alignment (one-to-one mapping), while others attend more broadly (gathering context about sequence length or digit identity from multiple positions). This division of labor emerges naturally from training.

Beam search vs greedy

Beam search ($B=3$) slightly outperforms greedy decoding on this task — it recovers from occasional early mistakes by maintaining alternative hypotheses. The accuracy difference is small (~2%) because the task is simple, but the principle matters for harder tasks like translation where greedy mistakes compound.

Length generalization

Testing on digit sequences longer than those seen in training reveals a generalization gap: the model handles 5-digit numbers perfectly (seen in training) but degrades on 7-digit numbers. This is a known limitation of transformer encoder-decoders — positional encodings and attention patterns don't always extrapolate cleanly beyond training lengths.

See cross-attention in action

A simulated cross-attention heatmap for number-to-words translation. Each cell shows how strongly decoder position $t$ (output word) attends to encoder position $i$ (input digit). Click different examples to see how the alignment pattern changes.

Takeaways & exercises

1. Encoder-decoder separates understanding from generation — bidirectional encoder + causal decoder, connected by cross-attention.
2. Cross-attention is the bridge — Q from decoder, K/V from encoder, learning soft alignment.
3. T5 unified NLP as text-to-text; BART used denoising pre-training.
4. Decoder-only dominates for general LLMs, but encoder-decoder remains best for tasks where input and output are fundamentally different.
5. Cross-attention weights are interpretable — they reveal learned alignment between input and output positions.

Further reading

• Vaswani et al. (2017) — Attention Is All You Need. The original encoder-decoder transformer.
• Bahdanau et al. (2015) — Attention for neural machine translation.
• Raffel et al. (2020) — T5. Text-to-text everything.
• Lewis et al. (2019) — BART. Denoising seq2seq.
• Radford et al. (2023) — Whisper. Speech encoder-decoder.
• P. Kulkarni — Module 08 notebook.