Multimodal

Making models see, hear, and read

For most of deep learning's history, models were unimodal: a language model processes text, an image classifier processes pixels, a speech recognizer processes audio. Multimodal models bridge these domains, learning shared representations that connect language, vision, audio, and more — enabling generation and understanding across modalities.

The core challenge: different modalities have fundamentally different structures. Text is discrete and sequential. Images are continuous and spatial. Audio is continuous and temporal. The model must learn that the word "cat," a photo of a cat, and the sound of a meow all refer to the same concept.

CLIP — contrastive vision-language

CLIP (Radford et al., 2021) learns a shared embedding space for images and text via contrastive learning: given a batch of image-caption pairs, train an image encoder and a text encoder so that matching pairs have high cosine similarity and non-matching pairs have low similarity.

Zero-shot classification: encode an image and a set of text labels ("a photo of a cat", "a photo of a dog"), predict the class with maximum cosine similarity. No task-specific training needed. CLIP matched supervised ResNet-50 on ImageNet — without ever seeing ImageNet labels during training.

Text-to-image — from DALL-E to Stable Diffusion

The text-to-image pipeline combines a text encoder (CLIP or T5) with an image generator (diffusion or flow model). The text embedding is injected into the denoising process via cross-attention — exactly the encoder-decoder mechanism from Chapter 08.

Image-to-text — visual understanding

The reverse: given an image, generate a description or answer questions. LLaVA (Liu et al., 2023) showed a simple approach: take a pretrained vision encoder (CLIP ViT), project its output into the token space of a pretrained LLM (LLaMA), and fine-tune on visual instruction data. The LLM treats image tokens the same as text tokens — no architectural changes needed.

GPT-4V, Claude, Gemini, and other frontier models now handle images natively. Visual understanding has become a standard capability, not a separate research area.

The convergence thesis — tokenize everything

The field is converging on a unified framework: tokenize everything, then model the tokens.

• Text is already tokens (BPE, SentencePiece).
• Images are tokenized via VQ-VAE, patch embeddings, or VAE latents.
• Audio is tokenized via EnCodec, SoundStream, or Whisper features.
• Video is tokenized frame by frame (spatial-temporal patches).

Once everything is tokens, a single transformer can model any modality and any cross-modal relationship. The modality-specific encoders handle compression; the transformer handles reasoning and generation.

Audio, video, and 3D

• Audio generation (MusicGen, Stable Audio): VQ-VAE tokenization of audio + autoregressive or diffusion generation. Text-to-speech, text-to-music, and audio editing use this framework.
• Video generation (Sora, Kling, Runway): extends image diffusion to the temporal dimension with 3D attention (spatial + temporal). The core challenge: temporal consistency across frames.
• 3D generation (DreamFusion, Zero-1-to-3): uses 2D diffusion priors to optimize 3D representations via score distillation sampling — a diffusion model guides 3D construction without 3D training data.

Building Mini-CLIP from scratch

Architecture

The notebook implements a complete CLIP system: a CNN image encoder and a transformer text encoder, both projecting into a shared 128-dimensional embedding space. Trained on MNIST images paired with text labels ("zero" through "nine") using InfoNCE contrastive loss.

Shared embedding space

t-SNE visualization of the trained embeddings shows that images and text of the same digit class cluster together in the shared space — a "3" image is close to the text "three" and far from "seven." The two modalities overlap, demonstrating that the contrastive objective has learned a genuinely shared semantic space.

Zero-shot classification

Using Mini-CLIP for zero-shot classification (compare image embedding to all 10 text embeddings, predict the closest) achieves ~85% accuracy on MNIST — without ever training a classifier. The model learned to match images to text labels purely from contrastive training. Per-class accuracy reveals that some digits (1, 0) are easier to distinguish than others (3, 5, 8) where visual similarity causes confusion.

Cross-modal retrieval

Given the text query "three," retrieve the top-5 most similar images from the test set — they're all 3s. Given an image of a 7, retrieve the most similar text — it's "seven." The embedding space supports bidirectional retrieval across modalities.

N×N similarity matrix

A 10×10 similarity matrix (one image per digit vs. all 10 text labels) shows a striking pattern: the diagonal is bright (high similarity for correct matches) and the off-diagonal is dark (low similarity for incorrect matches). The few bright off-diagonal cells reveal which digits the model confuses (e.g., "3" and "eight" have slightly elevated similarity because 3 and 8 share visual features).

CLIP similarity matrix

A simulated 8×8 cross-modal similarity matrix: images (rows) vs text labels (columns). Diagonal entries are high (correct matches); off-diagonal entries reveal which classes the model confuses. Click "shuffle" to simulate different training runs.

Takeaways & exercises

1. CLIP connects vision and language through contrastive learning in a shared embedding space — the foundation for text-to-image and zero-shot classification.
2. Text-to-image generation combines text encoders (CLIP/T5) with diffusion/flow generators — cross-attention is the bridge (same mechanism as Chapter 08).
3. Image-to-text (LLaVA approach): project visual features into a language model's token space. Simple, effective, now standard in frontier models.
4. The convergence thesis: tokenize everything — once all modalities are tokens, one transformer handles them all.
5. Contrastive learning doesn't generate — it creates the shared space that makes generation possible when combined with other models.

Further reading

• Radford et al. (2021) — CLIP: Learning Transferable Visual Models.
• Ramesh et al. (2021/2022) — DALL-E and DALL-E 2.
• Rombach et al. (2022) — Latent Diffusion / Stable Diffusion.
• Liu et al. (2023) — LLaVA: Visual Instruction Tuning.
• Brooks et al. (2024) — Sora: Video Generation.
• Défossez et al. (2022) — EnCodec: Audio Neural Codec.
• P. Kulkarni — Module 09 notebook. Mini-CLIP from scratch.