Fourth Foundations post in ML & AI, and the one that ties the other three together. Tokenization covered turning text into discrete units, How the Transformer Actually Works covered turning those units into contextual representations, and Vision Transformers covered doing the same thing to image patches. All three posts trained a model to represent one modality well. CLIP (Radford et al., 2021) asks a different question: what if you trained a text encoder and an image encoder at the same time, with the only goal being that a caption and its matching image end up close together in the same vector space?

Two towers, one shared space

CLIP is two separate encoders — an image encoder (a ResNet or a ViT, the paper tries both) and a text encoder (a 12-layer, 512-wide Transformer with 8 attention heads, 63M parameters, using the same byte-pair-encoding idea from the tokenization post with a 49,152-token vocabulary) — with no interaction between them until the very end. Each tower reduces its input to a single vector: an image becomes one embedding, a caption becomes one embedding. Both vectors are then projected into the same dimensionality and L2-normalized, so a plain cosine similarity between an image embedding and a text embedding is a meaningful number — 1 for identical direction, 0 for unrelated.

Diagram showing separate image and text encoder towers each producing an embedding vector, projected into a shared space, with an N-by-N similarity matrix below where the diagonal is highlighted as the correct pairs the training objective pulls together

Nothing here is architecturally new — it's the same encoder from the Transformer post, and the same patch-embedding trick from the ViT post. The entire contribution is the training objective that follows.

Contrastive pretraining

Take a batch of N (image, text) pairs scraped from the internet — CLIP's training set, called WIT, is 400 million such pairs. Encode all N images and all N captions, producing N image embeddings and N text embeddings. Compute the cosine similarity between every image embedding and every text embedding, giving an N×N matrix of scores. Exactly N of those N² pairs are correct (image i really does go with caption i); the rest are mismatched.

The training objective is a symmetric cross-entropy loss over that matrix: treat each row as a classification problem ("given this image, which of the N captions is the real one?") and each column as the mirror problem ("given this caption, which of the N images is the real one?"), and push the model to make the diagonal entries the highest value in both their row and their column. There's a learnable temperature parameter scaling the similarities before the softmax (initialized to the equivalent of 0.07 and clipped to prevent scaling by more than 100), controlling how sharply the model is pushed to separate correct pairs from incorrect ones.

Batch size matters more here than in most training setups, because the number of negative examples in the loss is the batch size minus one — every other caption in the batch acts as a negative for a given image. CLIP uses a very large minibatch of 32,768 pairs, which means every image is being contrasted against 32,767 wrong captions on every training step. The largest model in the paper (ViT-L/14) trained for 12 days on 256 V100 GPUs.

Zero-shot classification falls out for free

This is the part that made CLIP notable beyond being a good retrieval model. Once training is done, image classification on a completely new dataset needs no fine-tuning, no labeled examples from that dataset at all: turn each class label into a sentence — the paper's default template is "A photo of a {label}." — encode all of those sentences with the text tower, encode the image with the image tower, and pick whichever label's text embedding has the highest cosine similarity to the image embedding. Classification becomes a nearest-neighbor lookup in embedding space, using labels the model has never been trained to predict as a fixed output layer.

Two details the paper found mattered in practice:

  • The wording of the template matters. Just writing the bare class name as the caption underperforms "A photo of a {label}." by about 1.3 percentage points on ImageNet — the model was trained on real internet captions, which are sentences, not bare words, so giving it a sentence-shaped input at test time matches what it saw during training.
  • Ensembling templates helps more. Averaging the text embeddings produced by 80 different prompt templates ("a photo of a big {label}", "a black and white photo of a {label}", and so on) before comparing to the image gives a further 3.5 point improvement over the single best template.

With this setup, CLIP's best model matches the accuracy of a fully-supervised ResNet-50 on ImageNet — 76.2% top-1 — without using any of ImageNet's 1.28 million labeled training images. All the supervision came from captions written for entirely different purposes by people on the internet, never from anyone labeling images with ImageNet's specific 1,000 class names.

Why this is the right place to end the trio

Put next to the last two posts, the pattern is direct: tokenization decided what a "unit" of text is, the Transformer post described how to turn a sequence of units into a contextual representation, and the ViT post showed the same encoder works on image patches with no architecture change. CLIP's only remaining move is to point out that if both modalities produce a single vector, there was never a reason to keep their vector spaces separate — train them jointly against each other, and a huge amount of what used to require labeled data (image classification, image search by text query, guiding image generation toward a text prompt) becomes a similarity lookup instead. Most text-to-image systems built after 2021, including the CLIP guidance used in early diffusion models, lean directly on this shared space.

References

Contrastive-training diagram above is original artwork made for this post.