📏 Contrastive Learning: From Metric Learning to InfoNCE and SimCLR

A historical account of how pairwise margin losses became an information-theoretic objective

Table of Contents

• 🗺️ The Metric Learning Framing
• ⚖️ The Contrastive Loss (Hadsell et al., 2006)
• 🔺 The Triplet Loss (Schroff et al., 2015)
• 🔢 From Pairs to N-Pairs (Sohn, 2016)
• 📡 InfoNCE and Contrastive Predictive Coding (van den Oord et al., 2018)
  • 🔗 From NCE to InfoNCE: Statistical Roots
  • 🔑 The Categorical Cross-Entropy Derivation
  • 📐 The Mutual Information Lower Bound
  • 🌡️ The Role of the Critic
• 🖼️ Instance Discrimination: The Vision Bridge (Wu et al., 2018)
• 🧪 SimCLR: InfoNCE for Vision (Chen et al., 2020)
  • 🔭 Augmentation Strategy
  • 📐 NT-Xent as an InfoNCE Instance
  • 🔬 The Projector Head
  • 🌡️ Temperature and Hard Negatives
  • 🌐 Alignment and Uniformity on the Hypersphere
  • 🏋️ Semi-Supervised Learning and Scaling
• 🧭 Retrospective: The Historical Arc
• 📚 References

🗺️ The Metric Learning Framing

Contrastive learning is a family of representation learning methods unified by a single geometric goal: learn an embedding \(f_\theta : \mathcal{X} \to \mathbb{R}^d\) such that semantically similar inputs are mapped to nearby points and semantically dissimilar inputs are mapped to distant points.

Formally, let \(\mathcal{X}\) be the input space and let \(\sim\) denote a semantic equivalence (e.g. “same class”, “same image under augmentation”). We seek:

\[f_\theta(x) \approx f_\theta(x') \quad \text{whenever } x \sim x'\] \[\|f_\theta(x) - f_\theta(x'')\| \gg 0 \quad \text{whenever } x \not\sim x''.\]

In metric learning, similarity is defined by labels — pairs \((x, x^+)\) share a class, pairs \((x, x^-)\) do not. In self-supervised contrastive learning, similarity is defined by augmentation invariance — \((x, x^+)\) are two views of the same underlying image.

The history of contrastive learning traces a 15-year arc: from margin-based pairwise losses (2006) through triplets (2015), through the statistical mechanics of noise contrastive estimation and its generalization to mutual information bounds (2018), through the first vision applications via instance discrimination (2018), and culminating in SimCLR’s clean operationalization for large-scale vision (2020).

⚖️ The Contrastive Loss (Hadsell et al., 2006)

📐 Hadsell, Chopra, and LeCun introduced the original contrastive loss in the context of dimensionality reduction by learning an invariant mapping — a siamese network architecture that takes pairs of inputs and pushes them together or apart.

Setup. A siamese network \(f_\theta\) shares parameters across two branches. Given a labeled pair \((x_1, x_2)\) with label \(y \in \{0, 1\}\) (\(y = 0\) = same class / similar; \(y = 1\) = different class / dissimilar), define:

\[D_W = \|f_\theta(x_1) - f_\theta(x_2)\|_2.\]

Definition (Contrastive Loss).

\[\mathcal{L}_{\text{con}}(\theta;\, x_1, x_2, y) = (1 - y)\,\frac{D_W^2}{2} + y\, \frac{\max(0,\, m - D_W)^2}{2}\]

where \(m > 0\) is a margin hyperparameter.

Mechanics of each term:

• \((1-y) D_W^2 / 2\): when \(y = 0\) (similar pair), the loss is the squared distance — minimized by pulling the representations together.
• \(y \max(0, m - D_W)^2 / 2\): when \(y = 1\) (dissimilar pair), the loss is a hinge on the distance — penalized only if \(D_W < m\), i.e. if the dissimilar representations are too close. Once they are farther than \(m\) apart, the gradient vanishes.

The hinge structure is deliberate: pushing all dissimilar pairs infinitely far apart would dominate training and destabilize the embedding. The margin \(m\) creates a dead zone — dissimilar pairs farther than \(m\) contribute no gradient.

🔺 The Triplet Loss (Schroff et al., 2015)

The contrastive loss operates on pairs. A key limitation: the margin \(m\) is specified in absolute distance units, disconnected from the geometry of the local embedding neighborhood. The triplet loss, introduced by Schroff, Kalenichenko, and Philbin in FaceNet (2015), shifts from absolute distances to relative ordering.

Setup. A triplet consists of: - Anchor \(a\): the reference image - Positive \(p\): an image similar to \(a\) (same identity / class) - Negative \(n\): an image dissimilar to \(a\)

The goal is to ensure the anchor is closer to the positive than to any negative by at least a margin \(\alpha > 0\):

\[\|f_\theta(a) - f_\theta(p)\|_2^2 + \alpha < \|f_\theta(a) - f_\theta(n)\|_2^2.\]

Definition (Triplet Loss).

\[\mathcal{L}_{\text{trip}}(\theta;\, a, p, n) = \max\!\Bigl(0,\; \|f_\theta(a) - f_\theta(p)\|_2^2 - \|f_\theta(a) - f_\theta(n)\|_2^2 + \alpha\Bigr).\]

The loss is zero (no gradient) when the positive is already farther from the anchor than the negative is, by margin \(\alpha\). Otherwise, it penalizes the gap between positive and negative distances.

🔑 Triplet Mining

With \(N\) training examples, there are \(O(N^3)\) triplets. Most are easy — the positive is already much closer than the negative and the loss is zero. Training on easy triplets provides no gradient.

Hard negatives are negatives \(n\) where \(\|f_\theta(a) - f_\theta(n)\|^2 < \|f_\theta(a) - f_\theta(p)\|^2\) — the negative is closer to the anchor than the positive is. These are the most informative for learning.

FaceNet introduced semi-hard mining: select negatives where

\[\|f_\theta(a) - f_\theta(p)\|_2^2 < \|f_\theta(a) - f_\theta(n)\|_2^2 < \|f_\theta(a) - f_\theta(p)\|_2^2 + \alpha,\]

i.e. negatives that are farther than the positive but within the margin. These are neither trivial nor so hard they destabilize training.

🔢 From Pairs to N-Pairs (Sohn, 2016)

A structural limitation of both the contrastive and triplet losses is their local view: each optimization step updates parameters based on a single pair or triplet. Given a fixed anchor \(a\), we get one gradient signal — either from one positive or one negative.

Sohn (2016) generalized the triplet loss to the N-pairs loss, which simultaneously contrasts a single positive against \(N-1\) negatives in a single step:

Definition (N-pairs Loss). Given an anchor \(x\), a positive \(x^+\), and \(N-1\) negatives \(\{x_1^-, \ldots, x_{N-1}^-\}\), with embeddings \(z = f_\theta(x)\), \(z^+ = f_\theta(x^+)\), \(z_i^- = f_\theta(x_i^-)\):

\[\mathcal{L}_{N\text{-pairs}}(\theta;\, x, x^+, \{x_i^-\}) = \log\!\left(1 + \sum_{i=1}^{N-1} \exp\!\bigl(z^\top z_i^- - z^\top z^+\bigr)\right).\]

Equivalently, normalizing embeddings (working with inner products as similarities):

\[\mathcal{L}_{N\text{-pairs}} = -\log\frac{\exp(z^\top z^+)}{\exp(z^\top z^+) + \sum_{i=1}^{N-1} \exp(z^\top z_i^-)}.\]

This is a softmax cross-entropy: the loss is the negative log-probability assigned to the correct class (the positive) in a \((N)\)-way classification problem.

📡 InfoNCE and Contrastive Predictive Coding (van den Oord et al., 2018)

The conceptual leap from N-pairs to InfoNCE came through Contrastive Predictive Coding (CPC), introduced by van den Oord, Li, and Vinyals at DeepMind. CPC was designed for sequential data (speech, video) — learning representations by predicting the future from the present. But the loss function it introduced, InfoNCE, became the mathematical foundation of all modern contrastive learning.

🔗 From NCE to InfoNCE: Statistical Roots

InfoNCE does not appear from nowhere — it is a direct descendant of Noise Contrastive Estimation (NCE), introduced by Gutmann & Hyvärinen (2010) for fitting unnormalized density models without computing an intractable partition function.

The NCE problem. Suppose we have an unnormalized model \(\tilde{p}_\theta(x)\) and want to estimate \(p_\theta(x) = \tilde{p}_\theta(x) / Z(\theta)\) where \(Z(\theta) = \int \tilde{p}_\theta(x)\,dx\) is intractable. The maximum likelihood objective is:

\[\mathcal{L}_{\text{MLE}} = -\frac{1}{n}\sum_i \log p_\theta(x_i) = -\frac{1}{n}\sum_i \log \tilde{p}_\theta(x_i) + \log Z(\theta).\]

The \(\log Z(\theta)\) term is intractable, and its gradient \(\nabla_\theta \log Z(\theta) = \mathbb{E}_{x \sim p_\theta}[\nabla_\theta \log \tilde{p}_\theta(x)]\) requires sampling from the model itself. NCE replaces this with a binary classification problem — but the connection is not obvious and deserves unpacking.

Why a classifier can recover a density. The key insight comes from a basic Bayesian calculation. Suppose you mix data samples \(x^+ \sim p_d(x)\) with noise samples \(x^- \sim p_n(x)\) at ratio \(1:k\). Introduce a binary label \(C \in \{\text{data}, \text{noise}\}\); the prior probabilities are \(P(C = \text{data}) = \frac{1}{1+k}\) and \(P(C = \text{noise}) = \frac{k}{1+k}\).

Applying Bayes’ theorem for \(P(C = \text{data} \mid x)\), the marginal \(P(x)\) expands by the law of total probability:

\[P(x) = p_d(x) \cdot \frac{1}{1+k} + p_n(x) \cdot \frac{k}{1+k} = \frac{p_d(x) + k\,p_n(x)}{1+k}.\]

Substituting into Bayes’ theorem — the \((1+k)\) factors cancel:

\[h^*(x) = P(C = \text{data} \mid x) = \frac{p_d(x) \cdot \frac{1}{1+k}}{\frac{p_d(x) + k\,p_n(x)}{1+k}} = \frac{p_d(x)}{p_d(x) + k\, p_n(x)}.\]

Now rearrange: the odds ratio of the optimal classifier gives you the density ratio directly,

\[\frac{h^*(x)}{1 - h^*(x)} = \frac{p_d(x)}{k\, p_n(x)} \implies p_d(x) = k\, p_n(x) \cdot \frac{h^*(x)}{1 - h^*(x)}.\]

Since \(p_n\) and \(k\) are known, a perfect classifier uniquely determines \(p_d\) — without any integrals. This is the bridge: solving binary classification is equivalent to learning the density ratio \(p_d / p_n\).

How NCE exploits this. NCE parameterizes the classifier using the model,

\[h_\theta(x) = \frac{p_\theta(x)}{p_\theta(x) + k\, p_n(x)}, \qquad p_\theta(x) = \frac{\tilde{p}_\theta(x)}{e^c},\]

where \(c = \log Z(\theta)\) is treated as an additional scalar parameter to be learned by gradient descent — not computed by integration. The binary cross-entropy loss is then:

Definition (NCE Objective).

\[\mathcal{L}_{\text{NCE}}(\theta, c) = -\mathbb{E}_{x \sim p_d}\!\left[\log h_\theta(x)\right] - k\,\mathbb{E}_{y \sim p_n}\!\left[\log\bigl(1 - h_\theta(y)\bigr)\right].\]

This only ever evaluates \(\tilde{p}_\theta\) at individual sample points — no integral, no expectation under \(p_\theta\). At the minimum, the classifier achieves Bayes optimality: \(h_\theta = h^*\), which forces \(p_\theta(x) = p_d(x)\) everywhere. The scalar \(c\) converges to the true \(\log Z(\theta)\) as a byproduct — the model learns normalization automatically.

This is the statistical core of contrastive learning: estimating a density ratio by training a classifier against a reference distribution.

From NCE to InfoNCE. The N-pairs loss (from the metric learning lineage) and NCE (from the statistics lineage) converge at InfoNCE via two independent generalizations:

flowchart TD
    npairs["N-pairs loss (2016)
multiclass softmax
from metric learning"]
    nce["NCE (2010)
binary data-vs-noise
from statistics"]
    infonce["InfoNCE (2018)
K-way identification
density ratio as optimal critic"]
    npairs --> infonce
    nce --> infonce

InfoNCE generalizes the binary NCE classifier (\(K = 2\): data or noise) to a \(K\)-way identification task, while inheriting NCE’s key insight that the optimal solution is a density ratio:

The structural analogy \(p_n \leftrightarrow p(x)\) holds at the level of the math — both serve as the reference in the density ratio — but hides a deep conceptual shift. In NCE, negatives are synthetic: you fabricate a noise distribution \(p_n\) and draw fake samples from it. In InfoNCE, all \(K\) candidates are real data. There is no real-vs-fake distinction. The discrimination task is purely about conditioning: which of these real samples was drawn from \(p(x \mid c)\) rather than the unconditional \(p(x)\)?

The reason \(p(x)\) appears as the reference is not that it acts as noise — it is that the conditional-vs-marginal ratio \(p(x \mid c)/p(x)\) is exactly what mutual information measures: \(I(X; C) = \mathbb{E}[\log p(x \mid c)/p(x)]\). Distinguishing samples from \(p(x \mid c)\) vs. \(p(x)\) is MI estimation. NCE estimated a partition function; InfoNCE estimates an information-theoretic quantity.

🔑 The Categorical Cross-Entropy Derivation

Setup. We have a context representation \(c\) (the “query”) and a set of \(K\) candidate representations \(\{x_1, \ldots, x_K\}\), exactly one of which (\(x_+\), at position \(j^*\)) is the positive — the true future/paired sample. The rest are negatives, drawn independently from the data distribution \(p(x)\).

Let \(f_\theta(x, c)\) be a critic function (also called a score function) measuring compatibility between candidate \(x\) and context \(c\). The model assigns probability:

\[p(j \mid \{x_1, \ldots, x_K\}, c) = \frac{f_\theta(x_j, c)}{\sum_{k=1}^{K} f_\theta(x_k, c)}.\]

Definition (InfoNCE Loss).

\[\mathcal{L}_{\text{InfoNCE}} = -\mathbb{E}\!\left[\log \frac{f_\theta(x_+, c)}{\sum_{k=1}^{K} f_\theta(x_k, c)}\right]\]

where the expectation is over the choice of positive \(x_+\) and the \(K-1\) negatives drawn from \(p(x)\).

📐 The Mutual Information Lower Bound

The central theorem that gives InfoNCE its name:

Theorem (InfoNCE Lower Bound). Let \(X\) and \(C\) be jointly distributed random variables. Define

\[I_\theta(X; C) \triangleq \mathbb{E}\!\left[\log \frac{f_\theta(x, c)}{\frac{1}{K}\sum_{k=1}^K f_\theta(x_k, c)}\right]\]

where \(x_1, \ldots, x_K\) are \(K-1\) negatives plus one positive from \(p(x \mid c)\). Then:

\[I(X; C) \geq I_\theta(X; C) \geq \log K - \mathcal{L}_{\text{InfoNCE}}.\]

Proof sketch. We derive the lower bound \(\mathcal{L}_{\text{InfoNCE}} \geq \log K - I(X; C)\), equivalently \(I(X; C) \geq \log K - \mathcal{L}_{\text{InfoNCE}}\).

The optimal critic that minimizes \(\mathcal{L}_{\text{InfoNCE}}\) is the density ratio:

\[f^*(x, c) = \frac{p(x \mid c)}{p(x)}.\]

Why? The minimizer of \(-\mathbb{E}[\log q(j^* \mid \cdot)]\) over any probability distribution \(q\) is the true conditional \(p(j^* \mid \cdot)\). By Bayes:

\[p(j^* = j \mid \{x_k\}, c) \propto p(x_j \mid c) \cdot \prod_{k \neq j} p(x_k) \propto \frac{p(x_j \mid c)}{p(x_j)},\]

so the optimal critic is the density ratio \(p(x \mid c) / p(x)\).

Substituting \(f^*\) into \(\mathcal{L}_{\text{InfoNCE}}\):

\[\mathcal{L}_{\text{InfoNCE}}^* = -\mathbb{E}\!\left[\log\frac{p(x_+ \mid c) / p(x_+)}{\frac{1}{K}\sum_{k=1}^{K} p(x_k \mid c) / p(x_k)}\right].\]

Using Jensen’s inequality and the law of large numbers (the denominator concentrates to \(\mathbb{E}_{p(x)}[p(x \mid c)/p(x)] = 1\)):

\[\mathcal{L}_{\text{InfoNCE}}^* \approx -\mathbb{E}\!\left[\log\frac{p(x_+ \mid c)}{p(x_+)}\right] + \log K = -I(X;C) + \log K.\]

Hence \(I(X; C) = \log K - \mathcal{L}^*_{\text{InfoNCE}}\), and for any sub-optimal \(\theta\), \(I(X; C) \geq \log K - \mathcal{L}_{\text{InfoNCE}}(\theta)\). \(\square\)

Key consequence. Minimizing \(\mathcal{L}_{\text{InfoNCE}}\) over \(\theta\) maximizes a lower bound on \(I(X; C)\). The bound is tight as \(K \to \infty\) (more negatives = better MI estimate). This gives contrastive learning a principled information-theoretic interpretation: it maximizes mutual information between views.

🌡️ The Role of the Critic

The InfoNCE framework separates the architecture (how \(f_\theta\) is implemented) from the objective (the categorical cross-entropy). Different critic designs lead to different methods:

The bilinear critic in CPC allows the compatibility function to rotate and scale the context before comparing — more expressive but introduces \(O(d^2)\) parameters per prediction step.

🖼️ Instance Discrimination: The Vision Bridge (Wu et al., 2018)

Unsupervised Feature Learning via Non-Parametric Instance Discrimination (Wu, Xiong, Yu, Lin, CVPR 2018) occupies the crucial position between CPC and SimCLR: it applied the InfoNCE/NCE ideas to vision self-supervised learning for the first time, establishing that instance-level discrimination with no labels produces strong visual representations.

🎯 The Instance Discrimination Pretext Task

Definition (Instance Discrimination). Given a dataset \(\mathcal{D} = \{x_1, \ldots, x_n\}\) of \(n\) images, treat each image \(x_i\) as its own class \(y = i\). The pretext task is \(n\)-way classification: given a query, identify which training instance it came from.

This is a radical extension of the N-pairs framework: instead of \(N\) classes within a mini-batch, there are \(n\) classes across the entire training set (\(n = 1{,}281{,}167\) for ImageNet).

The non-parametric softmax. In standard \(n\)-way classification, the model learns a weight vector \(w_i \in \mathbb{R}^d\) per class. With \(n > 10^6\) classes this is prohibitively expensive. Instead, InstDisc uses the stored embedding \(v_i \in \mathbb{R}^d\) directly as the class prototype:

\[P(i \mid x) = \frac{\exp(v_i^\top f_\theta(x) / \tau)}{\sum_{j=1}^{n} \exp(v_j^\top f_\theta(x) / \tau)}.\]

The denominator sums over all \(n\) training embeddings — still intractable. InstDisc approximates it using NCE.

💾 The Memory Bank and NCE Approximation

Memory bank. InstDisc maintains a bank \(\mathcal{M} \in \mathbb{R}^{n \times d}\) storing the most recently computed \(\ell_2\)-normalized embedding for each training image. After each forward pass for image \(x_i\), the entry \(\mathcal{M}[i]\) is updated to the new embedding. The bank decouples the embeddings used as negatives from the current model parameters — a direct precursor to MoCo’s queue.

NCE approximation. To avoid computing the full \(n\)-way denominator, InstDisc draws \(m\) noise indices \(\{j_1, \ldots, j_m\}\) uniformly from \(\{1, \ldots, n\}\) and uses:

\[P_{\text{NCE}}(i \mid x) = \frac{\exp(v_i^\top f_\theta(x) / \tau)}{\exp(v_i^\top f_\theta(x) / \tau) + \frac{n}{m}\sum_{k=1}^{m} \exp(v_{j_k}^\top f_\theta(x) / \tau)}.\]

This is a binary classifier between the positive instance \(i\) and \(m\) noise instances — directly NCE with a uniform noise distribution over training indices. The \(n/m\) factor corrects for the subsampling. The full loss is:

\[\mathcal{L}_{\text{InstDisc}} = -\sum_{i \in \text{batch}} \log P_{\text{NCE}}(i \mid x_i).\]

🔑 What InstDisc Established

InstDisc demonstrated three things that directly enabled SimCLR:

1. Instance-level discrimination produces semantic representations. Despite never seeing class labels, InstDisc embeddings cluster by semantic category. Nearest-neighbor retrieval using InstDisc features achieves $$40% top-1 accuracy on ImageNet — competitive with early supervised methods, and achieved with zero labels.

2. Temperature is critical at \(\tau = 0.07\). The temperature in the non-parametric softmax controls the concentration of the probability distribution. InstDisc established this value experimentally — exactly the value SimCLR later adopted.

3. The memory bank solves the large-batch problem. By storing all \(n\) embeddings, InstDisc sidesteps the need for large mini-batches. The effective negative pool is the entire training set. The cost is staleness: memory bank embeddings may be up to \(n/B\) batches old.

🧪 SimCLR: InfoNCE for Vision (Chen et al., 2020)

A Simple Framework for Contrastive Learning of Visual Representations (SimCLR, Chen et al., 2020) distilled the CPC/InfoNCE framework into a clean recipe for visual self-supervised learning, eliminating specialized architectures and achieving state-of-the-art with remarkable simplicity.

The full SimCLR pipeline: two augmented views of a batch pass through a shared encoder \(f_\theta\) and projector \(g_\theta\); the NT-Xent loss maximizes agreement between positive pairs and pushes apart all others.

🔭 Augmentation Strategy

SimCLR’s core insight: the choice of data augmentation defines what the representation will be invariant to, and this choice is the single most important design decision.

The augmentation pipeline samples two views \((v, v')\) of each image \(x \sim p(x)\) by composing:

1. Random cropping — crops a random patch, resized to \(224 \times 224\). This is the most important augmentation: it forces the network to recognize objects across scale and position.
2. Random horizontal flip — 50% probability.
3. Color jitter — random brightness, contrast, saturation, hue perturbations with specified strength.
4. Random grayscale — 20% probability of converting to grayscale.
5. Gaussian blur — kernel size \(10\%\) of image size, \(\sigma \in [0.1, 2.0]\).

The key finding: color jitter + crop together are the dominant pair. Without color jitter, representations exploit color histograms (a shortcut) rather than learning semantic content. Without crops, two views of the same image are too similar and the task is too easy.

The augmentation pipeline applies a stochastic composition of crop, flip, color jitter, grayscale, and Gaussian blur to produce two randomly-transformed views \((v, v')\) from each image \(x\).

📐 NT-Xent as an InfoNCE Instance

Given a batch of \(N\) images, SimCLR produces \(2N\) embeddings by applying augmentation twice to each image. After passing through encoder \(f_\theta\) and projector \(g_\theta\), embeddings are \(\ell_2\)-normalized:

Batch preparation: \(N\) images are each augmented twice, yielding \(2N\) views. Positive pairs are the two augmented views of the same image (shaded); all other within-batch pairs are negatives.

\[\bar{z}_k = z_k / \|z_k\|_2, \quad k \in \{1, \ldots, 2N\}.\]

The \(2N\) embeddings are indexed as \(\bar{z}_1, \bar{z}_1', \bar{z}_2, \bar{z}_2', \ldots, \bar{z}_N, \bar{z}_N'\) where \((\bar{z}_k, \bar{z}_k')\) are the two views of image \(k\).

Definition (NT-Xent Loss). The normalized temperature-scaled cross-entropy loss for anchor \(\bar{z}_k\) is:

\[\ell_k = -\log \frac{\exp(\bar{z}_k^\top \bar{z}_k' / \tau)}{\displaystyle\sum_{m=1}^{2N} \mathbf{1}[m \neq k]\, \exp(\bar{z}_k^\top \bar{z}_m / \tau)}\]

where \(\tau > 0\) is the temperature. The full objective is symmetric:

\[\mathcal{L}_{\text{SimCLR}} = \frac{1}{2N} \sum_{k=1}^{N} \bigl(\ell_k + \ell_k'\bigr).\]

The \(2N \times 2N\) pairwise cosine similarity matrix. Diagonal blocks (same-image pairs) are the positives; the loss drives the off-diagonal entries down while pulling the positive entries up. Self-similarities (diagonal) are masked out.

For each anchor \(\bar{z}_k\), the NT-Xent loss is the negative log of the softmax probability assigned to its positive partner \(\bar{z}_k'\) in a \((2N-1)\)-way classification over all other embeddings in the batch.

NT-Xent is InfoNCE with critic \(f_\theta(x, c) = \exp(\bar{z}(x)^\top \bar{z}(c) / \tau)\), context \(c = v_k\) (one view), positive \(x_+ = v_k'\) (other view of same image), and \(K - 1 = 2(N-1)\) negatives from the batch (all other \(2N-2\) embeddings). The identity \(\mathcal{L}_{\text{NT-Xent}} = \mathcal{L}_{\text{InfoNCE}}\) holds exactly:

\[\ell_k = -\log\frac{f_\theta(\bar{z}_k', \bar{z}_k)}{\sum_{m \neq k} f_\theta(\bar{z}_m, \bar{z}_k)}.\]

Connection to the lower bound. By the InfoNCE theorem:

\[I(V; V') \geq \log(2N-1) - \mathcal{L}_{\text{SimCLR}}.\]

Minimizing \(\mathcal{L}_{\text{SimCLR}}\) approximately maximizes the mutual information between the two augmented views. The MI bound grows as \(\log(2N-1)\) — maximized by using the largest possible batch size.

🔬 The Projector Head

A surprising finding from Chen et al.: appending a 2–3 layer MLP projector \(g_\theta\) after the encoder \(f_\theta\), and discarding \(g_\theta\) at evaluation time, consistently improves linear probe accuracy by $$10 percentage points.

The encoder \(f_\theta\) (ResNet-50 in the original paper) maps each augmented view to a 2048-dimensional representation \(h = f_\theta(v)\), which is the vector used for downstream tasks.

The projector \(g_\theta\) — a 2-layer MLP with ReLU — maps \(h \in \mathbb{R}^{2048}\) to \(z \in \mathbb{R}^{128}\). Only \(z\) sees the NT-Xent loss; \(h\) is preserved for downstream linear probing.

Without a projector, the NT-Xent loss operates directly on the encoder output, forcing the encoder to simultaneously: 1. Be invariant to the augmentations (since \((v, v')\) must map to similar \(z\)) 2. Be informative for downstream tasks

These two goals conflict: some information discarded for augmentation invariance (e.g. exact crop location, color saturation level) is genuinely useless, but other information discarded (e.g. fine-grained texture, color identity for certain tasks) may matter downstream.

The projector acts as an information sink. Information discarded by the NT-Xent objective is removed in \(z = g_\theta(y)\) but preserved in \(y = f_\theta(x)\), which is never directly optimized by the contrastive loss. The projector absorbs the augmentation invariances, protecting the encoder.

flowchart LR
    x["image x"] --> f["f_theta
encoder"]
    f --> y["y in R^{d_f}
representation"]
    y --> g["g_theta
projector MLP"]
    g --> z["z in R^d
embedding"]
    z --> L["L_SimCLR
NT-Xent"]
    y -. "linear probe
at eval" .-> cls["classifier"]

🌡️ Temperature and Hard Negatives

The temperature \(\tau\) is a crucial hyperparameter with a precise effect on the gradient distribution.

For anchor \(\bar{z}_k\) with positive similarity \(s_+ = \bar{z}_k^\top \bar{z}_k'\) and negative similarities \(\{s_m\}_{m \neq k}\), the softmax probability assigned to negative \(m\) is:

\[p_m = \frac{\exp(s_m / \tau)}{\exp(s_+/\tau) + \sum_{m'\neq k} \exp(s_{m'}/\tau)}.\]

The gradient of \(\ell_k\) with respect to the positive similarity is:

\[\frac{\partial \ell_k}{\partial s_+} = \frac{-1}{\tau}(1 - p_+), \quad p_+ = \frac{\exp(s_+/\tau)}{\sum_m \exp(s_m/\tau)}.\]

The gradient with respect to negative similarity \(s_m\) is:

\[\frac{\partial \ell_k}{\partial s_m} = \frac{p_m}{\tau}.\]

Effect of \(\tau\): - Small \(\tau\): all softmax mass concentrates on the highest-similarity negatives (the hard negatives with \(s_m \approx s_+\)). The gradient is dominated by a few hard negatives — efficient but sensitive to noisy negatives (false negatives from the same class). - Large \(\tau\): softmax mass is spread uniformly. All negatives contribute equally — unbiased but inefficient, as most negatives are easy and provide little signal.

SimCLR uses \(\tau = 0.07\) — a small temperature that concentrates on hard negatives. This choice assumes negatives within a batch are unlikely to be semantically similar (reasonable at batch size 4096 with 1000 ImageNet classes), so hard negatives are genuinely hard, not false positives.

🌐 Alignment and Uniformity on the Hypersphere

Wang & Isola (2020) decompose the NT-Xent objective into two orthogonal geometric properties, revealing precisely what SimCLR optimizes on the unit sphere \(S^{d-1}\).

Setup. Since SimCLR \(\ell_2\)-normalizes embeddings, all representations lie on \(S^{d-1}\). Let \(p_{\text{pos}}\) denote the distribution over positive pairs \((z, z^+)\) and \(p_{\text{data}}\) the marginal distribution over individual embeddings.

Definition (Alignment Loss).

\[\mathcal{L}_{\text{align}} = \mathbb{E}_{(z, z^+) \sim p_{\text{pos}}}\!\left[\|z - z^+\|_2^2\right].\]

On the unit sphere, \(\|z - z^+\|_2^2 = 2 - 2\langle z, z^+\rangle\), so \(\mathcal{L}_{\text{align}} = 2\bigl(1 - \mathbb{E}[\langle z, z^+\rangle]\bigr)\). Minimizing alignment pulls positive pairs to the same point on \(S^{d-1}\).

Definition (Uniformity Loss).

\[\mathcal{L}_{\text{uniform}} = \log\,\mathbb{E}_{z,\, z' \overset{\text{iid}}{\sim} p_{\text{data}}}\!\left[\exp\!\bigl(-2\|z - z'\|_2^2\bigr)\right].\]

This is the log of the average pairwise Gaussian kernel over the embedding distribution. Since all embeddings lie on \(S^{d-1}\) (so \(\|z - z'\|^2 = 2 - 2\langle z, z'\rangle\)), the expression becomes \(\log\,\mathbb{E}[\exp(-4(1-\langle z, z'\rangle))]\). The kernel \(\exp(-2\|z - z'\|^2)\) is maximized when \(z = z'\) and small when points are far apart, so \(\mathcal{L}_{\text{uniform}}\) is minimized when the distribution is maximally spread on \(S^{d-1}\) — i.e. uniform on the hypersphere.

NT-Xent decomposes into alignment + uniformity. The NT-Xent loss for a single anchor \(z\) with positive \(z^+\) is:

\[\ell = \underbrace{-z^\top z^+ / \tau}_{\substack{\text{alignment:}\\ \text{pull positive}}} + \underbrace{\log \sum_{m} \exp(z^\top z_m / \tau)}_{\substack{\text{uniformity:}\\ \text{push all apart}}}.\]

In the large-batch limit where the batch approximates \(p_{\text{data}}\):

\[\mathcal{L}_{\text{NT-Xent}} \approx \frac{1}{\tau}\,\mathcal{L}_{\text{align}} + \frac{1}{\tau}\,\mathcal{L}_{\text{uniform}} + \text{const}.\]

🔑 The tension is fundamental. Alignment alone is minimized by mapping everything to one point — collapse. Uniformity alone (ignoring positives) is minimized by a perfectly uniform distribution with no positive structure. NT-Xent balances both: temperature \(\tau\) rescales the tradeoff, with small \(\tau\) sharpening the uniformity pressure (harder negative separation) and large \(\tau\) softening both terms.

🏋️ Semi-Supervised Learning and Scaling

The note so far has treated SimCLR purely as a self-supervised pre-training method evaluated via linear probing — freeze \(f_\theta\), fit a linear classifier on top. The paper also establishes SimCLR as a semi-supervised learner by fine-tuning the entire encoder on a small labeled subset.

Two downstream protocols.

The 1% result is the most striking: 85.8% top-5 accuracy matches AlexNet trained with 100× more labels. The encoder pre-trained on unlabeled data is already in a region of parameter space where a small labeled nudge generalizes well.

Why fine-tuning dominates linear probing at low label fractions. Linear probing is a strict test: the encoder must compress all task-relevant variation into a linearly separable structure before training ends. At 1% labels, a linear classifier is severely data-starved — it can overfit to the few labeled examples with no capacity to correct the encoder’s bias. Fine-tuning allows the encoder to adapt its invariances to the task, discarding irrelevant pre-training structure. The pre-trained initialization provides strong inductive bias while the labeled data steers it — a regime where contrastive pre-training pays a large dividend.

Self-training / distillation pipeline. The fine-tuned semi-supervised model can generate pseudo-labels on remaining unlabeled data, bootstrapping a knowledge distillation loop:

1. Pre-train \(f_\theta\) with NT-Xent on all unlabeled data.
2. Fine-tune \((f_\theta, \text{head})\) on the small labeled set \(\to\) teacher \(T\).
3. Apply \(T\) to unlabeled data to generate pseudo-labels \(\hat{y}_i = T(x_i)\).
4. Train a student \(S\) (possibly smaller) on the full labeled + pseudo-labeled corpus.

Each iteration tightens the pseudo-label quality; the contrastive pre-training ensures the teacher’s representations generalize to the unlabeled distribution.

Scaling behavior. Unlike supervised training, contrastive pre-training does not plateau quickly:

• Supervised ResNet-50 tops out between 90 and 300 epochs.
• SimCLR continues improving past 800 epochs. Each epoch presents fresh stochastic negative combinations drawn from the \(2N\)-way batch, so the effective contrast distribution is not exhausted by repeated passes over the dataset.

Three independent scaling axes all help:

🧭 Retrospective: The Historical Arc

The 15-year trajectory from contrastive loss to SimCLR traces a progression along three axes: how negatives are specified, how similarity is measured, and what theoretical grounding justifies the objective.

flowchart TD
    NCE["Noise Contrastive Estimation
Gutmann and Hyvarinen 2010
Binary density ratio estimation"]
    A["Contrastive Loss
Hadsell et al. 2006
Pairs + Euclidean margin"]
    B["Triplet Loss
Schroff et al. 2015
Triplets + relative ordering"]
    C["N-Pairs Loss
Sohn 2016
N-way softmax + inner product"]
    D["InfoNCE / CPC
van den Oord et al. 2018
MI lower bound + density ratio critic"]
    F["Instance Discrimination
Wu et al. 2018
Non-parametric softmax + memory bank"]
    E["NT-Xent / SimCLR
Chen et al. 2020
InfoNCE for vision + augmentation pairing"]
    A --> B
    B --> C
    C --> D
    NCE --> D
    D --> F
    F --> E

Key transitions:

1. Pairs → Triplets: shifted from absolute-distance constraints to relative-ordering constraints, removing the fragile margin \(m\).
2. Triplets → N-Pairs: moved all negatives into a single softmax, enabling efficient in-batch negative mining and revealing the classification interpretation.
3. NCE → InfoNCE: NCE estimated densities by binary classification against noise; InfoNCE generalized to \(K\)-way classification, replacing absolute density estimation with conditional density ratio estimation (\(p(x|c)/p(x)\)) and partition-function estimation with MI lower bounding.
4. InfoNCE → InstDisc: applied the NCE/InfoNCE framework to vision for the first time via instance-level discrimination; introduced the memory bank to decouple negatives from batch size; established \(\tau = 0.07\) and the non-parametric softmax.
5. InstDisc → SimCLR: replaced the identity pretext and memory bank with augmentation pairing and in-batch negatives; added the projector head; showed that augmentation composition is the dominant design variable.

SimCLR’s linear evaluation performance on ImageNet as a function of training epochs, batch size, and projector ablations. The key result: 76.5% top-1 at 1000 epochs with batch size 4096, a 7% absolute gain over prior SSL methods and matching supervised ResNet-50.

After self-supervised pre-training, the encoder \(f_\theta\) is frozen and a linear classifier is trained on top of the representations. SimCLR representations transfer to downstream tasks without any contrastive fine-tuning.

📚 References