🔍 Self-Supervised Vision: Contrastive Learning and Beyond

BYOL · SimSiam · Barlow Twins · VICReg — a pedagogical account of collapse-free self-supervised objectives

Table of Contents

• 🏗️ Background: The Joint-Embedding Framework
• 💥 The Collapse Problem
• ⚔️ Contrastive Baselines: SimCLR and MoCo
• 🔁 BYOL: Bootstrapping Without Negatives
• 🔀 SimSiam: Stop-Gradient Alone
• 📊 Barlow Twins: Redundancy Reduction
• 🔧 VICReg: Explicit Regularization
• 🗺️ Unified Perspective
• 📚 References

🏗️ Background: The Joint-Embedding Framework

Self-supervised learning (SSL) for vision aims to learn representations from unlabeled data. The dominant paradigm is the joint-embedding framework: train a network to produce similar representations for two different views of the same image.

Formal setup. Let \(\mathcal{X}\) be the image space and \(\mathcal{T}\) a distribution over stochastic augmentation functions \(t : \mathcal{X} \to \mathcal{X}\) (random crops, color jitter, grayscale, blur, etc.). Given an image \(x \in \mathcal{X}\), sample two views:

\[v = t(x), \quad v' = t'(x), \quad t, t' \overset{\mathrm{iid}}{\sim} \mathcal{T}.\]

An encoder \(f_\theta : \mathcal{X} \to \mathbb{R}^{d_f}\) maps images to representations, and a projector \(g_\theta : \mathbb{R}^{d_f} \to \mathbb{R}^d\) maps representations to the embedding space:

\[y = f_\theta(v), \quad z = g_\theta(y) \in \mathbb{R}^d.\]

Some methods add a predictor \(h_\theta : \mathbb{R}^d \to \mathbb{R}^d\) on one branch (discussed in §BYOL).

The downstream representation is always \(y = f_\theta(\cdot)\) — the projector \(g\) is discarded at inference time. This separation is empirically crucial: the projector absorbs information discarded by the SSL objective, protecting the encoder from over-specializing to augmentation invariances.

Notation summary.

💥 The Collapse Problem

The naive similarity-only objective is:

\[\mathcal{L}_{\text{naive}}(\theta) = \mathbb{E}_{x,\, t,\, t'}\bigl[\|z - z'\|^2\bigr].\]

This is minimized trivially by the constant solution: \(z = z' = \mathbf{c}\) for any fixed vector \(\mathbf{c} \in \mathbb{R}^d\), regardless of input. Every image maps to the same point — the representation carries no information.

Definition (Representational Collapse). We say the system has collapsed if the rank of the empirical embedding matrix \(Z \in \mathbb{R}^{N \times d}\) (rows = embeddings for \(N\) images) is much less than \(\min(N, d)\). Full collapse has \(\mathrm{rank}(Z) = 1\) (all embeddings identical up to a global mean shift).

Necessary condition for non-collapse. For \(z\) to be informative, the covariance matrix \(\Sigma = \mathrm{Cov}(z)\) must be full-rank (all eigenvalues \(> 0\)). Equivalently, the embedding distribution \(p(z)\) must have support over a non-degenerate \(d\)-dimensional set.

Every SSL method in this note introduces a mechanism specifically designed to prevent collapse. They differ in where they enforce diversity: in the negatives (contrastive), in the gradient flow (BYOL), in the correlation structure (Barlow Twins), or in the marginal variance (VICReg).

⚔️ Contrastive Baselines: SimCLR and MoCo

Contrastive methods prevent collapse by explicitly repelling embeddings of different images. The core insight: if all embeddings must be far from each other (except matched pairs), they cannot all collapse to a single point.

📐 The NT-Xent Loss (SimCLR)

Given a batch of \(N\) images, we produce \(2N\) embeddings \(\{z_1, z_1', z_2, z_2', \ldots, z_N, z_N'\}\) after \(\ell_2\)-normalization. Each \((z_k, z_k')\) is a positive pair; all other \(2(N-1)\) embeddings in the batch are negatives for \(z_k\).

The normalized temperature-scaled cross-entropy (NT-Xent) loss for anchor \(z_k\) paired with positive \(z_k'\) is:

\[\ell_k = -\log \frac{\exp\!\bigl(\mathrm{sim}(z_k,\, z_k') / \tau\bigr)}{\displaystyle\sum_{m=1}^{2N} \mathbf{1}[m \neq k]\, \exp\!\bigl(\mathrm{sim}(z_k,\, z_m) / \tau\bigr)}\]

where \(\mathrm{sim}(u, v) = u^\top v / (\|u\| \|v\|)\) is cosine similarity and \(\tau > 0\) is a temperature hyperparameter. The full loss averages over both orderings of each positive pair:

\[\mathcal{L}_{\text{SimCLR}} = \frac{1}{2N} \sum_{k=1}^{N} \bigl(\ell_k + \ell_k'\bigr).\]

Why temperature matters. Small \(\tau\) sharpens the softmax, concentrating gradient signal on the hardest negatives (those most similar to the anchor). Large \(\tau\) spreads attention uniformly. SimCLR uses \(\tau = 0.07\).

The batch-size bottleneck. The denominator sums over \(2(N-1)\) negatives. With too few negatives, the objective is easy and representations are poorly calibrated. SimCLR requires \(N \approx 4096\) — demanding hundreds of GPUs. This is the primary practical limitation of pure contrastive learning.

🔧 MoCo: The Memory Bank Fix

He et al. (MoCo, 2020) decouple the negative pool from the current batch by maintaining a queue of \(K \gg N\) past embeddings (e.g. \(K = 65\,536\)). To keep the queue consistent — embeddings produced by a slowly-changing encoder — MoCo introduces a momentum encoder with parameters \(\xi\):

\[\xi \leftarrow m\xi + (1 - m)\theta, \quad m \approx 0.999.\]

All negatives in the queue are produced by the momentum encoder; the query is produced by the online encoder \(\theta\). This allows large effective negative pools without requiring large batches.

🔁 BYOL: Bootstrapping Without Negatives

Bootstrap Your Own Latent (Grill et al., NeurIPS 2020) removes negatives entirely. The central question: without repulsion, what prevents collapse?

🏛️ Architecture

BYOL uses two networks with asymmetric structure:

flowchart LR
    v["view v"]
    vp["view v'"]
    fo["f_theta
online encoder"]
    go["g_theta
online projector"]
    qo["q_theta
predictor"]
    ft["f_xi
target encoder"]
    gt["g_xi
target projector"]
    sg["stop-grad"]
    loss["L_BYOL"]

    v --> fo --> go --> qo --> loss
    vp --> ft --> gt --> sg --> loss
    fo -. "EMA update" .-> ft
    go -. "EMA update" .-> gt

The online network comprises encoder \(f_\theta\), projector \(g_\theta\), and predictor \(q_\theta\). The target network has only encoder \(f_\xi\) and projector \(g_\xi\) — no predictor. This asymmetry is the key structural ingredient.

Target network update (EMA).

\[\xi \leftarrow \tau \xi + (1 - \tau)\theta, \quad \tau \in [0, 1).\]

\(\tau\) is annealed from \(0.996\) toward \(1.0\) over training. The target network receives no gradient updates — only EMA updates from the online network.

📐 The BYOL Loss

Given views \(v, v'\) of the same image \(x\), compute online and target embeddings:

\[z_\theta = g_\theta(f_\theta(v)), \quad z'_\xi = g_\xi(f_\xi(v')).\]

The online network predicts the target projection:

\[\mathcal{L}_{\text{BYOL}} = \bigl\| \bar{q}_\theta(z_\theta) - \bar{z}'_\xi \bigr\|^2 = 2 - 2 \cdot \frac{\langle q_\theta(z_\theta),\; \mathrm{sg}(z'_\xi)\rangle}{\|q_\theta(z_\theta)\|_2 \cdot \|\mathrm{sg}(z'_\xi)\|_2}\]

where \(\bar{u} = u / \|u\|_2\) denotes \(\ell_2\)-normalization and \(\mathrm{sg}(\cdot)\) is the stop-gradient operator (zeroes gradients during backpropagation). The full objective symmetrizes over both views:

\[\mathcal{L} = \mathcal{L}_{\text{BYOL}}(\theta, \xi;\, v, v') + \mathcal{L}_{\text{BYOL}}(\theta, \xi;\, v', v).\]

Gradients flow only through \(\theta\) — the target \(\xi\) is not differentiated.

🔑 Why Doesn’t BYOL Collapse?

Several mechanisms act jointly:

1. Stop-gradient asymmetry. The gradient \(\nabla_\theta \mathcal{L}\) depends on the online output \(q_\theta(z_\theta)\) but treats the target \(z'_\xi\) as a fixed constant. This breaks the symmetry between the two branches: the online network must predict the target, not merely agree with it. The predictor \(q_\theta\) must learn a non-trivial mapping from online embeddings to target embeddings, creating a non-degenerate optimization landscape.

2. Batch normalization. The projectors \(g_\theta\) and \(g_\xi\) contain batch normalization layers. For a batch of \(N\) projector outputs \(z_1, \ldots, z_N \in \mathbb{R}^d\), BN computes:

   \[\hat{z}_i = \frac{z_i - \mu_B}{\sigma_B + \epsilon}, \qquad \mu_B = \frac{1}{N}\sum_{k=1}^{N} z_k.\]

   The crucial observation is that \(\hat{z}_i\) is a relative quantity — it encodes how \(z_i\) deviates from the batch average, not its absolute position in \(\mathbb{R}^d\). Factoring out \(\mu_B\) makes this concrete. Since \(z_i - \mu_B = \frac{N-1}{N}(z_i - \bar{z}_{-i})\) where \(\bar{z}_{-i} = \frac{1}{N-1}\sum_{k \neq i} z_k\):

   \[\hat{z}_i = \frac{N-1}{N(\sigma_B + \epsilon)}\!\left(z_i - \bar{z}_{-i}\right).\]

   Up to a scalar factor, \(\hat{z}_i\) is the deviation of \(z_i\) from the mean of all other batch embeddings — “how does image \(i\) differ from a typical image in this batch?” The other \(N-1\) samples act as implicit negatives.

   Why this creates implicit repulsion. Suppose \(z_j\) begins drifting toward \(z_i\) (embeddings start to coalesce). As \(z_j \to z_i\), the batch mean \(\mu_B\) shifts toward \(z_i\), causing \(z_i - \mu_B\) to shrink — \(\hat{z}_i\) deflates toward zero. But the BYOL loss wants \(\hat{z}_i\) to match the (stop-gradient) target network output, which is generically non-zero. The loss therefore increases, generating a gradient that pushes \(z_i\) away from \(\mu_B\). Through the shared mean, this gradient also nudges \(z_j\) in the opposite direction. The result: every image’s attempt to match its target automatically resists other embeddings moving toward it.

   This coupling is exact via the chain rule. For \(j \neq i\):

   \[\frac{\partial \hat{z}_i}{\partial z_j} = -\frac{1}{N(\sigma_B + \epsilon)} \neq 0.\]

   The repulsive force that image \(i\)’s loss exerts on \(z_j\) is:

   \[\frac{\partial \mathcal{L}_i}{\partial z_j} = \underbrace{\frac{\partial \mathcal{L}_i}{\partial \hat{z}_i}}_{\text{loss gradient for image }i} \cdot \underbrace{\left(-\frac{1}{N(\sigma_B+\epsilon)}\right)}_{\text{leakage through }\mu_B}.\]

   The leakage factor \(-1/N(\sigma_B+\epsilon)\) is what propagates image \(i\)’s gradient signal into the parameters that generated \(z_j\). BYOL’s nominally non-contrastive loss therefore sends implicit cross-sample gradient signals across all pairs in the batch — structurally analogous to the repulsive denominator terms in NT-Xent.

   Why BN is insufficient alone. The repulsive force is loss-gradient-driven: it is proportional to \(\frac{\partial \mathcal{L}_i}{\partial \hat{z}_i}\), which vanishes when the loss is near zero. At full collapse \(z_k = \mathbf{c}\) for all \(k\): \(\mu_B = \mathbf{c}\), \(\sigma_B = 0\), and \(\hat{z}_i = 0\) for all \(i\); after learnable scale \(\gamma\) and bias \(\beta\), the BN output is the constant \(\beta\). The predictor trivially learns \(q(\beta) = \beta\), so \(\mathcal{L} = 0\) and \(\nabla_\theta\mathcal{L} = 0\). Collapse is still a valid zero-loss fixed point. BN acts like viscosity — it resists movement toward collapse during training but cannot prevent arriving there.

3. EMA bootstrap. The target \(\xi\) lags behind \(\theta\) by approximately \(1/(1-\tau)\) gradient steps. The online network must constantly track a moving target — there is no stable fixed point at which both networks output the same constant and the gradient vanishes.

Relative load-bearing of the three mechanisms:

🔍 The Optimal Predictor Perspective

The three mechanisms above explain why collapse is not a stable fixed point. Xiong (2024) offers a complementary account that reframes BYOL in terms of what the encoder is positively incentivized to do.

Setup. Treat \(q_\theta\) as approximately optimal for the current target \(\xi\). This is reasonable: the predictor is updated by gradient descent on every step, while \(\xi\) drifts slowly under EMA with \(\tau \approx 0.996\). From the encoder’s perspective, \(q_\theta\) looks essentially stationary.

A loss-minimizing predictor satisfies \(q^*(z_\theta) = \mathbb{E}[z'_\xi \mid z_\theta]\) — the conditional expectation of the target embedding given the online embedding. Substituting, the encoder’s effective loss is the conditional variance of the target:

\[\mathbb{E}\bigl[\|q^*(z_\theta) - z'_\xi\|^2\bigr] = \mathbb{E}\bigl[\mathrm{Var}(z'_\xi \mid z_\theta)\bigr].\]

The implicit instance-discrimination argument. The law of total variance decomposes the target’s variance:

\[\underbrace{\mathrm{Var}(z'_\xi)}_{\text{fixed by target network}} = \underbrace{\mathbb{E}[\mathrm{Var}(z'_\xi \mid z_\theta)]}_{\text{encoder minimizes}} + \underbrace{\mathrm{Var}(\mathbb{E}[z'_\xi \mid z_\theta])}_{\text{explained variance — must grow}}.\]

Since \(\mathrm{Var}(z'_\xi)\) is held fixed by the (slowly moving) target network, reducing the conditional variance forces the explained variance — the spread of the predictor’s outputs — to increase. The predictor can only produce spread-out outputs if \(z_\theta\) carries substantial information about which image generated the embedding. Concretely: the predictor achieves lower loss precisely when distinct images land in distinct embedding regions, because it can then look up the correct target rather than outputting a global average.

The encoder therefore has an implicit incentive to be discriminative — not because it is penalized for similarity between different images’ embeddings, but because the prediction task is only solvable when embeddings separate instances.

🔀 SimSiam: Stop-Gradient Alone

Simple Siamese Representation Learning (Chen & He, CVPR 2021) distills BYOL to its minimal core by removing the EMA target network entirely. The question it answers: is the momentum encoder necessary, or is stop-gradient alone sufficient to prevent collapse?

Surprisingly, stop-gradient plus a predictor suffices. EMA is a stabilizer, not a collapse-prevention mechanism.

🏛️ Architecture

Both branches share identical parameters \(\theta\) — there is no separate target network. A predictor \(h_\phi\) with its own parameters \(\phi\) sits atop the projector on both branches:

flowchart LR
    v["view v"]
    vp["view v'"]
    enc1["f_theta + g_theta
shared weights"]
    enc2["f_theta + g_theta
shared weights"]
    z1["z_1"]
    z2["z_2"]
    p1["p_1 = h_phi(z_1)"]
    p2["p_2 = h_phi(z_2)"]
    sgz2["sg(z_2)"]
    sgz1["sg(z_1)"]
    D1["D(p_1, sg(z_2))"]
    D2["D(p_2, sg(z_1))"]
    loss["L_SimSiam"]

    v --> enc1 --> z1 --> p1 --> D1 --> loss
    vp --> enc2 --> z2 --> p2 --> D2 --> loss
    z2 --> sgz2 --> D1
    z1 --> sgz1 --> D2

Both \(\theta\) (encoder + projector) and \(\phi\) (predictor) are updated by gradient descent — there is no EMA copy.

📐 The SimSiam Loss

Define the negative cosine similarity:

\[\mathcal{D}(p,\, z) = -\frac{p^\top z}{\|p\|_2 \cdot \|z\|_2}.\]

Let \(z_1 = g_\theta(f_\theta(v))\) and \(z_2 = g_\theta(f_\theta(v'))\). The SimSiam objective is:

\[\mathcal{L}_{\text{SimSiam}} = \frac{1}{2}\,\mathcal{D}\!\bigl(h_\phi(z_1),\;\mathrm{sg}(z_2)\bigr) + \frac{1}{2}\,\mathcal{D}\!\bigl(h_\phi(z_2),\;\mathrm{sg}(z_1)\bigr).\]

Gradients flow through \(h_\phi(z_i)\) — and back into \(z_i\) and then \(\theta\) — but not through \(\mathrm{sg}(z_j)\).

🔑 The EM Interpretation

Chen & He interpret SimSiam as an expectation-maximization (EM) algorithm with two alternating subproblems:

E-step — optimize the predictor, fix the encoder. For fixed \(\theta\), find:

\[h_\phi^* = \operatorname{argmin}_{\phi}\; \mathbb{E}\!\bigl[\mathcal{D}(h_\phi(z_1),\; \mathrm{sg}(z_2))\bigr].\]

The solution is proportional to the conditional expectation: \(h_\phi^*(z_1) \propto \mathbb{E}[z_2 \mid z_1]\). The optimal predictor computes the expected target embedding given the online embedding.

M-step — optimize the encoder, fix the predictor. For fixed \(h_\phi^*\), update:

\[\theta \leftarrow \theta - \eta\, \nabla_\theta\; \mathbb{E}\!\bigl[\mathcal{D}(h_\phi^*(z_1),\; \mathrm{sg}(z_2))\bigr].\]

This pushes the encoder to produce embeddings \(z_1\) that are better predicted by the current \(h_\phi^*\), i.e. more consistent with the target \(z_2\). The stop-gradient on \(z_2\) implements the decoupling between the two steps — without it, both objectives are conflated into a single gradient step that collapses.

The EM frame also clarifies why a predictor is necessary: the E-step only makes sense if there is a separate parameter \(\phi\) to optimize. With \(h = \mathrm{id}\), the E-step is vacuous and the M-step has a degenerate global minimum at constant embeddings.

📊 Barlow Twins: Redundancy Reduction

Barlow Twins (Zbontar et al., ICML 2021) prevents collapse by directly imposing a spectral constraint on the cross-correlation structure of the embeddings — driving the cross-correlation matrix of twin embeddings toward the identity.

🧬 Neuroscience Motivation

The method is named after Horace Barlow’s 1961 redundancy reduction hypothesis: efficient neural coding should minimize statistical redundancy between neurons. An efficient neural code decorrelates its outputs — no two neurons should carry redundant information. Barlow Twins operationalizes this principle as an SSL objective.

📐 The Cross-Correlation Matrix

Given a batch of \(N\) image pairs, let \(Z^A, Z^B \in \mathbb{R}^{N \times d}\) be the embedding matrices from the two views, each batch-normalized along the batch dimension (zero mean, unit variance per dimension). The cross-correlation matrix \(C \in [-1,1]^{d \times d}\) is:

\[C_{ij} = \frac{\displaystyle\sum_{b=1}^{N} z^A_{b,i}\, z^B_{b,j}}{\displaystyle\sqrt{\sum_{b=1}^{N}\bigl(z^A_{b,i}\bigr)^2} \cdot \sqrt{\sum_{b=1}^{N}\bigl(z^B_{b,j}\bigr)^2}}.\]

Since both matrices are batch-normalized, this simplifies to:

\[C = \frac{1}{N} \bigl(Z^A\bigr)^\top Z^B \in [-1,1]^{d \times d}.\]

Each entry \(C_{ij}\) is the cosine similarity between the \(i\)-th embedding dimension across view A and the \(j\)-th embedding dimension across view B.

🎯 The Barlow Twins Objective

\[\mathcal{L}_{\text{BT}} = \underbrace{\sum_{i=1}^{d} (1 - C_{ii})^2}_{\text{invariance term}} + \lambda \underbrace{\sum_{i=1}^{d} \sum_{j \neq i} C_{ij}^2}_{\text{redundancy-reduction term}}\]

Invariance term \((C_{ii} \to 1)\): Each diagonal entry \(C_{ii}\) is the correlation between the \(i\)-th dimension of \(Z^A\) and the \(i\)-th dimension of \(Z^B\) across the batch. Driving \(C_{ii} \to 1\) forces the representation to be invariant to augmentations along every dimension independently.

Redundancy-reduction term \((C_{ij} \to 0,\; i \neq j)\): Each off-diagonal entry \(C_{ij}\) is the correlation between different dimensions \(i\) (from view A) and \(j\) (from view B). Driving \(C_{ij} \to 0\) decorrelates the output dimensions — no two dimensions carry redundant information.

Combined target: \(C = I_d\). This is exactly the whitening condition — the embeddings from the two views are mutually decorrelated and individually normalized.

🔑 Why Doesn’t Barlow Twins Collapse?

If all embeddings collapse to a constant \(z = z' = \mathbf{c}\), then after batch normalization (which enforces zero mean and unit variance per dimension), the variance of each dimension collapses to zero — but BN explicitly prevents this by design. More fundamentally:

• The invariance term alone would be satisfied by collapse (any constant gives \(C_{ii} = 1\)).
• The redundancy-reduction term creates pressure for different dimensions to carry different information. At non-degenerate embeddings, the gradient of \(\sum_{i \neq j} C_{ij}^2\) with respect to \(z^A_{b,i}\) is non-zero and pushes dimensions apart.

The two terms work in tandem: invariance ensures view-consistency; redundancy reduction ensures richness (full-rank embeddings).

No large batches required. Barlow Twins does not need many negatives — the cross-correlation matrix \(C\) requires only enough samples to estimate \(d \times d\) correlations reliably. A batch of \(N = 2048\) suffices in practice.

🔧 VICReg: Explicit Regularization

Variance-Invariance-Covariance Regularization (Bardes, Ponce, LeCun, ICLR 2022) addresses collapse through three explicit regularization terms applied independently to each branch — without weight sharing, batch normalization of embeddings, or any cross-branch coupling beyond the invariance term.

🏛️ Architecture

VICReg uses two networks (which may be architecturally different — no weight sharing required) producing embeddings \(Z, Z' \in \mathbb{R}^{N \times d}\) for the two views. The variance and covariance regularizers are computed per branch independently.

📐 The Three Terms

1. 📏 Variance regularization (prevents norm collapse):

\[v(Z) = \frac{1}{d} \sum_{j=1}^{d} \max\!\Bigl(0,\; \gamma - \sqrt{\mathrm{Var}_b(z^j) + \varepsilon}\Bigr)\]

where \(z^j \in \mathbb{R}^N\) is the \(j\)-th column of \(Z\) (all batch values for dimension \(j\)), \(\mathrm{Var}_b(z^j) = \frac{1}{N}\sum_b (z_{b,j} - \bar{z}_j)^2\), and \(\gamma = 1\), \(\varepsilon > 0\) is a stability constant. This is a hinge loss on the per-dimension standard deviation: penalize if \(\sigma_j < \gamma\), no penalty if \(\sigma_j \geq \gamma\).

Failure mode prevented: Norm collapse — all samples map to the same value along dimension \(j\), so \(\sigma_j \to 0\).

2. 🔀 Covariance regularization (prevents dimensional collapse):

\[c(Z) = \frac{1}{d} \sum_{i \neq j} \bigl[\hat{C}(Z)\bigr]_{ij}^2, \qquad \hat{C}(Z) = \frac{1}{N-1}\sum_{b=1}^{N} (z_b - \bar{z})(z_b - \bar{z})^\top\]

This penalizes off-diagonal entries of the within-branch sample covariance matrix. Driving \(c(Z) \to 0\) forces embedding dimensions to be pairwise uncorrelated.

Failure mode prevented: Dimensional collapse — variance concentrated in a low-dimensional subspace, with most dimensions carrying redundant linear combinations of a few directions.

3. 📐 Invariance (aligns corresponding views):

\[s(Z, Z') = \frac{1}{N} \sum_{b=1}^{N} \|z_b - z'_b\|^2\]

Standard MSE between corresponding embeddings from the two branches.

🎯 The VICReg Objective

\[\mathcal{L}_{\text{VICReg}} = \lambda\, s(Z, Z') + \mu\bigl[v(Z) + v(Z')\bigr] + \nu\bigl[c(Z) + c(Z')\bigr]\]

Default hyperparameters: \(\lambda = \mu = 25\), \(\nu = 1\).

🔑 Why Doesn’t VICReg Collapse?

The three terms address three distinct failure modes:

At the optimum of \(\mathcal{L}_{\text{VICReg}}\): each dimension has \(\sigma_j \geq \gamma\) (variance), dimensions are decorrelated \(\hat{C}(Z) \approx \sigma^2 I_d\) (covariance), and corresponding embeddings agree \(z \approx z'\) (invariance). This requires \(Z\) and \(Z'\) to be individually whitened (up to a scale) and jointly aligned.

🗺️ Unified Perspective

The methods form a clean taxonomy based on how each prevents collapse:

🔑 Three Orthogonal Principles

Principle 1: Repulsion. Contrastive methods prevent collapse by explicitly pushing apart embeddings of different images. Sufficient but expensive: requires negatives, large batches, or a memory bank.

Principle 2: Gradient asymmetry. BYOL’s stop-gradient + EMA creates a directional optimization landscape — the online network chases the target, but gradients flow only through the online branch. This asymmetry prevents the trivial fixed point where both networks agree on a constant.

Principle 3: Distributional constraint. Barlow Twins and VICReg impose constraints on the statistical structure of the embedding distribution. Barlow Twins cross-couples both branches via \(C \to I_d\); VICReg regularizes each branch independently. Both prevent dimensional collapse by requiring embeddings to span the full \(d\)-dimensional space. VICReg is strictly more architecturally flexible — it drops the requirement for BN on embeddings and weight sharing, making it the most modular of the three.

📈 Empirical Performance

All three post-contrastive methods reach competitive performance with SimCLR/MoCo on ImageNet linear evaluation:

Numbers are approximate and depend on backbone/augmentation details.

📚 References