Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations

Yuhao Yang, Zhi Ji, Zhaopeng Li, Yi Li, Zhonglin Mo, Yue Ding, Kai Chen, Zijian Zhang, Jie Li, Shuanglong Li, Lin Liu — Baidu Inc. ArXiv 2503.02453, 2025.

Relations

Builds on: HSTU (Zhai et al., 2024), TIGER (Rajput et al., NeurIPS 2023) (no note yet), LIGER (Yang et al., 2024) (no note yet) Concepts used: Generative Recommender Systems — §4 (Generative Retrieval, RQ-VAE, COBRA)

Table of Contents

• 1. Motivation: The Information Loss Problem
• 2. Sparse and Dense Representations
  • 2.1 Sparse IDs via RQ-VAE
  • 2.2 End-to-End Learnable Dense Vectors
• 3. Cascaded Sequential Modeling
  • 3.1 Probabilistic Factorization
  • 3.2 Input Construction and Forward Pass
• 4. Training Objectives
  • 4.1 Sparse Cross-Entropy Loss
  • 4.2 Dense Contrastive Loss
• 5. Coarse-to-Fine Inference: BeamFusion
  • 5.1 Sparse ID Beam Search
  • 5.2 Conditional Dense Generation
  • 5.3 BeamFusion Ranking Score
• 6. Experiments
  • 6.1 Public Benchmarks
  • 6.2 Industrial Ablation
  • 6.3 Recall–Diversity Trade-off
• 7. References

1. Motivation: The Information Loss Problem 📐

Generative retrieval via RQ-VAE semantic IDs encodes each item \(x\) as a tuple \((c_1, c_2, \ldots, c_K)\) of discrete codeword indices. The encoder compresses the item’s dense embedding \(\mathbf{z} \in \mathbb{R}^d\) into \(K\) successive quantization residuals; the decoder reconstructs \(\hat{\mathbf{z}} = \sum_{k=1}^K \mathbf{e}_{c_k}\). By construction, the final residual \(\mathbf{r}_K = \mathbf{z} - \hat{\mathbf{z}}\) is discarded.

The information loss is irreducible. For finite \(K\) and codebook sizes, \(\|\mathbf{r}_K\| > 0\) almost surely. Two items can share an identical semantic ID yet differ in their true embeddings — a collision that makes them indistinguishable to the retrieval model. Increasing \(K\) or codebook size reduces the residual but never eliminates it, and increases inference cost.

LIGER (Yang et al., 2024) addressed this by appending a separately pre-trained dense embedding alongside the sparse ID at each sequence position. But using a frozen, separately-trained dense encoder introduces a representation mismatch: the dense vector is not adapted to the generative pretraining objective and carries no collaborative signal from interaction sequences.

COBRA’s solution is a cascade: generate the sparse ID first, then condition the generation of the dense vector on the produced sparse ID. This ordering allows the dense vector to refine the coarse categorical prediction — recovering intra-cluster detail that the sparse ID discards — while avoiding the mismatch of pre-trained fixed embeddings.

2. Sparse and Dense Representations

2.1 Sparse IDs via RQ-VAE

The sparse representation follows the standard RQ-VAE construction (see §4.1 of the concept note for full derivation). Item \(x\) is encoded from its textual attribute description via a Transformer text encoder, then quantized hierarchically into a \(K\)-tuple \((c_1, \ldots, c_K)\).

Each level has a codebook of \(V\) entries; \(K\) is the number of quantization levels. COBRA uses \(K=3\), \(V=32\) for public datasets and \(K=2\), \(V=32\) for the industrial dataset. The codebook is trained with EMA updates and random last-level assignment to achieve $$99% code uniqueness (see codebook collapse mitigation).

2.2 End-to-End Learnable Dense Vectors

Definition (COBRA Dense Encoder). A Transformer encoder processes the flattened item attribute tokens prefixed with a [CLS] token. The dense vector for item \(x\) is the [CLS] hidden state:

\[\mathbf{v}_x = \text{TransformerEncoder}([\texttt{[CLS]}, w_1, w_2, \ldots, w_L])_{[0]}\]

where \(w_1, \ldots, w_L\) are the item’s attribute tokens and the subscript \([0]\) selects the [CLS] position.

Critically, this encoder is trained end-to-end with the contrastive loss of §4.2 — not pre-trained and frozen. The dense vector therefore adapts to the recommendation objective: items that co-occur in user interaction sequences are pulled together in dense space, regardless of their attribute similarity.

3. Cascaded Sequential Modeling 🔑

3.1 Probabilistic Factorization

Definition (COBRA Factorization). Let \(S_{1:t} = ((x_1, \mathbf{v}_1), \ldots, (x_t, \mathbf{v}_t))\) be the user’s interaction history where \(x_i\) has sparse ID \(\text{ID}_i\) and dense vector \(\mathbf{v}_i\). COBRA factorizes the joint predictive distribution as:

\[P(\text{ID}_{t+1}, \mathbf{v}_{t+1} \mid S_{1:t}) = \underbrace{P(\text{ID}_{t+1} \mid S_{1:t})}_{\text{sparse prediction}} \cdot \underbrace{P(\mathbf{v}_{t+1} \mid \text{ID}_{t+1}, S_{1:t})}_{\text{dense refinement}}\]

The factorization is the product rule applied with \(\text{ID}_{t+1}\) as the first variable. This is not an independence assumption — the dense prediction is explicitly conditioned on the sparse ID.

Intuition. The sparse ID answers “which cluster does the next item belong to?” The dense vector, conditioned on the sparse ID, answers “where within that cluster is the next item?” The cascade decomposes a single hard prediction into two simpler sequential steps.

3.2 Input Construction and Forward Pass

Each sequence position \(t\) is represented by concatenating the sparse ID embedding with the dense vector:

\[\mathbf{h}_t = [\mathbf{e}(\text{ID}_t);\ \mathbf{v}_t] \in \mathbb{R}^{2d}\]

where \(\mathbf{e}(\text{ID}_t) = \sum_{k=1}^K \mathbf{e}_{c_k^{(t)}}\) is the sum of codeword embeddings. The full sequence history is thus:

\[S_{1:t} = [\mathbf{h}_1, \mathbf{h}_2, \ldots, \mathbf{h}_t] \in \mathbb{R}^{t \times 2d}\]

Sparse ID forward pass. A causal Transformer decoder processes \(S_{1:t}\) and a SparseHead projects the output at position \(t\) to logits over the codebook:

\[\mathbf{z}_{t+1} = \text{SparseHead}(\text{TransformerDecoder}(S_{1:t}))\]

Dense vector forward pass. The predicted sparse ID embedding \(\mathbf{e}(\widehat{\text{ID}}_{t+1})\) is appended to the history, and a DenseHead projects the decoder output at the new final position:

\[\bar{S}_{1:t} = [S_{1:t};\ \mathbf{e}(\widehat{\text{ID}}_{t+1})], \qquad \hat{\mathbf{v}}_{t+1} = \text{DenseHead}(\text{TransformerDecoder}(\bar{S}_{1:t}))\]

Note that the dense head sees the predicted sparse ID — not the ground-truth — at training time. The model is therefore trained to produce dense vectors conditioned on the predicted cluster, which matches inference behavior.

4. Training Objectives

4.1 Sparse Cross-Entropy Loss

The sparse ID loss is standard cross-entropy over codebook tokens at each level \(k \in [K]\). For a sequence of length \(T\):

\[\mathcal{L}_{\text{sparse}} = -\sum_{t=1}^{T-1} \log \frac{\exp\!\left(z_{t+1}^{\text{ID}_{t+1}}\right)}{\sum_j \exp\!\left(z_{t+1}^j\right)}\]

where \(z_{t+1}^j\) is the \(j\)-th logit of \(\mathbf{z}_{t+1}\).

4.2 Dense Contrastive Loss

The dense loss aligns the predicted vector \(\hat{\mathbf{v}}_{t+1}\) with the ground-truth item dense vector \(\mathbf{v}_{t+1}\) via an InfoNCE objective with in-batch negatives:

\[\mathcal{L}_{\text{dense}} = -\sum_{t=1}^{T-1} \log \frac{\exp\!\left(\cos(\hat{\mathbf{v}}_{t+1}, \mathbf{v}_{t+1}) / \tau\right)}{\sum_{j \in \mathcal{B}} \exp\!\left(\cos(\hat{\mathbf{v}}_{t+1}, \mathbf{v}_j) / \tau\right)}\]

where \(\mathcal{B}\) is the in-batch item set and \(\tau\) is a temperature hyperparameter. The cosine similarity metric is invariant to embedding scale, which stabilizes training when the encoder and decoder operate at different magnitudes.

Combined objective:

\[\mathcal{L} = \mathcal{L}_{\text{sparse}} + \mathcal{L}_{\text{dense}}\]

The two terms are summed without weighting hyperparameters — the ablation shows both are necessary and they operate on disjoint output heads.

5. Coarse-to-Fine Inference: BeamFusion 💡

5.1 Sparse ID Beam Search

At inference, beam search with width \(M\) generates the top-\(M\) sparse ID candidates from the model’s distribution:

\[\{\widehat{\text{ID}}_{T+1}^k\}_{k=1}^M = \text{BeamSearch}\!\left(\text{TransformerDecoder}(S_{1:T}),\ M\right)\]

Each beam \(k\) carries an associated log-probability \(\phi^k = \log P(\widehat{\text{ID}}_{T+1}^k \mid S_{1:T})\).

5.2 Conditional Dense Generation

For each beam \(k\), the sparse ID embedding is appended to the history and the DenseHead generates a conditional dense vector:

\[\hat{\mathbf{v}}_{T+1}^k = \text{DenseHead}\!\left(\text{TransformerDecoder}\!\left([S_{1:T};\ \mathbf{e}(\widehat{\text{ID}}_{T+1}^k)]\right)\right)\]

An ANN index over pre-computed item dense vectors is then queried for each beam:

\[\mathcal{A}_k = \text{ANN}\!\left(\hat{\mathbf{v}}_{T+1}^k,\ \mathcal{C}(\widehat{\text{ID}}_{T+1}^k),\ N\right)\]

where \(\mathcal{C}(\widehat{\text{ID}}_{T+1}^k)\) restricts the search to items in the beam’s code cluster, and \(N\) is the number of retrieved neighbors per beam.

5.3 BeamFusion Ranking Score

Each candidate item \(a \in \bigcup_{k=1}^M \mathcal{A}_k\) is scored by combining the sparse beam probability and the dense cosine similarity:

\[\Phi(\hat{\mathbf{v}}_{T+1}^k, \widehat{\text{ID}}_{T+1}^k, a) = \underbrace{\text{Softmax}(\tau \cdot \phi^k)}_{\text{sparse beam weight}} \cdot \underbrace{\text{Softmax}(\psi \cdot \cos(\hat{\mathbf{v}}_{T+1}^k, \mathbf{a}))}_{\text{dense similarity weight}}\]

The hyperparameters \(\tau\) and \(\psi\) control the precision–diversity trade-off: higher \(\tau\) concentrates weight on high-probability beams (lower diversity); higher \(\psi\) concentrates weight on the dense nearest neighbors (lower recall from lower-probability beams). Empirically, the optimal balance is \(\tau = 0.9\), \(\psi = 16\).

Final retrieval:

\[\mathcal{R} = \text{Top-}K\!\left(\bigcup_{k=1}^M \mathcal{A}_k,\ \Phi\right)\]

6. Experiments 📊

6.1 Public Benchmarks

COBRA is evaluated on three Amazon product review datasets (Beauty, Sports and Outdoors, Toys and Games) against TIGER and dense sequential baselines (SASRec, BERT4Rec, RecFormer, LIGER).

COBRA improves over TIGER by 9–23% across all metrics, with larger gains on NDCG (which penalizes lower-ranked correct items more heavily) than Recall — consistent with BeamFusion’s improvement in ranking quality, not just top-\(K\) coverage.

6.2 Industrial Ablation

Evaluated on Baidu’s advertising dataset (200M DAU). The ablation isolates the contribution of each component:

Both branches are necessary. Removing the sparse branch hurts more at low recall cutoffs (R@50: −48.2%) because beam search over semantic IDs is the primary structuring mechanism. Removing the dense branch hurts more at high recall cutoffs (R@800: −26.7%) because the dense vector is what distinguishes items within a cluster once the cluster is identified.

Online A/B test (January 2025, 10% traffic): +3.60% conversion rate, +4.15% ARPU.

6.3 Recall–Diversity Trade-off

The BeamFusion parameters \(\tau\) and \(\psi\) trade recall (broad candidate coverage) against diversity (distributing candidates across semantic clusters). Optimal performance at \(\tau = 0.9\), \(\psi = 16\); increasing \(\tau\) beyond 1.0 collapses the retrieval toward the single highest-probability beam, reducing diversity with diminishing recall gains.

7. References