GSLM2: Multi-Stream Speech Representation Learning Narrows the Gap Between Speech and Text Models

Speech language models (SLMs) hold promise for learning linguistic structure directly from speech, but they still lag significantly behind text models in capability and data scaling efficiency. Meta's research team identifies two bottlenecks behind this gap: the limited linguistic quality of existing speech units, and reliance on single-stream discrete units to represent speech. To address this, they propose GSLM2, which narrows the gap through multi-level, multi-stream representation learning.

Two Bottlenecks: Speech Unit Quality and Single-Stream Representation

Current generative speech language models typically encode speech into sequences of discrete units and then model them like a language model. However, these units tend to be more acoustic than linguistic. Meanwhile, although multi-codebook approaches are widely used in acoustic feature encoding, they are rarely used to represent speech semantics. The core idea of GSLM2 is to have the model learn complementary speech attributes across multiple levels of abstraction, rather than compressing all information into a single stream.

Multi-codebook vs. single-stream discrete units: In neural speech coding, multi-codebook refers to quantizing different components of speech with multiple parallel discrete codebooks (each containing a number of codewords) and then combining them to reconstruct the waveform; typical examples include acoustic codecs such as EnCodec and SoundStream. This approach preserves high-fidelity acoustic detail at low bitrates and has therefore long been used for speech compression and generation. Single-stream discrete units, by contrast, compress each frame of speech into a single sequence of discrete symbols—the form closest to text tokens and convenient for direct language model modeling—but with limited information capacity, it easily mixes acoustic information such as prosody and timbre with linguistic content in the same stream. GSLM2 starts from the premise that since speech carries multiple layers of information—phonemes, syllables, words, syntax—it should be represented hierarchically with multi-stream, multi-level units, much like multi-codebook coding, rather than forced into a single sequence.

SpidR 2.0: Multi-Level High-Quality Discrete Units

GSLM2's encoder, named SpidR 2.0, uses self-supervised learning and introduces differentiable prediction heads to induce high-quality discrete units at multiple levels. Compared with prior methods, SpidR 2.0 aims to produce units of higher linguistic quality, laying the foundation for subsequent multi-stream modeling. This design directly targets the first bottleneck—improving the linguistic quality of speech units.

Self-supervised speech representation and differentiable prediction heads: Self-supervised learning refers to learning representations by constructing training objectives from speech itself (e.g., contrastive learning, masked prediction) rather than relying on human transcriptions; wav2vec 2.0, HuBERT, and WavLM are representative works. The SpidR series further discretizes continuous representations on this basis. Differentiable prediction heads refer to attaching several lightweight prediction networks on top of the encoder output, each responsible for a prediction target at one level of abstraction (e.g., acoustic frame level, phoneme level, word level); because the entire path is end-to-end differentiable, gradients from discretization (e.g., vector quantization or Gumbel-Softmax) can flow back through these heads, allowing the encoder to be optimized directly for "producing high-quality discrete units" rather than learning continuous features first and clustering afterward. This is the key mechanism by which SpidR 2.0 can simultaneously induce higher-quality units at multiple levels.

Multi-Stream SLM: Shared Backbone and Cross-Stream Dependencies

After obtaining multi-level units, GSLM2 uses a multi-stream SLM for joint modeling. The model contains a shared temporal backbone, stream-specific predictors, and an autoregressive head that captures cross-stream dependencies. This architecture allows the model to process speech information at multiple levels of abstraction simultaneously rather than relying on a single stream. Through the shared backbone, the model uses data efficiently; through stream-specific predictors and the autoregressive head, it captures complementary relationships among streams.

Shared backbone and stream-specific predictors: Multi-stream modeling draws on ideas from multi-task learning and mixture-of-experts. The shared temporal backbone (typically a Transformer) reuses the same set of parameters across all streams, so units at different levels of abstraction share temporal context and amortize training data requirements; stream-specific predictors retain independent output heads for each stream, preventing statistical characteristics at different levels from interfering with one another. The cross-stream autoregressive head models dependencies between streams—for example, high-level semantic units constraining low-level acoustic units. This "shared + separate" structure balances parameter count and data efficiency, and is the architectural basis for GSLM2's stable gains as data and parameters scale.

Scaling Laws: Consistent Improvements with Data and Parameter Scale

The team conducted experiments at up to 6 million hours of training data and model scale of 6.5 billion parameters. Results show that GSLM2 consistently improves linguistic ability across different data and model scales, with higher data scaling efficiency. This means the multi-stream representation approach is not only effective but also highly scalable, offering a new path for data scaling of speech models.

Evaluation Performance: The Speech-Text Gap Narrowed Substantially

In read and conversational speech domains, GSLM2 substantially narrows the gap between speech models and text models on three tasks: sWuggy, sBLIMP, and tSC. According to the paper, the gap is reduced by about two orders of magnitude, while surpassing competing speech-text systems on all three tasks. This result shows that combining high-quality multi-level units with multi-stream modeling can significantly improve the linguistic competence of speech models.

sWuggy, sBLIMP, and tSC: These three tasks form a zero-shot speech language understanding evaluation suite used to measure whether a model has acquired linguistic knowledge similar to text models, without any fine-tuning. sWuggy presents a real pseudoword and a nonword altered by one phoneme, asking the model to judge which is more like a real word, testing lexical and phonological knowledge; sBLIMP presents pairs of grammatically correct and incorrect sentences (e.g., subject-verb agreement, negative concord), asking the model to choose the valid sentence, testing syntactic ability; tSC, or Topic Score Comprehension, tests semantic understanding with sentence pairs in which keywords are substituted. All three are evaluated in a "pairwise comparison, choose the higher probability" format, so they can directly compare the linguistic competence of speech models and text models on the same scale, making them a common yardstick for measuring the speech-text gap.

Open Source and Significance

GSLM2's training code and model checkpoints are open-sourced on GitHub (https://github.com/facebookresearch/gslm2). For speech language model research, this work provides a reproducible baseline and demonstrates the potential of multi-stream representation learning for narrowing the speech-text gap. In the future, further optimizing the linguistic quality of units, exploring more levels of abstraction, and extending the method to more speech domains remain directions worth attention.

One-line Summary

Meta proposes GSLM2, using multi-level multi-stream speech units to substantially narrow the linguistic capability gap between speech and text models.

Paragraph Summary

Meta's research team identifies two bottlenecks behind speech language models' lag behind text models: the limited linguistic quality of discrete speech units, and the use of only single-stream units to represent speech. They propose GSLM2: the encoder SpidR 2.0 generates high-quality discrete units at multiple levels of abstraction through self-supervised learning and differentiable prediction heads; a multi-stream SLM then jointly models them with an architecture comprising a shared temporal backbone, stream-specific predictors, and an autoregressive head capturing cross-stream dependencies. At up to 6 million hours of data and 6.5 billion parameters, GSLM2's linguistic ability improves consistently with data and parameter scaling, with higher data scaling efficiency. On three zero-shot tasks—sWuggy, sBLIMP, and tSC—it narrows the gap with text models by about two orders of magnitude and surpasses competing speech-text systems. Code and checkpoints are open-sourced.