GSLM2
A multi-level, multi-stream speech language model proposed by Meta that narrows the gap in linguistic capability between speech and text models through high-quality discrete units and multi-stream modeling.
Timeline (last 90 days)
The Meta research team proposes GSLM2, which narrows the capability gap between speech language models and text models through multi-level, multi-stream representation learning.
GSLM2's encoder is named SpidR 2.0, which uses self-supervised learning and introduces a differentiable prediction head to induce high-quality discrete units at multiple levels.
GSLM2 uses a multi-stream SLM for joint modeling, comprising a shared temporal backbone network, an independent predictor for each stream, and an autoregressive head that captures cross-stream dependencies.
The research team conducted GSLM2 experiments at up to 6 million hours of training data and a model scale of 6.5 billion parameters.
GSLM2 consistently improves language ability across different data scales and model scales, and its efficiency increases with data scaling.
On the three tasks sWuggy, sBLIMP, and tSC, GSLM2 narrows the gap between speech models and text models by about two orders of magnitude.
GSLM2's training code and model checkpoints have been open-sourced on GitHub at https://github.com/facebookresearch/gslm2.
All Facts (7)
The Meta research team proposes GSLM2, which narrows the capability gap between speech language models and text models through multi-level, multi-stream representation learning.
50%UnverifiedGSLM2's encoder is named SpidR 2.0, which uses self-supervised learning and introduces a differentiable prediction head to induce high-quality discrete units at multiple levels.
50%UnverifiedGSLM2 uses a multi-stream SLM for joint modeling, comprising a shared temporal backbone network, an independent predictor for each stream, and an autoregressive head that captures cross-stream dependencies.
50%UnverifiedThe research team conducted GSLM2 experiments at up to 6 million hours of training data and a model scale of 6.5 billion parameters.
50%UnverifiedGSLM2 consistently improves language ability across different data scales and model scales, and its efficiency increases with data scaling.
50%UnverifiedOn the three tasks sWuggy, sBLIMP, and tSC, GSLM2 narrows the gap between speech models and text models by about two orders of magnitude.
50%UnverifiedGSLM2's training code and model checkpoints have been open-sourced on GitHub at https://github.com/facebookresearch/gslm2.
50%