SteerTTS
A lightweight controllable TTS model with only 23M parameters. It employs explicit variance modeling and a hybrid attention-bidirectional Mamba-3 sequence modeling architecture, supports fine-grained prosody control (duration, pitch, energy, and so on), and can combine classifier guidance with a flow-matching refiner to improve synthesis quality.
Timeline (last 90 days)
SteerTTS is a TTS model project with only 23M parameters (including a frozen Vocos vocoder), released by a Reddit user.
SteerTTS adopts FastPitch-style explicit variance modeling: phonemes first pass through an encoder, three predictors for duration, pitch and energy each output control quantities, which are then aligned via a length regulator and finally decoded into 100-band log-mel spectra, handed to a frozen Vocos vocoder for waveform synthesis.
Each encoder and decoder block in SteerTTS contains, in order, multi-head self-attention, a bidirectional Mamba-3 branch and a convolutional FFN, where the Mamba-3 branch is formed by summing the forward and backward layers, using complex-valued states and trapezoidal discretization.
SteerTTS replaces all LayerNorms with adaptive versions (AdaLN) and uses a masking mechanism to distinguish "unspecified" from "explicitly set to zero": when a control item is omitted the model infers a reasonable value from timbre, labels and text, while passing {"rate": 0} explicitly requests that timbre's neutral speaking rate.
At inference, SteerTTS uses a classifier-guidance-like strategy, rendering a conditional version and an unconditional version separately and then extrapolating on duration, pitch, energy and mel spectra, thereby amplifying the control signal without retraining.
SteerTTS's optional flow-matching refiner uses a ConvNeXt backbone with 10 midpoint steps to enhance mel detail, at a cost of roughly 12x compute.
SteerTTS's Voicebox-style prompt variant supports voice cloning.
SteerTTS has 9.6M parameters in total for the acoustic part plus a 13.5M Vocos vocoder, with the refiner adding an extra 17.6M.
Evaluated with ±2 standard deviation requests on held-out LibriTTS-R speakers, SteerTTS achieves Spearman correlation coefficients between 0.87 and 0.97 for loudness, median pitch, H1–H2 and spectral tilt; however, ρ is only 0.46–0.79 for pauses, voicing ratio and CPPS, and CPPS leaks into pitch and level.
SteerTTS delivers an average gain of about 0.5 at guidance strength s=1, and at s≈1.5 roughly fulfills the requested magnitude, but cross-interference increases accordingly; WER stays stable at about 1% across all requests.
5 more timeline events
All Facts (15)
SteerTTS is a TTS model project with only 23M parameters (including a frozen Vocos vocoder), released by a Reddit user.
50%UnverifiedSteerTTS adopts FastPitch-style explicit variance modeling: phonemes first pass through an encoder, three predictors for duration, pitch and energy each output control quantities, which are then aligned via a length regulator and finally decoded into 100-band log-mel spectra, handed to a frozen Vocos vocoder for waveform synthesis.
50%UnverifiedEach encoder and decoder block in SteerTTS contains, in order, multi-head self-attention, a bidirectional Mamba-3 branch and a convolutional FFN, where the Mamba-3 branch is formed by summing the forward and backward layers, using complex-valued states and trapezoidal discretization.
50%UnverifiedSteerTTS replaces all LayerNorms with adaptive versions (AdaLN) and uses a masking mechanism to distinguish "unspecified" from "explicitly set to zero": when a control item is omitted the model infers a reasonable value from timbre, labels and text, while passing {"rate": 0} explicitly requests that timbre's neutral speaking rate.
50%UnverifiedAt inference, SteerTTS uses a classifier-guidance-like strategy, rendering a conditional version and an unconditional version separately and then extrapolating on duration, pitch, energy and mel spectra, thereby amplifying the control signal without retraining.
50%UnverifiedSteerTTS's optional flow-matching refiner uses a ConvNeXt backbone with 10 midpoint steps to enhance mel detail, at a cost of roughly 12x compute.
50%UnverifiedSteerTTS's Voicebox-style prompt variant supports voice cloning.
50%UnverifiedSteerTTS has 9.6M parameters in total for the acoustic part plus a 13.5M Vocos vocoder, with the refiner adding an extra 17.6M.
50%UnverifiedEvaluated with ±2 standard deviation requests on held-out LibriTTS-R speakers, SteerTTS achieves Spearman correlation coefficients between 0.87 and 0.97 for loudness, median pitch, H1–H2 and spectral tilt; however, ρ is only 0.46–0.79 for pauses, voicing ratio and CPPS, and CPPS leaks into pitch and level.
50%UnverifiedSteerTTS delivers an average gain of about 0.5 at guidance strength s=1, and at s≈1.5 roughly fulfills the requested magnitude, but cross-interference increases accordingly; WER stays stable at about 1% across all requests.
50%UnverifiedSteerTTS achieves an RTFx of 60–80 on an RTX 5070, and 4–8 on CPU without the refiner.
50%UnverifiedSteerTTS supports only English, uses a single training seed, and has not yet undergone human evaluation.
50%UnverifiedSteerTTS weights are licensed under CC BY-NC 4.0 and the code under the MIT license.
50%UnverifiedThe authors have not yet done a clean ablation comparing bidirectional Mamba with pure attention in small-scale non-autoregressive TTS.
50%UnverifiedThe SteerTTS project has open-sourced the model weights and evaluation tables at huggingface.co/WhissleAI/steer-tts.
50%