SteerTTS: A 23M-Parameter TTS Model with Hybrid Attention and Bidirectional Mamba-3 for Fine-Grained Prosody Control

In speech synthesis, controllability has always been tricky. Most TTS models can generate natural speech, but struggle to adjust speaking rate, pitch, or energy on demand—especially when you want to dynamically adjust prosody on every turn of a conversation. A Reddit user released a project called SteerTTS that attempts to solve this with just 23M parameters (including a frozen Vocos vocoder).

Architecture: Explicit variance modeling + hybrid sequence modeling

SteerTTS follows a FastPitch-style explicit variance modeling path: phonemes first pass through an encoder, where three predictors for duration, pitch, and energy each output control quantities, then are aligned through a length regulator, and finally decoded into 100-band log-mel spectrograms, which are synthesized into waveforms by a frozen Vocos vocoder. According to the author's description in the Reddit post, this design gives each control dimension its own prediction head, making subsequent conditional intervention easier.

The real differentiation lies in how sequence modeling is done inside each encoder and decoder block. Each block consists of multi-head self-attention, a bidirectional Mamba-3 branch, and a convolutional FFN in sequence. The Mamba-3 branch is formed by summing forward and backward layers, using complex-valued states and trapezoidal discretization. This hybrid structure attempts to use the state space model to handle local temporal details while attention captures global dependencies, reducing computational overhead.

Another key design is that all LayerNorms are replaced with adaptive versions (AdaLN), and a masking mechanism distinguishes "unspecified" from "explicitly set to zero." The author particularly emphasizes: when a control item is omitted, the model infers reasonable values from timbre, tags, and text; while passing {"rate": 0} explicitly requests the neutral speaking rate for that timbre. This distinction is important for automated pipelines—you don't want a missed parameter to be misinterpreted as "silence" or "zero rate."

Explicit variance modeling refers to decoupling prosodic attributes such as pitch (F0), energy, and duration from acoustic features in the TTS pipeline, using dedicated prediction heads to regress these scalar curves at the phoneme level separately, and then upsampling phoneme-level predictions to frame level through a length regulator. Compared to end-to-end implicit modeling, this approach gives each prosodic dimension an explicit intermediate representation, so at inference time you can directly perform numerical intervention, extrapolation, or replacement on these intermediate quantities without perturbing the entire acoustic model. FastSpeech 2 is the representative work of this paradigm, and FastPitch further feeds pitch embeddings back to the decoder to enhance pitch controllability; the cost is that predictors may entangle with each other, and cross-leakage of controls needs to be measured separately in evaluation.

Mamba is a class of sequence modeling structures based on selective state space models (SSM), whose core is to control state transitions with input-dependent parameters, thereby achieving content-dependent information routing like attention while maintaining inference and training costs that grow linearly with sequence length. Mamba-3 refers to a newer iteration of this series, introducing improvements such as complex-valued states and trapezoidal discretization to more precisely approximate continuous-time systems. The bidirectional variant described in the article sums the outputs of forward and backward SSM branches, so that each position obtains both left and right context simultaneously, which is conceptually akin to bidirectional RNNs. Compared with attention, SSMs excel at capturing smooth, local temporal dynamics but lack explicit pairwise interactions; placing both in parallel within the same block is precisely an attempt to strike a balance between long-range dependencies and local details.

Steering and refinement: Classifier guidance + flow matching refinement

At inference, SteerTTS adopts a strategy similar to classifier guidance: rendering a conditional version and an unconditional version separately, then extrapolating over duration, pitch, energy, and mel spectrograms. This can amplify control signals without retraining.

An optional flow matching refiner (ConvNeXt backbone, 10 midpoint steps) is used to enhance mel details, at the cost of roughly 12x computation. There is also a Voicebox-style prompt variant that supports voice cloning. The model totals 9.6M parameters for the acoustic part plus 13.5M for Vocos, with the refiner adding an additional 17.6M.

Classifier-Free Guidance (CFG) was originally used in diffusion models: during training, conditions are randomly dropped so that the same network can produce both conditional and unconditional predictions; at inference, extrapolation is performed as ε = ε_uncond + s·(ε_cond − ε_uncond), amplifying the difference in the conditional direction to enhance condition adherence, at the cost of reduced diversity. SteerTTS ports this idea to prosody control, treating "control conditions" as the guidance direction and extrapolating separately over duration, pitch, energy, and mel spectrograms, thus increasing control magnitude without retraining. The value of guidance strength s is a trade-off: if s is too small, the requested magnitude is under-delivered; if s is too large, audio quality and cross-interference worsen.

Flow Matching is a continuous-time generative modeling method that learns a velocity field from a simple prior to the data distribution, with the training objective of regressing that velocity field, and sampling performed by integrating with an ODE solver. The so-called "midpoint steps" refer to using second-order midpoint integration to obtain finer mel reconstruction with fewer steps; the 10 steps and roughly 12x computation mentioned in the article refer to the cost of this optional refinement path.

Control precision in practice: Four dimensions strong, three dimensions weak

The author evaluated on held-out LibriTTS-R speakers with requests of ±2 standard deviations, and the results show clear divergence. Spearman correlation coefficients for loudness, median pitch, H1–H2, and spectral tilt range from 0.87 to 0.97, indicating that requested values and read-back values are highly consistent. However, pauses, voicing proportion, and CPPS (cepstral peak prominence) perform weakly, with ρ only 0.46–0.79, and CPPS leaks into pitch and level—when adjusting CPPS, pitch and loudness change as well, showing obvious cross-interference.

Regarding guidance strength, the average gain is about 0.5 at s=1, and at s≈1.5 the requested magnitude is roughly delivered, but cross-interference increases accordingly. Commendably, WER remains stable at about 1% under all requests, indicating that prosody manipulation does not break intelligibility.

Spearman correlation coefficient ρ measures the monotonic consistency between requested control values and read-back measured values of synthesized speech; ρ close to 1 indicates perfectly consistent ordering, and it is a common metric for evaluating prosody controllability. H1–H2 in the article refers to the difference in amplitude between the first and second harmonics, an acoustic correlate of breathy voice; CPPS (cepstral peak prominence) is obtained from the ratio of the height to the width of the smoothed cepstral peak, and is regarded as a composite indicator of overall voice quality and phonatory stability, often correlated with roughness and effort; voicing proportion refers to the proportion of voiced frames in the entire utterance. These quantities are not independently predicted by the model but are indirectly influenced by underlying controls such as pitch and energy, so their weak correlation and cross-leakage reflect inherent acoustic coupling rather than a mere modeling defect.

Performance and limitations

In terms of speed, RTFx is 60–80 on an RTX 5070, and 4–8 on CPU without the refiner. The model supports only English, uses a single training seed, and has not yet undergone human evaluation. Weights are under CC BY-NC 4.0 license, and code under MIT.

The author poses an open question at the end of the post: in small-scale non-autoregressive TTS, how does bidirectional Mamba actually compare with pure attention? He admits that a clean ablation has not yet been done and hopes the community will share experience. This is precisely where hybrid architectures most need evidence—without ablation, it is hard to tell whether the gain from bidirectional Mamba-3 comes from the structure itself or from parameter configuration.

Multi-source perspective and positioning

This release currently has only a single Reddit source, while the Hugging Face model page provides the complete evaluation table. From a technical路线 perspective, SteerTTS is in the same lineage as FastPitch and FastSpeech 2, but introducing Mamba-3 in sequence modeling is a relatively new attempt. Its 23M size and explicit numerical control interface make it more suitable for turn-by-turn prosody adjustment in embedded or dialogue systems, rather than general-purpose TTS pursuing ultimate audio quality.

For developers, the masked AdaLN design is worth borrowing: distinguishing "unspecified" from "zero value" is a detail often overlooked in controllable generation systems. The cross-leakage of CPPS also reminds us that prosodic dimensions are not orthogonal, and drift in other dimensions when explicitly controlling one dimension needs to be measured and compensated.

The project has open-sourced model weights and the evaluation table at huggingface.co/WhissleAI/steer-tts.

One-line Summary

SteerTTS, with 23M parameters, uses explicit variance modeling and a hybrid Mamba-attention structure to achieve numerically steerable prosody TTS.

Paragraph Summary

SteerTTS is an open-source controllable TTS project released by a Reddit user, with 23M total parameters (9.6M acoustic model plus 13.5M frozen Vocos vocoder). It follows FastPitch-style explicit variance modeling, with independent prediction heads for duration, pitch, and energy, and designs each encoder-decoder block as a hybrid of multi-head self-attention, bidirectional Mamba-3, and convolutional FFN. All LayerNorms are replaced with masked AdaLN to distinguish "unspecified" from "explicit zero value." At inference, classifier guidance extrapolates control signals, with an optional flow matching refiner to enhance mel details. Evaluation on LibriTTS-R shows Spearman ρ of 0.87–0.97 for loudness, pitch, H1–H2, and spectral tilt, but only 0.46–0.79 for pauses, voicing proportion, and CPPS, with cross-leakage; WER remains stable at about 1%, and RTFx is 60–80 on an RTX 5070. Limitations include English-only support, a single training seed, lack of human evaluation, and no ablation validation of the bidirectional Mamba gain.