Lafa

Lafa is a data-efficient continuous-flow language model scaled to 8B parameters, employing a block-wise flow architecture.

In Lafa's block-wise flow architecture, the prefix uses causal attention, and within each block a bidirectional, asynchronous flow is used.

The research team trained Lafa at three scales—0.5B, 7B, and 8B—each using fewer than 30B tokens.

Lafa's training starts from a pretrained autoregressive model and adapts it through flow training, which, via the multi-token prediction effect, preserves or even improves the capabilities of the pretrained AR model.

Evaluation on instruction-following, reasoning, and coding tasks shows that the flow models learned by Lafa are comparable to or even better than autoregressive models in quality.

Lafa's flow model can be distilled into flow maps, enabling single-step or few-step generation and thereby reducing inference latency.

Lafa designs a self-speculative decoding scheme that uses flow-based drafts for autoregressive verification within a single model, requiring no additional draft model parameters or memory overhead.

At comparable quality, Lafa's self-speculative decoding scheme achieves 2.54x the throughput of autoregressive models, outperforming multi-token prediction and discrete diffusion methods.

Lafa's research shows that continuous flow can serve as a scalable foundation for language modeling, paving the way for the practical use of flow language models.

Current Lafa research is still limited to the 8B parameter scale, and performance at larger scales remains to be validated.

The impact of hyperparameter choices (such as block size and number of flow steps) in Lafa's block-wise flow architecture on performance has not yet been systematically analyzed.