No Language Pretraining Needed: Halcyon Trains Strong T2I Models from Scratch

Introduction: Is Language Pretraining Necessary for T2I?

Modern text-to-image (T2I) systems almost all rely on internet-scale language pretraining to understand text conditions. But a research team at Stanford University poses a fundamental question in the paper "Text-to-Image Generative Modeling Without Language Pretraining": Is language pretraining really necessary for strong T2I generation? Their answer is no—by introducing the Halcyon series of models, they demonstrate that training from scratch can match or even surpass baselines that rely on language pretraining.

Autoregressive models are a class of models that generate data by sequentially predicting the next element in a sequence. In text-to-image tasks, autoregressive models encode images into discrete token sequences and model them jointly with text tokens, thereby generating both text and images simultaneously in a "predict the next token" manner. This paradigm differs from diffusion models, which generate images through iterative denoising. Halcyon adopts an autoregressive objective, meaning it directly maximizes the likelihood of the joint image-text sequence, naturally supporting multimodal fusion.

Halcyon: An Early-Fusion Multimodal Autoregressive Model

Halcyon is a family of early-fusion multimodal autoregressive models, pretrained entirely from scratch on the GPIC dataset. GPIC contains 100 million image-text pairs, far smaller than typical internet-scale language corpora. Unlike mainstream approaches, Halcyon does not load any pretrained language model weights; instead, it fuses text and image tokens at an early stage and trains them uniformly with an autoregressive objective.

This design allows the model to learn image-text alignment from the very beginning, rather than first learning language and then adapting to images. The research team trained versions of different scales, including Halcyon-0.52B and Halcyon-1.6B, to verify the scalability of the method.

Early fusion refers to mapping tokens from different modalities (such as text and images) into the same representation space at the model input stage and letting them interact within the same Transformer. In contrast, late fusion typically encodes each modality separately first and then aligns them at higher layers. The advantage of early fusion is that the model can learn fine-grained cross-modal associations from the very first layer, avoiding the representation gap between modalities. Halcyon is trained entirely from scratch without a pretrained language model as a text encoder, making early fusion a key design for its unified modeling of images and text.

Performance Breakthrough: Less Compute, Better Results

On the generation quality metric gFD (generative Fréchet distance, lower is better), Halcyon's performance is impressive:

• Halcyon-0.52B achieves 48.0 gFD, outperforming LlamaGen-XL/8's 63.3 gFD, while using only 1/18.2 of the latter's pretraining compute.
• Halcyon-1.6B achieves 35.3 gFD, outperforming JiT-T2I-XXL/8++'s 41.0 gFD, with pretraining compute just 1/275 of the latter's.

This means that not only is language pretraining unnecessary, but it can also substantially reduce computational costs. The compute efficiency advantage of the 1.6B version in particular is practically attractive for resource-constrained research teams and companies.

gFD (generative Fréchet distance) is a commonly used metric for evaluating generated image quality, drawing on the idea of FID (Fréchet Inception Distance). It computes the Fréchet distance between the distributions of generated and real images in a feature space; a lower value indicates that the generated distribution is closer to the real distribution, i.e., higher image quality and better diversity. gFD typically uses a pretrained feature extractor (such as an Inception network) to embed images, so its absolute value is affected by the feature extractor, but relative comparisons under the same experimental setup are meaningful.

Controlled Experiments: Text Pretraining Brings No Gain

To clarify the role of language pretraining, the authors conducted controlled experiments. The results show that text-only pretraining does not improve T2I generation quality. This finding challenges the common assumption that "language understanding is a prerequisite for text-to-image generation." Instead, the early-fusion autoregressive objective itself enables the model to learn sufficient semantic alignment from image-text pairs.

In addition, the team explored multi-task learning: by replacing caption dropout with image-to-text (I2T) supervision, Halcyon simultaneously learns image captioning while maintaining competitive T2I quality under the same pretraining budget. This shows that the model can acquire inverse-task capabilities without sacrificing generation ability.

Caption dropout is a regularization technique commonly used in multimodal training: during training, the text description (caption) is dropped with a certain probability, forcing the model to generate based only on the image or partial context, thereby enhancing robustness and preventing over-reliance on text. The Halcyon team replaced it with image-to-text (I2T) supervision, i.e., explicitly training the model to generate descriptions from images. This multi-task learning approach not only preserves generation ability but also equips the model with inverse-task (image captioning) capability without additional pretraining budget.

Implications and Outlook

Halcyon's work opens a new path for T2I generation: breaking free from reliance on massive language models and building multimodal systems from scratch. Its significance lies not only in lowering the compute barrier but also in inspiring us to rethink the objective functions and data efficiency of multimodal pretraining. Of course, the current experiments are limited to the GPIC dataset and specific model scales; future work needs to verify generalization to larger scales and more diverse data. But at least Halcyon has proven—language pretraining is not a necessity for T2I.

Code and models are publicly available on the project page: https://cs.stanford.edu/~keshik/halcyon