OrderGPT: Learning Generation Order as a Modeling Choice — A New Breakthrough in AI Modeling

Discrete data generative modeling has long been dominated by two paradigms: GPT-style autoregressive models (generating token by token from left to right) and masked diffusion models (generating in arbitrary order). These seemingly opposed approaches are actually two extremes on the same axis — generation order. For structured discrete data without a natural traversal order (such as human pose or tabular data), neither extreme may be optimal. A recent paper titled OrderGPT proposes that the intermediate "single adapted order" is an overlooked middle ground, and presents a method that lets the model learn the generation order itself.

From Two Poles to a Continuum: Rethinking Generation Order

Autoregressive models like GPT decompose sequence probability into a left-to-right chain of conditional probabilities. This order is natural for natural language because language itself has a linear structure. Masked diffusion models allow generation in arbitrary order — essentially any-order autoregression. But many discrete domains have no natural notion of "left to right." Human pose, for example, consists of multiple joint coordinates, and tabular data consists of multiple fields. Forcing a left-to-right order onto such data may introduce unreasonable dependency assumptions.

The core insight of OrderGPT is that generation order itself can be a learnable modeling choice rather than a pre-fixed hyperparameter. Rather than choosing either extreme, the paper explores the middle path of "learning a single task-level order."

Any-Order Autoregressive modeling is a generalization of autoregressive modeling: it does not require the model to decompose the joint probability strictly in left-to-right order, but allows variables to be conditionally predicted one by one in any permutation. Mathematically, ( p(x) = \prod_{t} p(x_{\pi_t} \mid x_{\pi_{<t}}) ), where (\pi) is a permutation of the variables. Masked diffusion models can be viewed as a special case of any-order autoregression — at each step, a number of positions are randomly selected for prediction, and the predicted positions are gradually added to the "known" set. The importance of this perspective is that autoregression and diffusion are not fundamentally opposed model types; they differ only in the degree of constraint on generation order. OrderGPT leverages exactly this continuum framework, turning order from an externally given quantity into an internally learned one.

How Does OrderGPT Learn Generation Order?

OrderGPT builds on Any-Order GPT, which supports prediction in any order but requires the order to be explicitly given. OrderGPT's innovation is that it can learn a task-appropriate generation order from data without canonical-order supervision.

The mechanism has three steps:

1. Extract dependency signals from random prefixes: During training, the model predicts variables from randomly sampled reveal prefixes. The key observation is that such prediction produces recurring attention patterns in the backbone network, and these patterns provide model-relevant predictive dependency signals.
2. Aggregate into a dependency graph and derive priorities: The attention signals are aggregated into a reciprocal dependency graph, and a spectral teacher is then used to derive continuous generation priorities.
3. Lightweight OrderMLP predicts the order: A lightweight OrderMLP integrated into the GPT backbone learns to predict these priorities from attention, incorporating order learning into the architecture and completing it in a single training run.

The training process gradually shifts toward the learned order, after which the order is frozen and used for subsequent autoregressive training and generation. This means the model explores the order early in training and fixes it later to stabilize generation.

Spectral teacher refers to a method that uses the spectral decomposition of a graph Laplacian matrix to derive a node ordering. Given a reciprocal dependency graph representing the strength of dependencies between variables, one can construct its Laplacian matrix (L = D - A) ((D) is the degree matrix and (A) is the adjacency matrix) and take the eigenvector corresponding to the second-smallest eigenvalue (the Fiedler vector). The magnitudes of the components of this vector give a continuous node ordering: nodes with similar components are more tightly connected in the graph and should be placed at nearby generation steps. Compared with direct graph search or greedy ordering, the spectral method is more robust to noise and naturally outputs continuous priorities, which facilitates training OrderMLP with a regression objective.

Reveal prefix is a random partial observation of input variables during training: a subset of variables is randomly selected as known conditions (i.e., "revealed"), and the remaining variables serve as prediction targets. Under the any-order autoregressive framework, each training iteration samples a different reveal set, exposing the model to diverse conditioning combinations. OrderGPT's key observation is that, although the reveal set is random, when the model predicts a target variable, attention stably concentrates on certain revealed variables — this attention pattern, recurring across samples, is precisely the signal of the true dependency structure in the data. By aggregating attention under a large number of random prefixes, the model can estimate dependencies between variables without any order supervision.

Cross-Modal Validation: Language, Pose, and Tabular Data

The paper validates OrderGPT's adaptability across multiple modalities:

• Language: The model adaptively recovers a dependency structure close to left-to-right, showing that language's natural order is indeed one of the optima and that the method does not forcibly distort language modeling.
• Human pose: It learns an anatomically organized decomposition — generating joints according to human body structure rather than an arbitrary coordinate order.
• Tabular data: It discovers decompositions that place statistically and semantically related fields near each other, while improving downstream modeling metrics across all domains.

These results show that learning generation order not only does not harm language modeling but can also deliver substantial gains in domains lacking a natural order.

Significance and Outlook: Order Learning Opens New Space

The value of OrderGPT lies in turning generation order from a human prior into a learnable parameter. For structured discrete data, this offers a new option between autoregression and diffusion. The code is open-sourced on GitHub (Chenhe-yang/OrderGPT), and the community can further explore applications of order learning in more domains.

One direction worth considering: can order learning also feed back into language models? Although language has a natural order, certain tasks (such as code generation and mathematical reasoning) may have better generation orders. OrderGPT's framework provides a foundational tool for such exploration.

One-line Summary

OrderGPT lets models learn generation order from data, finding a middle path between autoregression and masked diffusion.

Paragraph Summary

OrderGPT proposes treating generation order as a learnable parameter rather than a pre-fixed hyperparameter. Built on any-order GPT, it extracts recurring attention patterns from predictions over random reveal prefixes, aggregates them into a reciprocal dependency graph, uses a spectral teacher to derive continuous generation priorities, and has a lightweight OrderMLP predict the order in a single training run. The order is explored early in training and frozen later for autoregressive generation. On language, the model recovers a structure close to left-to-right; on human pose, it learns an anatomically organized decomposition; on tabular data, it discovers decompositions where semantically related fields are adjacent and improves downstream metrics. This work turns generation order from a human prior into a learnable parameter, offering structured discrete data without a natural order a new option between autoregression and diffusion.