Maya a1 Option Order Sensitivity Study: The Hidden Bias of Non-Autoregressive Models

Title

Maya a1 Option Order Sensitivity Study: The Hidden Bias of Non-Autoregressive Models

Meta Description

New research reveals that the Laya non-autoregressive model is highly sensitive to option order: 25.8% of answers change when options are rearranged, with a 10.1 percentage point accuracy gap between first and last positions. Fine-tuning with shuffling reduces the flip rate to 7.2%.

Article Content

Can non-autoregressive models truly understand the meaning of options? A new study on the Laya model (referred to in the article as Maya a1) gives a concerning answer: the model's sensitivity to option order far exceeds expectations, and answers change depending on how options are arranged.

Research Background: Non-Autoregressive Decision Models

The Laya model uses a non-autoregressive architecture, where each option corresponds to a [MASK] token, and the model scores each option in a single forward pass and selects the highest score. The research team redesigned the checkpoint (421M parameters, 2-layer option head, no positional tensors, parallel layout present but unused) to test whether option position affects answers.

This design was originally intended to eliminate the left-to-right bias of autoregressive models, but experiments show that positional information may seep into decisions in more subtle ways.

Comparison Between Non-Autoregressive and Autoregressive Models

Autoregressive models (such as the GPT series) generate tokens one by one from left to right, with each token's generation depending on previously generated tokens. This sequential nature naturally introduces positional bias: the model may pay more attention to earlier options or information. Non-autoregressive models generate all output positions in parallel at once, independent of generation order, theoretically eliminating left-to-right bias. The Laya model adopts this architecture, assigning a [MASK] token to each option, scoring all options simultaneously through a single forward pass, and then selecting the option corresponding to the highest score. This design aims to improve inference efficiency and avoid order bias, but this study reveals that positional information can still influence decisions through other mechanisms (such as parameters in the option head or implicit patterns in training data).

Experimental Design: Cyclic Rotation and Random Permutation

The study used 951 Qwen-verified questions, applying all cyclic rotations to each question's options (using 6 random permutations for 12-option questions). Results show that the released checkpoint changes answers on 25.8% of questions (confidence interval 22.9-28.7%), with a flip rate of 19.7% on 605 English questions.

When the correct option is in first versus last position, accuracy differs by 10.1 percentage points. Flips are concentrated in regions where the model's own confidence is low: the lowest confidence tercile has a flip rate of 54%, while the highest tercile has only 8%.

Cyclic Rotation and Permutation Testing

Cyclic rotation is a systematic option rearrangement method: for N options, by cyclically shifting the option list by 1, 2, ... up to N-1 positions, N-1 different permutations are generated (the original order is retained as baseline). For example, the cyclic rotations of options [A,B,C] include [B,C,A] and [C,A,B]. This method effectively detects whether the model is sensitive to the absolute position of options while controlling for relative order changes between options. When the number of options is large (e.g., 12), the total number of permutations is enormous, so the study uses random sampling (e.g., 6 random permutations) to approximate evaluation. By comparing the model's answer consistency across different permutations, order sensitivity can be quantified.

Fine-Tuning Can Mitigate Order Sensitivity

The research team used the Laya trainer to fine-tune on 2,818 rows of teacher-verified data, adding option shuffling. After shuffling, the flip rate dropped to 7.2%, compared to 11.4% without shuffling, indicating that shuffling contributed a 4.2 percentage point improvement.

Confidence-weighted voting over orders achieved 69.7% accuracy, compared to 69.1% for single base pass, with 4.1x increase in computation and limited improvement.

The study also fine-tuned the same data using the authors' parallel layout, reducing the flip rate to 0.5%, showing that architectural choices have a significant impact on order sensitivity.

Confidence-Weighted Voting

Confidence-weighted voting is an ensemble strategy: predictions are made separately for multiple option permutations of the same question, and weighted voting is performed based on the model's confidence (e.g., softmax probability) for each prediction, ultimately selecting the option with the highest weighted score. Theoretically, if the model has higher confidence in predictions for certain permutations, these predictions should carry greater weight in voting, thereby improving overall accuracy. However, in this study, this method only improved accuracy from 69.1% to 69.7%, while increasing computation by 4.1x, indicating that the calibration between model confidence and prediction correctness is not ideal, and the gains from simple weighted voting are limited.

Multi-Source Validation and Limitations

This study was conducted by an independent team, providing cross-validation with the authors' original report. Labels were generated by LLMs, and the test set was filtered by teachers, which may introduce bias.

Order sensitivity is a common issue in non-autoregressive models, and this study provides a new perspective for model robustness evaluation.

Bias Risk of LLM-Generated Labels and Teacher Filtering

This study uses large language models (LLMs) to generate question labels and filters the test set through a "teacher" model (possibly a more powerful LLM or human rules). Although this pipeline can quickly build large-scale evaluation data, it has inherent biases: LLM-generated labels may reflect their own knowledge biases or hallucinations, and teacher filtering may introduce selection bias (e.g., retaining only samples the teacher model considers correct), causing the test set to not fully represent the true distribution. Additionally, while independent team cross-validation can increase credibility, if both parties use similar LLMs to generate labels, bias may still persist. Therefore, the generalizability of the conclusions should be treated with caution.

One-line Summary

Laya non-autoregressive model is highly sensitive to option order, with 25.8% of answers flipping.

Paragraph Summary

A study on the Laya non-autoregressive decision model (referred to in the article as Maya a1) found that the model is unusually sensitive to option order: answers change on 25.8% of questions when options are rearranged, with a flip rate of 19.7% on English questions. Accuracy differs by 10.1 percentage points when the correct option is in first versus last position, with flips concentrated in low-confidence regions (54% flip rate in the lowest tercile, only 8% in the highest). Adding option shuffling during fine-tuning reduces the flip rate from 11.4% to 7.2%, with shuffling contributing 4.2 percentage points; fine-tuning with parallel layout further reduces the flip rate to 0.5%. Confidence-weighted voting provides limited improvement (69.7% vs 69.1%) with 4.1x increase in computation. The study suggests that non-autoregressive models still have hidden positional biases, and architectural choices significantly impact robustness.