DNO Alignment in Practice: 8B Model Surpasses 70B in Win Rate

Direct Preference Optimization (DPO) and its variants have become mainstream methods for large model alignment, but they essentially fit a static preference dataset. Direct Nash Optimization (DNO) takes a different path: it models alignment as a two-player preference game and attempts to approximate the Nash equilibrium of that game. A new empirical study independently reproduces DNO-Prct, a practical variant of DNO, validates its effectiveness on Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, and provides many noteworthy engineering details.

How DNO-Prct Works

The theoretical goal of DNO is to align a general preference function rather than a fixed annotation set. Its practical version, DNO-Prct, uses an iterative process: in each round, online sampling is performed on the policy model, then an LLM scores the samples together with golden responses from a teacher model, large-margin preference pairs are selected, and finally a DPO-style contrastive update is performed. This process strings "generate—score—filter—update" into a closed loop, and in each round the model faces new samples produced by its current policy.

The research team independently reimplemented this process rather than directly using the original authors' code. The value of such independent reproduction lies in testing the reproducibility of the method itself, not merely reporting a tuning result. The experiments choose two open-weight small models as starting points, and use GPT-4o and larger open-weight models from the same family as annotators, respectively, to compare the differences between proprietary and open-source models as scorers.

Direct Preference Optimization (DPO) is a preference alignment method that does not require an explicit reward model. Based on the Bradley-Terry preference model, it transforms the reward maximization objective in RLHF into a classification loss over preference pairs, directly optimizing the policy model so that it favors chosen responses. DPO avoids the complexity of online reinforcement learning algorithms such as PPO, is stable to train, and is computationally efficient, making it one of the mainstream methods for large model alignment. However, its limitation is that it only fits a static preference dataset and cannot exploit new samples generated online by the policy model, making it prone to overfitting the annotation distribution. DNO-Prct introduces an online iterative mechanism on this basis to approximate a more general preference objective.

Nash equilibrium is a core concept in game theory, referring to a strategy combination in a two-player (or multi-player) game in which no party can obtain a better payoff by unilaterally changing its strategy. Under the DNO framework, alignment is modeled as a two-player game between the policy model and a preference function (or teacher model): the policy model generates responses, the preference function provides preference signals, and both sides continuously adjust their strategies. Ideally, when the distribution of responses generated by the policy model makes it impossible for the preference function to further distinguish better from worse, Nash equilibrium is reached. At this point the policy model is consistent with true preferences, corresponding to the theoretically optimal alignment. DNO-Prct approximates this equilibrium iteratively, but approximations such as finite-round iteration and LLM scoring prevent it from strictly converging.

Key Data: The 8B Model's Comeback

On the AlpacaEval 2.0 benchmark, with GPT-4o as judge, after three rounds of iteration, the length-controlled (LC) win rate showed significant improvement. Llama 3.1 8B Instruct's LC win rate against GPT-4o rose from 31.7% to 45.3%, and Mistral 7B Instruct v0.1 rose from 6.8% to 18.4%. More notably, the tuned 8B model surpassed Llama 3.1 70B Instruct on all reported win-rate metrics.

This result shows that the gain brought by DNO-Prct is not just a minor fluctuation on the benchmark, but sufficient for a small model to cross the scale gap in a specific evaluation. Of course, the evaluation here relies on GPT-4o as judge, and the specific win-rate numbers need to be understood in the context of the evaluation setup.

AlpacaEval 2.0 is a commonly used instruction-following evaluation benchmark containing 805 diverse user instructions covering writing, question answering, coding, and other scenarios. It uses an LLM as an automatic judge, comparing model responses with reference responses (usually from GPT-4) in pairwise fashion and computing win rates. To control length bias, AlpacaEval 2.0 introduces length-controlled (LC) win rate, i.e., the win rate after regression adjustment for response length, preventing models from gaining inflated win rates by generating longer responses. LC win rate focuses more on content quality than length, so it is widely used to evaluate alignment effectiveness. All win-rate improvements in this article are based on LC win rate, ensuring fair comparison.

How Much Gain Do Open-Source Annotators Preserve?

A practical question is: if GPT-4o is not used for annotation, can DNO-Prct still be effective? The study provides a quantitative answer. When using open-weight models from the same family as annotators, Llama retained 62% of the LC gain obtained with the GPT-4o annotator, and Mistral retained 82%. This means the scoring step does not necessarily depend on proprietary models; although open-source annotators lose some effectiveness, they can still maintain most of the benefit.

This finding has direct implications for teams with limited resources. It lowers the entry barrier for DNO-Prct from "must call a closed-source large model API" to "can build a scoring pipeline using larger models from the same family," which is friendlier in terms of both cost and data controllability.

Ablation Experiments: Which Steps Really Matter

The study conducted a set of ablation configurations, and the results show that three variants—removing golden responses, removing data look-back, or accepting only sample pairs preferred by the teacher—are 6 to 10 LC points lower than the full pipeline. This indicates that teacher golden responses and the preference-pair filtering strategy are key components of the pipeline, not dispensable decorations.

In contrast, increasing the number of online samples per prompt from 1 to 7 changes the LC win rate by at most 2.1 points; sweeping the margin from 1 to 5 changes it by at most 2.4 points in the independent pipeline. The impact of these two hyperparameters is relatively mild, leaving greater tolerance for tuning in engineering deployment.

Cost, Bias, and Theoretical Distance

The paper also reports the cost of the pipeline and discusses the deviation between the practical algorithm and the theory it approximates. The theoretical foundation of DNO is Nash equilibrium, but DNO-Prct makes many approximations in practice: finite-round iteration, LLM scoring replacing exact preferences, large-margin filtering, and so on. These approximations make the method runnable, but also mean it cannot strictly guarantee convergence to the theoretical objective.

One point that needs to be treated cautiously is that all results are from a single run, and GPT-4o serves simultaneously as teacher and evaluation judge. Under this setup, the reported gains may partly come from bias caused by the teacher and judge being from the same source. The research team discusses this, but a single run lacks variance estimates, and robustness across models and judges still needs more reproductions to confirm.

Same-source bias refers to the possibility that when the teacher model and the evaluation judge are the same model (such as GPT-4o), evaluation results may systematically favor the teacher model's style or preferences, thereby overestimating the true gain brought by the method. In the reproduction of DNO-Prct, GPT-4o both generates golden responses (teacher) and serves as the judge for AlpacaEval 2.0, which may cause the model to learn GPT-4o's specific preferences and obtain inflated win rates under a same-source judge. To mitigate this issue, practitioners should introduce a judge model different from the teacher (such as Claude or human evaluation), or at least conduct cross-judge validation to confirm the robustness of the gain.

Implications for Practitioners

For teams that want to do preference alignment on open-source models, this study provides several actionable lessons: teacher golden responses and the data look-back mechanism are worth retaining; online sample count and margin parameters do not need overly fine tuning; annotators can be replaced by open-source models from the same family, at the cost of discounted gains but unchanged direction. At the same time, it is best to introduce a judge different from the teacher in the evaluation stage to avoid same-source bias amplifying conclusions.

DNO-Prct demonstrates an alignment path from "fitting static preferences" to "iterative game approximation." Its performance on 8B models shows that this path has practical potential, but there is still some distance to go before it reaches a stable, reproducible engineering standard.

One-line Summary

Independent reproduction of DNO-Prct validates that its online iterative preference alignment significantly improves win rate on 8B models and surpasses 70B models.

Paragraph Summary

This article independently reproduces DNO-Prct, a practical variant of Direct Nash Optimization, which iteratively aligns models through online sampling, LLM scoring, large-margin preference-pair filtering, and DPO-style updates. On Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, after three rounds of iteration, the LC win rates against GPT-4o rise from 31.7% to 45.3% and from 6.8% to 18.4%, respectively, and the tuned 8B model surpasses Llama 3.1 70B on all reported win rates. Using open-source annotators from the same family preserves 62% and 82% of the gain, and ablation experiments show that teacher golden responses and data look-back are key components, while online sample count and margin parameters have smaller effects. The study also notes that single runs and same-source teacher-judge bias may introduce bias, and more reproductions are needed to confirm robustness.