NOPO: Solving One-Sided Verification with Negative-Only Reinforcement Learning

NOPO suppresses the policy using only trusted negative samples, improving pass@k in one-sided verifiability scenarios.
This article explains Negative-Only Policy Optimization (NOPO), which addresses the failure of RLVR in One-Sided Verifiability (OSV) scenarios where the verifier can only reliably falsify negative samples and positive samples may contain false positives. NOPO treats positive samples as unlabeled data, updates the policy only by suppressing trusted negative samples, and combines a reference-relative negative suppression loss with a group-level adapter for robust optimization. Across unit tests, constraint checking, and LLM self-verifiers, NOPO outperforms baselines on pass@k for code, logic, and math benchmarks, with gains persisting under inference-time scaling and cross-benchmark transfer.
Title
NOPO: Solving One-Sided Verification with Negative-Only Reinforcement Learning
Meta Description
This article explains NOPO (Negative-Only Policy Optimization), a method for One-Sided Verifiability (OSV) scenarios that performs policy optimization using only trusted negative samples, improving pass@k on code, logic, and math tasks, and discusses its technical principles and experimental performance.
Article Content
Reinforcement Learning with Verifiable Rewards (RLVR) performs well on tasks such as code generation and logical reasoning, but its core assumption—that a verifier exists that can reliably judge whether a solution is correct—is often difficult to satisfy in real-world scenarios. In many cases, we can only obtain cheap negative checks: we can reliably falsify incorrect solutions but cannot confirm correct ones. This asymmetry gives rise to a new research question: how can effective policy optimization be performed under such "One-Sided Verifiability" (OSV) conditions?
A recent paper proposes Negative-Only Policy Optimization (NOPO), which updates the policy using only trusted negative samples (solutions falsified by the verifier), and significantly improves pass@k metrics in scenarios such as unit tests, constraint checking, and LLM self-verifiers. This article provides an in-depth interpretation of the method's core ideas, technical details, and experimental performance.

One-Sided Verifiability: When the Verifier Only Says "No"
Traditional RLVR assumes that the verifier can reliably judge the correctness of a solution (i.e., two-sided verifiability). However, in many practical applications, complete verification is either too costly or simply infeasible. For example, in code generation, unit tests can only catch some errors and cannot prove that code is completely correct; in constraint satisfaction problems, a checker can quickly find solutions that violate constraints, but a solution that satisfies constraints is not necessarily globally optimal; in LLM self-verification, the model can identify some obvious errors but may let hidden defects slip through.
The paper formalizes this scenario as One-Sided Verifiability (OSV): the negative samples given by the verifier are reliable counterexamples (verified negatives), while positive samples are merely "non-falsified candidates" that may contain many false positives. This asymmetry makes standard RLVR unreliable—reinforcing those positive samples that "pass partial checks" may directly reward shortcut-taking behavior, causing the policy to learn solutions that superficially pass checks but are actually incorrect.
False Positive in the context of this article refers to solutions that are not falsified by the verifier but are actually incorrect. Since under OSV the verifier can only provide negative evidence, any solution that passes checks is merely a "non-falsified candidate" rather than a confirmed correct solution. The existence of false positives means that directly reinforcing these positive samples introduces erroneous learning signals, making the policy tend to generate outputs that are superficially compliant but substantively wrong. NOPO is designed precisely to avoid this risk by treating positive samples as unlabeled data and relying only on reliable negative samples for updates.
NOPO: Policy Optimization by Suppressing Only Negative Samples
To address the challenges of OSV, NOPO's core idea is: treat positive samples that pass verification as unlabeled data, and update the policy only by suppressing trusted negative samples. Specifically, NOPO combines two key components:
- Reference-Relative Negative Suppression Loss: When suppressing negative samples, use a reference policy as the baseline to prevent the policy from deviating too far from the initial distribution, thereby maintaining exploration ability. This relative loss design helps stabilize training and prevents the model from over-correcting on negative samples.
- Group-Level Adapter: Scale the update magnitude according to the proportion of non-falsified responses in each group. If most solutions in a group are non-falsified (i.e., the proportion of negative samples is low), the update magnitude is smaller; conversely, when the proportion of negative samples is high, the update magnitude increases. This allows the policy to adaptively leverage negative sample information and avoid over-suppression when the proportion of positive samples is too high.
In this way, NOPO bypasses dependence on the reliability of positive samples, learns only from reliable negative samples, and uses group-level information to adjust update strength, achieving robust policy optimization.
Reference Policy usually refers to the initial model before reinforcement learning fine-tuning (e.g., a pretrained or supervised fine-tuned model), whose distribution represents the model's existing prior knowledge. In NOPO, the reference-relative negative suppression loss uses the reference policy as an anchor and penalizes the probability difference between the current policy and the reference policy on negative samples, rather than directly maximizing the negative log-likelihood of negative samples. This relative form limits the magnitude of policy updates, preventing the model from forgetting general capabilities or collapsing into mode collapse due to over-suppressing negative samples, and is a key design for stable training.
Experimental Validation: Performance Improvements Across Multiple Scenarios
The paper evaluates NOPO on code, logic, and math benchmarks under three verifier settings: unit tests, constraint checking, and LLM self-verifiers. When oracle rewards (i.e., a perfect verifier) are unavailable, NOPO outperforms baseline methods on pass@k metrics. These gains persist under inference-time scaling and cross-benchmark transfer conditions, indicating that NOPO has good generalization and scalability.
You may not have noticed that NOPO's advantage lies in not requiring a perfect verifier—it only needs to reliably identify negative samples. This greatly lowers the threshold for applying RLVR in practice and provides a feasible reinforcement learning solution for scenarios where only partial verification signals are available.
pass@k is a commonly used evaluation metric in code generation and reasoning tasks, representing the probability that at least one solution passes verification (or meets correctness criteria) when k solutions are independently sampled from the model. Compared with pass@1, pass@k better reflects the diversity and coverage of the model's output distribution. In OSV scenarios, since the verifier can only falsify negative samples, pass@k computation usually relies on an oracle verifier (perfect verifier) for final judgment to distinguish truly correct solutions from false positives. NOPO's improvement in pass@k indicates that suppression using only negative samples can also indirectly improve the quality of generated positive samples.
Implications and Outlook
The proposal of NOPO opens a new direction for applying RLVR in weak-verification environments. Its "negative-only" strategy is not only applicable to code generation but can also be extended to any task with cheap negative checks, such as safety alignment, logical reasoning, and scientific hypothesis screening. Future work can further explore how to combine a small number of reliable positive samples with a large number of negative samples, and how to design more efficient group-level adapters.
For AI researchers and engineers, NOPO reminds us that when verification resources are limited, cleverly leveraging negative signals may be more practical than pursuing a perfect verifier. As model scale increases and tasks become more complex, such robust optimization methods will become increasingly important.
One-line Summary
NOPO suppresses the policy using only trusted negative samples, improving pass@k in one-sided verifiability scenarios.
Paragraph Summary
This article explains Negative-Only Policy Optimization (NOPO), a method aimed at addressing the failure of Reinforcement Learning with Verifiable Rewards (RLVR) in One-Sided Verifiability (OSV) scenarios. OSV means the verifier can only reliably falsify negative samples, while positive samples may contain false positives, causing standard RLVR to reward shortcut-taking behavior. NOPO treats positive samples as unlabeled data, updates the policy only by suppressing trusted negative samples, and combines a reference-relative negative suppression loss with a group-level adapter for robust optimization. Under three settings—unit tests, constraint checking, and LLM self-verifiers—NOPO outperforms baselines on pass@k metrics across code, logic, and math benchmarks, and the gains persist under inference-time scaling and cross-benchmark transfer. This method reduces dependence on perfect verifiers and provides a feasible solution for weak-verification environments.
Related articles

Gemini 4 Neon Hands-On: Half the Cost, Near-Flagship Performance
Gemini 4 Neon hands-on roundup: Intelligence Index 61 beats its own flagship Carbon; Coding Agents performance matches Opus 5.5 at half the cost. Includes pure-code 3D demos, official pricing, and failure cases.

Gemini 4 Neon vs Claude Opus 5.5 Hands-On Comparison: Which One Is Worth Switching To
Gemini 4 Neon vs Claude Opus 5.5 hands-on: official benchmarks lead on knowledge work and long context but lag in coding; independent tests give it a higher intelligence index yet slower, more verbose output. Two rounds of tests each won some, showing when to switch.

Gemini 4 Neon Builds a Minecraft Clone in One Shot: Magic and Faceplants
Gemini 4 Neon beats Claude Opus 5.5 on one-shot prompts, but cold-start tests expose broken water physics and weak bash reliability. We break down API pricing traps, the power of 8-agent Swarm, and when to use Neon vs GPT 6.2 Sol.