SpeCal: Calibrating Unsupervised RLVR Rewards with Hidden-State Spectra

SpeCal uses hidden-state spectra to calibrate unsupervised RLVR advantage values at both group and rollout levels.
Unsupervised RLVR lets models derive rewards from their own rollouts, bypassing ground-truth answers, but self-generated supervision reliability varies greatly across and within groups, and decoding results and reward statistics alone cannot reveal whether the learning structure is reliable. SpeCal instead leverages hidden-state spectra: at the group level, it combines spectral quality with a Gaussian band-pass gate on effective rank to select sampling groups with sufficient representational variation and effective dimensions in a preferred range; at the rollout level, it measures each response's conditional support in a leave-one-out spectral subspace, with the number of dominant directions set by group effective rank. These two signals calibrate the advantages induced by the original rewards without redefining the unlabeled reward, making it easy to integrate with existing pipelines and offering a lightweight path for exploiting unlabeled reasoning data.
Unsupervised reinforcement learning with verifiable rewards (RLVR) allows large models to improve their reasoning abilities even without ground-truth answers, but the reliability of self-generated rewards varies widely. A new paper proposes SpeCal, which extracts signals from hidden-state spectra to perform two-level calibration of reward advantages, offering unsupervised RLVR an approach that does not modify the reward definition and only corrects the learning signal.

The Bottleneck of Unsupervised RLVR: Where Do Rewards Come From, and Are They Reliable?
RLVR significantly improves the reasoning performance of large language models through verifiable reward signals, but it relies heavily on ground-truth answers or external verifiers, which prevents a large amount of unlabeled reasoning data from being used. The idea of unsupervised RLVR is to let the model derive scalar rewards from its own multiple rollouts, thereby bypassing external verification.
The problem is that the reliability of this self-generated supervision varies greatly across sampling groups and among different responses within a group. Existing methods mainly construct or refine rewards from decoding results and reward statistics—these endpoint signals can tell us what the model "answered," but they can hardly indicate whether the model's internal rollout representations have formed a reliable learning structure.
RLVR (Reinforcement Learning with Verifiable Rewards) is a training paradigm that optimizes a language model's reasoning ability using programmatically verifiable reward signals (such as matching the final answer to a math problem or passing unit tests for code). Unlike traditional RLHF, which relies on human preference models, RLVR rewards are deterministic and automatically judgeable, so the training signal has lower noise and is easier to scale. Typical implementations such as GRPO sample multiple responses to the same prompt and directly construct advantage values based on whether the verifier marks them correct or incorrect, skipping the value network. Its limitation lies precisely here: every sample requires a ground-truth answer or an executable verifier, and a large amount of open-ended reasoning data beyond math and code cannot be used directly—this is exactly the boundary that unsupervised RLVR attempts to break through.
Core Observation: Hidden-State Spectra Offer Another Perspective
The paper's key observation is that hidden-state spectra can characterize how a rollout group is organized and the degree of support each rollout within the group receives, thereby providing a complementary perspective on supervision reliability. In other words, rather than looking only at whether the final answer is correct, it is better to examine whether the representation structure formed by the model during generation "holds up."
This forms the design basis of SpeCal—a two-level spectral calibration framework for unsupervised RLVR.
Hidden-state spectrum refers to the singular value distribution obtained by performing singular value decomposition (SVD) on the hidden-state matrix of a group of rollouts at a certain layer (or several layers). Each row of the matrix corresponds to the representation vector of one rollout. The magnitudes of the singular values reflect how many orthogonal directions exhibit significant variation in the group's representations, and the rate of decay of the singular value spectrum characterizes whether the representations are concentrated in a few dominant directions. Effective rank is a scalar summary of this spectrum, usually obtained by exponentiating the information entropy of the normalized singular values: a value close to the total dimension indicates dispersed and diverse representations; a very low value indicates highly collinear representations and a lack of structural variation in sampling. Compared with a single scalar such as a reward score, spectral statistics carry overall information about the geometry of representations within the group, and can therefore serve as independent corroborating evidence for reward reliability.
Group-Level Calibration: Spectral Quality and a Gaussian Band-Pass Gate
At the group level, SpeCal combines spectral quality with a Gaussian band-pass gate that operates on the effective rank based on normalized entropy. This mechanism tends to select rollout groups whose representations vary sufficiently and whose effective dimension falls within a preferred range.
The intuition behind this design is: if a group has too little representational variation, its sampling lacks diversity; if its effective dimension deviates from a reasonable range, it may indicate excessive structural noise. The role of the band-pass gate is precisely to constrain group-level reliability to a "sweet spot."
The Gaussian band-pass gate borrows the idea of band-pass filtering from signal processing: only signals falling within a certain central range are allowed to pass, while those too high or too low are attenuated. Here, the input is the group effective rank (converted via normalized entropy), and the gate function constructs Gaussian weights around a preferred center μ and bandwidth σ. Groups with too low an effective rank (insufficient representational variation, redundant sampling) and groups with too high an effective rank (large structural noise, divergent representations) are both downweighted, and only groups falling within the "sweet spot" receive a gating coefficient close to 1. Spectral quality is a score for the overall shape of the singular value spectrum, used to measure whether the group's representations have a clear dominant structure. The two are multiplied to obtain the group-level reliability weight, which is then applied to the group's advantage values, achieving soft filtering without changing the reward definition.
Rollout-Level Calibration: Conditional Support in a Leave-One-Out Spectral Subspace
At the rollout level, the same group effective rank determines how many dominant directions to retain in a leave-one-out spectral subspace formed by the remaining rollouts. Each held-out rollout is measured for its conditional support within this subspace.
The significance of this is that whether a response is reliable depends not only on its own reward score, but also on whether the representation structure it forms relative to other responses in the same group receives sufficient support. The signals obtained at the two levels are ultimately used to calibrate the advantage values induced by the original rewards, without redefining the underlying unlabeled reward.
The leave-one-out spectral subspace is a cross-validation-style geometric construction: for a given rollout group, first remove the rollout to be evaluated, perform SVD on the hidden states of the remaining rollouts, and take the top k right singular vectors to span a subspace, where k is determined by the group's effective rank. Then project the held-out rollout's representation onto this subspace and measure the degree to which it shares structure with other samples in the group using the ratio of its projection energy to its total energy (or the projection residual)—this is the conditional support. High support indicates that the response's representation is consistent with the dominant structure within the group, making its reward more likely to be credible; low support may indicate an outlier representation or a lucky correct answer. The leave-one-out design avoids the bias of self-support, where the evaluated sample itself inflates the subspace.
Calibrating the Learning Signal Without Redefining the Reward
A notable design choice of SpeCal is "calibration rather than replacement." It preserves the original unsupervised reward definition and only corrects the advantages at the group and rollout levels. This decoupling makes the method easier to integrate with existing unsupervised RLVR pipelines and reduces the risk of introducing new biases.
From a broader perspective, this work suggests that the reliability problem of self-generated supervision may not need to be solved only at the answer level; structural information from the model's internal representations can also serve as a basis for calibration. For teams hoping to tap unlabeled reasoning data, this offers a lightweight path worth noting.
One-line Summary
SpeCal uses hidden-state spectra to calibrate unsupervised RLVR advantage values at both the group and rollout levels.
Paragraph Summary
Unsupervised RLVR lets models derive rewards from their own multiple rollouts, bypassing ground-truth answers, but the reliability of self-generated supervision varies greatly across and within groups, and decoding results and reward statistics alone make it hard to judge whether the learning structure is reliable. The paper proposes SpeCal, which instead leverages a complementary perspective based on hidden-state spectra: at the group level, it combines spectral quality with a Gaussian band-pass gate operating on effective rank to select sampling groups with sufficient representational variation and effective dimensions in a preferred range; at the rollout level, it measures each response's conditional support in a leave-one-out spectral subspace, with the number of dominant directions determined by the group effective rank. The two levels of signals are ultimately used to calibrate the advantage values induced by the original rewards without redefining the unlabeled reward, making it easy to integrate with existing pipelines and offering a lightweight path for tapping unlabeled reasoning data.
Related articles

The ARTEX Alert: How AI Agents Do Hackers' Work
FBI issues a 13-page alert on the AI agent tool ARTEX: attackers use it to automate penetration testing, with some cases achieving privileged access in under three hours. This article analyzes ARTEX's architecture, real attack cases, the authorization paradox, and AI agent security boundaries.

FBI Warns: Open-Source AI Pentesting Framework ARTEX Weaponized — From Competition Winner to Threat Dossier in Three Weeks
FBI threat bulletin warns that open-source AI pentesting framework ARTEX was weaponized — just three weeks from GitHub release to first weaponization, linked to South Korean financial breaches, supply chain attacks, and large-scale operations. Timeline, attack evidence, and defense recommendations.

Tokendoctor: The Money-Saving Solution That Cuts Claude Code Costs by 38%
Tokendoctor claims it can reduce Claude Code usage costs by 38% through token efficiency optimization. This article analyzes its principles, community reactions, and practical implications for developers.