[KongchangAI]
· 2 min read· 1,043 words

Learning Calibrated Decision Heads from Bandit Feedback: CAPS Wins

Learning Calibrated Decision Heads from Bandit Feedback: CAPS Wins

Naive probability rewards fail; CAPS effectively learns calibrated decision heads from bandit feedback.

This article explores fine-tuning decision heads using only logged bandit feedback (whether the chosen action was correct) via LoRA on Qwen3-0.6B-Base and 1.7B-Base. Using the Brier or log score of the sampled action's probability as a REINFORCE reward caused training collapse, with accuracy dropping from 0.636 to 0.111, while under full labels no bandit method beat a cross-entropy-plus-temperature baseline on ECE. The authors propose Chosen-Action Proper Scoring (CAPS), applying a proper scoring rule to the chosen action's probability. Learning from logged feedback only, CAPS is 2.6 points more accurate and 0.053 lower in ECE than confirmed-answer-only fine-tuning on Banking77, and 21.7 points more accurate on sentiment data when the logger often errs. Versus full-label training at N=5000, the gap is small (1.5 points lower on 0.6B, 0.6 points lower on 1.7B). Under noisy feedback, inverse correction recovers most clean calibration and early stopping prevents noise memorization. The study stresses careful reward design; CAPS offers an effective approach but still depends on log quality.

A decision head is a model component that takes a state and a list of options, and outputs a probability for each option in a single forward pass. Once deployed, it typically can only observe whether the chosen option was correct, i.e., logged bandit feedback. A new study fine-tunes decision heads via LoRA on Qwen3-0.6B-Base and 1.7B-Base using this feedback and reports three key findings.

rss source: Learning a calibrated decision head from logged bandit feedback

Failure of Naive REINFORCE Rewards

The team first examined using the Brier score or log score of the sampled action's own probability as a REINFORCE reward and found this signal unusable. Specifically, every wrong action received a reward at least as large as the gold action's; with a mean baseline, the wrong action with the lowest probability received the largest advantage; and with only two options, the signal was identically zero. On a set of ordered-scale problems, this reward design caused accuracy to plummet from 0.636 to 0.111—a severe performance collapse.

By contrast, treating the probability as a loss and optimizing it via the pathwise gradient avoids this collapse, and using the REINFORCE reward r or r minus p also achieves similar results. This shows that reward design must handle probability signals carefully rather than simply using probability scores directly as rewards.

REINFORCE vs. Pathwise Gradient: REINFORCE is a classic family of policy gradient estimators that updates parameters using the gradient of the log-probability multiplied by the reward. It essentially differentiates the score function of the sampled action, so its gradient has high variance and depends on the reward scale. The pathwise gradient (also called the reparameterized gradient) instead expresses random sampling as a deterministic function of independent noise, directly differentiating the probability output, which usually has lower variance. In the decision-head setting, if a probability score is used directly as the reward, the score-function term in REINFORCE couples with the probability term in the reward, leading to anomalous gradient directions; the pathwise gradient directly optimizes the probability itself (e.g., the Brier score), avoiding this coupling and thus being more stable.

Bandit Methods Do Not Beat Cross-Entropy Under Full Labels

In the fully labeled setting, the study compared four bandit methods against cross-entropy plus one temperature on the average expected calibration error (ECE) across four unseen label patterns. The results show that no bandit method beat this simple baseline. This means that when labels are plentiful, conventional supervised learning still has an advantage in calibration performance, and bandit methods bring no additional gain.

Expected Calibration Error (ECE): ECE is a common metric for measuring the consistency between model confidence and true accuracy. It bins predicted confidences into intervals, computes the absolute difference between average confidence and average accuracy within each bin, and takes a sample-proportion-weighted average. A lower ECE indicates more reliable output probabilities. Cross-entropy plus a temperature parameter is a simple post-hoc calibration method: a single temperature scalar is tuned on a validation set to scale the logits so as to minimize cross-entropy, thereby improving calibration. Bandit methods typically optimize the decision loss directly, which may sacrifice calibration performance, so they are not necessarily better than a temperature-scaled cross-entropy baseline on this metric.

CAPS: Learning Calibration from Logged Feedback Only

The study's core contribution is Chosen-Action Proper Scoring (CAPS), a direct method that applies a proper score to the probability of the chosen action. When learning only from logged feedback, CAPS beats fine-tuning on confirmed answers only—the latter discards negative samples. Specific results include:

  • On the Banking77 dataset, CAPS is 2.6 percentage points higher in accuracy, 0.053 lower in ECE, and has lower selective risk (based on three random seeds versus one baseline seed).
  • On sentiment data, when the logger frequently errs, CAPS is 21.7 percentage points higher in accuracy on one seed.
  • At N=5000, compared with full-label training, CAPS is 1.5 percentage points lower in accuracy on the 0.6B model and 0.6 percentage points lower on the 1.7B model (versus one full-label seed), while propensity weights brought no gain on this log.

In addition, under feedback noise with a known flip rate (one seed per flip rate), CAPS's confidence tracks the noisy channel on average, inverse correction recovers most of the clean calibration, and early stopping based on held-out log outcomes prevented memorizing the noise samples in all three runs.

Proper Scoring Rule: A proper scoring rule is a class of loss functions that attain their minimum when the predicted distribution matches the true distribution, thereby encouraging the model to output true probabilities. Common examples include the Brier score (mean squared error) and the log score (cross-entropy). The core idea of CAPS is to apply a proper scoring rule only to the probability of the chosen action, using the information in the logged feedback about whether the chosen action was correct to construct the loss, without needing labels for unchosen actions. Owing to the properties of proper scoring rules, minimizing this loss drives the model to output calibrated probabilities even when the training signal comes only from the chosen action.

Propensity Weights: In bandit feedback, the probability that each action is chosen is called the propensity score. Propensity weights usually refer to weighting samples by the inverse propensity score to correct for selection bias introduced by the logging policy, thereby estimating the value of the target policy without bias. However, when propensity scores are estimated inaccurately or the logging policy differs too much from the target policy, propensity weights introduce high variance or even bias. In this study, propensity weights brought no gain, possibly due to limitations in log quality or weight estimation.

Practical Implications and Limitations

This study reveals several key points about learning calibrated decision heads from bandit feedback: naive probability rewards cause training collapse, while CAPS provides an effective alternative. However, CAPS still has a gap compared with full-label training, and propensity weights are ineffective on certain logs, indicating that the method's effectiveness depends on log quality. Under noisy feedback, inverse correction and early stopping are necessary stabilizing measures. The code, adapters, logs, and per-record results have all been made public to facilitate reproduction and further research.

One-line Summary

No translation needed for one-line summary (see JSON field).

Paragraph Summary

See JSON field.

#bandit feedback#decision head#calibration#CAPS#REINFORCE#LoRA#Qwen3
Share:

Related articles