[KongchangAI]
· 2 min read· 1,347 words

Are System 1 Decision Models' Confidence and Decisions Trustworthy? Interpretability and Perturbation Auditing

Are System 1 Decision Models' Confidence and Decisions Trustworthy? Interpretability and Perturbation Auditing

Open audit benchmark reveals calibration, interference, and attribution traits of System 1 decision models, with 4B as the key watershed.

An open, deterministic audit benchmark systematically evaluated five open-weight System 1 decision models (0.8B to 27B) and System 2 baselines across NLI, adversarial syntactic shortcuts, clinical triage, prompt injection defense, and logic gate triggers. Findings: provider confidence is a closed-form dispersion transformation, not empirical accuracy; cross-question packing causes significant directional interference on models below 4B; models above 4B calibrate better than log-probability and JSON decoding baselines under syntactic shifts; prompt injection defense achieves 0.988–0.998 AUROC with zero false positives; single-pass inference enables interactive XAI, and S1-SHAP significantly improves OR gate attribution. Recommendations: treat confidence as a ranking signal, avoid batch packing with small models, and use attribution methods matching the output layer's mathematical structure.

System 1 decision models replace autoregressive generation with a single forward pass, directly outputting typed probability distributions, and are deployed in scenarios such as low-latency routing and automated triage. However, the statistical calibration and post-hoc interpretability of such models have previously lacked systematic characterization. An open, deterministic audit benchmark fills this gap by evaluating five open-weight models ranging from 0.8B to 27B and generative System 2 baselines across three dimensions: calibration under label-preserving perturbations, cross-question interference, and post-hoc attribution faithfulness.

System 1 Decision Model Audit Overview

Design and Coverage of the Audit Benchmark

The benchmark covers NLI (e-SNLI), adversarial syntactic shortcuts (HANS), synthetic clinical triage (N=900), prompt injection defense (Tensor Trust, N=250), and an 86-case logic gate trigger suite. System 2 baselines include single-token log-probability scoring and structured JSON/chain-of-thought decoding. This multi-task, multi-baseline design enables cross-model and cross-paradigm comparisons, rather than merely reporting performance on a single dataset.

The audit emphasizes determinism—identical inputs produce identical outputs—facilitating reproducibility and attribution analysis. The open benchmark positioning also means results can be verified by third parties, which is particularly important for high-stakes decision scenarios.

Finding 1: Confidence Is a Formula Output, Not Empirical Accuracy

The audit reveals that provider-reported confidence is an exact closed-form dispersion transformation— 1 - H(p)/ln K for models below 10B, and normalized margin for Clef-27B—rather than being calibrated against empirical accuracy. This means the model's "confidence" is essentially a mathematical transformation of the output distribution's shape, with no necessary connection to real-world correctness rates.

This finding has direct implications for deployment practice: if downstream systems use confidence thresholds as triggers for routing or human review, the actual error rate corresponding to the threshold may systematically deviate from expectations. The audit recommends treating confidence as a relative ranking signal rather than an absolute probability, and recalibrating with held-out data before deployment.

Finding 2: Cross-Question Interference Amplifies as Model Size Decreases

Packing multiple questions into a single request induces directional cross-question interference. On Tev1-0.8B, the binary flip rate reaches 11.7% to 22.4%; on Tev1-4B, this drops to 0.1% to 0.2%. The directional nature of the interference indicates that question ordering and grouping systematically affect outputs, rather than being random noise.

For batch API call scenarios, this means that packing strategies that save latency and cost may come at the expense of consistency. The audit results suggest that 4B parameters appears to be a watershed: models below this scale show significantly degraded decision stability under combined requests.

Finding 3: Models Above 4B Have Better Calibration Under Syntactic Shifts and Are Faster

System 1 models with 4B parameters and above not only are faster under syntactic shifts but also outperform both System 2 baselines in calibration. Their weighted ECE (ECE_w) is 0.030 to 0.087, compared to 0.144 for the log-probability baseline and 0.378 for the JSON decoding baseline.

The architectural advantage of a single forward pass becomes evident here: decision distributions can be obtained without token-by-token generation, reducing both latency and cumulative calibration drift during generation. The calibration disadvantage of JSON structured decoding is particularly prominent, contrasting with its popularity in scenarios requiring strict format output.

Weighted ECE (Expected Calibration Error) is a standard metric for measuring the deviation between model confidence and actual accuracy. It partitions predicted confidence into bins, computes the absolute difference between the mean confidence and empirical accuracy within each bin, and then sums these weighted by the sample size of each bin. The "w" in ECE_w typically refers to weighting by sample weights or class frequencies to mitigate bias from class imbalance. Lower ECE indicates better calibration: 0 represents perfect calibration, and values above 0.1 are generally considered poorly calibrated. In the audit, an ECE_w of 0.030–0.087 means the average deviation between the model's output confidence and true correctness rate is only a few percentage points, whereas the 0.378 of the JSON decoding baseline indicates severe over- or under-confidence. Note that ECE depends on binning strategy and sample size; when comparing across studies, confirm that binning methods are consistent.

Finding 4: High AUROC and Low Latency for Prompt Injection Defense

In the prompt injection defense task (N=250), models of 4B and above achieve AUROC of 0.988 to 0.998, recall up to 95.2%, and 0.0% false positives at 48 to 60 milliseconds latency. This combination—high recall, zero false positives, sub-100ms response—makes System 1 models practically valuable as a real-time security filtering layer.

Zero false positives is particularly critical: false positives in security filters directly harm normal user experience, while high recall ensures attack samples are not missed. The audit data shows that both can be achieved simultaneously in models above 4B.

Finding 5: Single-Pass Inference Makes Interactive XAI Feasible

A single forward pass enables 512-coalition XAI analysis to run interactively, taking 1.3 to 3.2 seconds per input. In contrast, attribution methods requiring multiple forward passes struggle to meet interactive requirements at the same coalition scale.

The audit also proposes S1-SHAP—KernelSHAP with a logit link—to mitigate attribution distortion from softmax saturation on disjunctive OR gates. In probability space, OR gate recovery rates are 57.1% to 82.1%; under S1-SHAP, they improve to 89.3% to 100.0% (p_adj = 0.004). This improvement targets the attribution challenge of disjunctive structures in the logic gate trigger suite: softmax compresses the probabilities of multiple activation paths, causing attribution methods to underestimate the contribution of any single path, whereas the logit link preserves the additive structure.

KernelSHAP is a model-agnostic attribution method within the SHAP (SHapley Additive exPlanations) framework that approximates Shapley values via kernel-weighted linear regression. It treats each feature as a "player" and estimates each feature's marginal contribution using model outputs on coalitions (feature subsets). Shapley values satisfy axioms such as additivity and symmetry, theoretically enabling fair contribution allocation. However, KernelSHAP by default models outputs in probability space; after softmax, probabilities are compressed to [0,1] and gradients in saturated regions are extremely small, causing attribution values to be distorted. The logit link takes the log-odds of probabilities, transforming outputs back into an additive real-number space so that contributions from multiple activation paths can be linearly superimposed. S1-SHAP uses KernelSHAP with a logit link precisely to avoid softmax saturation masking the contribution of a single path in disjunctive OR gate structures ("triggered if any condition is met"), thereby significantly improving OR gate recovery rates.

Implications for Deployment Practice

Taken together, the five findings show that System 1 decision models above 4B parameters exhibit usable calibration and defense performance, but vendor confidence cannot be used directly as accuracy. Packing multiple questions into requests introduces significant directional interference on models smaller than 4B, and batch sizes should be avoided or strictly limited.

Regarding interpretability, the single-pass inference architecture turns attribution analysis from offline reports into an interactive tool. The improvement of S1-SHAP on disjunctive logic gates demonstrates that attribution methods need to match the mathematical structure of the model's output layer—saturation problems in probability space can be mitigated in logit space.

The open and deterministic design of the audit benchmark itself provides a comparable baseline for subsequent model iterations. For teams adopting System 1 models in routing, triage, and security filtering, this audit offers concrete parameter-scale thresholds, calibration reference values, and attribution method recommendations.

One-line Summary

An open audit benchmark reveals calibration, interference, and attribution characteristics of System 1 decision models, with 4B as a key watershed.

Paragraph Summary

An open, deterministic audit benchmark systematically evaluated five open-weight System 1 decision models from 0.8B to 27B and System 2 baselines across five task types: NLI, adversarial syntactic shortcuts, clinical triage, prompt injection defense, and logic gate triggers. The audit found: provider confidence is a closed-form dispersion transformation rather than empirical accuracy; cross-question packing induces significant directional interference on models below 4B; models above 4B have better calibration than log-probability and JSON decoding baselines under syntactic shifts; prompt injection defense achieves 0.988–0.998 AUROC with zero false positives; single-pass inference enables interactive XAI, and S1-SHAP significantly improves OR gate attribution. The conclusions recommend treating confidence as a ranking signal, avoiding batch packing with small models, and adopting attribution methods that match the mathematical structure of the output layer.

#System 1 decision models#audit benchmark#calibration#cross-question interference#prompt injection defense#S1-SHAP#interpretability#XAI#KernelSHAP
Share:

Related articles