System 1 Decision Model
A class of AI models that directly output a decision distribution from a single forward pass, in contrast to autoregressive System 2 models. It features low latency and is suited to scenarios such as real-time routing, triage, and safety filtering.
Timeline (last 90 days)
An open, deterministic audit benchmark evaluated five open-weight System 1 decision models ranging from 0.8B to 27B as well as a generative System 2 baseline, covering three dimensions: calibration, cross-question interference, and post-hoc attribution faithfulness.
The audit recommends treating the confidence of the System 1 model as a relative ranking signal rather than an absolute probability, and recalibrating with held-out data before deployment.
System 1 models with 4B parameters or more have a weighted ECE under syntactic perturbation (ECE_w) of 0.030 to 0.087, versus 0.144 for the log-probability baseline and 0.378 for the JSON-decoding baseline.
In the prompt-injection defense task (N=250), System 1 models with 4B parameters or more achieve an AUROC of 0.988 to 0.998, a recall of up to 95.2%, and a 0.0% false positive rate at 48 to 60 milliseconds of latency.
The experimenters concluded that System 1 models are not incapable of perceiving patterns, but perform poorly when simultaneously handling prediction and decision assignment; overall performance improved significantly after decomposing the task.
All Facts (5)
An open, deterministic audit benchmark evaluated five open-weight System 1 decision models ranging from 0.8B to 27B as well as a generative System 2 baseline, covering three dimensions: calibration, cross-question interference, and post-hoc attribution faithfulness.
50%UnverifiedThe audit recommends treating the confidence of the System 1 model as a relative ranking signal rather than an absolute probability, and recalibrating with held-out data before deployment.
50%UnverifiedSystem 1 models with 4B parameters or more have a weighted ECE under syntactic perturbation (ECE_w) of 0.030 to 0.087, versus 0.144 for the log-probability baseline and 0.378 for the JSON-decoding baseline.
50%UnverifiedIn the prompt-injection defense task (N=250), System 1 models with 4B parameters or more achieve an AUROC of 0.988 to 0.998, a recall of up to 95.2%, and a 0.0% false positive rate at 48 to 60 milliseconds of latency.
50%UnverifiedThe experimenters concluded that System 1 models are not incapable of perceiving patterns, but perform poorly when simultaneously handling prediction and decision assignment; overall performance improved significantly after decomposing the task.
50%