System-1 Decision Model Benchmark: What Happens When Fast Models Meet Rigorous Evidence

When Agents Need to Make Quick Decisions

Modern AI agents constantly make countless small, precise judgments while executing tasks: which model to invoke, which tool to use, whether retrieved text is relevant, whether an input carries an injection attack. These decisions may seem trivial, but they directly impact the efficiency and cost of the entire agent system.

The conventional approach is to hand these judgments off to a large language model (LLM), but every LLM call comes with significant latency and cost. This is precisely why System-1 decision models emerged — they output classification probabilities in a single forward pass, promising to handle these decisions at a fraction of the cost and latency. Think of them as the machine equivalent of what psychologist Daniel Kahneman called "fast thinking."

A study published on arXiv (arXiv:2610.02267) conducts one of the more rigorous paired evaluations of this model class to date, and — remarkably — includes a self-audit of its own evaluation process, surfacing a number of issues worth paying close attention to.

The "System-1" label borrows from Kahneman's dual-process theory in Thinking, Fast and Slow: System-1 represents fast, automatic, intuitive thinking, while System-2 represents slow, deliberate reasoning. In AI agent architectures, System-1 decision models are typically small, fine-tuned classifiers or lightweight language models (often under 10 billion parameters) that skip multi-step reasoning and directly map inputs to a limited set of output classes (e.g., "relevant/irrelevant" or "call tool A/B/C"). By contrast, general-purpose LLMs like GPT-4 or Claude require a full autoregressive decoding process for the same kinds of judgments — single-call latency can range from hundreds of milliseconds to several seconds, and costs are orders of magnitude higher. This makes System-1 models appealing candidates for handling "routing and filtering" decisions in agent pipelines.

Two Models, Head to Head

The research team selected two representative System-1 models for comparison: the open-weight Laya and the hosted-service Jev. The evaluation covered 11 agent decision points, drawing data from 18 public sources to construct 7,283 base cases plus 6,640 robustness variants.

To ensure reliable conclusions, the study employed strict experimental design: byte-level identical inputs, paired statistical tests, and reproducibility checks across different hardware and dates. These practices are uncommon in AI benchmarking, yet they are precisely what separates genuine capability assessment from noise.

A Significant Accuracy Gap

The results show that the hosted model Jev is significantly more accurate on 9 of the 11 decision points, with margins ranging from +10.8 to +46.0 percentage points. This is a substantial performance gap, indicating that the hosted model has a meaningful advantage in engineering polish and data coverage.

Two Shared Weaknesses

Both models, however, exposed common blind spots. On zero-shot model routing — deciding which LLM to call — neither model outperformed random guessing, meaning System-1 models are currently unreliable for this task. On RAG relevance gating (judging whether retrieved text is actually relevant), the two models tied, with neither showing a clear advantage.

RAG (Retrieval-Augmented Generation) is the dominant approach for knowledge-augmented generation: before producing an answer, the system retrieves candidate text passages from an external knowledge base and appends them to the context for the LLM to reference. However, not all retrieved results are genuinely relevant — injecting irrelevant content wastes tokens and disrupts model reasoning, a phenomenon known as "context contamination." A relevance gating step is therefore often placed before the LLM to filter out noise. In theory, a System-1 model could perform this filtering with extremely low latency; but this study shows both models are evenly matched here and neither demonstrates a decisive advantage, suggesting that relevance judgment exceeds the capability boundaries of lightweight classifiers.

Robustness: The Open Model's Fragile Side

The open-weight model Laya performed poorly on robustness tests. When option order was reversed, it changed 30% of its answers — a sign that the model's judgments depend heavily on the positional arrangement of choices rather than their actual content.

More striking was its collapse when the candidate pool grew larger or more similar. When faced with 50 nearest-neighbor tools, Laya's accuracy on tasks with a single correct tool plummeted to 31%, while Jev maintained 98% accuracy. This contrast underscores an important point: in real-world agent environments full of distractors, stability may matter more than peak accuracy.

The Most Valuable Part: The Self-Audit

The most valuable contribution of this study is the team's rigorous audit of its own evaluation process. They proactively identified three analytical errors and one design confound — all of which had distorted the initial deployment benefit claims:

• Overlooked pre-filtering costs: The originally reported 23.9% cost savings, once corrected, shrank to just 4.3%. This is nearly an order-of-magnitude difference, a reminder that all hidden steps must be accounted for when calculating gains.
• Gate accuracy mistaken for end-to-end quality: A reported 58% figure diverged sharply from the true end-to-end 98%, conflating a local metric with overall system performance.
• In-sample threshold setting: When the 5% false-positive target was set, the miss rate on the held-out set reached as high as 17%, showing that threshold performance on training data cannot be directly extrapolated to deployment.
• The injection false-positive "channel effect": This effect disappeared when using channel-native content, indicating it was an artifact of experimental design rather than a real phenomenon.

The team also noted that two additional suspected confounds, upon investigation, did not change the final conclusions. This kind of intellectual honesty is rare in an AI evaluation landscape awash with inflated claims.

The "in-sample threshold" problem is a classic trap in ML evaluation, worth unpacking. In binary classification tasks (e.g., detecting injection attacks), researchers must choose a probability threshold to decide between outputting "yes" or "no." If that threshold is found by optimizing a target (such as keeping the false-positive rate below 5%) on the training or validation set, it will naturally overfit to those samples — the model's confidence distribution happens to align with the threshold's assumptions on that data, but in real deployment, the distribution may shift, causing the actual miss rate to far exceed expectations. The large gap between the 17% miss rate on the held-out set and the 5% target directly demonstrates this mechanism. The lesson for engineering teams: thresholds must be validated on a holdout set that is completely independent of training data and calibrated to the expected deployment distribution.

Implications for Practitioners

This study sends several clear signals. System-1 decision models can deliver real value on certain agent decision tasks, but they are not a universal solution — model routing and relevance gating remain genuinely hard problems.

On model selection, hosted models currently lead in accuracy and robustness, but the transparency and controllability of open-weight models still have value, provided their stability under increased distractors is thoroughly tested.

The deeper lesson is about evaluation itself. The paper's title, "Fast Models, Slow Evidence," cuts to the heart of the matter: models can be fast, but the evidence supporting deployment decisions must be able to withstand repeated scrutiny. Hidden costs, metric conflation, and in-sample overfitting are the traps that cause surface-level performance numbers to paint a wildly misleading picture.

The research team has open-sourced all cases, raw outputs, and analysis code on GitHub (David-DL-Space/sys1-eval), providing a reproducible foundation for follow-on research. For engineering teams building agent systems, this is an exceptionally valuable methodological reference.