Testing the Jev Decision Model: System 1 AI Performance and Improvements in Online Learning Environments

Recently, a class of models known as "System 1" decision models has been gaining traction in the AI community. Following TypeSafe's Jev and the open-weight Laya model, a wave of new products has emerged — including AWS Strands Decider, Cloudflare Clef, and OpenAI's Decisions API. A Reddit developer shared his experiments benchmarking the Jev model on sequential decision tasks, revealing both the capability boundaries and the improvement potential of this model class.

System 1 vs. System 2: A Cognitive Framework for Decision Models

The theoretical starting point for this experiment comes from Kahneman's dual-process theory, introduced in Thinking, Fast and Slow. Recognizing an emotion on someone's face the moment you see it is a System 1 response — effortless and automatic. Calculating 17×24, on the other hand, requires the deliberate, effortful reasoning of System 2.

The experimenter drew an interesting analogy: if evolution and social conditioning have made humans skilled at reading faces, then machines trained for decision-making should be equally capable of weighing options with precision — we want probabilistic judgment to become their "first instinct." That's precisely what System 1 decision models like Jev aim to achieve: fast, intuitive probability outputs for decisions, rather than the lengthy chain-of-thought reasoning characteristic of inference models.

A key question arises, however: to what extent do Jev's probability outputs depend on the specific text used to describe tasks and options? Are we inadvertently "baking in" biases from human-authored training corpora into decision models? To strip away textual framing and isolate pure decision-weighting ability, the experimenter designed an online learning game.

Kahneman's dual-process theory originated in cognitive psychology and has since been borrowed by AI researchers to distinguish between two fundamentally different model architecture philosophies. In the context of large language models, System 2 corresponds to "reasoning models" like OpenAI's o-series and DeepSeek^Ad-R1 — they generate extensive internal chain-of-thought before producing an answer, improving accuracy through explicit step-by-step derivation, but each call can take seconds or even tens of seconds with significantly higher token consumption. System 1 models attempt to bypass this process entirely, directly outputting probability distributions or decision weights, targeting "good enough" judgments at millisecond latency and minimal cost. This distinction is particularly significant in latency-sensitive applications like real-time bidding, recommendation systems, and automated trading — scenarios where a slow-but-accurate answer is often less valuable than a fast-and-reasonable approximation.

Experiment Design: A Four-Action, Sixty-Four-Round Online Learning Game

The core setup of the experiment is elegantly simple. The game consists of four actions played over 64 rounds. Before each decision, the model sees the historical loss records for each action. Jev's output probabilities are converted into allocation weights, and the model bears the weighted average loss across those weights.

The constraints are worth noting: all losses are fixed before the game begins and are revealed one round at a time. What's truly being tested is whether the model can identify and exploit patterns in the revealed sequence — the classic "regret minimization" problem in online learning.

The cleverness of this design lies in how it abstracts decision-making away from specific textual context, leaving only pure numerical trade-offs. If a model is merely reciting decision patterns from its training corpus, this abstract numbers game will expose it immediately.

Regret minimization is a foundational concept in online learning theory. "Regret" is defined as the difference between an algorithm's actual cumulative loss and the loss of the best fixed strategy in hindsight — essentially, "how much less could I have lost if I'd known the optimal action from the start?" A well-designed online learning algorithm should guarantee that regret grows far slower than linearly with the number of rounds, typically targeting O(√T). Classical algorithms like Multiplicative Weights (Hedge) and UCB (Upper Confidence Bound) have been proven to achieve this guarantee under different assumptions. Applying this framework to evaluating decision models means we're not just asking "how often did the model guess correctly," but rather "how much potential value did it waste compared to the theoretical optimum" — a stricter and more discriminating standard.

Out-of-the-Box Performance: Jev Degrades to "Follow-the-Leader"

The results were disappointing. Jev performed poorly in this environment: it placed nearly all weight on whichever action had historically performed best, exhibiting behavior that closely resembles the classic Follow-the-Leader (FTL) strategy.

FTL has its use cases — it performs well when one action is dominant throughout. But faced with simple alternating patterns, this strategy falls flat. When the optimal action switches between rounds, blindly following the historical winner leads to sustained losses.

This result confirmed the experimenter's concern: an out-of-the-box System 1 decision model behaves more like an intuitive "bet on the historical winner" than a genuine perceiver of sequential patterns. It fails to exploit the structural information hidden within the sequence.

Improvement Path 1: The Cost of Using a Reasoning Model

One straightforward improvement is to feed the problem into a reasoning model like Luna. In some environments, Luna performs significantly better — it can identify patterns in the sequence through explicit reasoning.

But the trade-off is obvious. Luna is a classic System 2 model, meaning substantially higher latency and cost. Using a heavyweight reasoning model to solve problems that a lightweight decision model should handle is rarely worthwhile in production deployments. This is precisely why System 1 models exist — we need fast, low-cost decision-making capability.

Improvement Path 2: Separating Prediction from Allocation

The second approach proposed by the experimenter is more insightful: let Jev handle only pattern recognition, rather than making direct choices.

Concretely, instead of asking the model to choose between actions, the model is asked to predict independently for each action — whether that action will incur a loss in the next round. These are called "Noul predictions." The experimenter then converts these predictions into two strategies:

• argmin strategy: places all weight on the action predicted to have the lowest loss
• softmin strategy: spreads weight across multiple actions when their predictions are close

The results were dramatic. Noul + argmin successfully exploited a perfect alternating pattern, while softmin reduced losses when rankings were unreliable. Separating the "prediction" and "allocation" steps produced a meaningful improvement.

The insight behind this finding is that System 1 models aren't incapable of perceiving patterns — they just perform poorly when simultaneously handling both prediction and decision allocation. Once the task is decomposed, letting the model focus on what it does well (local per-action prediction) while using deterministic rules for allocation, overall performance improves significantly.

The softmin strategy here is conceptually closely related to the exploration-exploitation tradeoff in reinforcement learning. The argmin strategy is pure exploitation — always selecting the currently predicted best action, maximizing gains when patterns are clear but highly sensitive to prediction errors. Softmin introduces a degree of exploration by spreading weight across multiple candidate actions, trading some potential gain for robustness against prediction uncertainty. This design philosophy mirrors the Hedge algorithm's exponential weighting mechanism: maintain spread when evidence is insufficient to differentiate options; gradually concentrate when one option's advantage becomes sufficiently clear. Migrating this logic from the algorithmic level to a hybrid architecture of "model prediction + deterministic rules" is arguably the most practically valuable engineering finding of this experiment.

Implications for Decision Model Training

The experimenter draws an optimistic conclusion: training environments purpose-built for these kinds of decision tasks have the potential to significantly improve System 1 model capabilities.

This insight has implications for the wave of decision APIs now emerging (such as OpenAI's Decisions API and AWS Strands Decider). It suggests that progress in decision models may depend not only on larger parameter counts or stronger reasoning capabilities, but on the design of training environments and task structures — teaching models to internalize probabilistic trade-offs as "first instinct" without relying on heavy inference machinery.

It's worth noting that this article is based on a single developer's experiment. The model names — Jev, Laya, Luna, Noul — all come from the original post. The experimenter has made the relevant code publicly available (Google Colab) for those who wish to reproduce and verify the results.

Conclusion

This small-scale experiment, while limited in scope, offers a valuable perspective: evaluating System 1 decision models requires more than looking at out-of-the-box performance — it means understanding how they fail on abstract decision tasks. Separating prediction from allocation, and designing training environments with targeted task structures, may be the key direction for these fast decision models to break through their current capability ceiling.