Open-Source Decision Model CLM Challenges Closed-Source Jev: Up to 13x Faster Latency

In the domain of AI Agent decision-making and verification, an open-source project called CLM (Contrastive Language Models) is generating significant discussion. Positioned as a self-hostable alternative to Jev — the closed-source model from TypeSafe AI — CLM builds a new set of projection heads on top of Qwen3-8B. Based on discussions in the Reddit community r/LocalLLaMA, CLM demonstrates clear advantages in latency and fine-tunability while maintaining functional parity — though it doesn't dominate across the board, as Jev still leads on several key dimensions.

What Is CLM: An Open-Source Mirror of Jev

CLM is not a large model trained from scratch. Instead, it layers a set of lightweight projection heads on top of the Qwen3-8B backbone, totaling only about 75MB. Its design goal is straightforward: serve as an open-weight, locally deployable alternative to Jev.

At the API and functional interface level, CLM fully replicates Jev's "System One" decision interface, supporting three core question primitives:

• Choice: Evaluates a set of discrete candidates and returns a categorical probability distribution;
• Noul: Outputs calibrated true/false probabilities for a given proposition or guardrail check;
• Score: Scores input according to an ordered rubric or scale.

Notably, CLM claims to provide Jev's complete primitive set — not a subset. Code originally written for the TypeSafe Jev client can be pointed directly at a clm-serve endpoint for a drop-in replacement (from clm import CLMClient, Choice, Noul, Score). This compatibility lowers the migration barrier considerably, especially for Agent systems already built on Jev's interface.

A Projection Head is a lightweight neural network module, typically consisting of a few fully connected layers, attached on top of a pre-trained backbone model. Its core function is to map the high-dimensional general-purpose representations (embeddings) produced by the backbone into a lower-dimensional space or probability distribution suited to a specific task. Because projection heads have very few parameters (CLM's total only ~75MB), the cost of training and fine-tuning them is far lower than full fine-tuning of an entire large model. This "frozen backbone + fine-tuned projection head" paradigm allows CLM to inherit Qwen3-8B's language understanding capabilities while adapting to decision-verification tasks at minimal cost — which is precisely why it can be compact yet claim full functional coverage.

CLM's Advantages: Low Latency, Fine-Tunability, and SOTA on Coding Verification

CLM's core competitiveness centers on three areas.

Latency and Decoupled Caching

Jev, as a proprietary cloud model, jointly evaluates state and question candidates. CLM separates the state head and action head into distinct processing steps. For Agents with persistent tool sets or fixed action spaces, CLM only needs to embed those actions once and cache them for reuse.

In benchmark tests on interactive browser Agents and game scenarios (such as T-Rex and Super Mario), CLM runs 4 to 13 times faster than Jev. This architectural decoupling essentially pre-computes and caches repeated calculations — a significant advantage for high-frequency Agent loops.

Open Weights and Fine-Tunability

Jev is a closed API; users cannot fine-tune it and can only prompt it via state and question instructions. CLM's projection heads are open-weight and extremely lightweight, allowing users to fine-tune them on their own Agent trajectories. This opens the door to deep domain-specific customization.

Coding Benchmark Verifier Performance

After fine-tuning on Agent trajectories, CLM achieves state-of-the-art performance on coding verification tasks: 87.6% on Terminal-Bench 2.1 and 81.6% on DeepSWE. By comparison, zero-shot Jev struggles on these benchmarks — achieving only around 71% on DeepSWE. This demonstrates that when domain data is available, the gains from fine-tuning are sufficient to outperform a larger closed-source model.

Terminal-Bench and DeepSWE are two representative benchmarks for evaluating AI Agent coding capabilities. Terminal-Bench focuses on completing command-line operations and system-level tasks in real terminal environments, testing an Agent's ability to use tools, recover from errors, and execute multi-step plans. DeepSWE focuses on software engineering scenarios, requiring Agents to understand codebase context and perform actual code modifications or debugging. Both share the characteristic of having deterministic, programmatically verifiable results, making them well-suited for measuring the accuracy of "verifier" models — i.e., models that judge whether a piece of code or an execution step is correct, rather than generating code themselves. This explains why projection-head models like CLM, which specialize in structured judgment, can match or even surpass general-purpose large models on these two benchmarks.

Jev's Moat: Generalization, Context, and Probability Calibration

Although CLM covers Jev's full functional surface, the current CLM-v0.1-8B still lags behind Jev on several dimensions that users should carefully weigh before choosing.

Zero-Shot Broad-Domain Knowledge

Jev is backed by a larger proprietary model and retains an edge-case accuracy advantage on zero-shot, open-domain tasks. Specific numbers: on the Berkeley Function Calling Leaderboard v4, Jev scores 99.2% versus CLM-8B's 95.2%; on WikiRacing, Jev achieves a perfect 30/30 while CLM-8B scores 26/30. The gaps are modest, but they remain meaningful in accuracy-sensitive scenarios.

Context Budget

Jev supports up to 64K token context out of the box. CLM-8B has only been tested and calibrated in the 2K–8K context range. While its Qwen3 backbone can technically accept longer prompts, representations beyond 8K have not been calibrated for the reference heads — meaning CLM's reliability in long-context scenarios is uncertain.

Probability Normalization

CLM computes probabilities via dot product plus softmax over the candidate set passed in each request, so its probabilities are inherently relative to the provided candidates. Jev's scores are internally calibrated against an absolute standard. This difference matters for application logic that needs to compare absolute probabilities across requests.

This distinction has important implications for real-world system design. Consider dot-product-plus-softmax relative probabilities: if an Agent provides candidate set {Action 1, Action 2} in request A, and {Action 1, Action 2, Action 3} in request B, even with identical semantic inputs, Action 1 will receive different probability values in each request because softmax redistributes probability mass across the current candidate set. This means CLM's outputs can only be used for within-request candidate ranking, not for cross-request absolute comparisons (e.g., "is the confidence in this request's best action higher than last request's best action?"). For Agent architectures that need to maintain global action quality assessments or compare decision confidence across time steps, this is a design constraint worth accounting for upfront.

Selection Guide: When to Choose CLM, When to Stick with Jev

Taking everything together: if you're worried that replacing Jev with CLM will sacrifice API functionality — the answer is no. You get the complete primitive set (Choice, Noul, Score), plus significant latency gains and zero API costs.

The trade-off is this: compared to TypeSafe's managed service, you'll give up some zero-shot generalization on niche out-of-domain tasks and be constrained by the current context calibration range.

In other words, for Agent systems with fixed action spaces, high call frequency, and available domain fine-tuning data (such as browser automation or coding verification), CLM is a highly compelling self-hosted choice. For general-purpose scenarios requiring broad knowledge, long context, and absolute probability calibration, Jev's closed-source advantages remain difficult to fully displace in the near term. As a v0.1 release, CLM's future iterations — especially expansion of its context calibration range — are worth watching closely.

It should be noted that this article is based on a single Reddit discussion thread and project self-reported data. The benchmark results have not been independently replicated by a third party. Readers should rely on their own testing in their specific scenarios when making actual selection decisions.