CLM: The Contrastive Language Model Architecture That Makes Decisions 13x Faster Than TiFF

As System One decision models come back into the spotlight, a new architecture from a Stanford and NVIDIA research team has been turning heads — the Contrastive Language Model (CLM). Its core claim is striking: decision-making speed up to 13x faster than TiFF-class models, at comparable accuracy. What makes it even more interesting is that this speed gain doesn't come from shrinking the model — it comes from a design philosophy fundamentally different from mainstream large language models.

Why "Non-Thinking" Models Are Making a Comeback

The vast majority of decisions in real software systems don't actually require a model to "reason" for 30 seconds. Questions like "which tool should I call next," "is this ticket urgent," or "which link should the agent click" demand millisecond-level responses — not seconds or minutes of waiting.

This is the fundamental motivation behind the resurgence of System One models — fast, non-reasoning, non-thinking. There have been plenty of open-source TiFF alternatives already, but most are still language models at heart: they read your text, read every option, and then write out an answer token by token, even when that answer is just a single word with a probability attached. This architecture gets expensive fast as options multiply or grow longer.

What CLM Actually Is: CLIP for Decision-Making

The most intuitive way to understand CLM is to think of it as "CLIP for decisions." CLIP puts images and text into the same embedding space — a photo of a dog ends up near the phrase "a photo of a dog." CLM does the same thing for "contexts" and "actions."

It consists of two encoders: a state encoder that reads "what's happening right now," and an action encoder that reads "what you can do." Take Mario encountering a Goomba — the state is the game screen, and the actions are move left, jump, or run right. To make a decision, the model embeds the state and each option, then scores them by distance, and finally converts those scores into probabilities via softmax — say, jump 80%, move right 15%, move left 5% — and Mario jumps.

CLM is designed to be compatible with TiFF's API, offering three primitives: yes/no questions, multiple choice, and scoring. But the underlying mechanism is identical across all three — "one state plus a closed set of candidates." A yes/no question is just multiple choice with two options (true and false). This also means that, like TiFF, CLM can never hallucinate an answer outside the list — but it can still pick the wrong one from the list, and it'll do so with confidence. "No hallucination" doesn't mean "always correct."

CLIP (Contrastive Language–Image Pre-training) is a multimodal model released by OpenAI in 2021. It uses contrastive learning to map images and text into the same high-dimensional vector space, so semantically related image-text pairs end up closer together. The core of this approach is the dual-encoder (dual-tower) architecture: two independent encoders process inputs from different modalities, and cosine similarity or dot products measure the degree of match in a shared space. The biggest advantage of the dual-tower architecture is that one side's embeddings can be pre-computed and cached — at query time, you only need to encode the other side and run a vector search, making computation extremely lightweight. This is the architectural root of CLM's speed advantage. The tradeoff is that neither encoder can "see" the other during computation, which prevents fine-grained cross-attention comparisons — which is exactly why CLM's accuracy drops when it needs to distinguish between highly similar candidates.

Contrastive Learning: Where the Free Negative Samples Come From

Getting the correct action to land near its corresponding state requires contrastive training. The training data consists of large numbers of "context–action" pairs — actions someone actually took in a given context. The model's job is simple: pull each context closer to its matching action, and push it away from all other actions. Repeat this millions of times, and contexts will reliably end up near the right actions.

The clever part is that you never need anyone to write "wrong answers." Suppose you sample 1,000 examples at a time. For Mario's screen, "jump" is the correct answer — and the actions from the other 999 contexts are naturally wrong for this one. Contrastive learning needs positive and negative samples, and those negatives come for free.

Contrastive learning is a self-supervised training paradigm whose core goal is to teach a model "what should be close to what, and what should be far from what" — rather than predicting specific labels. Its loss function is typically based on InfoNCE (a variant of Noise-Contrastive Estimation): for a positive pair, maximize their similarity while minimizing similarity to all other samples in the current batch (i.e., "in-batch negatives"). The larger the batch, the more in-batch negatives, and the stronger the discriminative pressure on the model, resulting in finer-grained representations. This is why CLM's second training stage specifically introduces hard negatives generated by Gemini — random negatives tend to be obviously different from the positive sample and easy to distinguish; hard negatives are semantically similar but incorrect, forcing the model to draw more precise boundaries in the embedding space. This jump drives accuracy from 52% to 69%.

The Real Source of Speed: Splitting and Caching

TiFF-class models feed the state and all options into the model together on every call. Double the options or make them longer, and you're reading twice as much content — plus decoding the answer token-by-token from a large model — and costs skyrocket. In the team's charts, constrained decoding with Qwen3-8B pushes latency above 4 seconds.

CLM splits the two apart. In most agent loops, the action set doesn't change between steps — only the state does. So CLM embeds the actions once and caches them. Each subsequent step only needs one embedding pass for the state, followed by a dot product against the options — and dot products are essentially zero-cost. In the Mario demo, this compresses 5 forward passes per step down to 1. As the number of candidates scales from 1 to 1,000, CLM's latency curve barely moves: near 1,024 options, it takes just 44 milliseconds, while TiFF takes 570 milliseconds — a roughly 13x gap.

Three-Stage Training With a Frozen Backbone

Why does CLM understand "right vs. wrong" better than a typical embedding model? Ask a vanilla Qwen3-8B embedding model "who wrote Romeo and Juliet" and Shakespeare comes in third — because "similar" isn't the same as "correct." After CLM's training, Shakespeare ranks first.

This comes from three-stage training. Stage one uses 60 million Q&A pairs from NVIDIA Nemotron data to learn general world knowledge. Stage two is the most critical: the team used Gemini to generate 30 million hard negatives, teaching the model to distinguish "sounds right" from "actually right." Order matters: pre-training alone scores 52% on the hard-negatives benchmark; adding the hard-negatives stage jumps it to 69%. But training with hard negatives from day one only peaks at 62% and overfits quickly. Learn the world first, then learn the nuances. Stage three turns it into an agent — approximately 1 million real agent trajectories, with 40% of the original Q&A data replayed as a buffer; without this, the hard-negatives score drops back from 69% to 56%.

Surprisingly, the 8-billion-parameter backbone is frozen — Qwen3-8B acts only as a reader and receives no gradients. What actually gets trained are the small projection heads on each side, each around 20 million parameters. Because the backbone is unchanged, the entire dataset can be embedded in one shot, and a full pre-training run takes only about an hour on a single RTX.

Constrained decoding is a common approach for getting structured output from mainstream large language models: during autoregressive generation of each token, the vocabulary is restricted or sampling is guided to force the model to produce only content matching a predefined format (e.g., JSON, enumerated values). This method requires no retraining and can reuse existing LLMs directly, but every inference still requires a complete autoregressive decoding pass — tokens generated one at a time, with the state and all candidates concatenated into a single context window. As the number of candidates and text length grow, KV cache usage and decoding steps scale together, driving latency sharply upward. The 4-second-plus latency mentioned for Qwen3-8B with constrained decoding is a direct manifestation of this bottleneck. CLM sidesteps autoregressive decoding entirely by turning decision-making into vector retrieval — this is the fundamental reason the two architectures differ by an order of magnitude in speed.

A Live Demo on DGX Spark — and the Real Weaknesses

The authors ran a full demo on a DGX Spark: Qwen3-8B as the encoder through a VLM service, with the team's CLM server on top (the service itself is about 75 MB; the full system uses roughly 25 GB of VRAM).

In a support ticket scenario involving repeated billing charges and unreachable customer service, a single call answered three questions: is it urgent (urgent 84%), which team should handle it (billing 99%), how frustrated is the customer (very angry). On a second call with the same ticket, the cached response came back in 2 milliseconds. The authors also built a list of 1,080 tools (Stripe, Trello, QuickBooks, Zoom, etc.) — TiFF has a cap of 255 options and can't fit them all in a single call. The first request took a few seconds to embed all the tools; subsequent requests ran at about 80 milliseconds, with virtually no difference between choosing from 1 tool or 1,000.

The weaknesses are just as clear, and they're the classic tradeoff of dual-encoder design: the state is encoded independently and never sees the options, so it can't compare candidates side by side. In testing, selecting from 8 tools achieved 86% accuracy; scaling to all 1,080 tools, accuracy plunged to 17%, mostly from confusing similarly named tools (like multiple refund tools). The WikiRacing task told the same story — sometimes precise, sometimes chaotic.

How to Use CLM as a Developer

At its core, CLM turns "decision-making" into "retrieval" — encode the context, look up the action, and make 1,000 options roughly as fast as 1. But while latency scales, accuracy doesn't scale proportionally.

The authors' practical advice is to use CLM as a first-stage coarse filter: use it to narrow thousands of tools or candidates down to the top 10 in milliseconds, then hand those off to a System Two model that can read options side by side for the final decision. If your action set is already small and stable (like a game loop or a handful of routing options), CLM works well on its own. Avoid throwing a thousand tools at an agent directly — a better approach is to split into sub-agents, each with its own smaller tool set.

The current version supports text only. The team has announced a multimodal version at around 5 billion parameters coming next month. Given that the scaling curves show the biggest gains from encoder size, a larger encoder is precisely what's needed to address the "confusing similar options" weakness — and that makes it worth watching.

The authors close with an observation: old algorithms are coming back. System One models have brought fast, non-reasoning models back into fashion, and CLM has brought contrastive learning — big around 2017–2018 — back with them. We may be far from having exhausted what existing algorithms can do.

This two-stage coarse-filter + fine-ranker architecture has well-established precedent in information retrieval: Stage one uses a dual-tower model (such as BM25 or vector search) to quickly recall the Top-K from millions of candidates; Stage two uses a cross-encoder to finely rerank a small candidate set. A cross-encoder concatenates the query and each candidate before feeding them into the model together, enabling full cross-attention and higher accuracy — but its computation scales linearly with the number of candidates, making it suitable only for small sets. Treating CLM as the "Stage 1 retriever" in this system, paired with a System Two model that can read options side by side as the "Stage 2 reranker," is currently the best engineering path to balancing latency and accuracy.