Replacing LLM Routing with Small Encoders: Boosting Reliability with Conformal Prediction

Why Rethink LLM Routing

Most production routing solutions still send every request through a generative large model. The upside is generality and flexibility, but the costs are equally obvious: high latency, high cost, and unpredictable behavior. An online workshop on Reddit proposed a different approach — using small bidirectional encoders to make routing decisions and reserving LLM calls for edge cases that genuinely need them.

The workshop was led by Ben Auffarth, an AI consultant and bestselling author with over 15 years of data science experience. The event ran on Friday, October 30, from 9:30 AM to 12:30 PM (ET) as a livestream. The core thesis is straightforward: replace LLM routing with a quantized small encoder, then layer on conformal prediction to calibrate uncertainty, yielding a faster, cheaper, and more predictable system.

The Current State of Routing: Every Request Goes Through a Generative Model

The Hidden Cost of Generative Routing

Today's common routing implementations tend to hand every request to a generative model. Even when a request is just simple intent classification or short-text matching, it still goes through a full large-model inference pass. This means inference cost can't scale down with request complexity, and latency stays stuck above the floor set by the generative model.

Predictability is even trickier. Generative model outputs carry randomness, which undermines the stability of routing decisions. For production systems with strict SLAs, this uncertainty translates directly into operational burden.

What Small Encoders Can Do

The workshop's proposed alternative is a bidirectional encoder. These models are naturally suited to classification and matching tasks, and once quantized, they can run efficiently on CPU with throughput far exceeding generative models. Routing is essentially a classification problem — determining which category a request belongs to and which processing path it should take — which falls squarely within an encoder's capabilities.

Bidirectional encoder refers to models like BERT: when encoding each token, they simultaneously use context from both left and right sides, so the resulting vector representation captures the overall semantics of the sentence well. This differs from GPT-style autoregressive models, which only look at context from left to right. Precisely because bidirectional encoders need just a single forward pass to obtain a full-sentence representation, they are especially well-suited for discriminative tasks like sentence classification, intent recognition, and semantic matching (e.g., Sentence-BERT) — and these tasks are exactly the essence of routing decisions. Their typical scale ranges from tens of millions to hundreds of millions of parameters, far smaller than generative models with billions of parameters, so latency is lower, they can be batched in parallel, and memory footprint is smaller. The downside is they lack generative capability and can't handle open-ended Q&A or tasks requiring reasoning chains — which is precisely why the article advocates reserving LLMs for edge cases.

Breaking Down the Tech Stack: From Encoder to Conformal Prediction

The workshop's hands-on content covers a complete technical pipeline that can be broken into several key stages.

Quantized Encoders and CPU Throughput

The first step is building a bidirectional encoder router and quantizing it to improve throughput on CPU. The direct benefit of quantization is lower inference cost, and deployment no longer depends heavily on GPUs. For high-frequency, low-complexity tasks like routing, CPU inference is often sufficient.

Quantization refers to converting model weights and/or activations from high-precision floating point (such as FP32 or FP16) to lower-bit representations, with common forms including INT8 and, in recent years, 4-bit schemes. Taking INT8 as an example, model size can be compressed to roughly a quarter of the original, memory bandwidth pressure drops accordingly, and integer operations are often faster than floating point on modern CPUs, significantly boosting throughput. In practice there are two categories: post-training quantization (PTQ) directly calibrates and converts an already-trained model and is the easiest option; quantization-aware training (QAT) simulates low-precision errors during training and typically yields smaller accuracy loss. Note that quantization does introduce some accuracy degradation — usually acceptable for classification tasks, but it introduces bias into probability outputs. This point directly relates to the article's later motivation for calibrating with conformal prediction: the calibration layer can absorb the confidence shifts introduced by quantization.

Offline Hard Negative Generation

Training a reliable router hinges on the quality of negative samples. The workshop emphasizes that hard negatives need to be generated offline, not constructed on the fly within the training loop. Offline generation keeps negative samples closer to the real distribution and makes the training process more controllable.

SetFit Contrastive Few-Shot Training

The training stage uses SetFit, a contrastive few-shot learning method. Few-shot means you don't need massive labeled data to get a usable routing model, and contrastive training helps the model better distinguish between similar intent categories. For teams with limited labeling budgets, this path is quite appealing.

SetFit is an efficient method for few-shot text classification, proposed by the Hugging Face team. Its core idea: first fine-tune a sentence-embedding encoder on a small number of labeled samples using contrastive learning, pulling same-class samples closer together and pushing different-class samples apart; then train a lightweight classification head (such as logistic regression) on these embeddings. The contrastive learning stage generates training signal by constructing positive pairs within the same class and negative pairs across classes, so only 8 to 16 labeled samples per class are needed to approach the performance of full fine-tuning, and training typically completes in minutes on a single GPU or even CPU. Compared with directly fine-tuning a classifier, SetFit is less prone to overfitting on small data. The "hard negatives" it requires are exactly what the article emphasizes generating offline — hard negatives are negative examples that are semantically very close to positive samples and easily confused, and training with them forces the model to learn finer-grained decision boundaries.

Calibrating Uncertainty with Split Conformal Prediction

A routing system must know when it's uncertain. The workshop introduces split conformal prediction to calibrate uncertainty, providing statistical coverage guarantees for the model's outputs. When the encoder's confidence on a request is insufficient, the system can escalate it to the LLM.

This design redefines the LLM's role as a "fallback" — invoked only in validated edge cases. It preserves the generative model's general capabilities while avoiding its waste on the vast majority of requests.

Conformal prediction is a model-agnostic uncertainty quantification framework: it doesn't assume model outputs follow a particular distribution, but instead uses a held-out calibration dataset to compute "nonconformity scores" between model scores and true labels, and based on these produces a prediction set such that the true label falls within this set with a pre-specified probability (e.g., 90%) — a property known as the coverage guarantee. The split variant is the simplest: partition data into a training set and a calibration set, and after training, compute the quantile threshold only on the calibration set, making computational cost extremely low and well-suited to production. For routing scenarios, its value lies in turning "the model's confidence" into a statistically grounded judgment — when the prediction set is too large or contains multiple categories, it means the encoder cannot make a clear decision, and the system escalates the request to the LLM accordingly. Compared with softmax probabilities, the coverage guarantee from conformal prediction doesn't depend on whether the model is well calibrated, making it more robust.

Where This Approach Applies

A small encoder router is not a silver bullet. It assumes routing decisions can be modeled as a classification or matching problem and that sufficient representative data exists for training. If a business's intent categories change frequently, or the requests themselves require generative understanding, a pure encoder approach will fall short.

Introducing conformal prediction also has prerequisites: a usable calibration dataset is needed. The quality of the calibration set directly determines whether uncertainty estimates are reliable. If the calibration data deviates too much from the actual traffic distribution, the coverage guarantee will be distorted.

From an engineering perspective, the real value of this approach lies in separating "expensive" and "cheap" processing paths. The vast majority of requests go through the lightweight encoder, while a small number of hard cases go through the LLM, with both cost and latency varying dynamically with traffic structure.

Who Should Attend

This workshop targets engineering and algorithm teams dealing with routing problems. If you have a large volume of requests to route, are sensitive to LLM inference costs, or want routing decisions to have quantifiable uncertainty bounds, this technical path is worth attention.

Presenter Ben Auffarth's background spans AI consulting and data science practice. The workshop takes a livestream-plus-online format, and the agenda and full details can be found on the event page. For teams looking to deploy lightweight routing in production, this is an opportunity to connect encoders, few-shot training, and conformal prediction into a coherent pipeline.

One-line Summary

Replace LLM routing with small bidirectional encoders plus conformal prediction, reserving large-model calls for edge-case hard examples.

Paragraph Summary

The article introduces an online workshop on an alternative to LLM routing. Production environments today commonly send every request through a generative large model, resulting in high latency, high cost, and unpredictable behavior. Workshop presenter Ben Auffarth proposes using bidirectional encoders for routing decisions: these models suit classification and matching tasks, and once quantized can run efficiently on CPU with throughput far exceeding generative models. The technical pipeline includes quantized encoders, offline hard negative generation, SetFit contrastive few-shot training, and split conformal prediction for calibrating uncertainty. When the encoder's confidence is insufficient, the request is escalated to the LLM as a fallback. The approach assumes routing can be modeled as a classification problem with representative data and a calibration set, and is not suitable for scenarios with frequently changing intents or requiring generative understanding.