LLMs Don't Need to Write the Answer—Just Point to It: Pointer-Style Models Explained

Why Does Your Model Keep "Writing" the Answer?

When you hand an LLM a customer support ticket along with four candidate labels, it could simply tell you which label matches—but instead, it spells out {"label": "billing"} token by token. The word "billing" was already in your prompt. Extraction tasks work the same way: the invoice number is right there in the PDF, yet the model retypes it for you—usually correctly, but that means it has the freedom to get it wrong.

This waste is baked into how LLMs read and write. They read your prompt once in a single pass, then generate the response token by token, running the model again for each output token. That's why output costs so much more than input: according to Anthropic's pricing, Sonnet 5.5 charges $2 per million input tokens but $10 per million output tokens.

For short labels, cost isn't the core issue. The real problem is this: a generator can "write" anything—"Billing," "billing_dispute," or even "Sure! The label is billing." So you end up adding parsers, retries, and schemas. JSON mode helps with format drift, but as one guide puts it, it's only "a format constraint, not a correctness guarantee."

Pointer-Style Models: Only "Pointing" to Answers from the Input

The pointer approach is completely different. It reads the input once, then scores candidates that already exist in the input: either each label you've provided, or every possible text span in the document. Answers can only come from the input itself.

This isn't a new concept. Pointer Networks (2015) attempted to select elements directly from the input as output; SQuAD (2016) defined each answer as "a span of text in the corresponding reading passage"; BERT answered extractive questions by scoring each token as "answer start/end"; and named entity recognition still returns start and end offsets to this day.

The four small task models from ScaleDown (called Constrained Decoding Models, or CDMs) follow exactly this path. Their joint article with LangChain summarizes it: "A CDM is trained for a single task and only returns answers that task permits, such as one label from your list, or certain text spans from the input."

Pointer Networks, proposed by Vinyals et al. in 2015, are neural network architectures that use attention mechanisms to point directly at positions in an input sequence. Unlike traditional sequence-to-sequence models, a pointer network's output vocabulary size is dynamically tied to input length—each output token corresponds to an index in the input rather than being generated from a fixed vocabulary. This makes them naturally suited to combinatorial optimization problems (like the traveling salesman problem), sorting, and later, widely-used extractive question answering and entity recognition. Their core advantage: outputs necessarily belong to the input set, making "fabrication" mechanically impossible. Modern extractive models (like BERT's QA head or NER's start/end classifiers) differ in implementation but are conceptually continuations of pointer networks—scoring the input rather than generating freely.

ScaleDown CDM's Four Models: Two That "Point," One That "Writes"

According to ScaleDown's documentation, the four models behave differently:

• Classify: Scores text against each label's rubric, returning top_label and scores that sum to 1.0. It points.
• Extract: Returns "text spans extracted exactly from the input," with confidence and start/end offsets. It points.
• Summarize: "Fluent paraphrasing, not lifting original sentences." It writes.
• Compress: Compresses long context. The documentation doesn't specify whether it selects or rewrites.

So the "pointer" concept only covers two of the four models. One naming caveat: what's usually called "constrained decoding" refers to using masks to constrain a regular LLM's tokens so it can't break your schema (e.g., Outlines, XGrammar)—here it's a product name.

Constrained Decoding in mainstream LLM engineering refers to forcing a model at inference time, via logits masking, to only output token sequences that conform to a specific grammar or schema—representative tools include Outlines, XGrammar, and Guidance. It's fundamentally different from the "pointer" concept: constrained decoding still has a generative model outputting token by token, just with a restricted set of legal tokens; pointer models don't generate new text at all—they only select existing spans from the input. ScaleDown naming its product "Constrained Decoding Models" can easily be confused with this general technique, but the former is a vendor-specific small task model, while the latter is a decoding-time constraint applicable to any LLM. Understanding this distinction helps assess capability boundaries and appropriate use cases.

The Real Advantage: Verifiable "Receipts"

Speed improvements are nice, but the key point is this: pointer-style answers come with their own proof. If Extract tells you the invoice number is at characters 8 through 17, you can slice your own string to verify it directly. A generative answer that transposes two digits looks identical to a correct one.

This verifiability has enormous engineering value—it turns "the model says it's right" into "you can independently verify it's right."

In information extraction, start/end offsets refer to the character start index and end index (typically left-closed, right-open) of the answer text in the original input string. They form a lightweight traceability mechanism: the application layer can use input[start:end] to slice and verify directly, without trusting the model's self-report. Offset reliability depends on how the tokenizer maps to character indices—if the model returns token indices rather than character indices, additional alignment steps are needed, especially with multi-byte characters (like Chinese). This is also a key detail when evaluating pointer-style extraction models: whether offsets can be used directly for slicing the original string determines whether "verifiability" is truly out-of-the-box or requires extra engineering adaptation.

Where It Fits in Your Application Architecture: Around the Main LLM, Not Replacing It

These small models aren't meant to replace your main LLM—they work around it: small models handle routine classification, extraction, and compression, while the general model handles what genuinely needs "writing."

A typical architecture: user requests go through Classify first, which routes to the main LLM; tool outputs go through Compress before reaching the main LLM; agent execution traces go to a second Classify to check for loops, terminating the run if the "looping" score exceeds 0.8; documents go to Extract, writing text spans with offsets to the database.

But this loop check has a prerequisite: the 0.8 threshold must be meaningful for your data. ScaleDown hasn't published calibration data, and Classify "always returns a winner—even when the model isn't sure." So in practice, add an other label and test on a few hundred of your own samples.

Half-Competitors: Platforms That Only Do "Label Selection"

TypeSafe's Jev, OpenAI's Decisions API, and Cloudflare's Clef can all answer "typed questions" with probabilities. That covers the Classify half. But none of them document support for "pointing at text spans."

If you arrange these platforms across three slots—"select labels, point at spans, summarize"—ScaleDown CDM fills all three, charging $0.05 per million input tokens; while Jev ($0.042), OpenAI Decisions ($0.10, secondary source, Beta), and Cloudflare Clef ($0.038–$0.24) only fill the "select labels" slot, leaving the other two empty. All four platforms provide free outputs.

The Boundaries of the Pointer Approach

Pointers aren't universal. There are clear limitations:

• The answer must be in the input. "How many line items are on this invoice?" requires addition—there's nothing to point at.
• A legal label isn't necessarily the correct label. It can point at the wrong thing.
• Summarize is writing, and Compress is unspecified. Don't assume the "can't fabricate text" property covers these two models.
• The big numbers come from ScaleDown itself. Tweets claim 200x faster and 467x cheaper; ScaleDown's own benchmark documentation says 8% to 19.5% accuracy improvement and 2.2x to 13.9x speed. No independent third party has verified either set of numbers.
• Your text leaves your servers. That's the inherent cost of a hosted API.

When to Use Pointer-Style Models

If the LLM call you're making is essentially selecting among given options or extracting spans from source text, then a generative model is doing redundant work and introducing unnecessary error surface. Pointer-style models turn these tasks back into multiple-choice and localization problems—answers come from the input, results can be independently verified, and cost and latency are significantly lower.

Conversely, scenarios requiring reasoning, computation, synthesized rewriting, or cross-document synthesis still belong with general-purpose LLMs. Combining the two, rather than choosing one over the other, is the pragmatic architectural choice.

One-line Summary

Pointer-style models select answers from the input rather than generating them, offering verifiability and cost advantages—but with clear boundaries.

Paragraph Summary

The article discusses the inefficiency of LLMs "rewriting" answers already present in the input token by token during classification, extraction, and similar tasks, and introduces ScaleDown's pointer-style small models (CDMs). Pointer-style models score existing candidates in the input to select labels or text spans, so answers can't fall outside the input range, and they come with start/end offsets that can be independently verified. Among ScaleDown's four CDMs, Classify and Extract "point," Summarize "writes," and Compress's behavior is unspecified. The article also compares competitors like TypeSafe Jev, OpenAI Decisions, and Cloudflare Clef, which only cover "label selection," and notes that the pointer approach can't handle computation, synthesized rewriting, or cross-document reasoning, with performance data still lacking independent verification.