An LLM Call Is Really a Classifier: It Should Return a Label and a Probability, Not a Sentence

Is your LLM call really "generating" anything? When tasks like ticket routing, content moderation, transaction review, or lead qualification are wrapped up as a single "please choose from the following options" prompt, it is no longer a generation task — it is a classifier. Classifiers have a mature methodology: label definitions, an annotated test set, meaningful probabilities, actionable thresholds, and an escalation path. But most LLM pipelines skip all four.

A typical scenario: a ticket-routing prompt lists six queues (billing, account, technical, sales, urgent, general) and asks for a JSON field queue. The response passes schema validation, the queue name is always one of six, the integration has never errored, latency is normal, and all six queues are growing "reasonably" on the dashboard. Then someone manually labels 500 tickets and finds that 40 of them (8%) were routed to the wrong queue. At 20,000 tickets per month, that means roughly 1,600 tickets per month landing in the wrong team's hands. There is no alert, because there is nothing in the system to alert on — every output is a valid queue name.

It Was Always a Classifier

When the answer is determined before the input arrives to be one of a fixed set, the task is a classification problem. Classification has a mature discipline, and most LLM pipelines skip it:

1. A label set with written definitions — the answer is determined by the definition, not by the everyday meaning of the label name.
2. An annotated evaluation set drawn from real traffic — large enough to measure the error rate you care about.
3. Scores that rank model errors — ideally calibrated so that "0.9" roughly means nine times out of ten correct.
4. Thresholds and an escalation path — above the threshold, pass it through; below, send it to a human.

The router only has a simplified version of the first item and nothing else. Its prompt returns only a label, not a score, so there is nothing to threshold and nowhere to send the "unsure" cases.

Structured output makes the problem harder to see, not easier. Constraining the response to a JSON schema only eliminates the one kind of error you would definitely have noticed — malformed output — while leaving all other kinds of errors untouched. According to TypeSafe AI's Jev, this is a model that generates no text at all: it takes a question, an input, and a pre-declared set of options, and returns a probability distribution over those options, so the answer can only fall within the options. Compared to parsing free text, this is real progress. But what it guarantees is only the format of the answer, and a study published around the same time shows that the scope of this guarantee is far smaller than it appears.

Calibration is the core quality metric for classifier probabilities: a calibrated model that outputs 0.9 means that across many samples of the same kind, roughly 90% really are in the correct class. An uncalibrated model can still have good ranking ability (i.e., high AUC, where high-score samples really are more often correct), but the values themselves cannot be used directly as probabilities. Modern neural networks generally suffer from overconfidence; Guo et al. in 2017 found that deep models on ImageNet had average confidence far higher than their actual accuracy, and temperature scaling — scaling the logits by a single temperature parameter T and then applying softmax — is a simple and surprisingly effective post-hoc correction. Platt scaling and isotonic regression apply to binary classification and monotonic non-parametric settings respectively. Calibration must be fitted on held-out annotated data, not the training set, or it will overfit the score distribution. For LLM pipelines, uncalibrated scores can still be used for ranking and bucketing, but only calibrated scores can serve as a trustworthy risk level in threshold rules.

A Legal Label Is Not a Correct Label

Sun, Xu, Shi, and Yang of Fudan University and the University of Science and Technology of China tested Jev and two isomorphic models in the paper "Type-Safe Is Not Error-Free" (September 2026): Laya based on ModernBERT and Open-Jev based on DeBERTa-v3. For all three, each option is a short name plus a definition. For example, the name "yes" comes with the definition "a human should inspect this run," and "no" comes with "no human attention needed."

The experimental design is extremely simple: the question, input, names, and definition texts are all identical; the only change is which name is attached to which definition, with the new pairing written out in the input. A model that reads the definition will pick the same definition in both versions and therefore return the other name; a model that reads the name keeps returning the same name, which now means the opposite. The test set is 1,200 binary decisions across four workflows: invoice reconciliation, agent trajectory triage, security alert classification, and customer service escalation.

Laya's results are as follows:

Option names	Decision flip rate	Original pairing AUC	Swapped pairing AUC
yes / no	76.9%	93.8%	23.2%
false / true	49.7%	96.2%	57.7%
rejected / accepted	45.8%	90.8%	63.6%
0 / 1 (neutral)	6.5%	94.2%	92.4%
A / B (neutral)	6.0%	94.2%	94.1%

Data excerpted from selected rows of Table 1 in the paper.

With yes/no naming, three-quarters of the decisions flipped. The 23.2% AUC is the more critical signal: it is below the 50% of random guessing, meaning the model's scores can still distinguish the two classes, just in the reversed direction. As the authors point out, AUC does not depend on the decision threshold, so "this error cannot be corrected by adjusting the threshold."

Jev itself is less extreme, but the pattern is consistent. With yes/no naming it flipped 32.5% of decisions, 30.4 percentage points higher than with 0/1 naming, and AUC dropped from 81.5% to 58.1%. Since Jev is not deterministic, the authors also ran the original version twice to see natural variation: at most only 1.33% of decisions changed between runs, meaning the yes/no flip rate is 24 times this noise floor.

Under all conditions, the type error rate was always 0%. Every answer was a legal option. The authors' conclusion is worth remembering: "Type safety guarantees where a decision can land, but not what the options mean to the model."

Two results point directly to the solution. After replacing the names with random five-character strings, all three models returned to the flip rates of neutral names, and their pre-swap accuracy was within 3.3 percentage points of that with neutral names — meaning is carried by the definition, and the models did follow the definitions. Polarity matters more than familiarity: pairs like "absent/present," "negative/positive," and "rejected/accepted," which Laya had never output during training, still raised its flip rate by 41 percentage points over neutral names.

Why This Still Matters When Names and Definitions Agree

No one would deliberately attach "no" to the definition of yes, so it is reasonable for production prompts to be concerned about this. The swap experiment is a measurement tool, not a production scenario; it is the cleanest way to determine whether the model is reading the definition or the name. On ordinary inputs the two agree, and accuracy cannot tell you which one it is following.

This becomes deadly wherever the definition is narrower than the word's meaning. In the router, the "urgent" queue is defined as "production systems are down or customers cannot log in." A ticket saying "I urgently need a copy of last month's invoice" belongs to the billing queue by definition and to the urgent queue by dictionary meaning. If the model weighs the name more than the definition, it will go to the wrong place. The same applies to "spam" (defined as excluding legitimate cold sales emails), "fraud" (defined as a specific set of chargeback patterns), and "qualified" (by the sales team's criteria rather than the word's meaning).

This is not new for language models in general. Wei et al. (2023) pointed out that small language models "ignore flipped labels in context and thus rely mainly on the semantic priors from pretraining," and that overriding these priors is "an emergent ability of model scale," while instruction tuning "both enhances the use of semantic priors and the ability to learn input-label mappings, but the former more strongly." Liusie, Manakul, and Gales (2023) found that in zero-shot prompt-based classifiers, "the precise choice of prompt template and label words can greatly affect performance, and semantically equivalent settings often show significant performance differences." What this 2026 paper adds is a clean measurement method on the latest class of models designed to avoid this problem.

AUC (Area Under the ROC Curve) measures a model's ability to rank positive and negative samples: the probability that, given one random positive and one random negative, the model gives the positive a higher score. An AUC of 0.5 is equivalent to random guessing, and 1.0 means perfect ranking. Its key property is that it does not depend on any specific decision threshold, so it independently reflects the model's discriminative power. In the paper, after swapping names and definitions the AUC fell to 23.2%, below 0.5, meaning the model's scores could still distinguish the two classes but the direction was systematically reversed — giving high scores to samples that should have been low. This reversal cannot be fixed by moving the threshold, because the threshold only changes the cut point, not the ranking direction. This also explains why looking only at accuracy misses the problem: when classes are roughly balanced, a direction-reversed model may still have accuracy around 30%–50%, looking merely like "poor performance," while an AUC below 0.5 clearly exposes the structural failure of semantic reversal.

Emergent ability and semantic prior are the key background for understanding this phenomenon. Wei et al.'s 2023 work showed that small language models, when faced with flipped labels in context, often ignore these new mappings and continue to rely on the semantic priors learned during pretraining — for example, linking "yes" to affirmation and "no" to negation. Overriding this prior and truly learning a new input-to-label mapping is regarded as an emergent ability that appears only with larger model scale. Although instruction tuning enhances both the use of priors and the ability to learn new mappings, the former is enhanced more noticeably. This explains why the model still answers by name after names and definitions are swapped: in the pretraining corpus, the binding between "yes/no" and meaning is far stronger than the temporary definitions in the prompt. Liusie et al. also found that in zero-shot prompt-based classifiers, the choice of label words greatly affects performance, and semantically equivalent settings often differ significantly. Therefore, putting meaning into definitions and using neutral IDs as names essentially cuts off the model's reliance on label-word priors and forces it to read the definitions you provide.

Run a Swap Experiment on Your Own Prompt

The most useful part is this: the test needs no labels, only your prompt, a batch of real input samples, and two extra calls per input. The core idea is — list options in the form "name: definition," compare the "definition selected by meaning" rather than the "name," and rerun after rotating all names by one position:

def build_prompt(text, options):
    listed = "\n".join(f"- {name}: {definition}" for name, definition in options)
    return (
        "Pick the option whose definition fits the ticket. "
        "Reply with the option name only.\n\n"
        f"Options:\n{listed}\n\nTicket:\n{text}"
    )

def picked_definition(text, options, classify):
    name = classify(build_prompt(text, options), allowed=[n for n, _ in options])
    return dict(options)[name]  # compare by meaning, not by name

def flip_rate(texts, names, definitions, classify, shift=1):
    original = list(zip(names, definitions))
    moved = list(zip(names[shift:] + names[:shift], definitions))  # shift=0 to measure run-to-run noise
    flips = sum(
        picked_definition(t, original, classify) != picked_definition(t, moved, classify)
        for t in texts
    )
    return flips / len(texts)

Run three variants on a few hundred real inputs:

1. Real names, rotated by one (shift=1) — this is the number you want.
2. Neutral names, rotated by one — replace the names with A, B, C and run again; this is the control group, because rotating neutral names should change nothing.
3. Real names, not rotated (shift=0) — run the same prompt twice; this is the noise floor, because non-zero temperature sampling changes some answers between identical runs.

If the first number is close to the other two, the model is reading your definitions. If it is much higher, part of your definition is just decoration and the label names are making some of the decisions. In that case the paper's fix is cheap: use neutral IDs as option names, put all meaning into definitions, and map IDs back to queue names in code. Then rerun the accuracy check, because this change is not automatically free — on problems with 3 to 16 options, even just switching Laya to neutral letters changes about half the decisions, even though the model's original accuracy on those problems was only 56.4%. Test your own prompt; do not assume.

One scope caveat: the paper tested typed decision models, not chat models that answer via JSON schema. Whether your model behaves like Laya, like Jev, or neither is exactly what these three numbers are meant to tell you.

A Probability You Can Threshold

The second thing the router lacks is a score. There are two common ways to get a score from an LLM, and only one is truly useful.

Ask the model to state its confidence. Adding a "confidence" field to the schema is easy, but it is basically uninformative. Xiong et al. found in a large-scale evaluation that LLMs "tend to be highly overconfident when verbally expressing confidence," with values "mostly falling in the 80% to 100% range, and usually multiples of 5, just like humans talking about confidence." In the router example above, the model reported 0.9 or higher on 488 of the 500 annotated tickets. There is no separable space for a threshold.

Read the probability of the label itself. If your provider exposes the token log-probability of the selected output (not all APIs or models support this), use the probability the model assigned to the chosen label. Single-token, mutually distinct option names make this straightforward. This is a real score, but do not assume it is calibrated. The GPT-4 technical report shows that "pretrained models are highly calibrated," but "after the post-training process, calibration is weakened," and its calibration plot is explicitly labeled "post-training significantly damages calibration." Typed decision models like Jev return distributions by design, which is a real advantage, but the same check still applies.

What you need to know is whether the scores rank the model's errors. Group the annotated set by score and count the proportion correct in each group. Here is an illustrative table a router might produce:

Probability of selected queue	Tickets	Correct	Correct proportion
Above 0.95	330	323	97.9%
0.80–0.95	90	81	90.0%
0.60–0.80	50	36	72.0%
Below 0.60	30	20	66.7%
Total	500	460	92.0%

Errors are not evenly distributed. One-third of the tickets (the 170 below 0.95) contain 33 of the 40 errors. This is exactly what makes a threshold viable.

The table also reveals that the scores are not fully calibrated: tickets below 0.60 are still two-thirds correct. If you want this number itself to be trustworthy, fit a calibration correction on a held-out annotated slice: temperature scaling (Guo et al. (2017) found it "surprisingly effective" for neural networks), Platt scaling, or isotonic regression on the raw scores. Reliability diagrams and these corrections are the subject of the calibration lecture in SophiArch's applied machine learning course, which walks through them in full on gradient boosting and random forest models — models whose raw probabilities likewise need this kind of correction.

Thresholds Are Determined by the Cost of Errors

A threshold is not a quality setting; it is a trade-off between two costs, both of which should be written down. Assume a misrouted ticket costs about 20 minutes of human labor (the wrong team reads it, determines it is not theirs, forwards it, and the customer is still waiting), while manually sorting a ticket takes about 1 minute and gets it right. Per 500 tickets:

Auto-route when probability is at least	Auto-routed	Missed errors	Sent to human	Cost (minutes)
No threshold	500	40	0	40 × 20 = 800
0.60	470	30	30	600 + 30 = 630
0.80	420	16	80	320 + 80 = 400
0.95	330	7	170	140 + 170 = 310

Under this cost structure, sending everything below 0.95 to a human cuts routing cost by about 60% (from 800 minutes to 310 minutes per 500 tickets). If manually sorting a ticket costs 3 minutes instead of 1, then the 0.80 threshold is better: 320 + 240 = 560 minutes versus 140 + 510 = 650 minutes. The right threshold comes from your cost structure, not from the model.

Tickets sent to humans come with another benefit: each one brings back a correct label, and precisely from the cases the model was least sure about. After a few months, this is a better training set than you could have carefully designed.

Beyond temperature scaling, it is also important to understand where log-probabilities come from. When a provider returns the token log-probability of the selected output, it reflects the model's distribution at the moment it generated that token, not confidence in the final class judgment. Only if the option names are single tokens and mutually distinct (avoiding tokenization ambiguities like "yes" versus "yes,") does this value approximate a class probability. Multi-token option names require multiplying or averaging the token probabilities, and mishandling this introduces bias. In addition, for models that have gone through RLHF or other post-training processes, the log-probabilities have been distorted by human preference data; the GPT-4 technical report explicitly states that post-training significantly damages calibration. Therefore, the first step after getting scores should be to draw a reliability diagram: bucket by score and count the actual accuracy in each bucket. Only after confirming that scores rank errors can you talk about thresholds and calibration corrections. For classification tasks, traditional classifiers such as logistic regression on TF-IDF features often give calibrated probabilities out of the box, which is one reason to include small classifiers as a candidate in the escalation ladder.

Climbing the Ladder

Once a decision has a label, a score, and a threshold, you can compare different implementations on equal footing. A reasonable order to try:

1. The model you are already using, with neutral option IDs, output constrained to the label set, and the probability of the selected label as the score. If accuracy, the swap test, and the score table are all acceptable at your volume, stop here.
2. The same model plus a calibration correction, fitted on a held-out label set, if you need probabilities to live up to their name (for example, if a downstream rule reads it as a risk level).
3. A small classifier trained on your own labels, at which point the escalation queue is continuously producing labels. Even logistic regression on TF-IDF features is worth trying as a baseline, because it is deterministic, its probabilities can be calibrated with standard tools, and it costs almost nothing to run. Whether fine-tuning or distilling a small model is worth going further is a separate cost question.
4. A dedicated decision model, such as Jev. These models return distributions by design and can be very fast. Run the swap test on it too, because this is exactly the class of model the paper studied.

The exit test is the same at every level: accuracy, the score table, and the swap flip rate, all run on your own annotated traffic. Not vendor benchmarks, not descriptions of how confident the model is in itself.

Five Questions for Any LLM Call That Returns a Label

• Does the answer always come from a fixed set? Then it is a classifier, no matter what the call looks like.
• Do you have a few hundred real inputs labeled by humans, and do you know the error rate on them?
• When you move a name to a different definition, how many decisions change — compared with neutral names, and compared with simply rerunning?
• Do the scores rank the errors at the bottom of the table? Is anything reading them as probabilities?
• Where do low-score cases go, and who labels them?

Back to the team from the beginning: they answered all five questions in two weeks. The billing queue stopped receiving password reset tickets, and the "urgent" queue became much quieter.

Key References

• Sun, Y., Xu, J., Shi, J., & Yang, Z. (2026). Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It.
• TypeSafe AI. Introducing System One models and Jev.
• Wei, J. et al. (2023). Larger language models do in-context learning differently.
• Liusie, A., Manakul, P., & Gales, M. (2023). Mitigating Word Bias in Zero-shot Prompt-based Classifiers.
• Xiong, M. et al. (2024). Can LLMs Express Their Uncertainty?.
• OpenAI (2023). GPT-4 Technical Report.
• Guo, C. et al. (2017). On Calibration of Modern Neural Networks.

One-line Summary

An LLM call with fixed options is really a classifier, and it needs four things: label definitions, an annotated set, rankable probabilities, and cost-driven thresholds.

Paragraph Summary

The article argues that LLM calls with fixed options — ticket routing, content moderation, transaction review — are really classifiers, not generation tasks, and need four elements: label definitions, an annotated evaluation set, rankable probabilities, and thresholds with an escalation path. Most pipelines lack all four. A 2026 Fudan/USTC study shows that merely swapping the pairing of option names and definitions, with yes/no naming, flips 76.9% of decisions and drops AUC from 93.8% to 23.2%, while the type error rate stays at 0%, showing that a legal format does not mean correct semantics. The authors provide a label-free swap experiment: compare real-name rotation, neutral-name rotation, and no rotation across three runs to determine whether the model is actually reading the definitions. Verbal confidence from models is generally overconfident, so you should use label token probabilities and calibrate them; thresholds should be determined by error costs rather than the model, and low-score human review also continuously produces high-quality labels.