Prompts or Probabilities? The Debate Over Evaluating Semantic Focus-Sensitivity in LLMs

The Divergence of Two Evaluation Paths

When it comes to evaluating the linguistic competence of language models, academia and industry have long followed two paths: one observes the model's token probability distribution over standard text, and the other directly prompts the model to make metalinguistic judgments, such as asking it "which sentence is more natural?" According to the rss-sourced study Prompts and probabilities in the linguistic assessment of language models: Evidence from semantic focus-sensitivity, these two paths yield intriguingly divergent results in the new domain of semantic focus-sensitivity.

Semantic focus-sensitivity refers to contrasts like "Lucy only SWAM yesterday" versus "Lucy only swam YESTERDAY," where meaning is distinguished solely by focus markers (stress, bold, italics). The study tested models of different sizes using textual focus markers such as bold and italics, and the results paint a clear picture: token probabilities were consistently above chance across all models, while prompting performance was highly correlated with model scale.

The token probability evaluation method typically means: given a sentence (or a pair of sentences), directly read the model's log probabilities for each token, then compute sentence probability or perplexity from these and compare the probability of different candidate sentences. It does not require the model to "answer a question"; it only uses the model's native output distribution as an autoregressive language model, making it simple to implement with continuous results and low variance. Prompting, by contrast, packages the task as a natural-language instruction, has the model generate judgments like "A is more natural" or "B is more natural," and then parses its output. The former measures what the model "would say," the latter measures what the model "would judge"—the cognitive levels they probe are not the same, and this is precisely the root of the divergence discussed below.

Probability Method: Stable but "Too Honest"

The most intriguing point in the study is that the token probability method has a "high floor but a limited ceiling." It was above random guessing across models of all sizes, showing that models do indeed capture the semantic differences brought by focus markers. The problem, however, is that language models themselves are not inclined to spontaneously generate "out-of-the-blue"-style focus markers in natural output—that is, models rarely use bold or italics on their own to mark focus in ordinary text, which causes probability-based evaluation to be "diluted" when encountering focus-sensitive data.

The researchers proposed a clever validation: replacing standard focus markers with randomly generated markers. Token probabilities remained above chance, but the performance drop was significantly larger than for prompting. This directly supports their explanation—the probability method's performance is constrained by the model's natural generation tendencies toward focus markers, not by the model failing to understand focus semantics.

Prompting Overtakes on Large Models

The prompting results are broadly consistent with prior research: worse than the probability method on small models, but on some large models, prompting substantially outperforms the probability method. This echoes earlier findings that prompting improves with larger models.

The implications are worth pondering. Metalinguistic prompting is no easy task for language models; it requires the model to step out of "text generation" mode and instead make judgments about text. But when the test data is stylistically atypical (for example, relying on bold or italics—markers the model does not often generate), prompting instead demonstrates greater robustness.

Metalinguistic judgment refers to the ability to talk about or evaluate language itself as an object—for example, judging "which sentence is more fluent" or "whether these two sentences mean the same thing." It is contrasted with using language (employing language for communication). For language models, metalinguistic prompting requires switching from "continuing text" to "evaluating text," a mode shift not directly covered by their pretraining objective, which is why small models often perform poorly. But as scale increases, models gradually acquire the ability to perform such tasks zero-shot, which also explains why prompting performance is highly correlated with model scale while the probability method does not depend on this mode shift.

Implications for Assessing Linguistic Competence

The authors' core argument is that this finding has important implications for evaluating the linguistic competence of language models, even if the models are not performing metalinguistic judgments through human-like introspection. In other words, the fact that prompting works does not mean the model "understands" focus in the linguistic sense, but it is indeed a more robust probing tool under certain data conditions.

This is a reminder for evaluation practice: a single method may systematically underestimate or overestimate model capabilities. The probability method is reliable on stylistically typical data but is dragged down by the model's generation preferences; prompting is more flexible on large models but is sensitive to model scale. The divergence between the two methods is itself first-hand information for understanding the boundaries of model linguistic competence.

For researchers and engineers focused on AI evaluation, the value of this work lies not in declaring which method is "better," but in revealing how the choice of evaluation method shapes our perception of model capabilities. On subtle linguistic phenomena like semantic focus, methodological self-awareness matters more than the conclusions themselves.

Background Notes

Out-of-the-blue is a term in semantics/pragmatics referring to a sentence thrown out abruptly, detached from any presupposed context. It contrasts with "contextually supported" sentences—for example, if Lucy has already been mentioned in context, then saying "Lucy only SWAM yesterday" is quite natural. Focus markers (especially contrastive focus) usually require contextual triggering, so forcibly adding bold or italics to an isolated sentence is itself somewhat "unnatural." This explains why models rarely spontaneously produce such markers in natural text: in training corpora, bold and italics are mostly used for headings, emphasis, or formatting rather than linguistic focus marking, so the model's generation distribution is systematically skewed and probability evaluation is correspondingly "diluted."

One-line Summary

A study comparing token probabilities and prompting for evaluating semantic focus-sensitivity finds systematic divergence between the two as model scale increases.

Paragraph Summary

This study compares two paths for evaluating language model linguistic competence: the token probability method and prompting, in a test scenario of semantic focus-sensitivity—sentence pairs distinguished in meaning solely by focus markers such as bold and italics. Results show that the token probability method is consistently above chance across all model sizes but is dragged down by the model's preference not to spontaneously generate out-of-the-blue focus markers; prompting, meanwhile, is highly correlated with model scale, underperforming the probability method on small models and substantially overtaking it on some large models. The researchers validated this by replacing focus markers with random markers: the probability method's performance drop was significantly larger than prompting's, supporting the "generation preference interference" hypothesis. The authors conclude that the choice of evaluation method systematically affects judgments about model linguistic competence, and methodological self-awareness matters more than any single conclusion.