Unverified50% confidenceFactExact time
The study "Prompts and probabilities in the linguistic assessment of language models: Evidence from semantic focus-sensitivity" found that on the semantic focus-sensitivity task, the token probability method performed consistently above chance on all tested models.
1
Sources
50%
Confidence
Long-term
Relevance
10/30/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedThere are two main approaches to evaluating the linguistic ability of language models: methods based on token probability distributions and methods based on prompting for metalinguistic judgments.79% similarVerifiedWhen generating text, language models default to selecting word sequences with higher conditional probability.76% similarVerifiedPerplexity measures a language model's ability to predict test text; the lower the value, the less surprised the model is by the text.75% similarUnverifiedA/B test outputs for large language models are highly random and subjective, and the same prompt may produce completely different results across two separate conversations.73% similarUnverifiedVocalGrad serves as a reminder: high scores of multimodal models on semantic tasks may mask gaps in perceptual ability, and when models rely on textual priors or linguistic reasoning to guess answers, benchmarks will overestimate their true ability.73% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1115548API
curl https://kongchang.com/api/v1/knowledge/claims/1115548MCP
get_claim(id=1115548)