Unverified50% confidenceFactExact time
The probability that public evaluation sets appear in training data changes with model updates, so scores on old evaluations lose comparability; API models undergo silent updates, and the same model name may correspond to different weights at different times.
1
Sources
50%
Confidence
Long-term
Relevance
11/3/2026
First Seen
Sources
Old Conclusions Meet New Models: Insights from LLM Replication Research
hackernewshackernews11/3/2026
Related Entities
Related Claims
UnverifiedThe probability calibration of model outputs is often biased, especially on patterns that appear familiar within the training data distribution, where overly high confidence is given.78% similarUnverifiedIn AI auditing, intersectionality is one of the hardest axes to measure, because when two demographic parameters change simultaneously, the sample size for the corresponding combination in the training data usually drops sharply, causing model scores to tend to be low.71% similarUnverifiedEarly in training, the model's output is close to a uniform distribution and α is low; as training progresses, the model learns to concentrate probability mass on the correct token, and α rises.71% similarUnverifiedDifferent vendors differ in training data, alignment, and retrieval capabilities, so the error distributions of their respective models do not overlap.70% similarUnverifiedThere is no stable mapping between a model's self-evaluation score and its actual accuracy; rephrase the same prompt and the score may drift substantially.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1130389API
curl https://kongchang.com/api/v1/knowledge/claims/1130389MCP
get_claim(id=1130389)