Unverified50% confidenceOpinionExact time
The conclusions of many LLM papers are in fact 'a certain model at a certain point in time exhibited a certain behavior', yet they are often stated as general claims about 'LLMs'; after a model generation change, these claims are neither falsified nor confirmed—they simply quietly lapse.
1
Sources
50%
Confidence
Long-term
Relevance
11/3/2026
First Seen
Sources
Old Conclusions Meet New Models: Insights from LLM Replication Research
hackernewshackernews11/3/2026
Related Entities
Related Claims
VerifiedThe output of LLMs is inherently non-deterministic; even with the same Prompt as input, the model may return results with different formats, content, or even logic at different times.72% similarUnverifiedIn the revision stage, LLMs can point out logical jumps, redundant sentences, or check whether arguments lack supporting evidence.71% similarUnverifiedIf a single LLM is made to simultaneously take on the three roles of observation, reasoning, and verification, the model tends to treat early inferences as implicit premises for subsequent inferences71% similarUnverifiedReproducing LLM research faces a special difficulty: the object of study (the model) itself is drifting, and under the same prompt and evaluation pipeline, the output distribution of a new version of the model may already be markedly different.71% similarUnverifiedIf LLM judges that are blind to omissions are widely used for scoring, benchmarking, or guiding model training (e.g., RLHF, RLAIF), the entire optimization loop may be misled, causing models to learn to generate content that appears complete but is actually incomplete.71% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1130384API
curl https://kongchang.com/api/v1/knowledge/claims/1130384MCP
get_claim(id=1130384)