Unverified50% confidenceCautionExact time
Under a paired design, any routing jitter, temperature sampling or re-running of unstable endpoints degrades paired data into unpaired data, invalidating the premises of the McNemar test.
1
Sources
50%
Confidence
Long-term
Relevance
10/19/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedCalling real models on every test suite run is an anti-pattern: it is slow, unstable, consumes request quotas, and makes it difficult to distinguish between code issues and changes in model responses when tests fail.67% similarUnverifiedThe sampling mentioned in decoding randomness includes temperature scaling, top-k and top-p, aimed at avoiding stiff, repetitive output, at the cost that low-probability tokens are occasionally selected and errors can be plausibly continued.67% similarUnverified"Temperature 0" is not equivalent to "a unique result": batch scheduling, floating-point nondeterminism, mixture-of-experts routing, quantization error, and provider backend redeployments can all cause the same input to produce different outputs.67% similarUnverifiedWhen all sampled outputs in a group are incorrect, the group-relative advantage collapses to zero, causing the model to receive no effective gradient feedback in high-uncertainty scenarios.66% similarUnverifiedEven with temperature set to zero, verdicts still flip between identical replicates; on difficult tasks, swapping the positions of two candidate answers can reverse most verdicts.66% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1066682API
curl https://kongchang.com/api/v1/knowledge/claims/1066682MCP
get_claim(id=1066682)