Unverified50% confidenceCautionExact time
The lack of reliable Evals leads to an "optimization illusion" — where metric numbers look good but actual user experience has not improved or has even regressed
1
Sources
50%
Confidence
Long-term
Relevance
10/10/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedManual evaluation is close to the user's actual experience, but has two shortcomings: instability (subjectivity) and low efficiency.77% similarUnverifiedThe paper points out that existing user simulators lack outcome calibration, meaning they cannot cause agents to fail in the right places in the right ways.72% similarUnverifiedIncreasing the number of agents does not necessarily improve reliability; homogeneous and mutually influencing agents may instead accelerate the formation of erroneous consensus.71% similarUnverifiedComplexity itself does not drive users away; what truly causes user churn is the sense of ambiguity that comes from a lack of clear goals and feedback.70% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/994886API
curl https://kongchang.com/api/v1/knowledge/claims/994886MCP
get_claim(id=994886)