Unverified50% confidenceOpinionExact time
Hamel Husain recommends starting with about 30 examples, continuously expanding until no new failure modes appear, then accumulating about 100 labeled examples per failure mode to validate the evaluator.
1
Sources
50%
Confidence
Long-term
Relevance
10/15/2026
First Seen
Sources
Related Entities
Related Claims
UnverifiedIn the benchmark's 60 runs, the early termination conditions were triggered 27 times, including 21 detections of lack of progress and 6 cases of model provider unavailability.74% similarUnverifiedIn a test that used Traework to fully simulate a product development workflow, the tool retrieved more than 100 sources during the research phase, but after manual verification only about 40 were truly usable, while the other 60 needed to be discarded or re-investigated.69% similarUnverifiedThe current dataset contains 770 recorded attempts across 29 model/configuration entries, but the unified test suite includes only 16 distinct tasks.67% similarUnverifiedThis benchmark uses 10 original short documents of 200 to 250 words each, with 18 planted anomalies, 37 scoring entries in total (18 dirty and 19 clean), testing 13 models with 74 prompts per model.67% similarUnverifiedWhen testing LLM citation behavior, the same prompt should be run at least 20–50 times, ideally expanded to more than 100 times.67% similar
Cite This Claim
Stable URI
https://kongchang.com/claim/1036264API
curl https://kongchang.com/api/v1/knowledge/claims/1036264MCP
get_claim(id=1036264)