Dan Klein
Co-founder and CTO of Scaled Cognition, and a guest on the Chain of Thought podcast. He focuses on research and applications of reasoning models and test-time compute.
Timeline (last 90 days)
Scaled Cognition co-founder and CTO Dan Klein believes large language models are "plausibility engines" that assemble seemingly plausible output token by token.
Dan Klein argues that a system with about 80% or 90% accuracy is still unreleasable if one in ten customers is told a flight is booked when it is not; a system that is correct every time in 70% of scenarios is more valuable.
Dan Klein argues that layering a second model on top of a system to check the first is a weak fix, because both can err and often fail on the same hard cases; he estimates that 80% checking 80% ends up at about 82%.
In episode 54 of the Chain of Thought podcast, Dan Klein proposed a framework for judging whether it is worth paying for a reasoning model: it is only worth paying when you can clearly explain what the extra thinking is buying.
Dan Klein believes that a system that is correct in most scenarios but whose error rate is enough to break customer operations such as booking a flight is inferior to a system that covers fewer scenarios but is correct every time, and only the latter can go live.
Once a task has been done a few times, planning should become memory: rather than solving it from scratch on every request, cache the known flow or directly call a standard model.
All Facts (6)
Scaled Cognition co-founder and CTO Dan Klein believes large language models are "plausibility engines" that assemble seemingly plausible output token by token.
50%UnverifiedDan Klein argues that a system with about 80% or 90% accuracy is still unreleasable if one in ten customers is told a flight is booked when it is not; a system that is correct every time in 70% of scenarios is more valuable.
50%UnverifiedDan Klein argues that layering a second model on top of a system to check the first is a weak fix, because both can err and often fail on the same hard cases; he estimates that 80% checking 80% ends up at about 82%.
50%UnverifiedIn episode 54 of the Chain of Thought podcast, Dan Klein proposed a framework for judging whether it is worth paying for a reasoning model: it is only worth paying when you can clearly explain what the extra thinking is buying.
50%UnverifiedDan Klein believes that a system that is correct in most scenarios but whose error rate is enough to break customer operations such as booking a flight is inferior to a system that covers fewer scenarios but is correct every time, and only the latter can go live.
50%UnverifiedOnce a task has been done a few times, planning should become memory: rather than solving it from scratch on every request, cache the known flow or directly call a standard model.
50%