Chain of Thought
Chain of Thought is a podcast focused on AI engineering practice, discussing topics such as open-source models, agent systems, and production deployment.
Core Facts
Chain of Thought能显著提升模型在数学推理、逻辑判断等需要多步推导的任务上的表现
The core method of Self-Consistency is to independently sample the same problem multiple times, generate multiple different chains of thought, and select the answer with the highest frequency through majority voting.
Timeline (last 90 days)
CoT is a thinking tool suited to complex, decomposable problems that need to be examined, not a universal method for all scenarios; for scenarios requiring rapid response, or domain judgments highly dependent on experience, an explicit reasoning chain is not necessarily better than trained intuition.
For high-risk scenarios, judging model reliability from CoT text alone is insufficient; it must also be combined with consistency across multiple samples, external verification, and other means.
Language models generally suffer from overconfidence, especially in CoT, where erroneous steps are often written in as fluent and as confident a tone as correct steps, making them hard for readers to distinguish.
One explanation for CoT's effectiveness is that it spreads computation that must be done all at once across multiple token generation steps, effectively adding "serial computation depth" to the model.
The classic implementation of CoT is few-shot prompting: give the model a few "question—step-by-step reasoning—answer" examples, and the model imitates this format.
On episode 12 of the Chain of Thought podcast, Logan Kilpatrick said that solving a domain-specific computer vision problem typically takes nine to twelve months (in the smooth case).
LlamaIndex co-founder Jerry Liu noted on episode 61 of the Chain of Thought podcast that when the application form is still open-ended, the framework layer still has value, because frameworks provide core abstractions that make it easier to build certain things.
On episode 57 of Chain of Thought, Snorkel AI's Alex Ratner said that to get models to do something different, you don't do brain surgery on neurons or reinvent the algorithm architecture — you go get data.
In episode 54 of the Chain of Thought podcast, Dan Klein proposed a framework for judging whether it is worth paying for a reasoning model: it is only worth paying when you can clearly explain what the extra thinking is buying.
The article recommends that safety research must concurrently develop observability methods that do not rely on the model's voluntary disclosure, as a fallback.
4 more timeline events
All Facts (18)
Chain of Thought能显著提升模型在数学推理、逻辑判断等需要多步推导的任务上的表现
90%VerifiedThe core method of Self-Consistency is to independently sample the same problem multiple times, generate multiple different chains of thought, and select the answer with the highest frequency through majority voting.
90%UnverifiedDeepSeek R1等具备推理能力的模型在做AI Agent时，其思维链本身就在进行类似Tree Search的规划
80%Unverified大模型的规划能力经历了四个进化阶段：基础输入输出、Chain of Thought（思维链）、Self-Consistency（自洽性）、Tree of Thought（思维树）
80%UnverifiedCoT is a thinking tool suited to complex, decomposable problems that need to be examined, not a universal method for all scenarios; for scenarios requiring rapid response, or domain judgments highly dependent on experience, an explicit reasoning chain is not necessarily better than trained intuition.
50%UnverifiedThe classic implementation of CoT is few-shot prompting: give the model a few "question—step-by-step reasoning—answer" examples, and the model imitates this format.
50%UnverifiedOne explanation for CoT's effectiveness is that it spreads computation that must be done all at once across multiple token generation steps, effectively adding "serial computation depth" to the model.
50%UnverifiedLanguage models generally suffer from overconfidence, especially in CoT, where erroneous steps are often written in as fluent and as confident a tone as correct steps, making them hard for readers to distinguish.
50%UnverifiedFor high-risk scenarios, judging model reliability from CoT text alone is insufficient; it must also be combined with consistency across multiple samples, external verification, and other means.
50%UnverifiedOn episode 12 of the Chain of Thought podcast, Logan Kilpatrick said that solving a domain-specific computer vision problem typically takes nine to twelve months (in the smooth case).
50%UnverifiedLlamaIndex co-founder Jerry Liu noted on episode 61 of the Chain of Thought podcast that when the application form is still open-ended, the framework layer still has value, because frameworks provide core abstractions that make it easier to build certain things.
50%UnverifiedOn episode 57 of Chain of Thought, Snorkel AI's Alex Ratner said that to get models to do something different, you don't do brain surgery on neurons or reinvent the algorithm architecture — you go get data.
50%UnverifiedIn episode 54 of the Chain of Thought podcast, Dan Klein proposed a framework for judging whether it is worth paying for a reasoning model: it is only worth paying when you can clearly explain what the extra thinking is buying.
50%UnverifiedThe article recommends that safety research must concurrently develop observability methods that do not rely on the model's voluntary disclosure, as a fallback.
50%UnverifiedThe article recommends that models should not be directly penalized based on chain-of-thought content, and that the chain of thought should be used as a diagnostic tool rather than a training signal.
50%UnverifiedAdding "let's think step by step" to a prompt can significantly improve the performance of large language models.
50%UnverifiedChain-of-thought output may simply be a rationalization output pattern learned by the model, not fully corresponding to the actual internal information processing path—a phenomenon referred to in AI interpretability research as the 'faithfulness problem'.
50%Unverified分步推理本质上是将复杂条件概率分解为多个较简单条件概率的乘积，降低每步预测难度
50%