The Evolution of LLM Reasoning: A Complete Guide from Chain-of-Thought to DeepSeek-R1

Introduction

The reasoning capability of Large Language Models (LLMs) has become one of the hottest research directions in AI today. From the introduction of Chain-of-Thought (CoT) prompting in 2022, to the groundbreaking debut of OpenAI's o1 model, to DeepSeek-R1 achieving reasoning breakthroughs as an open-source project, this technological evolution is profoundly reshaping our understanding of artificial intelligence.

The open-source GitHub project Awesome-LLM-Reasoning systematically organizes the development trajectory of this field and has already earned over 3,600 stars, making it an essential reference for researchers and developers looking to understand LLM reasoning technologies.

From Chain-of-Thought to Reasoning Models: The Complete Evolution

Chain-of-Thought: The Starting Point of LLM Reasoning

Chain-of-Thought (CoT) prompting is a milestone in LLM reasoning research. In 2022, Jason Wei and colleagues at Google first proposed this concept—including step-by-step reasoning examples in prompts to guide models to show their "thinking process" rather than directly outputting answers. This seemingly simple technique dramatically improved large model performance on mathematical reasoning, logical reasoning, and commonsense reasoning tasks.

CoT was built upon an important cognitive science insight: humans decompose complex problems into multiple intermediate steps rather than jumping directly from question to answer. Before CoT, researchers primarily relied on few-shot prompting to guide models through tasks, but this approach showed limited effectiveness on tasks requiring multi-step reasoning. CoT's breakthrough was that it didn't require modifying model parameters or architecture—it could activate the model's latent reasoning capabilities simply by changing the input format. Subsequent research also discovered Zero-shot CoT—simply adding instructions like "Let's think step by step" to the prompt could trigger step-by-step reasoning behavior without any examples, further demonstrating that large models already possess some degree of inherent reasoning potential.

The core insight of CoT is: making the model "show its work" inherently improves reasoning quality. This mirrors how humans benefit from "writing out steps" when solving problems—externalizing the thinking process not only facilitates verification but also effectively reduces errors from logical leaps.

From Prompt Engineering to Model Internalization: Upgrading the Reasoning Paradigm

After CoT, the research community rapidly produced a series of improvements:

• Self-Consistency: Sampling multiple reasoning paths and selecting the most consistent answer to improve reasoning robustness
• Tree-of-Thought: Extending linear chains of thought into tree-structured search, allowing models to explore multiple reasoning branches
• Graph-of-Thought: Further generalizing reasoning structures into graph structures, supporting more complex reasoning topologies
• Step-by-Step Verification: Introducing Process Reward Models (PRM) to verify each step of reasoning

Process Reward Models (PRM) represent a more granular evaluation approach compared to Outcome Reward Models (ORM). While ORM only focuses on whether the final answer is correct, PRM scores every intermediate step in the reasoning process. OpenAI's 2023 paper "Let's Verify Step by Step" systematically demonstrated PRM's superiority in mathematical reasoning tasks. Training PRM requires large amounts of step-level annotation data—annotators must judge whether each step in a reasoning chain is correct, making data acquisition costly. However, PRM's advantage lies in its ability to precisely locate where reasoning errors occur, thereby guiding models to improve specific reasoning steps rather than merely knowing whether the final result is right or wrong.

These methods enriched LLM reasoning paradigms from different dimensions, but most remained at the level of inference-time techniques. The true paradigm shift came from internalizing reasoning capabilities into the model training process.

OpenAI o1 vs. DeepSeek-R1: Two Reasoning Model Approaches Compared

OpenAI o1: The Pioneer of Closed-Source Reasoning Models

The OpenAI o1 series marked the birth of an entirely new category: "reasoning models." Unlike traditional LLMs, o1 engages in extended internal "thinking" before answering questions, generating a hidden chain of reasoning. This design enabled breakthrough performance on highly challenging tasks such as math competitions, programming, and scientific reasoning.

The key innovation of o1 is establishing test-time compute as a new dimension for improving model capabilities. Traditional Scaling Laws, proposed by Jared Kaplan and colleagues at OpenAI in 2020, identified power-law relationships between model performance and training data volume, model parameter count, and training compute. This law long guided the development direction of large models—bigger models, more data, longer training times. However, o1's test-time compute concept broke this single-dimensional thinking: even with fixed model parameters, investing more computational resources during inference (such as generating longer chains of thought, performing multiple sampling and verification passes) can significantly improve output quality. Some researchers call this the "Scaling Law for inference," and it opens a new path to improving AI capabilities without continuously increasing model size.

DeepSeek-R1: A Milestone in Open-Source Reasoning Models

DeepSeek-R1's release advanced reasoning model development from a different angle. As an open-source model, R1 not only demonstrated reasoning capabilities comparable to o1 but, more importantly, revealed the technical pathway for training reasoning models:

1. Reinforcement Learning-Driven Reasoning Training: R1 extensively uses reinforcement learning (RL) to train reasoning capabilities rather than relying solely on supervised learning
2. Emergent Reasoning Behaviors: During RL training, the model spontaneously learned reasoning strategies such as self-verification, backtracking, and reflection
3. Reasoning Capability Distillation: R1 proved that large model reasoning capabilities can be effectively distilled into smaller models

DeepSeek-R1's reinforcement learning training paradigm differs fundamentally from traditional supervised fine-tuning (SFT). In SFT, models learn to imitate human-annotated reasoning processes, meaning the model's reasoning capability ceiling is limited by annotation data quality. In RL training, models autonomously explore effective reasoning strategies through environmental interaction (such as attempting to solve problems and receiving correctness feedback). Specifically, R1 uses reinforcement learning algorithms like GRPO (Group Relative Policy Optimization), with the final answer correctness of math problems serving as the reward signal. During this process, the model spontaneously exhibited "aha moments"—suddenly learning self-correction and reflection behaviors, an emergent phenomenon extremely rare in pure supervised learning. This suggests that RL can help models discover reasoning strategies that human annotators might never have conceived.

Regarding distillation, Knowledge Distillation was originally proposed by Geoffrey Hinton and colleagues in 2015, with the core idea of transferring knowledge from a large model (teacher model) to a small model (student model). In the context of reasoning models, distillation is particularly significant: reasoning models typically need to generate very long chains of thought, resulting in extremely high inference costs. DeepSeek-R1 demonstrated that reasoning capabilities from a 671B-parameter large model can be distilled into 7B, 14B, or even 1.5B small models, with the distilled small models far outperforming non-reasoning models of equivalent size on reasoning tasks. Specific distillation methods include: using reasoning chains generated by the large model as training data for small models, or having small models learn to imitate the output distribution of the large model. This opens the door for practical deployment of reasoning models and edge computing applications.

The open-source significance of DeepSeek-R1 is profound. It enables the entire research community to explore and improve upon it, greatly accelerating the adoption and iteration of reasoning model technology.

The Awesome-LLM-Reasoning Project: A Technical Map for Researchers

A Structured Knowledge System

The project's core value lies in providing a structured technical map of LLM reasoning. For newcomers to the field, it offers a clear learning path; for experienced researchers, it serves as an efficient tool for tracking the latest developments. The project's 3,600+ stars and 200+ forks reflect the community's strong demand for such systematic resources.

Four Major Trends in Current LLM Reasoning Research

Based on the papers and technical directions collected by the project, current LLM reasoning research shows several clear trends:

• Scalability of Reasoning: How to ensure reasoning capabilities continuously improve with increased computational resources
• Reliability of Reasoning: How to reduce hallucinations and logical errors during reasoning
• Efficiency Optimization: How to reduce computational costs while maintaining reasoning quality
• Multimodal Reasoning: Extending reasoning capabilities from pure text to visual, code, and other multimodal scenarios

Future Directions for LLM Reasoning Capabilities

The development of LLM reasoning capabilities is at a critical juncture. From a technical perspective, three directions deserve focused attention:

Deep integration of reasoning and planning. Current reasoning models primarily excel at "solving given problems" but still fall short in scenarios requiring long-term planning and multi-step decision-making. Combining reasoning capabilities with Agent frameworks to give models autonomous planning and execution abilities is a direction full of potential. Agent frameworks refer to enabling LLMs to function as autonomous agents with the ability to perceive environments, formulate plans, use tools, and execute actions. Current mainstream Agent architectures include ReAct (Reasoning + Acting), AutoGPT, BabyAGI, and others. The core challenge of combining reasoning models with Agent frameworks is that reasoning models excel at deep thinking given complete information, but Agent scenarios require models to make decisions under incomplete information and dynamically adjust strategies based on environmental feedback. This involves a paradigm shift from "single-turn deep reasoning" to "multi-turn interactive reasoning." For example, an Agent with strong reasoning capabilities executing complex tasks needs to evaluate the current state, predict possible outcomes, and choose optimal actions at each step—essentially applying reasoning capabilities to sequential decision-making problems.

Cross-domain generalization of reasoning. Currently, reasoning models perform outstandingly in mathematics and programming but still have room for improvement in open-domain reasoning, causal reasoning, and other areas. How to truly generalize reasoning capabilities to all types of cognitive tasks remains a core challenge.

Interpretability of reasoning processes. As reasoning chains become increasingly long and complex, ensuring transparency and auditability of reasoning processes will become a critical threshold for reasoning models to reach practical applications.

Conclusion

From the inspirational breakthrough of Chain-of-Thought to the systematic innovations of OpenAI o1 and DeepSeek-R1, the pace of LLM reasoning evolution is remarkable. The Awesome-LLM-Reasoning project provides us with a panoramic window to observe this technological transformation.

For researchers and developers following AI development, understanding the technical trajectory of LLM reasoning capabilities not only helps grasp current research frontiers but also provides reference for future technology selection and research directions. The continued evolution of reasoning capabilities is likely the key path toward more powerful and reliable AI systems.

Key Takeaways

• Chain-of-Thought prompting pioneered LLM reasoning research, significantly improving model reasoning by externalizing thought processes
• OpenAI o1 established test-time compute as a new dimension for improving model capabilities, creating a new category of reasoning models
• DeepSeek-R1 revealed the RL-driven reasoning model training pathway through open source, accelerating the democratization of reasoning technology
• The Awesome-LLM-Reasoning project has become the field's most important systematic knowledge resource with 3,600+ stars
• LLM reasoning research is continuously evolving along four directions: scalability, reliability, efficiency, and multimodality