Stanford AI Course: Three Feedback Mechanisms That Enable Agents to Self-Evolve

Large language models excel at natural language processing tasks, and most of us are familiar with using them in chatbot scenarios. But to make these models truly useful for real-world tasks, they need the ability to interact with real environments, tools, and code—and learn from those interactions.

The Capability Boundaries of LLMs and the Agent Paradigm

Large Language Models (LLMs) acquire rich linguistic knowledge and reasoning abilities through pre-training on massive text corpora. However, pure text-based training has inherent limitations: a model's knowledge is frozen at the cutoff date of its training data and cannot perceive real-time information; it lacks experience interacting with the physical world, making it prone to hallucinations disconnected from reality. The Agent paradigm emerged to address this—it treats the LLM as the "brain" and equips it with the ability to perceive environments, use tools, and execute actions, thereby transcending the boundaries of pure language models. Through a closed loop of perception-decision-action, agents can accomplish complex tasks such as booking flights, writing and debugging code, and controlling robots. This represents a critical leap from "language understanding" to "embodied intelligence."

Lecture 4 of the Stanford AI Agents course focuses on three representative papers, systematically exploring how agents achieve self-evolution through different sources of feedback: ReAct (combining reasoning with action), RLEF (code generation based on execution feedback), and Constitutional AI (a feedback loop of model self-critique). The core distinction among these three techniques lies in where the feedback comes from: from interaction with the environment, from some form of critique mechanism, or from known correct answers serving as ground truth.

ReAct: Teaching Models to Think and Act Simultaneously

Humans think before they act. For example, when deciding whether to take a course, you first consider how it might help you. Once you have a reason, you act, then generate new reasoning based on your observations—whether you like or dislike the course—and decide on the next step. This "think-act" cycle is fundamental to how humans operate, and it can also effectively refine language model outputs.

Before ReAct, language models struggled to combine reasoning and action as isolated processes. The core challenge was the hallucination problem: models lack awareness of what is actually happening in the environment. If you ask Gemini, ChatGPT, or Claude "What's the temperature today?" they have no concept of current events unless they actually execute a search call to retrieve the information—they lack grounding in the real world.

The Reasoning-Action Loop in Language Space

ReAct proposes an elegantly simple abstraction: using prompts to make the model generate verbal reasoning traces. First, the model is prompted to "think step by step," then the reasoning trace is appended to the prompt, followed by asking what action should be taken based on this reasoning, and then executing the action. The key insight is that these thoughts occur in language space—they don't affect the environment themselves but guide the model to generate effective actions.

Chain-of-Thought Prompting

Chain-of-Thought (CoT) is a breakthrough prompting technique proposed by Google in 2022. The core idea is to include "Let's think step by step" or provide reasoning examples in the prompt, guiding the model to output intermediate reasoning steps rather than jumping directly to the answer. This approach significantly improves model performance on complex tasks like mathematical reasoning and commonsense QA. For example, for a problem like "Roger has 5 tennis balls. He buys 2 more cans of tennis balls, with 3 balls per can. How many tennis balls does he have now?" CoT prompts the model to output "Started with 5 → bought 2 cans → 3 per can → 2×3=6 → total 5+6=11" rather than blurting out the answer directly. Self-consistency is an enhanced version of CoT: it generates multiple reasoning paths for the same question and selects the final answer through majority voting, further improving accuracy.

This alternating approach (thought 1 → action 1 → thought 2 → action 2) closely mirrors the human process: you're thirsty, you walk to the kitchen, you find there's no water, so you decide to go to the supermarket. Each step informs the next. Take a question from HotpotQA as an example—"Besides the Apple Remote, what other device can control the program it was originally designed to interact with?"—neither standard prompting nor chain-of-thought alone can produce the correct answer. But ReAct, by searching "Apple Remote," discovering it controls the "Front Row" media center, then searching "Front Row software," ultimately arrives at the accurate response.

HotpotQA and Fever Benchmark Datasets

HotpotQA is a multi-hop question answering dataset released by Stanford in 2018, where questions require integrating information from multiple Wikipedia documents to answer, testing the model's ability to construct reasoning chains. For instance, a typical question might require first identifying entity A, then looking up entity B related to A, and finally answering a question about B. Fever (Fact Extraction and VERification) is a fact-checking dataset containing 185,000 claims, each requiring a judgment of "supported/refuted/insufficient information" along with Wikipedia evidence. The common characteristic of these two datasets is that they cannot be solved by relying solely on the model's parametric knowledge—external knowledge retrieval and multi-step reasoning are required. This is why they have become standard testing platforms for evaluating agent methods like ReAct. They represent the evolution from closed-domain QA to open-domain knowledge-intensive tasks.

ReAct's Effectiveness and Limitations

Experimental results show that ReAct outperforms chain-of-thought on the fact-checking task Fever, but doesn't always win on HotpotQA. The truly powerful combination is a fallback mechanism between ReAct and chain-of-thought with self-consistency—falling back to chain-of-thought when ReAct fails after several steps, or vice versa. This demonstrates clear value in appropriately combining the model's internal knowledge with external knowledge.

On the decision-making task WebShop (where the agent purchases items based on user instructions), ReAct scored 66.6, significantly outperforming imitation learning and imitation learning + RL baselines, but still far below the human expert score of 82.1. Errors in multi-step processes cascade and amplify over time, which is the main reason for the lower success rate. Additionally, ReAct requires extensive demonstrations when the action space is too large, and multi-step reasoning incurs higher inference costs.

You may not have noticed, but today's open-source models (like Qwen's thinking mode) have already "internalized" this capability—they've been distillation-trained on such trajectories, and tool calling and reasoning have become out-of-the-box capabilities.

Model Distillation and Capability Internalization

Knowledge Distillation is a technique for transferring the capabilities of a large "teacher model" to a smaller "student model." In the agent domain, distillation has special significance: encoding complex reasoning-action patterns directly into model weights so that agent behavior emerges without explicit prompting. For example, open-source models like Qwen2.5, trained on large volumes of ReAct-style trajectories, learn the pattern of "generate reasoning → call tool → parse result → continue reasoning," with tool calling becoming a natural part of the output. This internalization brings three major advantages: reduced token consumption during inference, less dependence on prompt engineering, and improved coherence in multi-turn interactions. However, the trade-off is reduced flexibility—internalized patterns are difficult to quickly adapt to new tools or task formats. The current research trend is a hybrid approach: core capabilities are distilled into the model, while specific tasks are adapted through prompting or fine-tuning.

RLEF: Building Stronger Coding Agents with Execution Feedback

Most students in the class are using Claude Code—a huge number of engineering and coding tasks are being delegated to these agents. To make coding agents powerful enough, the key is understanding user intent and obtaining feedback from generated code for iteration.

RLEF is one of the earliest papers to demonstrate that execution feedback can significantly improve the performance of coding LLMs. It provides an end-to-end reinforcement learning fine-tuning framework: actions are generated code, observations come from execution feedback—running tests and collecting pass/fail results, assigning binary rewards based on these results, then training with PPO.

PPO in Language Model Applications

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm proposed by OpenAI in 2017 and has become the de facto standard for RLHF (Reinforcement Learning from Human Feedback). In language model fine-tuning, PPO solves the instability problems of traditional policy gradient methods: by limiting the magnitude of policy changes per update (using KL divergence constraints), it prevents performance collapse from a single large update. The specific process is: old policy generates response → reward model scores it → compute advantage → update policy with clipped objective function → repeat. PPO's "proximal" property ensures stable convergence during training, enabling models like ChatGPT to learn steadily from human preferences. In RLEF, PPO's reward signal comes from code execution results (pass/fail tests) rather than human annotations, achieving an automated feedback loop.

Dual-Layer Testing Strategy: Public Tests and Private Tests

The core of RLEF is an iterative feedback loop that uses execution feedback at both training time and inference time. After receiving a natural language problem description, the model generates a code solution and evaluates it against a public test set. If it fails, execution feedback is fed back to the model for the next attempt, looping until it passes or reaches the maximum number of rounds. For solutions that pass, a private test set determines the reward, which is used in the PPO training loop.

Take detecting palindrome substrings as an example: the first-round basic solution fails public tests due to execution timeout; after receiving feedback, the model generates an optimized, better solution that passes public tests; it's then submitted to the private test set to obtain the reward signal. This separation of public and private testing cleverly prevents the model from simply memorizing test outputs, since it receives execution feedback rather than the answers themselves.

Why RLEF Works

Experiments show (albeit based on the older LLaMA 3.1 model) that after RLEF training on competitive programming data, the solve rate improves notably, with the x-axis on a logarithmic scale. Base models typically cannot benefit from failed solutions and execution feedback. What actually makes the difference is that showing execution feedback at each round enables the model to learn to locate and fix errors—and this capability generalizes to other benchmarks.

Competitive Programming and Algorithmic Problem Solving

Competitive programming (such as Codeforces, LeetCode, and the APPS dataset) is an important scenario for evaluating coding agent capabilities. These problems typically include: an algorithmic task described in natural language, input/output examples, hidden test sets, and strict time/space constraints. Difficulty ranges from simple array operations to complex dynamic programming and graph algorithms. Unlike real-world software engineering, competitive problems have clear correctness criteria (passing all test cases), making them ideal training scenarios for reinforcement learning—binary reward signals are clear and require no human annotation. Historically, AlphaCode was a DeepMind milestone in 2022, reaching the median level of human contestants on Codeforces. Current models like Claude and GPT-4 perform well on medium-difficulty problems but still lag on difficult problems requiring deep algorithmic insight—precisely the capability gap that methods like RLEF aim to bridge.

Ablation analysis shows that with RLEF, as rounds progress (round 1, round 2, round 3), erroneous outputs decrease and the model begins making targeted fixes; without the iterative loop, the model cannot make correct edits. This leverages both higher sample diversity and more precise modifications.

Classroom discussions also highlighted limitations: binary rewards may only work for simpler problems. For more complex ones, error traces or other metadata might be needed for more efficient debugging. Whether process reward models outperform outcome reward models remains an open question across different domains and benchmarks.

Constitutional AI: Teaching Models to Learn from AI Feedback

The traditional approach to building chatbot feedback loops involves collecting human preferences—showing two outputs and having humans judge which is more correct, useful, or specific, then training a reward model accordingly (i.e., RLHF). But this approach scales poorly: collecting tens of thousands of human annotations is extremely time-consuming and tedious.

The Human Annotation Bottleneck in RLHF

RLHF (Reinforcement Learning from Human Feedback) is the core technique for training dialogue models like ChatGPT. The pipeline includes: supervised fine-tuning → collecting human preference data → training a reward model → optimizing the policy with PPO. However, human annotation faces severe bottlenecks: high cost (each annotation requires multiple annotators to reach consensus), poor scalability (OpenAI collected hundreds of thousands of preference data points for GPT-4), high subjectivity (different annotators interpret "helpfulness" differently), and inability to cover long-tail scenarios. More critically, as model capabilities improve, more specialized domain experts are needed for annotation, further driving up costs. The breakthrough of Constitutional AI lies in replacing most human feedback with AI feedback—humans only need to write high-level principles (the constitution), after which the model autonomously generates training data. This "RLAIF" (Reinforcement Learning from AI Feedback) paradigm elevates human participation from "per-item annotation" to "principle design," achieving a shift from labor-intensive to knowledge-intensive.

Constitutional AI, proposed by Anthropic, is based on a key insight: since models can reason, human-written principles (a constitution) can be used to guide the model's self-improvement. Humans only need to write this constitution; they don't need to continuously participate in the loop afterward. This works because models' instruction-following capabilities are already strong enough—if you ask them to format responses in a certain way or judge whether a response exhibits a particular behavior, they can do it.

The Self-Critique and Revision Loop

Constitutional AI employs 16 principles forming a "constitution." The workflow is: use red team prompts, observe the model's output, have the model critique its own output based on the constitution (e.g., "Is this response harmful or unethical?" "Does it contain gender bias?" "Is it inappropriate for children?"), obtain revisions, then fine-tune the model on these revised trajectories. This is the supervised fine-tuning stage.

Red Teaming and Adversarial Prompts

Red teaming originates from cybersecurity and in AI safety refers to systematically testing model vulnerabilities with adversarial inputs. For language models, red teams design prompts that induce harmful outputs, such as: jailbreak prompts (bypassing safety mechanisms), indirect injection (embedding malicious instructions through seemingly harmless context), and role-play attacks (making the model pretend it has no restrictions). Anthropic uses red team datasets in Constitutional AI containing prompts involving violence, discrimination, illegal activities, and other topics, specifically designed to expose the model's harmful tendencies. This data isn't meant to train the model to produce harmful content, but to teach it to recognize and refuse such requests. The red team-blue team cycle (red team attacks, blue team defends, red team attacks again) has become standard practice in AI safety, with companies like OpenAI and Anthropic maintaining dedicated red teams.

Next comes the reinforcement learning stage: a preference model is trained based on first-stage responses and the constitution, then used to fine-tune the LLM so it produces the most thoughtful, respectful, and appropriate responses. This essentially replaces the extensive human feedback in RLHF with a preference model trained on the constitution.

The Trade-off Between Helpfulness and Harmlessness

Results show that simply having the model evaluate its own responses based on these principles can dramatically improve harmlessness scores while maintaining roughly comparable helpfulness—this is precisely the strength of the Claude model family. The key finding is that Constitutional AI combined with chain-of-thought achieves the optimal frontier of the helpfulness-harmlessness trade-off, outperforming pure RLHF.

One valuable point raised in discussion: having a model critique itself can sometimes be harder because the model may be overconfident in its own cognition. Using consensus from multiple models for critique often works better. Additionally, updating the constitution involves continual learning challenges—how to make a model forget certain knowledge or follow new rules remains an open research problem.

Conclusion: Feedback Loops Are the Key to Agent Self-Evolution

Together, these three papers outline the core approach to agent self-evolution—building effective feedback loops. ReAct combines reasoning and action in language space, achieving grounding through real-world interaction and producing interpretable decision trajectories; RLEF uses unit tests as execution feedback in the coding domain, enabling models to iteratively fix code; Constitutional AI uses human-written principles to drive model self-critique and improvement.

Their fundamental difference lies in the source of feedback, but they share a common truth: when a feedback loop carries sufficient signal, models can improve beyond their training data. As the lecturer noted, we are borrowing from human problem-solving approaches to design LLMs—decomposing tasks, exploring in parallel, leveraging memory and experience—but ultimately we must answer a deeper question: can search in reasoning space be automated? For tasks with well-defined search spaces (like games), the answer is yes; but for open-ended tasks requiring real-world experience, we still need to collect and utilize observations in language space. This is precisely the most exciting frontier of current agent research.