Why GRPO Can Train Reasoning LLMs But Can't Teach a Robot to Walk

GRPO dramatically reduced the training cost of DeepSeek's reasoning models, which raises a natural question: if it's cheap and effective, why isn't anyone using it for robot control? A technical discussion on Reddit offers two reasons, and these two reasons belong to entirely different categories of problems.

From REINFORCE to GRPO: The Removed Critic

This discussion follows up on an earlier survey of an algorithmic lineage: REINFORCE → PPO → GRPO. That survey summarized GRPO as "PPO minus the critic, plus a within-group mean baseline."

This summary directly leads to the question at the start of the article. If GRPO is cheaper and performs well enough on DeepSeek^Ad's reasoning models, why hasn't the same displacement effect occurred in control tasks? The author argues there are two answers, and they aren't the same kind of trouble: first, bias exists, and second, every step receives the same feedback signal.

The latter is essentially the same credit assignment problem from the REINFORCE era—once the critic is removed, it comes back.

What exactly is a critic (value function): In actor-critic algorithms like PPO, the critic is a value network trained in parallel with the policy network. It estimates the expected cumulative return V(s) obtainable from a given state while following the current policy. Its core role is to provide a per-state, per-timestep baseline: when the actual return exceeds V(s), the action was better than expected and should be reinforced; when it falls below V(s), it should be suppressed. This way, each timestep gets its own distinct "good or bad" signal, which is the key to solving the credit assignment problem. The cost is that the critic itself must be trained, tuned, is prone to estimation errors, and introduces additional memory and compute overhead.

What is the within-group mean baseline: GRPO removes the value network and instead samples a group of G outputs for the same prompt (or the same initial state), using the average reward of this group as the baseline. Each sample's advantage equals "its own score − the group mean score." This is natural in the LLM setting, because one generation is one complete sample, and the reward is terminal. But the within-group mean is a scalar shared across the entire trajectory, and it cannot distinguish the contribution of each individual step within the trajectory—this corresponds exactly to the "every step receives the same feedback signal" mentioned in the original text.

Credit assignment: This refers to the problem of attributing the final reward to specific decisions in sequential decision-making. If the reward only appears at the end of an episode, and there are dozens or hundreds of intermediate actions, how do you determine which step caused success or failure? REINFORCE multiplies a global return scalar by the log-probability gradient of each step, which inherently cannot localize credit; PPO mitigates this by using the critic to provide an independent advantage estimate for each step; GRPO, after removing the critic, loses this localization capability again in long-sequence control tasks.

LLMs Can Tolerate It, Robots Cannot

The credit assignment problem isn't fatal for LLMs because the LLM reinforcement learning setting has several special structural conveniences: there is only a terminal reward, the environment is relatively quiet, starting states can be replayed, and the model itself carries pretraining priors. Stacked together, these conditions make "every step receiving the same feedback" acceptable.

The robot side has none of these conveniences. The state changes every frame, physical feedback is continuous and non-replayable, and there is no pretraining prior to tell it "how the legs should move." The same feedback signal spread evenly across every timestep is equivalent to not telling it which step went wrong.

Controlled Experiments on Five Classic Control Tasks

The discussion cites the work of De Oliveira et al. (arXiv:2511.03527), which compared critic-free methods against PPO on five classic control tasks.

The result: except for CartPole, PPO wins on all tasks. Moreover, GRPO's group weighting mechanism did not yield better performance than a naive batch mean—meaning that the improvement of GRPO over REINFORCE did not materialize on control tasks.

CartPole is the only exception, but its task complexity is extremely low and insufficient to overturn the overall trend.

The five classic control tasks typically refer to discrete or continuous control environments in reinforcement learning benchmark libraries: CartPole (balancing a pole, with minimal state and short episodes), Acrobot (a two-link underactuated swing-up that needs to gain energy by swinging), MountainCar (a car that must oscillate back and forth in a valley to build enough energy to reach the top, with sparse rewards), and Pendulum (continuous inverted pendulum control), among others. These tasks share the common characteristics of continuously changing states, rewards that are often per-step or approximately sparse, and a demand for temporal precision in the policy that is far higher than in LLM generation—a single wrong step can cause the entire subsequent trajectory to collapse, making them highly sensitive to credit assignment precision.

Group weighting: This refers to GRPO not only using the within-group mean as a baseline, but also often normalizing the advantages of samples within the group (e.g., dividing by the within-group standard deviation) to reweight them. In theory, this makes gradients more stable and less affected by reward scale. But the paper's comparison shows that on control tasks, it differs little from a naive batch mean baseline—indicating that the benefit of this "improvement" depends on structural features specific to the LLM setting (such as reward variance distribution) and does not transfer to control tasks.

Why PPO might not have the upper hand on CartPole: CartPole has low state dimensionality, short episodes, and extremely low task difficulty—almost any reasonable algorithm can learn a decent policy, and at that point the critic's extra overhead and estimation error may actually drag down performance. This is exactly why the author emphasizes that "CartPole is the exception but cannot overturn the overall trend"—it's too simple and unrepresentative.

This Isn't "a Worse Algorithm," It's a Different Type of Problem

An easily misunderstood point: this conclusion does not show that GRPO is a bad algorithm. Its success on reasoning models is real; it's just that its success depends on a specific set of conditions—terminal rewards, replayable environments, and pretraining priors.

When these conditions don't hold, no one takes over the work that the removed critic used to do, and degradation appears. Control tasks happen to be exactly where all these conditions fail.

The author includes a disclosure at the end: the arguments and script were completed by the author, with Claude assisting with polishing and visualization.

The original article includes an interactive chart: https://sreejithb.com/grpo_dissect/single

Why pretraining priors are crucial: Before doing reinforcement learning, an LLM has already learned grammar, common sense, and basic reasoning patterns through large-scale text pretraining. The RL in the fine-tuning stage is really just preference alignment on an already quite strong policy—it doesn't need to learn from scratch "what a good answer is"; it only needs to adjust near its existing capabilities. Robot control has no such prior: a randomly initialized policy facing continuous motor signals has no idea "which direction the legs should swing" and must learn motion patterns from scratch using only the per-step reward signal. So when the reward signal is spread evenly and credit assignment is unclear, a robot can barely make progress, whereas an LLM can still fall back on priors.

The significance of replayable starting states: In the LLM setting, the same prompt can be sampled multiple times, and the environment (context) is identical and repeatable; GRPO's within-group comparison is built precisely on this. A robot, by contrast, faces continuously changing real physics; the same initial position can hardly be reproduced exactly, and with noise and irreversible physical feedback, the premise of within-group comparison itself does not hold. Together, these structural differences explain why an algorithm that achieved great success on reasoning models exhibits systematic degradation when transferred to control, rather than simply being "a worse algorithm."

One-line Summary

GRPO's missing critic brings back the credit assignment problem—LLMs can tolerate it, robots cannot, and PPO wins across the board on control tasks.

Paragraph Summary

The article asks a natural question: GRPO is cheap and effective, so why isn't anyone using it for robot control? There are two answers, and they belong to different categories. First, bias exists; second, every step receives the same feedback signal—essentially the credit assignment problem from the REINFORCE era, returning after the critic was removed. It isn't fatal for LLMs because they enjoy four structural conveniences: terminal rewards, a quiet environment, replayable starting states, and pretraining priors. Robots have none of these: states change continuously, feedback is not replayable, and there is no prior telling it how the legs should move—evenly spread feedback is equivalent to not telling it which step went wrong. De Oliveira et al. (arXiv:2511.03527) show that across five classic control tasks, PPO wins across the board except for CartPole, while GRPO's group weighting offers no advantage over a naive batch mean. The conclusion is not that GRPO is a bad algorithm, but that it depends on a specific set of conditions—and control tasks happen to be where all these conditions fail.