DeepMind Lets Gemini 4 Control a Humanoid Robot, Hits 7 Cornhole Shots in a Row

An experimental video from Bilibili shows a striking scene: the DeepMind team attempted to have the Gemini 4 model directly control a humanoid robot to complete a cornhole tossing game, and it hit the target 7 times in a row. This is not simple scripted execution, but a complete closed loop from learning via video, to simulation training, to real-robot iteration.

From Watching Video to Real-Robot Operation: Gemini 4's Learning Path

The starting point of the experiment is very intuitive — Gemini 4 is shown a video of a human completing the tossing task. The model needs to understand the action logic in the video, and is then prompted to run the Mojoku simulator for practice. According to the video content shared by the Bilibili creator, Gemini 4 demonstrated in this process the ability to "learn from experience, iterate, make mistakes, and review failure videos."

Each trial and error becomes the basis for the next adjustment. The model does not passively execute a preset program, but actively analyzes the video record of each failed toss, locates the problem, and then optimizes the action parameters. This reasoning ability based on video understanding allows it to gradually narrow the margin of error.

The Mojoku simulator belongs to the category of common physics simulation platforms in the field of robot learning. Its core role is to reproduce physical laws such as gravity, friction, and collision in a virtual environment, allowing the model to conduct large amounts of trial and error at a cost far lower than real-robot experiments. Compared with a real robot, a simulator can run thousands of tosses in parallel, and each failure can immediately generate video and state data for the model to review. However, there is usually a "reality gap" between simulation and the real robot — that is, it is difficult for the virtual environment to fully reproduce factors such as real-world materials, latency, and sensor noise. Therefore, migrating from simulation to the real robot often requires additional fine-tuning, which is also the technical background for the detail in this experiment of "adjusting the release timing by a few centimeters."

Iteration Details: Transfer from Simulation to Real Robot

During the simulation phase, Gemini 4 conducted multiple rounds of tossing practice. According to the video description, in the third round of practice it adjusted its stance width and optimized its arm-swing trajectory so that its hand passed cleanly over its hip, forming a smooth parabola. These fine adjustments were not manually written, but completed by the model itself based on feedback.

The real test was transferring from simulation to the real robot. The video mentions that the model observed the actual landing point of each beanbag and adjusted the release timing by a few centimeters until it successfully landed 7 shots in a row. This "few centimeters" precision adjustment reflects the model's ability to adapt to subtle differences in the physical world.

Model Reasoning Ability: Knowing "What Went Wrong"

The most noteworthy part of the experiment is that Gemini 4 can reason through video understanding and determine specifically which step went wrong. The video explicitly mentions: "Gemini 4 can reason through its video understanding ability and narrow down exactly where the problem is."

This ability means the model not only knows that an action failed, but can also locate the cause of the failure — whether the release timing was wrong, the stance was off, or the arm-swing trajectory deviated. Then, based on these judgments, it makes targeted corrections in subsequent attempts.

Reasoning driven by video understanding means that the model can not only recognize "what happened" in the video, but also infer "why it failed." In the field of embodied intelligence, this usually relies on a vision-language model (VLM) to encode video frame sequences into semantic representations, and then combine them with task goals for causal inference. For example, the model needs to distinguish between two different types of errors: "releasing too early causes the beanbag to fly too high" and "stance offset causes directional deviation." This ability differs from traditional reinforcement learning — the latter usually relies only on scalar reward signals and struggles to explain the specific reasons for failure. What Gemini 4 demonstrates is using video understanding as input for reasoning, turning "watching the replay" into an active means of problem localization, thereby making corrections more targeted rather than blind search.

The Significance of Learning from Experience

At the end of the video, it emphasizes: "It learns from its own experience, which is very exciting." This sentence points to the core value of the experiment. Traditional robot control often relies on precise modeling and manual parameter tuning, while Gemini 4 demonstrates a new path: letting the model understand the world through video, accumulate experience through trial and error, and locate problems through reasoning.

Hitting 7 times in a row may be just the beginning. When a model can watch videos, understand tasks, practice in simulation, adjust on the real robot, and extract lessons from each failure, it means the threshold for general-purpose robot control is being redefined.

This experiment also leaves some unresolved questions: Is the model's adjustment process fully autonomous? How exactly is the gap between simulation and the real robot bridged? These details may require more technical disclosure to verify. But in any case, the learning loop demonstrated by Gemini 4 in physical tasks has already provided a noteworthy sample for embodied intelligence.

Embodied AI emphasizes that agents acquire knowledge and skills through interaction with the physical world, rather than only processing static data. Traditional industrial robot control relies on precise kinematic modeling and manual parameter tuning, and every task requires engineers to debug repeatedly; the path of embodied intelligence is to let models "learn while doing" like humans. The reason Gemini 4's experiment has attracted attention is precisely that it combines the semantic understanding ability of large models with the closed loop of robot execution: watching videos to establish task representations, accumulating experience in simulation, and iterating on the real robot based on landing-point feedback. If this paradigm can generalize, it means that in the future robots can quickly master new skills by watching instructional videos, greatly lowering the deployment threshold. This is also the technical basis for the judgment that "the threshold for general-purpose robot control is being redefined."

One-line Summary

Gemini 4 hit 7 cornhole tosses in a row through video watching, simulation practice, and real-robot iteration.

Paragraph Summary

An experimental video from Bilibili shows DeepMind having Gemini 4 directly control a humanoid robot to complete a cornhole tossing game, hitting the target 7 times in a row. The experiment began by showing the model a video of a human tossing, after which the model practiced autonomously in the Mojoku simulator, reviewed failure videos, analyzed problems, and iteratively adjusted stance, arm-swing trajectory, and release timing. After transferring from simulation to the real robot, the model needed only to fine-tune the release timing by a few centimeters to achieve consecutive hits. The article emphasizes that Gemini 4's core ability lies in reasoning through video understanding, allowing it to locate specific causes of failure and make targeted corrections. This closed-loop learning path provides a noteworthy sample for embodied intelligence, while also leaving unresolved questions such as how the gap between simulation and the real robot is bridged.