Training a Robot Dog to Mimic a Real Dog Using a Single Phone Video: Full Open-Source RL Pipeline

From a Single Video to a Reinforcement Learning Policy

A developer recently shared a creative robotics project on Reddit: training a quadruped robot to mimic the movements of a real dog using nothing but an ordinary phone video. The project goes from raw video input all the way to a deployable reinforcement learning (RL) policy — a complete pipeline from "raw footage" to "runnable control strategy" — and it's fully open-source.

The project is built on Unitree's GO2 quadruped robot and trained in two major physics simulation platforms: Isaac Sim and Mujoco. The developer summed up the entire process in one line: "I taught a quadruped robot to imitate a real dog using only a video I filmed on my phone."

What makes this project compelling is how dramatically it reduces the data collection cost of traditional motion control. No expensive motion capture equipment, no multi-camera arrays — a single monocular video shot on a phone is all it takes to kick off training.

Technical Breakdown: From Video to Robot Control Policy

The project's technical pipeline can be broken down into three key stages, each corresponding to a classic challenge in computer vision and robotics.

Step 1: Extracting 3D Skeleton Poses from Monocular Video

The first step is reconstructing the dog's 3D skeletal pose from a standard monocular video. This is already a significant challenge — monocular video lacks depth information, so recovering 3D motion from a 2D image requires pose estimation and 3D reconstruction techniques. Compared to multi-view or depth camera setups, monocular reconstruction has a much lower barrier to entry, but accuracy is harder to control. The fact that this pipeline works at all speaks to the robustness of its design.

3D pose estimation from monocular video typically relies on two approaches: direct regression via deep learning (where models learn a mapping from 2D keypoints to 3D coordinates using large annotated multi-view or depth datasets), or methods that combine kinematic constraints and temporal information to infer depth from optical flow or skeletal motion patterns across frames. For animals (rather than humans), the difficulty increases further — publicly available annotated animal motion datasets are far fewer than human datasets, demanding stronger model generalization. The recent emergence of vision foundation models like DINO and ViTPose has made it possible to obtain reliable keypoint detection without large amounts of animal-specific labeled data, and this progress is a key technical enabler for low-barrier pipelines like this one.

Step 2: Motion Retargeting — Mapping Joint Motion

Once the dog's motion trajectory is extracted, a core challenge remains: the skeletal structure of a real dog doesn't match that of a robot dog. A real dog has a flexible spine and complex muscle systems, while the GO2 is a rigid mechanical structure with a different number of joints and degrees of freedom.

This is where motion retargeting comes in — mapping the dog's joint movements onto the robot's corresponding joints. This step determines whether the robot can preserve the essence of the original motion while still satisfying its own physical constraints.

Motion retargeting originated in computer animation, where it's used to transfer one character's movements to another with a different skeletal proportion or structure. In robotics, the problem is more complex: it's not just about handling differences in bone length, but also satisfying hard constraints like the robot's joint angle ranges and velocity limits, as well as the physical feasibility of ground contact. Common approaches include inverse kinematics (IK) solvers (computing joint angles from target end-effector positions) and optimization-based trajectory fitting (minimizing error between the target pose and robot pose while enforcing physical constraints). The quality of retargeting directly affects the quality of the reference trajectory used in RL training — if the reference trajectory is significantly distorted, the RL agent will either learn an unnatural gait or fail to converge due to noisy reward signals.

Step 3: Reinforcement Learning Training in Simulation

Finally, the developer trains the robot's control policy in physics simulators (Isaac Sim and Mujoco) using reinforcement learning. The training objective is to make the robot's motion as close as possible to the dog it "observed." RL plays a critical role here — it doesn't simply replay the motion, but learns to stably reproduce the target movement pattern even under real-world physical disturbances.

According to the author, the final result is that "the robot's movements genuinely resemble those of the dog it observed," validating the feasibility of this video-to-policy pathway.

Why This Open-Source Project Deserves Attention

Dramatically Lowers the Data Barrier for Motion Imitation

Traditional robot motion learning typically relies on professional motion capture systems or manually designed reference trajectories — expensive and time-consuming. A "just use a phone video" approach means anyone can source training material from everyday life: film a cat, a dog, or even another animal, and that footage could become a robot's motion curriculum. This democratization of data collection has meaningful implications for the entire field of embodied intelligence.

Sim-to-Real: The Classic Paradigm

The project follows the approach of "training RL policies in simulation," which is the dominant paradigm in current quadruped robotics research (Sim-to-Real). Training in Isaac Sim and Mujoco allows for large-scale GPU-accelerated parallel simulation while thoroughly validating policy stability before deployment. Supporting both simulation platforms also suggests the author has given thought to cross-environment generalization.

Sim-to-Real transfer is one of the core challenges in quadruped robotics research. Policies trained in simulation often degrade when deployed in the real world, due to the "Reality Gap" — modeling errors in motor dynamics, friction coefficients, sensor noise, contact models, and more. The dominant approach to addressing this is Domain Randomization — randomizing simulation parameters (such as mass, friction, and latency) during training to force the policy to be robust to environmental variation. Isaac Sim is built on the PhysX engine and supports large-scale GPU-parallelized simulation, enabling thousands of simultaneous simulation instances to accelerate training; Mujoco is known for its lightweight, accurate contact dynamics. The two platforms complement each other, and supporting both helps cross-validate a policy's generalization ability.

Fully Open-Source and Reproducible

The author repeatedly emphasized a core philosophy in their writeup: "In an age where robots dance and run through forests, we shouldn't just be spectators or subscribers — we should be owners." All code has been open-sourced, encouraging developers to build it themselves.

This attitude reflects a certain vision of technological democratization. As robotics capabilities become increasingly concentrated in a handful of large companies, open-source projects give ordinary developers the chance to truly understand, modify, and own these capabilities — rather than simply consuming polished demos from vendors.

Significance and Current Limitations

From a technical standpoint, this project links together three major modules — pose estimation, motion retargeting, and reinforcement learning — into a complete and reproducible pipeline with significant educational and research value. For developers looking to get started in quadruped robot motion control, it offers a rare end-to-end reference example.

That said, the limitations deserve honest acknowledgment. The accuracy of monocular video reconstruction, information loss during motion retargeting, and the Sim-to-Real gap are all unavoidable challenges when moving toward real-world deployment. What the author demonstrates is proof of concept; turning that into a system that can perfectly replicate arbitrary video-based motion in the real world still requires considerable engineering work.

Still, "teaching" a robot dog to mimic a real dog using just a phone video is, in itself, a remarkably imaginative endeavor. It reminds us that in the fast-moving world of embodied intelligence, creativity and open sharing can often drive progress further than expensive equipment ever could.

Original post and source code: https://postcapitalistrobots.substack.com/p/the-first-robot-trained-entirely