Pixel: A Full-Stack Robot Learning System with Software-Hardware Co-Design

Why Full-Stack Co-Design Is Key to Mobile Manipulation

General-purpose mobile manipulation requires robots to possess multiple capabilities simultaneously: capable hardware, scalable demonstration data collection, expressive whole-body control, and effective policy learning. However, these components are often developed independently, and mismatches between them can limit the ceiling of the entire system. Pixel's starting point is precisely to eliminate this fragmentation—it is a full-stack system with software-hardware co-design that jointly designs hardware, data, control, and learning within a single framework.

According to the study's description, Pixel's goal is not a single-point breakthrough but to provide an integrable, extensible foundation for studying how robot hardware, data, control, and learning can be designed together for robot learning. This positioning is quite representative in current robot learning research: more and more teams recognize that optimizing only policy learning while ignoring the alignment between hardware and data collection makes it difficult to achieve truly general mobile manipulation capabilities.

Hardware: Low-Cost Wheeled Humanoid with Human-Aligned Kinematics

Pixel's hardware platform is a low-cost wheeled humanoid robot with human-aligned kinematics and compliant grippers. These two design choices directly serve the learning objectives.

Human-aligned kinematics means the robot's joint layout approximates that of a human body, making it easier to transfer human demonstration videos or motion data to the robot and reducing the domain gap between human and robot data. Compliant grippers provide passive adaptability in contact-rich manipulation tasks, reducing reliance on precise force control. The wheeled chassis makes a trade-off in mobility—giving up the high terrain adaptability of legged locomotion in exchange for lower cost and a more stable mobile base, thereby concentrating research efforts on the overall coordination of mobile manipulation.

Human-aligned kinematics refers to configuring the robot's joint degrees of freedom and limb proportions to closely match the human body—for example, the rotation axes of the shoulder, elbow, and wrist aligned with those of a human, and arm length and torso proportions referenced to human scale. The main purpose of this design is to narrow the morphological gap between human demonstration data and robot execution. In robot learning, directly using human videos or motion capture data often faces the "morphological mismatch" problem—human motion trajectories cannot be directly mapped onto robots with different numbers of degrees of freedom or joint ranges. Human-aligned kinematics makes inverse kinematics mapping more direct, reducing distortion from retargeting, thereby allowing human data to play a greater role in policy learning.

Compliant grippers means the gripper has some passive compliance when contacting objects, typically achieved through elastic materials, underactuated mechanisms, or flexible joints. Compared with rigid grippers, compliant grippers can absorb position errors and impacts through passive deformation when grasping irregular objects or performing contact-rich operations, reducing reliance on high-precision force control and accurate pose estimation. This is particularly important for learning-based control: the policy does not need to precisely model contact forces, as the hardware itself provides a certain margin for error.

Data Collection: Multiple Teleoperation and Robot-Free Human Data Interfaces

Data is one of the bottlenecks in robot learning. Pixel provides multiple teleoperation interfaces in this regard, as well as robot-free human data-collection interfaces.

The significance of the latter lies in expanding data sources: if collecting demonstrations must rely on the robot itself, data scale will be limited by the number of hardware units and usage time. Introducing robot-free human data collection allows manipulation data to be accumulated without occupying robot resources, and then fused with robot data through a unified policy learning pipeline. Combined with the human-aligned kinematics design, such data maintains good correspondence with robot execution in morphology—this is precisely the direct benefit brought by co-design.

Teleoperation refers to a human operator controlling a robot in real time through input devices (such as joysticks, VR controllers, motion capture systems, or master-slave robotic arms) while recording joint trajectories, visual observations, and other data as demonstrations. It is one of the most commonly used data sources for robot imitation learning. Its advantage is that the data strictly corresponds to the robot itself; its disadvantages are slow collection speed, dependence on robot availability, and the need for operator training.

Robot-free human data collection does not use the robot itself but directly collects manipulation data through wearable devices, handheld tools, or video recording. Such data is low-cost and easily scalable, but the problem of alignment with robot morphology must be addressed. Through its human-aligned kinematics design, Pixel makes such data easier to map onto the robot in joint space, allowing it to be fused with teleoperation data in a unified pipeline.

Control and Learning: Unified Whole-Body Controller and Hierarchical Learning Stack

Pixel uses a unified whole-body controller that incorporates the actions of the mobile chassis and robotic arm into the same control framework, rather than handling navigation and manipulation separately. For tasks requiring simultaneous movement and manipulation, a unified controller can avoid interface friction between upper and lower modules.

On the learning side, Pixel adopts a general policy learning pipeline across human demonstrations and robot demonstrations, and builds a hierarchical learning stack on top of it, further supporting language-conditioned visuomotor control and long-horizon task execution. The hierarchical structure allows the system to decompose long-horizon tasks into manageable sub-goals, which are then executed by low-level visuomotor policies—a common and effective organizational approach in current long-horizon mobile manipulation research.

Unified whole-body controller differs from traditional hierarchical control: traditional approaches typically split the mobile chassis and robotic arm into a navigation layer and a manipulation layer, each with independent controllers and interfaces, which can easily cause conflicts or delays when simultaneous movement and manipulation are required. A unified controller incorporates all degrees of freedom of the chassis and arm into a single optimization or control problem, synchronizing movement and manipulation at the action level. This is crucial for mobile manipulation tasks, as many tasks require the robot to keep its end-effector stable or adjust its posture while moving.

A hierarchical learning stack typically includes high-level and low-level policies: the high level is responsible for generating sub-goals or abstract actions based on language instructions and visual observations, while the low level is responsible for translating sub-goals into specific joint or end-effector control commands. This structure alleviates the credit assignment problem in long-horizon tasks, allowing each layer to focus on decisions at different time scales. Language-conditioned visuomotor control means the policy receives both visual observations and natural language instructions as input and directly outputs actions, enabling the same policy to perform different tasks specified by language.

System Evaluation: Covering Hardware, Data, and Policy Learning

The research team evaluated the system across three dimensions—hardware, data collection, and policy learning—and demonstrated mobile manipulation behaviors requiring dexterity, active perception, navigation, and whole-body coordination.

These capability dimensions correspond precisely to several challenges in mobile manipulation: dexterity tests end-effector execution and contact control, active perception requires the robot to actively adjust its viewpoint to obtain information, navigation involves chassis movement, and whole-body coordination requires simultaneous movement and manipulation. Pixel's evaluation covers these aspects, indicating that its co-design is not effective only in a single component but produces results at the complete system level. The researchers also released a project homepage (pixel-robot.com/pixel) providing more results.

Insights from Co-Design

The value of Pixel is not just a robot or a set of algorithms. The question it raises is more noteworthy: when hardware, data interfaces, controllers, and learning pipelines are designed separately, which capabilities are limited by interface mismatches? Pixel's answer is to align these components within a unified framework—human-aligned kinematics align data sources, the unified whole-body controller aligns movement and manipulation, and the general learning pipeline aligns human and robot demonstrations. For researchers seeking to build general-purpose mobile manipulation systems, this full-stack perspective provides an extensible starting point.

As a note, the materials do not disclose specific hardware costs, data scales, or performance metrics, so this article does not make quantitative judgments about these figures. The system's actual generalization capability and reproducibility still await verification by the community on more tasks.

One-line Summary

Pixel integrates hardware, data, control, and learning through full-stack software-hardware co-design, breaking through the mismatch bottleneck in mobile manipulation systems.

Paragraph Summary

Pixel is a full-stack system with software-hardware co-design for general-purpose mobile manipulation, aimed at solving the system mismatch caused by independent development of hardware, data collection, control, and learning. On the hardware side, it uses a low-cost wheeled humanoid platform with human-aligned kinematics and compliant grippers, facilitating transfer of human demonstration data and contact-rich manipulation. On the data side, it provides multiple teleoperation interfaces and robot-free human data collection methods to expand demonstration data sources. On the control and learning side, it uses a unified whole-body controller to coordinate the chassis and robotic arm, and builds a hierarchical learning stack to support language-conditioned visuomotor control and long-horizon task execution. The evaluation covers core capabilities such as dexterity, active perception, navigation, and whole-body coordination, demonstrating the effectiveness of co-design at the complete system level.