LeRobot 3D: Building 3D Spatial Perception and Manipulation for the SO101 Robot

From 2D to 3D: A Critical Leap in Robot Learning

Most mainstream robot learning pipelines still rely primarily on 2D camera observations as input. While this may suffice for image classification or simple visual tasks, the key information required for real robotic arm manipulation is inherently three-dimensional: the position of objects relative to the robot, which regions are reachable, which are occluded, and where collision risks exist.

The fundamental limitation of 2D camera observations in robot manipulation stems from the loss of depth information. RGB images are projections of a 3D world — a process that irreversibly discards object depth, volume, and spatial relationships. This loss is tolerable for simple classification or detection tasks, but a robotic arm needs to know the exact distance between a target object and the end-effector, as well as the object's 3D bounding box, in order to plan safe and effective motion trajectories. Imitation learning methods like ACT (Action Chunking with Transformers) and Diffusion Policy have achieved impressive results with 2D observations, yet researchers have found that policies relying solely on RGB input suffer significantly degraded generalization once the scene changes slightly — such as a shift in lighting or object orientation. This is precisely where 3D spatial perception delivers its core value: providing a more robust environmental representation that is resilient to viewpoint changes and lighting conditions.

A developer recently shared an open-source project on Reddit called LeRobot 3D — a 3D-grounded teleoperation stack built for the SO101 robot. The project's core goal is to integrate "3D grounding" capabilities into the LeRobot ecosystem, enabling robots to truly "understand 3D space" rather than simply "see flat images."

LeRobot is an open-source robot learning framework led by Hugging Face, designed to lower the barrier to entry for robot imitation learning research. It provides a unified dataset format, pretrained models, and training pipelines, allowing researchers to focus on algorithmic innovation rather than infrastructure. SO101 is a low-cost, open-source robotic arm platform supported within the LeRobot ecosystem, with hardware costs in the hundreds of dollars — making real-robot experimentation accessible to university labs and individual developers. LeRobot 3D's role within this ecosystem is precisely to fill the gap in 3D perception infrastructure.

For researchers and developers looking to build robot systems with spatial reasoning capabilities, this addresses a recurring pain point: every new project requires building camera calibration, point cloud fusion, robot geometry tracking, and visualization infrastructure from scratch. LeRobot 3D aims to encapsulate these low-level capabilities into a reusable, standardized foundation.

Core Features Breakdown

According to the author, the codebase already supports a fairly complete 3D perception workflow.

Multi-Camera 3D Reconstruction

The project supports fusing data from one or more Intel RealSense depth cameras into a shared, real-time scene point cloud. Intel RealSense is Intel's consumer-grade depth-sensing camera line; its depth perception is primarily based on stereo vision — capturing images from two lenses at different angles, computing depth values per pixel via disparity, and generating point cloud data with XYZ coordinates. RealSense cameras are widely used in the robotics research community thanks to solid ROS driver support and relatively affordable pricing. That said, RealSense depth estimation accuracy can degrade under strong lighting or on highly reflective surfaces — which is a key motivation for the multi-camera fusion approach: complementing information across multiple viewpoints to cover the blind spots and noisy regions of any single sensor, while fundamentally alleviating the occlusion problem that 2D approaches struggle to overcome.

Robot Geometry Tracking

LeRobot 3D uses Forward Kinematics to synchronously track the SO101's URDF geometry model against the live scene. Forward kinematics is a fundamental robotics computation: given the angle of each joint, it precisely solves for the position and orientation of the end-effector and each link in the world coordinate frame. URDF (Unified Robot Description Format) is the standard XML format in the ROS ecosystem for describing a robot's physical structure, including link geometry, mass, and joint ranges and connections. By combining FK and URDF, the system can precisely compute the 3D spatial position of every link on the robotic arm at any moment — only by knowing both the robot's own geometry and the environment's point cloud can the system reliably determine whether a given motion trajectory is safe, providing a solid foundation for collision detection and reachability analysis.

Configuration-Driven Teleoperation

For teleoperation, users control one or more SO101 follower arms via a leader-follower matching mechanism driven by configuration files. This config-driven design lowers the setup barrier for multi-arm coordination, and experiment configurations can be directly reused and extended.

Real-Time 3D Visualization in the Browser

The project leverages Viser to deliver real-time 3D visualization in the browser, intuitively displaying the fused scene, the robot's overall geometry, and individual link states. Viser is a lightweight Python visualization library designed for 3D scenes; its key feature is streaming 3D scenes to the browser via WebSocket, eliminating the need for local OpenGL or rendering dependencies. Compared to the traditional RViz (ROS visualization tool), Viser's browser-native nature offers clear advantages for remote debugging, cloud deployment, and collaborative workflows — any device with a modern browser can access the live 3D scene. In embodied AI research, good visualization tools can significantly shorten debugging cycles: researchers can intuitively assess point cloud fusion quality and calibration results rather than printing numerical logs frame by frame.

Precise Alignment Between Camera and Robot

In any 3D perception system, calibration is the critical factor that determines the accuracy ceiling. If the camera coordinate frame and the robot coordinate frame are misaligned, even the finest point cloud cannot be translated into effective manipulation decisions.

LeRobot 3D provides a complete "camera-to-robot" calibration pipeline: users first perform a coarse manual alignment, and the system then uses a multi-scale ICP (Iterative Closest Point) algorithm to register the point cloud against the robot's URDF mesh model, finely optimizing the extrinsic parameters.

ICP is a classic algorithm in point cloud registration: it iteratively finds corresponding point pairs between two point clouds, computes the optimal rigid-body transformation (rotation + translation), until the alignment error converges below a threshold. Multi-scale ICP progressively optimizes through coarse-to-fine resolution levels, improving tolerance to initial alignment errors while accelerating convergence. Traditional robot calibration typically relies on checkerboard targets or ArUco markers, requiring additional physical tools and tedious data collection procedures.

A notable design choice here is worth highlighting: using the robot's own 3D model as the calibration reference rather than a traditional calibration target. The robot's geometry is precisely known (guaranteed by manufacturing tolerances), and using it for ICP registration eliminates the need for extra tools while keeping the workflow closer to the actual operating environment — a pragmatic engineering innovation.

Why This Matters

With the rise of open-source frameworks like LeRobot, the barrier to low-cost robot learning is dropping rapidly. But most out-of-the-box solutions remain at the 2D observation level, which to some extent limits a model's ability to understand complex manipulation tasks.

LeRobot 3D's value lies not in starting from scratch, but in filling the missing 3D infrastructure gap in the LeRobot ecosystem. Spatial reasoning is a necessary step toward more capable robot manipulation — whether for grasping, obstacle avoidance, or precision assembly, all depend on accurate modeling of the 3D environment. Standardizing and modularizing these capabilities can significantly accelerate downstream research iteration, and lays the groundwork for policy networks trained with the LeRobot framework to directly consume 3D spatial representations as input in the future.

Open-Source Status and Future Outlook

The project is currently hosted on GitHub (SergioMOrozco/lerobot_3d) and is under active development, with the author explicitly welcoming community contributions.

For developers exploring embodied AI or robotic arm manipulation research, LeRobot 3D offers a practical starting point. Embodied AI emphasizes that agents must develop and demonstrate intelligence through physical interaction with the real world — perception, action, and cognition form an inseparable closed loop in the physical world, and 3D spatial understanding is at the core of that loop. As an early-stage project, LeRobot 3D is currently built primarily around the SO101 hardware; its generalizability to other robot platforms and real-time performance at scale remain to be further validated.

But the direction is undeniably right: enabling robots to truly understand the 3D world is a critical step from demo-level showcases to practical, deployable manipulation. Open-source efforts focused on foundational infrastructure often advance the entire community more meaningfully in the long run than flashy one-off demonstrations.

Key Takeaways