Teaching Robots to Make Bubble Tea: The Hand Reconstruction Challenge Under Cup Occlusion

In the process of robots learning human manipulation skills, the quality of demonstration data determines the upper bound of imitation learning. But in real-world scenarios, hands and objects never get out of each other's way for the sake of "convenient data collection." A recent case that sparked discussion on Reddit is a perfect example: when teaching a robot to make bubble tea, the cup keeps hiding the fingers.

Why Fingers Hidden Behind a Cup Are a Real Problem

In the video, the worker holds the bubble tea cup in the most natural way — several fingers wrap around to the far side of the cup, so from any single viewpoint, some fingers are invisible. This is the correct grip to begin with, but for robotics teams collecting demonstration data, it becomes a headache: the hand reconstruction system needs to infer where the fingers actually are when it "can't see" them.

If workers were asked to change their grip just to expose every finger to the camera for data collection, the demonstrations captured would no longer be the same — what you'd record wouldn't be real operating habits, but a "performance" custom-made for the algorithm. For robot systems aiming to replicate genuine human manipulation strategies, this kind of data contamination is fatal.

Hand reconstruction is a subtask at the intersection of computer vision and robotics. Its goal is to recover the 3D pose of the hand from images or video, typically including the wrist's global position and orientation, the rotation angles of each finger joint, and in some work, the surface shape of a hand mesh. Mainstream methods fall roughly into two categories: one based on parametric hand models (such as MANO), which compress high-dimensional hand morphology into a small set of pose and shape parameters and then regress them; the other directly predicts joint keypoints from images and then fits a complete pose through inverse kinematics or optimization. Such systems are inherently ill-posed under monocular RGB input — depth information is missing and self-occlusion occurs frequently, so when fingers are blocked by objects, the model can only "guess" based on priors or temporal information. Demonstration collection often seeks not just joint angles but also fingertip-object contact positions, which demands higher precision and makes the occlusion problem even thornier.

MEgoVista's Approach: Not Single-Frame, but Whole-Sequence Optimization

The method mentioned by the team, MEgoVista, upgrades the problem from "single-frame inference" to "full-sequence optimization." Instead of looking in isolation at how much the hand is occluded in a given frame, it performs hand motion optimization at the scale of the entire recording.

Specifically, it combines two types of information: reliable visual observations — frames where fingers are clearly visible — and temporal continuity and anatomical constraints, which are prior rules such as fingers cannot teleport and joints have physiological limits. When a frame is blocked by the cup, the system can fill in the missing parts using the motion trends of surrounding frames and anatomical constraints.

The effectiveness of this strategy lies in the fact that occlusion is usually temporary. A finger wraps behind the cup and may emerge the next second. Whole-sequence optimization can use these "emergence" moments to anchor the "hidden" moments.

"Whole-sequence optimization" methodologically belongs to batch or trajectory-level estimation, in contrast to the common frame-by-frame independent inference. Frame-by-frame methods treat each frame as an independent sample — fast but prone to jitter or drift during occlusion; trajectory-level methods jointly solve all frames of a video, constructing an objective function containing data terms and regularization terms, and solve the entire motion trajectory at once using sliding windows or global optimization. Commonly used regularization terms include smoothness constraints on velocity and acceleration (to prevent fingers from "teleporting"), upper and lower bound constraints on joint angles, and anatomical priors such as left-right hand symmetry and coordinated finger movement. Such methods are more computationally expensive and usually non-real-time, but in exchange they yield temporally consistent solutions — especially suitable for post-processing already-recorded demonstration data rather than online inference during collection.

What to Look At When Judging Whether Reconstruction Is Reliable

The original post raises an open question: when a tracker "draws" fingers behind a cup, what do you check to judge whether the reconstruction is trustworthy?

This question is deeper than it appears. Because the reconstruction system can always output a result, but output ≠ correct. In robot demonstration collection, erroneous hand reconstruction directly misleads downstream imitation learning policies — the robot may learn an action a human never performed.

Several verification angles worth attention: whether the reconstruction is kinematically consistent with surrounding frames; whether finger joint angles fall within the human range of motion; whether inferences in occluded regions are corrected when fingers become visible again rather than ignored. These can all serve as signals for checking reconstruction reliability.

In robot imitation learning, demonstration data plays the role of "expert demonstration," and the policy network treats these hand trajectories as targets to imitate. If the reconstructed finger positions have systematic bias — for example, placing an occluded index finger incorrectly on the other side of the cup wall — the contact points, grasping force directions, and even the entire manipulation sequence the robot learns could be wrong, and such errors won't show up clearly in training loss because the problem lies in the data itself, not the optimization. Therefore, the industry typically also performs downstream validation: replaying reconstructed trajectories in simulation to check for unreasonable hand-object interpenetration; or cross-comparing the same manipulation segment using another viewpoint or another sensor (such as glove-based motion capture), using ground truth to measure whether reconstruction error rises significantly in occluded intervals.

What This Means for Small Teams

The value of this case isn't in which specific algorithm wins, but in exposing a structural contradiction in real demonstration collection: data naturalness and data observability often cannot be had at the same time.

For small robotics teams with limited resources, asking demonstrators to adjust their poses may seem like a low-cost compromise, but it quietly shifts the distribution of training data. The significance of methods like MEgoVista is that they demonstrate the feasibility of another path — instead of requiring humans to change their behavior, let the reconstruction algorithm adapt to occlusion. This idea applies to any robot learning task involving hand-object interaction, not just making bubble tea.

Paper link: https://arxiv.org/abs/2609.16684