HexaShell
A thin-shell framework for robot manipulation that forces a vision-language model (VLM) to directly write executable code to control the robot. It provides only raw RGB images and structured logs, serving to test the VLM's spatial reasoning and programming capabilities.
Timeline (last 90 days)
HexaShell is a "thin shell" framework that forces the VLM into the role of a full-stack coding agent, writing executable code directly for real robot operations, receiving only raw RGB images, and iteratively correcting after reading the logs of the code it writes.
In the HexaShell experiments, the robot failed its first grasp of a cylinder, and the VLM, based on the logs, raised the approach height by 4 cm and succeeded on the second attempt.
According to data published by the HexaShell project team, the VLM reached a 78% success rate within three attempts on the RLBench pick-and-place task.
HexaShell achieved a 62% success rate on the Meta-World door-opening task.
HexaShell achieved a 45% success rate on the custom tool-use (screwdriver insertion) task.
Among HexaShell's failure modes, spatial reasoning errors (depth misestimation) account for about 30% of total failures and are the biggest bottleneck when using RGB only.
Among HexaShell's failure modes, temporal ordering errors account for about 15%, and token-limit truncation accounts for about 10%.
Ablation experiments show that inserting a learned motion planner between the VLM and the robot raises the RLBench success rate to about 90%.
The deployment risks HexaShell exposes include unfiltered command permissions (an erroneous move_to() could drive the robot arm into an occupied area), depth-perception bottlenecks and token-limit issues.
The high-priority mitigation is to deploy a real-time safety wrapper that validates every motion primitive against joint limits, collision maps and velocity caps before execution.
1 more timeline events
All Facts (11)
The token-limit problem can be solved with chunked generation: the VLM outputs a short script, the robot runs it and streams new logs, and the VLM continues generating.
50%UnverifiedHexaShell is a "thin shell" framework that forces the VLM into the role of a full-stack coding agent, writing executable code directly for real robot operations, receiving only raw RGB images, and iteratively correcting after reading the logs of the code it writes.
50%UnverifiedAccording to data published by the HexaShell project team, the VLM reached a 78% success rate within three attempts on the RLBench pick-and-place task.
50%UnverifiedHexaShell achieved a 62% success rate on the Meta-World door-opening task.
50%UnverifiedHexaShell achieved a 45% success rate on the custom tool-use (screwdriver insertion) task.
50%UnverifiedAmong HexaShell's failure modes, spatial reasoning errors (depth misestimation) account for about 30% of total failures and are the biggest bottleneck when using RGB only.
50%UnverifiedAmong HexaShell's failure modes, temporal ordering errors account for about 15%, and token-limit truncation accounts for about 10%.
50%UnverifiedAblation experiments show that inserting a learned motion planner between the VLM and the robot raises the RLBench success rate to about 90%.
50%UnverifiedThe deployment risks HexaShell exposes include unfiltered command permissions (an erroneous move_to() could drive the robot arm into an occupied area), depth-perception bottlenecks and token-limit issues.
50%UnverifiedThe high-priority mitigation is to deploy a real-time safety wrapper that validates every motion primitive against joint limits, collision maps and velocity caps before execution.
50%UnverifiedIn the HexaShell experiments, the robot failed its first grasp of a cylinder, and the VLM, based on the logs, raised the approach height by 4 cm and succeeded on the second attempt.
50%