HexaShell Lets Robots Write Their Own Code to Fix Grasping Failures: VLM Programming Ability Put to a Real Test

In a lab in Zurich, a dual-arm robot tries to pick up a red block but misses. Instead of waiting for a human engineer to intervene, it outputs a short error log on its own, then generates a new piece of Python-style code that corrects the grasp height. On the second attempt, it succeeds. The whole process takes less than ten seconds—this is not a sci-fi scene, but the latest experiment from the HexaShell project.

HexaShell is a "thin-shell" framework that forces vision-language models (VLMs) to act as full-stack programming agents, writing executable code directly for real-world robot manipulation. Unlike previous "shell" papers that added precomputed depth maps, external planners, or curated action vocabularies, HexaShell strips away all convenience layers: the VLM receives only raw RGB images, writes robot code, reads robot logs, and then iterates on corrections. It is like a streamlined software development loop—compile, run, debug—replayed on metal limbs.

Case Study: How a Robot Debugs Its Own Grasping Task

The experimental task is to move a blue cylinder from a cluttered tray into a narrow slot. The lab's camera rig streams three calibrated RGB views to the VLM. The VLM generates a script and sends it to the robot for execution via HexaShell's API.

On the first attempt, the robot's gripper closes, but the cylinder slips—the VLM misestimated object depth from the RGB images. The robot then streams a log file containing joint angles, an image at the moment of grasping, and the error code GRASP_FAILURE.

The VLM reads the log, performs brief internal reasoning ("grasp failed → likely depth underestimated → increase approach height"), and replies with a corrected script: raise the approach height by 4 cm. HexaShell runs the new code, and the second attempt succeeds.

This case reveals three core constraints that HexaShell imposes on the VLM: 3D reasoning from RGB data alone, full code synthesis without a curated action set, and iterative error correction based on structured logs. These constraints expose the VLM's true programming ability and blind spots.

VLM (Vision-Language Model) is a class of multimodal large models that process both image and text inputs, such as GPT-4V, Claude 3 Vision, Gemini, and others. Its core capability is mapping visual information into a linguistic semantic space, thereby supporting tasks such as visual question answering, image captioning, and visual reasoning. In robotics, the unique value of VLMs lies in their ability to directly combine natural language instructions with visual scenes to generate executable action plans or code. However, VLMs are not designed for precise 3D geometric computation: when inferring depth, size, and spatial relationships from 2D images, they rely on statistical priors in training data rather than explicit geometric modeling. HexaShell exploits this characteristic as a "probe"—letting the VLM write robot code without depth sensors or pose annotations, thereby testing the boundaries of its intrinsic spatial reasoning and programming abilities.

Benchmarks: What the Numbers Mean

According to data released by the HexaShell project team, on the RLBench pick-and-place task, the VLM achieves a 78% success rate within three attempts; on the Meta-World door-opening task, the success rate is 62%; and on a custom tool-use (screwdriver insertion) task, the success rate is 45%.

The significance of these numbers lies in comparison: state-of-the-art model-free policies that receive depth images and object pose annotations typically report 80–85% success rates on the same pick-and-place task. HexaShell forces the VLM to infer geometry from raw RGB alone, narrowing the gap to single digits—demonstrating that VLMs already possess strong spatial reasoning capabilities.

But the failure modes are equally clear. Spatial reasoning errors (depth misestimation) account for about 30% of total failures and are the biggest bottleneck when using RGB alone; temporal ordering errors (such as releasing before reaching the target) account for about 15% and are simple script defects that compiler-style safety checks can catch; token-limit truncation (scripts cut off in long-horizon tasks) accounts for about 10%, dragging the tool-use benchmark success rate down to 45%.

An ablation study further reveals: adding a learned motion planner between the VLM and the robot can raise RLBench success rate to about 90%, but at that point the planner takes on most of the spatial reasoning work. Therefore, HexaShell's thin-shell design remains an appropriate probe for measuring the VLM's intrinsic programming ability.

RLBench is a widely used simulation benchmark platform in robot learning, built on CoppeliaSim, offering more than 100 manipulation tasks (such as pick-and-place, door opening, peg insertion, etc.), each with expert demonstrations and dense rewards. Its characteristics include diverse tasks, support for multi-view RGB-D observations, and standardized evaluation protocols, facilitating horizontal comparison across methods. Meta-World is another lightweight meta-reinforcement learning benchmark containing 50 tabletop robotic arm manipulation tasks, often used to test policy generalization and multi-task learning capabilities. The common limitation of both benchmarks is that they are mainly oriented toward single-arm fixed-base setups, with relatively structured observation and action spaces. HexaShell chooses them as testbeds partly for fair comparison with existing specialized policies, and partly because its conclusions require caution when transferred to real industrial scenarios—7-DOF redundant manipulators, mobile bases, and dynamic obstacles in real production lines introduce complexities not covered by the benchmarks.

Risks and Mitigations: The Distance from Lab to Production Line

What HexaShell exposes is not only model capability but also deployment risk. The most pressing is unfiltered command authority—a wrong move_to() could drive the robotic arm into a human-occupied area. The high-priority mitigation is to deploy a real-time safety wrapper that validates every primitive against joint limits, collision maps, and speed caps before execution.

The depth perception bottleneck requires a lightweight depth estimator as a preprocessor, adding a synthetic depth channel to the VLM's input tensor while maintaining the single visual tensor input format. The token-limit problem can be solved through chunked generation: the VLM outputs short scripts, the robot runs them and streams new logs, and the VLM continues generating—maintaining the iterative debugging loop while controlling prompt length.

In addition, current benchmarks are biased toward single-arm setups, whereas industrial production lines commonly use 7-DOF manipulators or mobile bases. Extending a hardware-agnostic primitive library (such as move_base, grasp_7dof) and testing on heterogeneous platforms are necessary steps toward productization.

A real-time safety wrapper is a key middleware layer in robot deployment, sitting between the high-level instruction generator (here, the VLM) and the low-level motion controller. Its core responsibility is to pre-validate every motion primitive to be executed (such as move_to(), grasp()): checking whether the target pose is within joint limits, whether the motion trajectory collides with known obstacles, and whether speed and acceleration exceed safety thresholds. In HexaShell's thin-shell architecture, VLM-generated code calls these primitives directly; without filtering, a single wrong coordinate parameter could cause the robotic arm to rush toward people or equipment at high speed. Safety wrapper implementations typically rely on kinematics solvers (such as IKFast, TRAC-IK) and collision detection libraries (such as FCL, Bullet), and run under hard real-time constraints. Notably, the wrapper can only intercept risks that are "formally verifiable"; semantic-level errors (such as grasping the wrong object) still require log feedback and iterative debugging to correct.

Industry Implications: Code-First Robot Workflows

HexaShell does not promise to instantly turn any VLM into a perfect robot programmer. Its value lies in refusing to give the model any crutches, thereby exposing the true ceiling of current multimodal models. The 78% pick-and-place success rate and 62% door-opening success rate show that even when forced to infer depth, geometry, and timing from raw images, VLMs can already match specialized policies on standard tasks.

For companies looking to rapidly prototype robot behaviors, HexaShell offers a "code-first" workflow: describe the goal in natural language, let the VLM generate a script, observe the robot execute it, and let the VLM debug itself. For safety-critical deployments, the thin-shell design requires a robust sandbox that checks every primitive before instructions reach the actuators.

From a product integration perspective, building a HexaShell-compatible SDK is a viable path—exposing a single run("<code>") call, a structured log format, and a safety wrapper. Early adopters can plug in any VLM (Claude-V, Gemini-V, or upcoming multimodal models) without rewriting the motion pipeline. R&D investment should focus on fine-tuning VLMs on robotics-relevant corpora (assembly manuals, CAD drawings, motion primitive documentation), because the thin shell amplifies any improvement.

In the long run, combining HexaShell with a modular verification layer—running generated scripts in high-fidelity simulation before real execution—can add a safety net while preserving the core idea. This minimalist, transparent, data-driven experiment may well be the starting point for robot programming's march toward autonomy.

One-line Summary

HexaShell's thin-shell framework forces VLMs to write and self-debug robot code from raw RGB images alone, achieving a measured 78% pick-and-place success rate.

Paragraph Summary

HexaShell is a minimalist robot programming framework that strips away all auxiliary layers—depth maps, external planners, and curated action libraries—forcing vision-language models to generate executable robot code from raw RGB images alone and to iterate corrections by reading structured logs. On the RLBench pick-and-place task, the VLM reaches a 78% success rate within three attempts, narrowing the gap to single digits versus specialized policies using depth and pose annotations (80–85%), proving that VLMs already possess strong spatial reasoning. But the failure modes are equally clear: depth misestimation accounts for 30%, temporal errors 15%, and token truncation 10%. At the deployment level, a real-time safety wrapper is needed to validate every primitive, chunked generation mitigates token limits, and extending a hardware-agnostic primitive library is key to productization. HexaShell's value lies in measuring the true programming ceiling of multimodal models in a transparent, data-driven way.