Claude vs. GPT Control a Robotic Arm to Paint: Which One Does It Better?

A Fascinating Robotic Arm Painting Experiment

A developer shared an intriguing experiment on Reddit: they connected two frontier AI models — Claude Fable 5.1 and GPT 6 Astra — to the same SO-101 robotic arm, had each one complete an identical painting task, and then compared the results.

The experiment was inspired by a post that had circulated on X (formerly Twitter), where someone had GPT 6 Astra control an SO-101 arm to draw the Golden Gate Bridge. The original post used image reproduction as a benchmark, but this developer changed the task: instead of recreating a reference image, the models were asked to fill predefined shape areas with a single solid color.

This change in task design is actually quite significant. Reproducing an image tests a model's grasp of overall composition, color, and proportion, whereas single-color shape filling focuses more narrowly on boundary recognition, path planning, and execution precision — the model needs to understand a shape's outline, plan the arm's movement trajectory, and fill the area as completely as possible without going out of bounds. This makes the evaluation criteria cleaner and easier to compare across models.

How Did the Two Models Perform?

According to the poster, the two models produced paintings that were remarkably close in terms of shape coverage — judging purely by how well the shapes were filled, there was no obvious gap between them.

But the devil is in the details. The author noted that there were "things worth discussing" between the two paintings — meaning there were subtle, qualitative differences in fill evenness, edge handling, and stroke consistency. These kinds of differences are precisely what reveal a model's nuanced capabilities in spatial reasoning and motor control. One model might fill more evenly but occasionally stray outside the boundary; the other might have cleaner edges but slightly lower coverage.

Interestingly, the author showed the results to a few friends and got a consensus that was "surprising." Rather than directly revealing which model came out ahead, they posed the question to the community and invited everyone to judge for themselves. This blind-evaluation approach is arguably more valuable than the author simply declaring a winner — it removes brand bias and lets the work speak for itself.

Why Experiments Like This Matter

Connecting a large language model to a physical robotic arm and having it paint may look like a novelty project, but it touches on an important direction in today's AI landscape: Embodied AI.

The Leap from Text to Physical Action

Large models are adept at handling text and images, but reliably controlling physical hardware requires translating high-level task understanding ("fill this shape") into a precise sequence of low-level action commands (joint angles, movement paths, stylus pressure, etc.). This translation is an intense test of a model's planning abilities and spatial understanding. Painting tasks happen to be an intuitive test vehicle — the quality of the result is immediately visible to the naked eye.

At the technical implementation level, using a large model to control a robotic arm typically relies on a "perception-reasoning-execution" pipeline. The model first captures the current scene through a camera (visual input), understands the spatial relationship between the shape outline and the current pen tip position, then generates an action sequence (e.g., "move right 5mm, press down 1mm"), which a driver then translates into motor control signals for each joint of the arm. Throughout this process, the model needs to continuously run a closed feedback loop — re-perceiving the scene after each stroke, checking for deviation from the planned path, and correcting in real time. This places high demands on the model's inference speed and understanding of spatial coordinate systems. Common technical approaches in the field today include: direct prompt engineering (having the model output coordinate sequences), Code-as-Action (having the model generate control scripts), and purpose-trained vision-action models (such as RT-2, π0, etc.). This experiment is closer to the first two approaches — an exploration of transferring a general-purpose large model to physical control scenarios in a "zero-shot" manner.

Standardized Hardware Creates Comparability

The experiment used the SO-101, a relatively widespread open-source robotic arm platform. Testing different models on the same hardware with the same task eliminates hardware variability, making the comparison fairer. This is also a methodology worth borrowing for community-driven model capability evaluations: control the variables, and focus on the model itself.

The SO-101 is a low-cost open-source robotic arm developed through a collaboration between Hugging Face and the hardware community, priced in the ~$100 range, designed to let researchers and developers replicate embodied AI experiments at minimal cost. It uses serial bus servo motors and integrates easily with open-source frameworks like LeRobot, making it simple to connect to Python control scripts or large model APIs. Thanks to its open-source nature and widespread adoption, the SO-101 has recently become the community's go-to "standard test bench" for validating embodied AI ideas — similar to how MNIST functions in computer vision: simple enough to be reproducible, yet capable of exposing real differences in model capabilities. This also means the methodology of this experiment is reasonably transferable — other developers could use the same hardware and task setup to independently verify or challenge its conclusions.

Task Design Choices Matter Enormously

By switching from "reproduce an image" to "fill a shape with one color," the author effectively redefined the difficulty and dimensions of the evaluation. This is a reminder that when assessing AI capabilities, the task design itself will greatly influence the conclusions. Filling shapes lowers the bar for color and composition, while amplifying scrutiny of execution precision and boundary control. Different tasks will make different models "look stronger," so when evaluating any AI comparison experiment, the first question to ask is: what exactly is it measuring?

Embodied AI refers to giving AI systems a "body" so they can perceive, reason, and act through real-time interaction with the physical environment, rather than operating purely in digital space. The core argument of this concept is that truly general intelligence requires embodied experience as a foundation — just as human cognition is formed through continuous interaction with the physical world. Unlike pure language or vision models, embodied AI faces the unique challenges of physical irreversibility, sensor noise, and real-time response requirements — a single erroneous command can directly cause a hardware collision or task failure, with no "undo" option. Current mainstream embodied AI research scenarios include robot navigation, object grasping, and bimanual manipulation. Painting tasks, with their visualizable results and intuitive evaluation criteria, have become a particularly compelling and communicable entry point into this field.

The Suspense of Community Judgment

The most compelling aspect of this post is actually the deliberate suspense the author built in — they didn't reveal which model painted better, and expressed surprise at the consensus direction their friends arrived at.

This approach is thought-provoking. In discussions about AI models, people often carry strong brand expectations: some naturally favor Claude's precision, others champion the GPT series' general-purpose capabilities. But when the brand labels are hidden and only the actual output is evaluated, the conclusions can be counterintuitive. The author's "surprise" is itself telling: a model's real-world performance doesn't always match our preconceptions.

It's worth noting that this is, at the end of the day, a personal, very small-sample informal experiment. The evaluations were based mainly on a handful of friends' subjective impressions and cannot serve as rigorous evidence of one model's superiority over the other. It functions more as a thought-provoking prompt: as large models are increasingly integrated into the physical world, how should we design fair, reproducible evaluation methods to measure their true capabilities on embodied tasks?

That may be the most valuable question this robotic arm painting showdown leaves us with.