Open-Source Chess Robot Arm: A Complete Breakdown of the Dual-CNN Vision System

An Open-Source Robot Arm That Plays Chess

Recently, a developer shared on Reddit a chess-playing robot arm project built with his team. This is no simple programming toy—it's a full engineering implementation spanning mechanical design, structural modeling, and software control. The team first completed the structural design and fabrication in SolidWorks, after which this developer took charge of programming, ultimately enabling the robot arm to play chess against a human opponent.

SolidWorks is a 3D computer-aided design (CAD) software developed by Dassault Systèmes, widely used in mechanical engineering and product design. In this robot arm project, SolidWorks was used not only for part modeling and assembly design but also for motion simulation and stress analysis, helping the team verify joint range of motion, load capacity, and structural integrity before actual fabrication. For open-source hardware projects, the exported 3D model files (such as STL or STEP formats) can be directly used for 3D printing or CNC machining, significantly lowering the barrier for others to reproduce the project.

What makes this project especially noteworthy is that it's fully open-source, with code hosted on a GitHub repository containing firmware, a graphical user interface (GUI), training scripts, and links to assets and datasets. For hobbyists looking to get started with robot vision and embedded control, this is a rare and valuable reference implementation.

System Architecture: A Complete Loop from Camera to Robot Arm

The entire system's workflow is clear and representative. An IP camera is mounted above the chessboard, capturing the board in real time and streaming video to a computer hidden beneath the table for inference. After the computer identifies the game state, it drives the robot arm to make its move.

An IP camera (Internet Protocol Camera) is a digital video device that transmits video data via network protocols. Unlike traditional USB cameras, it doesn't require a direct physical connection to the computer—instead, it transmits video streams over Wi-Fi or Ethernet. Common transmission protocols include RTSP (Real-Time Streaming Protocol) and the ONVIF standard. In this project, the IP camera is mounted directly above the chessboard for a top-down view while the computer sits under the table, eliminating the need for cable connections between the two—a practical solution for space-constrained tabletop setups. Developers typically use OpenCV's VideoCapture function to read from an RTSP stream URL, enabling real-time image acquisition.

This "Sense-Plan-Act" closed loop is the core paradigm of robotic systems. The Sense-Plan-Act architecture is a classic robot control framework first proposed by MIT's Artificial Intelligence Laboratory in the 1980s. In this paradigm, the perception layer gathers environmental information from sensors, the planning layer converts perception data into action plans, and the execution layer translates plans into physical actions through actuators (motors, pneumatic cylinders, etc.). Although alternative approaches like behavior-based architecture and end-to-end learning have since emerged, for tasks like chess—where rules are well-defined and the environment is controllable—the classic three-layer architecture remains the clearest and most debuggable choice. In this project, the camera handles perception, the vision models on the computer handle state recognition and decision-making, and the robot arm firmware handles precise execution—all three connected through network and serial communication to form a complete closed loop. The IP camera's advantage over USB cameras lies in its flexible wiring and ease of fixed-angle top-down mounting, which is particularly well-suited for chessboard recognition scenarios requiring a stable overhead perspective.

Dual-CNN Vision Recognition System Design

The most technically sophisticated part of the project is the vision recognition pipeline, which employs two independent Convolutional Neural Network (CNN) models that run inference separately on each square of the chessboard.

A Convolutional Neural Network (CNN) is a deep learning architecture specifically designed for processing grid-like data such as images. Its core idea is to use learnable filters (convolution kernels) in convolutional layers that slide across the image, extracting local features like edges, textures, and shapes. A typical CNN architecture consists of convolutional layers, pooling layers (for dimensionality reduction and translation invariance), and fully connected layers (for classification decisions). Compared to traditional image processing methods like template matching or color threshold segmentation, CNNs can learn more generalizable feature representations, offering greater robustness against interference factors such as lighting variations and differences in piece appearance.

Division of Labor Between the Two CNN Models

The first CNN model detects whether a piece is present on each square and, if so, its color (black or white). The second CNN model determines the piece's precise position within the square. This approach of decomposing a complex recognition task into two sub-tasks is a common and effective strategy in engineering practice.

Compared to training a single monolithic model to simultaneously recognize "presence/absence, color, and position," decoupling the task into two focused smaller models offers several clear advantages: each model has a more focused training objective, data annotation is simpler, and models are easier to converge and achieve high accuracy. Additionally, while per-cell inference requires more computation, it avoids the interference from mutual occlusion and perspective distortion that whole-board recognition would encounter, resulting in more robust identification.

It's worth noting that per-cell inference means the system needs to segment the chessboard image into 64 individual square images, then run both CNN models on each square, totaling 128 forward inference passes. While this is far more inference runs than performing a single object detection pass on the entire board image (e.g., using YOLO or SSD), each inference operates on a very small input image with extremely low per-pass computational cost, and it avoids the complex post-processing steps common in object detection such as anchor box design and Non-Maximum Suppression (NMS). On embedded or desktop-grade hardware, the total inference time for 64 small image classifications can typically be kept at the millisecond level, more than sufficient for chess—a scenario where real-time requirements aren't extreme.

Why the Position Model Is Indispensable

The second model's purpose is particularly interesting. Simply knowing whether each square has a piece isn't enough—the robot arm needs precise grasping coordinates. Pieces are often not perfectly centered within their squares, especially after human moves or the robot arm's own operations, where pieces may shift. A dedicated position estimation model that corrects the piece's actual coordinates within its square can significantly improve the robot arm's grasping success rate, reducing mistakes like knocking over or missing pieces.

AI-Assisted Development: A Strategic Split Between Handwritten Firmware and Auto-Generated GUI

The developer openly shared that he adopted a differentiated strategy throughout development: the firmware was primarily written by hand, while the remaining parts—especially the GUI—were largely built with AI assistance.

This division reflects a mature understanding among today's developers about AI coding tools. Firmware is the low-level software running on embedded hardware (such as microcontrollers), directly controlling motor drivers, sensor readings, and communication protocols. In a robot arm project, firmware must coordinate the movement of multiple servos or stepper motors, involving Inverse Kinematics calculations—computing the required rotation angle for each joint based on a target end-effector position. Firmware also manages serial communication (e.g., UART) or USB protocols to receive commands from the host computer, and implements real-time motion interpolation to ensure smooth arm movement. These tasks demand extremely precise timing—a minor timer configuration error could cause motor step loss or communication packet drops. The cost of errors is high and debugging is difficult, making the developer's choice to maintain direct control a prudent engineering judgment.

On the other hand, GUI code—with its high degree of pattern regularity, well-defined boundaries, and low cost of trial-and-error—plays perfectly to the strengths of AI coding assistants, which can rapidly generate usable interface frameworks and save enormous amounts of repetitive work.

This model of "human-led for critical parts, AI-accelerated for peripheral parts" may well be the practical path for individual developers and small teams to efficiently complete complex projects.

Practical Value and Technical Takeaways from the Open-Source Project

The completeness of this open-source project is commendable. The repository contains not only firmware and GUI code but also training scripts along with links to datasets and 3D model assets. This means others can reproduce the entire pipeline from scratch—whether their interest lies in mechanical design, embedded firmware development, or computer vision model training, they'll find relevant references.

For beginners in robot vision, this project offers several directly applicable lessons:

• Task decomposition beats monolithic models: Breaking recognition tasks into multiple focused sub-tasks is often more efficient and reliable than training a single complex model.
• Fixed viewpoints simplify vision problems: A fixed overhead camera layout transforms open-scene recognition challenges into controlled grid-based recognition.
• Use AI tools wisely: Decide whether to write code by hand or delegate to AI based on the code's criticality and complexity.

Conclusion

This chess-playing robot arm project, shared by a single developer, nonetheless presents the complete implementation path of a cross-disciplinary robotics system—bringing together mechanical design, embedded control, computer vision, and AI-assisted programming. It carries no flashy marketing, yet its fully open-source nature gives it genuine educational and reference value. For anyone interested in robotics, CNN-based visual recognition, or DIY hardware projects, this is a starting point well worth exploring in depth.