Building a Bouldering Analysis Tool with Computer Vision: A New Paradigm Where VLM Prompts Replace Model Training

A Climber's AI Experiment

Computer vision technology is moving from the lab into everyday life scenarios. Recently, a climbing enthusiast and developer shared a bouldering analysis tool built with Vision Language Models (VLMs) on Reddit, sparking considerable discussion. This project not only demonstrates the capabilities of modern visual AI but also reveals an important trend: we're shifting from a paradigm of "training specialized models" to "prompting general-purpose models."

The creator admits to being a bouldering beginner, noting that part of the motivation for building this tool was to better understand their own movements and routes while climbing. Although it's just a first demo, the technical combination and implementation approach are quite inspiring.

Bouldering is a major discipline within rock climbing, specifically referring to climbing lower rock faces or artificial walls (typically under 4-5 meters) without rope protection, with thick crash pads placed on the ground for safety. Unlike traditional rope climbing, bouldering emphasizes the technical difficulty and explosive power of individual moves, with each route (called a "problem") typically consisting of only 4-8 moves. In indoor climbing gyms, different colored holds represent different routes and difficulty grades, and climbers must complete the ascent using only holds of the designated color — which is why the project needs to precisely segment "blue holds," as they represent a specific climbing route.

Technical Architecture: Three AI Models Working in Concert

The core of this tool lies in the combined use of three AI models, each handling its own responsibility, ultimately assembling into a complete climbing movement analysis system.

Hold Segmentation: VLM Run's Visual Agent Orion

The most compelling aspect of the project is the use of VLM Run's visual agent Orion to segment all blue bouldering holds on the wall. The developer simply provides a natural language prompt, and Orion identifies and segments all blue holds in the frame with impressive results.

To understand why this step is so critical, one needs to first understand how Vision Language Models (VLMs) work. VLMs are a class of multimodal AI models capable of simultaneously understanding images and natural language. Unlike traditional computer vision models that can only perform predefined tasks, VLMs acquire broad visual understanding capabilities through pre-training on massive image-text paired datasets. Users can describe targets they want to identify in natural language, and the model can locate and segment those targets in images — even if those target categories never appeared in specialized training data. This capability is called "zero-shot" or "open-vocabulary" recognition, and is one of the most important breakthroughs in computer vision over the past two years.

VLM Run is a platform providing vision language model inference services, allowing developers to access visual understanding and segmentation capabilities through APIs. Orion is a Visual Agent on the platform focused on image segmentation tasks. Unlike traditional instance segmentation models (such as Mask R-CNN) that require training on specific category datasets, Orion accepts natural language instructions as input and can find and precisely segment corresponding regions in images based on text descriptions. This "prompt-to-segment" paradigm dramatically simplifies the development workflow.

The key insight here lies in the transformation of the computer vision workflow. The author specifically noted:

"What's interesting is that we can now directly prompt a VLM to segment all the holds, rather than creating a new dataset from scratch to train a specialized model."

In traditional computer vision pipelines, identifying targets in niche scenarios like climbing holds typically requires: collecting large amounts of images, manually annotating hold positions, training an object detection model, and iteratively fine-tuning. The entire process is time-consuming and labor-intensive, with a high barrier for individual developers. With VLM's general visual understanding capabilities, all of this is reduced to a single prompt.

Pose Estimation and Human Detection Models

Beyond hold recognition, the tool also needs to understand the climber's body movements. The developer chose two mature models from Hugging Face:

• ViTPose+ Huge: Responsible for pose estimation, tracking the climber's joint points and limb positions
• RT-DETR: Responsible for human detection, locating the climber within the frame

ViTPose+ is a human pose estimation model based on the Vision Transformer (ViT) architecture, developed by research teams from institutions including the University of Sydney. "Huge" refers to the model's parameter scale — it's the largest version in the ViTPose+ series with the strongest accuracy performance. The core task of Pose Estimation is detecting human keypoints (such as shoulders, elbows, wrists, knees, ankles, etc.), typically outputting 2D or 3D coordinates for 17-25 keypoints. Compared to traditional CNN-based methods (like OpenPose or HRNet), ViTPose+ leverages Transformer's global attention mechanism, performing more robustly in occlusion and complex pose scenarios. In climbing analysis, precise joint localization is crucial for determining whether hands and feet are in contact with holds.

RT-DETR (Real-Time Detection Transformer) is a real-time object detection model proposed by Baidu in 2023 — the first Transformer-based architecture to achieve real-time end-to-end detection performance. Traditional object detection models (like the YOLO series) rely on hand-designed post-processing steps (such as Non-Maximum Suppression/NMS), while the DETR series eliminates these steps through Transformer's set prediction mechanism, achieving truly end-to-end detection. RT-DETR achieves real-time inference speed while maintaining high accuracy, making it ideal for video analysis applications that require fast response. In this project, RT-DETR is responsible for quickly locating the climber's position in video frames, providing bounding boxes for subsequent pose estimation.

Notably, both models are hosted on Hugging Face, currently the largest open-source AI model community and hosting platform, often called "the GitHub of AI," hosting over 500,000 pre-trained models. Developers can directly download and use these models without training from scratch, with just a few lines of code through the unified API interface (Transformers library). This open-source ecosystem dramatically lowers the barrier to AI application development.

This combination allows the system to precisely capture real-time positions of hands, feet, and torso, providing the data foundation for subsequent movement analysis.

Feature Implementation: From Hold Detection to Climbing Path Visualization

By combining hold detection with pose estimation, the tool implements a complete climbing movement tracking logic:

• Hold activation visualization: When the climber's hand or foot touches a hold, the system highlights it as "activated"
• Completion detection: When both hands simultaneously touch the final hold, the route is judged as completed
• Path recording: By tracking the torso midpoint trajectory, the system draws a visualization of the entire climbing path

Regarding the technical implementation of path recording, the torso midpoint is a commonly used approximation of the body's center of gravity in pose estimation, typically calculated as the geometric center of the left/right shoulder keypoints and left/right hip keypoints. Although the body's true center of mass shifts with posture changes (approximately at navel level), the torso midpoint as a simplified indicator effectively reflects the climber's overall displacement on the wall. By recording this midpoint's coordinates frame by frame and connecting them, a visualized movement trajectory is generated. This trajectory can intuitively reveal whether the climber has unnecessary lateral swinging, whether their center of gravity stays close to the wall, and other key technical elements.

This design approach aligns well with bouldering's actual rules — bouldering routes typically use "both hands on the top hold" as the completion marker. By recording the torso trajectory, climbers can visually review their movement path on the wall, analyzing whether their movements are efficient and whether their route choices are optimal.

The Practical Value of AI-Assisted Sports Training

The value of this project lies not only in the technology itself but also in demonstrating the practical possibilities of AI-assisted sports analysis. Climbing is a sport with extremely high demands on body coordination, route planning, and force distribution. For beginners, it's difficult to objectively assess the quality of their movements on each attempt.

The creator mentioned that such a tool "can help climbers better understand their movements." Imagine being able to replay the center of gravity trajectory, the sequence and timing of each hold used during every climb — climbers could identify habitual movement deficiencies and make targeted improvements. This parallels the motion capture analysis already widespread in professional sports — in professional athletics, athletes use optical motion capture systems like Vicon for training analysis, with a single setup costing hundreds of thousands of dollars. With open-source models and VLMs, individual developers can achieve similar functionality at virtually zero hardware cost, requiring only a smartphone camera and cloud API call fees.

It's worth noting that the creator also paid tribute to community pioneers — specifically thanking Daniel Reiff and his bouldering computer vision project for the inspiration. This reflects the virtuous cycle of knowledge and inspiration sharing within the open-source community.

Paradigm Shift: Natural Language Prompts Replace Model Training

If there's one thing about this project worth the entire AI field reflecting on, it's the development paradigm shift it illustrates.

In the past, building computer vision applications for any vertical niche scenario (like "identifying blue climbing holds") required developers to complete the full pipeline of "data collection → annotation → training → deployment." This path requires not only large amounts of data and compute but also considerable machine learning expertise. Taking a simple hold detection model as an example, the traditional workflow might require: photographing hundreds of climbing wall images from different angles and lighting conditions, annotating the bounding box or segmentation mask for each hold in every image (annotation alone could require dozens of hours of manual labor), selecting an appropriate model architecture, configuring training hyperparameters, conducting multiple rounds of training and validation, and finally deploying as an inference service. The entire cycle is measured in weeks or even months.

Now, with the maturation of vision language model capabilities, many long-tail recognition tasks can be solved directly through natural language prompts. Developers' focus shifts from "training models" to "combining and orchestrating existing capabilities" — assembling VLM segmentation capabilities, ViTPose+'s pose estimation abilities, and RT-DETR's detection capabilities like building blocks to quickly prototype functional applications. This pattern aligns with the "microservices architecture" philosophy in software engineering: each AI model is a service focused on a specific task, and the developer's core competency lies in understanding each model's strengths and limitations, then designing reasonable data flows and interaction logic to chain them together.

This transformation dramatically lowers the barrier to AI application development. An amateur climbing enthusiast being able to build a fully functional movement analysis tool without constructing a specialized dataset was almost unimaginable just a few years ago.

Conclusion

Although the creator humbly states this is just a "first demo" with plenty of room for improvement, this project clearly demonstrates the practical potential of modern visual AI in sports analysis. It's a microcosm of technological democratization — when powerful computer vision models become readily accessible, creativity and imagination become the scarcer resources.

For developers, perhaps the next interesting application isn't about training the most powerful model, but about cleverly combining existing capabilities like VLMs, pose estimation, and object detection to solve real, specific, and even somewhat niche everyday problems.