BoT-Feedback: Making AI Coaching More Explainable with Biomechanical Evidence

When AI Coaches: Why Current Approaches Fall Short

Multimodal large language models (MLLMs) have demonstrated impressive capabilities in visual understanding and multimodal reasoning — yet they still have fundamental shortcomings when it comes to generating feedback on human movement. Imagine an AI watching an athlete's training video and offering improvement suggestions. That's the core job of a coach, and it's precisely where current models are weakest.

The root of the problem is that existing approaches typically infer instructional feedback directly from visual observation. This leads to three obvious failure modes: overly generic suggestions (vague advice like "make the movement cleaner"), poor explainability (no clear reasoning behind recommendations), and physically implausible hallucinations (guidance that violates basic principles of human biomechanics).

By contrast, how do real expert coaches work? They diagnose performance through explicit biomechanical reasoning — analyzing joint kinematics, posture, and body dynamics. This is precisely the core gap between human experts and current AI systems.

Multimodal large language models (MLLMs) are large language models capable of processing multiple input modalities simultaneously — text, images, video, and more. Unlike pure text models, MLLMs use visual encoders to convert images or video frames into feature vectors the model can understand, then fuse them with language model reasoning. Representative models include GPT-4V, Gemini, and LLaVA. In sports analysis scenarios, MLLMs must understand sequences of human motion across consecutive frames — a task that demands far more sophisticated temporal modeling and 3D spatial perception than ordinary image understanding. "Biomechanical hallucinations" refer to model-generated suggestions that violate human anatomy or the laws of movement mechanics — for example, asking a joint to bend in an impossible direction, or ignoring center-of-mass and balance constraints. Such errors are especially common in end-to-end models that lack structured physical knowledge constraints.

BoT-Feedback's Core Idea: Ground Reasoning in Evidence

The arXiv paper introducing BoT-Feedback proposes a four-stage reasoning framework called Biomechanics of Thought (BoT). Rather than letting the model give suggestions based on intuition, it mimics the diagnostic logic of a professional coach, proceeding step by step:

Four-Stage Progressive Reasoning

• Identify the action: First, determine what movement is being performed in the video;
• Localize key body regions: Identify the body parts most critical to that movement;
• Quantify the gap: Compare expert and student performances through quantitative biomechanical differences;
• Synthesize feedback: Based on the evidence gathered in the previous three steps, generate explainable instructional recommendations.

This structured reasoning chain transforms feedback from "black-box intuition" into "evidence-backed diagnosis," where every recommendation can be traced to a specific biomechanical basis.

Supporting Infrastructure: Data Parser and Alignment Strategy

Having the framework alone isn't enough to make this work. The research team designed two key components.

The first is a plug-and-play Biomechanical Data Parser (BDP), which converts raw video into structured biomechanical descriptors — essentially translating messy visual information into a "mechanical language" the model can understand.

The second is an alignment strategy that matches expert and student movements along the temporal dimension. This step is critical: only by precisely aligning two motion sequences on the time axis can meaningful frame-by-frame comparisons be made. Without it, quantitative analysis becomes impossible.

For evaluation, the team also constructed a benchmark dataset called BiomAF, containing paired teacher-student videos, 3D skeletons, biomechanical attributes, and expert instructional annotations. This multimodal, expert-labeled dataset provides a reproducible evaluation foundation for explainable action feedback research.

The "structured biomechanical descriptors" extracted by the BDP typically include quantitative metrics such as joint angles, angular velocities, limb segment orientations, and center-of-mass trajectories — data generally inferred from raw video by pose estimation models (such as MediaPipe, OpenPose, or deep learning-based 3D pose reconstruction networks). Temporal alignment faces a classic challenge: different learners complete the same movement at varying speeds and rhythms, so direct frame-by-frame comparison introduces substantial noise. Common alignment techniques include Dynamic Time Warping (DTW), an algorithm that elastically stretches or compresses the time axis while preserving temporal order, aligning key pose waypoints between two sequences to make quantitative comparisons physically meaningful.

Experimental Results: Small Models Can Rival Large Ones

The paper conducted experiments across twelve open-source and closed-source MLLMs, with clear conclusions: grounding reasoning in biomechanical evidence consistently improves feedback quality, explainability, and robustness, while substantially reducing biomechanical hallucinations.

The most compelling figure is the improvement in expert evaluation scores — BoT-Feedback raised the average expert score from 2.07 to 2.95, a relative improvement of approximately 40%. Even more noteworthy is the shift in the competitive landscape: this approach enables compact open-source MLLMs to approach the performance of much larger proprietary systems, narrowing the gap between open- and closed-source models on explainable action feedback generation.

This suggests that through structured reasoning informed by domain knowledge, model scale is no longer the only determining factor — a well-designed reasoning framework can go a long way toward compensating for a disadvantage in parameter count.

The "expert evaluation scores" referenced in the paper are human assessment metrics in which domain experts (sports scientists or coaches) rate model-generated feedback across dimensions typically including accuracy, explainability, specificity, and actionability. The improvement from 2.07 to 2.95 represents a significant gain for a human evaluation task — human evaluation scores are inherently subject to rater subjectivity, and scoring scales tend to be narrow, so an absolute improvement of 0.88 points corresponding to roughly 40% relative gain indicates that introducing a biomechanical reasoning chain had a substantive impact on output quality, not merely a marginal one. This evaluation approach is widely used in high-expertise domains such as medical Q&A and educational feedback, to compensate for the inability of automated metrics (such as BLEU and ROUGE) to capture domain-specific correctness.

Significance and Limitations

BoT-Feedback's value extends beyond sports coaching. It reflects a broader trend: when general-purpose large models struggle in specialized domains, rather than simply stacking more parameters, it's more effective to explicitly inject expert reasoning logic — in this case, biomechanics — into the model's reasoning chain.

This approach of "evidence grounding" has implications for any scenario requiring high explainability: medicine, rehabilitation training, sports science, and beyond. It upgrades AI output from "sounds reasonable" to "verifiable with evidence" — precisely the property most valued in high-stakes decision-making contexts.

Of course, as a newly published preprint, the methods still require broader empirical validation. The BDP's dependence on video quality and pose estimation accuracy, the coverage scope of the BiomAF benchmark, and real-world deployment effectiveness in actual instructional settings are all directions worth monitoring going forward. But regardless, the idea of grounding multimodal reasoning in biomechanical evidence provides a clear, quantifiable paradigm for explainable AI feedback.