MYOPIA Benchmark Reveals: Why Tree-Based Reasoning Fails in Vision-Language Models

Decision trees can turn a single classification by a vision-language model (VLM) into a chain of interpretable question-answer steps from root to leaf, but the accuracy cost of this "greedy, non-backtracking" reasoning path has long lacked systematic evaluation. A study titled MYOPIA: Understanding the Limits of Tree-Based Reasoning in Vision-Language Models delivers a less-than-optimistic answer: across all 12 paired experiments spanning four datasets and three models, tree traversal lost to zero-shot prompting without exception.

The research team's MYOPIA benchmark includes four LLM-generated, human-refined decision trees corresponding to GTSRB, MNIST, CIFAR-10, and MSCOCO, along with prompt templates, diagnostic protocols, and evaluation tools. The code is open-sourced on GitHub.

Why tree-based reasoning looks appealing

Decomposing image classification into hierarchical questions like "Is it circular? → Does it have edges? → …" is intuitively attractive. Each step's judgment can be read by humans, and errors can be traced to specific nodes—an appealing property for vision systems that require auditing.

The problem is that this kind of traversal is inherently greedy: once the model makes a mistake at a node, it can never return to the correct branch. The paper calls this mechanism "irreversible exclusion"—once a local judgment goes wrong, the correct category is permanently kicked out of the candidate set. Zero-shot prompting has no such constraint; it can weigh all evidence from the entire image in one shot.

This constitutes the structural risk of tree-based reasoning: the gains from interpretability may be offset by the cost of single points of failure.

Zero-shot Prompting refers to a reasoning approach where no examples are provided to the model—the task is simply described as an instruction for the model to answer. In visual classification scenarios, zero-shot prompting typically feeds the entire image together with a question like "What category is this?" into the VLM, letting the model produce an answer in one pass. It does not rely on a predefined tree structure, nor does it require multiple rounds of Q&A, so all evidence from the entire image can be weighed in parallel within a single forward pass. In contrast, decision tree traversal breaks the same image into several independent sub-questions, each focusing on a single local attribute, and the model cannot "look back" at previously excluded branches. This is precisely one of the fundamental reasons why zero-shot prompting consistently wins in the paper: it naturally avoids the single-point-of-failure constraint of tree-based reasoning, at the cost of sacrificing step-by-step interpretability.

Three models, four datasets: zero-shot wins across the board

The experiments cover three models: GPT-4o, Qwen-VL Max, and LLaMA-3.2 11B Vision. In all 12 "dataset × model" pairings, tree traversal accuracy was lower than zero-shot prompting, and paired bootstrap analysis consistently supported the zero-shot approach.

This is not an incidental phenomenon of a particular model type or dataset, but a stable pattern that holds across models and datasets.

Paired bootstrap analysis is a statistical method for evaluating the significance of performance differences between two approaches when sample sizes are limited. The procedure treats the same batch of test samples as paired data, records both methods' predictions for each pair, and then repeatedly resamples with replacement (bootstrap) to estimate the distribution and confidence interval of the accuracy difference. Compared with simply subtracting two accuracy rates, the paired design controls for variance introduced by individual sample difficulty, making it more sensitive for comparisons like "tree traversal vs. zero-shot" under the same data and model. The paper's statement that "paired bootstrap analysis consistently supports the zero-shot approach" means this gap is not accidentally driven by a particular batch of hard samples, but is statistically stable and reproducible, strengthening the conclusion's reliability across models and datasets.

Three diagnosed failure modes

The paper does not stop at the conclusion that "tree-based reasoning is worse," but breaks down where the gap comes from.

Attribute memory is stronger than image grounding. Models "memorize" the category attributes encoded in the tree better than they "see" them. The label–image accuracy gaps for GPT-4o, Qwen-VL Max, and LLaMA reach 9.5, 13.8, and 12.6 percentage points, respectively. In other words, the model knows what features a class "should have," but cannot reliably confirm whether those features are present in the image.

Erroneous decisions are irreversible. Once a local judgment goes wrong, the correct category is excluded, and no matter how accurate subsequent nodes are, the mistake cannot be undone.

Answer extraction introduces additional loss. Parsing the answers needed for tree nodes from model outputs itself involves model-dependent errors, further dragging down end-to-end accuracy.

Label–image accuracy gap is the core metric in this paper for diagnosing "attribute memory stronger than image grounding." Its procedure is: first, provide the model with only the category label (e.g., "stop sign") and have it answer attribute questions at tree nodes, recording accuracy as the "label-conditioned" upper bound; then have the model answer the same questions using only the real image, recording "image-conditioned" accuracy. The difference between the two reflects how much the model relies on category priors memorized during training rather than actually reading evidence from pixels. The gaps for GPT-4o, Qwen-VL Max, and LLaMA-3.2 are 9.5, 13.8, and 12.6 percentage points, respectively, indicating that even when a model can correctly state the features a class of objects "should have," grounding them in a specific image still causes a noticeable drop—this is precisely the most fragile link in tree-based reasoning.

What factors affect node accuracy

The study conducted a node-level audit of the four trees. There is a weak but consistent negative correlation between split entropy and node accuracy: the higher the entropy, the more error-prone the node judgment.

In contrast, structural metrics such as depth, branching factor, and within-node grounding differences do not show a stable relationship with accuracy. This means that simply making trees shallower or narrower may not solve the problem; the real bottleneck may lie in whether each node's question is "perceptually answerable"—whether the model can answer it based solely on image evidence.

Split entropy measures how "impure" the category distribution of sub-branches is when a tree node splits downward. The higher the entropy, the more mixed the candidate categories under that node, and the model must answer a vaguer, harder-to-define question (e.g., "Is this object square?" when the node contains road signs, digits, and animals simultaneously), making judgment more error-prone. The study observes a weak but consistent negative correlation between split entropy and node accuracy, meaning that even if overall structural metrics (depth, branching factor) are unstable, the "distinguishability" of the node question itself can still predict the probability of error. This finding supports the paper's "perceptual answerability" recommendation: rather than blindly compressing the tree's size, it is better to prioritize ensuring that each node's question can be reliably answered by the model based solely on image evidence.

Implications for tree-based visual reasoning design

The value of MYOPIA lies not only in pointing out flaws—it provides a framework for diagnosing hierarchical visual classification. Based on this, the paper's authors propose two directions:

First, prioritize perceptual answerability: when designing node questions, first confirm whether the model can directly obtain the required evidence from the image, rather than relying on category priors it memorized during training.

Second, allow recovery from early errors: introduce backtracking or parallel branching mechanisms to avoid a single mistake locking down the entire path.

For teams attempting to transform vision systems with decision trees, chain-of-thought, or similar hierarchical structures, this benchmark and diagnostic protocol are worth using as a pre-launch stress test. Interpretability is never free, and MYOPIA quantifies this cost through 12 experiments.