MYOPIA
A benchmark for evaluating the performance of vision-language models under tree-structured reasoning. It includes decision trees generated by LLMs and refined by humans, prompt templates, diagnostic protocols and evaluation tools, covering the GTSRB, MNIST, CIFAR-10 and MSCOCO datasets.
Timeline (last 90 days)
The MYOPIA benchmark contains four decision trees generated by LLMs and refined manually, corresponding to the GTSRB, MNIST, CIFAR-10, and MSCOCO datasets respectively.
The paper diagnoses three types of failure modes: attribute memory being stronger than image grounding, erroneous decisions being irreversible, and answer extraction introducing additional loss.
The paper's authors suggest prioritizing the perception of answerability, designing node questions by first confirming whether the model can directly obtain the required evidence from the image, rather than relying on category priors memorized during training.
All Facts (3)
The MYOPIA benchmark contains four decision trees generated by LLMs and refined manually, corresponding to the GTSRB, MNIST, CIFAR-10, and MSCOCO datasets respectively.
50%UnverifiedThe paper diagnoses three types of failure modes: attribute memory being stronger than image grounding, erroneous decisions being irreversible, and answer extraction introducing additional loss.
50%UnverifiedThe paper's authors suggest prioritizing the perception of answerability, designing node questions by first confirming whether the model can directly obtain the required evidence from the image, rather than relying on category priors memorized during training.
50%