COMPREHEND: Ultra-Fine-Grained Benchmark Understanding with Entailment Hierarchies

Why Do We Need Finer-Grained Prompt Classification?

For LLM practitioners, understanding what types of prompts users typically submit and how models perform on those categories is key to optimizing models and designing applications. Existing methods usually build taxonomies by hierarchically clustering user prompts and using an LLM to synthesize cluster names. However, these cluster names are often too general for humans to understand the detailed distribution of prompts, let alone accurately identify the specific strengths and weaknesses of LLMs.

To address this problem, researchers propose COMPREHEND—a method that automatically constructs an ultra-fine-grained entailment hierarchy. The method replaces cluster names in the taxonomy with sentences, providing precise cluster descriptions for finer-grained classification.

COMPREHEND's Core Idea: Sentences Instead of Labels

The key innovation of COMPREHEND is replacing traditional "cluster names" with "sentences" to build an entailment hierarchy. In natural language inference, entailment describes the logical inference relationship between two sentences (e.g., "A entails B" means if A is true then B is true). COMPREHEND uses this relationship to organize prompts: more specific prompt sentences entail more general descriptions, forming a hierarchical tree from abstract to concrete.

Compared with advanced taxonomy construction methods such as EvalTree, COMPREHEND shows significant improvements on the WildBench benchmark: the entailment/faithfulness metric rises from 22.6 to 41.7, and the informativeness metric rises from 16.3 to 39.1. These results indicate that sentence-based descriptions can more precisely capture the subtle differences among prompts, making the taxonomy more interpretable.

Natural Language Inference (NLI) and Entailment: Natural language inference is the task of judging the logical relationship between two sentences, typically including three relations: entailment, contradiction, and neutral. In COMPREHEND, researchers mainly use entailment: if sentence A is true, then sentence B must be true, then A is said to entail B (A → B). For example, "The model performs poorly on algebra word problems requiring multi-step reasoning" entails "The model performs poorly on math problems," because the former is a specific instance of the latter. Using this relationship, specific prompt sentences can be organized under more general descriptions to form a hierarchy.

Entailment Hierarchy: An entailment hierarchy is a tree-structured taxonomy based on logical entailment relations, where parent nodes represent more general descriptions and child nodes represent more specific instances. Unlike traditional clustering hierarchies, an entailment hierarchy does not require child nodes under the same parent to be similar to each other; instead, it requires each child node to entail its parent. This structure naturally supports navigation from abstract to concrete, facilitating fine-grained analysis. Construction typically requires computing entailment probabilities between sentence pairs and establishing parent-child relationships accordingly.

EvalTree: EvalTree is an advanced taxonomy construction method that builds a classification tree by hierarchically clustering user prompts and using an LLM to generate a name for each cluster. Unlike COMPREHEND, EvalTree uses cluster names as node labels, and these names tend to be rather general, making it difficult to precisely describe the subtle differences among prompts within a cluster. COMPREHEND achieves finer-grained classification by replacing cluster names with sentences and organizing the hierarchy based on entailment relations. Experiments show that on the WildBench benchmark, COMPREHEND significantly outperforms EvalTree on both entailment/faithfulness and informativeness metrics.

Experimental Validation: Weakness Reports Are More Predictive

In addition to taxonomy quality metrics, the study also examines the practical utility of weakness reports generated from the entailment hierarchy. When predicting response scores in the benchmark, weakness reports generated by COMPREHEND are more informative than those from EvalTree. This means that fine-grained classification not only helps humans understand model behavior but can also be directly used to diagnose model deficiencies—for example, identifying the specific prompt types on which a model is prone to fail.

This result echoes a practical need: practitioners don't just want to know "the model performs poorly on math problems"; they want to know "the model performs poorly on algebra word problems requiring multi-step reasoning." COMPREHEND approaches this goal through sentence-level descriptions.

Weakness Reports: Weakness reports are summary documents generated from a taxonomy that describe a model's shortcomings on specific prompt categories. Typically, researchers collect model response scores for prompts in each node (or category) of the taxonomy and use an LLM to generate natural language reports pointing out the specific prompt types on which the model is prone to fail. The quality of weakness reports depends on the granularity of the taxonomy: the finer the classification, the more precisely the reports can locate model deficiencies. In COMPREHEND, because the hierarchy is based on sentence-level descriptions, the generated weakness reports can more specifically point out the fine-grained prompt types on which the model underperforms, thereby being more informative when predicting response scores.

Open Source and Reproducibility

The research team has open-sourced all code in the GitHub repository (JackNapier20/COMPREHEND_public), making it easy for the community to reproduce and extend. For researchers who want to build taxonomies in their own domains, this provides an actionable foundation.

Limitations and Outlook

Although COMPREHEND greatly surpasses EvalTree on the metrics, it relies on the quality of LLM-generated sentence descriptions, which may introduce generation bias. In addition, the construction cost of the entailment hierarchy (e.g., computing entailment relations between sentences) is not discussed in detail in the original paper. Future work may need to explore more efficient entailment judgment methods and extend the approach to multilingual or multimodal prompt scenarios.

Overall, COMPREHEND offers a new perspective for LLM benchmark understanding: shifting from "naming clusters" to "describing with sentences," so that the taxonomy truly serves fine-grained model diagnosis.

One-line Summary

COMPREHEND replaces cluster names with sentences to build an entailment hierarchy, enabling ultra-fine-grained prompt classification and improving model weakness diagnosis.

Paragraph Summary

COMPREHEND is a method that automatically constructs an ultra-fine-grained entailment hierarchy, aiming to solve the problem that existing prompt taxonomies have cluster names that are too general, making it difficult to understand prompt distributions and identify model weaknesses. The method replaces traditional cluster names with sentences and uses entailment relations from natural language inference to organize prompts, forming a hierarchical tree from abstract to concrete. On the WildBench benchmark, COMPREHEND improves the entailment/faithfulness metric from 22.6 to 41.7 and the informativeness metric from 16.3 to 39.1, significantly outperforming EvalTree. Weakness reports generated from this hierarchy are more informative when predicting response scores and can help practitioners precisely locate model deficiencies. The code is open-sourced for community reproduction and extension. Future work needs to address issues such as generation bias and construction cost.