COMPREHEND
A method for automatically constructing ultra-fine-grained entailment hierarchies. It replaces cluster names in conventional taxonomies with sentences and organizes prompt categories based on entailment relations in natural language inference, enabling finer-grained prompt classification and model weakness analysis.
Timeline (last 90 days)
COMPREHEND is a method that automatically builds ultra-fine-grained entailment hierarchies, using sentences in place of the cluster names in traditional taxonomies.
On the WildBench benchmark, COMPREHEND's entailment/faithfulness metric rises from 22.6 for EvalTree to 41.7.
When predicting response scores in the benchmark tests, the weakness reports generated by COMPREHEND are more informative than those of EvalTree.
COMPREHEND's code is open-sourced at the GitHub repository JackNapier20/COMPREHEND_public.
COMPREHEND depends on the quality of the sentence descriptions generated by the LLM, which may introduce generation bias.
All Facts (5)
COMPREHEND is a method that automatically builds ultra-fine-grained entailment hierarchies, using sentences in place of the cluster names in traditional taxonomies.
50%UnverifiedOn the WildBench benchmark, COMPREHEND's entailment/faithfulness metric rises from 22.6 for EvalTree to 41.7.
50%UnverifiedWhen predicting response scores in the benchmark tests, the weakness reports generated by COMPREHEND are more informative than those of EvalTree.
50%UnverifiedCOMPREHEND depends on the quality of the sentence descriptions generated by the LLM, which may introduce generation bias.
50%UnverifiedCOMPREHEND's code is open-sourced at the GitHub repository JackNapier20/COMPREHEND_public.
50%