Compositional Equivalence
The phenomenon in which different module sequences produce the same function in end-to-end semantics; it is one of the key factors that inflate benchmark scores in compositional generalization evaluation.
Timeline (last 90 days)
Compositional equivalence refers to the phenomenon in which different module sequences produce the same function in end-to-end semantics.
If a benchmark contains both module coverage violations and compositional equivalence, shortcut learning is exacerbated.
Compositional generalization benchmark design should avoid test sets that contain compositional equivalence, control module coverage shift, and consider both factors simultaneously.
In the code output prediction task on the coding benchmark LiveCodeBench, the compositional generalization performance of the pretrained model Llama3-8B is affected by compositional equivalence.
Evaluating compositional generalization is not merely a training-test split problem; ignoring compositional equivalence and module coverage may lead to erroneous judgments about model capabilities.
All Facts (5)
Compositional equivalence refers to the phenomenon in which different module sequences produce the same function in end-to-end semantics.
50%UnverifiedIf a benchmark contains both module coverage violations and compositional equivalence, shortcut learning is exacerbated.
50%UnverifiedCompositional generalization benchmark design should avoid test sets that contain compositional equivalence, control module coverage shift, and consider both factors simultaneously.
50%UnverifiedIn the code output prediction task on the coding benchmark LiveCodeBench, the compositional generalization performance of the pretrained model Llama3-8B is affected by compositional equivalence.
50%UnverifiedEvaluating compositional generalization is not merely a training-test split problem; ignoring compositional equivalence and module coverage may lead to erroneous judgments about model capabilities.
50%