[KongchangAI]
Conceptcomposition equivalence

Compositional Equivalence

The phenomenon in which different module sequences produce the same function in end-to-end semantics; it is one of the key factors that inflate benchmark scores in compositional generalization evaluation.

Timeline (last 90 days)

Nov 11

Compositional equivalence refers to the phenomenon in which different module sequences produce the same function in end-to-end semantics.

Unverified50%
Nov 11

If a benchmark contains both module coverage violations and compositional equivalence, shortcut learning is exacerbated.

Unverified50%
Nov 11

Compositional generalization benchmark design should avoid test sets that contain compositional equivalence, control module coverage shift, and consider both factors simultaneously.

Unverified50%
Nov 11

In the code output prediction task on the coding benchmark LiveCodeBench, the compositional generalization performance of the pretrained model Llama3-8B is affected by compositional equivalence.

Unverified50%
Nov 11

Evaluating compositional generalization is not merely a training-test split problem; ignoring compositional equivalence and module coverage may lead to erroneous judgments about model capabilities.

Unverified50%

All Facts (5)

Source Articles