COGS
A benchmark for evaluating compositional generalization in neural networks and large language models, designed to expose deficiencies in systematic compositional abilities across these model types.
Timeline (last 90 days)
Researchers have designed test sets such as SCAN, COGS, and ARC to probe compositional generalization ability.
Benchmarks such as SCAN and COGS reveal clear shortcomings in large language models' systematic compositional abilities.
Lake, Baroni, and others validated on benchmarks such as SCAN and COGS that current deep learning models have significant shortcomings in systematic generalization.
All Facts (3)
Researchers have designed test sets such as SCAN, COGS, and ARC to probe compositional generalization ability.
50%UnverifiedBenchmarks such as SCAN and COGS reveal clear shortcomings in large language models' systematic compositional abilities.
50%UnverifiedLake, Baroni, and others validated on benchmarks such as SCAN and COGS that current deep learning models have significant shortcomings in systematic generalization.
50%