Masked Filler
A common protocol in long-context evaluation: to control for length as a variable, researchers replace irrelevant filler text with special placeholders or mask it directly so that the model cannot see this content. New research points out that this operation itself introduces additional interference.
Timeline (last 90 days)
On the MMLU benchmark, the masked-filler protocol reproduces the 11 to 18 percentage point performance drop caused by length (at 30K tokens).
In retrieval tasks, when the filler is masked, accuracy loss at 32K tokens is as high as 45 percentage points; when the same filler remains visible, the loss is zero.
The masked-filler protocol has a flaw: the masking operation itself changes the input distribution and introduces additional interference, rather than length being at work.
All Facts (3)
On the MMLU benchmark, the masked-filler protocol reproduces the 11 to 18 percentage point performance drop caused by length (at 30K tokens).
50%UnverifiedIn retrieval tasks, when the filler is masked, accuracy loss at 32K tokens is as high as 45 percentage points; when the same filler remains visible, the loss is zero.
50%UnverifiedThe masked-filler protocol has a flaw: the masking operation itself changes the input distribution and introduces additional interference, rather than length being at work.
50%