Single-loop Tax
A systematic performance degradation observed in recurrent Transformers caused by a mismatch between the number of loops during training and during inference. Its root cause is that intermediate representations do not fully converge under the parameter-sharing architecture.
Timeline (last 90 days)
The "single-loop tax" phenomenon was reproduced in a small recurrent Transformer setting, namely that the single-loop version has a systematic performance penalty relative to the multi-loop version.
The single-loop tax may stem from a representation bias caused by the mismatch between computational depth at training and at inference in recurrent architectures.
When reporting the performance of recurrent architectures, comparative results for single-loop and multi-loop must be given together to avoid the single-loop tax interfering with conclusions.
All Facts (3)
The "single-loop tax" phenomenon was reproduced in a small recurrent Transformer setting, namely that the single-loop version has a systematic performance penalty relative to the multi-loop version.
50%UnverifiedThe single-loop tax may stem from a representation bias caused by the mismatch between computational depth at training and at inference in recurrent architectures.
50%UnverifiedWhen reporting the performance of recurrent architectures, comparative results for single-loop and multi-loop must be given together to avoid the single-loop tax interfering with conclusions.
50%