[KongchangAI]
Conceptone-loop tax

Single-loop Tax

A systematic performance degradation observed in recurrent Transformers caused by a mismatch between the number of loops during training and during inference. Its root cause is that intermediate representations do not fully converge under the parameter-sharing architecture.

Timeline (last 90 days)

Nov 13

The "single-loop tax" phenomenon was reproduced in a small recurrent Transformer setting, namely that the single-loop version has a systematic performance penalty relative to the multi-loop version.

Unverified50%
Nov 13

The single-loop tax may stem from a representation bias caused by the mismatch between computational depth at training and at inference in recurrent architectures.

Unverified50%
Nov 13

When reporting the performance of recurrent architectures, comparative results for single-loop and multi-loop must be given together to avoid the single-loop tax interfering with conclusions.

Unverified50%

All Facts (3)

Source Articles