The delta-rule layers are recurrent, with each layer's state being independent of sequence length, thus eliminating the need for a KV cache that grows with context