Causal Mask
A masking mechanism in autoregressive language models that restricts the scope of attention, ensuring each position can attend only to itself and preceding tokens, thereby preventing information leakage.
Timeline (last 90 days)
In autoregressive language models, attention needs to be paired with a causal mask to ensure that each position can only attend to itself and preceding tokens, preventing information leakage.
When packing multiple questions, the causal mask needs to be modified: each question can see the full context, but the questions must be isolated from one another; if question two can see the content of question one, the order of asking affects the answers and reproducibility is broken. The author points out that Cloudflare's Clef does not do this isolation, which in his view would cause output drift after reordering.
The device-side causal mask is generated directly on the accelerator or applied as a kernel, eliminating the transfer and memory overhead of constructing it on the host and then copying it to the device.
In a decoder-only architecture, the causal mask ensures each token can only see itself and earlier tokens.
When predicting the Token at position t, a causal language model can only see the input at positions 1 to t, and cannot see future Tokens, which is usually implemented via a causal mask.
Normalizing flows share with large language models the same causal mask, KV-cache mechanism, and a consistent left-to-right sequence generation structure.
All Facts (6)
In autoregressive language models, attention needs to be paired with a causal mask to ensure that each position can only attend to itself and preceding tokens, preventing information leakage.
50%UnverifiedWhen packing multiple questions, the causal mask needs to be modified: each question can see the full context, but the questions must be isolated from one another; if question two can see the content of question one, the order of asking affects the answers and reproducibility is broken. The author points out that Cloudflare's Clef does not do this isolation, which in his view would cause output drift after reordering.
50%UnverifiedThe device-side causal mask is generated directly on the accelerator or applied as a kernel, eliminating the transfer and memory overhead of constructing it on the host and then copying it to the device.
50%UnverifiedIn a decoder-only architecture, the causal mask ensures each token can only see itself and earlier tokens.
50%UnverifiedWhen predicting the Token at position t, a causal language model can only see the input at positions 1 to t, and cannot see future Tokens, which is usually implemented via a causal mask.
50%UnverifiedNormalizing flows share with large language models the same causal mask, KV-cache mechanism, and a consistent left-to-right sequence generation structure.
50%