Agnes-3.0-Flash
A 33-billion-parameter hybrid-attention multimodal model supporting an extended context window of 262,144 tokens. It employs a 3:1 hybrid attention architecture alternating gated delta-rule recurrent layers with global attention layers, and supports joint understanding of text, images, and video as well as tool calling.
Timeline (last 90 days)
Agnes-3.0-Flash is a multimodal model with 33B parameters that supports joint understanding of text, images, and video
Agnes-3.0-Flash provides a context window of 262,144 tokens
Agnes-3.0-Flash offers adjustable reasoning effort and tool-calling capabilities
Agnes-3.0-Flash uses a hybrid attention decoder in which three out of every four layers run gated delta rule and the fourth layer runs standard global attention
Agnes-3.0-Flash has a total of 72 layers, consisting of 54 delta-rule recurrent layers and 18 global attention layers arranged in a 3:1 alternating pattern, with a hidden dimension of 5120 and a vocabulary size of 248,320
Agnes-3.0-Flash's global attention layers use 6:1 GQA with 24 query heads and 4 KV heads, a head dim of 256, RMS-norm applied to q and k, and sigmoid-gated output
Agnes-3.0-Flash's delta-rule layers are configured with 16 key heads, 48 value heads, head dim 128, a preceding causal convolution with kernel size 4, paired with gated RMS-norm, and recurrent states stored in fp32 precision
Agnes-3.0-Flash's feed-forward network uses SwiGLU with an intermediate dimension of 17408, with a parallel 2048-dimensional SwiGLU branch per layer
Agnes-3.0-Flash's positional encoding uses tri-axis rotation (text, height, width), with interleaved mrope segment ratios of 11:11:10, a base of 1e7, applied only to the first 25% (64 dimensions) of each head dim
Agnes-3.0-Flash is equipped with a standalone vision encoder: 27 layers, hidden dimension 1152, patch size 16, projected to 5120 dimensions after 2×2 spatial merging
3 more timeline events
All Facts (11)
Agnes-3.0-Flash is a multimodal model with 33B parameters that supports joint understanding of text, images, and video
50%UnverifiedAgnes-3.0-Flash provides a context window of 262,144 tokens
50%UnverifiedAgnes-3.0-Flash offers adjustable reasoning effort and tool-calling capabilities
50%UnverifiedAgnes-3.0-Flash uses a hybrid attention decoder in which three out of every four layers run gated delta rule and the fourth layer runs standard global attention
50%UnverifiedAgnes-3.0-Flash has a total of 72 layers, consisting of 54 delta-rule recurrent layers and 18 global attention layers arranged in a 3:1 alternating pattern, with a hidden dimension of 5120 and a vocabulary size of 248,320
50%UnverifiedAgnes-3.0-Flash's global attention layers use 6:1 GQA with 24 query heads and 4 KV heads, a head dim of 256, RMS-norm applied to q and k, and sigmoid-gated output
50%UnverifiedAgnes-3.0-Flash's delta-rule layers are configured with 16 key heads, 48 value heads, head dim 128, a preceding causal convolution with kernel size 4, paired with gated RMS-norm, and recurrent states stored in fp32 precision
50%UnverifiedAgnes-3.0-Flash's feed-forward network uses SwiGLU with an intermediate dimension of 17408, with a parallel 2048-dimensional SwiGLU branch per layer
50%UnverifiedAgnes-3.0-Flash's positional encoding uses tri-axis rotation (text, height, width), with interleaved mrope segment ratios of 11:11:10, a base of 1e7, applied only to the first 25% (64 dimensions) of each head dim
50%UnverifiedAgnes-3.0-Flash is equipped with a standalone vision encoder: 27 layers, hidden dimension 1152, patch size 16, projected to 5120 dimensions after 2×2 spatial merging
50%UnverifiedCompressing the number of layers that bear KV cache overhead to one quarter of the total layer count under ultra-long contexts can significantly reduce VRAM costs for long-context inference, and is an important direction in the architectural evolution of long-context models
50%