Visual Tokens
The basic units through which vision-language models process images. An image is divided into patches, which are mapped to feature vectors by a visual encoder and participate alongside text tokens in self-attention computation.
Timeline (last 90 days)
In CUAs, each screenshot is represented as thousands of visual tokens, which accumulate with the interaction history to form an extremely long context, and most visual tokens are either semantically irrelevant to the next action or temporally redundant across consecutive screenshots.
In language models, visual tokens are concatenated with text tokens and jointly participate in self-attention computation, whose computational overhead grows quadratically with the number of tokens.
Images are divided into fixed-size patches, converted into visual tokens by the visual encoder, then mapped to the same embedding space as text tokens through a projection layer, and fed together with text tokens into the language model for joint inference.
All Facts (3)
In CUAs, each screenshot is represented as thousands of visual tokens, which accumulate with the interaction history to form an extremely long context, and most visual tokens are either semantically irrelevant to the next action or temporally redundant across consecutive screenshots.
50%UnverifiedIn language models, visual tokens are concatenated with text tokens and jointly participate in self-attention computation, whose computational overhead grows quadratically with the number of tokens.
50%UnverifiedImages are divided into fixed-size patches, converted into visual tokens by the visual encoder, then mapped to the same embedding space as text tokens through a projection layer, and fed together with text tokens into the language model for joint inference.
50%