Concepthidden states / hidden state
Hidden State
The high-dimensional vector representation computed by each Transformer layer in a large language model for the input token sequence, encoding semantic, syntactic, and even pragmatic information about the tokens in their current context. The final layer's hidden state is typically passed to the output layer and mapped to a probability distribution.
Timeline (last 90 days)
Oct 14
Taking Llama-3.1-8B as an example, each token has a hidden state vector of dimension 4096 at each layer, encoding the semantic, syntactic, and even pragmatic information of that token in the current context.
Unverified50%