d_model
The vector dimension representing each token in a Transformer model, and the hidden-layer width shared by all layers within the model, determining the attention projection size and the feed-forward network expansion ratio.
Timeline (last 90 days)
In the Transformer encoder, the input matrix has shape (tokens × d_model), and the output after each layer has exactly the same shape.
d_model is the vector dimension representing each token in the Transformer, and also the hidden layer width shared by all layers inside the model.
In the paper "Attention Is All You Need," the base model uses d_model = 512.
A sequence of 6 tokens fed into an encoder with d_model = 512 forms a 6×512 matrix.
Each token is converted into a d_model-dimensional vector by looking up the Embedding Table, and N tokens form an N × d_model input matrix.
All Facts (5)
In the Transformer encoder, the input matrix has shape (tokens × d_model), and the output after each layer has exactly the same shape.
50%Unverifiedd_model is the vector dimension representing each token in the Transformer, and also the hidden layer width shared by all layers inside the model.
50%UnverifiedIn the paper "Attention Is All You Need," the base model uses d_model = 512.
50%UnverifiedA sequence of 6 tokens fed into an encoder with d_model = 512 forms a 6×512 matrix.
50%UnverifiedEach token is converted into a d_model-dimensional vector by looking up the Embedding Table, and N tokens form an N × d_model input matrix.
50%