Feedforward Network
A neural network structure that transmits information in one direction only and has no feedback loops. It can be fully decomposed into independent paths and has low causal integration.
Timeline (last 90 days)
The Component Interaction Consistency (CIC) signal proposed by Circe measures the degree of alignment between the direction of attention updates in early layers and the direction of attention-induced FFN output changes in later layers.
In the Transformer architecture, the input of each token is processed through multiple layers of self-attention and feed-forward networks, and each layer produces a high-dimensional vector representation called a "hidden state"; the hidden states of all layers and all tokens together constitute the "activation space."
All Facts (2)
The Component Interaction Consistency (CIC) signal proposed by Circe measures the degree of alignment between the direction of attention updates in early layers and the direction of attention-induced FFN output changes in later layers.
50%UnverifiedIn the Transformer architecture, the input of each token is processed through multiple layers of self-attention and feed-forward networks, and each layer produces a high-dimensional vector representation called a "hidden state"; the hidden states of all layers and all tokens together constitute the "activation space."
50%