Head-Level Score Modulation
A method for injecting structural information in multi-head attention that assigns independent modulation parameters to each attention head, applying distinct transformations to the scores so that different heads attend to different types of edge relations or subgraph patterns.
Timeline (last 90 days)
A study titled "Efficient Edge-aware Attention Network for Graph Generation" proposes an edge-aware attention layer that injects structural information through head-level score modulation and updates edge representations with a lightweight residual mechanism.
Head-level score modulation means assigning independent modulation parameters or functions to each attention head in multi-head attention, applying different transformations to the attention scores rather than having all heads share a single structural injection scheme.
Compared with adding a uniform edge bias to the scores, head-level score modulation allows different heads to attend to different types of edge relationships or subgraph patterns, enhancing the model's ability to express heterogeneous structural information and being naturally compatible with the block-wise computation flow of standard attention kernels.
All Facts (3)
A study titled "Efficient Edge-aware Attention Network for Graph Generation" proposes an edge-aware attention layer that injects structural information through head-level score modulation and updates edge representations with a lightweight residual mechanism.
50%UnverifiedHead-level score modulation means assigning independent modulation parameters or functions to each attention head in multi-head attention, applying different transformations to the attention scores rather than having all heads share a single structural injection scheme.
50%UnverifiedCompared with adding a uniform edge bias to the scores, head-level score modulation allows different heads to attend to different types of edge relationships or subgraph patterns, enhancing the model's ability to express heterogeneous structural information and being naturally compatible with the block-wise computation flow of standard attention kernels.
50%