BDH
An architecture proposed by Pathway in which a graph structure of neurons allows attention to emerge from local interactions. Its GPU implementation is equivalent to a low-rank ReLU transformation combined with linear attention, connecting the biologically inspired design to efficient attention computation.
Timeline (last 90 days)
BDH (Dragon Hatchling) combines linear attention in high-dimensional neural space with a low-rank GPU implementation, where its recurrent attention state is an N×D matrix (N much greater than D) rather than an N×N connectivity matrix.
In the BDH architecture, correlated neural activity produces Hebbian-like updates to connections, giving working memory a synaptic-level interpretation.
BDH's state does not persist into model parameters after sequence inference concludes, which is fundamentally different from long-term potentiation in biological synapses.
All Facts (3)
BDH (Dragon Hatchling) combines linear attention in high-dimensional neural space with a low-rank GPU implementation, where its recurrent attention state is an N×D matrix (N much greater than D) rather than an N×N connectivity matrix.
50%UnverifiedIn the BDH architecture, correlated neural activity produces Hebbian-like updates to connections, giving working memory a synaptic-level interpretation.
50%UnverifiedBDH's state does not persist into model parameters after sequence inference concludes, which is fundamentally different from long-term potentiation in biological synapses.
50%