ConceptHyper-Connections
Hyper-Connections
A technique for improving the residual pathways of Transformers, introducing multiple parallel residual streams or learnable connection weights to enhance the expressive capacity of information flow in the vertical direction.
Timeline (last 90 days)
Nov 13
Recursive used a model ladder starting at about 400 million parameters to validate architectural changes; the hyper-connections variant worked well at small scale but failed completely above 4 billion parameters, so it was abandoned.
Unverified50%