Git Re-Basin
Git Re-Basin is a neural network model merging method proposed by Ainsworth et al. in 2022. It uses the Hungarian algorithm or stochastic gradient descent to find neuron permutation matrices, aligning independently trained isomorphic networks to a shared coordinate system before interpolation. The approach substantially improves upon naive weight averaging.
Core Facts
Timeline (last 90 days)
When MAEM reaches a fixed capacity limit, it merges the least-used experts via the Git Re-Basin alignment technique, and distills the merged experts into a single unified fused expert using topology-aware pseudo-data synthesized from the MAE distributions.
Git Re-Basin alignment incurs additional computational cost when the number of experts is large, which is an engineering issue that needs to be considered in actual deployment.
Git Re-Basin remaps the neuron ordering of one model to the corresponding positions of another model by solving for a permutation matrix, which can significantly reduce the distance between the two models on the loss surface.
Git Re-Basin, proposed by Ainsworth et al. in 2022, is a representative work in permutation alignment methods.
Git Re-Basin is less effective on heterogeneous tasks with large distributional differences than in same-task fine-tuning scenarios.
Git Re-Basin uses the Hungarian algorithm or stochastic gradient descent to find the neuron permutation matrix that makes network B's activation distribution most closely match that of network A.
After neuron alignment, the loss landscape between two networks often exhibits a linear pattern, allowing interpolated models to maintain considerable performance, whereas naive averaging typically falls into high-loss barrier regions.
All Facts (7)
After neuron alignment, the loss landscape between two networks often exhibits a linear pattern, allowing interpolated models to maintain considerable performance, whereas naive averaging typically falls into high-loss barrier regions.
65%UnverifiedWhen MAEM reaches a fixed capacity limit, it merges the least-used experts via the Git Re-Basin alignment technique, and distills the merged experts into a single unified fused expert using topology-aware pseudo-data synthesized from the MAE distributions.
50%UnverifiedGit Re-Basin alignment incurs additional computational cost when the number of experts is large, which is an engineering issue that needs to be considered in actual deployment.
50%UnverifiedGit Re-Basin remaps the neuron ordering of one model to the corresponding positions of another model by solving for a permutation matrix, which can significantly reduce the distance between the two models on the loss surface.
50%UnverifiedGit Re-Basin is less effective on heterogeneous tasks with large distributional differences than in same-task fine-tuning scenarios.
50%UnverifiedGit Re-Basin uses the Hungarian algorithm or stochastic gradient descent to find the neuron permutation matrix that makes network B's activation distribution most closely match that of network A.
50%UnverifiedGit Re-Basin, proposed by Ainsworth et al. in 2022, is a representative work in permutation alignment methods.
50%