What's the Connection Between Model Merging and Knowledge Distillation? Exploring the Bridge Between Parameter Space and Output Distribution

An Underappreciated Research Question

In large language model (LLM) engineering practice, Model Merging and Knowledge Distillation (KD) are typically treated as two separate technical tracks. The former operates directly in parameter space — combining weights from multiple models through some algebraic method; the latter works at the output level — training a student model to mimic a teacher model's output distribution. A Reddit user recently posed a question that cuts to the heart of this divide: do these two approaches share any intrinsic connection?

The original poster's intuition is valuable: since one class of methods operates on the parameter space and the other on output distributions, understanding the bridge between them might offer deeper insight into the nature of LLMs. The question may seem basic, but it actually touches on a long-standing tension in deep learning — the mapping between the geometric structure of weight space and functional behavior.

The Fundamental Differences Between the Two Paradigms

Model Merging: Direct Weight Manipulation

The core assumption behind model merging is that the parameters of different models (typically derived from the same pretrained base) lie within a relatively smooth region amenable to linear interpolation. Common approaches include Weight Averaging, Task Arithmetic, and more sophisticated methods like TIES and DARE. What these all share is this: no training data or gradient backpropagation is required — a new model emerges purely from algebraic operations on weights.

These methods work because of an empirical observation: fine-tuned models tend to reside in the same "loss basin" on the loss surface, so linearly combining their weights doesn't cause catastrophic performance degradation. The strength of model merging lies in its extremely low cost and lack of data requirements, but it demands architectural consistency across models and offers limited fine-grained control over the merged model's behavior.

Task Arithmetic represents a significant recent advance in model merging and deserves special mention. The core idea is: subtract the pretrained base weights from a fine-tuned model's weights to obtain a "task vector" — a vector believed to encode task-specific capabilities. By adding or subtracting multiple task vectors, one can combine or cancel specific capabilities without retraining. For example, adding a "math reasoning vector" to a "multilingual vector" should theoretically yield a model with both capabilities. TIES (Trim, Elect Sign & Merge) and DARE (Drop And REscale) further address parameter conflicts that arise when stacking multiple task vectors — TIES resolves directional conflicts through sign voting, while DARE reduces parameter interference through random sparsification. The shared theoretical premise of all these methods is the linear decomposability of weight space, and whether this premise holds is precisely the central tension connecting model merging to distillation.

Knowledge Distillation: Aligning Output Behavior

Knowledge Distillation (KD) and its variants — such as Output Probability Distillation (OPD) — take a fundamentally different path. Rather than assuming composable weights, distillation transfers a teacher's "knowledge" by training a student model to fit the teacher's soft labels or intermediate representations. Distillation requires a training process, data, and gradient optimization, but crucially does not require the student and teacher to share the same architecture — and this is precisely what enables model compression.

In other words, distillation cares about functional behavioral equivalence, while merging cares about geometric composability in parameter space.

Theoretical Bridges Between the Two

The original poster's insight is that parameter space and output distribution are simply two perspectives on the same model. Theoretically, several threads can connect them.

Mode Connectivity is one key concept. Research has shown that different local minima on a neural network's loss surface are often connected by low-loss paths. If two models are connected in weight space, their output distributions should vary smoothly between them — which creates a potential bridge between "weight manipulation" and "output alignment." Merging is essentially finding a compromise point along this connected path, while distillation forcibly approximates a target distribution in output space.

Another angle worth noting: distillation can be seen as implicit, data-constrained model merging. When you distill a student model using the averaged outputs of multiple teacher models, you are effectively performing a "soft merge" at the output distribution level, while traditional weight averaging performs a "hard merge" at the parameter level. Both pursue similar goals — combining capabilities from multiple models — but operate in different spaces.

Conversely, some recent merging methods (such as Fisher information-weighted merging) have begun borrowing ideas from distillation: rather than simply averaging weights arithmetically, they weight parameters according to their importance to the output distribution — effectively injecting output distribution information back into the parameter-space operations.

The Fisher Information Matrix (FIM) plays a key bridging role here and warrants further explanation. The FIM measures how sensitive a model's output distribution is to its parameters — the higher a parameter's Fisher value, the greater the impact of changing it on the output. Fisher-weighted merging methods (such as Fisher Merging) exploit this property by assigning higher weights to parameters that contribute more to the output distribution, rather than simply averaging all parameters equally. This is essentially encoding "output distribution importance" back into parameter-space merging decisions, implicitly referencing output-level semantics without explicitly running a distillation training pipeline. FIM computation typically requires a small amount of calibration data, which makes it an intermediate form between "completely data-free merging" and "data-driven distillation" — and a concrete entry point for understanding the unity between the two paradigms.

Why This Question Deserves Deeper Investigation

Understanding the unity between these two approaches has direct practical implications. If it can be shown that certain merging operations are equivalent to specific forms of distillation, then in compute-constrained or data-unavailable settings, cheaper merging could substitute for expensive distillation. Conversely, in scenarios requiring precise behavioral control, distillation remains more reliable.

At a deeper level, this question touches on our understanding of LLM internals. The correspondence between the geometric structure of parameter space (which directions matter, which basins are connected) and the semantic behavior of output distributions remains an open problem. Bridging this gap would help explain why model merging sometimes "unexpectedly" preserves or even enhances certain capabilities, and would help design new knowledge transfer methods that balance efficiency with controllability.

Current State and Open Directions

It should be acknowledged honestly: systematic academic research on a "unified framework for model merging and distillation" remains insufficient. Most work either focuses on engineering improvements to merging algorithms or on distillation's compression effectiveness — rarely do papers rigorously compare the two within the same theoretical framework. The original poster's question reflects this gap precisely.

For researchers looking to enter this area, several potential entry points include: quantifying the KL divergence shift that merging operations induce on output distributions, investigating whether distillation objectives can be reformulated as constrained optimization in weight space, and exploring the information-theoretic equivalence conditions between multi-teacher distillation and multi-model merging. These are gaps waiting to be filled, and important pieces of the puzzle for understanding large model behavior.

For quantifying the equivalence between the two approaches from an information-theoretic perspective, KL Divergence (Kullback-Leibler Divergence) is a natural measurement tool. KL divergence measures the difference between two probability distributions and appears directly in distillation training objectives — the student model's output distribution is required to approximate the teacher's distribution by minimizing their KL divergence. In model merging, KL divergence can serve as an evaluation metric, measuring how much the output distribution shifts before and after merging. If a weight averaging operation results in extremely small KL divergence between the merged model and the original models, that operation can be considered approximately lossless at the output level — mathematically comparable to the ideal case in distillation where the student perfectly reproduces the teacher. Systematically measuring the KL divergence changes induced by different merging strategies and comparing them against equivalent distillation objectives is one actionable research path toward integrating both approaches into a unified framework.