Model Merging and Weight Interpolation

Knowledge distillation compressed teacher behavior into a smaller student. Model merging asks a different deployment question: if you already have several useful fine-tuned checkpoints from the same base model, can you combine their weights into one checkpoint without launching another training run?

Your model platform team runs separate assistant checkpoints for code generation, math reasoning, and general chat. All three came from the same base model, but serving them behind a router raises memory and operational costs and complicates rollback. A merge creates a candidate checkpoint without another gradient-training run, but retained behavior remains an evaluation question.

A hard prerequisite for direct tensor interpolation is compatible parameter structure: corresponding tensors must have compatible shapes and meanings. Embedding matrices and language-model heads also need an explicit tokenizer policy. You can't interpolate a 7B checkpoint with a 70B checkpoint, or blindly combine token rows from different vocabularies. Mergekit can construct a union tokenizer and assign fallback embeddings for missing tokens, but that's an explicit output-space choice that still needs evaluation.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} Same-base checkpoints are the conservative starting point for task-vector merging because their parameter coordinates share lineage; compatible shapes alone don't establish merge quality.

Four recipes compare how same-base specialist checkpoints combine, from simple averaging to sparse conflict-aware task-vector merging.

When averaging weights is worth testing

If someone told you to average the internal numbers of two trained neural networks, skepticism is correct. Weight averaging is plausible in some fine-tuning settings because nearby checkpoints can occupy a connected low-loss region in parameter space. It isn't safe merely because both endpoints are good models.

Fine-tuning moves a checkpoint from a shared starting point through weight space. Linear mode connectivity studies whether the straight interpolation path between endpoints crosses a high-loss barrier.^{[2]Reference 2Linear Mode Connectivity and the Lottery Ticket Hypothesis.https://arxiv.org/abs/1912.05671} The Model Soups paper motivates averaging in a specific setting: models fine-tuned from a shared pretrained initialization under different hyperparameters often admit useful averages in its evaluated vision and text-classification experiments.^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482} This is evidence for testing nearby same-lineage candidates, not a proof that differently specialized LLM checkpoints share one basin.

In concrete terms, evaluate points on $\theta(\alpha) = (1-\alpha)\theta_A + \alpha\theta_B$ rather than assuming the midpoint is usable. Two checkpoints can each be strong on their own evaluation and still interfere when combined, especially when they represent different tasks. Models with unrelated pretraining lineages are even poorer candidates for direct interpolation because their parameter coordinate systems weren't preserved by a common base.

The screening logic below treats interpolation measurements as evidence, rather than granting the midpoint a pass because the sources share lineage. In a real run, losses and task_scores come from the candidate checkpoints and held-out evaluations:

Same-base lineage: Same-base lineage makes a merge experiment defensible. It doesn't establish that a code-generation and math-reasoning merge retained either behavior. Only per-task evaluation does that.

The permutation invariance problem

Neural networks can exhibit permutation invariance: under appropriate corresponding reordering of incoming and outgoing weights, hidden-unit permutations can preserve the function a network computes.^{[4]Reference 4The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks.https://arxiv.org/abs/2110.06296} This helps explain why independently trained networks may be poorly aligned for naive averaging even if they solve similar tasks.

Git Re-Basin^{[5]Reference 5Git Re-Basin: Merging Models modulo Permutation Symmetries.https://arxiv.org/abs/2209.04836} addresses this by finding the optimal permutation $\pi$ that aligns the neurons of one model to match the other before merging:

$\theta_{\text{merged}} = \frac{1}{2}(\theta_A + \pi(\theta_B))$

Permutation invariance makes the failure mode concrete in the next figure. Two checkpoints can learn the same three hidden features but store them in different slot orders, so Git Re-Basin permutes one model before averaging.

Git Re-Basin uses permutation matching and reports merged independently trained MLP, CNN, and ResNet models in its studied settings, including a zero-barrier ResNet result on CIFAR-10.^{[5]Reference 5Git Re-Basin: Merging Models modulo Permutation Symmetries.https://arxiv.org/abs/2209.04836} That paper is a useful alignment concept, but it isn't evidence that an arbitrary pair of large language models can be repaired and merged. For LLM work, same-base candidates plus downstream evaluation remain the practical default here.

Why does Git Re-Basin apply $\pi$ before averaging?

Answer

Two models can represent the same function with neurons in different orders. The permutation aligns corresponding neurons first, so averaging combines matching features instead of unrelated ones.

The merging toolkit: from simple averages to conflict-aware methods

Several techniques exist for merging models, ranging from mathematical averages to geometric interpolation rules. Choice depends on source lineage, observed delta conflict, and the evaluations the output must pass.

Model soups (uniform averaging)

The simplest approach to merging is called Model Soups^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482}. It averages the weights of multiple fine-tuned models to create a single averaged model:

$\theta_{\text{merged}} = \frac{1}{N} \sum_{i=1}^{N} \theta_i$

To see the operation, imagine two scalars instead of billion-parameter tensors. If checkpoint A has a weight 2.0 and checkpoint B has 4.0, their equal average is 3.0. Whether that compromise retains behavior can't be determined from this parameter alone. Model Soups evaluates averaging models fine-tuned from a shared initialization over hyperparameter configurations and reports improved accuracy and robustness in its studied settings without the inference cost of an ensemble.^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482}

Uniform vs. greedy soups

The original Model Soups paper distinguishes between two selection strategies:

• Uniform Soups: Average all fine-tuned checkpoints with equal weight. Simple but risky, as a single poorly-performing checkpoint can drag down the overall quality.
• Greedy Soups: Iteratively add checkpoints to the soup only if they improve performance on a held-out validation set. Start with the best individual model, then test each remaining model: if adding it to the average improves the validation metric, keep it; otherwise, discard it.

In the original paper, greedy soups outperform uniform averaging in the reported experiments because the selection rule skips candidates that hurt its held-out validation metric.^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482} A similar selection rule is reasonable to test only when your validation slices represent the behavior you need to keep.

uniform_merge performs uniform averaging. It takes a list of model state dictionaries and an optional list of weights, and returns a single state dictionary containing the weighted average of their parameters:

Pros

Exceptionally simple to implement and requires no hyperparameters beyond the optional weighting scheme.

Cons

If the models are fine-tuned on wildly different tasks, direct averaging can cause destructive interference between conflicting parameters, degrading overall performance.

Task arithmetic

Instead of averaging absolute weights, Task Arithmetic^{[6]Reference 6Editing Models with Task Arithmetichttps://arxiv.org/abs/2212.04089} operates on task vectors (the difference between fine-tuned weights and the base model). Task vectors are specific directions from the base checkpoint (e.g., "walk 10 steps north"). Instead of averaging the final destinations of different hikers, you take the specific path each hiker took from base camp and combine them. By scaling these paths (e.g., "take half the steps north"), you can carefully mix different skills.

$\tau_i = \theta_i - \theta_0 \quad \text{(task vector for model } i \text{)}$

$\theta_{\text{merged}} = \theta_0 + \sum_{i=1}^{N} \lambda_i \cdot \tau_i$

Where $\theta_0$ is the base model, $\theta_i$ are fine-tuned models, $\tau_i$ are per-task deltas, and $\lambda_i$ are merge weights.

A concrete scalar walkthrough

Before you run this on billion-parameter tensors, try it on a single parameter. Imagine a base model where one weight is 2.0. A code-generation fine-tune pushes that weight to 2.5, so its task vector is +0.5. A math-reasoning fine-tune pushes it to 1.5, so its task vector is -0.5. If you want to favor the code delta without discarding the math delta, you might set $\lambda_{\text{code}} = 0.6$ and $\lambda_{\text{math}} = 0.4$:

$\text{merged weight} = 2.0 + 0.6(0.5) + 0.4(-0.5) = 2.0 + 0.3 - 0.2 = 2.1$

The result, 2.1, is a parameter nudge in the code-generation direction. If you had instead averaged the absolute fine-tuned weights (2.5 and 1.5), you'd get 2.0, cancelling both deltas at this coordinate. A scalar calculation can't establish how either capability performs; it only shows how task-vector coefficients act on weights.

Why did absolute averaging cancel the two specialist deltas in the scalar example?

Answer

The code-generation fine-tune moved this coordinate up by +0.5 and the math-reasoning fine-tune moved it down by -0.5. Averaging final weights lands this coordinate back at the base value, cancelling both deltas here. Task evaluation determines the broader behavioral effect.

A different coefficient pair shows the same tuning choice. Suppose a code-generation fine-tune nudges a weight to 2.4 (task vector +0.4) and a math-reasoning fine-tune nudges it to 1.8 (task vector -0.2). Setting $\lambda_{\text{code}} = 0.7$ and $\lambda_{\text{math}} = 0.5$ gives 2.0 + 0.28 - 0.10 = 2.18. Whether this coefficient pair keeps either capability is measured after the merge.

task_arithmetic_merge implements the recipe in code. It takes a shared base model, a list of fine-tuned models, and scaling coefficients. It subtracts the base to get task vectors, applies a scaling coefficient, and adds them back to return the merged model:

Advantage

Scaling coefficients control how much of each task vector enters the candidate checkpoint; evaluation determines retained behavior.

TIES-Merging (trim, elect sign, merge)

A major issue with simple averaging is that task vectors can directly contradict one another (e.g., one model increases a weight by 0.5, while another decreases it by 0.5). TIES-Merging^{[7]Reference 7TIES-Merging: Resolving Interference When Merging Modelshttps://arxiv.org/abs/2306.01708} addresses two forms of interference in its recipe: it trims low-magnitude deltas, elects a sign for each coordinate using total signed movement, and merges only values aligned with that elected sign.

TIES resolves a coordinate as if three specialists proposed parameter deltas for it. First, discard small proposals (Trim). Next, sum the size of increases against the size of decreases, rather than counting voters (Elect sign). Finally, average only proposals in the winning direction (Disjoint merge).

Step 1: Trim

First, it drops low-magnitude changes treated as redundant by the recipe. The trim function below takes a task vector and a density threshold, then keeps high-magnitude updates:

Step 2: Elect sign

When multiple task vectors update the same parameter, TIES-Merging resolves conflicting directions with $\gamma_m^p = \mathrm{sgn}(\sum_t \hat{\tau}_t^p)$.^{[7]Reference 7TIES-Merging: Resolving Interference When Merging Modelshttps://arxiv.org/abs/2306.01708} The elect_sign function sums the trimmed signed deltas and takes the resulting sign. This differs from a majority vote: one large update can outweigh two smaller opposing updates.

Step 3: Disjoint merge

Finally, it computes the average of only those task vectors whose updates match the consensus direction. The disjoint_merge function takes trimmed task vectors and their elected signs. It averages only the parameter updates that agree with the elected direction, zeroing out any dissenting values to produce the final merged task vector:

TIES-Merging outperforms compared baselines in the paper's evaluated vision and T5 task-vector settings, and its analysis identifies sign interference.^{[7]Reference 7TIES-Merging: Resolving Interference When Merging Modelshttps://arxiv.org/abs/2306.01708} For an LLM merge, use it as a candidate recipe when task vectors conflict, then compare per-task evaluation against simpler baselines.

DARE (drop and rescale)^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099}

DARE randomly drops entries from each task vector and rescales the survivors before a downstream merge. The rescaling preserves an entry's expected delta under the random mask; it doesn't by itself prove that the resulting model preserves a capability.

The paper studies redundancy in supervised fine-tuning (SFT) deltas and reports that its evaluated models can often tolerate dropping 90% of delta entries, and in some cases 99%, before merging. Its size ablation reports that WizardMath-70B remains effective at a 99% drop rate while the evaluated 7B and 13B variants fail there.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099} Treat that as experimental evidence for the paper's SFT models, not a default density setting for a new merge. DARE is a sparsification step, not a complete merge recipe: sparsified task vectors still need a merger such as averaging or TIES.

DARE's analysis also separates SFT deltas, which it observes are typically within roughly 0.002, from continued-pretraining deltas that approach 0.03; its drop-and-rescale approach becomes ineffective on the latter.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099} One candidate pipeline is DARE-TIES: DARE sparsifies each delta, then TIES resolves directional conflicts among surviving values. Mergekit exposes that composition as dare_ties.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}

$\tau_i^{\text{DARE}} = \frac{\tau_i \odot m}{1 - p}$

where $m$ is a random binary mask with drop rate $p$ and the $1/(1-p)$ factor preserves each entry's expectation under masking.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099}

Why does DARE divide by $1-p$ after dropping weights?

Answer

Only a fraction $1-p$ of task-vector entries survive. Dividing by $1-p$ keeps each randomly masked delta entry unbiased in expectation; retained task quality still requires evaluation.

dare_sparsify implements the DARE step itself. It sparsifies one task vector, after which you can pass the result to a downstream merge rule:

What the paper establishes

On the SFT models and tasks it evaluates, DARE finds substantial redundancy in task-vector entries and improves several downstream merge methods after drop-and-rescale preprocessing.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099} For a new checkpoint family, density remains a tuned parameter: compare unsparsified and DARE-preprocessed merges on every required task slice.

SLERP (spherical linear interpolation)

SLERP is a geometric interpolation rule. It follows an angular arc between normalized vector directions rather than the chord used by linear interpolation. It isn't an algorithm for discovering a low-loss path around an incompatible-model ridge.

Rather than interpolating directions along a straight line, SLERP^{[9]Reference 9Animating Rotation with Quaternion Curveshttps://dl.acm.org/doi/10.1145/325334.325242} interpolates along a sphere. Write the geometry in terms of normalized directions:

$$\begin{aligned} \text{SLERP}(\hat{\theta}_A, \hat{\theta}_B, t) &= \frac{\sin((1-t)\Omega)}{\sin \Omega} \hat{\theta}_A \\ &\quad + \frac{\sin(t\Omega)}{\sin \Omega} \hat{\theta}_B \end{aligned}$$

where $t$ is the interpolation factor (from 0 to 1) and $\Omega = \arccos(\hat{\theta}_A \cdot \hat{\theta}_B)$ is the angle between normalized weight vectors. Practical merge implementations may handle magnitude separately after interpolating direction, as the code does here.

SLERP was introduced for computer graphics to interpolate rotations represented as quaternions.^{[9]Reference 9Animating Rotation with Quaternion Curveshttps://dl.acm.org/doi/10.1145/325334.325242} That source establishes its geometry, not downstream quality for neural-network weight merges. Use a SLERP checkpoint as another candidate and measure loss and required tasks just as you would for a linear merge.

The slerp function performs this geometric interpolation. It takes two unnormalized vectors and an interpolation factor t, projects them onto a unit sphere, computes the interpolation, and returns the combined vector:

What it guarantees geometrically

In the equal-norm orthogonal-vector example above, the SLERP midpoint remains on the unit circle while a linear midpoint would have smaller norm. That's a geometric property of this example, not evidence that the midpoint preserves either model's behavior. For a pair of checkpoints, evaluate linear and spherical candidates against the same release gates.

Running a merge with mergekit

mergekit^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} is an open-source toolkit for merging language-model checkpoints. Its documented CLI supports YAML-defined merges with CPU or limited-VRAM execution, and mergekit-multi can run multi-stage recipes where later merges consume earlier outputs.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}

To use mergekit, you typically define a YAML configuration file that specifies the base model, the fine-tuned source models, their respective merging coefficients, and the desired algorithm. This configuration acts as the input to the CLI tool to generate the merged model:

For dare_ties, each source model's density controls the retained fraction of that source's task vector. Merge methods have different schemas, so don't treat merge_method as a drop-in switch: for example, mergekit's slerp method takes exactly two source models.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} If all required tokens are already in the base tokenizer, tokenizer.source: base pins the output vocabulary to that base. Modern mergekit configuration defaults to a union tokenizer, which adds tokens present in source vocabularies and assigns fallback embeddings where an input model lacks them.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} Either policy is an output-space decision that needs targeted evaluation.

Common Mistake: Assuming all specialist checkpoints use the same tokenizer because they began from the same base. If the code checkpoint added fill-in-the-middle tokens such as <fim_prefix>, choosing base drops those added output entries while choosing union introduces filled embeddings for models that lack them. Verify tokenizer vocabularies, choose the output policy explicitly, and test code-completion prompts that require those tokens.

This preflight check makes the output-vocabulary decision explicit before any expensive merge runs:

Once the configuration is set, you can execute the merge using the CLI tool. The command below takes the YAML configuration file and the output path, producing the final merged model on disk:

For hardware-specific and memory-saving flags, check mergekit-yaml --help because supported options vary by version.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}

Mergekit lets engineers build candidate checkpoints without a new gradient-training run. Iterate over parameters such as retained density and task weights only against a defined evaluation suite.

Run an iterative validation loop after every merge: evaluate each target slice, compare it to declared gates and source baselines, and retune or reject any candidate that misses a critical threshold.

An aggregate score is useful for ranking candidates, but a release decision should fail on any critical threshold miss:

Finding good coefficients automatically

Manually searching merge coefficients and layer selections can be expensive. Evolutionary model merging^{[10]Reference 10Evolutionary Optimization of Model Merging Recipes.https://sakana.ai/evolutionary-model-merge/} applies evolutionary search to optimize merge recipes against a supplied fitness evaluation. Its search space can include per-layer source choices, interpolation weights, and whether to merge in parameter space, data-flow space, or both.

Sakana AI reports using this approach to build EvoLLM-JP from Japanese-language and math-oriented models, optimizing for the evaluations selected in that work.^{[10]Reference 10Evolutionary Optimization of Model Merging Recipes.https://sakana.ai/evolutionary-model-merge/} The principle is straightforward: treat each merge configuration as a "genome," score it on validation data, mutate its parameters, and retain better-scoring candidates. The resulting model inherits the objective's coverage and blind spots, so release gates still need independent slices.

In practice, a small grid search is a transparent baseline: try lambda values in [0.2, 0.4, 0.6, 0.8] for each task vector, measure each required slice on held-out data, and retain only configurations that clear all release gates. For a code/math/chat merge, those slices should include code-generation tests, math word problems, and instruction-following prompts; a code-only objective doesn't protect math behavior.

Passthrough and frankenmerging

Not every merge averages weights. Mergekit's passthrough method copies selected tensors or layer ranges from source models into the output checkpoint.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} This is model splicing rather than interpolation: tensor dimensions must compose, while useful behavior across the splice remains an evaluation result.

For example, a recipe might copy layers 0-19 from checkpoint A and layers 20-31 from checkpoint B. Compatible dimensions make that artifact constructible; they don't show that B's later layers can interpret the hidden states produced by A's earlier layers.

Use passthrough as an experiment with explicit source lineage, layer boundaries, and the same per-task gates used for any other candidate. A successful build only proves shape compatibility.

Merging pitfalls and hard limits

Direct weight merging requires aligned parameter meaning, shape, and output-space assumptions. You can't directly interpolate a 7-billion-parameter checkpoint with a 70-billion-parameter checkpoint, or silently combine incompatible vocabulary mappings and treat token IDs as equivalent. A generated merged artifact is a candidate, not evidence of retained quality.

Avoid direct interpolation when:

1. Different architectures: Source models must share the exact same structural architecture, parameter count, and layer layout. You can't directly interpolate a MiniMax model with a Mistral model, nor an 8B model with a 72B model.
2. Different tokenizers: Direct interpolation assumes aligned embeddings and next-token heads. Mergekit can help with tokenizer union in compatible cases, but it doesn't make arbitrary vocabulary mismatches disappear.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}
3. Mismatched prompt formats or chat templates: These usually won't block the raw tensor merge, but they can make the merged checkpoint look broken at inference time because the prompt serialization no longer matches the behaviors learned during tuning.
4. Quantized-only checkpoints: If you want to merge, do it on dequantized or full-precision weights first, then quantize the final artifact. The merge math depends on real-valued deltas, not already-rounded integers. Treat the merged checkpoint as a new model and re-run your quantization calibration and evals.
5. Distant fine-tuning trajectories: Models fine-tuned on divergent data or with large update magnitudes may be poor interpolation candidates. Git Re-Basin demonstrates permutation alignment in studied MLP, CNN, and ResNet settings; it isn't an established repair step for arbitrary LLM merges.^{[5]Reference 5Git Re-Basin: Merging Models modulo Permutation Symmetries.https://arxiv.org/abs/2209.04836}
6. Critical precision tasks: Don't ship a merge when a required slice misses its threshold, even if an aggregate metric rises. A specialist baseline remains part of the comparison.

Diagnosing a broken merge

When a merge goes wrong, the model usually tells you quickly. Three common failure patterns, their causes, and their fixes are:

Don't predict quality from the merge rule alone. Model Soups^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482} reported gains over the best individual checkpoint in its evaluated shared-initialization experiments, including ImageNet settings. That result motivates trying a merge; it doesn't predict whether a new LLM merge will clear its specialist task gates.

After merging, run a full evaluation suite across all target tasks, not the aggregate score alone. Strong code-generation accuracy can mask a miss in math reasoning. Track per-task metrics independently and compare against declared thresholds and each source model's baseline.