Model Merging and Weight Interpolation

Distillation trains a smaller student from teacher signal. Model merging starts later: you already have useful fine-tuned checkpoints and want one checkpoint without another gradient-based training run.

Picture your platform team serving three same-base 8B assistants: Code, Math, and Chat. Routing among them costs extra memory and makes rollback messier. The first merge writes successfully and reports an aggregate score of 87.2. Then the slice report arrives: code 92.4, math 82.1 against an 85.0 gate, and chat 87.0. Keeping one model would simplify routing, but this candidate stays blocked. The failed Math slice is the useful starting point: a file on disk proves construction, not retained behavior.

Before choosing a recipe, check whether coordinates still mean the same thing. If token ID 417 means return in one tokenizer but <fim_prefix> in another, averaging row 417 blends unrelated output symbols. Direct tensor interpolation needs compatible shapes and meanings, and embedding matrices plus language-model heads need an explicit tokenizer policy. Mergekit can build a union tokenizer and assign fallback embeddings for missing tokens, but that's an output-space choice that still needs evaluation.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} Same-base checkpoints are the conservative starting point because their parameter coordinates share lineage. Matching shapes alone doesn't make a merge good.

Keep Code, Math, and Chat as our running trio. We'll compare several recipes, from a plain average to sparse, conflict-aware task-vector merging. First ask whether a straight path through weight space is even usable.

When averaging weights is worth testing

If someone told you to average the internal numbers of two trained neural networks, skepticism is correct. The useful question is narrower: do these nearby checkpoints share a low-loss region? Weight averaging is plausible in some fine-tuning settings, but two good endpoints don't certify the points between them.

Fine-tuning moves a checkpoint from a shared starting point through weight space. Linear mode connectivity asks whether the straight path between endpoints crosses a high-loss barrier.^{[2]Reference 2Linear Mode Connectivity and the Lottery Ticket Hypothesis.https://arxiv.org/abs/1912.05671} Model Soups found useful averages in its evaluated vision and text-classification settings when models shared a pretrained initialization and differed in hyperparameters.^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482} That supports testing nearby same-lineage candidates. It doesn't show that Code and Math share one basin.

Evaluate points on $\theta(\alpha) = (1-\alpha)\theta_A + \alpha\theta_B$ rather than assuming the midpoint is usable. Code and Math can each pass alone and still interfere along the path. Models with unrelated pretraining lineages are even poorer candidates because no common base preserved their parameter coordinates.

Now make the first prediction. If the Code and Math endpoints pass, does $\alpha=0.50$ pass too? The screen below treats interpolation measurements as evidence. In a real run, losses and task_scores come from held-out evaluations; these values are a worked Code-to-Math path whose midpoint misses both gates.

Same-base lineage: A shared base makes Code and Math worth testing together. It doesn't establish that either behavior survives. Per-task evaluation decides that.

The permutation invariance problem

Why can two networks that solve the same task disagree at nearly every coordinate? Neural networks can exhibit permutation invariance: under corresponding reordering of incoming and outgoing weights, hidden-unit permutations can preserve the function a network computes.^{[4]Reference 4The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks.https://arxiv.org/abs/2110.06296} A slot number is an address, not a feature identity, so independently trained networks may be poorly aligned for naive averaging.

Git Re-Basin^{[5]Reference 5Git Re-Basin: Merging Models modulo Permutation Symmetries.https://arxiv.org/abs/2209.04836} searches for a permutation $\pi$ that aligns the hidden units of one model to the other before merging:

$\theta_{\text{merged}} = \frac{1}{2}(\theta_A + \pi(\theta_B))$

Use a tiny three-feature example to see the failure. Two checkpoints learn A, B, and C but store them in different slot orders. Here $\theta_B$ stores B, C, A. Git Re-Basin searches for a permutation $\pi$ that restores A, B, C before averaging.

Git Re-Basin uses permutation matching and reports merged independently trained MLP, CNN, and ResNet models in its studied settings, including zero-barrier linear mode connectivity for sufficiently wide ResNets on CIFAR-10.^{[5]Reference 5Git Re-Basin: Merging Models modulo Permutation Symmetries.https://arxiv.org/abs/2209.04836} Alignment fixes one kind of coordinate mismatch. It isn't evidence that an arbitrary pair of large language models can be repaired and merged, so same-base candidates plus downstream evaluation remain our LLM default.

Why does Git Re-Basin apply $\pi$ before averaging?

Answer

Two models can represent the same function with neurons in different orders. The permutation aligns corresponding neurons first, so averaging combines matching features instead of unrelated ones.

From simple averages to conflict-aware methods

The first screen asks whether coordinates line up. The next asks what to do when they do: average endpoints, add deltas, filter conflicts, or change interpolation geometry. Choose against source lineage, observed delta conflict, and the evaluations the output must pass.

Model soups (uniform averaging)

Start with the deliberately boring control: average the endpoint weights and see what behavior you lose. This is Model Soups^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482}, applied to multiple fine-tuned models:

$\theta_{\text{merged}} = \frac{1}{N} \sum_{i=1}^{N} \theta_i$

Predict the tiny example before reading the code. If checkpoint A has a weight 2.0 and checkpoint B has 4.0, equal averaging returns 3.0. The dictionaries below stand in for tensor slices, so the arithmetic is visible without loading a checkpoint. A single coordinate still can't tell you whether Code, Math, or Chat behavior survived.

Model Soups evaluates models fine-tuned from a shared initialization over hyperparameter configurations and reports improved accuracy and robustness in its studied settings without the inference cost of an ensemble.^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482} That evidence motivates a baseline for nearby candidates, not a blanket rule for specialists.

The original paper distinguishes two selection strategies. Uniform soups average all fine-tuned checkpoints with equal weight, which is simple but risky: one bad checkpoint can drag the average down. Greedy soups start from the best individual model and add a candidate only if the resulting average improves a held-out validation metric. In the reported experiments, greedy soups outperform uniform averaging because the selection rule skips candidates that hurt that validation metric.^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482} That rule is only as useful as the validation slices behind it.

uniform_merge averages matching keys. greedy_keep adds a candidate only when the trial soup score rises. Run it as a control before introducing task-vector filtering.

Averaging is cheap to implement and has no hyperparameters beyond optional weights. On our Code, Math, and Chat trio, a poor aggregate result tells you little about which task caused the loss. Conflicting parameters can cancel when specialists move in opposite directions. To keep those directions visible, the next recipe subtracts the shared base first.

Task arithmetic

The average baseline hides why specialists moved. Task arithmetic^{[6]Reference 6Editing Models with Task Arithmetichttps://arxiv.org/abs/2212.04089} keeps that reference point: a task vector is the difference between fine-tuned weights and the shared base model.

$\tau_i = \theta_i - \theta_0 \quad \text{(task vector for model } i \text{)}$

$\theta_{\text{merged}} = \theta_0 + \sum_{i=1}^{N} \lambda_i \cdot \tau_i$

Here $\theta_0$ is the base model, $\theta_i$ are fine-tuned models, $\tau_i$ are per-task deltas, and $\lambda_i$ are merge weights.

A concrete scalar walkthrough

Before you run this on billion-parameter tensors, make one prediction on a single coordinate. Start with a base weight at 2.0. Code pushes it to 2.5, so its task vector is +0.5; Math pushes it to 1.5, so its task vector is -0.5. If Code gets $\lambda_{\text{code}} = 0.6$ and Math gets $\lambda_{\text{math}} = 0.4$, does the merged coordinate land above or below the base?

$\text{merged weight} = 2.0 + 0.6(0.5) + 0.4(-0.5) = 2.0 + 0.3 - 0.2 = 2.1$

The result, 2.1, is a net positive nudge at this coordinate. Averaging the absolute fine-tuned weights (2.5 and 1.5) would land at 2.0, cancelling both deltas here. The scalar doesn't predict capability retention. It shows what coefficients do before you inspect billions of such coordinates and then test the model.

Why did absolute averaging cancel the two specialist deltas in the scalar example?

Answer

The code-generation fine-tune moved this coordinate up by +0.5 and the math-reasoning fine-tune moved it down by -0.5. Averaging final weights lands this coordinate back at the base value, cancelling both deltas here. Task evaluation determines the broader behavioral effect.

A second coefficient pair exposes a tuning detail. Suppose Code nudges the coordinate to 2.4 (task vector +0.4) and Math nudges it to 1.8 (task vector -0.2). Setting $\lambda_{\text{code}} = 0.7$ and $\lambda_{\text{math}} = 0.5$ gives 2.0 + 0.28 - 0.10 = 2.18. The coefficients need not form a convex average; they scale deltas. Capability retention still comes from evaluation.

task_arithmetic_merge subtracts the base to get task vectors, scales them, and adds them back:

Scaling coefficients control how much of each task vector enters the candidate checkpoint. They don't resolve a coordinate where Code and Math ask for opposite directions. That conflict is the reason to move from task arithmetic to TIES.

TIES-Merging (trim, elect sign, merge)

Task arithmetic exposes conflict but doesn't resolve it. One specialist can increase a coordinate by 0.5 while another decreases it by 0.5. TIES-Merging^{[7]Reference 7TIES-Merging: Resolving Interference When Merging Modelshttps://arxiv.org/abs/2306.01708} turns that conflict into a three-step filter: trim small deltas, elect a sign from total signed movement, then average only values aligned with that sign.

The paper's PEFT analysis found that keeping the top 20% of magnitudes often matched keeping all values on its studied tasks.^{[7]Reference 7TIES-Merging: Resolving Interference When Merging Modelshttps://arxiv.org/abs/2306.01708} Treat that result as evidence for its settings. Trim is a noise filter, not a guaranteed compressor for a new LLM merge.

Use the three specialist rows in the figure as a prediction exercise. At p2, the two negative entries total -0.90, while the positive entry is +1.10. Which sign wins? Positive signed mass wins, and only the aligned +1.10 survives the disjoint merge. The resulting task vector still gets added back to the base, often with a global scale $\lambda$.

Step 1: Trim

Trim keeps the largest-magnitude fraction of each task vector. With four values and density=0.5, predict which two survive, then check the code: they are the two biggest absolute updates.

Step 2: Elect sign

When multiple task vectors update one parameter, TIES-Merging resolves conflicting directions with $\gamma_m^p = \mathrm{sgn}(\sum_t \hat{\tau}_t^p)$.^{[7]Reference 7TIES-Merging: Resolving Interference When Merging Modelshttps://arxiv.org/abs/2306.01708} Sum the trimmed signed deltas and take the resulting sign. This isn't a majority vote: one large update can outweigh two smaller opposing updates.

Step 3: Disjoint merge

Now average only non-zero updates that agree with the elected sign. Trimmed zeros stay out of the mean, so a discarded proposal can't dilute the surviving direction:

TIES-Merging outperforms compared baselines in the paper's evaluated vision and T5 task-vector settings, and its analysis identifies sign interference.^{[7]Reference 7TIES-Merging: Resolving Interference When Merging Modelshttps://arxiv.org/abs/2306.01708} For our LLM trio, use it as a candidate when delta signs conflict. Keep plain task arithmetic in the comparison so any gain has a visible baseline.

DARE (drop and rescale)[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099

TIES chooses a direction among surviving updates. DARE asks a different question first: how many update entries need to survive at all? It randomly drops entries from each task vector and rescales the survivors before a downstream merge. Rescaling preserves an entry's expected delta under the random mask; it doesn't prove that the model preserves a capability.

The paper studies redundancy in supervised fine-tuning (SFT) deltas and reports that its evaluated models can often tolerate dropping 90% of delta entries, and in some cases 99%, before merging. Its size ablation reports that WizardMath-70B remains effective at a 99% drop rate while the evaluated 7B and 13B variants fail there.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099} Treat that as evidence for the paper's SFT models, not as our trio's default density. DARE is a sparsification step, not a complete merge recipe: sparse task vectors still need a merger such as averaging or TIES.

DARE's analysis separates SFT deltas, which it observes are typically within roughly 0.002, from continued-pretraining deltas that approach 0.03; its drop-and-rescale approach becomes ineffective on the latter.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099} One candidate pipeline is DARE-TIES: DARE sparsifies each delta, then TIES resolves directional conflicts among survivors. Mergekit exposes that composition as dare_ties.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}

$\tau_i^{\text{DARE}} = \frac{\tau_i \odot m}{1 - p}$

where $m$ is a random binary mask with drop rate $p$ and the $1/(1-p)$ factor preserves each entry's expectation under masking.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099}

Why does DARE divide by $1-p$ after dropping weights?

Answer

Only a fraction $1-p$ of task-vector entries survive. Dividing by $1-p$ keeps each randomly masked delta entry unbiased in expectation; retained task quality still requires evaluation.

dare_sparsify applies a keep-mask and the $1/(1-p)$ rescale. DARE chooses its mask randomly; this example fixes the mask so you can check the arithmetic. With $p=0.5$, the last two entries survive and double: 0.1 becomes 0.2 and 0.6 becomes 1.2. Predict those values before running it.

On the SFT models and tasks it evaluates, DARE finds substantial redundancy in task-vector entries and improves several downstream merge methods after drop-and-rescale preprocessing.^{[8]Reference 8Language Models are Super Mario: Absorbing Capabilities from Homologous Models as a Free Lunchhttps://arxiv.org/abs/2311.03099} For a new checkpoint family, density remains a tuned parameter. Compare unsparsified and DARE-preprocessed candidates on every required task slice before deciding whether sparsity helped.

SLERP (spherical linear interpolation)

SLERP changes the path geometry, not the evidence standard. It follows an angular arc between normalized vector directions rather than the chord used by linear interpolation. It isn't an algorithm for discovering a low-loss path around an incompatible-model ridge.

Rather than interpolating directions along a straight line, SLERP^{[9]Reference 9Animating Rotation with Quaternion Curveshttps://dl.acm.org/doi/10.1145/325334.325242} interpolates along a sphere. Write the geometry in terms of normalized directions:

$$\begin{aligned} \text{SLERP}(\hat{\theta}_A, \hat{\theta}_B, t) &= \frac{\sin((1-t)\Omega)}{\sin \Omega} \hat{\theta}_A \\ &\quad + \frac{\sin(t\Omega)}{\sin \Omega} \hat{\theta}_B \end{aligned}$$

where $t$ is the interpolation factor (from 0 to 1) and $\Omega = \arccos(\hat{\theta}_A \cdot \hat{\theta}_B)$ is the angle between normalized weight vectors. Practical merge implementations may handle magnitude separately after interpolating direction, as the code does here.

SLERP was introduced for computer graphics to interpolate rotations represented as quaternions.^{[9]Reference 9Animating Rotation with Quaternion Curveshttps://dl.acm.org/doi/10.1145/325334.325242} That source establishes its geometry, not downstream quality for neural-network weight merges. Use a SLERP checkpoint as another candidate and measure loss and required tasks just as you would for a linear merge.

The slerp function interpolates direction on the unit sphere, then linearly interpolates magnitude. Parallel and opposite vectors make $\sin\Omega$ too small, so those cases fall back to a straight line.

Make the geometry concrete before running it. For equal-norm vectors [1, 0] and [0, 1], a linear midpoint is [0.5, 0.5] with norm about 0.707; SLERP should land at [0.707, 0.707] with norm 1.0. That prediction is about shape, not model quality.

In the equal-norm orthogonal-vector example above, the SLERP midpoint remains on the unit circle while a linear midpoint would have smaller norm. Keep the distinction clear: geometry explains the candidate's path; task and safety evaluations decide whether it ships. For a pair of checkpoints, score linear and spherical candidates against the same release gates.

Running a merge with mergekit

The concepts now become one artifact. mergekit^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} is an open-source toolkit for merging language-model checkpoints. Its documented CLI supports YAML-defined merges with CPU or limited-VRAM execution, and mergekit-multi can run multi-stage recipes where later merges consume earlier outputs.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}

Carry the Code, Math, and Chat trio into a YAML file with the base, source weights, density, and merge method. This is a recipe for a candidate, not a release decision:

For dare_ties, each source model's density is the keep fraction of its task vector: density = 1 - p in DARE's drop-rate notation. Merge methods have different schemas, so merge_method isn't a drop-in switch. For example, mergekit's slerp method takes exactly two source models.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}

Tokenizer choice changes the output contract. If all required tokens are in the base tokenizer, tokenizer.source: base pins that vocabulary. Current mergekit configuration defaults to a union tokenizer, which adds source tokens and assigns fallback embeddings where an input model lacks them.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} Either policy needs targeted evaluation. Before running the YAML, predict which policy preserves Code's special tokens, then use the preflight check below.

A written checkpoint is still a candidate. Run the YAML through mergekit-yaml, score every required slice, and promote only after all gates pass.

⚠️ Common mistake: Assuming all specialist checkpoints use the same tokenizer because they began from the same base. If the Code checkpoint added fill-in-the-middle tokens such as <fim_prefix>, choosing base drops those added output entries while choosing union introduces filled embeddings for models that lack them. Verify tokenizer vocabularies, choose the output policy explicitly, and test code-completion prompts that require those tokens.

This preflight check makes the output-vocabulary decision explicit before any expensive merge runs:

Once the YAML is set, run the merge. The command takes the config file and an output path:

For hardware-specific and memory-saving flags, check mergekit-yaml --help because supported options vary by version.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}

A successful write to disk proves that the recipe ran. It says nothing about Code, Math, or Chat. Evaluate each target slice, compare it with declared gates and source baselines, then retune or reject any candidate that misses a critical threshold.

Make the release call before reading the helper. Does an 87.2 aggregate outweigh Math's 82.1? No. An aggregate ranks candidates, but a release decision fails on any critical threshold miss:

Finding good coefficients automatically

The blocked Math slice gives coefficient search a concrete objective. Searching merge coefficients and layer selections by hand can be expensive. Evolutionary model merging^{[10]Reference 10Evolutionary Optimization of Model Merging Recipes.https://sakana.ai/evolutionary-model-merge/} searches merge recipes against a supplied fitness evaluation. The search space can include per-layer source choices, interpolation weights, and whether to merge in parameter space, data-flow space, or both.

Sakana AI reports using this approach to build EvoLLM-JP from Japanese-language and math-oriented models, optimizing for the evaluations selected in that work.^{[10]Reference 10Evolutionary Optimization of Model Merging Recipes.https://sakana.ai/evolutionary-model-merge/} The loop is simple: score a recipe on validation data, mutate its parameters, and keep better-scoring recipes. The search inherits the objective's coverage and blind spots, so independent release slices still matter.

Before reaching for evolution, use a small grid as a transparent baseline. Try lambda values in [0.2, 0.4, 0.6, 0.8] for each task vector, measure each required slice on held-out data, and keep only configurations that clear every release gate. For Code, Math, and Chat, include code-generation tests, math word problems, and instruction-following prompts. A code-only objective can't protect Math.

Passthrough and frankenmerging

Not every merge averages weights. Mergekit's passthrough method copies selected tensors or layer ranges from source models into the output checkpoint.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit} This is model splicing rather than interpolation: tensor dimensions must compose, while useful behavior across the splice remains an evaluation result.

Suppose a recipe copies layers 0-19 from Code and layers 20-31 from Math. Compatible dimensions make that artifact constructible; they don't show that Math's later layers can interpret the hidden states produced by Code's earlier layers.

Use passthrough as an experiment with explicit source lineage, layer boundaries, and the same per-task gates used for any other candidate. A successful build proves shape compatibility only.

Merging pitfalls and hard limits

Direct weight merging requires aligned parameter meaning, shape, and output-space assumptions. You can't directly interpolate a 7-billion-parameter checkpoint with a 70-billion-parameter checkpoint, or silently combine incompatible vocabulary mappings and treat token IDs as equivalent. A generated artifact is a candidate, not evidence of retained quality.

Use these stop signs before spending time tuning coefficients. Some block construction; others block release even when construction and average quality look good.

Avoid direct interpolation when:

1. Different architectures: Source models must share the same structural architecture, parameter count, and layer layout. You can't directly interpolate a Llama checkpoint with a Mistral checkpoint, or an 8B model with a 70B model.
2. Different tokenizers: Direct interpolation assumes aligned embeddings and next-token heads. Mergekit can help with tokenizer union in compatible cases, but it doesn't make arbitrary vocabulary mismatches disappear.^{[1]Reference 1Mergekit: Tools for merging pre-trained large language modelshttps://github.com/arcee-ai/mergekit}
3. Mismatched prompt formats or chat templates: These usually won't block the raw tensor merge, but they can make the merged checkpoint look broken at inference time because the prompt serialization no longer matches the behaviors learned during tuning.
4. Quantized-only checkpoints: If you want to merge, do it on dequantized or full-precision weights first, then quantize the final artifact. The merge math depends on real-valued deltas, not already-rounded integers. Treat the merged checkpoint as a new model and re-run your quantization calibration and evals.
5. Distant fine-tuning trajectories: Models fine-tuned on divergent data or with large update magnitudes may be poor interpolation candidates. Git Re-Basin demonstrates permutation alignment in studied MLP, CNN, and ResNet settings; it isn't an established repair step for arbitrary LLM merges.^{[5]Reference 5Git Re-Basin: Merging Models modulo Permutation Symmetries.https://arxiv.org/abs/2209.04836}
6. Critical precision tasks: Don't ship a merge when a required slice misses its threshold, even if an aggregate metric rises. A specialist baseline remains part of the comparison.

Diagnosing a broken merge

Diagnose in order. First verify the artifact contract: keys, shapes, tokenizer, output head, and chat template. Then compare behavior with source baselines on each task slice. Only after those checks should you tune coefficients or choose another merge rule.

When a merge goes wrong, the model usually tells you quickly:

The original Model Soups gains in shared-initialization ImageNet experiments justify trying an average. They don't predict that this Code, Math, and Chat merge will pass.^{[3]Reference 3Model Soups: Averaging Weights of Multiple Fine-tuned Models Improves Accuracy without Increasing Inference Timehttps://arxiv.org/abs/2203.05482} Promotion still requires a clean artifact, retained behavior, and every binding gate.