Model Merging and Weight Interpolation

Knowledge distillation compressed teacher behavior into a smaller student. Model merging asks a different deployment question: if you already have several useful fine-tuned checkpoints from the same base model, can you combine their weights into one checkpoint without launching another training run?

Your logistics team runs a customer-support LLM that answers shipping, catalog, and refund questions. After a peak season, you have three fine-tuned checkpoints from the same base model: one tuned on carrier-policy updates, one on product-search queries, and one on refund workflows. Serving all three behind a router adds cold-start latency and complicates rollback. Model merging asks whether you can combine those checkpoints into one deployable model that keeps useful behavior from each specialist without running another training job.

This chapter explains when interpolation is worth testing, where it fails, and how to evaluate merged models before shipping them.

In the previous articles on fine-tuning and alignment, you learned that a model checkpoint is a large vector of numbers, and that fine-tuning nudges those numbers in directions that improve a specific task. Model merging asks a natural follow-up question: what if you could combine those nudges? If one team fine-tuned a base model for Python code and another team fine-tuned the same base model for math reasoning, can you average their checkpoints and get a model that's good at both?

The attraction is practical: a merge creates a candidate checkpoint without another gradient-training run. Whether that candidate retains any target behavior is an evaluation question, not a property guaranteed by the merge recipe.

A hard prerequisite for direct tensor interpolation is compatible parameter structure: corresponding tensors must have compatible shapes and meanings. Embedding matrices and language-model heads also need an explicit tokenizer policy. You can't interpolate a 7B checkpoint with a 70B checkpoint, or blindly combine token rows from different vocabularies. Mergekit can construct a union tokenizer and assign fallback embeddings for missing tokens, but that is an explicit output-space choice that still needs evaluation.^[1] Same-base checkpoints are the conservative starting point for task-vector merging because their parameter coordinates share lineage; compatible shapes alone do not establish merge quality.

The illustration below compares four recipes covered in this chapter, from averaging to sparse conflict-aware task-vector merging.

When averaging weights is worth testing

If someone told you to average the internal numbers of two trained neural networks, skepticism is correct. Weight averaging is plausible in some fine-tuning settings because nearby checkpoints can occupy a connected low-loss region in parameter space. It is not safe merely because both endpoints are good models.

Think of fine-tuning as moving from a shared starting point through weight space. Linear mode connectivity studies whether the straight interpolation path between endpoints crosses a high-loss barrier.^[2] The Model Soups paper motivates averaging in a specific setting: models fine-tuned from a shared pretrained initialization under different hyperparameters often admit useful averages in its evaluated vision and text-classification experiments.^[3] This is evidence for testing nearby same-lineage candidates, not a proof that differently specialized LLM checkpoints share one basin.

In concrete terms, evaluate points on $\theta(\alpha) = (1-\alpha)\theta_A + \alpha\theta_B$ rather than assuming the midpoint is usable. Two checkpoints can each be strong on their own evaluation and still interfere when combined, especially when they represent different tasks. Models with unrelated pretraining lineages are even poorer candidates for direct interpolation because their parameter coordinate systems were not preserved by a common base.

The screening logic below treats interpolation measurements as evidence, rather than granting the midpoint a pass because the sources share lineage. In a real run, losses and task_scores come from the candidate checkpoints and held-out evaluations:

Key insight: Same-base lineage makes a merge experiment defensible. It does not establish that a shipping-policy and refund-workflow merge retained either behavior. Only per-task evaluation does that.

The permutation invariance problem

Neural networks can exhibit permutation invariance: under appropriate corresponding reordering of incoming and outgoing weights, hidden-unit permutations can preserve the function a network computes.^[4] This helps explain why independently trained networks may be poorly aligned for naive averaging even if they solve similar tasks.

Git Re-Basin^[5] addresses this by finding the optimal permutation $\pi$ that aligns the neurons of one model to match the other before merging:

$\theta_{\text{merged}} = \frac{1}{2}(\theta_A + \pi(\theta_B))$

The illustration below makes the failure mode concrete. Two checkpoints can learn the same three hidden features but store them in different slot orders, so Git Re-Basin permutes one model before averaging.

Git Re-Basin uses permutation matching and reports merged independently trained MLP, CNN, and ResNet models in its studied settings, including a zero-barrier ResNet result on CIFAR-10.^[5] That paper is a useful alignment concept, but it is not evidence that an arbitrary pair of large language models can be repaired and merged. For LLM work, same-base candidates plus downstream evaluation remain the practical default here.

The merging toolkit: from simple averages to conflict-aware methods

Several techniques exist for merging models, ranging from mathematical averages to geometric interpolation rules. Choice depends on source lineage, observed delta conflict, and the evaluations the output must pass.

Model soups (uniform averaging)

The simplest approach to merging is called Model Soups^[3]. It averages the weights of multiple fine-tuned models to create a single robust model:

$\theta_{\text{merged}} = \frac{1}{N} \sum_{i=1}^{N} \theta_i$

To see the operation, imagine two scalars instead of billion-parameter tensors. If checkpoint A has a weight 2.0 and checkpoint B has 4.0, their equal average is 3.0. Whether that compromise retains behavior cannot be determined from this parameter alone. Model Soups evaluates averaging models fine-tuned from a shared initialization over hyperparameter configurations and reports improved accuracy and robustness in its studied settings without the inference cost of an ensemble.^[3]

Uniform vs. greedy soups

The original Model Soups paper distinguishes between two selection strategies:

• Uniform Soups: Average all fine-tuned checkpoints with equal weight. Simple but risky, as a single poorly-performing checkpoint can drag down the overall quality.
• Greedy Soups: Iteratively add checkpoints to the soup only if they improve performance on a held-out validation set. Start with the best individual model, then test each remaining model: if adding it to the average improves the validation metric, keep it; otherwise, discard it.

In the original paper, greedy soups outperform uniform averaging in the reported experiments because the selection rule skips candidates that hurt its held-out validation metric.^[3] A similar selection rule is reasonable to test only when your validation slices represent the behavior you need to keep.

The following function demonstrates uniform averaging. It takes a list of model state dictionaries and an optional list of weights, and returns a single state dictionary containing the weighted average of their parameters:

Pros

Exceptionally simple to implement and requires no hyperparameters beyond the optional weighting scheme.

Cons

If the models are fine-tuned on wildly different tasks, direct averaging can cause destructive interference between conflicting parameters, degrading overall performance.

Task arithmetic

Instead of averaging absolute weights, Task Arithmetic^[6] operates on task vectors (the difference between fine-tuned weights and the base model). Think of task vectors like a set of specific directions (e.g., "walk 10 steps north"). Instead of averaging the final destinations of different hikers, you take the specific path each hiker took from base camp and combine them. By scaling these paths (e.g., "take half the steps north"), you can carefully mix different skills.

$\tau_i = \theta_i - \theta_0 \quad \text{(task vector for model } i \text{)}$

$\theta_{\text{merged}} = \theta_0 + \sum_{i=1}^{N} \lambda_i \cdot \tau_i$

Where $\theta_0$ is the base model, $\theta_i$ are fine-tuned models, $\tau_i$ are per-task deltas, and $\lambda_i$ are merge weights.

A concrete scalar walkthrough

Before you run this on billion-parameter tensors, try it on a single parameter. Imagine a base model where one weight is 2.0. A Python expert fine-tune pushes that weight to 2.5, so its task vector is +0.5. A math expert fine-tune pushes it to 1.5, so its task vector is -0.5. If you want a model that is slightly better at Python but keeps its math ability, you might set $\lambda_{\text{python}} = 0.6$ and $\lambda_{\text{math}} = 0.4$:

$\text{merged weight} = 2.0 + 0.6(0.5) + 0.4(-0.5) = 2.0 + 0.3 - 0.2 = 2.1$

The result, 2.1, is a parameter nudge in the Python-delta direction. If you had instead averaged the absolute fine-tuned weights (2.5 and 1.5), you'd get 2.0, cancelling both deltas at this coordinate. A scalar calculation cannot establish how either task performs; it only shows how task-vector coefficients act on weights.

You can use the same arithmetic for logistics. Suppose a shipping-policy fine-tune nudges a weight to 2.4 (task vector +0.4) and a refund-workflow fine-tune nudges it to 1.8 (task vector -0.2). Setting $\lambda_{\text{shipping}} = 0.7$ and $\lambda_{\text{refund}} = 0.5$ gives 2.0 + 0.28 - 0.10 = 2.18. Whether this coefficient pair keeps either workflow is measured after the merge.

The task_arithmetic_merge function below shows how this works in practice. It takes a shared base model, a list of fine-tuned models, and scaling coefficients. It subtracts the base to get task vectors, applies a scaling coefficient, and adds them back to return the merged model:

Advantage

Scaling coefficients control how much of each task vector enters the candidate checkpoint; evaluation determines retained behavior.

TIES-Merging (trim, elect sign, merge)

A major issue with simple averaging is that task vectors can directly contradict one another (e.g., one model increases a weight by 0.5, while another decreases it by 0.5). TIES-Merging^[7] addresses two forms of interference in its recipe: it trims low-magnitude deltas, elects a sign for each coordinate using total signed movement, and merges only values aligned with that elected sign.

Think of this process like a logistics operations committee proposing numeric budget changes. TIES resolves a coordinate in three stages. First, discard small proposals (Trim). Next, sum the size of increases against the size of decreases, rather than counting voters (Elect sign). Finally, average only proposals in the winning direction (Disjoint merge).

Step 1: Trim

First, it drops low-magnitude changes treated as redundant by the recipe. The trim function below takes a task vector and a density threshold, then keeps high-magnitude updates:

Step 2: Elect sign

When multiple task vectors update the same parameter, TIES-Merging resolves conflicting directions by total magnitude. The elect_sign function sums the trimmed signed deltas and takes the resulting sign. This differs from a majority vote: one large update can outweigh two smaller opposing updates.

Step 3: Disjoint merge

Finally, it computes the average of only those task vectors whose updates match the consensus direction. The disjoint_merge function takes trimmed task vectors and their elected signs. It averages only the parameter updates that agree with the elected direction, zeroing out any dissenting values to produce the final merged task vector:

TIES-Merging outperforms compared baselines in the paper's evaluated vision and T5 task-vector settings, and its analysis highlights sign interference.^[7] For an LLM merge, use it as a candidate recipe when task vectors conflict, then compare per-task evaluation against simpler baselines.

DARE (drop and rescale)^[8]

DARE randomly drops entries from each task vector and rescales the survivors before a downstream merge. The rescaling preserves an entry's expected delta under the random mask; it does not by itself prove that the resulting model preserves a capability.

The paper studies redundancy in supervised fine-tuning (SFT) deltas and reports that its evaluated models can often tolerate dropping 90% of delta entries, and in some cases 99%, before merging. Its size ablation reports that WizardMath-70B remains effective at a 99% drop rate while the evaluated 7B and 13B variants fail there.^[8] Treat that as experimental evidence for the paper's SFT models, not a default density setting for a new merge. DARE is a sparsification step, not a complete merge recipe: sparsified task vectors still need a merger such as averaging or TIES.

DARE's analysis also separates SFT deltas, which it observes are typically within roughly 0.002, from continued-pretraining deltas that approach 0.03; its drop-and-rescale approach becomes ineffective on the latter.^[8] One candidate pipeline is DARE-TIES: DARE sparsifies each delta, then TIES resolves directional conflicts among surviving values. Mergekit exposes that composition as dare_ties.^[1]

$\tau_i^{\text{DARE}} = \frac{\tau_i \odot m}{1 - p}$

where $m$ is a random binary mask with drop rate $p$ and the $1/(1-p)$ factor preserves each entry's expectation under masking.^[8]

The dare_sparsify function below shows the DARE step itself. It sparsifies one task vector, after which you can pass the result to a downstream merge rule:

What the paper establishes

On the SFT models and tasks it evaluates, DARE finds substantial redundancy in task-vector entries and improves several downstream merge methods after drop-and-rescale preprocessing.^[8] For a new checkpoint family, density remains a tuned parameter: compare unsparsified and DARE-preprocessed merges on every required task slice.

SLERP (spherical linear interpolation)

SLERP is a geometric interpolation rule. It follows an angular arc between normalized vector directions rather than the chord used by linear interpolation. It is not an algorithm for discovering a low-loss path around an incompatible-model ridge.

Rather than interpolating directions along a straight line, SLERP^[9] interpolates along a sphere. The cleanest way to write the geometry is in terms of normalized directions:

$$\begin{aligned} \text{SLERP}(\hat{\theta}_A, \hat{\theta}_B, t) &= \frac{\sin((1-t)\Omega)}{\sin \Omega} \hat{\theta}_A \\ &\quad + \frac{\sin(t\Omega)}{\sin \Omega} \hat{\theta}_B \end{aligned}$$

where $t$ is the interpolation factor (from 0 to 1) and $\Omega = \arccos(\hat{\theta}_A \cdot \hat{\theta}_B)$ is the angle between normalized weight vectors. Practical merge implementations may handle magnitude separately after interpolating direction, which is what the code below does.

SLERP was introduced for computer graphics to interpolate rotations represented as quaternions.^[9] That source establishes its geometry, not downstream quality for neural-network weight merges. Use a SLERP checkpoint as another candidate and measure loss and required tasks just as you would for a linear merge.

The slerp function demonstrates this geometric interpolation. It takes two unnormalized vectors and an interpolation factor t, projects them onto a unit sphere, computes the interpolation, and returns the combined vector:

What it guarantees geometrically

In the equal-norm orthogonal-vector example above, the SLERP midpoint remains on the unit circle while a linear midpoint would have smaller norm. That is a geometric property of this example, not evidence that the midpoint preserves either model's behavior. For a pair of checkpoints, evaluate linear and spherical candidates against the same release gates.

Running a merge with mergekit

mergekit^[1] is an open-source toolkit for merging language-model checkpoints. Its documented CLI supports YAML-defined merges with CPU or limited-VRAM execution, and mergekit-multi can run multi-stage recipes where later merges consume earlier outputs.^[1]

To use mergekit, you typically define a YAML configuration file that specifies the base model, the fine-tuned source models, their respective merging coefficients, and the desired algorithm. This configuration acts as the input to the CLI tool to generate the merged model:

If all required tokens are already in the base tokenizer, tokenizer.source: base pins the output vocabulary to that base. Modern mergekit configuration defaults to a union tokenizer, which adds tokens present in source vocabularies and assigns fallback embeddings where an input model lacks them.^[1] Either policy is an output-space decision that needs targeted evaluation.

Common Mistake: Assuming all logistics checkpoints use the same tokenizer because they began from the same base. If the catalog checkpoint added SKU tokens (<SKU-12345>), choosing base drops those added output entries while choosing union introduces filled embeddings for models that lack them. Verify tokenizer vocabularies, choose the output policy explicitly, and test SKU-heavy prompts.

This preflight check makes the output-vocabulary decision explicit before any expensive merge runs:

Once the configuration is set, you can execute the merge using the CLI tool. The command below takes the YAML configuration file and the output path, producing the final merged model on disk:

For hardware-specific and memory-saving flags, check mergekit-yaml --help because supported options vary by version.^[1]

Mergekit lets engineers build candidate checkpoints without a new gradient-training run. Iterate over parameters such as retained density and task weights only against a defined evaluation suite.

The diagram below shows the iterative validation loop you should run after every merge: evaluate each target slice, compare it to declared gates and source baselines, and retune or reject any candidate that misses a critical threshold.

An aggregate score is useful for ranking candidates, but a release decision should fail on any critical threshold miss:

Finding good coefficients automatically

Manually searching merge coefficients and layer selections can be expensive. Evolutionary model merging^[10] applies evolutionary search to optimize merge recipes against a supplied fitness evaluation. Its search space can include per-layer source choices, interpolation weights, and whether to merge in parameter space, data-flow space, or both.

Sakana AI reports using this approach to build EvoLLM-JP from Japanese-language and math-oriented models, optimizing for the evaluations selected in that work.^[10] The principle is straightforward: treat each merge configuration as a "genome," score it on validation data, mutate its parameters, and retain better-scoring candidates. The resulting model inherits the objective's coverage and blind spots, so release gates still need independent slices.

In practice, a small grid search is a transparent baseline: try lambda values in [0.2, 0.4, 0.6, 0.8] for each task vector, measure each required slice on held-out data, and retain only configurations that clear all release gates. For a logistics merge, those slices should include carrier-policy questions, refund scenarios, and catalog lookups; a shipping-only objective does not protect refund behavior.

Passthrough and frankenmerging

Not every merge averages weights. Mergekit's passthrough method copies selected tensors or layer ranges from source models into the output checkpoint.^[1] This is model splicing rather than interpolation: tensor dimensions must compose, while useful behavior across the splice remains an evaluation result.

For example, a recipe might copy layers 0-19 from checkpoint A and layers 20-31 from checkpoint B. Compatible dimensions make that artifact constructible; they do not show that B's later layers can interpret the hidden states produced by A's earlier layers.

Use passthrough as an experiment with explicit source lineage, layer boundaries, and the same per-task gates used for any other candidate. A successful build only proves shape compatibility.

Merging pitfalls and hard limits

Direct weight merging requires aligned parameter meaning, shape, and output-space assumptions. You cannot directly interpolate a 7-billion-parameter checkpoint with a 70-billion-parameter checkpoint, or silently combine incompatible vocabulary mappings and treat token IDs as equivalent. A generated merged artifact is a candidate, not evidence of retained quality.

You should avoid merging when:

1. Different architectures: Source models must share the exact same structural architecture, parameter count, and layer layout. You can't merge a MiniMax model with a Mistral model, nor an 8B model with a 72B model.
2. Different tokenizers: Direct interpolation assumes aligned embeddings and next-token heads. Mergekit can help with tokenizer union in compatible cases, but it doesn't make arbitrary vocabulary mismatches disappear.^[1]
3. Mismatched prompt formats or chat templates: These usually won't block the raw tensor merge, but they can make the merged checkpoint look broken at inference time because the prompt serialization no longer matches the behaviors learned during tuning.
4. Quantized-only checkpoints: If you want to merge, do it on dequantized or full-precision weights first, then quantize the final artifact. The merge math depends on real-valued deltas, not already-rounded integers. Treat the merged checkpoint as a new model and re-run your quantization calibration and evals.
5. Distant fine-tuning trajectories: Models fine-tuned on divergent data or with large update magnitudes may be poor interpolation candidates. Git Re-Basin demonstrates permutation alignment in studied MLP, CNN, and ResNet settings; it is not an established repair step for arbitrary LLM merges.^[5]
6. Critical precision tasks: Do not ship a merge when a required slice misses its threshold, even if an aggregate metric rises. A specialist baseline remains part of the comparison.

Diagnosing a broken merge

When a merge goes wrong, the model usually tells you quickly. Here are three common failure patterns, their causes, and how to fix them:

Do not predict quality from the merge rule alone. Model Soups^[3] reported gains over the best individual checkpoint in its evaluated shared-initialization experiments, including ImageNet settings. That result motivates trying a merge; it does not predict whether a new LLM merge will clear its specialist task gates.

After merging, run a full evaluation suite across all target tasks, not only the aggregate score. Strong shipping accuracy can mask a miss in refund handling. Track per-task metrics independently and compare against declared thresholds and each source model's baseline.

Practice checkpoints

Why is weight averaging worth testing at all? Shouldn't mixing model weights produce garbage? Shared-initialization checkpoints provide a plausible starting point because studies such as Model Soups observe useful averages in evaluated settings. Shared lineage is not a quality proof: sample the interpolation path and run task gates. Independently trained models can also encode equivalent functions with incompatible neuron orderings, making direct averaging particularly suspect.

How do you choose merge coefficients for three or more models? Start with equal weights as a baseline, then tune against a held-out validation set that reflects your target task mixture. Track each target capability separately. If equal weights cause interference, try task-vector coefficients, TIES, DARE-TIES, or a small grid search over weights and density values.

When should you merge models instead of using mixture-of-experts routing? Merge when you want one dense checkpoint with roughly one-model serving cost, simpler rollback, and no router path. Use MoE or explicit routing when tasks are highly distinct, specialists must stay sharp, or you can afford multiple experts plus routing complexity.

How does LoRA adapter merging differ from full-weight merging? LoRA adapters are low-rank delta weights attached to a frozen base, so adapter merging operates on a much smaller parameter set. That makes experiments cheaper, but it doesn't remove interference: two adapters can still push the same target matrix in conflicting directions.

What should you check first when a merge outputs incoherent text? Check architecture, tensor shapes, tokenizer output policy, and chat template before coefficient tuning. These checks identify whether inputs and output IDs still mean what the merge recipe assumed.

Try it yourself

The best way to learn merging is to reason through a concrete scenario before you touch a GPU.

Interview warm-up

Question: Why can't we average the weights of GPT-2 and Llama-3?

Reasoning: Even if you ignored the parameter-count mismatch, the two models have different architectures and different tokenizers. Their parameter entries therefore do not represent an aligned interpolation space. Same-base checkpoints are conservative direct-merge candidates; alignment methods such as Git Re-Basin provide research evidence in studied neural-network architectures, not a generic way to make GPT-2 and Llama compatible.

Grid-search challenge

You have a base model and two fine-tuned variants:

• code_ft improves Python accuracy from 40% to 70%
• math_ft improves algebra accuracy from 35% to 65%

You want a merged model that scores at least 60% on both tasks. You try Task Arithmetic with coefficients $\lambda_{\text{code}}$ and $\lambda_{\text{math}}$.

Given the table below, which coefficient pair is the best starting point?

Answer: None of the listed pairs reaches 60% on both tasks. The best starting point is 0.8, 0.8 because it gets closest at 64% Python and 59% algebra, which tells you the merge is almost there but still suffering from interference. In practice, you'd try TIES or DARE next, or continue tuning coefficients around that region, to lift algebra without giving up too much Python performance.

Debug a broken merge

You merge three e-commerce support models (refunds, shipping, catalog) using uniform averaging. The merged model answers refund questions perfectly but responds to shipping queries with seemingly random product names.

What is the most likely cause, and what should you check first?

Answer: Tokenizer/output-space mismatch is a plausible first suspect, but the symptom alone does not identify one root cause. Inspect each source tokenizer and chat template, then compare shipping behavior in the source checkpoints. If added SKU tokens are needed, choose and evaluate an explicit union tokenizer policy; otherwise retain a shared-base output vocabulary and investigate source interference.

Mastery check

Key concepts

• Model soups and uniform averaging
• Task vectors and task arithmetic
• Linear mode connectivity
• Git Re-Basin and permutation alignment
• TIES-Merging
• DARE sparsification
• SLERP
• mergekit workflow and tokenizer handling
• Per-task regression testing for merged checkpoints

Evaluation rubric

• Foundational: Explains why same-base fine-tuned checkpoints are reasonable averaging candidates but still require evaluation
• Intermediate: Computes task vectors and shows how scaling coefficients change the merged checkpoint
• Intermediate: Explains why absolute averaging can cancel opposing task deltas
• Advanced: Compares when to use model soups, task arithmetic, TIES, DARE, or SLERP
• Advanced: Diagnoses tokenizer mismatch, architecture mismatch, and coefficient overload in a broken merge
• Advanced: Designs a post-merge eval suite that blocks promotion when any critical capability misses its release threshold

Follow-up questions

Common pitfalls

"Same parameter count means mergeable"

Symptom: The merged model produces incoherent or unstable outputs right away. Cause: Matching parameter count is not enough. The checkpoints may differ in architecture details, tokenizer alignment, or base-model lineage. Fix: Verify shared architecture, layer shapes, tokenizer handling, and common base checkpoint before merging any weights.

"One aggregate score proves the merge worked"

Symptom: The merged checkpoint looks strong overall but one specialist workflow regresses badly in production. Cause: Aggregate metrics can hide per-task damage from interference. Fix: Run per-task checks against declared release thresholds and source baselines, then block promotion when any critical skill falls below threshold.

"More coefficient means more skill"

Symptom: Larger lambda values make outputs repetitive, brittle, or off-distribution. Cause: Oversized coefficients can push the merged checkpoint too far away from the stable base region. Fix: Start with small or equal weights, tune on held-out evals, and watch for regressions rather than assuming stronger scaling always helps.

"Tokenizer issues only matter at preprocessing time"

Symptom: Catalog-heavy prompts fail or produce random product tokens even though the merge completed. Cause: Embedding rows and LM-head rows no longer line up cleanly across token IDs. Fix: Check vocabulary equality first. If source models added tokens, use explicit tokenizer alignment such as mergekit union handling and then rerun evals.

"DARE is a full merge recipe by itself"

Symptom: Engineers sparsify task vectors and assume the job is done. Cause: DARE is a preprocessing step that drops and rescales delta entries; it still needs a downstream merge rule such as averaging or TIES. Fix: Treat DARE as sparsification first, then run a real merge rule and evaluate the combined checkpoint.