LoRA & Parameter-Efficient Tuning

Suppose an access-policy assistant already handles routine key reviews, but a new rotation procedure changes how stale keys must be escalated. You have 5,000 labeled tickets and one 48 GB training GPU. Full fine-tuning asks that machine to hold gradients and Adam state for every base weight. A thin adapter looks affordable, but it still has to learn the new policy without damaging ordinary reviews. Which constraint fails first: memory, adaptation capacity, or regression quality?

Call this fictional experiment policy-lora-v1. It changes how the assistant handles policy evidence, not where the current policy lives: frequently changing procedures still need a versioned source or retrieval. LoRA (Low-Rank Adaptation) freezes the pretrained weights and learns a small additive update inside selected linear layers. Full fine-tuning moves every eligible weight. Both still execute the base network.

The original GPT-3 175B experiments reported roughly 10,000 times fewer trainable parameters and three times lower GPU memory requirements than full fine-tuning with Adam.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} Those results belong to one paper setup. The calculations below are storage estimates, and the executable examples check small layers, not a completed policy-training run.

Why LoRA exists

Full fine-tuning a 70-billion parameter model is expensive for a specific reason: training stores more than model weights. It also keeps gradients and optimizer state for every trainable parameter. Before reading the table, fix the accounting recipe. It assumes low-precision parameters and gradients with two FP32 Adam moment tensors, but no separate FP32 master parameter copy.

Consider the memory breakdown for a 70B model using Adam with FP16 (half-precision) weights:

The 840 GB total covers model state only. All GB and MB here are decimal ($10^9$ and $10^6$ bytes). A real run also stores activations and temporary buffers. If the optimizer keeps a separate FP32 master copy, add another 280 GB, giving the 1.12 TB, 16-byte recipe from the distributed training lesson. Actual dtypes depend on the optimizer and training stack; this isn't a claim about every Adam implementation.

params_billion = 70
gb_per_byte_per_billion = 1

recipe = {
    "fp16_parameters": 2,
    "fp16_gradients": 2,
    "fp32_adam_moments": 8,
}
without_master = params_billion * sum(recipe.values()) * gb_per_byte_per_billion
with_master = without_master + params_billion * 4

print("bytes_per_parameter_without_master=", sum(recipe.values()))
print("full_finetuning_states_without_master_GB=", without_master)
print("full_finetuning_states_with_master_GB=", with_master)
print("frozen_fp16_base_floor_for_lora_GB=", params_billion * recipe["fp16_parameters"])

bytes_per_parameter_without_master= 12
full_finetuning_states_without_master_GB= 840
full_finetuning_states_with_master_GB= 1120
frozen_fp16_base_floor_for_lora_GB= 140

840 GB of model state already blows past the capacity of a single 80 GB GPU. Does adapting to a specific task really require moving all 70 billion parameters independently?

Pretrained transformers are massively over-parameterized. During pretraining, they construct rich internal representations of syntax, reasoning, and world knowledge. When we adapt policy-lora-v1 on 5,000 security tickets, we aren't asking the network to relearn the structure of language from scratch. We're steering existing representations toward updated escalation rules.

Aghajanyan et al. found that their studied pretrained models could approach full fine-tuning performance by optimizing a much smaller, randomly projected parameter subspace.^{[2]Reference 2Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.https://arxiv.org/abs/2012.13255} This intrinsic dimension counts free optimization coordinates; it doesn't establish the matrix rank of each layer's update. One scalar $\theta$ controlling $\Delta W=\theta I$ already gives a full-rank update whenever $\theta\ne0$.

Inspired by those results, Hu et al. proposed a separate intrinsic rank hypothesis and tested low-rank updates.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} LoRA constrains selected updates to $\Delta W=BA$, whose rank is at most $r$. That's a useful modeling choice to evaluate, rather than proof that every task's ideal update has low rank.

What LoRA saves, and what it doesn't

LoRA removes gradients and optimizer state for the frozen base model. The base weights still occupy memory, and the adapter keeps its own small trainable state.

Run the policy example at 2,048 tokens per ticket and the apparent bargain changes. Activation memory isn't eliminated. Activations are intermediate tensors retained for backpropagation. Many grow with sequence length, batch size, and depth; materialized attention scores additionally grow quadratically with sequence length. Backpropagation still needs paths through frozen operations to calculate adapter gradients. Long-context activations can outweigh adapter state or even base weights. Profile the peak before shrinking rank: compare activation recomputation (gradient checkpointing), smaller micro-batches, and an attention implementation that avoids the full score matrix.

Memory boundary: LoRA removes base-weight gradient and optimizer-state storage, not the need to execute the full base network. A batch of 2,048-token access-review tickets can still be dominated by activations. Measure peak memory under the intended context length and micro-batch size.

The running example uses 70B for planning, but QLoRA's published comparison uses 65B. Keep that boundary visible: under this 12-byte recipe, 65B full fine-tuning requires 780 GB, while the QLoRA paper fine-tuned a 65B model on a single 48 GB GPU.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314} That paper result is a demonstrated end-to-end configuration, not a universal guarantee for every sequence length or batch size.

The bars explain why policy-lora-v1 is worth testing. They don't tell us how much behavior the overlay can express. For one layer, full fine-tuning rewrites $W_0$. LoRA keeps $W_0$ frozen and learns $\Delta W = BA$, scaled by $\alpha / r$, so the layer uses $W_0 + \frac{\alpha}{r}BA$. The next section earns each symbol with a small layer.

The math: from a 4096×4096 layer

A layer small enough to count

Suppose one Transformer projection has a $4096 \times 4096$ weight matrix. Before naming LoRA's factors, predict the budget: if the update has rank $r=16$, will it contain millions of values like the base layer, or only thousands?

Full fine-tuning would train $4096 \times 4096 = 16,777,216$ parameters. A rank-16 LoRA update instead uses:

• Matrix $A$ has shape $16 \times 4096$ and has $65,536$ parameters.
• Matrix $B$ has shape $4096 \times 16$ and has $65,536$ parameters.
• Total LoRA parameters: $131,072$

That is 0.78% of the full matrix size, a 99.2% reduction. The capacity is smaller by design; the base layer still supplies the general representation.

Now follow one input vector $x$. The frozen layer produces $W_0x$, while the adapter compresses $x$ into rank space, expands it back out, and adds that correction:

$h = W_0 x + \frac{\alpha}{r} B A x = W_0 x + \frac{\alpha}{r} B(Ax)$

One path is the base response. A second path is the trainable overlay, scaled by $\alpha/r$, where $\alpha$ controls its explicit strength. Because both paths share the same input, merging them later can recover one dense matrix.

The parentheses in $B(Ax)$ reflect computational associativity. An ordinary unmerged LoRA forward can evaluate two thin products without materializing the full $d_{\text{out}} \times d_{\text{in}}$ update. Forming that matrix would take $\approx 2 \times 4096^2 \times 16$ operations. Instead, $Ax$ and $B(Ax)$ each take about 131K FLOPs per token, totaling about 262K versus 33.5M for this base projection. That's about 0.78% additional arithmetic for this projection, excluding scaling and addition. It doesn't predict 0.78% extra latency: kernel launches, memory traffic, batching, and backward also matter.

Why the product is low rank

The worked layer used equal input and output widths. In general, a full update $\Delta W$ has the same dimensions as the base weights, $d_{\text{out}} \times d_{\text{in}}$. LoRA factorizes that update into two smaller matrices:

$\Delta W = B \cdot A$

Read the shapes from right to left when checking the product:

• $A \in \mathbb{R}^{r \times d_{\text{in}}}$: the first matrix (maps from input to rank $r$)
• $B \in \mathbb{R}^{d_{\text{out}} \times r}$: the second matrix (maps from rank to output)
• $r$ is the factor width, conventionally called the rank. The product's actual rank is at most $r$; it starts at zero under the usual zero-B initialization.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}

In the figure, trace the dimensions: $B$ times $A$ recreates a matrix shaped like $W_0$, but only $A$ and $B$ carry gradients. The visual also keeps the forward order straight: $x$ enters $A$, then $B$, even though the written product is $BA$.

Count the saved parameters

The same arithmetic scales to any square projection. Use the table to put capacity and storage side by side before checking the count in code.

$\text{LoRA params} = r \cdot d_{\text{in}} + d_{\text{out}} \cdot r = 2r \cdot d$

Where $r$ is LoRA rank, and $d_{\text{in}}, d_{\text{out}}$ are matrix dimensions (equal to $d$ for square projection layers).

For $r=16$, the adapter trains 131K values instead of 16.7M. The task gets a rank-16 update, and that layer no longer stores a full gradient plus Adam pair for its frozen base weights.

def lora_parameter_count(d_in: int, d_out: int, rank: int) -> int:
    return rank * d_in + d_out * rank

d_in = d_out = 4096
rank = 16
full_params = d_in * d_out
lora_params = lora_parameter_count(d_in, d_out, rank)
ratio = lora_params / full_params

print(f"full_params={full_params:,}")
print(f"lora_params={lora_params:,}")
print(f"LoRA r={rank} trains {lora_params:,} params, {ratio:.2%} of full fine-tuning.")

full_params=16,777,216
lora_params=131,072
LoRA r=16 trains 131,072 params, 0.78% of full fine-tuning.

Start from the checkpoint

The first training step shouldn't erase the behavior we paid to pretrain. If both $A$ and $B$ start non-zero, the adapter injects random noise before it has seen a single label.

The original paper uses Gaussian A and zero B.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} For linear layers, PEFT's default instead uses Kaiming-uniform A and still zeros B.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} Both make the initial adapter contribution zero. This section uses that ordinary linear-layer initialization; other supported schemes can reparameterize the base or use data to choose factors.

Follow its forward and backward paths:

1. Step-zero identity: $B=0$ makes the additive correction zero. With the same base, inputs, and evaluation settings, the layer begins as $W_0x$. This doesn't guarantee a particular loss trajectory after updates.
2. Usable gradient flow on step 1: Let loss be $\mathcal{L}$ and incoming gradient with respect to output $h$ be $g = \nabla_h \mathcal{L} \in \mathbb{R}^{d_{\text{out}}}$. By the chain rule: $\nabla_B \mathcal{L} = \frac{\alpha}{r} g (Ax)^T \in \mathbb{R}^{d_{\text{out}} \times r}, \qquad \nabla_A \mathcal{L} = \frac{\alpha}{r} B^T g x^T \in \mathbb{R}^{r \times d_{\text{in}}}$ At $B=0$, A's loss gradient is zero. B can receive a nonzero gradient when the scale, incoming gradient, and $Ax$ are nonzero. Nonzero A and x alone aren't sufficient: x can lie in A's null space, and batch contributions can cancel. Once B moves, A can receive gradients on later steps; inspect them rather than assuming every batch updates both factors.

If both factors are zero, both loss gradients are zero and ordinary gradient updates can't leave that point. Reversing the asymmetry, zero A with nonzero B, can also begin as a no-op with A learning first. Two unconstrained random factors generally perturb the starting model; that isn't a theorem that every nonzero initialization is unstable.

Predict the null-space case before running this CPU experiment. The base contribution is fixed at zero, the scale is one, and the loss is $(h-1)^2/2$:

import torch

def factors(a, b):
    return (torch.tensor([a], dtype=torch.float64, requires_grad=True),
            torch.tensor([[b]], dtype=torch.float64, requires_grad=True))

for name, a, b, values, expected in [
    ("usual", [1., -1.], 0., [1., 0.], (False, True)),
    ("null_space", [1., -1.], 0., [1., 1.], (False, False)),
    ("both_zero", [0., 0.], 0., [1., 0.], (False, False)),
    ("reversed", [0., 0.], 1., [1., 0.], (True, False)),
]:
    A, B = factors(a, b)
    x = torch.tensor(values, dtype=torch.float64)
    h = (B @ (A @ x)).squeeze()
    assert h.item() == 0
    ((h - 1).square() / 2).backward()
    moved = (bool(A.grad.abs().sum() > 0), bool(B.grad.abs().sum() > 0))
    assert moved == expected
    print(f"{name}: A_grad_nonzero={moved[0]}, B_grad_nonzero={moved[1]}")

A, B = factors([1., -1.], 0.)
optimizer = torch.optim.SGD([A, B], lr=0.1)
x = torch.tensor([1., 0.], dtype=torch.float64)
for step in range(2):
    optimizer.zero_grad(set_to_none=True)
    h = (B @ (A @ x)).squeeze()
    ((h - 1).square() / 2).backward()
    assert bool(A.grad.abs().sum() > 0) == (step == 1)
    optimizer.step()
print("usual_second_step_A_grad_nonzero=True")

usual: A_grad_nonzero=False, B_grad_nonzero=True
null_space: A_grad_nonzero=False, B_grad_nonzero=False
both_zero: A_grad_nonzero=False, B_grad_nonzero=False
reversed: A_grad_nonzero=True, B_grad_nonzero=False
usual_second_step_A_grad_nonzero=True

Separate capacity from strength

Rank and scale answer two distinct questions. Rank says how many independent directions the adapter can express; $\alpha/r$ says how strongly the resulting correction enters the layer:

$\text{scaling} = \frac{\alpha}{r}$

• $r$ (rank): sets the inner bottleneck dimension, dictating adapter parameter count and expressive capacity.
• $\alpha$ (alpha): sets the fixed numerical scale of the learned correction.

The original paper uses $\alpha/r$ to reduce hyperparameter retuning when changing rank.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} It isn't a universal rank-stability rule. B starts at zero, so the initial correction is zero at every rank; its later size depends on factor initialization, gradients, and optimization as well as the explicit multiplier.

For ordinary LoRA scaling, $\alpha=r$ and $\alpha=2r$ produce multipliers of 1 and 2. Holding $\alpha/r$ constant makes a rank comparison easier to interpret, but it doesn't guarantee identical learned update magnitude. Sweep rank, scale, learning rate, and target-module coverage as coupled choices rather than universal defaults.

rsLoRA (rank-stabilized LoRA) uses $\alpha/\sqrt{r}$. Kalajdzievski derives this scaling under assumptions about initialization and input/gradient statistics, and finds higher-rank learning improvements in the evaluated setups.^{[5]Reference 5A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA.https://arxiv.org/abs/2312.03732} The theorem doesn't guarantee identical gradient norms or better quality in every run, and provides no universal rank-32 or rank-64 cutoff. PEFT exposes it with use_rslora=True.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

import math

configs = [
    {"rank": 8, "alpha": 16},
    {"rank": 32, "alpha": 16},
    {"rank": 32, "alpha": 64},
]

for config in configs:
    rank = config["rank"]
    alpha = config["alpha"]
    ordinary = alpha / rank
    rslora = alpha / math.sqrt(rank)
    print(
        f"r={rank:>2} alpha={alpha:>2} "
        f"ordinary={ordinary:.3f} rslora={rslora:.3f}"
    )

r= 8 alpha=16 ordinary=2.000 rslora=5.657
r=32 alpha=16 ordinary=0.500 rslora=2.828
r=32 alpha=64 ordinary=2.000 rslora=11.314

Where to inject LoRA

The rank tells us how much each adapter can express. We still need to choose where that capacity enters the block. Start with the narrow question: should policy-lora-v1 change only attention routing, or also the per-token transformation that follows it?

In the diagram, purple marks adapted weights and muted boxes stay frozen. Query ($W_Q$) and Value ($W_V$) projections were a central LoRA configuration.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} PEFT's default targets depend on the architecture; unset target_modules isn't a universal promise of Q/V-only adaptation.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} In its LLaMA-7B Alpaca comparison, QLoRA required adapters across all linear Transformer-block layers to match full fine-tuning.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314}

Common injection strategies

Target coverage trades adapter cost for capacity. Before choosing a larger rank, compare these three ways to spend the same kind of adapter budget:

Module names vary by architecture. Current Hugging Face PEFT exposes target_modules="all-linear" as a QLoRA-style shortcut, but on a PreTrainedModel that selection excludes the output projection (lm_head). Inspect the matched module names instead of trusting the shortcut, especially after adding special tokens.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Why adapt MLP layers?

Attention layers ($W_Q, W_K, W_V, W_O$) control how tokens route information across the context. MLP (Multi-Layer Perceptron) layers, also called FFNs (Feed-Forward Networks), transform each token after that routing. The LLaMA and Qwen2-style SwiGLU blocks shown here use gate_proj, up_proj, and down_proj; names and layouts vary across architectures.

Q/V targeting was a principal original LoRA baseline, and that paper also compared other attention targets.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} QLoRA's LLaMA-7B/Alpaca ablation found all-linear coverage important for matching its full-fine-tuning baseline.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314} That motivates a coverage experiment for policy-lora-v1, not a promise that low rank everywhere always beats higher rank in attention or that every procedural task requires MLP adapters. Record cost and held-out quality for each candidate.

The toy block below puts a number on that choice. With rank 16, how much more adapter state does all-linear targeting add than Q/V targeting?

dimensions = {
    "q_proj": (4096, 4096),
    "k_proj": (4096, 4096),
    "v_proj": (4096, 4096),
    "o_proj": (4096, 4096),
    "gate_proj": (4096, 11008),
    "up_proj": (4096, 11008),
    "down_proj": (11008, 4096),
}
rank = 16

def adapter_params(names):
    return sum(rank * (dimensions[name][0] + dimensions[name][1]) for name in names)

qv = adapter_params(["q_proj", "v_proj"])
all_linear = adapter_params(list(dimensions))
print("qv_adapter_params=", qv)
print("all_linear_adapter_params=", all_linear)
print("all_linear_vs_qv_ratio=", round(all_linear / qv, 2))

qv_adapter_params= 262144
all_linear_adapter_params= 1249280
all_linear_vs_qv_ratio= 4.77

When the trainable state isn't a weight update

LoRA isn't the only parameter-efficient fine-tuning method. Two older but still useful PEFT families learn prompt-like state instead of inserting low-rank updates inside weight matrices.

Prompt tuning trains a small set of continuous virtual tokens prepended to the input.^{[6]Reference 6The Power of Scale for Parameter-Efficient Prompt Tuning.https://arxiv.org/abs/2104.08691} They aren't human-readable words; they are learned embedding vectors that steer the frozen model. For policy-lora-v1, that means the control signal enters through each ticket's context window, rather than changing internal projection weights. The approach is cheap, but that entry point limits how directly it can change internal transformations.

Prefix tuning trains continuous key/value prefixes for Transformer layers.^{[7]Reference 7Prefix-Tuning: Optimizing Continuous Prompts for Generation.https://arxiv.org/abs/2101.00190} The base weights remain frozen, while each layer receives learned prefix state that attention can attend to. The control signal therefore reaches multiple layers directly, without becoming a LoRA weight update.

Compare methods by where their learned state enters the model. These are examples to test, rather than a universal ranking of adaptation strength:

Don't call every PEFT method "LoRA." LoRA modifies internal projections through low-rank adapters. Soft prompt tuning and prefix tuning leave those projections frozen and learn continuous prompt-like state instead. That distinction matters when you inspect a checkpoint or estimate what a new policy can change.

Implementing LoRA with Hugging Face PEFT

The peft (Parameter-Efficient Fine-Tuning) library wraps the factors, so you don't need to write $BAx$ by hand. Before reading the setup, predict its two important effects: all-linear should attach more adapters than Q/V targeting, and r=16, alpha=32 should make the ordinary scale $2$.

Verify the adapter on CPU first

This tiny, randomly initialized Qwen2 architecture downloads no checkpoint. It tests the actual PEFT wrapper with pinned PyTorch 2.13.0, Transformers 5.16.1, and PEFT 0.20.0. One synthetic four-token sequence checks no-op initialization, frozen base weights, two-step gradient flow, adapter save/reload, and merge equivalence. These are reproducible fixture versions, rather than a claim that they're the latest releases. They say nothing about policy accuracy or 7B-model memory.

import tempfile
import torch
from peft import LoraConfig, PeftModel, get_peft_model
from transformers import Qwen2Config, Qwen2ForCausalLM

torch.manual_seed(7)
config = Qwen2Config(
    vocab_size=32, hidden_size=16, intermediate_size=32,
    num_hidden_layers=1, num_attention_heads=2, num_key_value_heads=1,
    max_position_embeddings=32, attention_dropout=0.0,
    tie_word_embeddings=False,
)
base = Qwen2ForCausalLM(config).eval()
ids = torch.tensor([[1, 2, 3, 4]])
expected = base(ids).logits.detach().clone()
base_state = {name: value.detach().clone() for name, value in base.state_dict().items()}
base_snapshot = [(p, p.detach().clone()) for p in base.parameters()]
model = get_peft_model(base, LoraConfig(
    r=2, lora_alpha=4, target_modules="all-linear",
    lora_dropout=0.0, bias="none", task_type="CAUSAL_LM",
)).eval()
torch.testing.assert_close(model(ids).logits, expected)
targets = [name.rsplit(".", 1)[-1] for name, module in model.named_modules()
           if hasattr(module, "lora_A")]
assert set(targets) == {"q_proj", "k_proj", "v_proj", "o_proj",
                        "gate_proj", "up_proj", "down_proj"}
assert model.get_nb_trainable_parameters() == (512, 3920)

model.train()
optimizer = torch.optim.SGD([p for p in model.parameters() if p.requires_grad], lr=0.1)
model(ids, labels=ids).loss.backward()
a_grad = sum(p.grad.abs().sum().item() for n, p in model.named_parameters() if "lora_A" in n)
b_grad = sum(p.grad.abs().sum().item() for n, p in model.named_parameters() if "lora_B" in n)
assert a_grad == 0 and b_grad > 0
assert all(p.grad is None for p in model.parameters() if not p.requires_grad)
optimizer.step()
optimizer.zero_grad(set_to_none=True)
model(ids, labels=ids).loss.backward()
assert sum(p.grad.abs().sum().item() for n, p in model.named_parameters() if "lora_A" in n) > 0
optimizer.step()
assert all(torch.equal(p, saved) for p, saved in base_snapshot)

model.eval()
adapted = model(ids).logits.detach()
assert not torch.equal(adapted, expected)
with tempfile.TemporaryDirectory() as directory:
    model.save_pretrained(directory)
    fresh_base = Qwen2ForCausalLM(config)
    fresh_base.load_state_dict(base_state)
    reloaded = PeftModel.from_pretrained(fresh_base, directory).eval()
    torch.testing.assert_close(reloaded(ids).logits, adapted)
merged = model.merge_and_unload(safe_merge=True).eval()
torch.testing.assert_close(merged(ids).logits, adapted, atol=1e-6, rtol=1e-5)
assert not any("lora_" in name for name, _ in merged.named_parameters())
print("adapted modules:", targets)
print("trainable parameters: 512; total with adapters: 3920")
print("no-op, frozen base, two-step gradients, save/reload, merge: passed")

adapted modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj']
trainable parameters: 512; total with adapters: 3920
no-op, frozen base, two-step gradients, save/reload, merge: passed

The key/value projections here are narrower than the query projection because the configuration uses grouped-query attention. Count actual module shapes, not just the model's hidden width. With rank 2, Q and O each add 64 parameters, K and V each add 48, and the three MLP projections each add 96: 512 total.

Set up the larger training job

This is a real training-setup fragment, not a local smoke test. To run it, install torch, transformers, peft, and accelerate on a machine that can load the target model. Let your trainer or distributed launcher handle placement. Accelerate documents device_map="auto" for Big Model Inference, not as a distributed-training strategy.^{[8]Reference 8Accelerate Documentation: Big Model Inference.https://huggingface.co/docs/accelerate/usage_guides/big_modeling}

import torch
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B",
    dtype=torch.bfloat16,
)

# Configure LoRA
lora_config = LoraConfig(
    r=16,                          # rank
    lora_alpha=32,                 # scaling factor alpha/r = 32/16 = 2.0
    target_modules="all-linear",   # QLoRA-style targeting
    lora_dropout=0.05,             # regularization
    bias="none",                   # don't train biases
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Inspect this output rather than assuming how many modules matched.

When you run print_trainable_parameters(), you get trainable parameters, total parameters, and the percentage for this exact checkpoint and module selection. Save that receipt with the run. A shortcut is useful only if it matched the intended projections, and all-linear won't attach an adapter to lm_head.

Inspect stored adapter precision

Loading the base in BF16 doesn't force its adapter factors to stay BF16. PEFT normally promotes low-precision LoRA factors to FP32; autocast_adapter_dtype=False disables that promotion.^{[9]Reference 9PEFT Troubleshooting: Adapter Dtypeshttps://huggingface.co/docs/peft/developer_guides/troubleshooting} The name describes adapter storage handling, not PyTorch's operation-level autocast. This pinned CPU check inspects the parameters directly:

import torch
from torch import nn
from peft import LoraConfig, get_peft_model

for promote in [True, False]:
    base = nn.Sequential(nn.Linear(6, 4, bias=False).to(torch.bfloat16))
    wrapped = get_peft_model(base, LoraConfig(r=2, target_modules=["0"]),
                             autocast_adapter_dtype=promote)
    layer = wrapped.base_model.model[0]
    actual = layer.lora_A["default"].weight.dtype
    assert actual == (torch.float32 if promote else torch.bfloat16)
    assert layer.lora_B["default"].weight.dtype == actual
    assert layer.base_layer.weight.dtype == torch.bfloat16
    print(f"promote={promote}: base={layer.base_layer.weight.dtype}, adapter={actual}")

promote=True: base=torch.bfloat16, adapter=torch.float32
promote=False: base=torch.bfloat16, adapter=torch.bfloat16

Merge when the serving contract is single-tenant

For a single-adapter deployment, you can merge the overlay into the base weights. That removes the extra adapter path and produces one ordinary checkpoint for inference engines that support the base architecture. Keep the separate artifact until evaluation and rollback checks pass.

# After training completes
merged_model = model.merge_and_unload()
# merged_model is now a standard transformers model with no PEFT adapters
merged_model.save_pretrained("./my-adapted-model")

For ordinary LoRA, merge_and_unload() folds $\frac{\alpha}{r}BA$ into each adapted weight and returns the underlying dense model.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} Assign and use that return value. It isn't a promise to preserve an independent, untouched base: the wrapped layers and their weights are modified during unloading. Save the adapter first and reload a clean base for rollback. Compare merged and unmerged outputs in evaluation mode, with dropout disabled; finite-precision rounding can prevent bitwise equality.

base = [[1.0, 2.0], [3.0, 4.0]]
A = [[1.0, -1.0]]
B = [[0.5], [1.0]]
scale = 2.0
x = [2.0, 1.0]

def matvec(matrix, vector):
    return [sum(a * b for a, b in zip(row, vector)) for row in matrix]

adapter_matrix = [
    [scale * B[row][0] * A[0][col] for col in range(2)]
    for row in range(2)
]
merged = [
    [base[row][col] + adapter_matrix[row][col] for col in range(2)]
    for row in range(2)
]
unmerged_output = [
    value + delta
    for value, delta in zip(matvec(base, x), matvec(adapter_matrix, x))
]
print("merged_weights=", merged)
print("outputs_match=", matvec(merged, x) == unmerged_output)

merged_weights= [[2.0, 1.0], [5.0, 2.0]]
outputs_match= True

See the two paths without a framework

The standard-library layer below isolates the forward algebra. It leaves $W_0$ unchanged, creates a thin $r \times d_{\text{in}}$ matrix $A$, starts $B$ at zero, and adds $\frac{\alpha}{r}B(Ax)$. It has no autograd or optimizer. The forward path uses two thin matrix-vector products; only the explicit merge calculation materializes the dense update.

def matvec(matrix: list[list[float]], vector: list[float]) -> list[float]:
    if not matrix or any(len(row) != len(vector) for row in matrix):
        raise ValueError("matrix/vector dimensions don't match")
    return [sum(a * b for a, b in zip(row, vector)) for row in matrix]

def close(left: list[float], right: list[float], tol: float = 1e-9) -> bool:
    return len(left) == len(right) and all(abs(a - b) <= tol for a, b in zip(left, right))

class LoRALinear:
    def __init__(self, weight: list[list[float]], rank: int, alpha: float):
        if not weight or not weight[0] or any(len(row) != len(weight[0]) for row in weight):
            raise ValueError("weight must be nonempty and rectangular")
        if type(rank) is not int or rank <= 0:
            raise ValueError("rank must be a positive integer")
        d_out = len(weight)
        d_in = len(weight[0])
        self.weight = [row[:] for row in weight]
        self.A = [
            [0.02 * ((i * d_in + j) % 7 - 3) for j in range(d_in)]
            for i in range(rank)
        ]
        self.B = [[0.0 for _ in range(rank)] for _ in range(d_out)]
        self.scaling = alpha / rank

    def delta_w(self) -> list[list[float]]:
        d_out = len(self.B)
        d_in = len(self.A[0])
        rank = len(self.A)
        return [
            [
                self.scaling * sum(self.B[i][k] * self.A[k][j] for k in range(rank))
                for j in range(d_in)
            ]
            for i in range(d_out)
        ]

    def forward(self, x: list[float]) -> list[float]:
        base = matvec(self.weight, x)
        adapter = [self.scaling * v for v in matvec(self.B, matvec(self.A, x))]
        return [u + v for u, v in zip(base, adapter)]

weight = [
    [1.0, 0.0, -0.5, 0.25],
    [0.0, 1.0, 0.5, -0.25],
    [0.5, -0.5, 1.0, 0.0],
]
layer = LoRALinear(weight, rank=2, alpha=4)
x = [1.0, -0.5, 0.25, 2.0]
base_out = matvec(weight, x)
starts_as_noop = close(layer.forward(x), base_out)

layer.B[0][0] = 1.0
layer.B[2][1] = -0.5
moved = layer.forward(x)
adapter_changed_output = not close(moved, base_out)

merged = [
    [w + d for w, d in zip(w_row, d_row)]
    for w_row, d_row in zip(weight, layer.delta_w())
]
merge_matches = close(matvec(merged, x), moved)
assert starts_as_noop and adapter_changed_output and merge_matches
assert close(matvec(layer.delta_w(), x), [v - b for v, b in zip(moved, base_out)])

print(f"starts_as_noop={starts_as_noop}")
print(f"adapter_changed_output={adapter_changed_output}")
print(f"merge_matches={merge_matches}")
print(f"base_out={[round(v, 4) for v in base_out]}")
print(f"adapted_out={[round(v, 4) for v in moved]}")

starts_as_noop=True
adapter_changed_output=True
merge_matches=True
base_out=[1.375, -0.875, 1.0]
adapted_out=[1.285, -0.875, 1.105]

Choose rank from evidence

policy-lora-v1 now has a working adapter path. The next question is whether rank 8 is enough or rank 64 is worth its cost. Rank $r$ controls adapter capacity, but module coverage, data quality, and learning rate can matter just as much. Hold target modules fixed and adapter parameters scale linearly with $r$:

Use a held-out task set for the new policy and a regression set for capabilities you need to retain. A rank that lowers training loss while degrading general behavior isn't a better adapter. Look for the failure shape before buying capacity: underfit policy examples suggest a capacity or coverage problem, while regression suggests data, scale, or target-selection trouble.

reference_rank = 16
reference_adapter_parameters = 1_249_280  # toy all-linear block from the earlier example

for rank in [4, 8, 16, 64, 256]:
    params = reference_adapter_parameters * rank // reference_rank
    multiplier = rank / reference_rank
    print(f"r={rank:>3} params={params:>8,} relative_to_r16={multiplier:.2f}x")

r=  4 params= 312,320 relative_to_r16=0.25x
r=  8 params= 624,640 relative_to_r16=0.50x
r= 16 params=1,249,280 relative_to_r16=1.00x
r= 64 params=4,997,120 relative_to_r16=4.00x
r=256 params=19,988,480 relative_to_r16=16.00x

The LoRA and QLoRA papers report strong low-rank results on their studied tasks, but that doesn't make one rank universal.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314} Establish target coverage, define the two evaluation sets, then sweep rank only when the evidence says capacity is the bottleneck.

Decide after the first controlled comparison

By now, LoRA has bought a smaller trainable state, not a guaranteed answer. For policy-lora-v1, ask two operational questions: will the team need many policy variants, and does the new data require a broad change beyond the base model's learned domain? Dataset size alone can't answer either one.

The flow below turns those questions into a starting experiment. It points toward LoRA or QLoRA when iteration cost or memory dominates, and keeps full fine-tuning in the comparison when a large domain shift and sufficient hardware make its extra capacity plausible.

If many task variants, fast iteration, or cheap deployment matter, LoRA is a strong first comparison even with a large dataset. If the target documents sit far from the pretraining mix and the hardware can update everything, full fine-tuning may justify its extra cost. policy-lora-v1 looks narrower: the model already speaks the domain, and the rotation procedure changes one escalation behavior.

• LoRA keeps the original checkpoint recoverable, but an active adapter can still regress behavior. Fewer trainable parameters constrain the update and make rollback or adapter disablement a config change. They don't guarantee that enabled-adapter responses preserve general capabilities, so evaluate policy quality and regressions together.
• Full fine-tuning has higher capacity. With a large corpus, enough memory, and a target domain far from pretraining, every eligible weight can move. Pay that cost only when held-out results show that the narrower update can't express the required behavior.

Diagnose from the failure shape

An adapter run usually tells you which axis is wrong before it tells you which hyperparameter to change. Read the symptom against the path we just built: target modules affect where behavior can move, rank affects capacity, scale affects update strength, and merge affects serving. The following cases turn that map into concrete checks.

Validation loss is flat and the model isn't learning

If training loss barely moves, adapting only $W_Q$ and $W_V$ may be too narrow for the behavior change. Compare all-linear targeting, then inspect the matched module list. QLoRA reported its full-fine-tuning match when applying adapters to all linear layers in each Transformer block, but that result belongs to its tasks and setup, not a universal target rule.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314}

Adapter cost rises with rank but held-out quality doesn't

Rank may not be the limiting axis, or ordinary $\alpha/r$ scaling may make the comparison hard to read. Hold target coverage and scaling explicit, check data formatting, and try rsLoRA for a high-rank run with use_rslora=True instead of assuming more rank is enough.^{[5]Reference 5A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA.https://arxiv.org/abs/2312.03732[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Changing rank causes overfitting or underfitting

Changing $r$ without changing $\alpha$ also changes the explicit scale. For an isolated ordinary-LoRA rank comparison, hold $\alpha/r$ constant. For an optimization search, log rank and alpha separately so a quality change has an interpretable cause. Don't present one alpha rule as universal.

Inference latency is higher than expected

If one adapter owns the deployment, a runtime adapter path may be unnecessary overhead. Evaluate a merged artifact after training: $W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$ is a standard linear layer with no separate adapter path. Keep adapters separate intentionally when serving several tasks from one resident base.

A multi-GPU job uses device_map="auto" as its sharding plan

That mixes inference dispatch with distributed training. device_map="auto" belongs to Accelerate's Big Model Inference path. Let the trainer or launcher select supported training placement, and use FSDP, DeepSpeed, or a deliberate QLoRA setup when the model doesn't fit on one training device.^{[8]Reference 8Accelerate Documentation: Big Model Inference.https://huggingface.co/docs/accelerate/usage_guides/big_modeling}

LoRA on a sharded base

Adapters don't cancel the FSDP / ZeRO contract from distributed training. Build the optimizer from requires_grad=True tensors only, wrap at block granularity, and remember that a full-shard unit still gathers frozen weights for compute. Peak memory includes that materialization plus activations, so a 2,048-token policy-lora-v1 batch can OOM even when the adapter file is tiny. Resume, preemption, and deciding whether to continue or export belong to the next chapter.

PEFT notes say "LoRA" but the method trains prompt vectors

Soft prompt tuning learns input embeddings, prefix tuning learns per-layer prefix state, and LoRA learns low-rank updates inside weight matrices. Name the method by what trains. If the learned parameters don't form $A$ and $B$ inside a projection layer, it isn't LoRA.

New special tokens never appear in generated output

Adding tokenizer IDs doesn't resize model tensors. Call resize_token_embeddings(len(tokenizer)) before wrapping the model, then inspect input and output shapes. Resizing creates new rows; it doesn't train useful representations for them. all-linear excludes lm_head. For untied embeddings, train and save both the affected input embeddings and output head, or configure supported row-selective training. For tied weights, verify that the tying survives wrapping and saving. PEFT's trainable_token_indices support and weight-tying options are version-sensitive; inspect their behavior rather than assuming attention adapters learn new lexical rows.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Shape checks fail only at runtime

$A$ and $B$ may be transposed or initialized inconsistently. For $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, use $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$ so $BA$ matches $W_0$. Assert those dimensions while wrapping modules, and zero-initialize one factor so the adapter starts as a no-op.

Production sweep starting points

When policy-lora-v1 is ready for a run, don't start by sweeping every knob. Pick one reproducible baseline, change one or two axes, and log enough context to explain a win or regression. The candidates below are starting points, not production defaults:

During training, log task evaluation and regression evaluation beside validation loss. Fewer trainable parameters don't make overfitting impossible. If training loss falls while held-out policy quality falls, inspect data cleanup and dropout before rank. If neither loss nor policy quality moves, inspect formatting and target modules before buying more capacity.

QLoRA: going further

Standard LoRA often keeps its frozen base in BF16 or FP16. That would leave a 65B base at roughly 130 GB before activations, which doesn't fit the 48 GB machine in our scenario. QLoRA (Quantized LoRA)^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314} quantizes the frozen base's supported weights to 4-bit and dequantizes as needed for computation. Other base tensors can remain in higher precision. Trainable factors also use higher precision, often FP32 in current PEFT. Gradients pass through the base computation to update adapters; the frozen weights have no parameter gradients or optimizer state.

Use consistent model sizes when comparing numbers. For a 65B model, the raw storage floor and the paper's measured feasibility statement are different kinds of evidence:

The raw 4-bit payload is only one storage term. Quantization metadata, adapters, adapter optimizer state, activations, and temporary buffers still contribute to the QLoRA training footprint. Calculate the floor below, then compare it with peak allocation from the actual run:

parameters_billion = 65
bf16_base_gb = parameters_billion * 2
nf4_raw_payload_gb = parameters_billion * 4 / 8
double_quant_overhead_bits_per_param = 8 / 64 + 32 / (64 * 256)
nf4_plus_constants_gb = parameters_billion * (4 + double_quant_overhead_bits_per_param) / 8

print(f"bf16_frozen_base_floor_GB={bf16_base_gb:.1f}")
print(f"nf4_raw_payload_GB={nf4_raw_payload_gb:.1f}")
print(f"nf4_plus_quant_constants_GB={nf4_plus_constants_gb:.1f}")
print("paper_device_capacity_GB=48")

bf16_frozen_base_floor_GB=130.0
nf4_raw_payload_GB=32.5
nf4_plus_quant_constants_GB=33.5
paper_device_capacity_GB=48

The output is a payload estimate, not a fit guarantee. Activations still scale with sequence length and micro-batch size, so a long-context QLoRA run can OOM even when the quantized base fits.

QLoRA combined three memory techniques. In the paper's evaluated tasks, that combination preserved 16-bit fine-tuning performance while making 4-bit adapter training practical:

1. NormalFloat4 (NF4): A 4-bit index selects one of sixteen nonuniform values, chosen using a normal-distribution model and normalized to $[-1,1]$. A block's absolute maximum rescales its weights into that range. The actual codebook has seven negative values, zero, and eight positive values. It isn't a sign/exponent/mantissa encoding or a guarantee of minimum MSE for arbitrary weights. Constructing levels directly from the normal inverse CDF's $Q(0)$ and $Q(1)$ would give infinite endpoints; use the finite codebook instead.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314[10]Reference 10bitsandbytes: NF4 Codebook and Nested Scale Quantization (8336490)https://github.com/bitsandbytes-foundation/bitsandbytes/blob/833649043474794b8fe7a4136e0c40faf077b2e0/bitsandbytes/functional.py}
2. Double quantization: One FP32 scale per 64 weights costs $32/64=0.5$ bits per weight. Quantizing those scales to eight-bit values, with an FP32 scale per 256 of them, gives $8/64+32/(64\cdot256)\approx0.127$ bits per weight. The paper calls its scale format eight-bit floats; current bitsandbytes stores indices into an eight-bit codebook and also keeps a centering offset. This large-tensor estimate saves about 3.03 decimal GB at 65B, excluding small extra metadata and incomplete blocks.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314[10]Reference 10bitsandbytes: NF4 Codebook and Nested Scale Quantization (8336490)https://github.com/bitsandbytes-foundation/bitsandbytes/blob/833649043474794b8fe7a4136e0c40faf077b2e0/bitsandbytes/functional.py}
3. Paged optimizers: CUDA managed memory lets optimizer-state pages migrate between GPU and host under memory pressure. This differs from ordinary pinned host buffers. Paging can make room during activation peaks, but doesn't eliminate all OOM causes and transfers can slow the iteration. It doesn't require the backward computation and optimizer step to run simultaneously. The paper's reported equal speed at batch size 16 belongs to its 65B/48 GB comparison.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314[11]Reference 11bitsandbytes Optimizers: Paged Optimizershttps://huggingface.co/docs/bitsandbytes/explanations/optimizers}

Trace one normalized block with a rounded copy of the published NF4 codebook. Can positive and negative values use exactly symmetric levels? Does zero survive? This cell checks lookup arithmetic, not packing, double quantization, a CUDA kernel, or task accuracy:

codebook = [-1., -.696193, -.525073, -.394917, -.284441, -.184773, -.091050,
            0., .079580, .160930, .246112, .337915, .440710, .562617, .722957, 1.]
weights = [-.2, 0., .2, 2.]
scale = max(abs(value) for value in weights)
indices = [min(range(16), key=lambda i: abs(value / scale - codebook[i]))
           for value in weights]
restored = [scale * codebook[i] for i in indices]
assert len(codebook) == 16 and indices[1] == 7 and restored[1] == 0
assert sum(value < 0 for value in codebook) == 7
assert sum(value > 0 for value in codebook) == 8
print("four_bit_indices=", indices)
print("dequantized_weights=", [round(value, 5) for value in restored])

four_bit_indices= [6, 7, 8, 15]
dequantized_weights= [-0.1821, 0.0, 0.15916, 2.0]

Keep four precision choices separate: packed base storage, dequantized base computation, adapter parameter storage, and optimizer state. The paper uses BF16 computation; the fragment below requests that compute dtype but doesn't force every tensor to BF16. PEFT's k-bit preparation can upcast unquantized tensors, and adapter promotion can leave factors in FP32.^{[9]Reference 9PEFT Troubleshooting: Adapter Dtypeshttps://huggingface.co/docs/peft/developer_guides/troubleshooting[12]Reference 12PEFT 0.20.0: prepare_model_for_kbit_traininghttps://github.com/huggingface/peft/blob/a5526d27a9d47d1e8264d5e1b1f96c0fdc79464e/src/peft/utils/other.py}

Dequantization's temporary storage depends on the implementation. The pinned current bitsandbytes backward path calls dequantize_4bit to materialize a weight matrix before computing input gradients. Don't assume the entire training path dequantizes only into SRAM/registers or has no additional device-memory allocation.^{[13]Reference 13bitsandbytes: Four-Bit Matrix Autograd (8336490)https://github.com/bitsandbytes-foundation/bitsandbytes/blob/833649043474794b8fe7a4136e0c40faf077b2e0/bitsandbytes/autograd/_functions.py} Frozen base weights have no parameter gradients; gradients through their operations can still be needed upstream.

QLoRA is a strong candidate when a frozen BF16 or FP16 base doesn't fit on the available training hardware. It trades memory for quantization complexity. Compare policy quality and regressions against an affordable higher-precision baseline so a memory win doesn't hide a behavior loss.

The following GPU setup fragment follows PEFT's QLoRA path: quantize at load time, prepare for k-bit training, then add adapters.^{[14]Reference 14PEFT Documentation: Quantization.https://huggingface.co/docs/peft/main/developer_guides/quantization} It requires a compatible bitsandbytes installation and BF16-capable hardware. It hasn't been run as a full GPU training job here. Pin the base revision and library versions, supply the trainer/data configuration, and measure peak memory before claiming the run fits.

import torch
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B",
    quantization_config=quantization_config,
    dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(
    model,
    LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)

One export option is to reload the exact source base revision in BF16 or FP16, attach the saved adapter, merge, then optionally re-quantize. This isn't numerically identical to the training path: the adapter trained against a quantized approximation of the base. Re-evaluate the merged and re-quantized artifacts separately. Quantized merge support depends on the backend; don't assume every format supports the same operation.

Keep the base shared when serving many adapters

Training and serving pull in opposite directions. Merging makes one adapter easy to run, but it destroys the shared-base advantage as soon as another policy needs a different overlay. Keep policy-lora-v1 unmerged when several tasks should share one resident base.

When deploying fine-tuned models to production, choose between two serving contracts:

• Merged artifact: Fold $\frac{\alpha}{r}BA$ into the base and serve it through an engine that supports that architecture and dtype. There are no adapter-factor tensors or low-rank kernels in the exported model. Ordinary inference still has kernel dispatch, KV caches, and other runtime costs; merging doesn't promise zero overall latency overhead from every deployment change.
• Shared base with dynamic adapters: Route requests to their adapters while keeping the common base resident. Adapters can be loaded as needed; how many fit or run concurrently depends on their ranks, targets, dtype, request shapes, and the engine. Fifty merged 14 GB artifacts would occupy 700 GB on disk, but don't have to be resident on GPUs simultaneously. Fifty assumed 128 MB adapters plus one 14 GB base occupy 20.4 GB as artifacts; host/GPU residency is a separate calculation.

Modern multi-tenant serving frameworks make dynamic adapter routing practical through two key innovations:

1. Heterogeneous batching (Punica): The paper's custom primitive is Segmented Gather Matrix-Vector multiplication (SGMV).^{[15]Reference 15Punica: Multi-Tenant LoRA Servinghttps://arxiv.org/abs/2310.18547} Requests with different adapters can share base computation instead of requiring separate whole-model passes. With tokens stored as rows and our weight orientation, the base projection is $XW_0^T$ and adapter i adds $\frac{\alpha_i}{r_i}(X_iA_i^T)B_i^T$. Adapter selection is per request; the low-rank operations use the corresponding factors.
2. Unified Paging (S-LoRA): A common GPU pool stores both KV-cache tensors and adapter weights in non-contiguous pages. Prefetching upcoming adapters reduces loading waits; the pool reduces fragmentation from changing sequence lengths and adapter ranks.^{[16]Reference 16S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} This engine-managed paging differs from the CUDA managed-memory optimizer mechanism above. Neither paper establishes the toy artifact calculation as a GPU serving benchmark.

Predict the storage before reading the calculation: 50 merged 14 GB checkpoints should cost 700 GB, while one 14 GB base plus 50 small adapters should be much smaller.

base_checkpoint_gb = 14
adapter_mb = 128
tenant_count = 50

merged_full_models_gb = base_checkpoint_gb * tenant_count
shared_base_plus_adapters_gb = base_checkpoint_gb + adapter_mb * tenant_count / 1000

print(f"merged_full_models_GB={merged_full_models_gb:.1f}")
print(f"shared_base_plus_adapters_GB={shared_base_plus_adapters_gb:.2f}")
print(f"storage_reduction_x={merged_full_models_gb / shared_base_plus_adapters_gb:.1f}")

merged_full_models_GB=700.0
shared_base_plus_adapters_GB=20.40
storage_reduction_x=34.3

Advanced: separate magnitude from direction

A LoRA variant called DoRA (Weight-Decomposed Low-Rank Adaptation)^{[17]Reference 17DoRA: Weight-Decomposed Low-Rank Adaptation.https://arxiv.org/abs/2402.09353} asks whether separating weight magnitude from direction can improve adaptation.

For a nonzero weight vector, magnitude is $m=\|w\|_2$ and unit direction is $v=w/\|w\|_2$. Liu et al.'s VL-BART query/value case study compared changes in these quantities across layers and checkpoints:

• Full fine-tuning showed correlation -0.62 between the study's magnitude-change and directional-change statistics.
• LoRA showed 0.83, and DoRA showed -0.31.

Those are measured patterns in that case study, not universal correlations or proof that an additive update cannot rotate a vector without changing its norm. Full fine-tuning also applies additive weight updates; DoRA makes the magnitude parameter explicit.

DoRA resolves this by explicitly factorizing each weight into a learnable magnitude vector and a directional matrix updated via low-rank adaptation. For a PyTorch linear weight $W_0 \in \mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$, the row-wise formulation is:

$V = W_0 + \frac{\alpha}{r}BA, \qquad W_{\text{DoRA}} = m \odot \frac{V}{\|V\|_{\text{row}}}$

where:

• $V \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ is the directional matrix carrying the standard LoRA update.
• $\|\cdot\|_{\text{row}}$ takes the row-wise $L_2$ norm (dim=1 in PyTorch). The paper uses the transposed, column-vector convention; don't transfer its axis label unchanged to this layout.
• $m \in \mathbb{R}^{d_{\text{out}}}$ is initialized to $\|W_0\|_{\text{row}}$ and broadcast as $(d_{\text{out}},1)$. The displayed divisions assume nonzero row norms.
• $\odot$ denotes element-wise broadcast multiplication across rows.

This design preserves the initialization invariant: on step zero, $B = 0 \implies V = W_0$. Because $m^{(0)} = \|W_0\|_{\text{row}}$, the initial weight is: $W^{(0)} = \|W_0\|_{\text{row}} \odot \frac{W_0}{\|W_0\|_{\text{row}}} = W_0$ The model begins with zero perturbation to the pretrained base weights.

The forward equation doesn't specify the whole training algorithm. The paper's Section 4.3 and PEFT 0.20.0 detach the denominator's norm from the gradient graph: its value is recomputed from the updated direction, but backward treats that value as constant. Differentiating the displayed normalization without this stop-gradient gives a different training rule.^{[17]Reference 17DoRA: Weight-Decomposed Low-Rank Adaptation.https://arxiv.org/abs/2402.09353[18]Reference 18PEFT DoRA implementationhttps://github.com/huggingface/peft/blob/a5526d27a9d47d1e8264d5e1b1f96c0fdc79464e/src/peft/tuners/lora/dora.py}

Make that distinction measurable. With one row $v=(3,4)$, magnitude $m=5$, and input $x=(1,0)$, both forward paths output 3. Differentiating $5v_1/\|v\|$ normally gives $(0.64,-0.48)$. Treating the denominator as constant gives $(1,0)$. The following PEFT check sets $A=I$ and $B=0$, so B's gradient exposes that difference directly:

import torch
from torch import nn
from peft import LoraConfig, get_peft_model

base = nn.Sequential(nn.Linear(2, 1, bias=False))
with torch.no_grad():
    base[0].weight.copy_(torch.tensor([[3., 4.]]))
model = get_peft_model(base, LoraConfig(
    r=2, lora_alpha=2, target_modules=["0"], use_dora=True,
))
layer = model.base_model.model[0]
with torch.no_grad():
    layer.lora_A["default"].weight.copy_(torch.eye(2))
    layer.lora_B["default"].weight.zero_()
output = model(torch.tensor([[1., 0.]]))
output.sum().backward()
peft_gradient = layer.lora_B["default"].weight.grad.flatten()

direction = torch.tensor([3., 4.], requires_grad=True)
normalized_output = 5 * direction[0] / direction.norm()
full_gradient, = torch.autograd.grad(normalized_output, direction)
torch.testing.assert_close(output.flatten(), normalized_output.detach().view(1))
torch.testing.assert_close(peft_gradient, torch.tensor([1., 0.]))
torch.testing.assert_close(full_gradient, torch.tensor([.64, -.48]))
print("same forward output:", round(output.item(), 2))
print("full-normalization gradient:", [round(v, 2) for v in full_gradient.tolist()])
print("PEFT detached-norm gradient:", [round(v, 2) for v in peft_gradient.tolist()])

same forward output: 3.0
full-normalization gradient: [0.64, -0.48]
PEFT detached-norm gradient: [1.0, 0.0]

Matching forward values doesn't prove that a handwritten DoRA implementation trains like PEFT. The detached denominator can change the directional matrix's norm during an update; the next forward recomputes that norm. The explicit magnitude still controls the normalized row's length.

During training, $BA$ changes the directional matrix while $m$ separately controls row magnitudes. When training concludes, a supported DoRA adapter can be merged into one dense matrix, removing the adapter and normalization branches from inference: $W_{\text{merged}} = m \odot \frac{W_0 + \frac{\alpha}{r}BA}{\|W_0 + \frac{\alpha}{r}BA\|_{\text{row}}}$

PEFT exposes DoRA with use_dora=True.^{[18]Reference 18PEFT DoRA implementationhttps://github.com/huggingface/peft/blob/a5526d27a9d47d1e8264d5e1b1f96c0fdc79464e/src/peft/tuners/lora/dora.py} For a square layer ($d_{\text{in}} = d_{\text{out}} = d$), adding $d$ magnitude scalars adds only $d / (2rd) = 1/(2r)$ extra parameters over standard LoRA (just 3.12% extra trainable parameters at $r=16$). Before merging, training requires additional normalization steps, so evaluate both quality gains and training throughput.

import math

d_in = 6
d_out = 4
rank = 2
lora_params = rank * (d_in + d_out)
dora_magnitude_params = d_out

# A rectangular weight makes a row/column mix-up visible.
weight = [
    [1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
    [0.0, 2.0, 0.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 3.0, 0.0, 0.0, 0.0],
    [0.0, 0.0, 0.0, 4.0, 1.0, 1.0],
]
row_norms = [math.sqrt(sum(value ** 2 for value in row)) for row in weight]
assert d_in != d_out
assert len(row_norms) == d_out
assert dora_magnitude_params == len(row_norms)
assert len(weight) == d_out
assert all(len(row) == d_in for row in weight)

square_d_in = square_d_out = 4096
square_rank = 16
square_lora_params = square_rank * (square_d_in + square_d_out)
square_dora_magnitude_params = square_d_out

print("lora_adapter_params=", lora_params)
print("dora_extra_magnitude_params=", dora_magnitude_params)
print(f"dora_extra_vs_lora={dora_magnitude_params / lora_params:.2%}")
print("fixture_weight_shape=", (d_out, d_in))
print("fixture_magnitude_broadcast_shape=", (len(row_norms), 1))
print("fixture_row_norms=", [round(value, 3) for value in row_norms])
print("square_lora_adapter_params=", square_lora_params)
print("square_dora_extra_magnitude_params=", square_dora_magnitude_params)
print(f"square_dora_extra_vs_lora={square_dora_magnitude_params / square_lora_params:.2%}")

lora_adapter_params= 20
dora_extra_magnitude_params= 4
dora_extra_vs_lora=20.00%
fixture_weight_shape= (4, 6)
fixture_magnitude_broadcast_shape= (4, 1)
fixture_row_norms= [1.0, 2.0, 3.0, 4.243]
square_lora_adapter_params= 131072
square_dora_extra_magnitude_params= 4096
square_dora_extra_vs_lora=3.12%

The rectangular count makes the dimension visible: rank 2 on a four-by-six PyTorch weight has 20 LoRA parameters and four magnitude scalars. Its final row norm is $\sqrt{18}$, not 4, because the whole row contributes. The square example alone would hide a transposed-axis bug.

Check that axis against the same PEFT version used above. A six-input, four-output linear layer should expose four magnitude parameters:

import torch
from torch import nn
from peft import LoraConfig, get_peft_model

base = nn.Sequential(nn.Linear(6, 4, bias=False))
model = get_peft_model(base, LoraConfig(
    r=2, lora_alpha=2, target_modules=["0"], use_dora=True,
))
layer = model.base_model.model[0]
magnitude = layer.lora_magnitude_vector["default"].weight
assert magnitude.shape == (4,)
torch.testing.assert_close(magnitude, layer.base_layer.weight.norm(dim=1))
print("stored magnitude shape:", tuple(magnitude.shape))

stored magnitude shape: (4,)

Practice: change one constraint

1. In the CPU fixture, replace all-linear with Q/V targeting. Predict 64 + 48 = 112 trainable parameters before changing the assertion. The base weight count should stay unchanged.
2. Set both LoRA factors to zero before the backward pass. Confirm neither learns. Restore random A and zero B, then verify A receives a gradient on the second optimization step.
3. Re-run merge equivalence after nonzero adapter updates. Save and reload a separate adapter against a fresh copy of the exact base; compare logits with dropout disabled.
4. For a real model, budget frozen weights, trainable state, and activation memory separately. No result above establishes that a 7B or 65B GPU run fits or improves policy quality.

What to check before moving on