LoRA & Parameter-Efficient Tuning

Suppose an access-policy assistant already handles routine key reviews, but a new rotation procedure changes how stale keys must be escalated. You have 5,000 labeled tickets and one 48 GB training GPU. Full fine-tuning asks that machine to hold gradients and Adam state for every base weight. A thin adapter looks affordable, but it still has to learn the new policy without damaging ordinary reviews. Which constraint fails first: memory, adaptation capacity, or regression quality?

policy-lora-v1 is the run we need. The base model already knows language and support patterns; this job should change one behavior slice. LoRA (Low-Rank Adaptation) freezes the pretrained weights and learns a small overlay inside selected linear layers. Full fine-tuning moves every eligible weight, so we're comparing a full update with a constrained update on the same base computation.

The original GPT-3 175B experiments make the pressure concrete: adapting only query and value projections at rank 4 reduced trainable parameters by 10,000x and GPU memory from 1.2 TB to 350 GB, about 3x relative to full fine-tuning with Adam.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} Those are results from one paper setup, not a promise for every checkpoint. We’ll use them to build the accounting, then test where the constraint moves.

Why LoRA exists

Full fine-tuning a 70-billion parameter model is expensive for a specific reason: training stores more than model weights. It also keeps gradients and optimizer state for every trainable parameter. Before reading the table, fix the accounting recipe. It assumes low-precision parameters and gradients with two FP32 Adam moment tensors, but no separate FP32 master parameter copy.

Consider the memory breakdown for a 70B model using Adam with FP16 (half-precision) weights:

The 840 GB total covers model state only. A real run also stores activations and temporary buffers. If the optimizer keeps a separate FP32 master copy, add another 280 GB, giving the 1.12 TB, 16-byte recipe from the distributed training lesson. Keep those categories separate: a table of parameter state isn't a peak-memory measurement.

840 GB of model state, before activations, already exceeds a single 80 GB GPU. The next question is whether the policy update really needs all 70 billion directions. Aghajanyan et al. found that useful fine-tuning changes often occupy a low-dimensional subspace of the full parameter space.^{[2]Reference 2Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.https://arxiv.org/abs/2012.13255} LoRA turns that observation into a constraint: learn the update with two thin matrices, while the full base matrix stays fixed.

What LoRA saves, and what it doesn't

LoRA removes gradients and optimizer state for the frozen base model. The base weights still occupy memory, and the adapter keeps its own small trainable state.

Run the policy example at 2,048 tokens per ticket and the apparent bargain changes. Activation memory isn't eliminated. Activations (intermediate tensors from the forward pass) still depend on sequence length, batch size, and model depth. Freezing weights can change which tensors autograd retains, so don't assume activation bytes are identical across implementations. Measure peak memory, then try checkpointing, smaller micro-batches, or memory-efficient attention if the long-context case is the failure.

Memory boundary: LoRA removes base-weight gradient and optimizer-state storage, not the need to execute the full base network. A batch of 2,048-token access-review tickets can still be dominated by activations. Measure peak memory under the intended context length and micro-batch size.

The running example uses 70B for planning, but QLoRA's published comparison uses 65B. Keep that boundary visible: under this 12-byte recipe, 65B full fine-tuning is 780 GB, while the QLoRA paper fine-tuned a 65B model on one 48 GB GPU.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} The paper result is a demonstrated end-to-end setup, not a 48 GB rule for every model, sequence length, or batch.

The bars explain why policy-lora-v1 is worth testing. They don't tell us how much behavior the overlay can express. For one layer, full fine-tuning rewrites $W_0$. LoRA keeps $W_0$ frozen and learns $\Delta W = BA$, scaled by $\alpha / r$, so the layer uses $W_0 + \frac{\alpha}{r}BA$. The next section earns each symbol with a small layer.

The math: from a 4096×4096 layer

A layer small enough to count

Suppose one Transformer projection has a $4096 \times 4096$ weight matrix. Before naming LoRA's factors, predict the budget: if the update has rank $r=16$, will it contain millions of values like the base layer, or only thousands?

Full fine-tuning would train $4096 \times 4096 = 16,777,216$ parameters. A rank-16 LoRA update instead uses:

• Matrix $A$ has shape $16 \times 4096$ and has $65,536$ parameters.
• Matrix $B$ has shape $4096 \times 16$ and has $65,536$ parameters.
• Total LoRA parameters: $131,072$

That is 0.78% of the full matrix size, a 99.2% reduction. The capacity is smaller by design; the base layer still supplies the general representation.

Now follow one input vector $x$. The frozen layer produces $W_0x$, while the adapter compresses $x$ into rank space, expands it back out, and adds that correction:

$h = W_0 x + \frac{\alpha}{r} B A x$

One path is the base response. A second path is the trainable overlay, scaled by $\alpha/r$, where $\alpha$ controls its explicit strength. Because both paths share the same input, merging them later can recover one dense matrix.

One detail decides whether the first forward pass matches the checkpoint: the adapter product must start at zero. A common construction initializes $A$ randomly and $B$ to zeros, so $BA=0$ before training. Initializing both factors randomly would perturb the pretrained behavior immediately.

Why the product is low rank

The worked layer used equal input and output widths. In general, a full update $\Delta W$ has the same dimensions as the base weights, $d_{\text{out}} \times d_{\text{in}}$. LoRA factorizes that update into two smaller matrices:

$\Delta W = B \cdot A$

Read the shapes from right to left when checking the product:

• $A \in \mathbb{R}^{r \times d_{\text{in}}}$: the first matrix (maps from input to rank $r$)
• $B \in \mathbb{R}^{d_{\text{out}} \times r}$: the second matrix (maps from rank to output)
• $r$ is the rank, a tiny number such as 8, 16, or 64 compared to the model's dimensions (often 4096 or more)^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}

Why does LoRA save parameters?

Answer

It learns two skinny matrices instead of one full-size update matrix. When $r$ is much smaller than $d_{\text{in}}$ and $d_{\text{out}}$, $A$ and $B$ contain far fewer trainable values than $\Delta W$.

In the figure, trace the dimensions: $B$ times $A$ recreates a matrix shaped like $W_0$, but only $A$ and $B$ carry gradients. The visual also keeps the forward order straight: $x$ enters $A$, then $B$, even though the written product is $BA$.

Count the saved parameters

The same arithmetic scales to any square projection. Use the table to put capacity and storage side by side before checking the count in code.

$\text{LoRA params} = r \cdot d_{\text{in}} + d_{\text{out}} \cdot r = 2r \cdot d$

Where $r$ is LoRA rank, and $d_{\text{in}}, d_{\text{out}}$ are matrix dimensions (equal to $d$ for square projection layers).

For $r=16$, the adapter trains 131K values instead of 16.7M. The task gets a rank-16 update, and that layer no longer stores a full gradient plus Adam pair for its frozen base weights.

Start from the checkpoint

The first training step should not erase the behavior we paid to pretrain. If both $A$ and $B$ start non-zero, the adapter adds a random perturbation before it has seen a label.

The original LoRA setup initializes one factor randomly and the other to zero.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} A common pattern is:

• $A$ starts with small random values.
• $B$ starts at zero.

That makes $B \cdot A = 0$ at step zero, so $\Delta W = 0$. policy-lora-v1 therefore starts with the checkpoint's behavior and learns a correction from there.

⚠️ Common mistake: Initializing both $A$ and $B$ with small random values because "random init worked for the base model." If both matrices start non-zero, the first forward pass outputs pretrained predictions plus random noise, and the model immediately drifts from its checkpoint. Always zero-initialize one factor.

Separate capacity from strength

Rank and scale answer different questions. Rank says how many independent directions the adapter can express; $\alpha/r$ says how strongly the resulting correction enters the layer:

$\text{scaling} = \frac{\alpha}{r}$

• $r$ (rank): determines adapter capacity.
• $\alpha$ (alpha): determines the explicit scale of its signal.

If rank increases but $\alpha$ stays fixed, what happens to $\alpha/r$?

Answer

It gets smaller. Each rank component contributes less scale, which helps keep the adapter update from growing only because you chose a higher rank.

For ordinary LoRA scaling, $\alpha=r$ and $\alpha=2r$ produce multipliers of 1 and 2. Holding $\alpha/r$ constant makes a rank comparison easier to interpret, but it doesn't guarantee identical learned update magnitude. Sweep rank, scale, learning rate, and target-module coverage as coupled choices rather than universal defaults.

rsLoRA (rank-stabilized LoRA) changes the multiplier from $\alpha/r$ to $\alpha/\sqrt{r}$. The rsLoRA paper argues that the original $1/r$ scaling can collapse gradients as rank grows, making higher-rank adapters learn no better than small ones. Hugging Face PEFT exposes the variant with use_rslora=True.^{[4]Reference 4A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA.https://arxiv.org/abs/2312.03732[5]Reference 5PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Where to inject LoRA

The rank tells us how much each adapter can express. We still need to choose where that capacity enters the block. Start with the narrow question: should policy-lora-v1 change only attention routing, or also the per-token transformation that follows it?

In the diagram, purple marks adapted weights and muted boxes stay frozen. The original LoRA paper focused on Query ($W_Q$) and Value ($W_V$) projections in attention, which remains PEFT's default when target_modules is unset.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685[5]Reference 5PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} QLoRA's experiments instead applied adapters to every linear layer in each Transformer block and matched its full-fine-tuning baseline on the reported tasks.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314}

Common injection strategies

Target coverage trades adapter cost for capacity. Before choosing a larger rank, compare these three ways to spend the same kind of adapter budget:

Module names vary by architecture. Current Hugging Face PEFT exposes target_modules="all-linear" as a QLoRA-style shortcut, but on a PreTrainedModel that selection excludes the output projection (lm_head). Inspect the matched module names instead of trusting the shortcut, especially after adding special tokens.^{[5]Reference 5PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Why adapt MLP layers?

Attention layers ($W_Q, W_K, W_V, W_O$) control how tokens route information to one another. MLP (Multi-Layer Perceptron) layers, also called FFNs (Feed-Forward Networks), transform each token after that routing. All-linear targeting gives the adapter access to both paths, but it costs more trainable state. If a Q/V adapter misses the new policy, test whether adding the MLP path fixes held-out examples before increasing rank everywhere.

The toy block below puts a number on that choice. With rank 16, how much more adapter state does all-linear targeting add than Q/V targeting?

When the trainable state isn't a weight update

LoRA isn't the only parameter-efficient fine-tuning method. Two older but still useful PEFT families learn prompt-like state instead of inserting low-rank updates inside weight matrices.

Prompt tuning trains a small set of continuous virtual tokens prepended to the input.^{[6]Reference 6The Power of Scale for Parameter-Efficient Prompt Tuning.https://arxiv.org/abs/2104.08691} They aren't human-readable words; they are learned embedding vectors that steer the frozen model. For policy-lora-v1, that means the control signal enters through each ticket's context window, rather than changing internal projection weights. The approach is cheap, but that entry point limits how directly it can change internal transformations.

Prefix tuning trains continuous key/value prefixes for Transformer layers.^{[7]Reference 7Prefix-Tuning: Optimizing Continuous Prompts for Generation.https://arxiv.org/abs/2101.00190} The base weights remain frozen, while each layer receives learned prefix state that attention can attend to. The control signal therefore reaches multiple layers directly, without becoming a LoRA weight update.

Choose by the path the new behavior needs to use:

Don't call every PEFT method "LoRA." LoRA modifies internal projections through low-rank adapters. Soft prompt tuning and prefix tuning leave those projections frozen and learn continuous prompt-like state instead. That distinction matters when you inspect a checkpoint or estimate what a new policy can change.

Implementing LoRA with Hugging Face PEFT

The peft (Parameter-Efficient Fine-Tuning) library wraps the factors, so you don't need to write $BAx$ by hand. Before reading the setup, predict its two important effects: all-linear should attach more adapters than Q/V targeting, and r=16, alpha=32 should make the ordinary scale $2$.

This is a real training-setup fragment, not a local smoke test. To run it, install torch, transformers, peft, and accelerate on a machine that can load the target model. Let your trainer or distributed launcher handle placement. Accelerate documents device_map="auto" for Big Model Inference, not as a distributed-training strategy.^{[8]Reference 8Accelerate Documentation: Big Model Inference.https://huggingface.co/docs/accelerate/usage_guides/big_modeling}

When you run print_trainable_parameters(), you get trainable parameters, total parameters, and the percentage for this exact checkpoint and module selection. Save that receipt with the run. A shortcut is useful only if it matched the intended projections, and all-linear won't attach an adapter to lm_head.

Merge when the serving contract is single-tenant

For a single-adapter deployment, you can merge the overlay into the base weights. That removes the extra adapter path and produces one ordinary checkpoint for inference engines that support the base architecture. Keep the separate artifact until evaluation and rollback checks pass.

The merge_and_unload() call computes $W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$ for every adapted linear layer and returns a base-model artifact with the update folded into its weights.^{[5]Reference 5PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} The result follows the ordinary dense path, with no separate adapter matrix multiply. Keep the adapters separate when one resident base must switch among tasks or tenants. Also remember that merge_and_unload() returns a model rather than mutating the original in place.

See the two paths without a framework

The short standard-library layer below strips away PEFT's module hooks. It freezes $W_0$, creates a thin $r \times d_{\text{in}}$ matrix $A$, starts $B$ at zero, and adds $\frac{\alpha}{r}BAx$. Predict its three checks: the first pass should match the base, changing $B$ should change the output, and adding $BA$ into $W_0$ should match the unmerged path.

Choose rank from evidence

policy-lora-v1 now has a working adapter path. The next question is whether rank 8 is enough or rank 64 is worth its cost. Rank $r$ controls adapter capacity, but module coverage, data quality, and learning rate can matter just as much. Hold target modules fixed and adapter parameters scale linearly with $r$:

Use a held-out task set for the new policy and a regression set for capabilities you need to retain. A rank that lowers training loss while degrading general behavior isn't a better adapter. Look for the failure shape before buying capacity: underfit policy examples suggest a capacity or coverage problem, while regression suggests data, scale, or target-selection trouble.

Your platform team wants policy-lora-v1 to cover three new incident-runbook policies. You have 5,000 labeled access tickets, each averaging 512 tokens. Would you start with $r=8$ or $r=64$?

Answer

Compare a low-cost candidate such as $r=8$ or $r=16$ before paying four times rank-16 adapter state for $r=64$. Choose from held-out policy accuracy, regression behavior, and memory/latency measurements rather than assuming a larger adapter fixes a data or prompt-format problem.

The LoRA and QLoRA papers report strong low-rank results on their studied tasks, but that doesn't make one rank universal.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} Establish target coverage, define the two evaluation sets, then sweep rank only when the evidence says capacity is the bottleneck.

Decide after the first controlled comparison

By now, LoRA has bought a smaller trainable state, not a guaranteed answer. For policy-lora-v1, ask two operational questions: will the team need many policy variants, and does the new data require a broad change beyond the base model's learned domain? Dataset size alone can't answer either one.

The flow below turns those questions into a starting experiment. It points toward LoRA or QLoRA when iteration cost or memory dominates, and keeps full fine-tuning in the comparison when a large domain shift and sufficient hardware make its extra capacity plausible.

If many task variants, fast iteration, or cheap deployment matter, LoRA is a strong first comparison even with a large dataset. If the target documents sit far from the pretraining mix and the hardware can update everything, full fine-tuning may justify its extra cost. policy-lora-v1 looks narrower: the model already speaks the domain, and the rotation procedure changes one escalation behavior.

• LoRA keeps the original checkpoint recoverable, but an active adapter can still regress behavior. Fewer trainable parameters constrain the update and make rollback or adapter disablement a config change. They don't guarantee that enabled-adapter responses preserve general capabilities, so evaluate policy quality and regressions together.
• Full fine-tuning has higher capacity. With a large corpus, enough memory, and a target domain far from pretraining, every eligible weight can move. Pay that cost only when held-out results show that the narrower update can't express the required behavior.

Diagnose from the failure shape

An adapter run usually tells you which axis is wrong before it tells you which hyperparameter to change. Read the symptom against the path we just built: target modules affect where behavior can move, rank affects capacity, scale affects update strength, and merge affects serving. The following cases turn that map into concrete checks.

Validation loss is flat and the model isn't learning

If training loss barely moves, adapting only $W_Q$ and $W_V$ may be too narrow for the behavior change. Compare all-linear targeting, then inspect the matched module list. QLoRA reported its full-fine-tuning match when applying adapters to all linear layers in each Transformer block, but that result belongs to its tasks and setup, not a universal target rule.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314}

Adapter cost rises with rank but held-out quality doesn't

Rank may not be the limiting axis, or ordinary $\alpha/r$ scaling may make the comparison hard to read. Hold target coverage and scaling explicit, check data formatting, and try rsLoRA for a high-rank run with use_rslora=True instead of assuming more rank is enough.^{[4]Reference 4A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA.https://arxiv.org/abs/2312.03732[5]Reference 5PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Changing rank causes overfitting or underfitting

Changing $r$ without changing $\alpha$ also changes the explicit scale. For an isolated ordinary-LoRA rank comparison, hold $\alpha/r$ constant. For an optimization search, log rank and alpha separately so a quality change has an interpretable cause. Don't present one alpha rule as universal.

Inference latency is higher than expected

If one adapter owns the deployment, a runtime adapter path may be unnecessary overhead. Evaluate a merged artifact after training: $W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$ is a standard linear layer with no separate adapter path. Keep adapters separate intentionally when serving several tasks from one resident base.

A multi-GPU job uses device_map="auto" as its sharding plan

That mixes inference dispatch with distributed training. device_map="auto" belongs to Accelerate's Big Model Inference path. Let the trainer or launcher select supported training placement, and use FSDP, DeepSpeed, or a deliberate QLoRA setup when the model doesn't fit on one training device.^{[8]Reference 8Accelerate Documentation: Big Model Inference.https://huggingface.co/docs/accelerate/usage_guides/big_modeling}

LoRA on a sharded base

Adapters don't cancel the FSDP / ZeRO contract from distributed training. Build the optimizer from requires_grad=True tensors only, wrap at block granularity, and remember that a full-shard unit still gathers frozen weights for compute. Peak memory includes that materialization plus activations, so a 2,048-token policy-lora-v1 batch can OOM even when the adapter file is tiny. Resume, preemption, and deciding whether to continue or export belong to the next chapter.

PEFT notes say "LoRA" but the method trains prompt vectors

Soft prompt tuning learns input embeddings, prefix tuning learns per-layer prefix state, and LoRA learns low-rank updates inside weight matrices. Name the method by what trains. If the learned parameters don't form $A$ and $B$ inside a projection layer, it isn't LoRA.

New special tokens never appear in generated output

Resizing the tokenizer while leaving lm_head frozen can create this symptom. target_modules="all-linear" skips the output projection on a Hugging Face PreTrainedModel, so an adapter can shift hidden states while a frozen head still maps them into the old vocabulary. Add lm_head (and usually embed_tokens) to modules_to_save, target them explicitly, or use PEFT trainable_token_indices for the new token IDs.^{[5]Reference 5PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Shape checks fail only at runtime

$A$ and $B$ may be transposed or initialized inconsistently. For $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, use $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$ so $BA$ matches $W_0$. Assert those dimensions while wrapping modules, and zero-initialize one factor so the adapter starts as a no-op.

Production sweep starting points

When policy-lora-v1 is ready for a run, don't start by sweeping every knob. Pick one reproducible baseline, change one or two axes, and log enough context to explain a win or regression. The candidates below are starting points, not production defaults:

During training, log task evaluation and regression evaluation beside validation loss. Fewer trainable parameters don't make overfitting impossible. If training loss falls while held-out policy quality falls, inspect data cleanup and dropout before rank. If neither loss nor policy quality moves, inspect formatting and target modules before buying more capacity.

QLoRA: going further

Standard LoRA often keeps its frozen base in BF16 or FP16. That would leave a 65B base at roughly 130 GB before activations, which doesn't fit the 48 GB machine in our scenario. QLoRA (Quantized LoRA)^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} stores the frozen base in 4-bit and dequantizes values as needed for computation. The trainable LoRA factors ($A$ and $B$) stay in a higher precision such as BF16. Gradients still pass through the base computation to update the adapters, but the frozen weights don't receive gradient or optimizer-state tensors.

Use consistent model sizes when comparing numbers. For a 65B model, the raw storage floor and the paper's measured feasibility statement are different kinds of evidence:

The raw 4-bit payload is only one storage term. Quantization metadata, adapters, adapter optimizer state, activations, and temporary buffers still contribute to the QLoRA training footprint. Calculate the floor below, then compare it with peak allocation from the actual run:

The output is a payload estimate, not a fit guarantee. Activations still scale with sequence length and micro-batch size, so a long-context QLoRA run can OOM even when the quantized base fits.

QLoRA combined three memory techniques. In the paper's evaluated tasks, that combination preserved 16-bit fine-tuning performance while making 4-bit adapter training practical:

1. NF4 (Normal Float 4-bit): NF4 chooses quantile-based values for normally distributed weights. Under the QLoRA paper's assumptions, it's information-theoretically optimal for that distribution.
2. Double quantization: Quantization needs constants that rescale blocks of 4-bit values for compute. QLoRA quantizes those constants too, reducing average overhead from $32/64 = 0.5$ bits per parameter to $8/64 + 32/(64 \cdot 256) = 0.127$ bits, a reduction of 0.373 bits per parameter.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314}
3. Paged optimizers: Unified Memory can absorb optimizer-state spikes from long-sequence minibatches. It smooths those peaks; it doesn't make an under-sized device or an unbounded context safe.

QLoRA is a strong candidate when a frozen BF16 or FP16 base doesn't fit on the available training hardware. It trades memory for quantization complexity. Compare policy quality and regressions against an affordable higher-precision baseline so a memory win doesn't hide a behavior loss.

Current PEFT's documented QLoRA-style setup quantizes the base at load time, prepares it for k-bit training, and then adds adapters:^{[9]Reference 9PEFT Documentation: Quantization.https://huggingface.co/docs/peft/main/developer_guides/quantization}

If you want a merged deployment artifact after QLoRA training, load the base model in BF16 or FP16, merge the adapter, and optionally re-quantize for inference. That sequence matters: merge against a higher-precision base, then measure the quantized serving artifact separately.

Keep the base shared when serving many adapters

Training and serving now pull in opposite directions. Merging makes one adapter easy to run, but it destroys the shared-base advantage as soon as another policy needs a different overlay. Keep policy-lora-v1 unmerged when several tasks should share one resident base.

In a merged-checkpoint setup, 50 clients means 50 full copies of the model. With LoRA, load the frozen base once and keep per-task adapters separately. Adapter size depends on rank, target modules, dtype, and model width, so calculate it instead of promising a fixed number. One resident base can serve access reviews, key-rotation exceptions, and quota-escalation cases.

Choose between two serving contracts:

• Single-tenant: merge once ($W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$) and serve a standalone model with no separate adapter-path overhead.
• Multi-tenant: keep $W_0$ resident, load adapters on demand, and apply them during inference.

Systems like S-LoRA^{[10]Reference 10S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} push this pattern further by keeping many adapters in CPU memory, paging active ones to GPU memory, and batching requests that target different adapters in the same serving step. That design can share one base across many tenant adapters, but its batching and paging behavior must still be measured.

Predict the storage before reading the calculation: 50 merged 14 GB checkpoints should cost 700 GB, while one 14 GB base plus 50 small adapters should be much smaller.

Advanced: separate magnitude from direction

A LoRA variant called DoRA (Weight-Decomposed LoRA)^{[11]Reference 11DoRA: Weight-Decomposed Low-Rank Adaptation.https://arxiv.org/abs/2402.09353} starts from a different question: what if an update needs to change how large a weight vector is and which direction it points, but ordinary LoRA entangles those changes? DoRA splits each pretrained weight into:

• a learnable magnitude vector $m$ (one scalar per column)
• a direction matrix that LoRA then adapts

Standard LoRA learns one additive update $BA$, so each column's length and direction change together. DoRA normalizes the updated direction, then restores a learned column magnitude:

$W' = m \frac{W_0 + BA}{\|W_0 + BA\|_c}$

Here $\|\cdot\|_c$ is the column-wise vector norm. $m$ starts as the column norms of $W_0$, and $B$ still starts at zero, so $W' = W_0$ at step zero. With the article's convention $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, each column contains $d_{\text{out}}$ values, but there are $d_{\text{in}}$ columns. The magnitude therefore has $d_{\text{in}}$ scalars and broadcasts as a $1 \times d_{\text{in}}$ row. A framework that stores fan-in/fan-out weights transposed may name the corresponding axis differently, so check its stored shape before mapping dimension labels. After training, magnitude and direction can be folded into an ordinary dense matrix, so a merged single-adapter deployment has no separate DoRA path.

The DoRA paper reports gains over plain LoRA on its evaluated tasks, especially at low ranks, while aiming to close the gap to full fine-tuning.^{[11]Reference 11DoRA: Weight-Decomposed Low-Rank Adaptation.https://arxiv.org/abs/2402.09353} The extra magnitude parameters are one scalar per column, which is cheap next to $2rd$ for a square layer. Current PEFT exposes DoRA with use_dora=True on LoraConfig. Its docs warn that DoRA has more runtime overhead than LoRA before merging and recommend merging for inference.^{[5]Reference 5PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} Treat it as a measured quality/cost experiment, not a free switch.

The rectangular count makes the dimension visible: rank 2 on a four-by-six weight has 20 LoRA parameters and six magnitude scalars. The square lines keep the 4096-wide overhead comparison. If d_out replaces d_in in the magnitude count or column loop, an assertion fails and the printed (1, 6) shape exposes the mistake.

Run the calculator with your model's real projection shapes, target modules, rank, dtype, and adapter count. Save its parameter and storage table beside your evaluation receipt, then compare LoRA, QLoRA, DoRA, and full SFT against one quality gate before choosing a training policy.