LoRA & Parameter-Efficient Tuning

Distributed training showed how to make full-model updates fit by spreading weights, gradients, and optimizer state across GPUs. LoRA asks a different question: when the behavior change is narrow, can you avoid updating the full model at all?

Suppose a warehouse support model already handles returns, but a new carrier policy changes how damaged items should be escalated. The model doesn't need to relearn language, invoices, or general support behavior. It needs a targeted update for one slice of behavior.

That is the problem LoRA solves. It adapts a large model without updating every weight. This lesson explains low-rank adapters, where they attach, how they train, and why they are useful for domain-specific behavior.

Full fine-tuning updates every connection in the model, which is slow, expensive, and memory-heavy. LoRA (Low-Rank Adaptation) freezes the pretrained model and adds a small set of trainable adapter weights that modify its behavior. In practice, this often cuts the trainable parameter count by orders of magnitude. In the original GPT-3 175B experiments, LoRA reduced trainable parameters by 10,000x and GPU memory by about 3x relative to full fine-tuning.^[1]

Why LoRA exists

Full fine-tuning a 70-billion parameter model is expensive because training stores more than model weights. It also needs gradients and optimizer state for every trainable parameter. Start by naming the accounting recipe: this table assumes low-precision parameters and gradients with two FP32 Adam moment tensors, but no separate FP32 master parameter copy.

Here's the memory breakdown for a 70B model using Adam with FP16 (half-precision) weights:

This is only the model-state part of training memory. Real runs also pay for activations. A training stack that keeps a separate FP32 master copy adds another 280 GB, producing the 1.12 TB, 16-byte recipe from the distributed-training lesson.

For most teams, this makes full fine-tuning impractical. LoRA builds on the empirical observation from Aghajanyan et al. that the update directions needed during fine-tuning often live in a surprisingly low-dimensional subspace of the full parameter space.^[2] In other words, you don't need to adjust every single weight independently. Most of the task-specific signal can be captured by changing the weights along a much smaller number of "important directions." LoRA exploits exactly that structure by learning the update as a product of two thin matrices instead of one giant matrix.

What LoRA saves, and what it doesn't

LoRA saves memory mostly by eliminating gradients and optimizer state for the frozen base model. That is the big win. The frozen weights still need to stay in memory, and the adapters carry their own small gradient and optimizer state.

That means activation memory is not eliminated. Activations (the intermediate tensors computed during the forward pass) still scale with sequence length, batch size, and model depth, so you can still run out of memory on long contexts even with tiny adapters. Freezing weights can change exactly which tensors autograd retains, so do not claim activation bytes are identical without measuring your implementation. For long contexts, evaluate gradient checkpointing, smaller micro-batches, and memory-efficient attention kernels alongside LoRA.

Key Insight: LoRA removes base-weight gradient and optimizer-state storage, not the need to execute the full base network. A batch of 2,048-token warehouse support tickets can still be dominated by activations. Measure peak memory under the intended context length and micro-batch size.

The visual below switches from the 70B planning example to a 65B reference point so its QLoRA column matches the demonstrated 65B run from the QLoRA paper.^[3]

The overlay intuition

Think of full fine-tuning as rebuilding an entire fulfillment routing table because you need one new carrier rule. It's expensive, slow, and you might accidentally change routes that already worked.

LoRA is like keeping the original routing table frozen but adding a small overlay for the rows that need adaptation. Requests still flow through the original table, with the overlay adding a compact correction.

• The base table: The frozen pre-trained weights ($W_0$).
• The overlay: The low-rank matrices ($A$ and $B$).
• The final route: The sum of both ($W_0 + \frac{\alpha}{r}BA$).

The mathematical term for this is low-rank decomposition. Even though a weight matrix is large, the change needed to learn a new task can have compact structure. LoRA represents that update by multiplying two narrow matrices together instead of learning one full-size update matrix.

The math: from intuition to formula

A worked example with small numbers

Before we write the general formula, work through a concrete example. Suppose a weight matrix $W$ has dimensions $4096 \times 4096$ (a typical Transformer projection layer).

Full fine-tuning would train $4096 \times 4096 = 16,777,216$ parameters.

With LoRA rank $r = 16$ (the low-rank dimension of the adapter update):

• Matrix $A$ has shape $16 \times 4096$ and has $65,536$ parameters.
• Matrix $B$ has shape $4096 \times 16$ and has $65,536$ parameters.
• Total LoRA parameters: $131,072$

That's 0.78% of the full matrix size, a 99.2% reduction.

Here's how the forward pass changes. For an input vector $x$, the original output is $W_0 x$. With LoRA, the output becomes:

$h = W_0 x + \frac{\alpha}{r} B A x$

The first term is the frozen base matrix. The second term is the low-rank overlay, scaled by $\alpha / r$, where $\alpha$ is a scaling hyperparameter that controls the adapter's strength.

Low-rank decomposition

Instead of learning a full update matrix $\Delta W$ with the same dimensions as the original weights ($d_{\text{out}} \times d_{\text{in}}$), LoRA factorizes it into two smaller matrices:

$\Delta W = B \cdot A$

where:

• $A \in \mathbb{R}^{r \times d_{\text{in}}}$: the first matrix (maps from input to rank $r$)
• $B \in \mathbb{R}^{d_{\text{out}} \times r}$: the second matrix (maps from rank to output)
• $r$ is the rank, a tiny number (typically 8, 16, or 64) compared to the model's dimensions (often 4096 or more)^[1]

The figure below shows the split between the frozen base path and the trainable adapter path.

Parameter savings: concrete calculation

Let's verify the savings for a realistic projection layer.

$\text{LoRA params} = r \cdot d_{\text{in}} + d_{\text{out}} \cdot r = 2r \cdot d$

Where $r$ is LoRA rank, and $d_{\text{in}}, d_{\text{out}}$ are matrix dimensions (equal to $d$ for square projection layers).

For $r=16$, the adapter trains less than 1% as many parameters as the full matrix. That is the core tradeoff: you give the task a constrained update space, and in return you remove most gradient and optimizer-state memory for that layer.

Initialization strategy

This is a subtle but important detail. The adapter path should start as a no-op. If both $A$ and $B$ started with arbitrary non-zero values, the model would begin with random noise added to its predictions and drift away from the pretrained checkpoint.

In the original LoRA setup, one factor is initialized randomly and the other is initialized to zero.^[1] A common pattern is:

• $A$ starts with small random values.
• $B$ starts at zero.

This ensures that at the very start of training, $B \cdot A = 0$, so $\Delta W = 0$. The model starts with the same behavior as the pretrained model and gradually learns the adaptation.

Common Mistake: Initializing both $A$ and $B$ with small random values because "random init worked for the base model." If both matrices start non-zero, the first forward pass outputs pretrained predictions plus random noise, and the model immediately drifts from its checkpoint. Always zero-initialize one factor.

Understanding alpha ($\alpha$)

The scaling factor $\frac{\alpha}{r}$ matters. It acts as a multiplier for the adapter's contribution:

$\text{scaling} = \frac{\alpha}{r}$

• $r$ (rank): Determines the capacity of the adapter.
• $\alpha$ (alpha): Determines the strength of the adapter's signal.

For ordinary LoRA scaling, $\alpha=r$ and $\alpha=2r$ produce multipliers of 1 and 2. Holding $\alpha/r$ constant is useful when you want a rank comparison without also changing this explicit multiplier, but it does not guarantee identical learned update magnitude. Treat rank, scaling, learning rate, and target-module coverage as coupled sweep axes rather than universal defaults.

Where to inject LoRA

When adapting a Transformer with LoRA, you must decide which weight matrices will receive the adapters. The diagram below illustrates the typical structure of a Transformer block, highlighting where adapters are commonly injected during the attention and feed-forward phases.

The :::primary class marks the LoRA-adapted weights and the :::muted class marks the frozen weights. The original LoRA paper focused on the Query ($W_Q$) and Value ($W_V$) projections in attention.^[1] QLoRA found that adapting every linear layer in the Transformer block was needed to match full-fine-tuning results in its experiments.^[3]

Common injection strategies

Depending on the complexity of your task and your compute budget, you can choose to inject LoRA adapters into a subset of the available linear layers. Here are the most common strategies used in practice:

Module names vary by architecture. Current Hugging Face PEFT exposes target_modules="all-linear" as the QLoRA-style shortcut, but you should still inspect which modules were matched on your model.^[4]

Why adapt MLP layers?

Attention layers ($W_Q, W_K, W_V, W_O$) control how tokens route information to one another. MLP (Multi-Layer Perceptron) layers, also called FFNs (Feed-Forward Networks), perform per-token feature transformations inside the block. All-linear targeting gives the adapter access to both paths, at the price of more trainable parameters; compare it against narrower targeting on held-out examples.

This toy transformer block makes the budget change visible:

Soft prompts and prefix tuning

LoRA is not the only parameter-efficient fine-tuning method. Two older but still useful PEFT families learn prompt-like state instead of adding low-rank updates inside weight matrices.

Prompt tuning trains a small set of continuous virtual tokens prepended to the input.^[5] These aren't human-readable words. They're learned embedding vectors that steer the frozen model toward a task, like adding a small learned routing slip before every warehouse ticket. Prompt tuning is simple and cheap, but it has limited capacity because the adaptation only enters through the context window.

Prefix tuning trains continuous key/value prefixes for transformer layers.^[6] Instead of changing model weights, it gives each layer learned prefix state that attention can attend to. That makes prefix tuning more expressive than plain soft prompts, because the learned control signal reaches multiple layers directly.

Use this decision rule:

Don't call every PEFT method "LoRA." LoRA modifies internal projections through low-rank adapters. Soft prompt tuning and prefix tuning leave those projections frozen and learn continuous prompt-like state instead.

Implementing LoRA with Hugging Face PEFT

The peft (Parameter-Efficient Fine-Tuning) library makes this easy. You don't need to write the matrix multiplication manually. The following code demonstrates how to apply LoRA to an official Qwen2.5 base model. It uses the common all-linear starting point for dense decoder models.

This is a real training setup fragment, not a tiny local smoke test. To run it, install torch, transformers, peft, and accelerate, and use a machine that can load the target model. Let your trainer or distributed launcher place trainable models; Accelerate documents device_map="auto" as a big-model inference dispatch path rather than a distributed-training strategy.^[7]

When you run print_trainable_parameters(), you see trainable params, all params, and percentage for the exact model and module selection. Record it with each run; a shortcut is only useful if it matched the intended projection layers.

Merging the adapter to remove the adapter path

For a single-adapter deployment, you often merge the adapter back into the base weights. This removes the small runtime cost of the adapter path and produces a single, ordinary model checkpoint you can serve with any inference engine.

The merge_and_unload() call computes $W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$ for every adapted linear layer and returns a base-model artifact with the adapter update folded into its weights.^[4] The resulting artifact uses a standard dense layer path: no separate adapter matrix multiply remains, and the adapted behavior is baked into the weights. Keep adapters separate instead when one resident base must switch among many tasks or tenants.

Under the hood: the forward pass

To understand what peft is doing, here is a simplified implementation of a LoRA layer in PyTorch. This custom linear layer replaces a standard nn.Linear module. It takes an input tensor x, computes both the frozen base model projection and the trainable low-rank adaptation, and returns the combined output scaled by alpha.

Choosing the right rank

Rank $r$ controls adapter capacity, but module coverage, data quality, and learning rate can matter as much as rank. Holding the target modules fixed, adapter parameters scale linearly with $r$:

Use a held-out task set and a regression set for capabilities you need to retain. A rank that lowers training loss while degrading general behavior is not a better adapter.

The LoRA and QLoRA papers report strong results at low ranks in their studied tasks, but that does not make one rank universal.^[1][3] Establish module coverage first, then sweep rank only as evaluation evidence demands.

LoRA vs full fine-tuning: when to choose which

Deciding between full fine-tuning and parameter-efficient methods depends less on raw sample count and more on domain shift, GPU budget, and whether you need one model or many variants.

Choosing between LoRA and full fine-tuning (FFT) is a tradeoff between resource efficiency and model capacity.

Raw sample count is a weak proxy. Five thousand long conversations can contain more total tokens than fifty thousand short prompts, and serving requirements often matter more than dataset size anyway.

If you need many task variants, fast iteration, or cheap deployment, LoRA often still wins even with a fairly large dataset. If the target domain is radically different from pretraining and you have enough memory to update everything, full fine-tuning can warrant its extra cost.

• LoRA keeps the original base checkpoint recoverable, but an active adapter can still regress behavior. Because fewer parameters train, LoRA constrains update capacity and makes rollback or adapter disablement straightforward. It does not guarantee that responses with the adapter enabled preserve general capabilities. Evaluate both target-task quality and regression behavior.
• Full fine-tuning has higher capacity. If you have a very large corpus, enough memory, and a target domain that's radically different from pre-training (for example, moving from web text to highly specialized logistics documents with custom entity schemas and multi-step fulfillment rules), full fine-tuning lets the model reshape every weight to capture those new patterns. In such cases the extra cost can be justified.

Common mistakes and how to fix them

Symptom: Validation loss is flat and the model isn't learning

Cause: You might be adapting only $W_Q$ and $W_V$ when the task needs deeper behavioral changes. Fix: Compare all-linear targeting. QLoRA reported its full-fine-tuning match when applying adapters to all linear layers in each Transformer block.^[3] Inspect the matched module list so you know what the shortcut actually selected.

Symptom: Adapter cost rises with rank but held-out quality does not

Cause: Rank is not the limiting axis, or ordinary $\alpha/r$ scaling has made the comparison harder to interpret. Fix: Hold target coverage and the scaling policy explicit while comparing ranks. Check data and module coverage before increasing rank. For high-rank experiments, rsLoRA (rank-stabilized LoRA) changes scaling from $\alpha / r$ to $\alpha / \sqrt{r}$ to improve high-rank optimization behavior.^[8][4]

Symptom: You changed rank and now the model overfits or underfits

Cause: Mismatched alpha. If you change $r$ without changing $\alpha$, the effective scaling changes. Fix: For an isolated ordinary-LoRA rank comparison, hold $\alpha/r$ constant; for an optimization search, track rank and alpha as separate parameters. Do not present one alpha rule as universal.

Symptom: Inference latency is higher than expected in production

Cause: You are applying an adapter path at runtime in a deployment that only needs one adapter. Fix: For a single-adapter deployment, evaluate merging after training. The merged weights are $W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$, a standard linear layer with no adapter-path overhead. Keep adapters separate intentionally for multi-adapter serving.

Symptom: A multi-GPU training job uses device_map="auto" as its sharding plan

Cause: Inference dispatch and distributed training were confused. device_map="auto" is documented in Accelerate's Big Model Inference path. Fix: Let the trainer or launcher select supported training placement, and use FSDP, DeepSpeed, or a deliberate QLoRA setup when the model does not fit on one training device.^[7]

Symptom: Your PEFT notes say "LoRA" but the method trains prompt vectors

Cause: You're mixing separate PEFT families. Soft prompt tuning learns input embeddings. Prefix tuning learns per-layer prefix state. LoRA learns low-rank updates inside weight matrices. Fix: Name the method by what trains. If the learned parameters do not form $A$ and $B$ adapter matrices inside a projection layer, it is not LoRA.

Symptom: Your implementation shape-checks only after a late runtime error

Cause: $A$ and $B$ were transposed or initialized inconsistently. For a base matrix $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, LoRA uses $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$, so $BA$ matches $W_0$. Fix: Assert dimensions when wrapping modules, and zero-initialize one factor so the adapter path starts as a no-op.

Production sweep starting points

When training LoRA models with libraries such as peft or trl, define a small first sweep and log each axis. These are candidates to test, not production defaults:

When monitoring LoRA training, track task evaluation and regression evaluation alongside validation loss. Fewer trainable parameters do not make overfitting impossible. If validation quality falls while training loss decreases, compare dropout, rank, and data cleanup. If the model does not learn the task, inspect formatting and target modules before assuming more rank is the fix.

QLoRA: going further

Standard LoRA usually keeps the frozen base model in BF16 or FP16. QLoRA (Quantized LoRA)^[3] takes this a step further by storing the frozen base model in 4-bit and dequantizing on the fly for compute. The trainable LoRA adapters ($A$ and $B$) stay in a higher precision such as BF16. Autograd must still propagate signals through base-layer computation to reach earlier adapters, but it does not allocate base-weight gradient or optimizer-state tensors. The 4-bit storage is for weights you are not training.

Use consistent model sizes when comparing numbers. For a 65B model, the raw storage floor and the paper's measured feasibility statement are different kinds of evidence:

The raw 4-bit payload is smaller than the actual QLoRA training footprint because quantization metadata, adapters, optimizer state, activations, and temporary buffers remain:

Activations still scale with sequence length and micro-batch size, so a long-context QLoRA run can still OOM even though the quantized base fits.

QLoRA introduces three key innovations to make 4-bit training possible without losing accuracy:

1. NF4 (Normal Float 4-bit): NF4 chooses quantile-based values for normally distributed weights. Under the QLoRA paper's assumptions, it is information-theoretically optimal for that distribution.
2. Double quantization: Quantization requires constants that scale blocks of 4-bit values back into compute precision. QLoRA quantizes these constants as well, reducing the average overhead by about 0.37 bits per parameter, from roughly 0.5 to 0.127 bits per parameter.^[3]
3. Paged optimizers: This feature uses Unified Memory to smooth the memory spikes that show up during long-sequence minibatches, instead of crashing as soon as optimizer state briefly exceeds GPU memory.

QLoRA is a strong candidate when the frozen BF16 or FP16 base model does not fit on available training hardware. It trades memory for quantization complexity, so compare task and regression quality against an affordable higher-precision baseline where possible.

Current PEFT's documented QLoRA-style setup quantizes the base at load time, prepares it for k-bit training, and then adds adapters:^[9]

If you want a merged deployment artifact after QLoRA training, first load the base model in BF16 or FP16, merge the adapter, and then optionally re-quantize for inference.

Multi-adapter serving

A major production advantage of LoRA is that one frozen base model can serve many task-specific adapters.

In a traditional setup, if you have 10 custom models for 10 different clients, you need to load 10 full copies of the model. With LoRA, you load the frozen base model once and keep per-task adapters separately. Adapter size depends on rank, target modules, dtype, and model width, so calculate it instead of promising a fixed number. A shared base model can serve one adapter for fashion returns, another for electronics warranties, and a third for bulk-shipping refunds while the base weights stay resident.

Two common deployment patterns dominate:

• Single-tenant: merge once ($W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$) and serve a standalone model with no separate adapter-path overhead.
• Multi-tenant: keep $W_0$ resident, load adapters on demand, and apply them during inference.

Systems like S-LoRA^[10] push this further by keeping many adapters in CPU memory, paging the active ones to GPU memory, and batching requests that target different adapters in the same serving step. That lets one base model serve thousands of tenant-specific adapters without loading thousands of full checkpoints.

This simplified storage calculation shows why separate adapters matter for multi-tenant deployments:

Advanced: DoRA (weight-decomposed LoRA)

A LoRA variant called DoRA (Weight-Decomposed LoRA)^[11] separates each pretrained weight vector into two pieces:

• a learnable magnitude term (one scalar per output channel)
• a direction term that's adapted with standard LoRA

Why this works

Standard LoRA learns one additive low-rank update $BA$. That update can change both magnitude and direction, but the same two factors must represent both effects.

DoRA explicitly factors the pretrained weight vector into its magnitude (a learnable scalar per output dimension) and its direction (a unit vector). The direction is then adapted with the usual low-rank LoRA matrices, while the magnitude scalars are trained directly. This separation gives the adapter an extra knob: it can boost or attenuate specific channels without needing to encode that scaling inside the low-rank matrices.

The DoRA paper reports improvements over plain LoRA on its evaluated tasks, especially at low ranks, while aiming to reduce the quality gap to full fine-tuning.^[11] The magnitude parameter adds one value per output channel for each adapted matrix. Because magnitude and direction can be recombined into ordinary linear weights after training, a merged single-adapter deployment can still use a dense linear path.

Current PEFT exposes DoRA through use_dora=True on LoraConfig, but its documentation warns of greater runtime overhead than LoRA and recommends evaluating merge or ephemeral-offload options for inference.^[4] Treat DoRA as a measured quality/cost experiment, not a free switch.

Check your understanding

Before moving on, make sure you can answer these questions without looking back:

1. Parameter count: For a $4096 \times 4096$ weight matrix, how many trainable parameters does LoRA with $r=16$ introduce? What percentage is that of the full matrix?

2. Initialization: Why must matrix $B$ start at zero? What would happen if both $A$ and $B$ started with random values?

3. Rank vs alpha: If you increase rank from $r=8$ to $r=32$ but keep $\alpha=16$ fixed, what happens to the explicit scaling factor $\frac{\alpha}{r}$? Can that factor alone tell you the trained adapter's final update magnitude?

4. Serving: Your production setup loads one base model and keeps 50 tenant adapters in CPU RAM, paging them to GPU on demand. A colleague suggests merging all adapters into separate full models to simplify the stack. What do you lose?

Solution sketches

1. $2 \times 16 \times 4096 = 131,072$ parameters. That's $131,072 / 16,777,216 \approx 0.78\%$ of the full matrix.

2. $B$ starts at zero so that $B \cdot A = 0$ at initialization. If both started random, the model would output pretrained predictions plus random noise from the very first forward pass, causing immediate drift.

3. The explicit scaling factor drops from $16/8 = 2.0$ to $16/32 = 0.5$. Raising $\alpha$ proportionally (for example, to $\alpha=64$) keeps that explicit multiplier fixed, but it doesn't guarantee the same learned update magnitude or quality. Compare rank and alpha with evaluation.

4. You'd lose the memory savings. Merging 50 adapters into 50 full models means loading 50 complete weight sets instead of one base plus 50 small adapter files. Adapter paging becomes impossible.

Practice checkpoints

What you should be able to defend

By the end of this lesson, you should be able to:

• State the LoRA update as $\Delta W = BA$ and give the dimensions of $A$, $B$, and $W_0$.
• Explain why the base weights freeze while only $A$ and $B$ receive gradients.
• Calculate the parameter savings for rank 16 on a $4096 \times 4096$ projection: 131,072 adapter parameters instead of 16,777,216 full-matrix parameters.
• Decide when to adapt only attention projections and when to use all-linear targeting across attention and MLP projections.
• Explain why QLoRA reduces base-weight memory with 4-bit quantization but still leaves activation memory as a training bottleneck.
• Distinguish LoRA weight adapters from soft prompt tuning and prefix tuning.

Key takeaways

• Low intrinsic rank: Fine-tuning often updates a small subspace of the model's weights. LoRA exploits this by training low-rank matrices $A$ and $B$.
• Training efficiency: LoRA often cuts trainable parameters by 100x to 10,000x, depending on rank and target modules.
• Memory caveat: Biggest savings come from optimizer states and gradients. Activation memory still matters.
• Injection strategy: Compare all-linear against a narrower baseline; it exposes more block projections at a higher adapter cost.
• Serving flexibility: Merge for single-tenant serving without a separate adapter path, or keep adapters separate for multi-tenant serving.
• Modern variants: QLoRA reduces frozen-base memory through quantization. DoRA adds a magnitude/direction parameterization whose quality and overhead must be measured.