LoRA & Parameter-Efficient Tuning

Distributed training makes full-model updates fit by spreading weights, gradients, and optimizer state across GPUs. LoRA starts from a narrower bet: when the behavior change is small, can you avoid updating the full model at all?

Suppose an access-policy assistant already handles routine key reviews, but a new rotation policy changes how stale keys should be escalated. The model doesn't need to relearn language or general support behavior. It needs a targeted update for one slice of behavior. That's the problem LoRA solves.

Full fine-tuning can update every eligible trainable weight, which is slow, expensive, and memory-heavy. LoRA (Low-Rank Adaptation) freezes the pretrained model and adds a small set of trainable adapter weights that modify its behavior. In practice, this often cuts the trainable parameter count by orders of magnitude. In the original GPT-3 175B experiments, LoRA reduced trainable parameters by 10,000x and GPU memory by about 3x relative to full fine-tuning.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}

Why LoRA exists

Full fine-tuning a 70-billion parameter model is expensive because training stores more than model weights. It also needs gradients and optimizer state for every trainable parameter. Start by naming the accounting recipe: this table assumes low-precision parameters and gradients with two FP32 Adam moment tensors, but no separate FP32 master parameter copy.

Consider the memory breakdown for a 70B model using Adam with FP16 (half-precision) weights:

This is only the model-state part of training memory. Real runs also pay for activations. A training stack that keeps a separate FP32 master copy adds another 280 GB, producing the 1.12 TB, 16-byte recipe from the distributed-training lesson.

For most teams, this makes full fine-tuning impractical. LoRA builds on the empirical observation from Aghajanyan et al. that the update directions needed during fine-tuning often live in a surprisingly low-dimensional subspace of the full parameter space.^{[2]Reference 2Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.https://arxiv.org/abs/2012.13255} Most task-specific signal can be captured by changing the weights along a much smaller number of "important directions." LoRA exploits that structure by learning the update as a product of two thin matrices instead of one giant matrix.

What LoRA saves, and what it doesn't

LoRA saves memory mostly by eliminating gradients and optimizer state for the frozen base model. The frozen weights still need to stay in memory, and the adapters carry their own small gradient and optimizer state.

That means activation memory isn't eliminated. Activations (the intermediate tensors computed during the forward pass) still scale with sequence length, batch size, and model depth, so you can still run out of memory on long contexts even with tiny adapters. Freezing weights can change exactly which tensors autograd retains, so don't claim activation bytes are identical without measuring your implementation. For long contexts, evaluate gradient checkpointing, smaller micro-batches, and memory-efficient attention kernels alongside LoRA.

Memory boundary: LoRA removes base-weight gradient and optimizer-state storage, not the need to execute the full base network. A batch of 2,048-token access-review tickets can still be dominated by activations. Measure peak memory under the intended context length and micro-batch size.

The visual below switches from the 70B planning example to a 65B reference point so its QLoRA column matches the demonstrated 65B run from the QLoRA paper.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314}

The overlay intuition

Full fine-tuning rewrites the whole weight matrix. LoRA keeps the base weights frozen and adds a small learned overlay:

• base path: frozen pretrained weights ($W_0$)
• adapter path: low-rank matrices ($A$ and $B$)
• effective weight: $W_0 + \frac{\alpha}{r}BA$

The useful update can have compact structure even when the base matrix is large.

The math: from intuition to formula

A worked example with small numbers

Start with a concrete example before the general formula. Suppose a weight matrix $W$ has dimensions $4096 \times 4096$ (a typical Transformer projection layer).

Full fine-tuning would train $4096 \times 4096 = 16,777,216$ parameters.

With LoRA rank $r = 16$ (the low-rank dimension of the adapter update):

• Matrix $A$ has shape $16 \times 4096$ and has $65,536$ parameters.
• Matrix $B$ has shape $4096 \times 16$ and has $65,536$ parameters.
• Total LoRA parameters: $131,072$

That's 0.78% of the full matrix size, a 99.2% reduction.

The forward pass changes by adding a low-rank update. For an input vector $x$, the original output is $W_0 x$. With LoRA, the output becomes:

$h = W_0 x + \frac{\alpha}{r} B A x$

The first term is the frozen base matrix. The second term is the low-rank overlay, scaled by $\alpha / r$, where $\alpha$ is a scaling hyperparameter that controls the adapter's strength.

Low-rank decomposition

Instead of learning a full update matrix $\Delta W$ with the same dimensions as the original weights ($d_{\text{out}} \times d_{\text{in}}$), LoRA factorizes it into two smaller matrices:

$\Delta W = B \cdot A$

where:

• $A \in \mathbb{R}^{r \times d_{\text{in}}}$: the first matrix (maps from input to rank $r$)
• $B \in \mathbb{R}^{d_{\text{out}} \times r}$: the second matrix (maps from rank to output)
• $r$ is the rank, a tiny number such as 8, 16, or 64 compared to the model's dimensions (often 4096 or more)^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}

This visual shows the split between the frozen base path and the trainable adapter path.

Parameter savings: concrete calculation

Verify the savings for a realistic projection layer.

$\text{LoRA params} = r \cdot d_{\text{in}} + d_{\text{out}} \cdot r = 2r \cdot d$

Where $r$ is LoRA rank, and $d_{\text{in}}, d_{\text{out}}$ are matrix dimensions (equal to $d$ for square projection layers).

For $r=16$, the adapter trains less than 1% as many parameters as the full matrix. That's the core tradeoff: you give the task a constrained update space, and in return you remove most gradient and optimizer-state memory for that layer.

Initialization strategy

One initialization detail matters: the adapter path should start as a no-op. If both $A$ and $B$ start non-zero, the model adds random noise before training learns anything.

In the original LoRA setup, one factor is initialized randomly and the other is initialized to zero.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} A common pattern is:

• $A$ starts with small random values.
• $B$ starts at zero.

This makes $B \cdot A = 0$ at the start of training, so $\Delta W = 0$. The model begins with pretrained behavior and learns the adaptation gradually.

Common Mistake: Initializing both $A$ and $B$ with small random values because "random init worked for the base model." If both matrices start non-zero, the first forward pass outputs pretrained predictions plus random noise, and the model immediately drifts from its checkpoint. Always zero-initialize one factor.

Understanding alpha ($\alpha$)

The scaling factor $\frac{\alpha}{r}$ matters. It acts as a multiplier for the adapter's contribution:

$\text{scaling} = \frac{\alpha}{r}$

• $r$ (rank): Determines the capacity of the adapter.
• $\alpha$ (alpha): Determines the strength of the adapter's signal.

For ordinary LoRA scaling, $\alpha=r$ and $\alpha=2r$ produce multipliers of 1 and 2. Holding $\alpha/r$ constant is useful when you want a rank comparison without also changing this explicit multiplier, but it doesn't guarantee identical learned update magnitude. Treat rank, scaling, learning rate, and target-module coverage as coupled sweep axes rather than universal defaults.

Where to inject LoRA

When adapting a Transformer with LoRA, you must decide which weight matrices receive adapters. The diagram compares three common coverage choices across attention and feed-forward projections.

The :::primary class marks the LoRA-adapted weights and the :::muted class marks the frozen weights. The original LoRA paper focused on the Query ($W_Q$) and Value ($W_V$) projections in attention.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} QLoRA found that adapting every linear layer in the Transformer block was needed to match full-fine-tuning results in its experiments.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314}

Common injection strategies

Target coverage trades adapter cost for capacity. Compare these common strategies:

Module names vary by architecture. Current Hugging Face PEFT exposes target_modules="all-linear" as the QLoRA-style shortcut, but you should still inspect which modules were matched on your model.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Why adapt MLP layers?

Attention layers ($W_Q, W_K, W_V, W_O$) control how tokens route information to one another. MLP (Multi-Layer Perceptron) layers, also called FFNs (Feed-Forward Networks), perform per-token feature transformations inside the block. All-linear targeting gives the adapter access to both paths, at the price of more trainable parameters; compare it against narrower targeting on held-out examples.

This toy transformer block makes the budget change visible:

Soft prompts and prefix tuning

LoRA isn't the only parameter-efficient fine-tuning method. Two older but still useful PEFT families learn prompt-like state instead of adding low-rank updates inside weight matrices.

Prompt tuning trains a small set of continuous virtual tokens prepended to the input.^{[5]Reference 5The Power of Scale for Parameter-Efficient Prompt Tuning.https://arxiv.org/abs/2104.08691} These aren't human-readable words. They're learned embedding vectors that steer the frozen model toward a task, like adding a small learned control prefix before every access ticket. Prompt tuning is simple and cheap, but it has limited capacity because the adaptation only enters through the context window.

Prefix tuning trains continuous key/value prefixes for transformer layers.^{[6]Reference 6Prefix-Tuning: Optimizing Continuous Prompts for Generation.https://arxiv.org/abs/2101.00190} Instead of changing model weights, it gives each layer learned prefix state that attention can attend to. That makes prefix tuning more expressive than plain soft prompts, because the learned control signal reaches multiple layers directly.

Use this decision rule:

Don't call every PEFT method "LoRA." LoRA modifies internal projections through low-rank adapters. Soft prompt tuning and prefix tuning leave those projections frozen and learn continuous prompt-like state instead.

Implementing LoRA with Hugging Face PEFT

The peft (Parameter-Efficient Fine-Tuning) library handles adapter wrapping for you. You don't need to write the matrix multiplication manually. This code applies LoRA to an official Qwen2.5 base model using the common all-linear starting point for dense decoder models.

This is a real training setup fragment, not a tiny local smoke test. To run it, install torch, transformers, peft, and accelerate, and use a machine that can load the target model. Let your trainer or distributed launcher handle device placement; Accelerate documents device_map="auto" as a big-model inference dispatch path rather than a distributed-training strategy.^{[7]Reference 7Accelerate Documentation: Big Model Inference.https://huggingface.co/docs/accelerate/usage_guides/big_modeling}

When you run print_trainable_parameters(), you see trainable params, all params, and percentage for the exact model and module selection. Record it with each run; a shortcut is only useful if it matched the intended projection layers.

Merging the adapter to remove the adapter path

For a single-adapter deployment, you often merge the adapter back into the base weights. This removes the small runtime cost of the adapter path and produces a single, ordinary model checkpoint for inference engines that support the base architecture.

The merge_and_unload() call computes $W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$ for every adapted linear layer and returns a base-model artifact with the adapter update folded into its weights.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} The resulting artifact uses a standard dense layer path: no separate adapter matrix multiply remains, and the adapted behavior is baked into the weights. Keep adapters separate instead when one resident base must switch among many tasks or tenants.

Under the hood: the forward pass

To understand what peft is doing, build a simplified LoRA layer in PyTorch. This custom linear layer replaces a standard nn.Linear module. It takes an input tensor x, computes both the frozen base model projection and the trainable low-rank adaptation, scales the adapter path by $\alpha/r$, and adds the two outputs.

Choosing the right rank

Rank $r$ controls adapter capacity, but module coverage, data quality, and learning rate can matter as much as rank. Holding the target modules fixed, adapter parameters scale linearly with $r$:

Use a held-out task set and a regression set for capabilities you need to retain. A rank that lowers training loss while degrading general behavior isn't a better adapter.

The LoRA and QLoRA papers report strong results at low ranks in their studied tasks, but that doesn't make one rank universal.^{[1]Reference 1LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} Establish module coverage first, then sweep rank only as evaluation evidence demands.

LoRA vs full fine-tuning: when to choose which

Deciding between full fine-tuning and parameter-efficient methods depends less on raw sample count and more on domain shift, GPU budget, and whether you need one model or many variants.

Choosing between LoRA and full fine-tuning (FFT) is a tradeoff between resource efficiency and model capacity.

Raw sample count is a weak proxy. Five thousand long conversations can contain more total tokens than fifty thousand short prompts, and serving requirements often matter more than dataset size anyway.

If you need many task variants, fast iteration, or cheap deployment, LoRA often still wins even with a fairly large dataset. If the target domain is radically different from pretraining and you have enough memory to update everything, full fine-tuning can warrant its extra cost.

• LoRA keeps the original base checkpoint recoverable, but an active adapter can still regress behavior. Because fewer parameters train, LoRA constrains update capacity and makes rollback or adapter disablement straightforward. It doesn't guarantee that responses with the adapter enabled preserve general capabilities. Evaluate both target-task quality and regression behavior.
• Full fine-tuning has higher capacity. If you have a very large corpus, enough memory, and a target domain that's radically different from pre-training (for example, moving from web text to highly specialized security-policy documents with custom entity schemas and multi-step approval rules), full fine-tuning lets the model reshape every weight to capture those new patterns. In such cases the extra cost can be justified.

Common mistakes and how to fix them

Symptom: Validation loss is flat and the model isn't learning

• Cause: You might be adapting only $W_Q$ and $W_V$ when the task needs deeper behavioral changes.
• Fix: Compare all-linear targeting. QLoRA reported its full-fine-tuning match when applying adapters to all linear layers in each Transformer block.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} Inspect the matched module list so you know what the shortcut selected.

Symptom: Adapter cost rises with rank but held-out quality doesn't

• Cause: Rank isn't the limiting axis, or ordinary $\alpha/r$ scaling has made the comparison harder to interpret.
• Fix: Hold target coverage and the scaling policy explicit while comparing ranks. Check data and module coverage before increasing rank. For high-rank experiments, rsLoRA (rank-stabilized LoRA) changes scaling from $\alpha / r$ to $\alpha / \sqrt{r}$ to improve high-rank optimization behavior.^{[8]Reference 8A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA.https://arxiv.org/abs/2312.03732[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Symptom: You changed rank and now the model overfits or underfits

• Cause: Mismatched alpha. If you change $r$ without changing $\alpha$, the effective scaling changes.
• Fix: For an isolated ordinary-LoRA rank comparison, hold $\alpha/r$ constant; for an optimization search, track rank and alpha as separate parameters. Don't present one alpha rule as universal.

Symptom: Inference latency is higher than expected in production

• Cause: You're applying an adapter path at runtime in a deployment that only needs one adapter.
• Fix: For a single-adapter deployment, evaluate merging after training. The merged weights are $W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$, a standard linear layer with no adapter-path overhead. Keep adapters separate intentionally for multi-adapter serving.

Symptom: A multi-GPU training job uses device_map="auto" as its sharding plan

• Cause: Inference dispatch and distributed training were confused. device_map="auto" is documented in Accelerate's Big Model Inference path.
• Fix: Let the trainer or launcher select supported training placement, and use FSDP, DeepSpeed, or a deliberate QLoRA setup when the model doesn't fit on one training device.^{[7]Reference 7Accelerate Documentation: Big Model Inference.https://huggingface.co/docs/accelerate/usage_guides/big_modeling}

Symptom: Your PEFT notes say "LoRA" but the method trains prompt vectors

• Cause: You're mixing separate PEFT families: soft prompt tuning learns input embeddings, prefix tuning learns per-layer prefix state, and LoRA learns low-rank updates inside weight matrices.
• Fix: Name the method by what trains; if the learned parameters don't form $A$ and $B$ adapter matrices inside a projection layer, it isn't LoRA.

Symptom: New special tokens never appear in generated output

• Cause: You resized the tokenizer but left lm_head frozen under LoRA-only targeting. The adapter can shift hidden states, but a frozen output projection still maps them into the old vocabulary basis.
• Fix: Add lm_head (and usually embed_tokens) to modules_to_save, target them explicitly, or use PEFT trainable_token_indices for the new token IDs.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora}

Symptom: Your implementation shape-checks only after a late runtime error

• Cause: $A$ and $B$ were transposed or initialized inconsistently. For a base matrix $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, LoRA uses $A \in \mathbb{R}^{r \times d_{\text{in}}}$ and $B \in \mathbb{R}^{d_{\text{out}} \times r}$, so $BA$ matches $W_0$.
• Fix: Assert dimensions when wrapping modules, and zero-initialize one factor so the adapter path starts as a no-op.

Production sweep starting points

When training LoRA models with libraries such as peft or trl, define a small first sweep and log each axis. These are candidates to test, not production defaults:

When monitoring LoRA training, track task evaluation and regression evaluation alongside validation loss. Fewer trainable parameters don't make overfitting impossible. If validation quality falls while training loss decreases, compare dropout, rank, and data cleanup. If the model doesn't learn the task, inspect formatting and target modules before assuming more rank is the fix.

QLoRA: going further

Standard LoRA usually keeps the frozen base model in BF16 or FP16. QLoRA (Quantized LoRA)^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314} takes this a step further by storing the frozen base model in 4-bit and dequantizing on the fly for compute. The trainable LoRA adapters ($A$ and $B$) stay in a higher precision such as BF16. Autograd must still propagate signals through base-layer computation to reach earlier adapters, but it doesn't allocate base-weight gradient or optimizer-state tensors. The 4-bit storage is for weights you aren't training.

Use consistent model sizes when comparing numbers. For a 65B model, the raw storage floor and the paper's measured feasibility statement are different kinds of evidence:

The raw 4-bit payload is smaller than the actual QLoRA training footprint because quantization metadata, adapters, optimizer state, activations, and temporary buffers remain:

Activations still scale with sequence length and micro-batch size, so a long-context QLoRA run can still OOM even though the quantized base fits.

QLoRA introduced three techniques that preserved the paper's evaluated 16-bit fine-tuning performance while making 4-bit adapter training practical:

1. NF4 (Normal Float 4-bit): NF4 chooses quantile-based values for normally distributed weights. Under the QLoRA paper's assumptions, it's information-theoretically optimal for that distribution.
2. Double quantization: Quantization requires constants that scale blocks of 4-bit values back into compute precision. QLoRA quantizes these constants as well, reducing the average overhead by about 0.37 bits per parameter, from roughly 0.5 to 0.127 bits per parameter.^{[3]Reference 3QLoRA: Efficient Finetuning of Quantized Language Models.https://arxiv.org/abs/2305.14314}
3. Paged optimizers: This feature uses Unified Memory to smooth the memory spikes that show up during long-sequence minibatches, instead of crashing as soon as optimizer state briefly exceeds GPU memory.

QLoRA is a strong candidate when the frozen BF16 or FP16 base model doesn't fit on available training hardware. It trades memory for quantization complexity, so compare task and regression quality against an affordable higher-precision baseline where possible.

Current PEFT's documented QLoRA-style setup quantizes the base at load time, prepares it for k-bit training, and then adds adapters:^{[9]Reference 9PEFT Documentation: Quantization.https://huggingface.co/docs/peft/main/developer_guides/quantization}

If you want a merged deployment artifact after QLoRA training, first load the base model in BF16 or FP16, merge the adapter, and then optionally re-quantize for inference.

Multi-adapter serving

A major production advantage of LoRA is that one frozen base model can serve many task-specific adapters.

In a traditional setup, if you have 10 custom models for 10 different clients, you need to load 10 full copies of the model. With LoRA, you load the frozen base model once and keep per-task adapters separately. Adapter size depends on rank, target modules, dtype, and model width, so calculate it instead of promising a fixed number. A shared base model can serve one adapter for access reviews, another for key-rotation exceptions, and a third for quota-escalation cases while the base weights stay resident.

Two common deployment patterns dominate:

• Single-tenant: merge once ($W_{\text{merged}} = W_0 + \frac{\alpha}{r}BA$) and serve a standalone model with no separate adapter-path overhead.
• Multi-tenant: keep $W_0$ resident, load adapters on demand, and apply them during inference.

Systems like S-LoRA^{[10]Reference 10S-LoRA: Serving Thousands of Concurrent LoRA Adapters.https://arxiv.org/abs/2311.03285} push this further by keeping many adapters in CPU memory, paging the active ones to GPU memory, and batching requests that target different adapters in the same serving step. That lets one base model serve thousands of tenant-specific adapters without loading thousands of full checkpoints.

This simplified storage calculation shows why separate adapters matter for multi-tenant deployments:

Advanced: DoRA (weight-decomposed LoRA)

A LoRA variant called DoRA (Weight-Decomposed LoRA)^{[11]Reference 11DoRA: Weight-Decomposed Low-Rank Adaptation.https://arxiv.org/abs/2402.09353} separates each pretrained weight vector into two pieces:

• a learnable magnitude term (one scalar per output channel)
• a direction term that's adapted with standard LoRA

Why this works

Standard LoRA learns one additive low-rank update $BA$. That update can change both magnitude and direction, but the same two factors must represent both effects.

DoRA explicitly factors the pretrained weight vector into its magnitude (a learnable scalar per output dimension) and its direction (a unit vector). The direction is then adapted with the usual low-rank LoRA matrices, while the magnitude scalars are trained directly. This separation gives the adapter an extra knob: it can boost or attenuate specific channels without needing to encode that scaling inside the low-rank matrices.

The DoRA paper reports improvements over plain LoRA on its evaluated tasks, especially at low ranks, while aiming to reduce the quality gap to full fine-tuning.^{[11]Reference 11DoRA: Weight-Decomposed Low-Rank Adaptation.https://arxiv.org/abs/2402.09353} The magnitude parameter adds one value per output channel for each adapted matrix. Because magnitude and direction can be recombined into ordinary linear weights after training, a merged single-adapter deployment can still use a dense linear path.

Current PEFT exposes DoRA through use_dora=True on LoraConfig, but its documentation warns of greater runtime overhead than LoRA and recommends evaluating merge or ephemeral-offload options for inference.^{[4]Reference 4PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora} Treat DoRA as a measured quality/cost experiment, not a free switch.

Check your understanding

Answer these from memory before you continue:

1. Parameter count: For a $4096 \times 4096$ weight matrix, how many trainable parameters does LoRA with $r=16$ introduce? What percentage is that of the full matrix?

2. Initialization: Why does standard LoRA initialize $B$ to zero while $A$ is random? What would happen if both factors started with random values?

3. Rank vs alpha: If you increase rank from $r=8$ to $r=32$ but keep $\alpha=16$ fixed, what happens to the explicit scaling factor $\frac{\alpha}{r}$? Can that factor alone tell you the trained adapter's final update magnitude?

4. Serving: Your production setup loads one base model and keeps 50 tenant adapters in CPU RAM, paging them to GPU on demand. A colleague suggests merging all adapters into separate full models to simplify the stack. What do you lose?

Solution sketches

1. $2 \times 16 \times 4096 = 131,072$ parameters. That's $131,072 / 16,777,216 \approx 0.78\%$ of the full matrix.

2. Standard LoRA uses random $A$ and zero $B$, so $B \cdot A = 0$ and the adapter starts as a no-op while $B$ can receive a useful first-step gradient. Zero $B$ is a convention, not the only mathematical option: zero $A$ with random $B$ also makes $B \cdot A = 0$, but changes the optimization dynamics. If both factors started random, the adapter would perturb the pretrained output from the first forward pass.

3. The explicit scaling factor drops from $16/8 = 2.0$ to $16/32 = 0.5$. Raising $\alpha$ proportionally (for example, to $\alpha=64$) keeps that explicit multiplier fixed, but it doesn't guarantee the same learned update magnitude or quality. Compare rank and alpha with evaluation.

4. You'd lose the memory savings. Merging 50 adapters into 50 full models means loading 50 complete weight sets instead of one base plus 50 small adapter files. Adapter paging becomes impossible.

Practice checkpoints

What to check before moving on

Check that you can:

• State the LoRA update as $\Delta W = BA$ and give the dimensions of $A$, $B$, and $W_0$.
• Explain why the base weights freeze while only $A$ and $B$ receive gradients.
• Calculate the parameter savings for rank 16 on a $4096 \times 4096$ projection: 131,072 adapter parameters instead of 16,777,216 full-matrix parameters.
• Decide when to adapt only attention projections and when to use all-linear targeting across attention and MLP projections.
• Explain why QLoRA reduces base-weight memory with 4-bit quantization but still leaves activation memory as a training bottleneck.
• Distinguish LoRA weight adapters from soft prompt tuning and prefix tuning.

Adapter decisions to defend

• Fine-tuning often updates a small subspace of the model's weights. LoRA exploits this by training low-rank matrices $A$ and $B$.
• LoRA often cuts trainable parameters by 100x to 10,000x, depending on rank and target modules.
• The biggest savings come from optimizer states and gradients. Activation memory still matters.
• Compare all-linear against a narrower baseline; it exposes more block projections at a higher adapter cost.
• Merge for single-tenant serving without a separate adapter path, or keep adapters separate for multi-tenant serving.
• QLoRA reduces frozen-base memory through quantization. DoRA adds a magnitude/direction parameterization whose quality and overhead must be measured.

Practice drill

Create an adapter selection scorecard for one domain adaptation:

1. Compare full fine-tuning, LoRA, QLoRA, prompt tuning, and prefix tuning on trainable parameters, base memory, activation memory, serving path, and rollback cost.
2. Calculate adapter parameters for two ranks and two target-module sets.
3. Log matched modules so q_proj/v_proj or all-linear choices are explicit rather than assumed.
4. Define held-out task, regression, and latency gates before choosing rank or merge strategy.

The scorecard should prevent a lower training loss from hiding worse regression behavior or serving complexity.