Deep Dive - Light-PEFT

Your LoRA experiment can fit its trainable-parameter budget and still miss its GPU memory or latency budget. The profiler shows why: every batch still reads the frozen backbone, builds its activations, and evaluates every attention head and feed-forward coordinate. The previous chapter's hosted Tinker loop saves optimizer state, but freezing weights doesn't make their forward path disappear.

That leaves an adaptation decision: which dense coordinates can this task afford to lose, and what evidence will let you remove them without silently damaging quality? Dropping a LoRA rank makes the update smaller. It doesn't make the frozen matrix W smaller.

Light-PEFT, a parameter-efficient fine-tuning (PEFT) method, takes a deliberately sharp route. It spends a short prefix scoring heads, feed-forward dimensions, adapter modules, and adapter ranks, then physically removes the quiet coordinates before the longer fine-tuning run.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Follow one handoff through this chapter: cost ledger → task-local scores → saved records → physical slice → measured receipt. We'll use four heads, five FFN coordinates, and a rank-four adapter so each later cut has a visible owner.

We're reading the Association for Computational Linguistics (ACL) 2024 research release at the exact commit below, inspected on September 2, 2026. Here ACL names the conference, not an access control list. The equations map onto NLU/estimate.py, NLU/train.py, and the bundled PEFT fork, not the current upstream PEFT API. Paper numbers stay inside their hardware protocol; a modern serving or reinforcement-learning system still has to prove the rest.

Start with cost that PEFT leaves behind

Before the notation, make a prediction. If the base weight is frozen and adapter rank r falls, which part of a token's path gets shorter? The low-rank branch does. The dense backbone path remains unless we change its shape.

Suppose a Transformer layer receives hidden states X. With ordinary LoRA, a frozen weight W receives a low-rank update:^{[2]Reference 2LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}

$$W' = W + sAB,$$

where A maps from the hidden dimension to rank r, B maps back, and s is a scale. These equations use row-vector activations: A has shape [d_in, r] and B has shape [r, d_out]. PyTorch stores the transposes: lora_A.weight is [r, d_in] and lora_B.weight is [d_out, r]. Thus a rank cut removes mathematical A columns/B rows, or stored A rows/B columns. Only the adapter factors need optimizer updates. The layer still evaluates XW' for every token.

The phrase frozen describes parameter updates, not execution. Read one training step as a dependency chain:

1. Read frozen weights and trainable adapter weights.
2. Run forward through every surviving layer.
3. Keep activations or checkpoint enough information to reproduce them.
4. Backpropagate through the paths needed to reach trainable parameters, including frozen operations between adapters and loss.
5. Store gradients only for trainable parameters, then update the adapter.

PEFT removes a large optimizer and gradient burden. It doesn't automatically remove backbone FLOPs, backbone weight reads, or activation memory. The paper measures this directly: as OPT grows, forward time and peak memory rise even though LoRA avoids most backbone parameter gradients.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

A tiny cost ledger

Use one layer with hidden size d = 8, sequence length S = 4, and a feed-forward width of 32. Before reading the table, predict what changing r can touch: the last row only. The rough dense projection count for one token is proportional to:

The adapter may add only a small number of trainable parameters, while the frozen 768 projection pairs per token still run. For S = 4, that's 3,072 projection multiply-accumulate pairs (6,144 FLOPs if one multiplication and one addition count separately). This excludes attention-score and value aggregation products, softmax, normalization, activation, and memory movement.

If a task never uses one attention head, or if many FFN coordinates carry little task signal, PEFT alone has no reason to skip them. A task-specific structured mask can make those coordinates absent from the second run.

Why quantization answers a different question

Quantization stores weights in fewer bits, reducing their storage footprint while kernels may dequantize values for computation. Quantizing a frozen base doesn't remove additional base optimizer state: ordinary LoRA already avoids that state. Structured pruning instead changes how many rows, columns, heads, or dimensions exist. Light-PEFT studies the second lever. Its method can be combined with quantization in principle, but the paper's experiments don't establish a QLoRA combination.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/[3]Reference 3QLoRA: Efficient Finetuning of Quantized LLMshttps://arxiv.org/abs/2305.14314}

What does ordinary LoRA fail to remove?

Answer

It removes most trainable base-model state, but it still executes the frozen backbone forward and backward path. Activations, frozen weight reads, attention work, and FFN work remain unless another method changes the model shape or execution path.

A research release, not a platform

Before copying a pruning script into a training service, identify what you're holding. The source at commit 7b10c22026fe6fa283d127353a50d6ed7176a8aa is a small academic code release for the Findings of ACL 2024 paper. Its authors are Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, and Weiping Wang. Their listed affiliations are the Institute of Information Engineering, Chinese Academy of Sciences, and the School of Cyber Security, University of Chinese Academy of Sciences.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Neither the paper nor repository names a company behind the project; primary ownership evidence points to those academic institutions and the gccnlp repository account.

Paper authors and code contributors are related evidence, not interchangeable labels. The gccnlp repository account publishes this snapshot; the GitHub contributors page is a separate activity view and doesn't prove that every paper author committed every file.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa[5]Reference 5Light-PEFT Contributorshttps://github.com/gccnlp/Light-PEFT/graphs/contributors}

License wording needs the same care: the repository root has no repository-wide license file. The bundled peft/ subtree contains an Apache-2.0 LICENSE, and its setup metadata identifies that package as a PEFT fork. It's correct to call that subtree Apache-2.0 licensed. It's not correct to call the full Light-PEFT repository Apache licensed.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa[6]Reference 6Apache License in Light-PEFT peft Subdirectoryhttps://github.com/gccnlp/Light-PEFT/blob/7b10c22026fe6fa283d127353a50d6ed7176a8aa/peft/LICENSE}

This distinction affects reuse. A research reader can inspect the code, paper, experiment settings, and history. A production team still needs a license review for every file, model checkpoint terms, dependency terms, and a maintenance plan. There's no service-level guarantee, release cadence, or current model catalog encoded by this checkout. The right handoff is a source map and an experiment receipt, not a promise that the repository is a supported runtime.

Repository maturity is equally narrow. There's no GitHub Actions workflow and no focused test suite for the Light-PEFT pruning path. The bundled peft/tests files exercise inherited configuration and adapter behavior, not the two-phase mask-to-surgery contract. Static Python compilation succeeds at the pinned commit, but that only proves files parse. It doesn't validate mask learning, tensor slicing, checkpoint restoration, or reported performance.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa} That gap is useful context for the rest of the chapter: every claim that matters needs a file, shape, or measurement you can inspect.

Repository map: follow the handoff

The checkout is direct. It doesn't hide the two-phase boundary behind a package command. Read it by asking one question: what does estimate.sh produce that train.sh must consume?

Read the table from left to right: command, recorder, surgery helper, then consumer. This package is a snapshot, not a general adapter framework. Its custom model map registers RoBERTa sequence-classification and multiple-choice paths.

The paper evaluates LoRA and Adapter on RoBERTa-Large, OPT-1.3B, and OPT-6.7B, but the repository's runnable NLU path is much narrower than that sentence might suggest.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Dependency age is part of the experiment

The release pins torch==2.0.1, transformers==4.31.0, datasets==2.13.0, numpy==1.25.1, scikit_learn==1.3.0, fairscale==0.4.13, and fsspec==2023.6.0. These versions explain how the original code was tested. They don't promise compatibility with current Transformers model classes, CUDA drivers, or a newer PEFT API.

An environment receipt should carry the pinned commit, Python and CUDA versions, GPU model and memory, and an exact pip freeze or lock export. Add the dataset revision and preprocessing settings, random seed, command line for both phases, and hashes for saved mask and adapter files. Those fields let a later reader tell whether a changed result came from code, data, hardware, or a different mask.

Without those fields, a speed or accuracy number is hard to reproduce. A green install log doesn't prove that the second command loaded the intended reduced shape.

Lifecycle: estimate, prune, continue

Light-PEFT separates estimation from physical pruning. During the first phase, masks and importance scores are trainable or recorded alongside PEFT updates. After a short prefix of training, the code selects survivors, slices tensors, writes records, and starts another trainer with those smaller tensors. The important handoff is this: a soft score becomes a shape-changing decision only after the saved records are checked.

Let t be the full training step count and t′ the estimation prefix. For GLUE the paper keeps estimation around 5% of total steps; SuperGLUE stays within 10%; QA uses 10%. The appendix reports that extending estimation beyond about 6.8% didn't improve BoolQ.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

The two shell scripts make that boundary visible. Read the diagram as a file handoff: estimate.sh learns evidence, peft_dir carries it, train.sh performs tensor surgery, and the new trainer receives a smaller model.

Those commands encode four handoffs:

1. Estimate. Build the baseline model and PEFT modules, initialize attention and FFN scalar masks to one, and train masks and PEFT parameters together for the estimation prefix.
2. Record. Evaluation and train-end callbacks save mask histories, PEFT module output ratios, and rank importance. Those records are the evidence that the next phase can inspect.
3. Slice and restore. The second process reads peft_dir, selects survivors, slices backbone and matching adapter feature dimensions, wraps the reduced base, restores filtered adapter weights, then slices adapter ranks. The estimate process owns the saved evidence; surgery consumes it.
4. Continue and measure. Start a new trainer with draw_and_training=True, then compare the complete wall-clock run, including estimation and continuation.

The mask is a score during phase one. A pruned dimension is a missing row or column during phase two. Before you run the second command, predict the failure if those rows and columns disagree: the adapter state can't fit the new base. For that reason, train.sh must use a compatible checkpoint and adapter state.

Masked Early Pruning of the foundation model

The first method narrows the cost question to the frozen Transformer: which coordinates contribute enough to justify running them? It doesn't guess individual weight values. It learns two trainable mask families, one scalar per attention head and one scalar per FFN intermediate dimension.

Attention head masks

Start with one attention layer. During estimation, every head still exists, but a scalar can turn its contribution up or down. If that scalar ends near zero, predict what happens next: the output is gated first; the head disappears only when the pruning helper slices its feature range.

Let layer i have N_H heads. A head output is multiplied by scalar m_A^(i). For one head:

$$h^{(i)}_j = m_{A,j}^{(i)} \cdot \operatorname{head}^{(i)}_j(X).$$

When m_A is near zero, the head contributes little to the task loss under this early trajectory. The mask starts at one, so initial behavior matches the unmasked model. Gradient descent can move individual values, while an L1 term makes small values attractive. That is task-local evidence, not a lifetime verdict about the head.

The repository implements this in models/hf_roberta/modeling_roberta.py. When config.mask_training is true, each attention block creates early_attn_mask shaped like (1, heads, 1, 1) and multiplies attention probabilities by it. At pruning time prune_heads builds an index of retained head coordinates and slices query, key, value, and output linear layers.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

That slice is structured. It removes a whole head's feature range, not random individual weights. Tensor cores and framework kernels can see a smaller dense matrix. The score creates a candidate; the slice creates the possible steady-state saving.

FFN dimension masks

The FFN has a different shape contract. If one intermediate coordinate leaves, both projections around that coordinate must agree. Predict the bug if fc1 loses a column but fc2 still expects the old width: the next matrix multiplication won't line up.

An FFN block has an intermediate activation:

$$z = \operatorname{Act}(X(W_{fc1}+\Delta W_{fc1})),$$

followed by a mask and the down projection:

$$\operatorname{FFN}(X) = (z \odot m_F)(W_{fc2}+\Delta W_{fc2}).$$

Here m_F has one scalar per intermediate coordinate. In the source, early_ffn_mask is shaped like (1, intermediate_size). prune_inter_neurons slices intermediate.dense output rows and output.dense input columns using the surviving indices.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Removing an FFN coordinate therefore requires both sides of the bottleneck to agree. Slicing only fc1 leaves fc2 expecting the old width. Slicing only fc2 leaves an activation with a mismatched width. The helper changes both. Keep that paired-index rule in mind when you inspect a saved mask.

L1 penalty is a pressure, not a final mask

The mask values still need a selection rule. A small value is a hint, not yet a deletion, so the mask trainer adds pressure toward sparse values:

$$\mathcal{L}_{mask} = \mathcal{L} + \lambda_A\lVert m_A\rVert_1 + \lambda_F\lVert m_F\rVert_1.$$

trainer_base.py loops through attention and layer modules, sums the absolute values, and multiplies by l1_loss_coef. The shell example sets l1_loss_coef=1e-4. The paper's appendix reports λ_A = λ_F = 1 × 10^-4 for its main experiments.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

L1 doesn't decide an exact number of heads or neurons by itself. Ask what would happen if every mask stayed above zero: a later ratio and quantile would still choose survivors. L1 makes the score distribution easier to rank, then the requested ratio sets a capacity budget.

Layer-wise heads versus global FFN dimensions

Before choosing a threshold, decide who is allowed to compete. For attention heads, the helper defaults to layer-wise pruning. It computes a quantile separately within each layer, then prunes values at or below that layer's threshold. Every layer therefore loses roughly the requested fraction, subject to ties.

For FFN dimensions, train.sh passes --ffn_pruning_method "global". The helper accepts pruning_method="global", sets quantile_axis=None, and computes one threshold across all layers and dimensions. A layer with uniformly useful coordinates can keep more dimensions while another layer loses more. Global ranking spends the budget where scores are lowest; layer-wise ranking protects each layer's quota.

These are the repository's budget choices, not proof that head scores are intrinsically incomparable or that FFN scores are automatically comparable across layers. Check scale and survivor counts before trusting a global cut.

Why does a quantile tie matter?

Answer

The source uses a strict > comparison after computing a quantile. Equal scores at the threshold can produce fewer survivors than a simple percentage calculation. Record the selected indices rather than only the requested ratio.

Random pruning is a control, not evidence

get_early_model uses the saved mask arrays by default. With random_pruning=True, it creates seeded random scores and uses them in place of recorded values. Treat that branch as a control: it asks whether a claimed gain depends on task evidence rather than on any structured shape reduction.

Random pruning still changes tensor shapes. It isn't a no-op baseline. Compare it with the same seed policy, ratio, hardware, and total time before attributing performance to mask learning.

Multi-Granularity Early Pruning of PEFT

Once the backbone has candidate survivors, turn the same question toward the adapter. The second method sees adapter capacity at two scales. At module scale, should an entire LoRA or Adapter insertion remain at this weight or sub-layer? At rank scale, if it remains, how many of its low-rank coordinates are useful? A large module can barely change the task trajectory, while a small surviving module can still spend rank and compute.

Module removal can cut many small operations. Rank removal can retain a module while reducing its internal width. The paper reports both because increasing the number of structured modules can affect forward and backward time more than increasing the rank of one module under a fixed parameter budget.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Module importance from output change

Make the module decision from what it changes, not from its parameter count. If a module's output barely moves relative to the path it would have taken without the adapter, predict that its score will be close to zero.

For LoRA, the repository and paper compare the adapter output to the frozen weight output:

$$I_M = \frac{\lVert XW_{down}W_{up}\rVert_2}{\lVert XW\rVert_2}.$$

For an Adapter inserted after a sub-layer with output h:

$$I_M = \frac{\lVert f(hW_{down})W_{up}\rVert_2}{\lVert h\rVert_2}.$$

An output ratio near zero says the module changed this task's activations little during the observed prefix. The code collects pos_importance_scores, averages the processed history, and writes peft_module_records.pkl.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa} The word observed matters: a quiet prefix can still hide a later task slice.

The displayed ratio suppresses implementation details: the fork's numerator includes LoRA scaling and dropout, its denominator includes the base linear bias, and MaskTrainer averages over non-padding tokens. There is no denominator epsilon in the forward ratio. The callback later replaces non-finite observations with zero, which can conceal instability and depress a module score. Log invalid counts before that sanitization; a cleaned score is not evidence of numerical health.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

The estimate places modules on all targeted weights so it can compare positions. This can cost more during the short prefix than a usual LoRA run. The paper's argument is that a small search cost can pay for a longer reduced run.

The helper prune_peft_modules takes module scores, computes a global quantile, and returns adapter parameter keys for surviving modules. get_early_peft_model filters the adapter state dict to those keys, sets reserved_modules, and enables prune_module so future wrapping skips removed positions.

Rank importance from first-order Taylor scores

Module pruning asks whether an insertion earns a place. Rank pruning asks which paired low-rank directions earn width. If setting one rank coordinate to zero would change the loss a lot, its first-order score should be large.

For parameter W_{i,j} in either down or up projection, the paper writes the first-order term as:

$$I_{W_{i,j}} = \frac{\partial \mathcal{L}}{\partial W_{i,j}}W_{i,j}.$$

The rank score sums terms for one column of W_down and the matching row of W_up. This treats a rank coordinate as a unit. Removing only one side would break the low-rank product, so both matrices are sliced. The paper's displayed equation doesn't put absolute-value bars around this product.

The repository makes a more specific implementation choice. In MaskTrainer.training_step, each PEFT module computes weight * grad for its down and up matrices, averages along the complementary axis, takes absolute values of those means, stacks the two vectors, and sums them into importance_scores. The callback averages the later half of the history, replaces NaN and infinity with zero, multiplies by the current mask when present, and writes peft_mask_records.pkl.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

The source's prune_peft_rank then computes a quantile across rank scores. For LoRA it slices lora_A rows and lora_B columns. If every rank is selected for removal, the module gets rank zero and its adapter is disabled. The same pattern exists for the bottleneck Adapter fork.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Layer-wise versus global adapter ranking

The helper accepts peft_pruning_method. A layer-wise choice computes a threshold inside each module. A global choice compares rank coordinates across all matching adapter modules. arguments.py defaults both ffn_pruning_method and peft_pruning_method to layerwise. The example train.sh overrides both to global. Predict the consequence before running: global rank pruning can spend most capacity in a few modules; layer-wise pruning computes a separate threshold per module. Record those flags in the receipt; copying only the ratios isn't enough.

Global rank pruning can concentrate capacity in a few modules. Layer-wise pruning sets separate thresholds but doesn't guarantee a nonzero rank: tied scores can remove an entire module even at a modest ratio.

Tiny numerical walk-through

Now make the cuts by hand. Use four attention heads in one layer, with the same numbers as the score map above. Before reading the result, predict which head falls below a layer-wise 25% cut:

Request an attention pruning ratio of 0.25. With NumPy's default linear interpolation, the layer-wise 25th percentile is 0.425 for this tiny list. The source keeps values strictly above that threshold, so head 2 is removed and heads 1, 3, and 4 survive. Query, key, value, and output slices now use three head ranges instead of four. The useful distinction is visible: the threshold chooses head 2; the four synchronized slices make the choice physical.

Next, change the competition. Use five FFN coordinates across two layers and predict which two a global 40% cut removes:

For a global 40% pruning ratio, sort values: 0.07, 0.15, 0.31, 0.42, 0.63. Linear interpolation gives 0.15 + 0.6 × (0.31 − 0.15) = 0.246, removing two low coordinates and leaving 0.31, 0.42, 0.63. The five-coordinate pool is a small arithmetic example, not the rectangular unpruned RoBERTa mask layout. A global cut can remove several coordinates from one layer if that is where the low scores are.

For the adapter, let three LoRA insertions produce output ratios 0.04, 0.31, and 0.11. Predict the module cut first. A module ratio of 0.33 removes the lowest third, likely the 0.04 module. For the surviving module, suppose rank scores are [0.02, 0.17, 0.03, 0.22] and rank pruning ratio is 0.50. Keep the two largest coordinates, ranks 2 and 4 in this local example. The adapter's down projection keeps the corresponding rows and its up projection keeps matching columns.

The arithmetic separates three decisions:

• a score says how much a coordinate changed the observed task trajectory;
• a quantile turns scores into a budget;
• a physical slice turns the budget into dense smaller tensors.

None of those steps says that a coordinate is universally unimportant. A new task, sequence distribution, adapter target set, or reward can rank it differently.

Code path from shell to saved artifacts

Now follow the handoff in the files. Before running anything, predict what the first command owns: it should create evidence, not a reduced model. The default scripts/estimate.sh pins one GPU (CUDA_VISIBLE_DEVICES=2 in the example), GLUE SST-2, RoBERTa-Large, LoRA rank 8, sequence length 128, batch size 32, five epochs, and max_steps=800. It passes --early_search True --use_mask_trainer True --use_peft True and writes under:

That path depends on DATASET_NAME, model_name_or_path, adapter name, and peft_imp. The source's p2-first_order string controls how PEFT importance is recorded. It isn't a generic method name recognized by current PEFT releases.

What estimate.py records

At on_train_begin, SuperGLUECallback creates one list per encoder layer for attention and FFN masks. It also writes init_peft_mask_records.pkl with initial PEFT mask tensors. The callback is establishing the history that later pruning will treat as evidence.

At each evaluation, the callback records current mask values. At train end it records one final value, stacks each layer's history, and saves:

It also processes each PEFT module's score history. The code averages the latter half of the history, sanitizes NaN and infinity, and writes:

These files are small relative to the model, but they are part of the run's provenance. Hash them and keep the seed with the command. If those files are missing or point to another run, the second phase has no trustworthy handoff.

What train.py reads

The second command consumes that handoff. The default scripts/train.sh points peft_dir at the estimate output and sets:

Read those knobs in two groups. The ratios request how much to remove; the method flags decide which scores compete. This command asks for a 75% module cut and a 50% rank cut among surviving modules, then lets global FFN and rank thresholds spend those cuts across layers. Keep the ratios but switch to layer-wise ranking and you've run a different experiment.

It passes --draw_and_training True --use_peft True. Pause before copying those flags: the ratios are script defaults, not a 1:1 copy of Table 1. The paper's 72% retained LoRA row prunes 5/16 of the heads (0.3125) and 1/3 of the FFN dimensions. If you want that row, change the flags; don't assume train.sh already encodes it.

get_trainer.py calls get_early_model, which loads the two .npy arrays, chooses the final recorded step, and prunes RoBERTa heads and FFN dimensions. get_early_peft_model then loads adapter state, selects modules, slices adapter feature dimensions to match the reduced base, wraps that base, restores filtered weights, and finally prunes ranks. Rank and module pickle reads are conditional on nonzero pruning ratios.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Partial rank pruning changes actual A/B tensor shapes without updating the module's nonzero r or recomputing scaling. The original scale therefore survives the slice. Rebuilding a modern adapter with alpha/new_rank would change its output. Export and reload need explicit per-module shapes and scale, not just the original configuration. Also treat .pkl files and the old torch.load checkpoint path as trusted-code inputs: don't load artifacts from an untrusted source.

The second command isn't a resume in the optimizer sense. It constructs a new model and trainer after tensor surgery. set_mask(init_mask=False) resets surviving backbone masks to one and disables their training/forward gates. The saved scalar magnitudes select coordinates; they aren't preserved as learned gates in continuation. If you need an uninterrupted optimizer trajectory, this code isn't that path.

Failure modes at the estimate-to-train handoff

If estimation succeeds but continuation fails, classify the boundary before changing a ratio. A missing record points backward to the producer; a shape error points forward to the surgery and state pairing. The table gives each symptom a first check:

For a concrete diagnosis, suppose all four record files exist but train.py raises an adapter-key mismatch. Don't lower the pruning ratio first. Compare the base revision, target module list, and surviving shapes with the estimate receipt; the producer and consumer disagree about identity or layout. A linear shape error points to the same handoff after surgery, while a missing .npy points back to the estimate output path.

Keep command output in a receipt. The printed "pruning complete" line proves only that control flow reached the helper. It doesn't prove shape correctness, throughput gain, or task quality. Those require the next gates: inspect shapes, run a batch, then measure the complete path.

A copy-runnable mask simulator

The following standard-library example lets you test the score-to-index handoff without a GPU. Before running it, predict three things: the head threshold should keep indices [0, 2, 3], the global FFN cut should keep [1, 2, 4], and every retained rank should appear in both adapter projections. The example mirrors score selection, strict > thresholding, and rank pairing without importing Torch or NumPy. It doesn't emulate Transformer kernels. Its value is the invariant: every retained rank is present in both adapter projections, and every selected backbone coordinate is explicit.

The output is a logic check, not a performance result. This simulator's quantile is a transparent teaching implementation. The repository uses NumPy's quantile behavior, so a production reproduction should compare exact selected indices rather than copying this helper into a trainer.

Try changing the rank ratio to 0.75. The retained pair list shrinks, but every pair still has matching down and up coordinates. Try changing one backbone score to a tie at the threshold. The strict comparison then makes the survivor count depend on the quantile value.

Check the product, not just the index labels

The next standalone CPU example uses PyTorch's stored orientation without importing PyTorch. It tests every subset of a rank-four adapter. A physically sliced product must equal a full-width product with the removed rank activations zeroed, at the same scale. This is an algebra check, not evidence that pruning preserves the original model's output or task accuracy.

Neither example runs a Transformer, learns masks, checks an exported checkpoint, or reproduces GPU timings. The upstream helper also needs explicit guards for empty survivor sets and invalid histories. In particular, its random rank branch constructs an array and subsequently treats it like a name-keyed dictionary; the module helper accepts random_shuffle but doesn't use it. Don't interpret those flags as a validated end-to-end random control.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Paper evidence: read numbers with their boundary

Now separate the questions a benchmark can answer. The paper validates Light-PEFT on GLUE, SuperGLUE, and question-answering tasks. Its foundation models are RoBERTa-Large, OPT-1.3B, and OPT-6.7B. Its PEFT structures are LoRA and Adapter. Those are the tested combinations, not a universal compatibility matrix.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Table 1 trade-off

Table 1 compares LoRA on RoBERTa-Large with Light-PEFT variants. Before reading the numbers, predict what a faster row has to give up or account for: search overhead, retained capacity, or task score. Read each row as one quality, capacity, and timing choice. The values below are paper-reported averages and speed measurements, not claims about current hardware.

The average covers the paper's MNLI, QNLI, QQP, and SST-2 columns, not the full GLUE suite. Training speed includes estimation before pruning, measured on a single NVIDIA TITAN RTX 24GB GPU with batch size 32 and sequence length 128. The caption doesn't establish an end-to-end download-to-export stopwatch. The 72% row prunes 5/16 of the heads and 1/3 of the FFN intermediate dimensions, then also prunes PEFT modules and ranks.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Notice the trade: the 67% retained configuration uses the same rounded 0.3M trainable parameter count as the 72% configuration, but gives up 0.3 GLUE average points for the reported 0.2× additional speedup. The useful comparison is a Pareto choice under a declared task and budget. "More pruning is always better" isn't a conclusion the table can support.

Figure 4 training cost

The mechanism makes a second prediction: if physical widths shrink, model-weight memory and activation memory should both fall, while the size of the fall still depends on the workload.

Figure 4 measures RoBERTa-Large on an NVIDIA RTX 3090 with batch size 32 and sequence length 128. In that experiment, the Light-PEFT setting retains 67% of foundation parameters and 0.3M trainable parameters. The paper reports 32% lower model-weight memory, 40% lower activation memory, and 39% lower peak memory. Its ten-batch forward and backward timing is 2.2× faster than baseline LoRA.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

The peak-memory result follows the mechanism taught earlier. Removing dimensions reduces both weight storage and intermediate activation shapes. The 2.2× number isn't a claim that every batch, sequence length, or GPU sees that ratio. It includes the measured workload and implementation.

Table 5 inference boundary

Hold the foundation at 52% retained and look at the module column. What can change the inference row without changing that backbone? The number of surviving adapter insertions. Section 5.3.2 places the efficiency experiments on an RTX 3090; Table 5 uses OPT-6.7B with batch size 96 and maximum length 100. It reports load memory and inference speed for foundation pruning and optional module pruning.

The paper therefore reports up to 1.6× inference speed and 48% lower load memory in this OPT-6.7B protocol. "48%" compares 6.5G with 12.5G approximately, and the 1.6× row also removes 75% of PEFT modules. Keep those axes together when reproducing the result.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

What the paper leaves untested

The evidence stops at those combinations. It doesn't establish support or benefit for:

• modern frontier MoE post-training;
• multi-task adapter fleets sharing one base;
• online reinforcement-learning distribution shifts;
• QLoRA plus Light-PEFT combinations;
• distributed large-scale jobs with tensor, pipeline, or expert parallelism;
• current CUDA, ROCm, or accelerator kernel stacks;
• continual pruning while adapters are loaded per request.

A capability claim beyond that scope needs a model-specific implementation, loader test, benchmark protocol, and license check.

Permanent base masks change serving semantics

Training has handed us a shape-specific artifact. Imagine a compiled plan for 16 heads receiving a 12-head pruned base. Can the server reuse its cached tensors? No. A shared-base adapter service relies on compatible base dimensions and coordinate identities. Ordinary multi-adapter serving makes the same bet: request A selects adapter A, request B selects adapter B, and both still fit the same frozen matrix dimensions. A structured Light-PEFT run physically removes heads and FFN coordinates from that base.

That creates a serving choice:

The paper's "plug-and-play" claim refers to switching masks and PEFT modules in its tested setup. It doesn't automatically mean that one production base can serve arbitrary adapters with one static kernel plan.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/} The deployment decision comes after the slice, because the slice determines which artifacts can share a shape.

KV caches and compiled kernels add more state. A cache keyed only by model name and adapter ID can return incompatible tensors if the base shape or pruning seed changes. Include base revision, shape signature, and adapter revision in cache keys. The cache key is part of the pruning contract, not an implementation detail to fill in later.

Reinforcement-learning distribution shifts

Early scores come from a prefix of one data distribution. Ask whether that evidence is still valid after the model changes. In supervised fine-tuning, later batches can already differ from the prefix. In online RL, the policy changes the data it receives, so the shift can be stronger:

1. The initial policy samples trajectories from distribution D_0.
2. Early masks rank coordinates using rewards and gradients from D_0.
3. Pruning changes policy capacity and removes some routes.
4. The new policy samples D_1, which can contain states absent from D_0.
5. A removed head or FFN coordinate may be important in D_1 even if it looked quiet in D_0.

This is a research concern, not a measured failure in the paper's static GLUE or QA protocol. An RL evaluation should log policy version, mask version, trajectory distribution, reward slices, and divergence before and after pruning. A held-out online stream can test whether reward collapse follows the physical cut. The important question is causal: did the reward change because of the physical cut, or because the policy changed the data?

If a system needs reversible exploration, keep the unpruned base or a set of checkpoints. A one-way physical slice is cheap to run but expensive to undo without rebuilding the original model.

Operational fragility and safe experiment design

The code is concise because it assumes a controlled research machine. A production adaptation fails at handoffs, not at the idea's headline: a state dict can have familiar keys and still target the wrong shapes, or a speedup can disappear once estimation is counted. Treat each contract below as a gate.

Shape contract

Before loading adapter state, predict the failure from one mismatched dimension, then verify:

• base model identifier and revision;
• number of layers;
• surviving attention heads per layer;
• hidden size and head dimension;
• surviving FFN width per layer;
• target module names;
• LoRA rank or Adapter bottleneck width;
• bias and dropout configuration.

The source slices tensors in place and restores state dict entries by name. A different Transformers implementation can rename a module or store transposed weights. A name match isn't enough; compare tensor shapes before the load. If the shape signature changes, stop before the adapter can fail deeper in the forward pass.

Numerical contract

Mask penalties and first-order scores depend on gradients. A score change can come from numerical settings rather than task evidence, so record:

• precision and autocast state;
• loss scale if mixed precision is enabled;
• gradient clipping;
• learning rate and mask learning-rate multipliers;
• NaN and infinity counts;
• evaluation interval used to sample mask history.

The repository uses separate attention and FFN mask learning-rate coefficients. MaskTrainer has constructor defaults of 100 and 50, but get_trainer.py passes arguments.py values, and those dataclass defaults are both 1. The example shells don't override them. A port that silently keeps the constructor defaults, or changes the multipliers, changes the search problem.^{[4]Reference 4Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa} Record the values that reach the trainer, not only the values shown in a constructor.

Measurement contract

A speed claim can be true and still answer the wrong question if the stopwatch starts after the expensive search. Time the complete workflow:

The paper's Table 1 explicitly includes estimation before pruning. A benchmark that starts its stopwatch after the masks exist will overstate speedup. A benchmark that includes data download but not model load will answer a different question. State both wall-clock definitions.

Failure isolation

Run checks in dependency order:

1. Environment and baseline. Import the pinned dependency set, load the tokenizer and base model without PEFT, then run one baseline forward and backward batch.
2. Estimation. Run one estimation batch, inspect mask ranges, and confirm all four record files exist.
3. Surgery. Rebuild one pruned model, print its shapes, and restore one filtered adapter state dict.
4. Smoke test. Run one post-pruning batch. Only then start the full continuation.

This order separates environment errors from score errors and shape errors. It also avoids burning a long GPU job before discovering that the saved peft_dir points to another dataset.

Scope comparison with nearby methods

At this point, compare mechanisms rather than labels. Light-PEFT combines pruning with PEFT; it isn't a replacement name for every efficient fine-tuning approach.

Light-PEFT's efficiency comes from paying for a change in the model's dense shapes. That makes its steady-state run different from a method that only gates execution or stores lower-precision weights. Because the method rewrites dense dimensions, its runtime must accept the resulting shape.

Questions to ask before adapting the code

When moving beyond RoBERTa NLU, use these as go-or-stop questions in order. A “yes” at one gate doesn't imply a “yes” at the next.

Does the model expose the same structures?

The code knows RoBERTa attention and layer classes. A decoder-only model may store fused QKV, grouped-query heads, rotary embeddings, parallel residual paths, or gated FFN matrices. A head slice must respect those layouts. An MoE layer adds expert routing and capacity rules that this code doesn't model.

Can a global score compare coordinates?

Global FFN and rank quantiles assume scores share a meaningful scale. Layer normalization, module placement, and gradient scale can make one layer's numbers incomparable. Normalize or keep thresholds local only after checking the score distribution.

Is physical pruning legal for the deployment runtime?

A training fork can slice PyTorch linear layers. A fused inference engine may require a separate kernel for each head count, tensor-parallel shard, or expert layout. Prove that the loader, compiler, quantizer, and serving cache all understand the new shape.

Does the task distribution stay stable?

Static labels and online rewards answer different questions. For multi-task or RL use, hold out shifts and re-evaluate after every pruning boundary. Early importance isn't a lifetime contract.

Can the experiment be undone?

Keep original base weights, original adapter state, selected indices, and a manifest. A mask vector alone can't reconstruct removed matrix rows unless the original checkpoint remains available.

What a pruning review must defend

A credible review should connect the repository's choices to an artifact that another engineer can inspect:

• Separate trainable-state savings from frozen projection, activation, and attention costs; count estimation before claiming a training speedup.
• Trace the exact commit from score history through selected coordinate indices, adapter feature slicing, restoration, and rank slicing. Explain the stored A/B orientation and preserved scale.
• Report per-layer head and FFN survivors, per-module rank shapes, non-finite score counts, and a post-pruning batch result. A ratio or a successful load alone isn't enough.
• Keep paper-reported timings separate from local measurements, and distinguish the four-task GLUE average from a full-suite score.
• Name the deployment's base revision, selected coordinates, compatible adapters, rollback artifact, and unresolved license or runtime constraints.

Follow-up questions

Two pruned adapters have identical tensor shapes but kept different original head indices. Can they share a pruned base?

Answer

Not on shape equality alone. Each adapter's coordinates refer to specific original base features. Check the selected indices and base revision as well as dimensions; otherwise a valid matrix multiplication can compute the wrong function.

A port reloads rank-four weights as rank two and recomputes alpha/r. The tensors load cleanly. What should you test next?

Answer

Compare outputs at the pruning boundary with the original scale. The pinned implementation preserves its scale during partial rank slicing; recomputing alpha/r can rescale the update even when every shape matches. Then test task quality and export/reload equivalence separately.

Mastery quiz

Course complete: Return to the curriculum roadmap. You've now read seventeen project repositories, from vLLM's paged serving through Tinker's hosted LoRA loop and this Light-PEFT research checkout. Keep them as a continuing source-reading loop: maintain source maps, test important claims against pinned code and primary sources, and use measured receipts to separate promising research from proven production behavior. For a career next step, turn those receipts into portfolio evidence with AI Engineer Portfolio Projects That Get Interviews, using a run manifest as the proof-of-skill checklist.