Deep Dive - Light-PEFT

Parameter-efficient fine-tuning (PEFT) freezes most backbone weights and learns a small update. That saves optimizer state and gradient storage, but it doesn't make the frozen network disappear from every training step. A frozen layer still computes activations, and those activations still carry information needed by backward.

Light-PEFT asks a sharper question: after a task has shown which heads, feed-forward dimensions, adapter modules, and adapter ranks it uses, why keep paying for the rest? The paper's answer is a short task-specific estimation phase followed by physical structured pruning and a longer fine-tuning phase.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

This chapter reads the pinned research release rather than treating Light-PEFT as a maintained product. It connects the equations to NLU/estimate.py, NLU/train.py, and the small PEFT fork, then separates paper evidence from what a modern model-serving or reinforcement-learning system would still need to prove.

Start with cost that PEFT leaves behind

Suppose a Transformer layer receives hidden states X. With ordinary LoRA, a frozen weight W receives a low-rank update:

$$W' = W + sAB,$$

where A maps from the hidden dimension to rank r, B maps back, and s is a scale. Only A and B need optimizer updates. The layer still evaluates XW' for every token.

The phrase frozen describes parameter updates, not execution. A training step still has this shape:

1. Read frozen weights and trainable adapter weights.
2. Run forward through every surviving layer.
3. Keep activations or checkpoint enough information to reproduce them.
4. Backpropagate the loss through the whole computation graph.
5. Store gradients only for trainable parameters, then update the adapter.

PEFT removes a large optimizer and gradient burden. It doesn't automatically remove backbone FLOPs, backbone weight reads, or activation memory. The paper measures this directly: as OPT grows, forward time and peak memory rise even though LoRA avoids most backbone parameter gradients.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

A tiny cost ledger

Use one layer with hidden size d = 8, sequence length S = 4, and a feed-forward width of 32. A rough dense projection count for one token is proportional to:

The adapter may add only a small number of trainable parameters. The frozen 768 projection pairs per token still run. For S = 4, that is 3,072 rough projection pairs before counting softmax, normalization, activation, and memory movement.

If a task never uses one attention head, or if many FFN coordinates carry little task signal, PEFT alone has no reason to skip them. A task-specific structured mask can make those coordinates absent from the second run.

Why quantization answers a different question

Quantization stores weights in fewer bits. It can reduce load and optimizer memory, but training kernels may dequantize values for computation. Quantization changes bytes per value; structured pruning changes how many rows, columns, heads, or dimensions exist. Light-PEFT studies the second lever. Its method can be combined with quantization in principle, but the paper's experiments don't establish a QLoRA combination.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

A research release, not a platform

The source at commit 7b10c22026fe6fa283d127353a50d6ed7176a8aa is a small academic code release for the Findings of ACL 2024 paper. The paper authors are Naibin Gu, Peng Fu, Xiyu Liu, Bowen Shen, Zheng Lin, and Weiping Wang. Their listed affiliations are the Institute of Information Engineering, Chinese Academy of Sciences, and the School of Cyber Security, University of Chinese Academy of Sciences.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/} The paper and repository don't name a company behind the project; primary ownership evidence points to those academic institutions and the gccnlp repository account.

Paper authors and code contributors are related evidence, not interchangeable labels. The pinned Git history has six commits, all with commit identity gccnlp and email domain iie.ac.cn; it has no tags or GitHub releases. The GitHub contributors page is a separate activity view and shouldn't be used to infer that every paper author committed every file.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa[3]Reference 3Light-PEFT Contributorshttps://github.com/gccnlp/Light-PEFT/graphs/contributors}

License wording needs the same care. The repository root has no repository-wide license file. The bundled peft/ subtree contains an Apache-2.0 LICENSE, and its setup metadata identifies that package as a PEFT fork. It's correct to call that subtree Apache-2.0 licensed. It's not correct to call the full Light-PEFT repository Apache licensed.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa[4]Reference 4Apache License in Light-PEFT peft Subdirectoryhttps://github.com/gccnlp/Light-PEFT/blob/7b10c22026fe6fa283d127353a50d6ed7176a8aa/peft/LICENSE}

This distinction affects reuse. A research reader can inspect the code, paper, experiment settings, and history. A production team still needs a license review for every file, model checkpoint terms, dependency terms, and a maintenance plan. There is no service-level guarantee, release cadence, or current model catalog encoded by this checkout.

Repository maturity is equally narrow. There is no GitHub Actions workflow and no focused test suite for the Light-PEFT pruning path. The bundled peft/tests files exercise inherited configuration and adapter behavior, not the two-phase mask-to-surgery contract. Static Python compilation succeeds at the pinned commit, but that only proves files parse. It doesn't validate mask learning, tensor slicing, checkpoint restoration, or reported performance.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Repository map: follow the handoff

The checkout is intentionally direct. It doesn't hide the two-phase boundary behind a package command.

The package is a snapshot, not a general adapter framework. The custom model map currently registers RoBERTa sequence-classification and multiple-choice paths. The paper evaluates LoRA and Adapter on RoBERTa-Large, OPT-1.3B, and OPT-6.7B, but the repository's runnable NLU path is much narrower than that sentence might suggest.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Dependency age is part of the experiment

The release pins torch==2.0.1, transformers==4.31.0, datasets==2.13.0, numpy==1.25.1, scikit_learn==1.3.0, fairscale==0.4.13, and fsspec==2023.6.0. These versions explain how the original code was tested. They don't promise compatibility with current Transformers model classes, CUDA drivers, or a newer PEFT API.

An environment receipt should include:

• the pinned commit;
• Python and CUDA versions;
• GPU model and memory;
• exact pip freeze or lock export;
• dataset revision and preprocessing settings;
• random seed;
• command line for both phases;
• saved mask and adapter file hashes.

Without those fields, a speed or accuracy number is hard to reproduce. A green install log doesn't prove that the second command loaded the intended reduced shape.

Lifecycle: estimate, prune, continue

Light-PEFT separates estimation from physical pruning. During the first phase, masks and importance scores are trainable or recorded alongside PEFT updates. After a short prefix of training, the code selects survivors, slices tensors, writes records, and starts another trainer with those smaller tensors.

Let t be the full training step count and t′ the estimation prefix. The paper uses early prefixes around 5% to 10% for its NLU studies and reports that extending estimation beyond about 6.8% didn't improve BoolQ in its appendix experiment.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

The lifecycle is:

1. Build the baseline model and PEFT modules.
2. Set attention and FFN scalar masks to one.
3. Train masks and PEFT parameters together for the estimation prefix.
4. Record mask histories at evaluation and train-end callbacks.
5. Record PEFT module output ratios and rank importance.
6. Select survivors with layer-wise or global quantiles.
7. Slice backbone linear layers and adapter matrices.
8. Save arrays, pickles, model config, and adapter state under peft_dir.
9. Start continued training with draw_and_training=True.
10. Include estimation and continuation time in the comparison.

The mask is a score during phase one. A pruned dimension is a missing row or column during phase two. That change is why the second command must use a compatible checkpoint and adapter state.

Masked Early Pruning of the foundation model

The first method targets the frozen Transformer. It has two trainable mask families: one scalar per attention head and one scalar per FFN intermediate dimension.

Attention head masks

Let layer i have N_H heads. A head output is multiplied by scalar m_A^(i). For one head:

$$h^{(i)}_j = m_{A,j}^{(i)} \cdot \operatorname{head}^{(i)}_j(X).$$

When m_A is near zero, the head contributes little to the task loss under this early trajectory. The mask starts at one, so initial behavior matches the unmasked model. Gradient descent can move individual values, while an L1 term makes small values attractive.

The repository implements this in models/hf_roberta/modeling_roberta.py. When config.mask_training is true, each attention block creates early_attn_mask shaped like (1, heads, 1, 1) and multiplies attention probabilities by it. At pruning time prune_heads builds an index of retained head coordinates and slices query, key, value, and output linear layers.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

That slice is structured. It removes a whole head's feature range, not random individual weights. Tensor cores and framework kernels can see a smaller dense matrix.

FFN dimension masks

An FFN block has an intermediate activation:

$$z = \operatorname{Act}(X(W_{fc1}+\Delta W_{fc1})),$$

followed by a mask and the down projection:

$$\operatorname{FFN}(X) = (z \odot m_F)(W_{fc2}+\Delta W_{fc2}).$$

Here m_F has one scalar per intermediate coordinate. In the source, early_ffn_mask is shaped like (1, intermediate_size). prune_inter_neurons slices intermediate.dense output rows and output.dense input columns using the surviving indices.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Removing an FFN coordinate requires both sides of the bottleneck to agree. Slicing only fc1 leaves fc2 expecting the old width. Slicing only fc2 leaves an activation with a mismatched width. The helper changes both.

L1 penalty is a pressure, not a final mask

The mask trainer adds:

$$\mathcal{L}_{mask} = \mathcal{L} + \lambda_A\lVert m_A\rVert_1 + \lambda_F\lVert m_F\rVert_1.$$

trainer_base.py loops through attention and layer modules, sums the absolute values, and multiplies by l1_loss_coef. The shell example sets l1_loss_coef=1e-4. The paper's appendix reports λ_A = λ_F = 1 × 10^-4 for its main experiments.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

L1 doesn't decide an exact number of heads or neurons by itself. The pruning ratio and quantile decide that later. L1 makes the score distribution easier to rank, then the requested ratio sets a capacity budget.

Layer-wise heads versus global FFN dimensions

For attention heads, the helper defaults to layer-wise pruning. It computes a quantile separately within each layer, then prunes values at or below that layer's threshold. Every layer therefore loses roughly the requested fraction, subject to ties.

For FFN dimensions, train.sh passes --ffn_pruning_method "global". The helper accepts pruning_method="global", sets quantile_axis=None, and computes one threshold across all layers and dimensions. A layer with uniformly useful coordinates can keep more dimensions while another layer loses more.

This difference is intentional. Heads are repeated semantic routes within each layer. FFN dimensions are a large shared pool where global ranking can spend the budget unevenly.

Random pruning is a control, not evidence

get_early_model uses the saved mask arrays by default. With random_pruning=True, it creates seeded random scores and uses them in place of recorded values. That branch can test whether a claimed gain depends on task evidence rather than any structured shape reduction.

Random pruning still changes tensor shapes. It isn't a no-op baseline. Compare it with the same seed policy, ratio, hardware, and total time before attributing performance to mask learning.

Multi-Granularity Early Pruning of PEFT

The second method sees adapter capacity at two scales:

1. Module scale: should this entire LoRA or Adapter insertion remain at this weight or sub-layer?
2. Rank scale: if it remains, how many of its low-rank coordinates are useful?

Module removal can cut many small operations. Rank removal can retain a module while reducing its internal width. The paper reports both because increasing the number of structured modules can affect forward and backward time more than increasing the rank of one module under a fixed parameter budget.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Module importance from output change

For LoRA, the repository and paper compare the adapter output to the frozen weight output:

$$I_M = \frac{\lVert XW_{down}W_{up}\rVert_2}{\lVert XW\rVert_2}.$$

For an Adapter inserted after a sub-layer with output h:

$$I_M = \frac{\lVert f(hW_{down})W_{up}\rVert_2}{\lVert h\rVert_2}.$$

An output ratio near zero says the module changed this task's activations little during the observed prefix. The code collects pos_importance_scores, averages the processed history, and writes peft_module_records.pkl.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

The LoRA estimate temporarily places modules on all targeted weights so it can compare positions. This can cost more during the short prefix than a usual LoRA run. The paper's argument is that a small search cost can pay for a longer reduced run.

The helper prune_peft_modules takes module scores, computes a global quantile, and returns adapter parameter keys for surviving modules. get_early_peft_model filters the adapter state dict to those keys, sets reserved_modules, and enables prune_module so future wrapping skips removed positions.

Rank importance from first-order Taylor scores

For parameter W_{i,j} in either down or up projection, the paper writes the first-order term as:

$$I_{W_{i,j}} = \frac{\partial \mathcal{L}}{\partial W_{i,j}}W_{i,j}.$$

The rank score sums terms for one column of W_down and the matching row of W_up. This treats a rank coordinate as a unit. Removing only one side would break the low-rank product, so both matrices are sliced. The paper's displayed equation doesn't put absolute-value bars around this product.

The repository makes a more specific implementation choice. In MaskTrainer.training_step, each PEFT module computes weight * grad for its down and up matrices, averages along the complementary axis, takes absolute values of those means, stacks the two vectors, and sums them into importance_scores. The callback averages the later half of the history, replaces NaN and infinity with zero, multiplies by the current mask when present, and writes peft_mask_records.pkl.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

The source's prune_peft_rank then computes a quantile across rank scores. For LoRA it slices lora_A rows and lora_B columns. If every rank is selected for removal, the module gets rank zero and its adapter is disabled. The same pattern exists for the bottleneck Adapter fork.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Layer-wise versus global adapter ranking

The helper accepts peft_pruning_method. A layer-wise choice computes a threshold inside each module. A global choice compares rank coordinates across all matching adapter modules. The shell defaults to global for FFN dimensions but leaves PEFT method explicit, so an experiment receipt should record both choices.

Global rank pruning can concentrate capacity in a few modules. That may be useful for one task, but it makes a shared serving policy less predictable. Layer-wise rank pruning keeps every module alive at some width but can leave low-value modules with tiny ranks.

Tiny numerical walk-through

Use four attention heads in one layer. After the estimation prefix, suppose the mask values are:

Request an attention pruning ratio of 0.25. With NumPy's default linear interpolation, the layer-wise 25th percentile is 0.425 for this tiny list. The source keeps values strictly above that threshold, so head 2 is removed and heads 1, 3, and 4 survive. Query, key, value, and output slices now use three head ranges instead of four.

Now use five FFN coordinates across two layers:

For a global 40% pruning ratio, sort values: 0.07, 0.15, 0.31, 0.42, 0.63. A threshold around the second value removes two low coordinates, leaving 0.31, 0.42, 0.63. Actual NumPy quantile interpolation and strict comparison decide ties, so always inspect the saved indices.

For adapter module ratios, let three LoRA insertions produce 0.04, 0.31, and 0.11. A module ratio of 0.33 removes the lowest third, likely the 0.04 module. For the surviving module, suppose rank scores are [0.02, 0.17, 0.03, 0.22] and rank pruning ratio is 0.50. Keep the two largest coordinates, ranks 2 and 4 in this local example. The adapter's down projection keeps the corresponding rows and its up projection keeps matching columns.

The arithmetic separates three decisions:

• a score says how much a coordinate changed the observed task trajectory;
• a quantile turns scores into a budget;
• a physical slice turns the budget into dense smaller tensors.

None of those steps says that a coordinate is universally unimportant. A new task, sequence distribution, adapter target set, or reward can rank it differently.

Code path from shell to saved artifacts

The default scripts/estimate.sh chooses one GPU, GLUE SST-2, RoBERTa-Large, LoRA rank 8, sequence length 128, batch size 32, five epochs, and max_steps=800. It passes --early_search True --use_mask_trainer True --use_peft True and writes under:

The exact path depends on DATASET_NAME, model_name_or_path, adapter name, and peft_imp. The source's p2-first_order string controls how PEFT importance is recorded. It isn't a generic method name recognized by current PEFT releases.

What estimate.py records

At on_train_begin, SuperGLUECallback creates one list per encoder layer for attention and FFN masks. It also writes init_peft_mask_records.pkl with initial PEFT mask tensors.

At each evaluation, the callback records current mask values. At train end it records one final value, stacks each layer's history, and saves:

It also processes each PEFT module's score history. The code averages the latter half of the history, sanitizes NaN and infinity, and writes:

These files are small relative to the model, but they are part of the run's provenance. Hash them and keep the seed with the command.

What train.py reads

The default scripts/train.sh points peft_dir at the estimate output and sets:

It passes --draw_and_training True --use_peft True. get_trainer.py calls get_early_model, which loads the two .npy arrays, chooses the final recorded step, and prunes RoBERTa heads and FFN dimensions. It then calls get_early_peft_model, which loads the adapter state and two pickle files, filters modules, slices rank matrices, wraps the reduced model, and restores the filtered adapter state.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

The second command isn't a resume in the optimizer sense. It constructs a new model path and a new trainer after tensor surgery. If you need an uninterrupted optimizer trajectory, this code isn't that path. It's a short search followed by continued task training on a changed architecture.

Why estimate.sh and train.sh can fail separately

Keep command output in a receipt. The printed "pruning complete" line proves only that control flow reached the helper. It doesn't prove shape correctness, throughput gain, or task quality.

A copy-runnable mask simulator

The following standard-library example mirrors score selection, strict > thresholding, and rank pairing without importing Torch or NumPy. It doesn't emulate Transformer kernels. Its value is the invariant: every retained rank is present in both adapter projections, and every selected backbone coordinate is explicit.

The simulator's quantile is a transparent teaching implementation. The repository uses NumPy's quantile behavior, so a production reproduction should compare exact selected indices rather than copying this helper into a trainer.

Try changing the rank ratio to 0.75. The retained pair list shrinks, but every pair still has matching down and up coordinates. Try changing one backbone score to a tie at the threshold. The strict comparison then makes the survivor count depend on the quantile value.

Paper evidence: read numbers with their boundary

The paper validates Light-PEFT on GLUE, SuperGLUE, and question-answering tasks. Its foundation models are RoBERTa-Large, OPT-1.3B, and OPT-6.7B. Its PEFT structures are LoRA and Adapter. Those are the tested combinations, not a universal compatibility matrix.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Table 1 trade-off

The GLUE table compares LoRA on RoBERTa-Large with Light-PEFT variants. The values below are paper-reported averages and speed measurements, not claims about current hardware.

The table's speed includes estimation before pruning. It was measured on a single NVIDIA TITAN RTX 24GB GPU with batch size 32 and sequence length 128. "1.6×" therefore means total wall-clock behavior under that protocol, not a guaranteed kernel multiplier on another GPU.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

Notice the trade: the 67% retained configuration uses the same rounded 0.3M trainable parameter count as the 72% configuration, but gives up 0.3 GLUE average points for the reported 0.2× additional speedup. The useful comparison is a Pareto choice under a declared task and budget, not "more pruning is always better."

Figure 4 training cost

Figure 4 measures RoBERTa-Large on an NVIDIA RTX 3090 with batch size 32 and sequence length 128. The Light-PEFT setting retains 67% of foundation parameters and 0.3M trainable parameters. The paper reports 32% lower model-weight memory, 40% lower activation memory, and 39% lower peak memory. Its ten-batch forward and backward timing is 2.2× faster than baseline LoRA.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

The peak-memory result follows the mechanism taught earlier. Removing dimensions reduces both weight storage and intermediate activation shapes. The 2.2× number is not a claim that every batch, sequence length, or GPU sees that ratio. It includes the measured workload and implementation.

Table 5 inference boundary

Table 5 uses OPT-6.7B with batch size 96 and maximum length 100. It reports load memory and inference speed for foundation pruning and optional module pruning.

The paper therefore reports up to 1.6× inference speed and 48% lower load memory in this OPT-6.7B protocol. "48%" compares 6.5G with 12.5G approximately, and the 1.6× row also removes 75% of PEFT modules. Keep those axes together when reproducing the result.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

What the paper leaves untested

The evidence doesn't establish support or benefit for:

• modern frontier MoE post-training;
• multi-task adapter fleets sharing one base;
• online reinforcement-learning distribution shifts;
• QLoRA plus Light-PEFT combinations;
• distributed large-scale jobs with tensor, pipeline, or expert parallelism;
• current GLM-5.2 or DeepSeek-V4 model implementations;
• current CUDA, ROCm, or accelerator kernel stacks;
• continual pruning while adapters are loaded per request.

GLM-5.2 and DeepSeek-V4 are research directions for asking whether task-specific structured pruning transfers to newer architectures. Their names are not support evidence for this pinned release. A capability claim needs a model-specific implementation, loader test, benchmark protocol, and license check.

Permanent base masks change serving semantics

Ordinary multi-adapter serving relies on one shared base shape. Request A can select adapter A, request B can select adapter B, and both use the same frozen matrix dimensions. A structured Light-PEFT run physically removes heads and FFN coordinates from that base shape.

That creates a serving choice:

The paper's "plug-and-play" claim refers to switching masks and PEFT modules in its tested setup. It doesn't automatically mean that one production base can serve arbitrary adapters with one static kernel plan.^{[1]Reference 1Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruninghttps://aclanthology.org/2024.findings-acl.447/}

KV caches and compiled kernels add more state. A server that caches a plan for 16 heads can't reuse it for a 12-head pruned base. A cache keyed only by model name and adapter ID can return incompatible tensors if the base shape or pruning seed changes. Include base revision, shape signature, and adapter revision in cache keys.

Reinforcement-learning distribution shifts

Early scores come from a prefix of one data distribution. In supervised fine-tuning, later batches can already differ from the prefix. In online RL, the policy changes the data it receives, so the shift can be stronger:

1. The initial policy samples trajectories from distribution D_0.
2. Early masks rank coordinates using rewards and gradients from D_0.
3. Pruning changes policy capacity and removes some routes.
4. The new policy samples D_1, which can contain states absent from D_0.
5. A removed head or FFN coordinate may be important in D_1 even if it looked quiet in D_0.

This is a research concern, not a measured failure in the paper's static GLUE or QA protocol. An RL evaluation should log policy version, mask version, trajectory distribution, reward slices, and divergence before and after pruning. A held-out online stream can test whether reward collapse follows the physical cut.

If a system needs reversible exploration, keep the unpruned base or a set of checkpoints. A one-way physical slice is cheap to run but expensive to undo without rebuilding the original model.

Operational fragility and safe experiment design

The code is concise because it assumes a controlled research machine. A production adaptation needs extra checks.

Shape contract

Before loading adapter state, verify:

• base model identifier and revision;
• number of layers;
• surviving attention heads per layer;
• hidden size and head dimension;
• surviving FFN width per layer;
• target module names;
• LoRA rank or Adapter bottleneck width;
• bias and dropout configuration.

The source slices tensors in place and restores state dict entries by name. A different Transformers implementation can rename a module or store transposed weights. A name match isn't enough; compare tensor shapes.

Numerical contract

Mask penalties and first-order scores depend on gradients. Record:

• precision and autocast state;
• loss scale if mixed precision is enabled;
• gradient clipping;
• learning rate and mask learning-rate multipliers;
• NaN and infinity counts;
• evaluation interval used to sample mask history.

The repository uses separate attention and FFN mask learning-rate coefficients in MaskTrainer, then defaults both to one in arguments.py. A port that silently changes those multipliers changes the search problem.^{[2]Reference 2Light-PEFT Source Repository at 7b10c22https://github.com/gccnlp/Light-PEFT/tree/7b10c22026fe6fa283d127353a50d6ed7176a8aa}

Measurement contract

Time the complete workflow:

The paper's Table 1 explicitly includes estimation before pruning. A benchmark that starts its stopwatch after the masks exist will overstate speedup. A benchmark that includes data download but not model load will answer a different question. State both wall-clock definitions.

Failure isolation

Run these checks separately:

1. Import the pinned dependency set.
2. Load tokenizer and base model without PEFT.
3. Run one baseline forward and backward batch.
4. Run one estimation batch and inspect mask ranges.
5. Confirm all four record files exist.
6. Rebuild one pruned model and print shapes.
7. Restore one filtered adapter state dict.
8. Run one post-pruning batch.
9. Run the full continuation only after those checks pass.

This order separates environment errors from score errors and shape errors. It also avoids burning a long GPU job before discovering that the saved peft_dir points to another dataset.

Scope comparison with nearby methods

Light-PEFT combines pruning with PEFT; it isn't a replacement name for every efficient fine-tuning approach.

Light-PEFT's efficiency comes from paying for a change in the model's dense shapes. That makes its steady-state run different from a method that only gates execution or stores lower-precision weights.

Questions to ask before adapting the code

When moving beyond RoBERTa NLU, ask these in order:

Does the model expose the same structures?

The code knows RoBERTa attention and layer classes. A decoder-only model may store fused QKV, grouped-query heads, rotary embeddings, parallel residual paths, or gated FFN matrices. A head slice must respect those layouts. An MoE layer adds expert routing and capacity rules that this code doesn't model.

Can a global score compare coordinates?

Global FFN and rank quantiles assume scores share a meaningful scale. Layer normalization, module placement, and gradient scale can make one layer's numbers incomparable. Normalize or keep thresholds local only after checking the score distribution.

Is physical pruning legal for the deployment runtime?

A training fork can slice PyTorch linear layers. A fused inference engine may require a separate kernel for each head count, tensor-parallel shard, or expert layout. Prove that the loader, compiler, quantizer, and serving cache all understand the new shape.

Does the task distribution stay stable?

Static labels and online rewards answer different questions. For multi-task or RL use, hold out shifts and re-evaluate after every pruning boundary. Early importance isn't a lifetime contract.

Can the experiment be undone?

Keep original base weights, original adapter state, selected indices, and a manifest. A mask vector alone can't reconstruct removed matrix rows unless the original checkpoint remains available.

Takeaways

• PEFT reduces trainable state, but frozen backbone forward, backward activation, and weight-read costs remain.
• Light-PEFT spends a short estimation prefix to learn task-local evidence before physical structured pruning.
• Attention heads use scalar masks and layer-wise ranking in the reference path.
• FFN intermediate dimensions use scalar masks and can use a global threshold.
• L1 pressure helps masks shrink, while ratios choose the final capacity budget.
• PEFT module pruning ranks output change; rank pruning aggregates first-order weight × gradient scores.
• estimate.py writes .npy mask histories and .pkl PEFT records; train.py reads them before slicing tensors.
• The paper's tested scope is RoBERTa-Large, OPT-1.3B, OPT-6.7B, LoRA, and Adapter.
• Table 1, Figure 4, and Table 5 numbers are paper-reported under their stated hardware, batch, sequence, and timing protocols.
• The full repository has no root license; only bundled peft/LICENSE is Apache-2.0.
• Permanent base shapes complicate shared-base multi-adapter serving, cache keys, and reversible RL exploration.
• GLM-5.2, DeepSeek-V4, frontier MoE post-training, multi-task fleets, QLoRA combinations, and distributed large-scale jobs remain research directions, not support evidence.

Mastery quiz

Course complete: Return to the curriculum roadmap. You have now read seventeen project repositories. Keep source maps, test important claims against pinned code and primary sources, and use measured receipts to separate promising research from proven production behavior. Use your run manifest as final proof-of-skill checklist.