Privacy-Preserving Machine Learning

An enterprise customer support copilot is in its final pre-launch security review. A red-team analyst enters a snippet from an internal troubleshooting ticket: "Incident 4092: Customer reported recurring billing error on...". The model immediately completes the sentence with an unmasked credit card tail, a personal home address, and an internal note recording that the customer entered psychiatric medical leave.

The engineering team is stunned. They ran regex scrubbers across the training corpus before fine-tuning. Names were stripped, and user identifiers were replaced with random hashes. Yet the model still reproduced the customer's private history verbatim.

Scrubbing explicit names leaves unique phrase combinations, rare tokens, and semantic correlations completely intact. Deep language models have billions of parameters; when an unusual sequence appears in training data, gradient descent fits it tightly. Redaction can't change that fundamental memorization dynamic.

Differential privacy frames this problem as a mathematical neighboring-dataset contract: if every single record contributed by one protected person vanished before training began, how much could the probability distribution of released model outputs change?^{[1]Reference 1Calibrating Noise to Sensitivity in Private Data Analysis.https://people.csail.mit.edu/asmith/PS/sensitivity-tcc-final.pdf[2]Reference 2The Algorithmic Foundations of Differential Privacy.https://www.microsoft.com/en-us/research/publication/algorithmic-foundations-differential-privacy/}

Answering that question requires choosing the exact unit you promise to protect, bounding each unit's mathematical influence, injecting calibrated randomness, and tracking cumulative privacy loss across every released checkpoint.

We'll work within the central privacy model: a trusted infrastructure service accesses raw records to run training, but the released model weights and generated text must satisfy provable privacy bounds. Differential privacy doesn't replace database access controls or scrub internal logs. It establishes a formal limit on what an external observer can extract from the final system.

Why redaction fails against modern extraction attacks

Redaction treats privacy as a syntactic text-matching task. If a word matches an entity dictionary or regex pattern, you mask it. Real privacy attacks operate on semantic statistics and model capacity.

Modern privacy evaluation focuses on three primary attack vectors:

1. Training data memorization and extraction: Overparameterized neural networks easily memorize rare or unique training sequences. Attackers probe models with targeted prompt prefixes, greedy decoding, or beam search to extract memorized strings verbatim. Security teams evaluate this vulnerability using canary insertion: inserting artificial secrets with controlled repetition counts $r$ into the training data, then measuring how many exposures allow an attacker to reconstruct the secret. Without mathematical protection, models can memorize sequences seen only once or twice.
2. Membership inference attacks (MIA): An adversary tests whether a known document $x$ belonged to the training set $D$ ^{[3]Reference 3Membership Inference Attacks against Machine Learning Modelshttps://arxiv.org/abs/1610.05820}. Because standard training minimizes empirical loss on training examples, the model exhibits systematically lower loss $\mathcal{L}(\theta; x)$ and lower perplexity on members than on non-members. Attackers calibrate this signal by comparing the target model's loss to a public reference model's loss ($\mathcal{L}_{\text{target}}(x) - \mathcal{L}_{\text{ref}}(x)$) or zlib compression entropy. If the target model finds a sentence significantly more probable than the reference model does, it flags that record as training data.
3. Attribute reconstruction: An attacker combines partial knowledge of a user's record with query access to the trained model to infer omitted sensitive fields, such as diagnostic codes or salary brackets.

A failed empirical attack doesn't prove that your model is secure. It only proves that the specific auditor, threshold, and feature set you tested couldn't separate members from non-members. Differential privacy provides a mathematical upper bound across all possible attacks, including adversaries with arbitrary background knowledge.

Why does a passing membership inference test fail to guarantee that a customer's support ticket can't be extracted?

Answer

An attack test only establishes an empirical lower bound on privacy leakage for one specific adversary, interface, and test split. It doesn't bound what a stronger adversary with better auxiliary data or different decoding strategies can extract. Differential privacy provides a worst-case mathematical bound over all possible adversaries.

Start with the unit you promise to protect

Before configuring an epsilon or tuning noise scales, write one concrete contract: "one protected unit is ___." Two datasets are neighbors when they differ by that single protected unit.

In enterprise support data, that unit might be one row, one conversation ticket, or one customer account.

Suppose Alice submitted twelve support messages across three tickets. A row-level guarantee bounds what happens when one message is removed. A user-level guarantee bounds what happens when all twelve messages disappear simultaneously. That's the exact same corpus under two completely different neighboring relations, yielding two radically different privacy promises.

The definition also depends on the adjacency convention:

• Add/remove adjacency: Neighbor $D'$ contains one fewer (or one additional) protected unit than $D$, so $|D \Delta D'| = 1$. Dataset size changes by one.
• Replace-one adjacency: Neighbor $D'$ replaces one protected unit's complete value with another arbitrary valid value, keeping total population size $N$ fixed, so $|D \Delta D'| = 2$.

Replacing a vector bounded in $[-C, +C]$ can swing an aggregate sum from $-C$ to $+C$, spanning a distance of $2C$. Removing that vector only changes the sum by $C$. Replacing doubles the sensitivity bound compared to removal. You can't swap conventions without updating your sensitivity calculations.

Turn this into Cynthia Dwork's formal condition. For every possible output event $S \subseteq \text{Range}(M)$, a randomized mechanism $M$ satisfies $(\varepsilon,\delta)$-differential privacy if:^{[1]Reference 1Calibrating Noise to Sensitivity in Private Data Analysis.https://people.csail.mit.edu/asmith/PS/sensitivity-tcc-final.pdf[2]Reference 2The Algorithmic Foundations of Differential Privacy.https://www.microsoft.com/en-us/research/publication/algorithmic-foundations-differential-privacy/}

$$P(M(D) \in S) \le e^{\varepsilon} P(M(D') \in S) + \delta$$

The probabilities reflect the internal randomness of mechanism $M$, not randomness in picking the training dataset. The condition must hold in both directions ($D$ to $D'$ and $D'$ to $D$) for every neighboring pair and every measurable output event $S$.

Parameter $\varepsilon$ (epsilon) is the privacy budget. It sets a strict multiplicative ceiling on the log likelihood ratio an observer can gain about any individual's presence. When $\varepsilon = 0.5$, $e^{0.5} \approx 1.65$, meaning the odds of any model output can shift by at most $65\%$. When $\varepsilon = 8.0$, $e^{8.0} \approx 2981$, leaving a wide window for distinguishing members.

Parameter $\delta$ (delta) provides additive slack for unlikely tail events where the strict $e^{\varepsilon}$ ratio fails. In machine learning, $\delta$ isn't a minor failure percentage. Setting $\delta = 1/N$ is fatal: a mechanism could publish one raw customer record with probability $1/N$ and still satisfy the inequality mathematically! For a corpus of $N = 100{,}000$ training records, $\delta$ must be strictly smaller than $10^{-5}$, typically $10^{-6}$ or $10^{-7}$. Setting $\delta = 0$ gives pure differential privacy.

Consider a toy mechanism you can evaluate by hand. A support customer answers a sensitive yes/no question truthfully with probability $0.75$ and reports the opposite answer with probability $0.25$:

The largest possible probability ratio for either output is $0.75 / 0.25 = 3.0$. This randomized-response mechanism satisfies pure differential privacy with $\varepsilon = \ln 3 \approx 1.10$ and $\delta = 0$. It's a local differential privacy mechanism because noise is added on the client device before anyone else observes the response.^{[2]Reference 2The Algorithmic Foundations of Differential Privacy.https://www.microsoft.com/en-us/research/publication/algorithmic-foundations-differential-privacy/}

Training an LLM uses the central model instead. Gradients from thousands of examples are pooled, clipped, and perturbed inside a secure compute boundary before weights leave the cluster.

A machine learning engineer trains an LLM on 50,000 patient records and reports a privacy guarantee with epsilon = 2.0 and delta = 0.05. Why is this delta value unacceptable?

Answer

Delta represents the probability that the differential privacy bound fails completely. A delta of 0.05 means the mechanism could leak raw unperturbed records for 5% of the dataset (2,500 patients) while technically satisfying the mathematical inequality. In machine learning, delta must be strictly smaller than 1/N, typically 10^-5 or 10^-6 for this corpus size.

Mechanisms and sensitivity scaling

Differential privacy adds noise proportional to the query's global sensitivity. For any function $f$ mapping a dataset to a vector space, sensitivity measures the maximum change between any two neighboring datasets $D$ and $D'$:

$$\Delta_1 f = \max_{D \sim D'} \|f(D) - f(D')\|_1, \qquad \Delta_2 f = \max_{D \sim D'} \|f(D) - f(D')\|_2$$

Two foundational mechanisms inject this noise:

The Laplace mechanism adds independent noise drawn from $\text{Lap}(\Delta_1 f / \varepsilon)$ to each coordinate. It guarantees pure $(\varepsilon, 0)$-differential privacy. For scalar counts or low-dimensional database queries, Laplace noise is simple and effective.

In deep neural networks, the Laplace mechanism fails completely. In a model with $d$ parameters, the expected $L_1$ norm of independent Laplace noise scales linearly with $d / \varepsilon$. When updating a 100-million-parameter model, adding coordinate-wise Laplace noise injects so much variance that optimization collapses immediately.

The Gaussian mechanism adds noise drawn from $\mathcal{N}(0, \sigma^2 I)$, where standard deviation $\sigma$ satisfies:

$$\sigma \ge \frac{\Delta_2 f \sqrt{2 \ln(1.25/\delta)}}{\varepsilon}$$

The Gaussian mechanism delivers $(\varepsilon, \delta)$-differential privacy. Its required noise scales with $L_2$ sensitivity ($\Delta_2 f$) rather than $L_1$ sensitivity. Because the $L_2$ norm of a vector is bounded by its maximum coordinate times $\sqrt{d}$ rather than $d$, the Gaussian mechanism injects vastly less total variance into high-dimensional gradient vectors. That makes the Gaussian mechanism the standard engine for private deep learning.

Differentially private stochastic gradient descent

Standard stochastic gradient descent computes the loss gradient over a mini-batch, averages those vectors, and steps the parameters. That average is directly vulnerable to outliers: if Alice's private record has an unusually large gradient with norm $100.0$ while normal samples have norm $0.5$, Alice's record pulls the aggregate step toward her exact tokens.

Differentially private stochastic gradient descent (DP-SGD) fixes this vulnerability by bounding each contribution individually before summation ^{[4]Reference 4Deep Learning with Differential Privacyhttps://arxiv.org/abs/1607.00133}.

The DP-SGD algorithm proceeds in five explicit phases on each training step $t$:

1. Poisson subsampling: Form batch $B_t$ by including each training record independently with sampling probability $q = L / N$, where $L$ is the target batch size and $N$ is total dataset size.
2. Per-sample backpropagation: For each example $i \in B_t$, compute its individual parameter gradient $g_i(\theta_t) = \nabla_\theta \mathcal{L}(x_i, \theta_t)$ separately before any batch reduction occurs.
3. Per-sample clipping: Enforce an $L_2$ norm threshold $C > 0$ on each vector: $$\bar{g}_i(\theta_t) = g_i(\theta_t) \cdot \min\left(1, \frac{C}{\|g_i(\theta_t)\|_2}\right)$$ If $\|g_i\|_2 \le C$, the gradient is untouched. If $\|g_i\|_2 > C$, the gradient keeps its exact direction but its length shrinks to $C$.
4. Summation and Gaussian noise injection: Sum the clipped vectors and add calibrated Gaussian noise scaled to sensitivity $C$: $$\widetilde{g}_t = \sum_{i \in B_t} \bar{g}_i(\theta_t) + \mathcal{N}\left(0, \sigma^2 C^2 I\right)$$ Adding or removing one sample changes the sum by at most $C$, so the $L_2$ sensitivity of the summation is $\Delta_2 = C$.
5. Descent step: Divide the noisy aggregate by target batch size $L$ and update parameters: $$\theta_{t+1} = \theta_t - \eta_t \frac{\widetilde{g}_t}{L}$$

A frequent bug in custom implementations is clipping the batch average after aggregation: clip(mean(gradients), C). This bounds the final update vector's length to $C$, but neighboring dataset averages can still point in opposite directions, creating a worst-case difference of $2C$ between outputs. Clipping must occur per sample before aggregation to establish sensitivity $C$ for the batch sum.

To verify this geometry, test per-sample vector clipping on concrete gradient vectors. Outliers project onto the unit ball while in-bounds gradients remain untouched:

The outlier [3.0, 4.0] with length $5.0$ scales down to [0.6, 0.8], whose length is $\sqrt{0.6^2 + 0.8^2} = 1.0$. The gradient's direction is preserved, but its ability to pull the model parameters toward its private values is clamped.

Now consider what happens when Alice contributes multiple messages. If Alice wrote two tickets with gradients $0.5$ and $5.0$, example-level clipping bounds each message to $1.0$, leaving Alice with a combined influence of $1.5$. A row-level clip doesn't bound Alice's full user-level influence.

User-level fine-tuning groups each user's records, sums their gradients, and clips that combined user vector ^{[5]Reference 5Fine-Tuning Large Language Models with User-Level Differential Privacyhttps://arxiv.org/abs/2407.07737}. Grouping records per user binds total customer influence before aggregation. Here we test Alice's two tickets against Bao's single ticket using a fixed public normalizer of two:

Removing Alice changes the normalized sum by exactly $0.5$, which equals $C / m = 1.0 / 2$. User-level grouping enforces that Alice's entire presence can't shift the pre-noise aggregate by more than $C$.

Why must DP-SGD compute gradients per sample before batch reduction, rather than clipping the batch average gradient?

Answer

Clipping the batch average bounds only the length of the final step, but neighboring datasets could still produce batch averages that point in opposing directions, creating an unbounded difference. Per-sample clipping bounds each individual contribution to norm C, guaranteeing that adding or dropping one sample shifts the batch sum by at most C.

Account for every exposed release

Clipping bounds sensitivity for one training step. Training an LLM takes thousands of steps. A privacy accountant tracks how privacy loss accumulates over the full optimization run ^{[6]Reference 6Compute DP-SGD Privacyhttps://opacus.ai/api/compute_dp_sgd_privacy.html}.

If you apply basic composition, privacy budgets add linearly: running $T$ steps with step budget $\varepsilon_0$ yields total $\varepsilon = T \cdot \varepsilon_0$. A training run of $5{,}000$ steps with $\varepsilon_0 = 0.1$ would report $\varepsilon = 500$, an astronomically useless number.

Modern DP-SGD implementations rely on the Moments Accountant or Rényi Differential Privacy (RDP) ^{[4]Reference 4Deep Learning with Differential Privacyhttps://arxiv.org/abs/1607.00133[7]Reference 7RDPAccountanthttps://opacus.ai/api/accounting/rdp.html}. RDP measures the Rényi divergence of order $\alpha$ between neighboring output distributions:

$$D_{\alpha}(M(D) \| M(D')) = \frac{1}{\alpha - 1} \ln \int \left( \frac{P(M(D) = s)^\alpha}{P(M(D') = s)^{\alpha - 1}} \right) ds$$

For a Gaussian mechanism with Poisson subsampling probability $q = L / N$, Rényi divergences compose linearly across steps:

$$\varepsilon_{\text{total}}(\alpha) = T \cdot \varepsilon_{\text{step}}(\alpha) \approx T \cdot \frac{\alpha q^2}{\sigma^2}$$

Converting RDP back to $(\varepsilon, \delta)$-DP at the end of training yields an asymptotic bound:

$$\varepsilon \approx \frac{q \sqrt{2 T \ln(1/\delta)}}{\sigma}$$

Notice the factor $q = L / N$. This is subsampling amplification: because an attacker only sees an individual's data if that individual was included in the Poisson batch, privacy protection is amplified by $q$. In large datasets where $q = 10^{-3}$ or $10^{-4}$, subsampling amplification reduces cumulative epsilon by orders of magnitude compared to naive composition.

You can evaluate this directly with Opacus's RDPAccountant. We compare cumulative epsilon across 300 and 600 training steps under fixed sampling rate $q = 0.01$ and noise multiplier $\sigma = 1.0$:

Doubling training steps from 300 to 600 increases epsilon from $1.451$ to $1.748$, rather than doubling it to $2.90$. Sublinear scaling allows practical models to train for thousands of steps while maintaining strict privacy budgets.

Once training finishes, what about serving customer inference requests? Under the post-processing theorem, applying any arbitrary function to the output of an $(\varepsilon,\delta)$-differentially private algorithm without accessing private data costs zero additional privacy budget ^{[2]Reference 2The Algorithmic Foundations of Differential Privacy.https://www.microsoft.com/en-us/research/publication/algorithmic-foundations-differential-privacy/}. Serving ten million inference queries from the released model doesn't spend any extra training epsilon.

Post-processing applies only to operations that touch public data and the released weights. If you compute validation scores on private held-out records, run hyperparameter grid searches on private logs, or save unnoised intermediate checkpoints, those operations expose private data outside the accounted transcript and spend separate privacy budget.

Release policies track this cumulative exposure across candidate models using release ledgers:

The release ledger blocks the candidate because total epsilon ($3.2$) exceeds the release cap ($3.0$). Using Decimal prevents floating-point rounding errors from approving unauthorized model deployments.

The privacy-utility trade-off in modern LLMs

Applying DP-SGD to a multi-billion-parameter language model exposes a brutal geometric obstacle: the curse of dimensionality.

When you add Gaussian noise $Z \sim \mathcal{N}(0, \sigma^2 C^2 I_d)$ to a model with $d$ parameters, the expected Euclidean norm of the injected noise vector is:

$$\mathbb{E}[\|Z\|_2] \approx \sigma C \sqrt{d}$$

In a 7-billion-parameter model, $\sqrt{d} \approx \sqrt{7 \times 10^9} \approx 83{,}666$. If you set clipping threshold $C = 1.0$ and noise multiplier $\sigma = 1.0$, each update step injects a random noise vector with norm exceeding $80{,}000$ into the weights!

True gradient sums from a standard mini-batch of 64 or 128 examples have norm on the order of $50$ to $100$. The random noise vector dwarfs the learning signal by a factor of nearly a thousand. Full-parameter DP-SGD on large models fails to converge unless you scale batch sizes into the tens of thousands ($L \ge 65{,}536$), which requires immense compute cluster memory.

Parameter-efficient fine-tuning with DP-LoRA eliminates this dimensionality trap ^{[8]Reference 8LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685[5]Reference 5Fine-Tuning Large Language Models with User-Level Differential Privacyhttps://arxiv.org/abs/2407.07737}.

Instead of updating all weights $W_0 \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$, Low-Rank Adaptation freezes the pre-trained base model completely and injects trainable low-rank decomposition matrices:

$$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A, \qquad A \in \mathbb{R}^{r \times d_{\text{in}}}, \quad B \in \mathbb{R}^{d_{\text{out}} \times r}$$

Setting rank $r = 8$ reduces the number of trainable parameters from billions down to a few million (often $< 0.1\%$ of total parameters).

In DP-LoRA, gradients are computed, clipped, and perturbed only for matrices $A$ and $B$. The dimension $d_{\text{train}}$ drops from $7 \times 10^9$ to $5 \times 10^6$. Because noise norm scales with $\sqrt{d_{\text{train}}}$, the total injected noise norm drops by a factor of $\sqrt{7000 / 5} \approx 37\times$. The gradient signal-to-noise ratio jumps dramatically, allowing DP-LoRA to learn domain tasks with high fidelity under strict privacy budgets.

This architecture creates an important pre-training privacy asymmetry:

1. Pre-training corpus exposure: The foundation model was trained on trillions of public and crawled web tokens without differential privacy. It may already have memorized text from those public sources. DP fine-tuning doesn't purge or scrub pre-existing foundation model memorization.
2. Fine-tuning corpus protection: The fine-tuning phase uses DP-SGD on proprietary domain data (such as medical records or customer support logs). The resulting adapter weights guarantee that an attacker querying the fine-tuned model can't extract or reconstruct private records from that fine-tuning dataset beyond the accounted $(\varepsilon, \delta)$ bound.

By anchoring on frozen pre-trained representations and training only low-rank adapters with DP-SGD, teams ship enterprise copilots that master complex domain tasks while providing provable privacy guarantees.