Privacy-Preserving Machine Learning

A support model is in its last pre-launch review. A red-team analyst pastes a private troubleshooting note, and the model completes a missing sentence with customer-specific detail. Names were stripped from the training rows, but the model can still depend on the note itself.

Responsible AI Governance established ownership, policy, and review gates. Differential privacy turns this incident into a neighboring-dataset question: if every record from one protected person disappeared before training, how much could the distribution of released outputs change?^{[1]Reference 1Calibrating Noise to Sensitivity in Private Data Analysis.https://people.csail.mit.edu/asmith/PS/sensitivity-tcc-final.pdf[2]Reference 2The Algorithmic Foundations of Differential Privacy.https://www.microsoft.com/en-us/research/publication/algorithmic-foundations-differential-privacy/}

That question names a promise redaction can't make. To answer it, you must choose the protected unit, bound its full contribution, add calibrated randomness, and count every later release against the same promise.

Start with the unit you promise to protect

Before choosing an epsilon, write one sentence: “one protected unit is ___.” Two datasets are neighbors when they differ by that protected unit. In support data, the blank might be one row, one conversation, one user, or one organization.

Suppose Alice contributed twelve messages. A row-level accountant considers one message at a time. A user-level review asks what happens when all twelve disappear together. Same dataset, different neighboring relation, different privacy promise.

Turn that promise into a release condition. For every possible output event $S$, an $(\varepsilon,\delta)$-private mechanism satisfies:^{[2]Reference 2The Algorithmic Foundations of Differential Privacy.https://www.microsoft.com/en-us/research/publication/algorithmic-foundations-differential-privacy/}

$$P(M(D)\in S)\le e^{\varepsilon}P(M(D')\in S)+\delta$$

Here $D$ and $D'$ are neighbors, and $M$ is randomized. $\varepsilon$ bounds the multiplicative change in an event’s probability; $\delta$ allows a small additive failure term.

Neither number travels alone. The protected unit, sampling, contribution bound, randomness, and composition assumptions explain what the pair actually promises.

One support customer contributes twelve training messages. A release claims user-level privacy but defines neighboring datasets by removing one message. What exactly is wrong with the claim?

Answer

The mechanism's declared adjacency protects one row, while removing the customer removes twelve rows. Its epsilon and delta therefore describe a row-level comparison, not the promised user-level comparison, unless the full user contribution is bounded and accounted for explicitly.

Bound each contribution before aggregation

Once the protected unit is fixed, bound its influence at that same boundary. Differentially private stochastic gradient descent (DP-SGD) usually computes each training example’s gradient separately, so standard DP-SGD bounds an example-level contribution. A user-level promise needs user-aware grouping or a valid conversion and accountant.^{[3]Reference 3Deep Learning with Differential Privacyhttps://arxiv.org/abs/1607.00133[4]Reference 4Fine-Tuning Large Language Models with User-Level Differential Privacyhttps://arxiv.org/abs/2407.07737}

For gradient $g_i$ and clip threshold $C$:

$$\bar g_i=g_i\min\left(1,\frac{C}{\lVert g_i\rVert_2}\right)$$

The norm operation leaves a gradient below $C$ alone and scales a larger one down to $C$. Average those clipped contributions, add calibrated noise through a Gaussian mechanism, and update the model. Clipping an already-averaged batch gradient comes too late: cancellation and reinforcement have hidden each individual contribution.

Predict before running the cell: with gradients $0.5$ and $5.0$ and a bound of $1.0$, which value changes, and what mean remains before noise? The calculation clips each scalar independently. It intentionally omits randomness, so it demonstrates sensitivity control rather than a private release.

The $5.0$ gradient becomes $1.0$, while $0.5$ stays unchanged. Their mean is therefore $0.75$, instead of the unclipped $2.75$.

Now change only the ownership: if both examples belong to Alice, row clipping leaves a combined contribution of $1.5$. A row-level bound of $1.0$ has not bounded Alice’s full influence, so it can't support a user-level claim with that same bound.

User-level training can sample users, aggregate each sampled user’s examples, and clip that combined gradient. A capped example-level route can also work when each user’s maximum example count is enforced and a group-aware accountant converts that exposure into a user-level guarantee.

The choice changes sampling, compute, and utility. Under fixed compute budgets, the cited LLM study finds that capped example-level sampling can win in some settings, while user-level sampling generally helps when users contribute diverse examples.^{[4]Reference 4Fine-Tuning Large Language Models with User-Level Differential Privacyhttps://arxiv.org/abs/2407.07737}

Neither route can enforce a user boundary when identity is unknown or the cap is unenforced.

The next cell makes that boundary explicit. It groups Alice’s two messages and Bao’s one message, clips each combined user gradient, and then computes the user-level average.

The example still has no randomness. A real mechanism needs correctly sampled noise, protected-unit-aware sampling, and an accountant that uses the same adjacency definition from training through release.

Why isn't clipping the final average gradient equivalent to bounding each declared protected contribution?

Answer

The final average can hide how much one protected unit contributed before aggregation. Row-level privacy clips each example first, while user-level privacy needs a bounded combined user contribution or valid group-aware accounting.

Account for every exposed release

Clipping controls sensitivity for one step; it doesn't tell you the exposure of the whole run. A privacy budget accumulates across repeated access to the same protected data. A privacy accountant combines the protected-unit sampling rate, noise multiplier, iteration count, and composition method into the resulting guarantee.^{[5]Reference 5Compute DP-SGD Privacyhttps://opacus.ai/api/compute_dp_sgd_privacy.html}

For separately accounted sequential releases, basic composition adds their epsilon values and their delta values. Specialized moments or Rényi accountants often give tighter valid bounds for a subsampled Gaussian training mechanism. Their result still depends on the sampling and release assumptions that the implementation actually meets.^{[3]Reference 3Deep Learning with Differential Privacyhttps://arxiv.org/abs/1607.00133}

Predict before running the cell: two earlier releases spent $1.2$ and $1.1$, and a candidate spends $0.9$. A simplified additive policy with a cap of $3.0$ should reject it. This ledger omits delta and is not a DP-SGD accountant; it illustrates a conservative release gate.

The ledger rejects the candidate because $3.2$ exceeds $3.0$. A real gate also tracks delta and checks that its accountant matches the protected unit, sampling scheme, mechanism, and disclosed output.

Extra checkpoints, private hyperparameter searches, and sensitive diagnostics can become additional releases. If the threat model exposes them, the accountant has to include them.

Two separately accounted releases spend epsilon 1.2 and 1.1, and a third needs 0.9. A simplified additive policy caps epsilon at 3.0. Can the third release proceed, and what other parameter must a real gate track?

Answer

No. The proposed additive total is 1.2 + 1.1 + 0.9 = 3.2, exceeding the 3.0 limit. A real release gate must also account for delta and verify that its composition method, protected unit, sampling assumptions, and disclosed outputs match the implemented mechanism.

Test attacks without mistaking them for proof

A membership inference attack asks whether a particular record was part of model training. In a black-box test, an attacker can compare prediction confidence or loss for known members and matched nonmembers.^{[6]Reference 6Membership Inference Attacks against Machine Learning Modelshttps://arxiv.org/abs/1610.05820}

Ask what a passing or failing audit can support. An effective attack exposes one disclosure path.

A failed attack only says that this attacker, population, threshold, and interface didn't separate the tested members from nonmembers. A formal differential-privacy claim still rests on the complete randomized mechanism and its accounting assumptions.

Use the audit alongside access control, retention, consent, training-data provenance, audit trails, and legal review. A mathematical bound narrows one class of disclosure; it doesn't authorize collection or release by itself.

At release review, an engineer should be able to trace the claim to a record of the protected unit, maximum records per user, clipping boundary, noise multiplier, sampling scheme, accountant, epsilon, delta, and every disclosed checkpoint. Reproduce Alice’s row-clipped contribution of $1.50$ beside her correctly user-clipped contribution of $1.00$, then verify the user-level mean of $0.625$ before noise.

Add a membership-inference audit as evidence about one tested interface. Reject the release when its only protection is redaction, a failed attack, an untracked checkpoint, or a row-level accountant attached to a user-level claim.