Data Labeling and Human Feedback

Continue the fictional model-promotion assistant: it proposes candidate C17 for a 10% rollout. Two reviewers inspect competing answers. Candidate A cites the required policy approval and canary metric threshold; Candidate B claims that it has already migrated production traffic and attempts an unauthorized deployment command. A preference label seems straightforward. Then the reviewers notice a corporate email in the context and discover that the prompt is reserved for a safety benchmark. Even a clear human judgment can be the wrong training record.

Privacy-Preserving ML examined how training algorithms limit what weights reveal about private contributions. That defense doesn't sanitize traces before raters read them or make their judgments correct. Here, decide which data you are authorized to use, limit reviewer exposure, write a rubric, and separate training from evaluation. A versioned dataset record documents those decisions; access controls enforce who may read the data. Hashes help detect some overlaps, but they don't lock a test set. Our teaching fixtures build promotion-feedback-v12 while reserving attack-014 for promotion-eval-v5. All counts and selector results below are synthetic examples, not measurements from a deployed assistant.

Clean data before it reaches selectors or raters, and reserve evaluation cases before choosing training items. Keep a development set for tuning selector hyperparameters and model checkpoints. The private test supports a final comparison; repeated tuning against its results can compromise independence even when no test row enters training. Unsafe traces require incident triage rather than automatic approval as targets.

Separate records by the job they do

Before anyone labels a single trace, decide what exact question its record must answer. A demonstration says what behavior to imitate; a pointwise assessment scores how one answer fares against a concrete standard; a preference pair says which of two competing answers wins; an evaluation fixture asks whether a future model candidate improved; an incident record preserves forensic evidence of what broke. The promotion assistant requires all five data modalities for distinct operational jobs:

Pointwise assessments need anchored rubrics. One reviewer's 4 can be another's 2 on a vague helpfulness scale. Explicit criteria (5 = cites the required canary suite and approval ID; 3 = mentions the suite but omits approval; 1 = invents authorization) reduce that ambiguity. They don't eliminate drift: review shared examples and revisit the anchors when the task changes. Treat 1–5 ratings as ordinal unless you can justify the distances between scores.

Preference pairs answer a different question: given the same prompt $x$, which completion should win? InstructGPT collected demonstrations for SFT, then rankings of 4 to 9 outputs per prompt. It converted a ranking of $K$ outputs into $\binom{K}{2}$ comparisons for a reward model used by PPO.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} Under the Bradley-Terry formulation, the probability that $y_w$ beats $y_l$ is modeled using a scalar reward $r_\psi$:

$$P(y_w \succ y_l \mid x) = \sigma\left(r_\psi(x, y_w) - r_\psi(x, y_l)\right) = \frac{1}{1 + \exp\left(-\left(r_\psi(x, y_w) - r_\psi(x, y_l)\right)\right)}$$

Direct Preference Optimization (DPO) optimizes policy $\pi_\theta$ on pairs $(x, y_w, y_l)$ without fitting a separate reward model.^{[2]Reference 2Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} For a reward $r$ and its optimal KL-regularized policy $\pi_r^*$, DPO derives $r(x, y) = \beta \log \frac{\pi_r^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$. The prompt-only term $Z(x)$ cancels in reward differences. Parameterizing the policy yields:

$$\mathcal{L}_{\text{DPO}}(\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$

Here $\beta > 0$ controls the strength of the KL penalty against reference model $\pi_{\text{ref}}$, which must give positive probability to the compared completions for these log ratios. Earlier summarization systems also used human comparisons to fit reward models.^{[3]Reference 3Learning to Summarize from Human Feedback.https://arxiv.org/abs/2009.01325}

Relative preference doesn't certify that the winner is safe. If A dumps private evaluation notes and B executes an unauthorized deployment command, choosing A increases its relative preference under the loss. It doesn't necessarily teach a specific leakage behavior, but neither answer meets this pipeline's target requirements. Our policy marks the pair BOTH_BAD and routes it to incident review. Other objectives can learn rankings between unacceptable answers; a binary export still needs an explicit policy for what its labels mean.

Try the routing decision before looking at the code. Prompt attack-014 must route to frozen evaluation and incident review, eligible-102 to the preference queue, and leak-008 to incident review. A single trace can have two non-training jobs, but a reserved or unsafe trace must never enter a training queue.

If attack-014 enters training, it no longer measures unseen generalization. A non-reserved incident can later inspire a hand-crafted gold correction or clean preference record, but the raw unsafe trace isn't an approved target. Preserve that lineage and verify any derived record against holdout sets before promotion.

Redact before selection or review

The redaction boundary belongs at ingestion. Production text must be sanitized before it reaches active selectors, embedding services, annotation platforms, external review vendors, or debug logs. Each downstream system can store cached copies or embeddings even if a candidate is never selected for human labeling.

InstructGPT split prompts by user ID and filtered personally identifiable information from its training split. Its API prompts came from the Playground with customers informed of training use; the paper says it didn't use production API customer data.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That is a particular collection policy, not permission to reuse arbitrary application logs.

For a promotion-assistant trace, preserve what changes the policy judgment:

• Candidate ID, rollout traffic percentage, cited evaluation suite, and policy rule.
• Whether an automated promotion, exception approval, or credential access required sign-off.
• The assistant's proposed action and final deployment gate decision.
• A stable pseudonymous case identifier for joining review metadata later.

Strip what the human reviewer doesn't need:

• Engineer names, personal email addresses, phone numbers, or corporate user IDs.
• Free-form employee chat messages unrelated to the deployment decision.
• API keys, session tokens, or internal credentials copied into retrieved context.

The fixture below replaces three known formats with pseudonyms, including a commit ID that this task doesn't require reviewers to see. Repeated values keep the same placeholder within a trace. This is pseudonymization, not anonymity or a complete PII detector: writing style, surrounding facts, and unsupported secret formats can reveal sensitive information. Presidio's own documentation warns that automated detection cannot guarantee finding all sensitive data.^{[4]Reference 4Presidio: Data Protection and De-identification SDK.https://github.com/data-privacy-stack/presidio}

The mapping exists in memory here; it hasn't been written to disk. In a production pipeline, an encrypted, access-controlled key vault holds reversible mappings. The annotation queue receives only cleaned text and a redaction version tag. Placeholders reset for each trace, so <OWNER_1> in two separate records doesn't link them to the same engineer. Rejecting existing placeholder patterns prevents raw user input containing <OWNER_1> from colliding with generated tokens.

Write the rubric before collecting preferences

Two completions to the same C17 rollout request can sound fluent while one exposes internal evaluation notes and the other triggers an unauthorized API call. Asking reviewers to "pick the better response" leaves standards unstated. Thumbs-down feedback is equally uninformative: it signals that something failed, but gives no clue whether the failure was a security breach, a hallucinated citation, or a minor formatting quirk. For the promotion assistant, write a strict hierarchical rubric:

1. Safety and authorization: Reject any response that executes a canary rollout without authorization, exposes protected test notes, or treats untrusted retrieved text as a verified sign-off.
2. Technical correctness: Prefer a response that accurately cites candidate ID, evaluation suite, and rollout traffic percentage.
3. Actionability: Prefer a response that specifies the required next operational step: propose-for-approval, exception review, or security review.
4. Clarity and tone: Among equally safe, correct, and actionable answers, prefer direct, professional language without conversational fluff.

Safety outranks style under this rubric. Two unsafe candidates receive BOTH_BAD. Equally acceptable answers can receive a tie; missing evidence calls for abstention. Those are different outcomes, so the annotation interface should expose both rather than forcing a winner.

Try a less obvious pair: A says, "The suite passed; request approval before changing traffic." B says, "The suite passed; approval is already recorded." Both are fluent. If reviewers can't inspect the approval record, can they determine correctness? They need more evidence, not a style tiebreaker. Give both reviewers the same authorized context, hide model identities where practical, and balance answer order.

The router below checks safety and correctness criteria. Reviewer judgments populate these attributes. Safe Candidate A versus incorrect Candidate B chooses A. Leaking Candidate C versus unauthorized Candidate D escalates both to incident review.

An unsafe rejected answer can supply a negative contrast when the chosen answer is reviewed as safe and correct. The code implements only the first two rubric levels: two safe, correct answers still need actionability and clarity review. It cannot infer a genuine tie from those flags. Keep unresolved disputes and ties out of this binary export unless its loss handles them explicitly. Type annotations don't validate runtime inputs, so the dataclasses reject unknown or non-Boolean flags.

Store enough provenance to replay the judgment

To explain why a pair entered promotion-feedback-v12, preserve enough evidence to reconstruct its judgment while limiting access to private identifiers:

Collect only reviewer metadata required for quality control. Restrict access and define retention policies; bias analysis isn't an excuse to amass unnecessary personal details about annotators.

Measure agreement, then repair disagreement

Two qualified reviewers can reach different conclusions. Inter-annotator agreement (IAA) describes consistency in the reviewed sample; some coefficients compare it with a specified chance model. High agreement doesn't prove fairness, correctness, or consistency on cases you didn't sample. Measure independent reviews before adjudication: agreed labels produced by a consensus meeting aren't an independent reliability study.

Raw agreement can hide a base-rate effect. Suppose Reviewer 1 approves 92% of cases and Reviewer 2 approves 88%. Independent biased coins with those approval rates would agree with probability:

$$p_e = (0.92 \times 0.88) + (0.08 \times 0.12) = 0.8192 \approx 0.819$$

A raw agreement of 84% is close to that 81.92% benchmark. Cohen's kappa measures agreement relative to chance estimated from each reviewer's label proportions.^{[5]Reference 5A Coefficient of Agreement for Nominal Scaleshttps://doi.org/10.1177/001316446002000104} The chance model is a comparison baseline, not evidence that either person literally guessed.

Let $N$ denote the total number of duplicate-reviewed cases evaluated across $k$ categories. Let $n_{ij}$ denote the count of items that Reviewer 1 assigned to category $i$ and Reviewer 2 assigned to category $j$. The observed proportional agreement $p_o$ sums the diagonal:

$$p_o = \frac{1}{N} \sum_{i=1}^k n_{ii}$$

Next, compute the marginal proportions for each rater. For category $i$, Reviewer 1's marginal probability is $p_{i\cdot} = \frac{1}{N} \sum_{j=1}^k n_{ij}$, and Reviewer 2's marginal probability is $p_{\cdot i} = \frac{1}{N} \sum_{j=1}^k n_{ji}$. Under the null hypothesis that ratings are statistically independent, the expected chance agreement $p_e$ is:

$$p_e = \sum_{i=1}^k p_{i\cdot} p_{\cdot i}$$

Cohen's kappa measures the fraction of agreement achievable beyond chance that was actually observed:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

If $p_e < 1$, perfect matching gives $\kappa = 1$; matching chance gives $\kappa = 0$, and worse-than-chance agreement gives $\kappa < 0$. If both reviewers use only the same single category, $p_o = p_e = 1$ and kappa is undefined.

Fleiss' kappa supports nominal categories with a constant $m \ge 2$ ratings per item; different people may review different items. It uses pooled category proportions for chance agreement.^{[6]Reference 6Fleiss' and Randolph's kappa multi-rater agreement measurehttps://www.statsmodels.org/dev/generated/statsmodels.stats.inter_rater.fleiss_kappa.html} It is not identical to Cohen's kappa when two named raters have different marginals. Let $N$ be the item count, $k$ the category count, and $n_{ij}$ the number of ratings of item $i$ assigned to category $j$.

The proportion of all assignments to category $j$ across the entire dataset is:

$$p_j = \frac{1}{N m} \sum_{i=1}^N n_{ij}$$

The agreement $P_i$ measures the fraction of agreeing ordered rater pairs out of $m(m - 1)$:

$$P_i = \frac{1}{m(m - 1)} \sum_{j=1}^k n_{ij}(n_{ij} - 1) = \frac{1}{m(m - 1)} \left( \sum_{j=1}^k n_{ij}^2 - m \right)$$

The overall mean agreement $\bar{P}$ and the expected chance agreement $\bar{P}_e$ are:

$$\bar{P} = \frac{1}{N} \sum_{i=1}^N P_i, \quad \bar{P}_e = \sum_{j=1}^k p_j^2, \quad \kappa_{\text{Fleiss}} = \frac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e}$$

For missing labels or varying rater counts, consider Krippendorff's alpha with the appropriate nominal or ordinal distance.^{[7]Reference 7Computing Krippendorff's Alpha-Reliabilityhttps://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf} Weighted Cohen's kappa is another option for two raters on an ordinal scale.^{[8]Reference 8cohen_kappa_scorehttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html} The unweighted formulas here treat every category mismatch alike.

InstructGPT reported $72.6 \pm 1.5\%$ raw agreement among its training labelers.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That 2022 task-specific result isn't a universal acceptance threshold. Our synthetic table has 120 duplicate-reviewed cases and three categories: A better, B better, and Tie.

From this table:

$$p_o = \frac{42 + 38 + 27}{120} = \frac{107}{120} \approx 0.892$$ $$p_e = \frac{46 \times 48 + 44 \times 42 + 30 \times 30}{120^2} = \frac{2208 + 1848 + 900}{14400} = \frac{4956}{14400} \approx 0.344$$ $$\kappa = \frac{107 \times 120 - 4956}{14400 - 4956} = \frac{7884}{9444} \approx 0.835$$

Compute from full counts before rounding. Both functions reject malformed counts and return None when their chance-corrected denominator is zero:

The example clears a locally chosen threshold ($0.835 \ge 0.65$); 0.65 is not a standard reliability certificate. Report the sample, rater population, category mix, and uncertainty. A bootstrap should resample independent cases or source groups, not pretend correlated traces are independent. Investigate the 13 disagreements:

1. Senior domain arbitration: A designated staff engineer or safety specialist reviews contentious pairs (for example, whether an ambiguous canary log entry constitutes valid sign-off) to issue an authoritative ruling.
2. Rubric defect triage: When raters split repeatedly on the same failure pattern, the rubric itself is ambiguous. File a documentation bug, update the guideline version (e.g. promotion-rubric-v4.1), and attach an anchored gold exemplar.
3. Subjective preference retention: Tone or formatting may admit several reasonable choices. Preserve individual judgments or mark a tie when appropriate. Decide whether the training objective needs a shared policy, a distribution of preferences, or user-specific behavior instead of assuming every difference has one correct answer.

Agreement metrics depend heavily on class balance. When one category dominates, $p_e$ approaches $1.0$ and kappa becomes unstable or undefined.^{[8]Reference 8cohen_kappa_scorehttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html} Always report raw agreement, contingency counts, and category distributions alongside kappa.

Select cases that might teach something new

With a finite budget, which unlabeled cases should people review next? Active learning uses a selection rule to seek more useful labels. The rule estimates value; it doesn't know the eventual training gain.^{[9]Reference 9Active Learning Literature Surveyhttps://burrsettles.com/pub/settles.activelearning.pdf} Random sampling is a useful representative baseline and audit method. It can also supply valuable training cases.

Compare three approaches:

1. Uncertainty sampling: Queries items where the current model policy is most uncertain. Common formulations include:
   • Least confidence: $U_{\text{LC}}(x) = 1 - \max_y P(y \mid x)$
   • Margin sampling: $U_{\text{Margin}}(x) = 1 - (P(y_1 \mid x) - P(y_2 \mid x))$ (queries cases with the narrowest gap between top two candidates)
   • Shannon entropy: $H(x) = -\sum_y P(y \mid x) \log_2 P(y \mid x)$
   • Failure mode: Uncertainty may reflect corrupted text or ambiguous labels. A batch can also contain many rephrasings of the same difficult case.
2. Diversity sampling: Spreads the labeling quota across different regions of representation space.
   • Core-set selection: Choose $B$ centers to minimize the largest distance from a pool item to its nearest center: $\min_{S \subseteq \mathcal{U}, |S|=B} \max_{x \in \mathcal{U}} \min_{s \in S} \|z_x - z_s\|_2$.^{[10]Reference 10Active Learning for Convolutional Neural Networks: A Core-Set Approachhttps://arxiv.org/abs/1708.00489}
   • Embedding clustering: Cluster trace embeddings and select actual examples near centroids, or select medoids. A $k$-means centroid needn't be a real example; geometric clusters needn't match functional domains.
   • Failure mode: Distance can emphasize irrelevant or already easy cases.
3. Hybrid active selection: Our heuristic adds uncertainty to a normalized farthest-first distance bonus inspired by $k$-center selection: $$x^* = \arg\max_{x \in \mathcal{U} \setminus S} \left[ \lambda \cdot U(x) + (1 - \lambda) \cdot \frac{\min_{s \in S} \text{dist}(z_x, z_s)}{\max_{u \in \mathcal{U} \setminus S} \min_{s \in S} \text{dist}(z_u, z_s)} \right]$$

Here $U(x) \in [0,1]$ is a supplied uncertainty score and $\lambda \in [0,1]$ balances it against distance. Seed the nonempty set $S$ with a highest-uncertainty item; if every remaining distance is zero, give all items a zero distance bonus. This is not the summed-similarity facility-location objective, and adding $U$ doesn't inherit a pure $k$-center approximation guarantee. Sener and Savarese's method also accounts for already labeled centers; our toy starts without them. Its CNN results aren't a guarantee for LLM preference training.^{[10]Reference 10Active Learning for Convolutional Neural Networks: A Core-Set Approachhttps://arxiv.org/abs/1708.00489}

For this binary example, $p_a$ estimates the probability that a reviewer prefers A, not the probability of an LLM's next token. The supplied probabilities assume a usable binary choice; ties, both-bad cases, and missing evidence need their own routes. Predict the ranking before running:

All three metrics produce the same ranking here: for two classes, each increases as $p_a$ approaches 0.5.^{[9]Reference 9Active Learning Literature Surveyhttps://burrsettles.com/pub/settles.activelearning.pdf} They can rank multiclass cases differently. High uncertainty may flag a useful edge case or corrupted text; an overconfident predictor can miss a dangerous error. Audit calibration and retain representative random review.

Try four review slots across six eligible candidates. These coordinates and normalized $U$ scores are invented independently of the preceding probability example. Uncertainty alone picks three canary cases plus one injection case. Will a distance bonus change the batch?

The hybrid batch reaches all four known domains. It might still lose to random selection after training: distance and uncertainty are proxies, and learning value depends on the task, representation, labels, and training procedure.

Test selection without contaminating evaluation

An active selector can construct an impressive queue and still degrade the downstream system. Always evaluate a selection policy as an empirical experiment against a randomized baseline:

1. Reserve training, development, and private test partitions before running selection.
2. Draw comparable batches from the training pool: one randomly, one via the selector.
3. Label both batches under identical rubrics with the same reviewer pool.
4. Start from the same baseline checkpoint and use comparable training settings and compute budgets.
5. Evaluate downstream gains across development benchmarks, including security and accessibility slices.
6. Repeat across selection and training seeds; compare gains, uncertainty, and total review cost.

Tune the selector on development results, reserving promotion-eval-v5 for a final, pre-registered release check. Iteratively altering selector hyperparameters after observing test scores adapts the selector to the test suite, destroying independence.^{[11]Reference 11Cross-validation: evaluating estimator performancehttps://scikit-learn.org/stable/modules/cross_validation.html}

Choose the independence boundary before splitting: related traces may share a user, source document, incident, or template ancestor. Keep such groups together when the claim requires unseen groups, and consider a time-based holdout for future behavior. A broad skill such as "resist prompt injection" can appear in both training and test; withholding one skill entirely answers a different question.

IDs alone miss renamed records. Content fingerprints can find some duplicates. This example applies Unicode NFC normalization, case folding, and whitespace collapsing, then uses the full SHA-256 digest. That policy suits this prose fixture; case or whitespace changes may alter code and identifiers, so don't apply it blindly to every artifact.

The fingerprint flags sneaky-override as identical under this normalization policy. It isn't a salted hash, authentication mechanism, access barrier, or anonymity guarantee. Guessable prompts can be matched against published digests, so keep sensitive registries protected. Check prompts, chosen/rejected completions, demonstrations, and relevant retrieved material, not just the prompt column.

For possible paraphrases, combine overlap heuristics with trusted source lineage. This English toy uses token-trigram Jaccard overlap and a supplied family tag. The tag identifies a shared template ancestor, not every example of the same skill:

No hit means no overlap detected by these checks, not proof of independence. Short texts with fewer than three tokens produce no trigrams; this tokenizer also misses non-ASCII language structure. Paraphrases can evade overlap thresholds, while shared boilerplate can cause false positives. Investigate suspected overlaps and derive family tags from host-controlled source metadata: a nonempty caller-provided tag isn't trusted lineage.

If contamination is found before training, remove the source and its derivatives and recheck the candidate dataset. If the model has already trained on it, deleting a row or editing the manifest doesn't undo that exposure. Retrain from an uncontaminated checkpoint or use a fresh independent test for the claim.

Measure label efficiency, not label volume

Use hypothetical results to practice the calculation: each batch has 400 accepted labels. Supplied development scores change from 61% to 64% for random selection and to 68% for hybrid selection. This code doesn't train or evaluate a model:

For these supplied scores, hybrid uses 57.1 accepted labels per percentage point gained versus 133.3 for random. Equal accepted-label counts aren't equal spending: include rejected attempts, triage, selector inference, and training costs. Repeated runs and paired evaluation uncertainty matter before claiming a reliable improvement.

Treat model judges as assistants, not ground truth

An LLM judge can help prioritize review, but test its behavior on your task. The 2023 MT-Bench study found position and verbosity effects and investigated possible self-enhancement bias. Its authors explicitly said their evidence couldn't establish that last effect conclusively.^{[12]Reference 12Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Don't convert findings about particular judges into a guarantee about every current model.

Run a swap test on a human-reviewed gold set, with model identities hidden and answers presented in both orders. Map position labels back to answer identities before comparing. The following supplied labels illustrate the audit; no hosted judge is called:

Original-order agreement is perfect, but two mapped choices change. That fails our illustrative check. Three examples don't establish a judge's accuracy or a universal 90% threshold. Repeat balanced comparisons with recorded prompts and decoding settings: stochastic variation can also cause flips, so one inconsistent pair doesn't establish systematic position bias. Audit ties, unsafe choices, and difficult task slices as well as aggregate agreement.

Constitutional AI used human-authored principles for model critiques/revisions and AI preference feedback, followed by supervised training and reinforcement learning.^{[13]Reference 13Constitutional AI: Harmlessness from AI Feedback.https://arxiv.org/abs/2212.08073} Its original method didn't require human harmfulness labels. For our assistant, human review and task-specific calibration remain design choices needed to assess whether synthetic labels follow our policy. Neither synthetic labels nor human labels belong in training if their source is a reserved test case.

Promote a versioned feedback dataset

When a batch clears review, package its provenance metadata with the artifact. Document source windows, redaction and rubric versions, agreement reports, and snapshot digests. Datasheets for Datasets proposes documentation of motivation, composition, collection, recommended uses, and related questions.^{[14]Reference 14Datasheets for Datasetshttps://arxiv.org/abs/1803.09010}

This synthetic manifest records a batch's intended documentation fields. The code checks key presence only; it doesn't validate types, verify reports, reconcile counts, or inspect the dataset:

Gate the full data release

A manifest doesn't prove its claims. Link the actual dataset snapshot to its reports and training run, and verify that counts and versions refer to that artifact. The following local gate checks trusted, supplied summaries: completed overlap checks, no reported overlap, a local agreement threshold, and no unresolved cases. It neither performs those audits nor guarantees safe model behavior:

The draft reports an ID overlap and is blocked. The repaired fixture passes these supplied-summary checks. In a real pipeline, changing the findings list isn't a repair: fix and re-audit the artifact before training an SFT or DPO candidate. Its resulting behavior still needs evaluation.

Try the review boundaries