Data Labeling and Human Feedback

CodeAssist's code-review assistant can't pass the governance gate from the last lesson without a trustworthy dataset record. Its team has attack traces, CI failure investigations, and ordinary code-review conversations, but raw logs aren't training data. A few contain private details. Others should remain frozen tests forever. In some rows, both answers are bad, so choosing a "winner" would teach the model the wrong lesson.

Build the missing data pipeline: turn reviewed traces into versioned feedback data while preserving a private evaluation set that can honestly measure whether the assistant improves.

Separate records by the job they do

Human feedback isn't one kind of label. A model team commonly needs several different artifacts:

Pointwise assessments need anchored criteria so reviewers interpret a category or score consistently. Preference pairs answer a different question: given the same prompt, which answer is better? InstructGPT used human-written demonstrations and ranked model outputs in its post-training pipeline.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} Direct Preference Optimization (DPO) uses preference pairs to optimize a policy without first fitting a separate reward model.^{[2]Reference 2Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Those methods don't mean every reviewed log belongs in training. Holdout examples and safety incidents have different jobs.

The first routing rule is strict: a case reserved for evaluation doesn't enter the training queue. It may also be an incident that needs investigation.

Routing seems administrative until leakage happens. If attack-014 is trained on, the next evaluation no longer answers "does the system generalize to this attack?" It answers "did it remember a case we revealed?"

Redact before selection or review

The governance chapter required minimized evidence. The same rule applies earlier in the feedback pipeline: production text must be cleaned before it reaches a selection service, external reviewer, or annotation interface.

For a code-review trace, preserve what changes the judgment:

• Repository, diff hunk, failing test, and cited policy rule.
• Whether a command, deploy, or secret access required approval.
• The assistant's proposed action and final gate decision.
• A stable pseudonymous case ID for joining records later.

Remove what the reviewer doesn't need:

• Engineer name, email, phone number, or full commit identifier.
• Free-form incident notes unrelated to the label.
• Secrets, tokens, or internal credentials copied into retrieved context.

This small example uses stable replacements so two occurrences of the same identifier still match during review.

The reversible mapping, if it must exist, belongs in a separate access-controlled store. The annotation record needs the redaction version and the cleaned text, not the engineer's identity.

Write the rubric before collecting preferences

A preference interface is only as good as the decision rule behind it. "Choose the better answer" leaves reviewers to invent their own priorities. For CodeAssist's code-review assistant, write a rubric in descending order:

1. Safety and authorization: Reject any answer that reveals protected notes, exports secrets, or runs a privileged command outside the approval rule.
2. Technical correctness: Prefer an answer that accurately cites the failing test, diff hunk, or repo policy.
3. Actionability: Prefer a response that states the next safe command, evidence link, or reviewer handoff.
4. Clarity and tone: Among equally safe and correct answers, prefer direct and respectful language.

Safety outranks style. If both candidates break the first rule, record BOTH_BAD and send the case to incident review. Don't create a chosen/rejected training pair from two unsafe answers.

Store enough provenance to replay the judgment

Each accepted preference record should identify:

Collect only reviewer attributes needed for a legitimate analysis, protect them with access controls, and define their retention. Bias analysis isn't an excuse to collect personal information without a purpose.

Measure agreement, then repair disagreement

Two trained reviewers can still read a rule differently. Inter-annotator agreement (IAA) measures whether labels are consistent enough for their intended use.

For two reviewers choosing A, B, or Tie, Cohen's kappa compares observed agreement with agreement expected from each reviewer's label frequency.^{[3]Reference 3A Coefficient of Agreement for Nominal Scaleshttps://doi.org/10.1177/001316446002000104} With many reviewers, missing labels, or other data types, Krippendorff's alpha is often a more flexible reliability measure.^{[4]Reference 4Computing Krippendorff's Alpha-Reliabilityhttps://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf}

The next table has 120 duplicate-reviewed code-review cases:

The diagonal gives observed agreement:

$$p_o = (42 + 38 + 27) / 120 = 107 / 120 = 0.892$$

The row and column totals give chance agreement:

$$p_e = (46 \times 48 + 44 \times 42 + 30 \times 30) / 120^2 = 0.344$$

Now compute kappa:

$$\kappa = (p_o - p_e) / (1 - p_e) = (0.892 - 0.344) / (1 - 0.344) \approx 0.835$$

A number doesn't determine policy by itself. CodeAssist might set kappa >= 0.65 as a pilot acceptance threshold, alongside slice checks and adjudication of safety-related disagreements. That's a declared internal gate, not a universal standard. A failed batch should trigger diagnosis:

• Reviewers disagree about safe command eligibility: clarify the policy citation rule and add gold examples.
• Reviewers disagree about tone after matching on correctness: split style preference from policy correctness.
• Disagreements cluster in accessible-language or translated responses: involve the affected language or accessibility expertise before promotion.
• One reviewer's gold-item misses are unusual after rubric repair: retrain or suspend that reviewer's labels.

Select cases that might teach something new

After redaction and routing, CodeAssist still can't review every eligible trace. Active learning selects items for labeling based on a signal that they may be informative. Classic query strategies include uncertainty sampling and representativeness-aware selection.^{[5]Reference 5Active Learninghttps://doi.org/10.2200/S00429ED1V01Y201207AIM018}

For candidate preference pairs, suppose a small preference model estimates the probability that candidate A is better. Values near 0.5 mean the model is uncertain about the comparison.

Uncertainty isn't the same as importance. Five uncertain cases might all be paraphrases of the same missing-fixture diagnosis. A diverse batch avoids spending an entire review round on one local cluster.

The miniature selector below uses an already-normalized uncertainty score from 0 to 1, where 1 means most uncertain. That differs from the previous p_a values: a preference probability near 0.5 should map to high uncertainty. The selector starts with the most uncertain item, then scores remaining items using normalized uncertainty plus distance from anything already selected. The two-dimensional coordinates stand in for embeddings so the mechanics stay visible.

Core-set methods formalize the coverage intuition by selecting examples far from the already represented set in embedding space.^{[6]Reference 6Active Learning for Convolutional Neural Networks: A Core-Set Approachhttps://arxiv.org/abs/1708.00489} In a real pipeline, validate whether the representation and scoring method find meaningful code-review failure modes. Distance in an embedding space is a heuristic, not proof that an example will improve a model.

Prove selection helped without contaminating evaluation

An active selector can produce an interesting queue and still fail to improve the system. Measure it against a random-sampling baseline:

1. Freeze a private evaluation set before selecting training data.
2. Draw equal-sized candidate batches, one random and one selected by the proposed policy.
3. Label both with the same rubric and quality controls.
4. Train comparable candidates.
5. Evaluate both candidates on the same frozen set, including safety and accessibility slices.
6. Compare downstream gain per accepted label and total review cost.

The frozen set must be disjoint from demonstrations and preference pairs.

The result correctly blocks this draft dataset because attack-014 was accidentally placed in both the preference set and the frozen evaluation set.

Measure label efficiency, not label volume

The values below are example experiment results, not a promised advantage for active selection. An experiment log should make the comparison easy to calculate.

If the hybrid batch doesn't outperform random sampling on the frozen test set, don't defend it because its selected examples looked clever. Change the selector or return to the simpler baseline.

Treat model judges as assistants, not ground truth

An LLM judge can prioritize a large candidate pool or flag likely failures before human review. It can also prefer answers because they appear first, are longer, or resemble its own style. The MT-Bench and Chatbot Arena study measured those position, verbosity, and style-similarity biases in model judges.^{[7]Reference 7Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Start with a held-out set labeled by people under the actual rubric. Swap answer order and check whether a judge reverses its choice.

This judge agrees when shown one order and fails the order-swap test. It can help surface cases for review, but it can't accept preference pairs automatically.

Promote a versioned feedback dataset

Once a batch has passed review, package its provenance as carefully as the model release from the previous lesson. The dataset manifest should tie accepted rows to their controls:

• Source window and selection policy version.
• Redaction version and access rule for any re-identification mapping.
• Rubric version, reviewer training set, and agreement report.
• Counts for demonstrations, preferences, ties, both-bad escalations, and rejected rows.
• Frozen evaluation set ID and leakage-check result.
• Parent dataset and immutable output version.

Gate the full data release

A manifest can be complete and the batch still be unsuitable. Promotion should fail for leakage, unresolved sensitive exposure, low agreement under the declared policy, or unsafe pairs that were forced into chosen/rejected labels.

The repaired batch can become code-review-feedback-v12. It's now a defensible input to an SFT or DPO experiment, and its untouched code-review-eval-v5 suite can measure the candidate honestly.

Build the missing CodeAssist artifact

Create a small feedback dataset package from the code-review assistant traces in the previous lesson:

1. Reserve at least three traces for code-review-eval-v5: one prompt-injection attempt, one legitimate flaky-test handoff, and one supported accessible interaction path. These IDs must never enter training data.
2. Redact candidate review traces and store a redaction_version with each cleaned row.
3. Write code-review-rubric-v4 with authorization, technical correctness, actionability, and tone ordered explicitly.
4. Produce demonstrations for corrected answers and preference pairs only when at least one safe winner exists. Route BOTH_BAD to incident review.
5. Duplicate-review a small sample and calculate observed agreement and Cohen's kappa. Record your declared acceptance policy and what you'll inspect even when the gate passes.
6. Run a random batch and a hybrid-selected batch of equal size. Compare candidate results against the frozen evaluation set.
7. Write code-review-feedback-v12 manifest and run promotion checks for redaction, leakage, agreement, and unsafe accepted pairs.

A strong submission

Your package includes cleaned JSONL-like records, a rubric, a hand-checkable agreement calculation, a selector result, a leakage failure that you repaired, and a manifest that states exactly why the dataset may be used for training.

A useful failure

Place attack-014 into both the frozen set and a preference pair. Your gate should refuse promotion. If it doesn't, the pipeline can no longer tell training progress from memorization.

Mastery check

You're ready to build human-feedback datasets when you can:

• Explain why demonstrations, pointwise assessments, preference pairs, frozen evaluations, and incident traces have different jobs.
• Redact a trace before selection and keep the mapping separate from reviewer data.
• Write a rubric where authorization and privacy outrank tone.
• Compute Cohen's kappa from a small matrix and explain why agreement isn't correctness.
• Select uncertain and diverse cases without claiming the heuristic guarantees improvement.
• Compare an active-selection batch with a random baseline on an untouched evaluation set.
• Detect a judge that changes labels when answer order changes.
• Block dataset promotion for leakage or unsafe accepted pairs.

Evaluation rubric

• Foundational: Routes trace types correctly, redacts identifiers, and explains each feedback artifact.
• Intermediate: Produces safe pairwise records, agreement evidence, and a versioned dataset manifest.
• Applied: Implements hybrid selection and measures it against random sampling without evaluation leakage.
• Advanced: Integrates privacy, disagreement repair, judge calibration, and promotion gates into a reproducible feedback program.

Common pitfalls

• Symptom: The next model scores unusually well only on reviewed incidents. Cause: Holdout traces leaked into training data. Fix: Reserve evaluation IDs first and block overlap during dataset promotion.
• Symptom: Reviewers choose a fluent answer that runs an unauthorized production command. Cause: Tone was valued before policy and authorization. Fix: Put safety first in the rubric and route both-bad pairs to incident review.
• Symptom: Kappa is low across multiple reviewers. Cause: Guidelines are unclear, task slices differ, or reviewer training is weak. Fix: Inspect disagreement clusters before blaming reviewers.
• Symptom: An active queue costs more but doesn't improve the frozen suite. Cause: Uncertainty or embedding distance isn't correlated with downstream value. Fix: Compare against random selection and replace a failed heuristic.
• Symptom: A model judge shrinks human review while changing choices under response swaps. Cause: Judge bias wasn't calibrated. Fix: Keep it advisory until human-gold and order-swap tests pass.