Data Labeling and Human Feedback

CodeAssist's code-review assistant can't pass the governance gate from the last lesson without a trustworthy dataset record. Its team has attack traces, CI failure investigations, and ordinary code-review conversations, but raw logs aren't training data. A few contain private details. Others should remain frozen tests forever. In some rows, both answers are bad, so choosing a "winner" would teach the model the wrong lesson.

Build the missing data pipeline: turn reviewed traces into versioned feedback data while preserving a private evaluation set that can honestly measure whether the assistant improves.

The same boundary should be explicit in the system design. Redaction and routing happen before selection or review, while the frozen lane reaches the release gate without becoming training data.

The training lane can change between runs. The evaluation lane and its IDs must stay independent so a score still measures generalization.

Why can't CodeAssist feed every production trace into training?

Answer

Logs may contain private data, unsafe or ambiguous examples, and cases needed for independent evaluation. A feedback pipeline must redact, route, review, version, and hold out evidence before any training step.

Separate records by the job they do

Human feedback isn't one kind of label. A model team commonly needs several different artifacts:

Pointwise assessments need anchored criteria so reviewers interpret a category or score consistently. Preference pairs answer a different question: given the same prompt, which answer is better? InstructGPT used human-written demonstrations and ranked model outputs in its post-training pipeline.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} Direct Preference Optimization (DPO) uses preference pairs to optimize a policy without first fitting a separate reward model.^{[2]Reference 2Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Those methods don't mean every reviewed log belongs in training. Holdout examples and safety incidents have different jobs.

The first routing rule is strict: a case reserved for evaluation doesn't enter the training queue. It may also be an incident that needs investigation.

Routing seems administrative until leakage happens. If attack-014 is trained on, the next evaluation no longer answers "does the system generalize to this attack?" It answers "did it remember a case we revealed?"

Redact before selection or review

The governance chapter required minimized evidence. The same rule applies earlier in the feedback pipeline: production text must be cleaned before it reaches a selection service, external reviewer, or annotation interface.

For a code-review trace, preserve what changes the judgment:

• Repository, diff hunk, failing test, and cited policy rule.
• Whether a command, deploy, or secret access required approval.
• The assistant's proposed action and final gate decision.
• A stable pseudonymous case ID for joining records later.

Remove what the reviewer doesn't need:

• Engineer name, email, phone number, or full commit identifier.
• Free-form incident notes unrelated to the label.
• Secrets, tokens, or internal credentials copied into retrieved context.

This small example uses stable replacements so two occurrences of the same identifier still match during review.

The reversible mapping, if it must exist, belongs in a separate access-controlled store. The annotation record needs the redaction version and the cleaned text, not the engineer's identity.

Why run redaction before active selection rather than immediately before displaying an item to a rater?

Answer

Selection systems, embeddings, queues, and debug logs can also retain sensitive content. Redacting at ingestion reduces the number of systems exposed to unnecessary identifiers.

Write the rubric before collecting preferences

A preference interface is only as good as the decision rule behind it. "Choose the better answer" leaves reviewers to invent their own priorities. For CodeAssist's code-review assistant, write a rubric in descending order:

1. Safety and authorization: Reject any answer that reveals protected notes, exports secrets, or runs a privileged command outside the approval rule.
2. Technical correctness: Prefer an answer that accurately cites the failing test, diff hunk, or repo policy.
3. Actionability: Prefer a response that states the next safe command, evidence link, or reviewer handoff.
4. Clarity and tone: Among equally safe and correct answers, prefer direct and respectful language.

Safety outranks style. If both candidates break the first rule, record BOTH_BAD and send the case to incident review. Don't create a chosen/rejected training pair from two unsafe answers.

Store enough provenance to replay the judgment

Each accepted preference record should identify:

Collect only reviewer attributes needed for a legitimate analysis, protect them with access controls, and define their retention. Bias analysis isn't an excuse to collect personal information without a purpose.

Measure agreement, then repair disagreement

Two trained reviewers can still read a rule differently. Inter-annotator agreement (IAA) measures whether labels are consistent enough for their intended use.

For two reviewers choosing A, B, or Tie, Cohen's kappa compares observed agreement with agreement expected from each reviewer's label frequency.^{[3]Reference 3A Coefficient of Agreement for Nominal Scaleshttps://doi.org/10.1177/001316446002000104} With many reviewers, missing labels, or other data types, Krippendorff's alpha is often a more flexible reliability measure.^{[4]Reference 4Computing Krippendorff's Alpha-Reliabilityhttps://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf}

The next table has 120 duplicate-reviewed code-review cases:

The diagonal gives observed agreement:

$$p_o = (42 + 38 + 27) / 120 = 107 / 120 = 0.892$$

The row and column totals give chance agreement:

$$p_e = (46 \times 48 + 44 \times 42 + 30 \times 30) / 120^2 = 0.344$$

Now compute kappa:

$$\kappa = (p_o - p_e) / (1 - p_e) = (0.892 - 0.344) / (1 - 0.344) \approx 0.835$$

A number doesn't determine policy by itself. CodeAssist might set kappa >= 0.65 as a pilot acceptance threshold, alongside slice checks and adjudication of safety-related disagreements. That's a declared internal gate, not a universal standard. A failed batch should trigger diagnosis:

• Reviewers disagree about safe command eligibility: clarify the policy citation rule and add gold examples.
• Reviewers disagree about tone after matching on correctness: split style preference from policy correctness.
• Disagreements cluster in accessible-language or translated responses: involve the affected language or accessibility expertise before promotion.
• One reviewer's gold-item misses are unusual after rubric repair: retrain or suspend that reviewer's labels.

Does a high kappa prove the labels are correct or fair?

Answer

No. It shows reviewers agree beyond chance under this rubric. They may agree on a flawed rule or miss a stakeholder's needs, so accuracy, slice review, safety escalation, and affected-user access still matter.

Select cases that might teach something new

After redaction and routing, CodeAssist still can't review every eligible trace. Active learning selects items for labeling based on a signal that they may be informative. Classic query strategies include uncertainty sampling and representativeness-aware selection.^{[5]Reference 5Active Learninghttps://doi.org/10.2200/S00429ED1V01Y201207AIM018}

For candidate preference pairs, suppose a small preference model estimates the probability that candidate A is better. Values near 0.5 mean the model is uncertain about the comparison.

Uncertainty isn't the same as importance. Five uncertain cases might all be paraphrases of the same missing-fixture diagnosis. A diverse batch avoids spending an entire review round on one local cluster.

The miniature selector below uses an already-normalized uncertainty score from 0 to 1, where 1 means most uncertain. That differs from the previous p_a values: a preference probability near 0.5 should map to high uncertainty. The selector starts with the most uncertain item, then scores remaining items using normalized uncertainty plus distance from anything already selected. The two-dimensional coordinates stand in for embeddings so the mechanics stay visible.

Core-set methods formalize the coverage intuition by selecting examples far from the already represented set in embedding space.^{[6]Reference 6Active Learning for Convolutional Neural Networks: A Core-Set Approachhttps://arxiv.org/abs/1708.00489} In a real pipeline, validate whether the representation and scoring method find meaningful code-review failure modes. Distance in an embedding space is a heuristic, not proof that an example will improve a model.

Prove selection helped without contaminating evaluation

An active selector can produce an interesting queue and still fail to improve the system. Measure it against a random-sampling baseline:

1. Freeze a private evaluation set before selecting training data.
2. Draw equal-sized candidate batches, one random and one selected by the proposed policy.
3. Label both with the same rubric and quality controls.
4. Train comparable candidates.
5. Evaluate both candidates on the same frozen set, including safety and accessibility slices.
6. Compare downstream gain per accepted label and total review cost.

The frozen set must be disjoint from demonstrations and preference pairs.

The result correctly blocks this draft dataset because attack-014 was accidentally placed in both the preference set and the frozen evaluation set.

ID overlap is necessary but not sufficient. Training on paraphrases of frozen attack prompts, template twins, or shared retrieved chunks still inflates "generalization." Hold out by template family when you can, and reject near-duplicates against frozen prompts with an n-gram or embedding check before promotion.

This output deliberately shows a detector miss: the three-gram check returns no hit for the paraphrase, so it can't authorize promotion. The independent template-family control blocks the same case. Assign template families from the review schema before selection, not from a model's generated text. A larger system can add a calibrated embedding-neighbor check against the frozen prompt bank, but no one lexical or semantic signal should be enough to permit training on a possible holdout twin.

Measure label efficiency, not label volume

The values below are example experiment results, not a promised advantage for active selection. An experiment log should make the comparison easy to calculate.

If the hybrid batch doesn't outperform random sampling on the frozen test set, don't defend it because its selected examples looked clever. Change the selector or return to the simpler baseline.

Treat model judges as assistants, not ground truth

An LLM judge can prioritize a large candidate pool or flag likely failures before human review. It can also prefer answers because they appear first, are longer, or resemble its own style. The MT-Bench and Chatbot Arena study measured those position, verbosity, and style-similarity biases in model judges.^{[7]Reference 7Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Start with a held-out set labeled by people under the actual rubric. Swap answer order and check whether a judge reverses its choice.

This judge agrees when shown one order and fails the order-swap test. It can help surface cases for review, but it can't accept preference pairs automatically.

Promote a versioned feedback dataset

Once a batch has passed review, package its provenance as carefully as the model release from the previous lesson. The dataset manifest should tie accepted rows to their controls:

• Source window and selection policy version.
• Redaction version and access rule for any re-identification mapping.
• Rubric version, reviewer training set, and agreement report.
• Counts for demonstrations, preferences, ties, both-bad escalations, and rejected rows.
• Frozen evaluation set ID and leakage-check result.
• Parent dataset and immutable output version.

Gate the full data release

A manifest can be complete and the batch still be unsuitable. Promotion should fail for leakage, unresolved sensitive exposure, low agreement under the declared policy, or unsafe pairs that were forced into chosen/rejected labels.

The repaired batch can become code-review-feedback-v12. It's now a defensible input to an SFT or DPO experiment, and its untouched code-review-eval-v5 suite can measure the candidate honestly.