Experiment Design and A/B Testing

A support assistant can produce a fluent access-policy answer and still fail users. The answer might arrive too slowly, cite the wrong policy, or lead users to reopen a ticket five minutes later. A model improvement matters only when a careful measurement says it helps.

In Decoding Algorithms, you logged the retrieval evidence and decoder settings that produced an answer. Hold that generator fixed and test one proposed change: rewriting a customer's query before retrieval. An A/B test, also called a randomized controlled experiment, gives randomly assigned customers either the existing system (control) or the new system (treatment) and compares their outcomes.

Use the result to make a defensible decision: ship the rewrite, keep investigating, or roll it back.

Test one change at a time

Suppose an engineer asks:

My service-account key has been stale for 12 days. Can it rotate automatically?

Your existing pipeline retrieves policy passages from the raw question and generates a reply. A candidate treatment first rewrites the request into a retrieval-focused query such as stale service-account key rotation within 14 days, then sends the retrieved evidence through the same prompt, model, and decoder settings.

Write a hypothesis that can fail:

Query rewriting will increase resolved key-rotation conversations while keeping grounded-answer audit failures, p95 latency, and human escalation inside their budgets.

Here, p95 latency is the response time that 95 percent of enrolled conversations finish at or below. It protects customers in the slower tail, not average users alone.

An experiment brief records the decision before results tempt you to change it:

For this compact teaching gate, guardrails use observed deltas. A production brief should also predeclare how uncertainty is handled for each guardrail; a point estimate barely inside a budget isn't strong evidence that the actual regression is acceptable.

Random assignment supports causal conclusions only when assignment, instrumentation, and stopping rules stay trustworthy. Practical online experiment systems make this pre-launch contract explicit.^{[1]Reference 1Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}

Assign a customer once

Why assign by customer account rather than message? One user might ask a follow-up question after seeing the first answer. If the first message uses treatment and the follow-up uses control, the experiences interfere: the second outcome depends partly on what happened in the first.

A stable hash gives each enrolled customer the same arm on every request. The salt includes an experiment name and version so a later experiment can assign independently.

If the same customer appears twice in the output, the assigned arm must match twice. In a service, persist the experiment version and arm on each event as well. Hashing is deterministic; logging makes it auditable.

Define an outcome from events

The assistant shouldn't get credit merely because it produced an answer. Resolution requires an observed customer outcome. Here, one conversation counts as resolved only if the customer marks it solved and doesn't open a human escalation within ten minutes.

This next lab turns raw product events into that metric. It also records two guardrails: whether an audit says the answer is grounded in retrieved policy and the response latency in milliseconds.

The small rows teach the definition, not the launch result. Notice that the treatment row with escalation_after_min=18 still counts as resolved because its escalation arrives outside the locked ten-minute window. That isn't automatically correct or wrong: it makes the metric contract inspectable. If later escalations matter, choose a longer window or track them separately before launch. A real analysis also needs enough customers to distinguish an actual lift from chance variation.

Read lift from counts

After the planned window, the worked analysis uses a compact teaching fixture of 2,000 customers per arm so the arithmetic stays easy to check by hand. That fixture is not powered for the locked MDE of +2.0 percentage points. The planning lab later shows you need about 9,246 customers per arm at 80% power for that MDE. Keep the roles separate: use 2,000-row tables to practice lift, intervals, and launch gates; use the powered enrollment estimate when you design a real test for the brief.

The absolute lift is the treatment rate minus the control rate:

$$\widehat{\Delta} = \hat{p}_{treatment} - \hat{p}_{control} = \frac{840}{2000} - \frac{760}{2000} = 0.04$$

The treatment resolves 4.0 additional conversations per 100 enrolled customers. That's a 4.0 percentage point lift.

The relative lift compares the absolute change with the original baseline:

$$\frac{0.04}{0.38} \approx 0.105$$

That's a 10.5 percent relative lift. Always name which one you report; saying "up 10.5 points" would be wrong.

Put uncertainty around the lift

If you reran the experiment with different customers, the counts wouldn't be identical. A confidence interval is a procedure for describing this sampling uncertainty. For two reasonably large binary-outcome arms, a normal-approximation interval is a useful first calculation.

For each arm, $\hat{p}$ is its observed resolution rate and $n$ is its number of enrolled customers. The estimated standard error of the difference is:

$$SE(\widehat{\Delta}) = \sqrt{ \frac{\hat{p}_{control}(1 - \hat{p}_{control})}{n_{control}} + \frac{\hat{p}_{treatment}(1 - \hat{p}_{treatment})}{n_{treatment}} }$$

A rough 95 percent interval is $\widehat{\Delta} \pm 1.96 \times SE$. The number 1.96 is the standard-normal cutoff that leaves about 2.5 percent in each tail.

The estimate is +4.0 points and this approximation gives an interval of about +1.0 to +7.0 points. Because the lower bound is above zero, this planned analysis gives evidence for a positive treatment effect, assuming assignment and instrumentation are sound. It doesn't prove every future rollout will gain four points.

For rare outcomes, heavily clustered customers, many simultaneous comparisons, or business-critical launches, choose the inference method with a statistician or a mature experimentation platform before running the test.

Plan for a meaningful win

A test with too little traffic may return "uncertain" even when the treatment helps. Before launching, declare a minimum detectable effect (MDE): the smallest true lift the planned test can detect with its target power. Our team chooses +2.0 percentage points because a smaller resolution improvement wouldn't justify maintaining the rewrite service. This brief also uses +2.0 points as its minimum observed lift worth shipping. Keep the two fields conceptually separate: MDE determines planned enrollment, while the launch threshold determines the decision rule.

See what sample size does while the observed rates stay at 38 and 42 percent:

With 200 or 500 customers per arm, a good-looking +4.0 point estimate is still compatible with no improvement. At 2,000 per arm, this observed +4.0 pp effect's interval finally excludes zero in the teaching fixture, but that is not the same as being powered to detect the locked design MDE of +2.0 pp (about 9,246 per arm below). More enrollment narrows uncertainty; it doesn't make treatment better by itself.

Power is the probability that a planned test will detect an effect of the chosen size when that effect truly exists. For equal-sized arms and a binary rate near the baseline, this approximation provides a planning estimate:

$$n \approx \frac{2p(1-p)(z_{1-\alpha/2} + z_{power})^2}{\delta^2}$$

Here, $p$ is baseline resolution rate, $\delta$ is the MDE, $\alpha$ is the false-positive budget, and power is the desired detection probability.

This formula is for planning, not post-result storytelling. Actual platform planning may account for unequal allocation, repeated customers, variance reduction, multiple metrics, or sequential analysis.

Check traffic before outcomes

A sample ratio mismatch (SRM) means observed assignment counts disagree suspiciously with the intended split. If the design says 50/50 but your data has 2,350 treatment customers and 1,650 control customers, investigate before reading resolution lift. Assignment code, event logging, eligibility filters, or bot removal may differ by arm.

This lab computes a chi-square diagnostic for a planned 50/50 allocation. A one-degree-of-freedom value above 10.83 is a deliberately strict alert threshold corresponding to a very small tail probability, about 0.1 percent.

An SRM alert doesn't identify the bug. It says your comparison hasn't earned trust yet. Stop, diagnose the event path, and restart or reanalyze only when you understand what changed.

Don't stop the first time noise looks good

Suppose treatment truly has no effect: both arms resolve 40 percent of conversations. If you run one planned two-sided check at the end with a 5 percent false-positive threshold, false wins should occur about 5 percent of the time. If you inspect the ordinary interval after every batch and stop at the first apparent win, noise receives many chances to cross the threshold.

This A/A simulation uses two identical arms, 5,000 repeat experiments, 20 looks, and a fixed seed. It tests this exact stopping policy, not a universal percentage for every experiment design.

Ordinary fixed-horizon intervals aren't valid for a decision triggered by continuous monitoring. Johari and colleagues define always-valid p-values and confidence intervals for sequential decisions, where repeated observation is part of the planned method.^{[2]Reference 2Peeking at A/B Tests: Why it matters, and what to do about it.https://doi.org/10.1145/3097983.3097992} For a first experiment, the straightforward rule is enough: choose enrollment in advance and analyze once.

Reduce noise with pre-experiment data

Power doesn't always require more users. CUPED (Controlled-experiment Using Pre-Experiment Data) uses a measurement collected before assignment, such as whether a user resolved a prior access-policy issue, to remove predictable user-to-user variation from the outcome.

Let $Y$ be the experiment outcome and $X$ be a pre-experiment covariate. CUPED forms:

$$Y_{adjusted} = Y - \theta(X - \bar{X}), \qquad \theta = \frac{\operatorname{cov}(Y, X)}{\operatorname{var}(X)}$$

Since $X$ was measured before treatment, it can't have been caused by treatment. When $X$ predicts $Y$, the adjusted outcome varies less while targeting the same treatment difference. Deng, Xu, Kohavi, and Walker introduced this approach for online experiments and reported substantial variance reductions in Bing experiments.^{[3]Reference 3Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED)https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf}

This simulation gives returning users different baseline resolution probabilities, then assigns treatment independently. Compare the standard error before and after CUPED.

In a finite sample, the raw and adjusted estimates won't be numerically identical. The adjustment must use only a pre-treatment signal and reduce uncertainty. Never adjust for a quantity treatment could affect, such as latency measured after the rewrite is enabled; that can bias the comparison.

Treat AI evaluation as a stack

An online experiment answers whether users benefit under live use. It shouldn't be the first time you discover that a treatment emits unsupported access-policy claims. For an AI feature, use layers:

If you use a large language model (LLM) as a judge for an offline rubric, treat it as a measured evaluator rather than truth: pin its model and prompt version and compare it with human labels on a retained calibration set. Zheng et al. study LLM judges as approximations to human preference and document position bias, verbosity bias, a bias toward answers produced by the judge itself, and reasoning limitations.^{[4]Reference 4Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

This lab makes the configuration receipt explicit. A comparison with two different decoder versions is rejected before anybody reads its lift.

Make the launch decision auditable

Now combine the evidence. This brief requires a positive interval, an observed lift of at least +2.0 points, guardrails inside their budgets, and trustworthy traffic. This last lab is a small launch gate that prints each condition instead of burying the call in a slide deck. Like the earlier lift tables, it reuses the compact 2,000-customer teaching fixture so you can check every line by hand. For the locked MDE of +2.0 points, a powered enrollment plan still targets about 9,246 customers per arm; rerun the same gate at that scale when you design a real launch.

This gate separates evidence from business value. positive_lift_interval asks whether the interval's lower bound is above zero. observed_lift_threshold asks whether the point estimate clears the predeclared +2.0 point bar. It doesn't prove the true lift exceeds +2.0 points: the interval begins at +1.0. A more conservative brief could require the lower bound to clear +2.0 points and would keep this result under investigation.

Guardrails can look safe on a point estimate while their uncertainty still crosses the budget. A stronger brief predeclares an uncertainty rule for each guardrail. This gate uses non-inferiority for grounding: both the point estimate and the upper end of a rough interval must stay under budget. For grounding failures as a rate, the unpooled SE is:

$$SE(\widehat{\Delta}_g) = \sqrt{ \frac{\hat{g}_{c}(1 - \hat{g}_{c})}{n_{c}} + \frac{\hat{g}_{t}(1 - \hat{g}_{t})}{n_{t}} }$$

With the teaching fixture, grounding rates are 1.10% for control and 1.15% for treatment, so $\widehat{\Delta}_g = +0.05$ pp. The unpooled SE is about 0.333 pp, which puts the rough 95% interval at about $[-0.60, +0.70]$ pp. Its upper end exceeds the +0.5 pp budget, so grounding_noninferiority fails even though the observed change is small.

For these teaching numbers, the lift and point-estimate checks pass, but the uncertainty-aware grounding rule keeps the decision at ROLL BACK OR INVESTIGATE. Change treatment latency to 1490 and rerun: resolution still improves, but the speed budget also fails. A treatment doesn't earn launch by winning only its favorite metric.

Practice: break the launch gate

1. Change treatment latency from 1260 to 1490. Which check fails, and why doesn't the resolution lift override it?
2. Restore latency, then change minimum_ship_lift from 0.02 to 0.05. Which check fails even though the interval still excludes zero?
3. Restore the threshold, then set srm_alert to True. Why should analysis pause before anybody argues about the observed lift?

Expected observations

1. latency_budget fails because treatment is now 310 ms slower than control, beyond the locked 150 ms budget.
2. observed_lift_threshold fails because the observed +4.0 point lift is below the new +5.0 point business threshold. Statistical evidence and launch value answer different questions.
3. traffic_integrity fails. SRM can signal broken assignment, filtering, or logging, so the comparison hasn't earned a launch decision.

Freeze the experiment receipt

Save the assignment unit and hash rule, metric query and outcome window, planned analysis schedule, sample-ratio-mismatch check, effect estimate with uncertainty, guardrails, and final decision. A reviewer should be able to rerun the gate from those artifacts without reconstructing policy from a dashboard.