Experiment Design and A/B Testing

A support assistant can produce a fluent refund answer and still fail customers. Perhaps the answer arrives too slowly. Perhaps it cites the wrong policy. Perhaps customers reopen a ticket five minutes later. A model improvement matters only when a careful measurement says it helps.

In Decoding Algorithms, you logged the retrieval evidence and decoder settings that produced an answer. Now you will hold that generator fixed and test one proposed change: rewriting a customer's query before retrieval. An A/B test, also called a randomized controlled experiment, gives randomly assigned customers either the existing system (control) or the new system (treatment) and compares their outcomes.

The goal isn't a green dashboard tile. It's a defensible decision: ship the rewrite, keep investigating, or roll it back.

Test One Change At A Time

Suppose a shopper asks:

I bought an annual plan 12 days ago. Can I get a refund?

Your existing pipeline retrieves policy passages from the raw question and generates a reply. A candidate treatment first rewrites the request into a retrieval-focused query such as annual plan refund within 30 days, then sends the retrieved evidence through the same prompt, model, and decoder settings.

Write a hypothesis that can fail:

Query rewriting will increase resolved refund conversations while keeping grounded-answer audit failures, p95 latency, and human escalation inside their budgets.

Here, p95 latency is the response time that 95 percent of enrolled conversations finish at or below. It protects customers in the slower tail, not only the average user.

An experiment brief records the decision before results tempt you to change it:

For this compact teaching gate, guardrails use observed deltas. A production brief should also predeclare how uncertainty is handled for each guardrail; a point estimate barely inside a budget isn't strong evidence that the actual regression is acceptable.

Random assignment supports causal conclusions only when assignment, instrumentation, and stopping rules stay trustworthy. Practical online experiment systems make this pre-launch contract explicit.^[1]

Assign A Customer Once

Why assign by customer account rather than message? One shopper might ask a follow-up question after seeing the first answer. If the first message uses treatment and the follow-up uses control, the experiences interfere: the second outcome depends partly on what happened in the first.

A stable hash gives each enrolled customer the same arm on every request. The salt includes an experiment name and version so a later experiment can assign independently.

If the same customer appears twice in the output, the assigned arm must match twice. In a service, persist the experiment version and arm on each event as well. Hashing is deterministic; logging makes it auditable.

Define An Outcome From Events

The assistant shouldn't get credit merely because it produced an answer. Resolution requires an observed customer outcome. For this chapter, one conversation counts as resolved only if the customer marks it solved and doesn't open a support ticket within ten minutes.

This next lab turns raw product events into that metric. It also records two guardrails: whether an audit says the answer is grounded in retrieved policy and the response latency in milliseconds.

The small rows teach the definition, not the launch result. Notice that the treatment row with ticket_after_min=18 still counts as resolved because its ticket arrives outside the locked ten-minute window. That isn't automatically correct or wrong: it makes the metric contract inspectable. If later tickets matter, choose a longer window or track them separately before launch. A real analysis also needs enough customers to distinguish an actual lift from chance variation.

Read Lift From Counts

After the planned window, suppose each arm contains 2,000 enrolled customers.

The absolute lift is the treatment rate minus the control rate:

$$\widehat{\Delta} = \hat{p}_{treatment} - \hat{p}_{control} = \frac{840}{2000} - \frac{760}{2000} = 0.04$$

The treatment resolves 4.0 additional conversations per 100 enrolled customers. That is a 4.0 percentage point lift.

The relative lift compares the absolute change with the original baseline:

$$\frac{0.04}{0.38} \approx 0.105$$

That is a 10.5 percent relative lift. Always name which one you report; saying "up 10.5 points" would be wrong.

Put Uncertainty Around The Lift

If you reran the experiment with different customers, the counts wouldn't be identical. A confidence interval is a procedure for describing this sampling uncertainty. For two reasonably large binary-outcome arms, a normal-approximation interval is a useful first calculation.

For each arm, $\hat{p}$ is its observed resolution rate and $n$ is its number of enrolled customers. The estimated standard error of the difference is:

$$SE(\widehat{\Delta}) = \sqrt{ \frac{\hat{p}_{control}(1 - \hat{p}_{control})}{n_{control}} + \frac{\hat{p}_{treatment}(1 - \hat{p}_{treatment})}{n_{treatment}} }$$

A rough 95 percent interval is $\widehat{\Delta} \pm 1.96 \times SE$. The number 1.96 is the standard-normal cutoff that leaves about 2.5 percent in each tail.

The estimate is +4.0 points and this approximation gives an interval of about +1.0 to +7.0 points. Because the lower bound is above zero, this planned analysis gives evidence for a positive treatment effect, assuming assignment and instrumentation are sound. It doesn't prove every future rollout will gain four points.

For rare outcomes, heavily clustered customers, many simultaneous comparisons, or business-critical launches, choose the inference method with a statistician or a mature experimentation platform before running the test.

Plan For A Meaningful Win

A test with too little traffic may return "uncertain" even when the treatment helps. Before launching, declare a minimum detectable effect (MDE): the smallest true lift the planned test is designed to detect with its target power. Our team chooses +2.0 percentage points because a smaller resolution improvement wouldn't justify maintaining the rewrite service. This brief also uses +2.0 points as its minimum observed lift worth shipping. Keep the two fields conceptually separate: MDE determines planned enrollment, while the launch threshold determines the decision rule.

See what sample size does while the observed rates stay at 38 and 42 percent:

With 200 or 500 customers per arm, a good-looking +4.0 point estimate is still compatible with no improvement. More enrollment narrows uncertainty; it doesn't magically make treatment better.

Power is the probability that a planned test will detect an effect of the chosen size when that effect truly exists. For equal-sized arms and a binary rate near the baseline, this approximation provides a planning estimate:

$$n \approx \frac{2p(1-p)(z_{1-\alpha/2} + z_{power})^2}{\delta^2}$$

Here, $p$ is baseline resolution rate, $\delta$ is the MDE, $\alpha$ is the false-positive budget, and power is the desired detection probability.

This formula is for planning, not post-result storytelling. Actual platform planning may account for unequal allocation, repeated customers, variance reduction, multiple metrics, or sequential analysis.

Check Traffic Before Outcomes

A sample ratio mismatch (SRM) means observed assignment counts disagree suspiciously with the intended split. If the design says 50/50 but your data has 2,350 treatment customers and 1,650 control customers, investigate before reading resolution lift. Assignment code, event logging, eligibility filters, or bot removal may differ by arm.

This lab computes a chi-square diagnostic for a planned 50/50 allocation. A one-degree-of-freedom value above 10.83 is a deliberately strict alert threshold corresponding to a very small tail probability, about 0.1 percent.

An SRM alert doesn't identify the bug. It says your comparison hasn't earned trust yet. Stop, diagnose the event path, and restart or reanalyze only when you understand what changed.

Don't Stop The First Time Noise Looks Good

Suppose treatment truly has no effect: both arms resolve 40 percent of conversations. If you run one planned two-sided check at the end with a 5 percent false-positive threshold, false wins should occur about 5 percent of the time. If you inspect the ordinary interval after every batch and stop at the first apparent win, noise receives many chances to cross the threshold.

The following A/A simulation uses two identical arms, 5,000 repeat experiments, 20 looks, and a fixed seed. It demonstrates this exact stopping policy, not a universal percentage for every test design.

Ordinary fixed-horizon intervals aren't valid for a decision triggered by continuous monitoring. Johari and colleagues define always-valid p-values and confidence intervals for sequential decisions, where repeated observation is part of the planned method.^[2] For a first experiment, the straightforward rule is enough: choose enrollment in advance and analyze once.

Reduce Noise With Pre-Experiment Data

Power doesn't always require more customers. CUPED (Controlled-experiment Using Pre-Experiment Data) uses a measurement collected before assignment, such as whether a customer resolved a prior refund issue, to remove predictable customer-to-customer variation from the outcome.

Let $Y$ be the experiment outcome and $X$ be a pre-experiment covariate. CUPED forms:

$$Y_{adjusted} = Y - \theta(X - \bar{X}), \qquad \theta = \frac{\operatorname{cov}(Y, X)}{\operatorname{var}(X)}$$

Since $X$ was measured before treatment, it can't have been caused by treatment. When $X$ predicts $Y$, the adjusted outcome varies less while targeting the same treatment difference. Deng, Xu, Kohavi, and Walker introduced this approach for online experiments and reported substantial variance reductions in Bing experiments.^[3]

This simulation gives returning customers different baseline resolution probabilities, then assigns treatment independently. Compare the standard error before and after CUPED.

In a finite sample, the raw and adjusted estimates won't be numerically identical. What matters is that adjustment uses only a pre-treatment signal and reduces uncertainty. Never adjust for a quantity treatment could affect, such as latency measured after the rewrite is enabled; that can bias the comparison.

Treat AI Evaluation As A Stack

An online experiment answers whether customers benefit under live use. It shouldn't be the first time you discover that a treatment emits unsupported refund claims. For an AI feature, use layers:

If you use a large language model (LLM) as a judge for an offline rubric, treat it as a measured evaluator rather than truth: pin its model and prompt version and compare it with human labels on a retained calibration set. Zheng et al. study LLM judges as approximations to human preference and document position, verbosity, self-enhancement, and reasoning limitations.^[4]

This lab makes the configuration receipt explicit. A comparison with two different decoder versions is rejected before anybody reads its lift.

Make The Launch Decision Auditable

Now combine the evidence. This brief requires a positive interval, an observed lift of at least +2.0 points, guardrails inside their budgets, and trustworthy traffic. This last lab is a small launch gate that prints each condition instead of burying the call in a slide deck.

This gate separates evidence from business value. positive_lift_interval asks whether the interval's lower bound is above zero. observed_lift_threshold asks whether the point estimate clears the predeclared +2.0 point bar. It doesn't prove the true lift exceeds +2.0 points: the interval begins at +1.0. A more conservative brief could require the lower bound to clear +2.0 points and would keep this result under investigation.

For these planned numbers and this predeclared rule, the launch gate says SHIP. Change treatment latency to 1490 and rerun: resolution still improves, but the speed budget fails. A treatment doesn't earn launch by winning only its favorite metric.

Practice: Break The Launch Gate

1. Change treatment latency from 1260 to 1490. Which check fails, and why doesn't the resolution lift override it?
2. Restore latency, then change minimum_ship_lift from 0.02 to 0.05. Which check fails even though the interval still excludes zero?
3. Restore the threshold, then set srm_alert to True. Why should analysis pause before anybody argues about the observed lift?

Expected observations

1. latency_budget fails because treatment is now 310 ms slower than control, beyond the locked 150 ms budget.
2. observed_lift_threshold fails because the observed +4.0 point lift is below the new +5.0 point business threshold. Statistical evidence and shipping value answer different questions.
3. traffic_integrity fails. SRM can signal broken assignment, filtering, or logging, so the comparison hasn't earned a launch decision.

Common Pitfalls

Explain The Decision Without Looking Back

Without looking back, explain why this experiment changes only the query rewrite while holding index, generator, prompt, and decoder fixed. Then answer these checks.

Evaluation rubric

• Foundational: Defines control, treatment, enrolled customer, primary outcome, and guardrails before viewing results.
• Intermediate: Computes lift and an uncertainty interval, then identifies SRM and optional stopping as reasons to withhold a launch call.
• Advanced: Produces an auditable AI launch report with pinned system receipts, justified MDE, grounding checks, latency budgets, and a valid analysis plan.

Reuse This Experiment Pattern

• A stable customer assignment function with a versioned experiment key.
• A metric definition based on observable resolution events rather than answer fluency.
• A planned analysis that reports absolute lift, uncertainty, sample size assumptions, and SRM status.
• A launch report that keeps groundedness and latency beside the primary outcome.

Key Terms

• Control: Existing system used as comparison baseline.
• Treatment: Candidate system containing one intended change.
• Randomization unit: Entity kept in one arm, such as a customer account.
• Guardrail: Metric that must remain acceptable even when primary metric improves.
• MDE: Minimum detectable effect worth designing the experiment to detect.
• SRM: Sample ratio mismatch, an assignment-integrity alarm.
• CUPED: Pre-experiment covariate adjustment that can reduce outcome variance.