508 study terms

Glossary

Browse AI, ML, and LLM terms with intuition, examples, decision rules, common mistakes, and related terms.

A/B testing

Core

Also known as A/B test, online controlled experiment, split test.

Definition

A/B testing (also called an online controlled experiment or split test) is a randomized comparison of two product or model variants under a predeclared assignment policy and primary metric. Eligible users, tenants, conversations, or requests are assigned to control (current system) or treatment (candidate change). You estimate whether treatment moves the metric more than sampling noise would allow, while guardrail metrics keep latency, cost, safety, and reliability from quietly regressing.

Randomization is what makes the comparison causal rather than anecdotal. When assignment is fair and logging is complete, differences in outcomes between arms come from the treatment (plus noise), not from who self-selected into a cohort. In AI products the hard parts are sticky assignment so multi-turn memory does not leak across arms, choosing a primary metric that matches the product decision, watching guardrails (latency, cost, safety, abstention), and separating "better answers" from "users stick around longer because the system is slower but nicer."

Example

You ship a new cross-encoder reranker. For two weeks, 10% of eligible support chats get sticky assignment to treatment (hash of conversation id + experiment salt). Primary metric: human acceptance of the drafted reply within 5 minutes. Guardrails: p95 latency under 2.5s, citation-present rate, and unauthorized-access answers at 0.

After $n = 12{,}400$ eligible chats ( $n_T \approx 1{,}240$ , $n_C \approx 11{,}160$ ), acceptance is $0.41$ control vs $0.46$ treatment, so $\hat{\delta} = 0.05$ . The Bernoulli SE above is about $0.015$ , so a 95% interval is roughly $0.05 \pm 0.029$ (about $+2$ to $+8$ percentage points). That interval excludes zero, but a predeclared MDE of $+3$ pp at 80% power would have needed far more treatment traffic at a 10% split (equal-allocation planning is on the order of a few thousand per arm; a 10% canary multiplies total volume). Expand only if the analysis set matches the power plan, every guardrail is green, and SRM is clean.

Before celebrating, check sample ratio mismatch against the planned 90/10 split, and check that treatment did not change who became "eligible" (for example by failing earlier and dropping out of logging). If treatment fell back to the old reranker on 18% of turns, report intent-to-treat (assigned arm) and as-served (actual model) results. Intent-to-treat answers "what happens when we assign this arm in production"; as-served answers "how good is the model when it actually runs."

Formula

For a binary primary metric with control rate $p_C$ and treatment rate $p_T$ , a common large-sample estimate of the absolute lift and its standard error is

\hat{\delta} = \hat{p}_T - \hat{p}_C, \qquad \mathrm{SE}(\hat{\delta}) \approx \sqrt{\frac{\hat{p}_T(1-\hat{p}_T)}{n_T} + \frac{\hat{p}_C(1-\hat{p}_C)}{n_C}}.

An approximate 95% confidence interval is $\hat{\delta} \pm 1.96\,\mathrm{SE}(\hat{\delta})$ . Planning sample size for a minimum detectable absolute lift $\delta$ , significance $\alpha$ , and power $1-\beta$ is approximately

n \approx \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \bigl(p_C(1-p_C) + p_T(1-p_T)\bigr)}{\delta^2}

per arm under equal allocation (normal approximation to two proportions). A 10% treatment canary still needs about the same treatment-arm $n_T$ for power, so total traffic is about $n_T / 0.1$ (roughly $10\times$ the treatment-arm size). Relative to a planned 50/50 equal-allocation total of $2n_T$ , that is about $5\times$ more overall traffic, not $10\times$ . Continuous metrics use the same structure with outcome variance $\sigma^2$ in place of Bernoulli variance. Variance-reduction methods such as CUPED shrink effective $\sigma^2$ when a pre-experiment covariate is predictive and strictly pre-assignment.

Diagram

How it works

Hypothesis and unit. State one intentional change (new prompt, reranker, model route) and the assignment unit: user, tenant, conversation, or request. For chat agents, sticky per conversation or user usually beats per-request flips.
Metrics. Predeclare one primary metric (for example, human acceptance of a drafted reply) and guardrails that can veto a win (p95 latency, cost per request, citation-present rate, safety failures).
Assignment. Hash a stable key with an experiment salt into a bucket; route a fixed share of eligible traffic to treatment. Log arm, eligibility, and version on every event.
Integrity. Before reading the primary metric, check sample ratio mismatch (observed arm counts vs planned split), fallback/as-served rates, and that eligibility filters did not differ by arm.
Analysis. Compare arms with a planned test or interval at a fixed horizon (or a predeclared sequential rule). Report both intent-to-treat (by assigned arm) and as-served (actual model) when fallbacks are common.

Offline evals and shadow traffic are upstream gates. An A/B test answers the live question offline fixtures cannot: does the change help real users under real load?

Experiment contract

Declare before peeking:

Field	Examples
Assignment unit	User, tenant, conversation, request
Eligibility	Surface, locale, consent, bot filter
Primary metric	Acceptance, resolution, preference win rate
Guardrails	p95 latency, cost, safety, abstention
Split and stickiness	10% treatment, sticky for 14 days
MDE, power, $\alpha$	+3 pp, 80% power, $\alpha=0.05$
Stopping rule	Fixed horizon or predeclared sequential looks
Exclusions	Internal dogfood, bot traffic, incomplete logs

Changing the primary metric after seeing results invalidates the statistical story the power plan assumed. For multi-turn agents, sticky per conversation usually beats per-request flips so memory, style, and tool state stay on one arm.

Why it matters here

Offline accuracy, answer style, latency, cost, and live behavior often disagree. A notebook win can raise latency enough to erase user benefit, or improve judge scores while safety failures climb. A/B tests are how you decide whether a model, prompt, or retrieval change is worth shipping under real traffic, not only whether a frozen eval looked good. They sit next to offline evaluation, shadow evaluation, canaries, and traffic splits as the causal step in the release path.

Where it appears

Computer-Use / GUI / Browser AgentsPure coordinate clicking is brittle. If the CI dashboard runs an A/B test that moves the Failed tests tab twenty pixels down, or if a different viewport collapses the sidebar, or if a responsive layout kicks in on a slightly narrower...Capstone: Product RankingUse the judged fixture to catch regressions. Use a controlled A/B experiment to compare shopper outcomes. [Experiment Design and A/B Testing](/learn/experiment-design-ab-testing) already made the assignment unit, primary metric, and...Capstone: Production ML PipelineDon't confuse that gate with an A/B test. A canary is limited live exposure with abort rules: if errors, latency, or a later labeled cost breach the policy, you pull the candidate back. An A/B test is a powered comparison with sticky...