A/B testing
Definition
A/B testing compares two product or model variants on live traffic or a controlled sample, using a predeclared metric and split policy.
Intuition
Treat an A/B test as a measurement contract. It says who sees each variant, which metric decides the outcome, and when the experiment is allowed to stop.
A/B testing helps separate signal from noise. It asks whether an observed difference is large enough, stable enough, and measured cleanly enough to support a decision.
Example
Route 10% of eligible requests to a new reranker and compare answer acceptance against the current reranker on the same eligibility rules.
Study check: state the estimand, the sample, and the assumption that would break the conclusion first.
Decision rule
Use A/B testing when offline evaluation isn't enough and the change can be exposed safely to a controlled slice of traffic.
Use A/B testing when sampling variation, experiment design, forecasting, or uncertainty could change what you ship. State assumptions before interpreting the number.
Common mistake
Changing the metric, audience, or stopping rule after seeing partial results turns the experiment into a story rather than evidence.
Do not report A/B testing without the data-generating process. Seasonality, repeated users, peeking, and sample-ratio problems can invalidate clean-looking math.
Why it matters here
Many AI systems need live evidence because offline accuracy, latency, and user behavior can disagree.