Experiment Design and A/B Testing

An on-call engineer stares at a dashboard showing a critical payment service at 7% error rate. The service owner acknowledged the page two minutes ago. The incident assistant suggests an immediate canary rollback in clean prose. That reply can still cite the wrong runbook, take ten seconds to answer, or send the engineer to a human page five minutes later. Should the team ship a query rewrite?

In Decoding Algorithms, you controlled how retrieved runbook passages turned into an answer and logged the decoding policy. Here we focus on an incident assistant and ask a product question: does rewriting its retrieval query help engineers resolve outages? Keep generation and model weights fixed so the experiment isolates that retrieval change.

An A/B test, also called a randomized controlled experiment, assigns eligible incidents randomly to either the existing system (control) or the new system (treatment) and compares their outcomes. It estimates a defined effect for the enrolled population under stated assumptions. Future incidents can differ and require ongoing evaluation.

That measurement supports one of three clear decisions: ship the rewrite, keep investigating, or roll it back.

View larger

Test one change at a time

Start with the incoming request that both arms must answer:

The canary is at 7% errors and the owner acked. Can we roll it back?

Your existing pipeline retrieves runbook passages directly from the raw question and generates an answer. The control path keeps that exact behavior.

A candidate treatment first rewrites the request into a retrieval-focused query: canary rollback error-rate policy owner acknowledgement. The policy forbids adding unsupported thresholds; a model can still violate that instruction, so audit its rewrites. Both arms send their retrieved evidence through the same generator prompt template, model, and decoder. Under trustworthy randomization and measurement, their outcome difference estimates the effect of assignment to the rewrite policy, including its latency overhead and failures.

One-change experiments simplify attribution. A deliberately designed factorial experiment can study multiple changes and their interaction; silently changing the decoder in one arm can't isolate a rewrite-only effect.

Write a hypothesis that can fail:

Query rewriting will increase the resolved-incident rate while keeping grounded-answer audit failures, p95 latency, and human pages inside their budgets.

Here, p95 latency represents the 95th percentile of first-request completion times, recorded once per enrolled incident. It protects the slower tail rather than hiding it inside an average. Predeclare how timeouts enter the metric; silently dropping unfinished requests could make a failing treatment look faster.

Lock the decision rules before results tempt you to change them:

The brief specifies an MDE, or minimum detectable effect: a true effect size at which the planned enrollment should give target power under the planning model. It isn't a hard detection boundary. Keep that target separate from the later launch threshold.

The grounding rule asks whether the data rule out a regression as large as its budget. That's a non-inferiority question: the interval has to stay inside the budget, not merely look small. Speed and human pages remain point-estimate screens in this compact example, not statistical assurances of safety. A deployable brief needs uncertainty rules for those guardrails too. p95 uncertainty requires request-level data, not only the two reported percentiles.

We call this binary metric “resolved” for the lab. It measures usable answer without early escalation, not restored service health. An engineer can mark an answer useful while the service stays down. Conversely, an appropriate escalation can protect users while counting as a metric failure. Validate the proxy against recovery outcomes and audit whether treatment changes marking behavior. The initial incident page before enrollment isn't a new escalation.

The Python labs use the standard library and require Python 3.12+ for Random.binomialvariate.^{[1]Reference 1Python random: binomialvariate and seeded simulationhttps://docs.python.org/3.14/library/random.html#random.binomialvariate} Seeded outputs were checked with Python 3.14.7; synthetic rows and simulated outcomes aren't production measurements.

Random assignment supports causal conclusions only when assignment, instrumentation, and stopping rules stay trustworthy.^{[2]Reference 2Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/} The first choice in that contract is the unit. What receives one arm and keeps it for every later message?

Assign an incident once

Assign the incident, not each message. An engineer might ask a follow-up after seeing the first answer. If the first message uses treatment and the follow-up uses control, the second outcome depends partly on the first answer. The unit has received both policies, so the contrast is no longer clean.

To define a single treatment outcome for each unit, causal inference commonly uses the Stable Unit Treatment Value Assumption (SUTVA):^{[3]Reference 3Causal Inference: What Ifhttps://miguelhernan.org/whatifbook}

1. No interference between units: Unit $i$'s outcome depends only on unit $i$'s assigned treatment, unaffected by the assignment of unit $j$.
2. No consequential hidden treatment versions: Each arm specifies a well-defined policy. Unrecorded versions shouldn't change the counterfactual outcome associated with that arm.

If messages are the units and earlier answers affect later outcomes, the no-interference assumption across those message units fails. If incidents are the units, changing arms mid-incident fails to deliver either promised incident-level policy. Sequentially randomized treatments can be studied, but require a different estimand and analysis. SUTVA isn't a general ban on multi-turn experiments.

Two possible paths of interference across incidents are:

• Behavioral spillover: An on-call engineer sees a treatment recommendation that clarifies a tricky canary rollback procedure, and then uses that learned knowledge 20 minutes later on a control incident.
• Shared compute contention: If the treatment's query rewriter makes extra LLM inference calls or triggers heavy vector database scans, it can exhaust shared GPU worker pools, increasing latency for control incidents hosted on the same infrastructure.

When important spillovers occur mainly within teams, team-level randomization can contain them. Team membership alone doesn't require that design: choose the unit for the intended effect and interference pattern. For equal cluster sizes $m$ with common within-cluster correlation, a standard planning approximation inflates variance by:

Here $m$ is the number of incident outcomes per cluster and $\rho_{\text{ICC}}$ the intracluster correlation coefficient. With $m=20$ and ICC $0.10$, the factor is $2.9$, not $1$. Unequal sizes and different estimators require a more specific calculation.^{[4]Reference 4Changing cluster composition in cluster randomised controlled trials: design and analysis considerationshttps://link.springer.com/article/10.1186/1745-6215-15-184} More messages within one team can't replace independently randomized teams, and team randomization doesn't remove shared-infrastructure spillover.^{[5]Reference 5Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Workhttps://www.microsoft.com/en-us/research/articles/why-tenant-randomized-a-b-test-is-challenging-and-tenant-pairing-may-not-work/}

A stable hash gives each enrolled incident the same arm on every request. Include an experiment name and version in the salt so a later experiment can get a fresh assignment. Lock the salt before outcomes arrive, keep incident IDs immutable, and don't let callers choose IDs to obtain a preferred arm. Hashing approximates random allocation here; it doesn't force exactly equal arm sizes.

import hashlib

EXPERIMENT = "incident-query-rewrite:v1"

def arm_for(incident_id: str) -> str:
    payload = f"{EXPERIMENT}:{incident_id}".encode()
    bucket = int.from_bytes(hashlib.sha256(payload).digest()[:8], "big") % 100
    return "treatment" if bucket < 50 else "control"

incidents = ["inc-014", "inc-014", "inc-001", "inc-021"]
for incident in incidents:
    print(incident, arm_for(incident))

assert arm_for("inc-014") == arm_for("inc-014")
print("repeat assignment is stable")

inc-014 control
inc-014 control
inc-001 treatment
inc-021 treatment
repeat assignment is stable

If the same incident appears twice in the output, the assigned arm must match twice. In a service, persist the experiment version and arm on each event as well. Hashing makes the choice deterministic; logging makes that choice auditable.

Define an outcome from events

The assistant shouldn't get credit merely because it produced an answer. Metric success requires a usable-answer mark by ten minutes after enrollment and no new human escalation page through that deadline. A genuinely absent mark counts as unresolved. Missing telemetry is different: investigate measurement failure rather than silently treating unobserved events as confirmed failures or successes.

Keep incidents whose treatment rewrite timed out in the treatment denominator. This intention-to-treat (ITT) comparison estimates the effect of assignment to the rewrite policy, failures included. Comparing only successful rewrites selects on a treatment-affected event and generally loses the randomized causal comparison. If 5% time out, dropping them might preferentially remove difficult cases, although overload can cause timeouts independently of query difficulty. ITT doesn't recover missing outcomes by itself.

Wait until every enrolled incident has its full follow-up window. A nine-minute-old incident with no page isn't yet a ten-minute success. The rows below are already matured incident summaries, not individual chat messages; solved means a usable mark arrived within the locked window.

The next lab turns matured event summaries into that metric. It also records whether an audit says the answer is grounded in the retrieved runbook and response latency in milliseconds. Read the rows before looking at aggregate counts: which page timing should disqualify a usable answer?

incident_outcomes = [
    {"arm": "control", "solved": True,  "escalation_after_min": None, "grounded": True,  "latency_ms": 1080},
    {"arm": "control", "solved": True,  "escalation_after_min": 4,    "grounded": True,  "latency_ms": 1200},
    {"arm": "control", "solved": False, "escalation_after_min": 2,    "grounded": False, "latency_ms": 980},
    {"arm": "treatment", "solved": True, "escalation_after_min": None, "grounded": True, "latency_ms": 1120},
    {"arm": "treatment", "solved": True, "escalation_after_min": 18,   "grounded": True, "latency_ms": 1240},
    {"arm": "treatment", "solved": True, "escalation_after_min": 6,    "grounded": True, "latency_ms": 1280},
]
# Each fixture incident has already been observed for 20 minutes.
incident_outcomes = [{**row, "observed_minutes": 20} for row in incident_outcomes]

def resolved(row: dict) -> bool | None:
    if row["observed_minutes"] < 10:
        return None  # Pending is not an unresolved outcome.
    escalation_too_soon = row["escalation_after_min"] is not None and row["escalation_after_min"] <= 10
    return row["solved"] and not escalation_too_soon

for arm in ("control", "treatment"):
    rows = [row for row in incident_outcomes if row["arm"] == arm]
    assert all(resolved(row) is not None for row in rows)
    successes = sum(resolved(row) for row in rows)
    grounded = sum(row["grounded"] for row in rows)
    print(f"{arm}: resolved={successes}/{len(rows)} grounded={grounded}/{len(rows)}")

assert resolved({**incident_outcomes[0], "observed_minutes": 9}) is None
assert not resolved({**incident_outcomes[0], "escalation_after_min": 10})
print("pending window is not scored; a page at exactly 10 minutes disqualifies")

control: resolved=1/3 grounded=2/3
treatment: resolved=2/3 grounded=3/3
pending window is not scored; a page at exactly 10 minutes disqualifies

The small rows teach the definition, not the launch result. The treatment row with escalation_after_min=18 still counts as resolved because its page arrives outside the locked ten-minute window. That makes the metric contract inspectable and deterministic.

If later pages matter, choose a longer window or track them separately before launch. The metric also needs enough incidents to distinguish an actual lift from chance variation. The next step is to turn these event-level decisions into an effect estimate.

View larger

Read lift from counts

After the planned window, start with a compact teaching fixture of 2,000 incidents per arm. The counts are small enough to check by hand. Before reading the table, predict which arm has the higher resolution rate and what could still make that apparent win misleading.

This fixture isn't powered for the locked MDE of +2.0 percentage points. The planning lab's shortcut gives 9,246 incidents per arm; its fuller two-proportion approximation gives 9,336 for 80% power.

Keep the roles separate: use the 2,000-row tables to practice lift, intervals, and launch gates; use the powered enrollment estimate when you design a real test for the brief.

Before subtracting the rates, name the target quantity. The estimand is the average effect of assignment to the rewrite policy on resolution among eligible incidents, with each incident weighted once. It includes rewrite failures and follow-up behavior. Random assignment lets us estimate that target by subtracting the two assigned-arm rates, under the no-interference and measurement assumptions already stated.

For this fixture, every enrolled incident has a completed answer and the same blinded audit. In production, predeclare how absent answers and missing audits enter grounding and availability metrics. If only a sample is audited, use its denominator and a predeclared sampling design; don't selectively omit treatment failures or divide sampled failures by all incidents.

The absolute lift is the treatment rate minus the control rate. It answers a practical question: how many additional incidents resolve per 100 enrolled incidents?

The treatment arm has 4.0 additional metric successes per 100 enrolled incidents in this fixture. That's a 4.0 percentage point estimated lift, subject to sampling uncertainty and the proxy's limitations.

The relative lift compares that absolute change with the original baseline. It answers a different question: how large is the change compared with the control rate?

That's a 10.5 percent relative lift. Name which one you report; saying "up 10.5 points" would be wrong because points describe the absolute difference here.

control_resolved, control_n = 760, 2000
treatment_resolved, treatment_n = 840, 2000

p_control = control_resolved / control_n
p_treatment = treatment_resolved / treatment_n
absolute_lift = p_treatment - p_control
relative_lift = absolute_lift / p_control

print(f"control rate: {p_control:.1%}")
print(f"treatment rate: {p_treatment:.1%}")
print(f"absolute lift: {absolute_lift * 100:.1f} percentage points")
print(f"relative lift: {relative_lift:.1%}")

control rate: 38.0%
treatment rate: 42.0%
absolute lift: 4.0 percentage points
relative lift: 10.5%

Put uncertainty around the lift

If you reran the experiment with different incidents, the counts wouldn't be identical. The observed lift is one draw from a larger process, so it needs a measure of how much it could move.

In Hypothesis Tests, Intervals, and pass@k you put a confidence interval around a paired lift. Here the two arms contain independent incidents, so the standard error adds their two Bernoulli variances. For reasonably large binary-outcome arms, a normal-approximation interval is a useful first calculation.

For each arm, $\hat{p}$ is its observed resolution rate and $n$ is its number of enrolled incidents. The estimated standard error of the difference is:

A rough 95 percent interval is $\widehat{\Delta} \pm 1.96 \times SE$. The number 1.96 is the standard-normal cutoff that leaves about 2.5 percent in each tail.

Across repeated experiments satisfying these assumptions, this procedure aims to cover the true effect about 95% of the time. The particular interval we calculate isn't a 95% probability statement about a fixed true effect, and it doesn't cover a different population automatically.

from math import sqrt

control_resolved, control_n = 760, 2000
treatment_resolved, treatment_n = 840, 2000

p_control = control_resolved / control_n
p_treatment = treatment_resolved / treatment_n
diff = p_treatment - p_control
se = sqrt(
    p_control * (1 - p_control) / control_n
    + p_treatment * (1 - p_treatment) / treatment_n
)
low = diff - 1.96 * se
high = diff + 1.96 * se

print(f"estimated lift: {diff * 100:.1f} pp")
print(f"standard error: {se * 100:.2f} pp")
print(f"approximate 95% interval: [{low * 100:.1f}, {high * 100:.1f}] pp")

estimated lift: 4.0 pp
standard error: 1.55 pp
approximate 95% interval: [1.0, 7.0] pp

The estimate is +4.0 points and this approximation gives an interval of about +1.0 to +7.0 points. Because the lower bound is above zero, this planned analysis gives evidence for a positive treatment effect, assuming assignment and instrumentation are sound.

Use the interval to quantify uncertainty about the defined effect. Predicting a future experiment's effect requires a prediction interval. The planning question remains: was 2,000 per arm enough to detect the +2.0 point win you said you cared about? Future rollouts need their own evaluation.

Rare outcomes, clustered incidents, and multiple comparisons require inference methods that account for those features. Select the method before outcomes arrive.

More rows aren't always more evidence

Suppose a logging join copies every incident outcome onto four message rows. Both arm rates stay unchanged, so the lift still looks right. But treating those copies as independent observations divides the estimated standard error by two. No new incidents were enrolled; the extra precision is fictional.

from math import sqrt

p_c, p_t, n_incidents = 0.38, 0.42, 2000
lift = p_t - p_c
for copies in (1, 4):
    assumed_n = n_incidents * copies
    se = sqrt((p_c * (1 - p_c) + p_t * (1 - p_t)) / assumed_n)
    low, high = lift - 1.96 * se, lift + 1.96 * se
    label = "unique incidents" if copies == 1 else "duplicated message rows (WRONG)"
    print(f"{label}: SE={se * 100:.2f} pp, CI=[{low * 100:.1f}, {high * 100:.1f}] pp")

unique incidents: SE=1.55 pp, CI=[1.0, 7.0] pp
duplicated message rows (WRONG): SE=0.77 pp, CI=[2.5, 5.5] pp

Duplicating rows changes the denominator in the formula, not the amount of independent evidence. Aggregate once per randomized incident before computing this interval.

Real within-team correlation is usually less extreme than identical copies, but the same independence issue applies. If teams are randomized, use a team-aware estimator and uncertainty calculation, such as cluster-robust inference or resampling whole teams. Choosing a cluster method doesn't itself fix interference between clusters. The assignment unit, estimand, and analysis must agree.^{[5]Reference 5Why Tenant-Randomized A/B Test is Challenging and Tenant-Pairing May Not Workhttps://www.microsoft.com/en-us/research/articles/why-tenant-randomized-a-b-test-is-challenging-and-tenant-pairing-may-not-work/}

Plan for a meaningful win

A test with too little traffic may return "uncertain" even when the treatment helps. Before launching, declare a minimum detectable effect (MDE): a true effect size at which you want a specified probability of rejecting the no-effect null. Smaller effects can still be detected, and an effect at the MDE can still be missed.

Power planning relates these parameters; power is the complement of Type II error:

1. False positive rate (Type I error, $\alpha$): The budget for false alarms under the null. This lab chooses $\alpha = 0.05$ two-sided ($z_{1-\alpha/2} \approx 1.96$).
2. False negative rate (Type II error, $\beta$): The risk of missing an effect at the chosen MDE. Here $\beta = 0.20$.
3. Statistical power ($1 - \beta$): The probability of rejecting the null at that alternative. Here it's 80% ($z_{1-\beta} \approx 0.8416$). These common choices aren't universal requirements.

Hold the observed rates at 38 and 42 percent, then watch what sample size does to the interval:

from math import sqrt

p_control = 0.38
p_treatment = 0.42
lift = p_treatment - p_control

for n in (200, 500, 2000):
    se = sqrt(p_control * (1 - p_control) / n + p_treatment * (1 - p_treatment) / n)
    low = lift - 1.96 * se
    high = lift + 1.96 * se
    print(f"{n:>4} per arm: [{low * 100:>4.1f}, {high * 100:>4.1f}] pp")

200 per arm: [-5.6, 13.6] pp
 500 per arm: [-2.1, 10.1] pp
2000 per arm: [ 1.0,  7.0] pp

With 200 or 500 incidents per arm, a good-looking +4.0 point estimate is still compatible with no improvement. At 2,000 per arm, this observed +4.0 pp effect's interval finally excludes zero in the teaching fixture.

That isn't the same as being powered to detect the locked design MDE of +2.0 pp, which needs about 9,246 incidents per arm below. More enrollment narrows uncertainty; it doesn't make treatment better by itself.

A shortcut for two equally allocated independent arms with approximately equal outcome variances gives the following per-arm size:

For binary proportions near baseline rate $p$, outcome variance is $\sigma^2 = p(1 - p)$:

Notice the $\delta^2$ in the denominator: halving the MDE from +2.0 pp to +1.0 pp quadruples the required sample size.

from math import ceil
from statistics import NormalDist

baseline_rate = 0.38
mde = 0.02
alpha = 0.05
target_power = 0.80

normal = NormalDist()
z_alpha = normal.inv_cdf(1 - alpha / 2)
z_power = normal.inv_cdf(target_power)
n_per_arm = ceil(
    2 * baseline_rate * (1 - baseline_rate) * (z_alpha + z_power) ** 2 / mde**2
)

print(f"MDE: {mde * 100:.1f} pp")
print(f"target power: {target_power:.0%}")
print(f"planning estimate: {n_per_arm:,} incidents per arm")

MDE: 2.0 pp
target power: 80%
planning estimate: 9,246 incidents per arm

This formula is for planning, not post-result storytelling. Actual platform planning may account for unequal allocation, repeated incidents, variance reduction, multiple metrics, or sequential analysis.

For comparison, the fuller independent-proportions calculation at baseline 0.38, true lift 0.02, equal allocation, and two-sided alpha 0.05 gives about 79.6% power at 9,246 per arm and rounds up to 9,336 per arm for 80%. That small difference explains why the shortcut is useful for sizing the project but shouldn't be mistaken for the final analysis specification.^{[6]Reference 6power_proportions_2indephttps://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.power_proportions_2indep.html}

An 80% power target isn't an 80% chance of satisfying the whole launch gate. If the true lift is exactly +2.0 points, a roughly symmetric estimate falls above the +2.0-point observed-lift threshold only about half the time. Requiring guardrails to pass can reduce launch probability further. Power the actual decision you intend to make, not just its easiest condition.

Use the estimate to decide how much traffic to enroll before the first outcome arrives. Predeclare a minimum calendar duration too when weekdays, on-call rotations, or novelty effects matter. Hitting the sample count during one unusual shift doesn't establish performance across those conditions.

None of that math is trustworthy if the two arms didn't actually receive the traffic you planned.

Check traffic before outcomes

A sample ratio mismatch (SRM) means observed assignment counts disagree suspiciously with the intended split. If the design says 50/50 but your data has 2,350 treatment incidents and 1,650 control incidents, pause before reading resolution lift. Assignment code, event logging, eligibility filters, or bot removal may differ by arm.

This lab computes a chi-square diagnostic for a planned 50/50 allocation. A one-degree-of-freedom value above 10.83 is a deliberately strict alert threshold corresponding to a very small tail probability, about 0.1 percent.

This lab uses a 0.001 threshold. Microsoft's 2020 write-up describes an even stricter 0.0005 threshold. Such a check diagnoses allocation or data-quality problems; treatment effects require a separate test.^{[7]Reference 7Diagnosing Sample Ratio Mismatch in A/B Testinghttps://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/}

observed = {"control": 1650, "treatment": 2350}
expected_each = sum(observed.values()) / 2

chi_square = sum(
    (count - expected_each) ** 2 / expected_each
    for count in observed.values()
)
alert_threshold = 10.83

print("planned split: 50% / 50%")
print(f"observed: control={observed['control']}, treatment={observed['treatment']}")
print(f"chi-square statistic: {chi_square:.1f}")
print("pause analysis for SRM investigation:", chi_square > alert_threshold)

planned split: 50% / 50%
observed: control=1650, treatment=2350
chi-square statistic: 122.5
pause analysis for SRM investigation: True

An SRM alert doesn't identify a bug, and even correctly randomized traffic can occasionally trigger the diagnostic by chance. It says the comparison needs investigation. Diagnose the event path before making a lift claim.

If the first look at lift looks unbelievable, treat instrumentation as the leading hypothesis until you can replay the events.^{[2]Reference 2Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}

Check distinct enrolled incident IDs, not message counts. Treatment could legitimately change follow-ups, so expecting a 50/50 split of messages would test the wrong denominator. No SRM alert means this check found no allocation mismatch, not that assignment or every outcome field is proven correct. For cluster randomization, check the planned cluster allocation; unequal incident counts can be legitimate.

Don't stop the first time noise looks good

Suppose treatment truly has no effect: both arms resolve 40 percent of incidents. One planned two-sided 5% test should falsely detect a difference about 5% of the time. That includes apparent benefits and apparent harms. It isn't a 5% rate of positive wins alone. Predict what happens when the same cutoff is checked 20 times and the first crossing ends the test.

Inspecting the ordinary interval after every batch gives noise many chances to cross. Cumulative looks share earlier observations, so they aren't independent tests; multiplying twenty independent 95% survival probabilities isn't the right calculation either. The stopping policy, not the treatment, changes the false-alarm rate.^{[8]Reference 8Peeking at A/B Tests: Why it matters, and what to do about it.https://doi.org/10.1145/3097983.3097992}

This A/A simulation, with two identical arms, uses 5,000 repeat experiments, 20 looks, and a fixed seed. Each look adds 100 new incidents per arm by drawing a binomial count, which is the same as 100 independent Bernoulli trials at 40%.

It tests this exact stopping policy. Other experiment designs need thresholds matched to their decision rules.

import random

rng = random.Random(11)
trials = 5_000
looks = 20
batch_size = 100
true_rate = 0.40

planned_alarms = 0
peek_alarms = {5: 0, 10: 0, 20: 0}

for _ in range(trials):
    control_successes = 0
    treatment_successes = 0
    crossed = {schedule: False for schedule in peek_alarms}

    for look in range(1, looks + 1):
        control_successes += rng.binomialvariate(batch_size, true_rate)
        treatment_successes += rng.binomialvariate(batch_size, true_rate)
        n = look * batch_size
        p_control = control_successes / n
        p_treatment = treatment_successes / n
        se = (p_control * (1 - p_control) / n + p_treatment * (1 - p_treatment) / n) ** 0.5
        significant = se > 0 and abs(p_treatment - p_control) / se > 1.96
        for schedule in crossed:
            if look % (looks // schedule) == 0:
                crossed[schedule] = crossed[schedule] or significant
        if look == looks:
            planned_alarms += significant

    for schedule in peek_alarms:
        peek_alarms[schedule] += crossed[schedule]

print(f"planned final look, either direction: {planned_alarms / trials:.1%}")
for schedule, alarms in peek_alarms.items():
    print(f"stop at first crossing, {schedule:2} looks: {alarms / trials:.1%}")

planned final look, either direction: 4.6%
stop at first crossing,  5 looks: 14.2%
stop at first crossing, 10 looks: 19.2%
stop at first crossing, 20 looks: 24.9%

View larger

Ordinary fixed-horizon intervals aren't valid for a decision triggered by continuous monitoring. Repeated observation changes the statistical procedure.

Repeated looks aren't the only way to multiply chances. Trying many variants, metrics, or subgroups and reporting whichever interval excludes zero also needs multiplicity control. Predeclare the family of claims and a method such as Bonferroni, or label new subgroup discoveries exploratory and confirm them with fresh data.^{[9]Reference 9NIST/SEMATECH e-Handbook: Bonferroni's Methodhttps://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm}

Sequential testing and continuous monitoring

Product teams want dashboards that update in real time and allow early stopping when a treatment is winning convincingly or hurting users. Two principled mathematical frameworks make this possible without inflating false alarms:

Alpha spending functions

Group sequential designs allocate a total false-alarm budget across looks using a cumulative alpha spending function $A(t)$. The information fraction $t=I_n/I_{\max}$ is approximately $n/N_{\max}$ when information grows proportionally with enrollment; it isn't automatically the same for every metric or design.

Lan and DeMets allow flexible look times under the design's information and monitoring assumptions. One function approximating O'Brien-Fleming spending, for total two-sided error $\alpha$, is:^{[10]Reference 10gsDesign: Lan-DeMets spending functions and sequential boundarieshttps://keaven.github.io/gsDesign/reference/sfLDOF.html}

This is a cumulative error budget. The p-value cutoff and z boundary at each look are calculated from that budget. At $\alpha=0.05$, little error is spent early and $A(1)=0.05$. Actual boundaries must account for correlations across looks and earlier crossings. The gsDesign six-look example has final critical z about 2.045, rather than the single-look value 1.96.^{[10]Reference 10gsDesign: Lan-DeMets spending functions and sequential boundarieshttps://keaven.github.io/gsDesign/reference/sfLDOF.html} Enrollment may need to increase to retain the desired power. Comparing each p-value with $A(t)$ alone doesn't implement the design.

Mixture sequential probability ratio tests (mSPRT)

The mixture Sequential Probability Ratio Test (mSPRT) is one way to construct time-valid inference for a specified observation model.^{[11]Reference 11Always Valid Inference: Continuous Monitoring of A/B Testshttps://pubsonline.informs.org/doi/10.1287/opre.2021.2135}

For a simple null with known density $f_0$, independent observations, common support, and a fixed probability mixing distribution $H$ chosen before seeing outcomes, a mixture likelihood ratio is:

Under those assumptions, $\Lambda_0=1$ and $\Lambda_n$ is a non-negative null martingale with expectation 1. By Ville's inequality:

The bound controls ever crossing $1/\alpha$ for this null, including optional stopping. One always-valid p-value is $\min(1,1/\max_{0\le s\le n}\Lambda_s)$. Inverting valid tests across candidate parameter values can construct a confidence sequence with simultaneous coverage over time.^{[11]Reference 11Always Valid Inference: Continuous Monitoring of A/B Testshttps://pubsonline.informs.org/doi/10.1287/opre.2021.2135}

An A/B null of equal unknown success rates is composite: its shared baseline is a nuisance parameter. Plugging the observed control rate into $f_0$ doesn't automatically give the martingale above. Use a sequential method derived for the actual two-arm model. Changes to eligibility, outcome distributions, or missing-data behavior also need to satisfy that method's assumptions; time validity doesn't control a family of unrelated metrics.

Here's a deliberately smaller problem: one stream with a known null success rate 0.40 and equal prior mass on alternatives 0.20 and 0.60. The outcome sequence is a hand-chosen fixture, not an A/B experiment. Predict why the evidence at step 20 can fall while the running p-value stays low.

null_rate = 0.40
alternatives = (0.20, 0.60)
ratios = [1.0, 1.0]
running_max = 1.0
outcomes = [1] * 10 + [0] * 10

for n, outcome in enumerate(outcomes, 1):
    for j, rate in enumerate(alternatives):
        increment = rate / null_rate if outcome else (1 - rate) / (1 - null_rate)
        ratios[j] *= increment
    mixture_ratio = sum(ratios) / len(ratios)
    running_max = max(running_max, mixture_ratio)
    if n in (10, 20):
        p_anytime = min(1.0, 1.0 / running_max)
        print(f"n={n}: current LR={mixture_ratio:.3f}, max LR={running_max:.3f}, anytime p={p_anytime:.4f}")

n=10: current LR=28.833, max LR=28.833, anytime p=0.0347
n=20: current LR=0.509, max LR=28.833, anytime p=0.0347

For either alternative $q$, the null expectation of one increment is $0.4(q/0.4)+0.6((1-q)/0.6)=1$. That's the mechanism behind the martingale. Ten failures reduce current evidence, but don't undo the earlier crossing. Long streams need numerically stable likelihood calculations; this tiny example demonstrates the stopping logic.

Reduce noise with pre-experiment data

Power doesn't always require more incidents. CUPED (Controlled-experiment Using Pre-Experiment Data) uses a predictive measurement collected before treatment. For a new incident, that could be a frozen summary of its service's resolution history from before the experiment started. It can't be a prior-period outcome of the same not-yet-created incident.

That signal can remove predictable unit-to-unit variation from the outcome.

Let $Y$ be the experiment outcome and $X$ be a pre-experiment covariate. CUPED forms an adjusted metric:

With random assignment and an unaffected covariate, $\mathbb{E}[\bar X_{treatment}-\bar X_{control}]=0$. Using the same fixed coefficient $\theta$ in both arms and the same centering value leaves the expected difference unchanged:

Now compute the variance of the adjusted metric:

Differentiating with respect to $\theta$ and setting to zero yields the optimal coefficient:

Plugging $\theta^*$ back into the variance equation produces the variance reduction formula:

Here $\rho$ is the Pearson correlation between $X$ and $Y$, assuming finite variances and $\operatorname{Var}(X)>0$. For this optimal one-covariate linear adjustment, outcome variance is multiplied by $1-\rho^2$. A poorly estimated coefficient needn't improve precision, and optimizing variance of an arm contrast may need arm-specific variance and allocation considerations.

Deng, Xu, Kohavi, and Walker introduced this approach and reported about 50% variance reduction on Bing experiments, enough to cut runtime or traffic roughly in half. Using the same metric from a one-week pre-period reduced queries-per-user variance by more than 45%.^{[12]Reference 12Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED)https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf}

This simulation gives each independent unit a historical flag available before treatment, then assigns treatment independently. Treat the flag as a simplified precomputed feature, not an earlier response to the experiment. Predict which estimate should have the smaller standard error.

After running it, replace the flag with 1 - baseline_flag and reverse the coefficient's sign. Why does a negatively correlated baseline carry the same predictive information?

import random
from statistics import correlation, covariance, fmean, variance

rng = random.Random(27)
n = 20_000
arm = [rng.randrange(2) for _ in range(n)]
baseline_flag = [rng.binomialvariate(1, 0.45) for _ in range(n)]
resolved_now = [
    rng.binomialvariate(1, min(1.0, max(0.0, 0.18 + 0.46 * prior + 0.04 * assigned)))
    for prior, assigned in zip(baseline_flag, arm)
]
theta = covariance(resolved_now, baseline_flag) / variance(baseline_flag)
mean_baseline = fmean(baseline_flag)
adjusted = [y - theta * (x - mean_baseline) for y, x in zip(resolved_now, baseline_flag)]

def lift_and_se(outcome: list[float]) -> tuple[float, float]:
    control = [value for value, assigned in zip(outcome, arm) if assigned == 0]
    treatment = [value for value, assigned in zip(outcome, arm) if assigned == 1]
    lift = fmean(treatment) - fmean(control)
    se = (variance(control) / len(control) + variance(treatment) / len(treatment)) ** 0.5
    return lift, se

raw_lift, raw_se = lift_and_se(resolved_now)
cuped_lift, cuped_se = lift_and_se(adjusted)
variance_reduction = 1 - variance(adjusted) / variance(resolved_now)

print(f"raw estimate:   lift={raw_lift * 100:.2f} pp, se={raw_se * 100:.2f} pp")
print(f"CUPED estimate: lift={cuped_lift * 100:.2f} pp, se={cuped_se * 100:.2f} pp")
print(f"outcome variance reduced: {variance_reduction:.1%}")
for label, lift, se in (("raw", raw_lift, raw_se), ("CUPED", cuped_lift, cuped_se)):
    print(f"{label} approximate 95% interval: [{(lift - 1.96 * se) * 100:.2f}, {(lift + 1.96 * se) * 100:.2f}] pp")
print(f"covariate correlation: {correlation(resolved_now, baseline_flag):.3f}")

raw estimate:   lift=4.68 pp, se=0.70 pp
CUPED estimate: lift=4.39 pp, se=0.61 pp
outcome variance reduced: 22.1%
raw approximate 95% interval: [3.32, 6.04] pp
CUPED approximate 95% interval: [3.19, 5.59] pp
covariate correlation: 0.470

View larger

In a finite sample, raw and adjusted estimates differ. Estimating $\theta$ from these same outcomes also means the fixed-coefficient expectation argument isn't an exact finite-sample proof of unbiasedness. The code uses a common pooled coefficient and a large-sample plug-in SE. Predeclare the adjustment; use suitable regression-adjusted or cross-fitted inference for richer models instead of searching for whichever adjustment gives the biggest lift.

Adjusted outcomes can fall outside zero and one. That's acceptable: they're numerical variance-reduction scores, not new Bernoulli labels. Report the adjusted lift and its uncertainty, and check whether the standard error actually improves.

Don't use a treatment-affected quantity, such as rewrite-period latency, as a CUPED baseline. Conditioning on it can block part of the treatment effect or introduce collider bias. The key requirement is that the covariate is unaffected by assignment; the original paper also considers an entry-day feature recorded during the experiment under that assumption.^{[12]Reference 12Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (CUPED)https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf} Frozen pre-experiment data makes the boundary easier to audit.

Bandits versus fixed A/B tests

When should you run a traditional fixed-horizon A/B test, and when should you deploy a multi-armed bandit algorithm?

This choice centers on the classic exploration versus exploitation tradeoff:

• Fixed-allocation A/B testing: Uses predeclared assignment probabilities, such as 50/50 or 90/10. Equal allocation is efficient for equal variances and costs, but isn't required. Its main goal here is estimating a defined effect and checking guardrails. Passing a gate can lead to a staged rollout.
• Reward-seeking multi-armed bandits: Adapt allocation using observed rewards, balancing exploration and exploitation. In the stationary setting, a common objective is low cumulative pseudo-regret $R(T)=\sum_{t=1}^T(\mu^*-\mu_{a_t})$, where $\mu_{a_t}$ is the selected arm's true mean and $\mu^*$ the best one.^{[13]Reference 13A Tutorial on Thompson Samplinghttps://arxiv.org/abs/1707.02038} Best-arm identification is a related objective rather than the same problem.

In Thompson Sampling, each arm maintains a posterior over its success probability. For stationary Bernoulli rewards, a Beta prior gives a conjugate posterior. Draw $\theta_k\sim\operatorname{Beta}(\alpha_k,\beta_k)$ for each arm and select the largest draw. In our product, route each new incident, keep its follow-ups assigned, and update only when its outcome matures. The lab simplifies this to immediate rewards and independent pulls.

import random

rng = random.Random(42)
p_control = 0.38
p_treatment = 0.42

alpha_c, beta_c = 1, 1
alpha_t, beta_t = 1, 1
n_pulls = 2000

for _ in range(n_pulls):
    sample_c = rng.betavariate(alpha_c, beta_c)
    sample_t = rng.betavariate(alpha_t, beta_t)
    if sample_t > sample_c:
        reward = 1 if rng.random() < p_treatment else 0
        alpha_t += reward
        beta_t += 1 - reward
    else:
        reward = 1 if rng.random() < p_control else 0
        alpha_c += reward
        beta_c += 1 - reward

pulls_c = alpha_c + beta_c - 2
pulls_t = alpha_t + beta_t - 2
mean_c = (alpha_c - 1) / pulls_c
mean_t = (alpha_t - 1) / pulls_t

print(f"control pulls: {pulls_c} ({pulls_c / n_pulls:.1%}), sample mean: {mean_c:.1%}")
print(f"treatment pulls: {pulls_t} ({pulls_t / n_pulls:.1%}), sample mean: {mean_t:.1%}")
print(f"allocation ratio: {pulls_t / pulls_c:.2f}x")

control pulls: 463 (23.2%), sample mean: 38.2%
treatment pulls: 1537 (76.8%), sample mean: 41.7%
allocation ratio: 3.32x

This seeded run routes 76.8% of pulls to the arm with the higher known simulation rate. It doesn't compare realized outage recovery with a fixed-allocation counterfactual. Another run can favor the worse arm for a while.

Adaptive routing needs different inference and operational choices:

1. Bias isn't always upward. Adaptive sampling, stopping, and selecting which arm to report can introduce different biases, including opposite signs.^{[14]Reference 14Are sample means in multi-armed bandits positively or negatively biased?https://proceedings.neurips.cc/paper_files/paper/2019/file/65b1e92c585fd4c2159d5f33b5030ff2-Paper.pdf} One observed sample mean doesn't establish its repeated-run bias.
2. Ordinary inference isn't automatically calibrated. Allocation and arm sample sizes depend on earlier rewards even when each arm has conditionally independent stationary rewards. Use estimators and sequential uncertainty methods designed for that policy, with routing propensities and enough continued exploration. Adaptation doesn't make causal inference impossible.
3. The reward doesn't encode every guardrail. A resolution-only algorithm can favor a variant with unacceptable latency or grounding. Constrained or contextual bandits can incorporate more structure, but need a justified reward, constraint design, and monitoring.^{[13]Reference 13A Tutorial on Thompson Samplinghttps://arxiv.org/abs/1707.02038}

Choose the design for the objective, outcome delay, traffic, and inference requirements. A fixed allocation is a straightforward fit for our rewrite-only causal question. A bandit can fit ongoing routing with a measurable reward, but nonstationarity, delayed incident outcomes, and several guardrails complicate this toy model. Neither “marketing only” nor “no scientific learning” defines bandits.

Pair answer evaluations on the same prompt

In conversational AI and incident management, prompts vary wildly in difficulty. Some incidents describe simple permission lockouts; others describe complex distributed deadlocks. In a standard between-subject A/B test (incident A gets Control, incident B gets Treatment), this prompt difficulty variance adds directly to the standard error of the lift.

Two related designs should be distinguished. Search interleaving mixes two ranked result lists into one presentation and attributes user clicks using a specified credit rule.^{[15]Reference 15Large-scale Validation and Analysis of Interleaved Search Evaluation.https://doi.org/10.1145/2094072.2094078} Paired answer evaluation generates both candidates for the same prompt and compares them with a judge, as in the original Chatbot Arena.^{[16]Reference 16Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} Showing two complete answers side by side isn't search-list interleaving.

For numerical scores on paired prompts:

Positive covariance reduces the variance of a difference. A shared additive prompt effect cancels exactly in the model below. Real systems can respond differently to the same prompt, so estimate covariance from paired outcomes before predicting the precision gain. A head-to-head preference vote also needs inference for that vote metric, not necessarily a difference of two numerical scores.

import random
from statistics import fmean, variance

rng = random.Random(42)
n_evals = 1000

# Synthetic shared prompt effect on an unbounded numerical score.
# Gaussian scores aren't restricted to a real 1-10 rating scale.
query_difficulty = [rng.gauss(5.0, 1.8) for _ in range(n_evals)]
queries_c = [rng.gauss(5.0, 1.8) for _ in range(n_evals)]
queries_t = [rng.gauss(5.0, 1.8) for _ in range(n_evals)]

# Independent A/B test (two distinct sets of queries)
scores_c_indep = [q + rng.gauss(0, 0.6) for q in queries_c]
scores_t_indep = [q + 0.45 + rng.gauss(0, 0.6) for q in queries_t]
lift_indep = fmean(scores_t_indep) - fmean(scores_c_indep)
se_indep = (variance(scores_c_indep) / n_evals + variance(scores_t_indep) / n_evals) ** 0.5

# Paired test: both candidates scored on the same queries.
scores_c_paired = [q + rng.gauss(0, 0.6) for q in query_difficulty]
scores_t_paired = [q + 0.45 + rng.gauss(0, 0.6) for q in query_difficulty]
diffs = [t - c for t, c in zip(scores_t_paired, scores_c_paired)]
lift_paired = fmean(diffs)
se_paired = (variance(diffs) / n_evals) ** 0.5
var_reduction = 1 - (se_paired / se_indep) ** 2

print(f"independent A/B: lift={lift_indep:.3f}, SE={se_indep:.3f}")
print(f"paired test: lift={lift_paired:.3f}, SE={se_paired:.3f}")
print(f"variance reduction: {var_reduction:.1%}")

independent A/B: lift=0.570, SE=0.085
paired test: lift=0.443, SE=0.027
variance reduction: 89.5%

In this particular simulation, the variance of the lift estimate falls 89.5%, and SE falls from 0.085 to 0.027. Both designs score 2,000 answers; the paired design uses 1,000 shared prompts. This isn't a measured gain for real models or a universal interleaving speedup.

Randomize answer order, blind identities when possible, predeclare tie handling, and check judge reliability. A preferred answer isn't automatically grounded or useful in live incident response. If an engineer sees both answers, their later behavior reflects that combined experience; the preference comparison doesn't estimate the original one-answer incident-resolution effect.

Treat AI evaluation as a stack

An online experiment answers whether the on-call benefits under live use. It shouldn't be the first time you discover that a treatment emits unsupported rollback claims. Check the treatment in layers, each with a different question:

If you use a large language model as a judge for an offline rubric, treat it as a measured evaluator rather than truth. Pin its model and prompt version, then compare it with human labels on a retained calibration set.

Zheng et al. study LLM judges as approximations to human preference and document position bias, verbosity bias, self-enhancement bias (favoring answers the judge model itself produced), and reasoning limitations.^{[16]Reference 16Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

This lab makes the configuration receipt explicit. A comparison with two different decoder versions is rejected before anybody reads its lift.

control = {
    "index": "runbooks-2026-05-01",
    "generator": "incident-generator-v7",
    "generator_prompt": "incident-generator-prompt-v3",
    "decoder": {"selection": "greedy", "schema": "incident-answer-v2"},
}
treatment = {
    "index": "runbooks-2026-05-01",
    "generator": "incident-generator-v7",
    "generator_prompt": "incident-generator-prompt-v3",
    "decoder": {"selection": "greedy", "schema": "incident-answer-v2"},
    "new_component": "query-rewrite-v1",
}

locked_fields = ("index", "generator", "generator_prompt", "decoder")

def validate_receipt(control, treatment):
    mismatches = [field for field in locked_fields if control[field] != treatment[field]]
    if mismatches:
        raise ValueError(f"unplanned changes: {mismatches}")
    return mismatches

mismatches = validate_receipt(control, treatment)

print("intentional treatment change:", treatment["new_component"])
print("unplanned mismatches:", mismatches)
print("locked fields match:", not mismatches)

bad_treatment = {**treatment, "decoder": {"selection": "greedy", "schema": "incident-answer-v3"}}
try:
    validate_receipt(control, bad_treatment)
except ValueError as error:
    print("rejected:", error)
else:
    raise AssertionError("unplanned decoder change was accepted")

intentional treatment change: query-rewrite-v1
unplanned mismatches: []
locked fields match: True
rejected: unplanned changes: ['decoder']

Make the launch decision auditable

The receipt check compares a few declared configuration fields. It doesn't verify the deployed code, outcome logs, randomization, or no-interference assumption. Keep those integrity checks in the experiment record too.

Now combine the evidence. The brief requires a positive interval, an observed lift of at least +2.0 points, guardrails inside their budgets, and trustworthy traffic.

This last lab prints each condition instead of burying the call in a slide deck. Like the earlier lift tables, it reuses the compact 2,000-incident teaching fixture so you can check every line by hand.

For the locked MDE of +2.0 points, the fuller independent-arm calculation targets about 9,336 incidents per arm before accounting for other design requirements. The smaller fixture below is arithmetic practice, not permission for an unplanned early look. Predict which guardrail could keep a positive lift from shipping.

from math import sqrt

report = {
    "control": {"n": 2000, "resolved": 760, "p95_latency_ms": 1180, "escalated": 310, "grounding_failures": 22},
    "treatment": {"n": 2000, "resolved": 840, "p95_latency_ms": 1260, "escalated": 306, "grounding_failures": 23},
    "minimum_ship_lift": 0.02,
    "srm_alert": False,
}

if report["srm_alert"]:
    raise ValueError("Investigate SRM before computing lift")

c = report["control"]
t = report["treatment"]
p_c = c["resolved"] / c["n"]
p_t = t["resolved"] / t["n"]
lift = p_t - p_c
se = sqrt(p_c * (1 - p_c) / c["n"] + p_t * (1 - p_t) / t["n"])
low = lift - 1.96 * se

latency_delta = t["p95_latency_ms"] - c["p95_latency_ms"]
escalation_delta = t["escalated"] / t["n"] - c["escalated"] / c["n"]
grounding_delta = t["grounding_failures"] / t["n"] - c["grounding_failures"] / c["n"]

grounding_control = c["grounding_failures"] / c["n"]
grounding_treatment = t["grounding_failures"] / t["n"]
grounding_se = sqrt(
    grounding_control * (1 - grounding_control) / c["n"]
    + grounding_treatment * (1 - grounding_treatment) / t["n"]
)
grounding_upper = grounding_delta + 1.96 * grounding_se

# Evidence rule was declared in the brief, before outcomes were observed.
checks = {
    "traffic_integrity": not report["srm_alert"],
    "positive_lift_interval": low > 0,
    "observed_lift_threshold": lift >= report["minimum_ship_lift"],
    "latency_budget": latency_delta <= 150,
    "escalation_budget": escalation_delta <= 0.005,
    "grounding_budget": grounding_delta <= 0.005,
    "grounding_noninferiority": grounding_upper < 0.005,
}

for name, passed in checks.items():
    print(f"{name}: {'PASS' if passed else 'FAIL'}")
print(
    f"lift={lift * 100:.1f} pp, low_end={low * 100:.1f} pp, "
    f"ship_threshold={report['minimum_ship_lift'] * 100:.1f} pp, latency_delta={latency_delta} ms, "
    f"grounding_delta={grounding_delta * 100:.2f} pp, grounding_upper={grounding_upper * 100:.2f} pp"
)
print("teaching gate:", "PASS" if all(checks.values()) else "HOLD AND INVESTIGATE")

traffic_integrity: PASS
positive_lift_interval: PASS
observed_lift_threshold: PASS
latency_budget: PASS
escalation_budget: PASS
grounding_budget: PASS
grounding_noninferiority: FAIL
lift=4.0 pp, low_end=1.0 pp, ship_threshold=2.0 pp, latency_delta=80 ms, grounding_delta=0.05 pp, grounding_upper=0.70 pp
teaching gate: HOLD AND INVESTIGATE

View larger

This gate separates evidence from business value. positive_lift_interval asks whether the interval's lower bound is above zero. observed_lift_threshold asks whether the point estimate clears the predeclared +2.0 point bar. Those checks answer different questions.

The interval begins at +1.0, so the evidence supports a positive lift while allowing effects below +2.0 points. A more conservative brief could require the lower bound to clear +2.0 points and would keep this result under investigation.

Guardrails can look safe on a point estimate while their uncertainty still crosses the budget. Our brief already declared the grounding rule: the upper endpoint must stay below +0.5 points. Using the upper end of a two-sided 95% interval corresponds to an approximate one-sided 97.5% bound, not a one-sided 95% bound. Keep that choice fixed rather than switching cutoffs after seeing the result.

For grounding failures as a rate, the unpooled SE is:

With the teaching fixture, grounding rates are 1.10% for control and 1.15% for treatment, so $\widehat{\Delta}_g = +0.05$ pp. The unpooled SE is about 0.333 pp, which puts the rough 95% interval at about $[-0.60, +0.70]$ pp. Its upper end exceeds the +0.5 pp budget, so grounding_noninferiority fails even though the observed change is small.

For these teaching numbers, the lift and point-estimate checks pass, but the grounding rule keeps the decision at HOLD AND INVESTIGATE. The interval includes no change and still extends into an unacceptable regression, so the result lacks precision to rule out that harm.

If this were the planned final look, “investigate” wouldn't mean adding incidents and rerunning the same ordinary interval until the guardrail passes. That would introduce the stopping problem from the peeking simulation. Keep the inconclusive result in the receipt; plan a fresh confirmatory experiment, or follow a continuation rule already justified by the sequential design. Widening the grounding budget after seeing its interval would change the launch question too.^{[11]Reference 11Always Valid Inference: Continuous Monitoring of A/B Testshttps://pubsonline.informs.org/doi/10.1287/opre.2021.2135}

The rare-event grounding interval is only a normal approximation with 22 and 23 failures. A real launch needs a preselected method suitable for those counts and audit sampling. Even a passing teaching gate wouldn't resolve the omitted latency and human-page uncertainty rules. It also reads an SRM flag rather than validating all instrumentation.

Change treatment latency to 1490 and rerun: resolution still improves, but the speed budget also fails. A treatment doesn't earn launch by winning only its favorite metric.

Practice: break the launch gate

Change one field at a time and predict the gate before rerunning it. These are counterfactual exercises, not permission to edit a live experiment's brief after seeing results:

1. Change treatment latency from 1260 to 1490. Which check fails, and why doesn't the resolution lift override it?
2. Restore latency, then change minimum_ship_lift from 0.02 to 0.05. Which check fails even though the interval still excludes zero?
3. Restore the threshold, then set srm_alert to True. Why should analysis pause before anybody argues about the observed lift?

Expected observations

1. latency_budget fails because treatment is now 310 ms slower than control, beyond the locked 150 ms budget.
2. observed_lift_threshold fails because the observed +4.0 point lift is below the new +5.0 point business threshold. Statistical evidence and launch value answer different questions.
3. The code raises ValueError: Investigate SRM before computing lift. SRM can signal broken assignment, filtering, or logging, so the outcome calculation doesn't run. It isn't merely a red badge added after displaying an untrustworthy lift.

Freeze the experiment receipt

Save the assignment unit and hash rule, metric query and outcome window, planned analysis schedule, sample-ratio-mismatch check, effect estimate with uncertainty, guardrails, and final decision. Those artifacts let a reviewer rerun the gate without reconstructing policy from a dashboard. A trustworthy experiment ends as a reproducible decision record, not a screenshot of a favorable number.