Causal Inference and Counterfactuals

An AI assistant resolves 54% of its tickets, while the old workflow resolves 76%. A release report calls the assistant harmful. Then the tickets are split by risk: the assistant is ten percentage points better inside both easy and difficult groups. Which result should drive the rollout decision?

Experiment Design and A/B Testing Foundations introduced fair comparisons. Causal inference asks whether a change caused an outcome, then makes the assignment assumptions behind that conclusion explicit.^{[1]Reference 1Causal Inference in Statistics: A Primerhttps://bayes.cs.ucla.edu/PRIMER/} Start with the counts, not the label on the report.

Watch the aggregate reverse the group results

Keep risk measured before routing. Within each group, the assistant's observed success rate is ten percentage points higher. Before looking at the totals, predict what the aggregate should say if the groups had the same mix.

The totals compare different populations. The assistant receives 90 difficult tickets; control receives 90 easy tickets. That unequal mix flips the aggregate direction.

This reversal is Simpson's paradox: group composition changes the direction of the aggregated association. The observational counts alone still don't establish a causal improvement.

The next cell computes both within-group effects and the reversed aggregate from the original counts.

The reversal disappears once both workflows are compared against the same risk distribution. That repairs the visible mix, but it still leaves a deeper question: for one assistant-handled ticket, what would control have done?

Separate observed outcomes from missing alternatives

Take one difficult ticket that received the assistant. We observe its treated outcome, but we don't observe what the same ticket would have done under control. Those two branches are the potential outcomes $Y(1)$ and $Y(0)$; their difference $Y(1)-Y(0)$ is that ticket's individual treatment effect.

Only one branch is observed for each ticket. If the assistant handles it, the control outcome is a counterfactual. A logging system can't recover that missing outcome by looking at another, potentially different, ticket.

The average treatment effect averages those individual differences over a named target population:

$$\operatorname{ATE}=E[Y(1)-Y(0)]$$

To estimate it, name the target population, treatment, outcome window, assignment mechanism, and assumptions. "Assistant tickets resolved faster" isn't enough without those details.

Why can't the control group's 76% aggregate success rate serve as the missing outcome for every assistant ticket?

Answer

The control group contains a much larger share of easy tickets. Its aggregate outcome describes a different population, so it is not a valid counterfactual for the harder assistant-assigned cases.

Draw the assignment mechanism

The table shows an imbalanced mix, but not why routing produced it. Suppose the system sends difficult tickets to the assistant more often. Risk exists before routing and influences both assignment and resolution, which makes it a confounder.

The following directed acyclic graph (DAG) turns that routing story into an explicit causal graph.

The path assignment ← risk → resolution is a backdoor path: risk connects assignment to resolution before the assistant acts. Backdoor adjustment for observed pre-treatment risk can block that path if risk captures the relevant common causes.^{[1]Reference 1Causal Inference in Statistics: A Primerhttps://bayes.cs.ucla.edu/PRIMER/}

Don't adjust for a variable created after the assistant acts, such as the assistant's own generated escalation note. It can lie on the causal path or introduce a fresh selection bias, so it can't replace pre-treatment risk.

Ticket risk is recorded before routing, while an escalation note is created by the assistant after routing. Which variable can support the stated backdoor adjustment, and why can't the other replace it?

Answer

Pre-treatment risk can block a common-cause path when it captures the relevant assignment and outcome differences. The assistant-generated escalation note occurs after treatment, so conditioning on it can remove part of the effect being estimated or introduce a new selection bias.

Standardize both arms to one population

Choose a target population with 70% low-risk and 30% high-risk tickets. The low-risk contribution is $0.7 \times 0.10 = 0.07$; the high-risk contribution is $0.3 \times 0.10 = 0.03$. Add them to get the target-population effect:

$$\operatorname{ATE}=0.7(0.90-0.80)+0.3(0.50-0.40)=0.10$$

Both groups improve by the same ten percentage points, so this weighted result is also ten percentage points. If subgroup effects differed, the target-population weights would change the answer.

The next cell continues the earlier session and applies one shared population distribution to both arms.

The arithmetic answers a target-population comparison. It identifies a causal effect only under explicit assumptions:

The smallest strata contain only ten tickets, so their observed rates are statistically uncertain. Standardizing both arms against a shared risk mix repairs one documented assignment imbalance; it doesn't prove that the adjusted ten-point estimate is exact or eliminate unmeasured confounding.

Both observed risk groups show a ten-point advantage, but a hidden customer-priority field affects both assistant routing and resolution. Does standardizing only by recorded risk identify the causal effect?

Answer

Not necessarily. The arithmetic still returns a risk-adjusted ten-point association, but an unmeasured common cause violates conditional exchangeability. Without measuring and appropriately adjusting for customer priority, or changing the assignment design, that association isn't justified as an identified causal effect.

Refuse effects when the alternative has no support

A propensity score estimates the probability that a ticket receives the assistant given pre-treatment features. Now test the assignment boundary: if every high-risk ticket always receives the assistant, there are no high-risk control outcomes in the log. No amount of weighting can invent that missing comparison.

Randomization makes treatment assignment independent of pre-treatment risk in expectation. It doesn't remove missing outcomes, attrition, interference, or measurement errors. A randomized design still needs a clear estimand and honest analysis.

Use a treatment-comparison worksheet to replay the decision. Preserve the low-risk and high-risk counts, both raw aggregate rates, and the declared 70%/30% target weights. Calculate the misleading -22 percentage-point aggregate and the +10 percentage-point standardized association.

Then inspect each risk group: record treated and control sample sizes, name the pre-treatment adjustment variables, and check overlap. Add rows for unmeasured confounding, shared outcome windows, interference, and the uncertainty from ten-ticket strata. If an identification check fails, label the result an observational association instead of a causal effect.