Distributions and Sampling

The previous chapter estimated one unknown rate: the precision of a fraud-review queue. Real AI products emit more than one kind of measurement. An e-commerce support agent may resolve a request or escalate it, route a request into an intent, call tools several times, and take a variable number of seconds to answer.

A distribution describes the values a random quantity can produce and how likely those values are. A wrong distribution does more than make a chart look odd. It can make a simulator produce impossible latency, hide retry bursts, or claim a launch is safe when it isn't.^[1][2]

One agent, four random quantities

Suppose you are planning a load test for an agent that answers order-support requests. For now, the parameter values below are illustrative assumptions, not measured production results. A real launch would estimate them from logged and reviewed traffic.

The phrase first simulation model matters. A distribution is a claim about data shape. If traces disagree with the claim, you revise the model instead of defending the convenient formula.

Exact outcomes and measured ranges

The first split is between discrete and continuous quantities.

For a discrete quantity, you can ask for the probability of one exact value, such as P(tool_calls = 2). For a continuous response time, one exact infinitely precise value has probability zero under the model. Ask for a range or a percentile instead: P(time > 15 seconds) or the p95 response time.

Before sampling, ask:

1. What values are allowed?
2. Which parameter sets the rate, center, or spread?
3. Which tail or threshold would hurt users or cost?
4. What observation would prove this model is a poor fit?

That fourth question separates an engineering simulation from random-number decoration.

Bernoulli: one request either resolves or escalates

Begin with the smallest distribution. Let R = 1 if the agent resolves an order-support request without a human, and R = 0 if it escalates. Assume a planning value of p = 0.72.

The expected value is:

$$E[R] = 0 \times 0.28 + 1 \times 0.72 = 0.72$$

Because R is encoded as 0 or 1, its long-run average is the resolution rate.

A batch of requests still moves around that long-run rate. Here, the distribution is held fixed at 0.72, while six independent 100-request batches vary.

The previous chapter taught you how to put an interval around a rate measured from real labels. Here, you are running the distribution forward to see the outcomes it could generate.

Categorical: one request takes one route

A customer request isn't always binary. It may be exactly one of several intent labels. For the route used by the support agent, suppose the planning mix is:

The four probabilities sum to 1.00 because every request receives one route in this simplified taxonomy. A categorical distribution draws one of those labels according to the supplied weights.

The sampled counts won't equal the expected counts exactly. That is ordinary variation. An unknown label such as chargeback_legal would be a different problem: the taxonomy or the sampler contract is wrong.

Poisson: a first baseline for tool-call counts

Tool calls are counts. A simple baseline is a Poisson distribution with parameter lambda, written $\lambda$. If lambda = 2.2, the simulated agent makes 2.2 tool calls per request on average.

For a Poisson quantity C, the probability of exactly k calls is:

$$P(C=k) = \frac{\lambda^k e^{-\lambda}}{k!}$$

With $\lambda=2.2$, compute a few values before sampling:

The Poisson model makes a stronger claim than "counts can't be negative." Its mean and variance are both $\lambda$. That assumption can be reasonable for independent events arriving at a steady rate. It's often questionable for an agent: a difficult refund request may trigger a search, a policy lookup, a retry, and a handoff together. Bursts create overdispersion, where variance is much larger than the mean.^[1][2]

Use a stress-test fixture to make that failure visible. The steady stream comes from the Poisson baseline. The bursty stream mixes ordinary requests with a small fraction of hard requests requiring many tools.

When the variance-to-mean ratio is far above 1, the Poisson baseline understates burst behavior. You don't need a more advanced count distribution yet. You need the habit: inspect the fit before trusting cost or capacity conclusions.

Lognormal: positive time with a slow tail

End-to-end agent response time can't be negative, and tool-assisted requests often form a slower right tail. A lognormal baseline encodes those two facts: if T is lognormal, then log(T) is normally distributed.

Choose parameters with care. NumPy's mean and sigma arguments for lognormal describe log seconds, not seconds. Setting mu = log(6) gives a median response time of 6 seconds. With sigma = 0.55, the mean is higher than the median because slow requests pull the right tail outward.

$$\text{median}(T) = e^\mu \qquad E[T] = e^{\mu + \sigma^2/2}$$

The median describes a typical request; p95 tells you about a noticeably slow slice. Reporting only the mean would conceal the operational question customers feel: how long do the slow responses take?

Failure case: a normal latency model invents time

A normal distribution may look convenient because it takes a familiar mean and standard deviation. For positive response time, it can generate impossible negative seconds.

This isn't a harmless inconvenience. If a simulator violates the physical shape of the metric, it can't be trusted to estimate service-level risk without repair.

Monte Carlo: repeat draws to estimate a question

Monte Carlo simulation means using repeated random samples to approximate a quantity you care about. For the Bernoulli resolution rate, larger simulated batches sit more tightly around the assumed p. The familiar 1 / sqrt(N) behavior returns: multiplying sample size by four roughly halves standard-error scale.

Simulation answers a conditional question: "What happens if these assumptions are the generating process?" It doesn't validate p = 0.72, lambda = 2.2, or the latency parameters. Those must come from data, review design, and production measurement.

Build it: a launch-simulation report

Now put the shapes together. The report below uses one seeded generator and prints the quantities an engineer could review before planning capacity.

Use that output as a simulator report, not a production claim. The rate, label mix, count tail, and latency tail are only as credible as their fitted parameters and assumption checks.

The LLM connection: tokens are categorical samples

The support-agent labels above form a categorical distribution because one label is selected from a finite list. An LLM uses the same mathematical family at a much larger scale: after producing a logit for each candidate token, softmax converts the logits into probabilities, and a decoder may sample one token from the weighted vocabulary.^[3]

Take a tiny continuation for the prompt Your refund is. Four tokens are enough to see the mechanics:

Later chapters teach decoding controls in depth. For now, retain the bridge: a sampling decoder doesn't choose from raw logits directly. It draws from a probability distribution derived from those logits. A greedy decoder is different: it selects the highest-probability token instead of drawing a sample. For open-ended text generation, nucleus sampling chooses a dynamic high-probability token set and renormalizes it before sampling, rather than sampling from an unreliable long tail.^[4]

A review checklist for simulated metrics

Before believing output from a random simulator, write down its contract:

The seed also belongs in the report. A seed makes one simulated run reproducible. It doesn't make an incorrect model correct.

Practice: challenge a candidate launch model

A teammate proposes this e-commerce support-agent simulator:

Write a review with four parts:

1. Name one parameter that needs evidence from logs or reviewed traffic.
2. Identify the impossible or missing state.
3. State one count diagnostic for the tool-call assumption.
4. State which latency summary should appear in the report.

Mastery check

Key concepts

• Bernoulli and categorical distributions
• discrete versus continuous values
• Poisson assumptions and overdispersion
• lognormal positive measurements
• Monte Carlo simulation
• seeded reproducibility
• token sampling as a categorical draw

Evaluation rubric

• Foundational: Identifies whether a support-agent measurement is binary, categorical, count-valued, or positive continuous.
• Developing: Explains the roles of p, categorical probabilities, lambda, mu, and sigma in plain English.
• Intermediate: Computes a Bernoulli expectation and several Poisson probabilities, then reads simulated output without confusing a sample with a parameter.
• Applied: Builds a seeded simulation report with rate, label mix, tool-call threshold, and p95 response time.
• Advanced: Rejects a distribution whose generated values or dispersion contradict real system behavior, and connects categorical sampling to LLM tokens.

Follow-up questions

Common pitfalls

• Symptom: Simulated response time becomes negative. Cause: a symmetric normal model was used for positive time with large spread. Fix: use a positive baseline and validate it against response-time traces.
• Symptom: Mean tool cost looks fine while costly requests spike. Cause: a Poisson baseline hid bursty tool loops. Fix: compare variance with mean and report threshold exceedance.
• Symptom: Simulation never routes damaged-item tickets. Cause: categorical support omitted a real label. Fix: define and validate the taxonomy before sampling.
• Symptom: Two reviewers can't reproduce output. Cause: sampler state or seed is missing. Fix: pass one named seeded generator through the simulation.
• Symptom: A team treats simulated rates as measured performance. Cause: parameters were assumed rather than estimated. Fix: fit and check parameters from representative evaluation or production data.