Distributions and Sampling

Estimating one unknown rate is only the start. Real AI products emit several kinds of measurements: an incident assistant may resolve a request or escalate it, route a request into an intent, call tools several times, and take a variable number of seconds to answer.

A distribution describes the values a random quantity can produce and how likely those values are. A wrong distribution does more than make a chart look odd. It can make a simulator produce impossible latency, hide retry bursts, or claim a launch is safe when it isn't.^{[1]Reference 1Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html[2]Reference 2Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/}

One agent, four random quantities

Suppose you're planning a load test for an agent that helps engineers handle CI failures, rollbacks, access requests, and log triage. For now, the parameter values below are illustrative assumptions, not measured production results. A real launch would estimate them from logged and reviewed traffic.

The phrase first simulation model matters. A distribution is a claim about data shape. If traces disagree with the claim, you revise the model instead of defending the convenient formula.

Exact outcomes and measured ranges

The first split is between discrete and continuous quantities.

For a discrete quantity, you can ask for the probability of one exact value, such as P(tool_calls = 2). For a continuous response time, one exact infinitely precise value has probability zero under the model. Ask for a range or a percentile instead: P(time > 15 seconds) or the p95 response time.

Before sampling, ask:

1. What values are allowed?
2. Which parameter sets the rate, center, or spread?
3. Which tail or threshold would hurt users or cost?
4. What observation would prove this model is a poor fit?

That fourth question separates an engineering simulation from random-number decoration.

Bernoulli: one request either resolves or escalates

Begin with the smallest distribution. Let R = 1 if the agent resolves an incident-assistant request without a human, and R = 0 if it escalates. Assume a planning value of p = 0.72.

The expected value is:

$$E[R] = 0 \times 0.28 + 1 \times 0.72 = 0.72$$

Because R is encoded as 0 or 1, its long-run average is the resolution rate.

A batch of requests still moves around that long-run rate. Here, the distribution is held fixed at 0.72, while six independent 100-request batches vary.

This builds on interval estimates from finite labels. A confidence interval uses a finite sample to quantify uncertainty about an unknown population or process rate under its sampling assumptions; the labels already observed aren't themselves uncertain. A distribution runs an assumed process forward so you can inspect the outcomes it could generate.

Categorical: one request takes one route

An incident-assistant request isn't always binary. It may be exactly one of several route labels. For the route used by the assistant, suppose the planning mix is:

The four probabilities sum to 1.00 because every request receives one route in this simplified taxonomy. A categorical distribution draws one of those labels according to the supplied weights.

The sampled counts won't equal the expected counts exactly. That's ordinary variation. An unknown label such as database_restore would be a different problem: the taxonomy or the sampler contract is wrong.

Poisson: a first baseline for tool-call counts

Tool calls are counts. A simple baseline is a Poisson distribution with parameter lambda, written $\lambda$. If lambda = 2.2, the simulated agent makes 2.2 tool calls per request on average.

For a Poisson quantity C, the probability of exactly k calls is:

$$P(C=k) = \frac{\lambda^k e^{-\lambda}}{k!}$$

With $\lambda=2.2$, compute a few values before sampling:

The Poisson model makes a stronger claim than "counts can't be negative." Its mean and variance are both $\lambda$. That assumption can be reasonable for independent events arriving at a steady rate. It's often questionable for an agent: a difficult rollback request may trigger a deploy lookup, a runbook lookup, a retry, and a handoff together. Bursts create overdispersion, where variance is much larger than the mean.^{[1]Reference 1Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html[2]Reference 2Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/}

Use a stress-test fixture to make that failure visible. The steady stream comes from the Poisson baseline. The bursty stream mixes ordinary requests with a small fraction of hard requests requiring many tools.

When the variance-to-mean ratio is far above 1, the Poisson baseline understates burst behavior. You don't need a more advanced count distribution yet. You need the habit: inspect the fit before trusting cost or capacity conclusions.

Lognormal: positive time with a slow tail

End-to-end agent response time can't be negative, and tool-assisted requests often form a slower right tail. A lognormal baseline encodes those two facts: if T is lognormal, then log(T) is normally distributed.

Choose parameters with care. NumPy's mean and sigma arguments for lognormal describe log seconds, not seconds. Setting mu = log(6) gives a median response time of 6 seconds. With sigma = 0.55, the mean is higher than the median because slow requests pull the right tail outward.

$$\text{median}(T) = e^\mu \qquad E[T] = e^{\mu + \sigma^2/2}$$

The median describes a typical request; p95 tells you about a noticeably slow slice. Reporting only the mean would conceal the operational question customers feel: how long do the slow responses take?

Failure case: a normal latency model invents time

A normal distribution may look convenient because it takes a familiar mean and standard deviation. For positive response time, it can generate impossible negative seconds.

This isn't a harmless inconvenience. If a simulator violates the physical shape of the metric, it can't be trusted to estimate service-level risk without repair.

Monte Carlo: repeat draws to estimate a question

Monte Carlo simulation means using repeated random samples to approximate a quantity you care about. For the Bernoulli resolution rate, larger simulated batches sit more tightly around the assumed p. The batch-rate standard deviation shrinks like 1/sqrt(N): moving from N=100 to N=10,000 divides it by about 10.

Simulation answers a conditional question: "What happens if these assumptions are the generating process?" It doesn't validate p = 0.72, lambda = 2.2, or the latency parameters. Those must come from data, review design, and production measurement.

Queueing delays and the utilization knee

A simulation that samples latency in isolation ignores how requests interact. In production, requests wait in queues.

Little's Law states that the average number of requests in a stable system $L$ equals the arrival rate $\lambda$ times the average time $W$ a request spends in the system:

$$L = \lambda W$$

For an M/M/1 queue with Poisson arrivals, exponential service times, one server, and $\lambda < \mu$, utilization is $\rho = \frac{\lambda}{\mu}$. Average total time in the system $W$ and average time waiting in the queue $W_q$ are:

$$W = \frac{1}{\mu - \lambda} \qquad W_q = \frac{\lambda}{\mu(\mu - \lambda)}$$

As utilization approaches 100%, both quantities rise sharply. Consider a service with a mean service time of 50 ms ($\mu = 20$ requests/second):

• At $\lambda = 15$ req/s ($\rho = 0.75$), total time is 0.20 seconds and queue time is 0.15 seconds.
• At $\lambda = 19$ req/s ($\rho = 0.95$), total time is 1.00 second and queue time is 0.95 seconds.

Real services may violate every M/M/1 assumption, and this mean-time formula says nothing directly about P99 latency. Use it as a capacity warning, then validate arrival patterns, service-time tails, and timeout behavior with production traces.

Statistical rejection of simulation models

Before using a simulator to size hardware or plan an SLA, challenge it against empirical production traces. Visual diagnostics like quantile-quantile (Q-Q) plots compare observed quantiles against candidate-distribution quantiles and expose tail mismatches.

The Kolmogorov-Smirnov (KS) statistic measures the maximum vertical distance between an empirical cumulative distribution function (CDF) and a candidate CDF. The usual one-sample KS null distribution assumes the candidate distribution and its parameters were fixed before seeing the sample. Estimating parameters on the same rows invalidates that textbook p-value; use a parametric bootstrap or a test designed for estimated parameters. A small p-value rejects the candidate under the test assumptions; a large p-value only means the sample failed to reject it.

This controlled check specifies both candidates before sampling, then challenges them with positive latency traces:

Build it: a launch-simulation report

Now put the shapes together. The report below uses one seeded generator and prints the quantities an engineer could review before planning capacity.

Use that output as a simulator report, not a production claim. The rate, label mix, count tail, and latency tail are only as credible as their fitted parameters and assumption checks.

The LLM connection: tokens are categorical samples

The incident-assistant labels above form a categorical distribution because one label is selected from a finite list. An LLM uses the same mathematical family at a much larger scale: after producing a logit for each candidate token, softmax converts the logits into probabilities, and a decoder may sample one token from the weighted vocabulary.^{[3]Reference 3Deep Learning.https://www.deeplearningbook.org/}

Take a tiny continuation for the prompt Deploy is. Four tokens are enough to see the mechanics:

Later chapters teach decoding controls in depth. For now, retain the bridge: a sampling decoder doesn't choose from raw logits directly. It draws from a probability distribution derived from those logits. A greedy decoder is different: it selects the highest-probability token instead of drawing a sample. For open-ended text generation, nucleus sampling chooses a dynamic high-probability token set and renormalizes it before sampling, rather than sampling from an unreliable long tail.^{[4]Reference 4The Curious Case of Neural Text Degeneration.https://arxiv.org/abs/1904.09751}

A review checklist for simulated metrics

Before believing output from a random simulator, write down its contract:

The seed also belongs in the report. A seed makes one simulated run reproducible. It doesn't make an incorrect model correct.

Practice: challenge a candidate launch model

A teammate proposes this incident-assistant simulator:

Write a review with four parts:

1. Name one parameter that needs evidence from logs or reviewed traffic.
2. Identify the impossible or missing state.
3. State one count diagnostic for the tool-call assumption.
4. State which latency summary should appear in the report.