Probability for Machine Learning

An abuse detector sends an API signup to manual review. On labeled signup data, it catches 95 percent of abusive signups. That number sounds reassuring until a reviewer asks a different question: when a signup is flagged, how often is it actually abusive?

If abuse is rare, even a modest false-alarm rate can fill the review queue with clean signups. That queue determines whether reviewers can keep up, so confusing detector recall with queue risk can turn into a costly operating decision.

Start by counting the flagged pile, before any formula renames it.^{[1]Reference 1Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html[2]Reference 2Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/}

Start with the population

Suppose a fully labeled reference population contains 10,000 signups from one API signup surface. These illustrative counts make the calculation concrete. Pick one row uniformly at random, so every signup is equally likely to be chosen.

An abusive signup is one of 100 rows in this population, so its probability is a fraction:

Read the result in plain language: before the detector says anything, 1 percent of signups are abusive.

That starting probability is the prior: the abuse rate before learning whether this signup was flagged. The posterior will be the probability after we observe that evidence.

The sample space, written $\Omega$, is the set of possible outcomes of our random pick: all 10,000 rows. An event is a subset of those outcomes. Let $A$ be the 100 abusive rows and $B$ the rows the detector flags. Probability assigns each event a value from 0 to 1; the whole sample space has probability 1.^{[3]Reference 3Deep Learning.https://www.deeplearningbook.org/}

For this example, frequencies are exact probabilities for a random pick from the reference population. Using them for future traffic adds an assumption: that traffic is sufficiently similar. Estimate its prior from a representative labeled sample, not just reviewed tickets. A queue selected by the detector overrepresents flagged signups and won't generally reveal the abuse rate among all signups.

Now reduce each selected signup to one bit: let Y = 1 when it's abusive and Y = 0 when it's clean. This is a random variable, a number whose value depends on which case you picked.

Its expectation is the probability-weighted average of its possible values. A 1 contributes with probability 0.01 and a 0 with probability 0.99:

That 0.01 is the base rate written as an average instead of a fraction. A 0/1 variable like this is a Bernoulli variable, and its expectation is the probability of seeing the 1 outcome.

The average doesn't tell you what one pick looks like. Variance measures the average squared distance from the expectation, Var[Y] = E[(Y - E[Y])^2].

Here the 100 ones sit 0.99 above the mean, and the 9,900 zeros sit 0.01 below it. Weight their squared distances by their frequencies:

For any Bernoulli variable with success probability p, this simplifies to p(1 - p). Here, variance describes the spread of individual 0/1 outcomes around the 1 percent average.

It isn't the same as uncertainty in an estimated rate or variation across product slices and repeated training runs. Later statistics chapters separate estimation noise from true distribution shifts.

The rows make this concrete: encode labels as 0s and 1s, and their mean must equal $P(\text{abusive})$. Before running the snippet, predict its last two lines from the prior and Bernoulli variance.

abuse_indicator = [1] * 100 + [0] * 9_900

prior = sum(abuse_indicator) / len(abuse_indicator)
variance = sum((value - prior) ** 2 for value in abuse_indicator) / len(abuse_indicator)

print(f"signups: {len(abuse_indicator):,}")
print(f"prior = E[Y]: {prior:.4f}")
print(f"Var[Y]: {variance:.4f}")

assert prior == 0.01
assert abs(variance - prior * (1 - prior)) < 1e-12

signups: 10,000
prior = E[Y]: 0.0100
Var[Y]: 0.0099

Evidence changes the question

Now let the detector provide evidence. Before looking at the posterior, predict what each rate is counting: the abusive pile, or the clean pile?

These are class-conditional rates. The true-positive rate, also called recall, asks: among abusive signups, what fraction are flagged?

The false-positive rate asks the parallel question for clean signups: what fraction are flagged anyway?

Once a flag arrives, product has to ask a different question:

Given that this signup was flagged, how likely is it abusive?

Notation makes the two questions visible:

Those lines aren't interchangeable. The first counts inside the abusive pile; the second counts inside the flagged pile.

Read the vertical bar as "given." What appears to its right determines which rows you count inside.

Count the flagged pile

Hold off on Bayes rule. Build the review queue directly from the two base piles.

Abusive signups:

Clean signups:

Now put the flagged signups into one pile.

Pause before dividing: which source should dominate the flagged queue? Five percent sounds small, but it acts on 9,900 clean signups. That produces more rows than 95 percent of the 100 abusive signups.

Now divide the abusive rows by the full flagged pile:

A flagged signup is about 16 percent likely to be abusive, not 95 percent likely. Recall measured the detector inside abusive rows; posterior risk measures the queue after the detector filters everyone.

Make the accounting executable. This small program prints both source piles before it prints the posterior, so each denominator stays visible.

total_signups = 10_000
abuse_signups = 100
clean_signups = total_signups - abuse_signups
true_positive_rate = 0.95
false_positive_rate = 0.05

true_flags = round(abuse_signups * true_positive_rate)
false_flags = round(clean_signups * false_positive_rate)
flagged_signups = true_flags + false_flags
posterior = true_flags / flagged_signups

print(f"true flags:  {true_flags}")
print(f"false flags: {false_flags}")
print(f"all flags:   {flagged_signups}")
print(f"P(abuse | flagged): {posterior:.3f}")

assert (true_flags, false_flags, flagged_signups) == (95, 495, 590)

true flags:  95
false flags: 495
all flags:   590
P(abuse | flagged): 0.161

The table fixes both the population and its detector outcomes, so its posterior is exact. If 0.95 and 0.05 instead describe chances for new signups, the realized counts will vary around their expected values.

Simulate 100,000 independent new signups to see that variation. rng.random() draws in [0, 1), so comparing it with 0.01 produces abuse about 1 percent of the time. A second draw applies the appropriate flag probability. The fixed seed makes this run reproducible.^{[4]Reference 4random: Generate pseudo-random numbershttps://docs.python.org/3/library/random.html}

import random

rng = random.Random(12)
total_signups = 100_000
abusive_count = 0
true_flags = 0
all_flags = 0

for _ in range(total_signups):
    abusive = rng.random() < 0.01
    flag_probability = 0.95 if abusive else 0.05
    flagged = rng.random() < flag_probability
    if abusive:
        abusive_count += 1
    if flagged:
        all_flags += 1
        if abusive:
            true_flags += 1

simulated_posterior = true_flags / all_flags

print(f"abusive signups: {abusive_count}")
print(f"true flags:      {true_flags}")
print(f"all flags:       {all_flags}")
print(f"simulated posterior: {simulated_posterior:.1%}")
print(f"counted posterior:   {95 / 590:.1%}")

assert abs(simulated_posterior - 95 / 590) < 0.02

abusive signups: 983
true flags:      933
all flags:       5823
simulated posterior: 16.0%
counted posterior:   16.1%

Independence and fixed rates are assumptions of this simulation, not findings about real signups. A coordinated burst can link many rows. Drawing distinct rows from the fixed 10,000-row population without replacement also makes their labels dependent: selecting one abusive row leaves 99 among the remaining 9,999 rows. A seed repeats the chosen experiment; it doesn't establish that its sampling assumptions fit production.

Conditioning means narrowing the world

Conditional probability is a filter followed by a count. It changes which rows are eligible before you divide.

For $P(\text{abusive} \mid \text{flagged})$, first keep only flagged rows. The denominator isn't all 10,000 signups; it's the 590 signups that survived the filter.

Ask "among which cases?" before you divide. Conditioning throws away the 9,410 passed signups. Those 95 abusive flags sit inside 590 flagged rows, not inside 10,000 signups.

View larger

Joint and marginal probability

Once the filter is visible, name the other views of the same rows. A joint probability is the chance that two events happen together. In this population, 95 signups are both abusive and flagged, so:

A marginal probability is the chance of one event without splitting by the other. To get the marginal flag rate, add every row that can produce a flag:

These views are linked by one identity. When $P(B) > 0$, conditional probability is the joint probability divided by the probability of the world you conditioned on:

Read in the other direction, the same identity gives the multiplication rule: a joint probability is a conditional probability times the probability of its conditioning event. The two conditional forms below require both $P(A) > 0$ and $P(B) > 0$.

Check it against the counts: $P(\text{abusive} \mid \text{flagged}) = 0.0095 / 0.059 \approx 0.161$, the same posterior as before.

Bayes rule after the counts

The count is clear now, so give it a compact name. Bayes' theorem, also called Bayes rule, is the formula version of the flagged-pile count.

For events $A$ and $B$ with $P(A) > 0$ and $P(B) > 0$:

For this example:

When $B$ is the observed evidence, $P(B \mid A)$ is its likelihood under hypothesis $A$. It asks how compatible a flag would be with the hypothesis that the signup is abusive. Bayes rule combines that compatibility with the prior to obtain the posterior.

The denominator $P(B)$ is the marginal probability of a flag: how often a flag happens at all. It must include both source piles, weighted by how common abusive and clean signups are:

Before substituting, predict what belongs in the denominator: only true flags, or every flag? Both source piles belong there:

With that denominator in hand, compute the posterior:

Same answer as the count table. Bayes rule is the compact form of the same accounting: track where the flagged signups came from before you divide.

Likelihood compares explanations for observed labels

Probability starts with a candidate rate and asks what outcomes it could produce.

Likelihood starts with labels already observed and compares candidate rates as explanations for those labels. The expression can look identical because the quantity being varied is different.^{[1]Reference 1Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html[2]Reference 2Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/}

Suppose reviewers label three flagged signups [abuse, clean, abuse]. At each fixed candidate rate $p$, assume independent labels with that same abuse probability. Knowing one label then doesn't change the probabilities for the others under that candidate model. Here $p$ describes the flagged queue, not all signups.

Try $p = 0.80$. The probability of this ordered sequence is $0.80 \times 0.20 \times 0.80 = 0.128$. Holding the labels fixed and varying $p$ gives the likelihood:

Of these three candidates, 0.80 has the largest likelihood, just ahead of 0.50. It isn't the best value over every possible $p$: the maximum occurs at the observed fraction, $p = 2/3$. The table also doesn't say there's a 12.8 percent chance that $p = 0.80$. Likelihood isn't a probability distribution over candidate rates.

If we record only "two abusive signups out of three" and ignore order, three sequences qualify. The Binomial count likelihood is therefore $3p^2(1-p)$, rather than $p^2(1-p)$ for the one ordered sequence. The factor 3 changes the probability of the observed event, but not which candidate rate maximizes likelihood.^{[5]Reference 5NIST/SEMATECH e-Handbook: Binomial Distributionhttps://www.itl.nist.gov/div898/handbook/eda/section3/eda366i.htm}

With many labels, multiplying likelihood terms or summing their log values leads to maximum-likelihood training, which Statistics and Uncertainty names explicitly.

Independence means no update

If a flag appears at the same rate in abusive and clean signups, predict whether it should update your belief. It shouldn't: equal rates carry no information about the event.

When $P(B) > 0$, seeing $B$ leaves the probability of $A$ unchanged if the events are independent:

Start with a broken abuse detector:

Out of 10,000 signups, this detector produces:

The flagged pile is still 1 percent abusive:

Independence holds here because $P(A \text{ and } B) = 20 / 10{,}000 = 0.002$ equals $P(A)P(B) = 0.01 \times 0.20$. Independent events can still happen together; mutually exclusive events can't.

Abusive and clean are mutually exclusive, but not independent: learning that a signup is abusive changes its probability of being clean from 0.99 to zero.

This detector creates review work without changing the queue's abuse rate.

Make that prediction executable: when both classes are flagged at 20 percent, expect the posterior to equal the prior and the update to be zero.

def posterior_if_flagged(prior, true_positive_rate, false_positive_rate):
    true_flags = true_positive_rate * prior
    false_flags = false_positive_rate * (1 - prior)
    return true_flags / (true_flags + false_flags)

prior = 0.01
posterior = posterior_if_flagged(prior, 0.20, 0.20)

print(f"prior:     {prior:.3f}")
print(f"posterior: {posterior:.3f}")
print(f"update:    {posterior - prior:+.3f}")

assert abs(posterior - prior) < 1e-12

prior:     0.010
posterior: 0.010
update:    +0.000

Two flags aren't necessarily two pieces of evidence

Return to the useful detector with 95 percent recall and a 5 percent false-positive rate. Suppose a second service copies its flag into another field. Seeing both fields doesn't narrow the queue any further: they describe the same 590 signups, so abuse risk stays at 16.1 percent.

Two distinct detectors require a joint rate. With flags $B_1$ and $B_2$, the multiplication rule within the abusive class is:

We can replace the first factor with $P(B_2\mid A)$ only if the two flags are conditionally independent given abuse. To multiply the false-positive rates, we need the corresponding independence within the clean class too. This is different from assuming the flags are independent across all signups.^{[3]Reference 3Deep Learning.https://www.deeplearningbook.org/}

Compare the two assumptions. The copied field has the same joint rates as one detector. Under a different, hypothetical model where both detectors have the original rates and are conditionally independent within each class, the joint rates are $0.95^2$ and $0.05^2$.

prior = 0.01

for description, abuse_joint_rate, clean_joint_rate in [
    ("copied flag", 0.95, 0.05),
    ("conditionally independent flags", 0.95**2, 0.05**2),
]:
    abuse_mass = prior * abuse_joint_rate
    clean_mass = (1 - prior) * clean_joint_rate
    risk = abuse_mass / (abuse_mass + clean_mass)
    print(f"{description}: {risk:.1%}")

copied flag: 16.1%
conditionally independent flags: 78.5%

The 78.5 percent result is justified by that hypothetical independence model. It is wrong for a copied flag. For real detectors that share features or rules, measure joint flag rates within each class instead of multiplying their separate rates without evidence.

Same detector, different world

Freeze detector performance first:

Change only the population. Before reading the posterior column, predict which world makes a flag most trustworthy.

Detector performance stayed fixed; the traffic mix changed. The same detector produces a 16 percent posterior in a 1 percent-abuse world and a 95 percent posterior when abuse is already half the traffic.

That shift can happen across traffic sources or time periods. Holding both detector rates fixed isolates the effect of the base rate; real traffic shifts can change all three rates. Remeasure them before carrying an old queue policy forward.

A posterior function that matches the table

Turn that accounting into a reusable function. Its flow follows the table: validate each probability, form the true-flag and false-flag masses, then divide true flags by all flags.

Before running the sweep, predict its shape. The detector stays fixed while the prior moves from 1 percent to 50 percent, so the posterior should rise.

Put this in probability_demo.py; it rejects invalid probabilities and zero-probability evidence so undefined decisions surface early.

def check_probability(x, name):
    if isinstance(x, bool) or not isinstance(x, (int, float)):
        raise TypeError(f"{name} must be a number, not a Boolean")
    if not 0 <= x <= 1:
        raise ValueError(f"{name} must be between 0 and 1")

def flagged_posterior(prior, true_positive, false_positive):
    check_probability(prior, "prior")
    check_probability(true_positive, "true_positive")
    check_probability(false_positive, "false_positive")

    true_flags = true_positive * prior
    false_flags = false_positive * (1 - prior)
    all_flags = true_flags + false_flags

    if all_flags == 0:
        raise ValueError("evidence probability must be greater than 0")

    return true_flags / all_flags

def main():
    priors = [0.01, 0.10, 0.50]

    for prior in priors:
        posterior = flagged_posterior(prior, 0.95, 0.05)
        print(prior, round(posterior, 3))

    try:
        flagged_posterior(1.4, 0.95, 0.05)
    except ValueError as error:
        print(error)

    try:
        flagged_posterior(0.01, 0.0, 0.0)
    except ValueError as error:
        print(error)

if __name__ == "__main__":
    main()

0.01 0.161
0.1 0.679
0.5 0.95
prior must be between 0 and 1
evidence probability must be greater than 0

The invalid prior is rejected instead of silently clipped. The final call represents a detector that never flags anyone. Conditioning on its flag is undefined because the denominator is zero; returning the prior would invent an answer.

The type check also rejects True, which Python otherwise accepts as an integer, and the range check rejects NaN and infinities.^{[6]Reference 6Built-in Typeshttps://docs.python.org/3/library/stdtypes.html} This helper uses ordinary floating-point probabilities. Extremely tiny products can underflow to zero in code even when the mathematical evidence is possible; the log-space section below explains that separate numerical limit.

A threshold changes two probabilities at once

A detector can emit a score rather than just a flag. A threshold turns that score into review evidence: flag when the score exceeds the threshold. On the same scored dataset, raising the threshold can only remove signups from the queue. Recall and false-positive rate can't increase, but precision needn't improve at every threshold.

Use this illustrative measurement table for the same 1 percent abuse population. In a real system, measure both rates on labeled data rather than treating them as permanent detector properties.

At 10,000 signups, broad review catches 95 abusive signups and falsely flags 495 clean ones. Strict review catches 80 and falsely flags 99, producing a queue of 179. Its abuse fraction is $80/179 \approx 44.7\%$.

Run both policies through the same accounting. Review rate is the fraction of all signups flagged; precision is the fraction of that queue that is abusive. This helper assumes its three inputs are valid probabilities.

def review_metrics(prior, recall, false_positive_rate):
    true_flags = recall * prior
    false_flags = false_positive_rate * (1 - prior)
    review_rate = true_flags + false_flags
    if review_rate == 0:
        raise ValueError("review rate must be greater than 0")
    precision = true_flags / review_rate
    return review_rate, precision

prior = 0.01
policies = [
    ("broad review", 0.95, 0.05),
    ("stricter review", 0.80, 0.01),
]

for name, recall, false_positive_rate in policies:
    review_rate, precision = review_metrics(prior, recall, false_positive_rate)
    print(
        f"{name:16} review={review_rate:6.2%} "
        f"abuse_in_queue={precision:6.2%} recall={recall:6.2%}"
    )

try:
    review_metrics(prior, recall=0.0, false_positive_rate=0.0)
except ValueError as error:
    print("empty queue:", error)

broad review     review= 5.90% abuse_in_queue=16.10% recall=95.00%
stricter review  review= 1.79% abuse_in_queue=44.69% recall=80.00%
empty queue: review rate must be greater than 0

The stricter policy sends fewer signups to review and raises the abuse fraction in that queue, which is precision. It catches fewer abusive signups, so recall falls.

That is the precision-recall tradeoff shown by the output.

Probability exposes the options; product cost and safety policy choose between them. A reviewer team may value queue cleanliness, coverage, or both.

A threshold can also send no signups to review. Then queue precision is undefined because no flagged pile exists, so code should reject or explicitly represent the empty queue instead of dividing by zero.

A score isn't automatically a probability

A model may emit a score such as 0.80 instead of a thresholded flag. Before treating it as risk, ask: among similarly scored signups, what fraction is actually abusive?

A model is calibrated for this event and population when cases assigned a given probability have a matching event frequency. For scores near 0.80, we'd expect an abuse fraction near 80 percent across sufficiently many comparable signups. Individual outcomes can still differ from their assigned probabilities.

Guo et al. found that then-standard image classifiers could achieve high classification accuracy while their confidence estimates were poorly calibrated.

Measure the match; a number in $[0, 1]$ isn't enough to establish calibrated risk.^{[7]Reference 7On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599}

Inspect one small score bucket. Its six labels contain three abusive signups. Compare that observed fraction with the average score, then express the difference in percentage points.

predicted_risk = [0.80, 0.82, 0.78, 0.81, 0.79, 0.80]
observed_abuse = [1, 0, 1, 0, 1, 0]

advertised_risk = sum(predicted_risk) / len(predicted_risk)
observed_rate = sum(observed_abuse) / len(observed_abuse)
gap = advertised_risk - observed_rate

print(f"average predicted risk: {advertised_risk:.0%}")
print(f"observed abuse rate:    {observed_rate:.0%}")
print(f"empirical gap:         {100 * gap:.0f} percentage points")

average predicted risk: 80%
observed abuse rate:    50%
empirical gap:         30 percentage points

Six signups can't establish how a deployed model is calibrated. They do show what to measure: predicted risk against observed abuse in a comparable bucket.

Use labeled data representative of the population where the probabilities will be used. If you fit a calibration adjustment, keep its fitting data separate from the model's training data and evaluate the adjusted probabilities on another held-out set. Checking only reviewed flags won't establish calibration among all signups.^{[8]Reference 8Probability calibrationhttps://scikit-learn.org/stable/modules/calibration.html}

Calibration also doesn't imply useful ranking. A model that assigns every signup risk 0.01 is calibrated in this exact 1 percent-abuse population, yet cannot rank one signup above another. In a 10 percent-abuse population, the same constant prediction is no longer calibrated. Name the event and population whenever you claim calibrated probabilities.

Statistics and Uncertainty teaches how much evidence you need before trusting the measured gap.

Other quantities need other distributions

The abuse label was a Bernoulli 0/1 because one signup either is or isn't abusive. Other ML quantities can take different values, so they need different sets of possible values.

A probability distribution describes how probability is spread over those values; choosing one is a modeling assumption, not a fact that the data proves.^{[1]Reference 1Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html[2]Reference 2Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/}

Use the possible values as a first sanity check.

For Bernoulli, E[Y] = p and Var[Y] = p(1 - p). Categorical class probabilities are nonnegative and sum to one.

For a Binomial count, the mean and variance are np and np(1 - p). For 100 independent signups with p = 0.01, the expected abuse count is 1 and its variance is 0.99. "Expected count 1" doesn't mean every batch contains exactly one abusive signup.^{[5]Reference 5NIST/SEMATECH e-Handbook: Binomial Distributionhttps://www.itl.nist.gov/div898/handbook/eda/section3/eda366i.htm}

For a Poisson count, the mean and variance both equal $\lambda$, the expected number of arrivals in the chosen interval. At an average of two alerts per minute, a five-minute count has $\lambda = 10$, provided the process assumptions hold. Bursty, coordinated abuse may violate those assumptions.^{[9]Reference 9NIST/SEMATECH e-Handbook: Poisson Distributionhttps://www.itl.nist.gov/div898/handbook/eda/section3/eda366j.htm}

A Normal variable isn't bounded to [0, 1], so it isn't a default model for probabilities. Distributions and Sampling later lets you simulate these families and test whether their assumptions fit observed data.

Modeling bridge: The same choice appears in language models. Next-token prediction is categorical: one token out of a vocabulary. Training then cares about the probability assigned to the observed token.

A token model multiplies probabilities

For one observed target token, training rewards probability assigned to that token.

The observed token is the target class. A one-hot target assigns 1 to that class and 0 to every other vocabulary entry. Its cross-entropy contribution is negative log probability, -log(p), where p is the model's probability for the target. Python's math.log uses the natural logarithm, so these losses are measured in nats.^{[3]Reference 3Deep Learning.https://www.deeplearningbook.org/}

Predict the ordering before running the snippet: the observed token with probability 0.01 should incur the largest loss.

import math

for target_probability in [0.90, 0.50, 0.01]:
    loss = -math.log(target_probability)
    print(f"target probability={target_probability:>4.2f}  loss={loss:>5.3f}")

target probability=0.90  loss=0.105
target probability=0.50  loss=0.693
target probability=0.01  loss=4.605

Lower probability costs more because the observed token was more surprising under the model.

For a sequence, extend the same idea one token at a time. The model's joint probability follows the chain rule: multiply the conditional probability of each next token given its preceding context.

Here, $t_i$ is the token at position $i$, and $\prod$ means multiply the terms for positions 2 through $n$. Each term conditions on the preceding tokens; the chain rule doesn't assume tokens are independent. If there's a prompt, every term also conditions on that prompt.

View larger

For three token probabilities, $0.5 \times 0.2 \times 0.1 = 0.01$. Their log probabilities add to the log of that product. With long sequences, summing logs also avoids underflow from multiplying tiny numbers.

For the four tokens in the figure, log(0.216) is about -1.532477, so the summed negative log likelihood is about 1.532 nats. The mean loss per token is about 0.383 nats. Adding the four logs after rounding each to two decimal places instead gives -1.54: keep full precision for calculations and round the final display.

With model parameters fixed, 0.216 is the probability of this sequence. Hold the observed tokens fixed and vary the model parameters, and the same expression becomes their likelihood. "Negative log likelihood" names the loss used for that parameter-fitting perspective.

Make the numeric failure visible by multiplying 200 token probabilities in Python:

import math

token_probability = 0.01
token_count = 200

raw_product = token_probability ** token_count
log_probability = token_count * math.log(token_probability)

print(f"raw product in float: {raw_product}")
print(f"log probability:       {log_probability:.1f}")
print(f"finite in log space:   {math.isfinite(log_probability)}")

raw product in float: 0.0
log probability:       -921.0
finite in log space:   True

The mathematical probability is $10^{-400}$, not zero. It's too small to represent as a nonzero value in Python's usual binary64 floating-point format, so this calculation underflows to 0.0.^{[10]Reference 10Floating-Point Arithmetic: Issues and Limitationshttps://docs.python.org/3/tutorial/floatingpoint.html}

Keep the finite log probability instead. Converting it back with math.exp would underflow again. A genuinely zero token probability is different: its mathematical log is negative infinity, and math.log(0) raises ValueError. Later language-modeling chapters use log-space calculations for cross-entropy and perplexity.

A tiny individual probability can also round to zero before you take its log. Neural-network implementations therefore compute log probabilities directly from logits with an operation such as log_softmax. Computing log(softmax(logits)) as two separate operations can lose a value that the combined calculation preserves. Summing logs only helps if the individual logs were computed without that earlier underflow.

Common mistakes

Most probability bugs begin before the formula. Someone changed the event, population, or denominator without saying so.

When a dashboard number looks implausible, ask one question:

Among which cases am I counting?

Once that answer is explicit, the denominator usually stops being mysterious.

Try it yourself

Use the same 10,000-signup population, with 100 abusive signups. Try the counts first, then use code to check your arithmetic.

1. A signup isn't flagged by the original detector. How many abusive and clean signups are in that group? Compute its abuse probability.
2. An improved detector keeps 95 percent recall but lowers the false-positive rate to 1 percent. Compute its review count and posterior. Why isn't this the same scenario as raising the original detector's threshold?
3. Reviewers can handle 200 of these 10,000 signups. Compare broad review, strict review, and the improved detector. Which queues fit, and how many abusive signups does each miss? Use these as worked counts; tomorrow's queue depends on incoming traffic.
4. Someone proposes using the review queue's abuse rate as next month's prior for all signups. Explain the selection error and suggest a better sampling plan.
5. Compute -log(0.80) and -log(0.10). Then call math.log(0). Explain how a genuinely zero probability differs from a nonzero sequence probability that underflows.

Solution checks

Compare your denominators as well as your answers.

A complete probability claim

For this reference population, a complete claim is:

Of the 590 signups flagged by broad review, 95 are abusive: 16.1 percent. Strict review raises that fraction to 44.7 percent, but misses 20 of the 100 abusive signups instead of 5.

The probabilities describe two queues. They don't choose a policy for you. That decision also needs review capacity, the cost of false alarms, and the harm of missed abuse.