Probability

From certainty to likelihood

Every chapter before this one asked what is true, given the rules. This chapter asks a different question: what is likely, given only part of the picture. That question keeps mattering long after the exam ends.

A doctor reads a test result. A bank scores a loan application. A spam filter reads an email. Each one starts with a prior belief, folds in new evidence, and lands on a sharper estimate. Conditional probability, Bayes’ theorem, and the random-variable machinery ahead are three instances of exactly that one move.

This is the book’s closing chapter because updating a belief on partial evidence is the most-used move in the world outside the exam hall. First comes conditional probability itself, restricting attention to what is now known. Then come the theorems that push that update through a whole partition of cases. Last comes the machinery that turns updated uncertainty into a single number worth acting on.

↑ Back to top

Probability, once you know something

For events $A$ and $B$ with $P(B) > 0$, the conditional probability of $A$ given $B$ is $P(A | B) = P(A \cap B)/P(B)$.

Read it as a restriction, not a new rule. $B$’s own probability becomes the new whole; $P(A | B)$ asks what fraction of that whole also lies in $A$. Nothing about $A$ or $B$ separately changes — only the ground being measured against shrinks to $B$.

*The condition $P(B) > 0$ is not a technicality.* “Given $B$” presumes $B$ can actually happen. Dividing by $P(B) = 0$ is undefined, not zero — a definition that ignored this would be asking a question with no event behind it.

Conditioning on a fixed event $B$ still behaves like an ordinary probability measure, once $P(B) > 0$ is assumed. Three properties follow directly from the definition, not as new assumptions but as consequences already sitting inside it.

$P(S | B) = 1$: once $B$ has happened, the whole sample space is certain relative to it. $0 \leq P(A | B) \leq 1$ for any event $A$, because $A \cap B$ can never carry more probability than $B$ itself carries. And the complement rule reads exactly as it did before conditioning: $P(A' | B) = 1 - P(A | B)$, where $A'$ is the complement of $A$ inside $S$.

None of these three needed a separate proof from scratch. Each one drops straight out of treating “conditioned on $B$” as an ordinary probability measure on its own — restricted to the smaller sample space $B$.

↑ Back to top

Turning the definition around

Multiply both sides of $P(A | B) = P(A \cap B)/P(B)$ by $P(B)$. The definition turns into a working tool. The result is the multiplication theorem: $P(A \cap B) = P(B) \cdot P(A | B) = P(A) \cdot P(B | A)$ — valid whichever event is conditioned on.

The same idea extends to three events: $P(A \cap B \cap C) = P(A) \cdot P(B | A) \cdot P(C | (A \cap B))$. Each new event conditions on everything already assumed to have happened before it.

This is the tool for “first this, then that” probabilities. Picture a sequence of draws or trials where the second event’s chances depend on the first having already occurred, and the third’s depend on both.

↑ Back to top

Independent events

Events $A$ and $B$ are independent if $P(A \cap B) = P(A) \cdot P(B)$. That is the defining test. It is checked directly from the probabilities involved, never assumed from how the events happen to be described.

When both $P(A) > 0$ and $P(B) > 0$, this is equivalent to $P(A | B) = P(A)$ and $P(B | A) = P(B)$. Conditioning on one event leaves the other’s probability completely unchanged.

Independence is a property of the probabilities involved, not of the words used to describe the events. Two events described in unrelated language can still fail this test. Two events that sound connected can still pass it — the product rule is the only check that counts.

Mutually exclusive means $A \cap B = \emptyset$ — the two events cannot both happen. Independent means $P(A \cap B) = P(A) \cdot P(B)$ — knowing one happened tells you nothing about the other’s chances. The two words sound like they might describe the same kind of separateness; they do not.

For events with $P(A) > 0$ and $P(B) > 0$, they are actually opposites. Mutually exclusive forces $P(A \cap B) = 0$, which cannot equal $P(A) \cdot P(B)$ when both factors are positive — so such events are never independent.

*If $A$ happens, a mutually exclusive $B$ becomes impossible.* That is the most one event can possibly tell you about another — the far end of dependence, not the middle ground of independence.

For independent events $A_1, A_2, \ldots, A_n$, the product rule chains straight through: $P(A_1 \cap A_2 \cap \ldots \cap A_n) = P(A_1) \cdot P(A_2) \cdot \ldots \cdot P(A_n)$.

No conditioning term appears anywhere in that product. Each event is independent of every other, so the individual probabilities multiply directly and the chain is complete. This is the working rule behind “the probability that all $n$ trials succeed”.

Pairwise independence is weaker than this full chain, and the two should not be confused. Checking that every pair of events satisfies the product rule does not by itself guarantee the full $n$-way product holds. The chain rule above assumes independence across the whole set, not just two at a time.

↑ Back to top

Splitting the sample space into cases

A partition of the sample space $S$ is a set of events $E_1, E_2, \ldots, E_n$ that between them cover every case exactly once. Three conditions make this precise: pairwise disjoint ($E_i \cap E_j = \emptyset$ for $i \neq j$), union equal to all of $S$, and each with $P(E_i) > 0$.

Every outcome in $S$ then belongs to exactly one $E_i$ — mutually exclusive, collectively exhaustive cases, with nothing left over and nothing counted twice.

This is the structure the next two theorems both build on. Break a hard problem into cases, solve each case, recombine. Neither the theorem of total probability nor Bayes’ theorem works without a genuine partition sitting underneath it.

$E_1, E_2, \ldots, E_n$ partition the sample space $S$, and $A$ is any event. The theorem of total probability says $P(A) = \sum_i P(E_i) \cdot P(A | E_i)$.

Each term in that sum is “the chance of being in case $E_i$” multiplied by “the chance $A$ happens, given case $E_i$”. Summing that product over every case recombines the case-by-case pieces into the plain, unconditional probability of $A$.

The theorem follows from the multiplication theorem, applied to each case in turn, then added. The cases are disjoint, so their separate contributions to $P(A)$ never overlap. Nothing here needs a new proof technique — only the partition and the tool already in hand.

↑ Back to top

Reversing the condition: from evidence back to cause

$E_1, E_2, \ldots, E_n$ partition $S$, and $A$ is any event with $P(A) > 0$. Bayes’ theorem reverses the condition: $P(E_i | A) = (P(E_i) \cdot P(A | E_i)) / (\sum_j P(E_j) \cdot P(A | E_j))$.

The numerator is the multiplication theorem applied to case $E_i$ alone. The denominator is the theorem of total probability — $P(A)$ written out as a sum over every case, not just case $i$. What comes out is the posterior probability of case $E_i$, now that $A$ is known to have happened.

*Read it as: start from the prior $P(E_i)$, weight it by how well case $E_i$ explains the evidence.* That weight is the likelihood, $P(A | E_i)$. Then rescale so the posteriors across every case sum to 1 — every number inside the theorem was already available from the partition.

The numerator of Bayes’ theorem already measures how likely a case is — prior times likelihood, for that one case. It is tempting to reuse that same number in the denominator too, since it already captures the case being asked about.

That shortcut drops something Bayes’ theorem needs. The denominator is $\sum_j P(E_j) \cdot P(A | E_j)$, summed over every partition element. It is the theorem of total probability computing the plain $P(A)$ — a number that does not care which hypothesis a student happens to be asking about.

Two urns are chosen with $P = 1/2$ each; let $R$ be “a red ball is drawn”, with $P(R | A) = 2/3$ and $P(R | B) = 1/3$. Correctly, $P(A | R) = (1/2 \cdot 2/3) / [(1/2 \cdot 2/3) + (1/2 \cdot 1/3)] = (1/3)/(1/2) = 2/3$. Dropping urn $B$’s term from the denominator gives $(1/3)/(1/3) = 1$ — a posterior of certainty, even though urn $B$ could plainly have produced a red ball too.

A denominator missing even one partition term is not a rounding error. It silently promotes one hypothesis to certainty. Every case in the partition has to appear in the sum, whether or not it is the case the question happens to be about.

↑ Back to top

Turning outcomes into numbers

A random variable $X$ is a function that assigns a real number to every outcome of a random experiment. Rolling two dice and recording their sum, counting heads in three tosses, timing how long a bulb lasts — each turns an outcome into a number, and that number is what $X$ names.

Its probability distribution lists every value $X$ can take, $x_1, x_2, \ldots, x_n$, alongside the probability of each, $p_1, p_2, \ldots, p_n$ — usually shown as a two-row table, values on top, probabilities below.

*Two conditions make a distribution valid: every $p_i$ satisfies $0 \leq p_i \leq 1$, and $\sum_i p_i = 1$.* The probabilities of all possible values have to exhaust the whole sample space exactly once — nothing left over, nothing counted twice.

The mean, or expectation, of a discrete random variable $X$ is $E(X) = \sum_i x_i \cdot p_i$. Each value is weighted by its own probability, then added up across every value $X$ can take.

The variance, $op(\text{Var})(X) = E(X^2) - [E(X)]^2$, measures spread around that mean, where $E(X^2) = \sum_i x_i^2 \cdot p_i$ is the mean of the squared values rather than of $X$ itself.

*A small variance means $X$’s outcomes cluster near $E(X)$; a large one means they scatter widely, even when the mean is identical.* Two random variables can share the exact same mean and still behave very differently — variance is the number that tells them apart.

↑ Back to top

The book’s last move: update, don’t guess

Every technique in this chapter is the same move wearing different clothes: start with what you know, fold in new evidence, land on a sharper answer.

Conditional probability narrows the sample space to what is now known. The multiplication theorem chains events in sequence. Total probability and Bayes’ theorem push that update through an entire partition of cases, turning a prior into a posterior. The random-variable machinery — distribution, mean, variance — turns that updated uncertainty into a single number worth acting on.

Chapters 1 through 12 asked what is true, given fixed rules, and every answer was exact. This closing chapter asked what is likely, given partial information, and every answer came with a margin. That is the harder, more honest question. It keeps being asked long after the exam is over: a diagnosis read off a test result, a spam filter’s next guess, a weather forecast revised as clouds gather. All of them are this chapter’s Bayes’ theorem, run again.

↑ Back to top

Practice set

Exercise 13.1 practises computing conditional probabilities directly from the definition. Problems range from a described sample space, counted directly, to given probability values plugged straight into the conditional-probability formula. *They apply the complement rule and the bound $P(A|B) \leq 1$ along the way*.

Exercise 13.2 practises the multiplication theorem and tests given pairs of events for independence. Problems chain two- and three-event sequences, contrast independent events with mutually exclusive ones, and apply the chain rule where several trials are all independent.

Exercise 13.3 practises the theorem of total probability and Bayes’ theorem together. Problems set up a partition, compute $P(A)$ by summing over every case, then reverse the condition with Bayes’ theorem to find a posterior. Always sum prior-times-likelihood over the full partition in the denominator — never just the matching term.

Exercise 13.4 practises building a probability distribution table for a random variable. Problems check the two validity conditions — each probability in $[0, 1]$, all of them summing to $1$ — then *compute the mean $E(X)$ and the variance*.

The miscellaneous exercise mixes every technique in the chapter without labelling which one applies. Problems draw on conditional probability, the multiplication and total-probability theorems, and Bayes’ theorem — recognising the right tool is itself part of the exercise.