Math — Bayesian Probability

A progressive implementation of Bayes' theorem — building the complete pipeline from likelihood through posterior probability, using a drug trial scenario with NumPy vectorization, then extending to continuous-range queries with the Beta distribution.

---

Learning Objectives

---

Task-by-Task Reference

Each task below highlights the unique challenge it posed and the new technique introduced — techniques from earlier tasks are not repeated.

---

Task 0 — Likelihood (0-likelihood.py)

Challenge: Compute the binomial likelihood P(x∣n,P)P(x \mid n, P)P(x∣n,P) for an entire vector of hypothetical probabilities — the first step in any Bayesian analysis.

Approach: For a drug trial with nnn patients and xxx side-effect cases, compute (nx)Px(1−P)n−x{n \choose x} P^x (1-P)^{n-x}(xn​)Px(1−P)n−x for every probability in the NumPy array PPP. The binomial coefficient is computed iteratively to avoid factorials. All operations are vectorized over PPP.

New techniques introduced:

Key takeaway: The likelihood answers "how probable is this data under each hypothetical parameter value?" It's the engine of Bayesian inference — all downstream calculations depend on it.

---

Task 1 — Intersection (1-intersection.py)

Challenge: Combine the likelihood with prior beliefs — computing P(data∣P)⋅P(P)P(\text{data} \mid P) \cdot P(P)P(data∣P)⋅P(P), the joint probability of data and parameters.

Approach: Call likelihood() from Task 0, then multiply the result element-wise by the prior array Pr. Validate that Pr has the same shape as P and sums to 1 (using np.isclose for floating-point safety). The intersection represents the unnormalized posterior.

New techniques introduced:

Key takeaway: The intersection is the numerator of Bayes' theorem: prior × likelihood. It's the "unnormalized posterior" — proportional to the posterior but not yet summing to 1.

---

Task 2 — Marginal Probability (2-marginal.py)

Challenge: Compute the marginal probability P(data)P(\text{data})P(data) — the normalizing constant that makes the posterior a valid probability distribution.

Approach: Call intersection() from Task 1, then sum all intersection values with np.sum(). The marginal is a single number — the total probability of observing the data under all hypotheses, weighted by prior beliefs. Bundles local copies of likelihood() and intersection() for self-containment.

New techniques introduced:

Key takeaway: The marginal probability P(data)P(\text{data})P(data) is the denominator of Bayes' theorem. It's a single number that normalizes the posterior: every possible way the data could occur, weighted by prior belief.

---

Task 3 — Posterior Probability (3-posterior.py)

Challenge: Compute the posterior probability P(P∣data)P(P \mid \text{data})P(P∣data) — the updated belief about each hypothesis after observing the data. This is the final output of Bayesian inference.

Approach: Call intersection() and marginal() from previous tasks, then divide: posterior = intersection / marginal. The result sums to 1 — it's a proper probability distribution. The posterior answers: "given the data, what should I now believe about each possible probability PPP?"

New techniques introduced:

Key takeaway: The posterior is the goal of Bayesian analysis — it tells you what to believe after seeing the data. The entire pipeline (likelihood → intersection → marginal → posterior) implements Bayes' theorem: P(H∣D)=P(D∣H)⋅P(H)P(D)P(H \mid D) = \frac{P(D \mid H) \cdot P(H)}{P(D)}P(H∣D)=P(D)P(D∣H)⋅P(H)​.

---

Task 100 — Continuous Posterior (100-continuous.py)

Challenge: Compute the posterior probability that the parameter ppp falls within a specific continuous range [p1,p2][p_1, p_2][p1​,p2​] — moving from discrete point probabilities (Task 3) to interval probabilities using the Beta distribution's cumulative distribution function.

Approach: With a uniform prior, the posterior for a binomial likelihood is Beta(x+1, n−x+1)\text{Beta}(x+1,\, n-x+1)Beta(x+1,n−x+1). Use scipy.special.betainc(a, b, x) — the regularized incomplete beta function, which is the CDF of the Beta distribution. The probability that p∈[p1,p2]p \in [p_1, p_2]p∈[p1​,p2​] is betainc(a,b,p2)−betainc(a,b,p1)\text{betainc}(a, b, p_2) - \text{betainc}(a, b, p_1)betainc(a,b,p2​)−betainc(a,b,p1​). Validates nnn (positive int), xxx (non-negative int ≤n\le n≤n), p1p_1p1​ and p2p_2p2​ (floats in [0,1][0,1][0,1], p2>p1p_2 > p_1p2​>p1​) with specific error messages.

New techniques introduced:

Key takeaway: The Beta distribution is the conjugate prior for the binomial — when the prior is uniform (Beta(1,1)\text{Beta}(1,1)Beta(1,1)), the posterior is Beta(x+1, n−x+1)\text{Beta}(x+1,\, n-x+1)Beta(x+1,n−x+1). The incomplete beta function (scipy.special.betainc) gives exact continuous probabilities without numerical integration. This extends the discrete Bayesian pipeline to answer "what's the probability the true rate is between 30% and 50%?"

---

Technique Inventory

---

Bayesian Pipeline (Conceptual)

Prior P(H)  ──┐
               ├──→ Intersection ──→ Marginal P(D) ──→ Posterior P(H|D) ──→ Continuous Query
Likelihood    ──┘     P(D|H)·P(H)      Σ P(D|H)·P(H)      Intersection/Marginal    Beta CDF subtraction
P(D|H)                                                                              P(p₁ < p < p₂)

Task 100 extends the pipeline beyond point probabilities: after computing the posterior, use the Beta CDF to answer range queries like "what's the probability ppp is between p1p_1p1​ and p2p_2p2​?"

---

Resources

• Bayes' Theorem — Wikipedia
• NumPy Array Operations
• Binomial Distribution
• Think Bayes — Allen Downey