Statistics › 🧠 Advanced & Bayesian Modeling

Beta & Bayesian A/B Testing

Turn conversion counts into full distributions of belief, then ask the question a p-value can't: how likely is B actually better than A?

AdvancedUpper-undergraduate probability
💡
The big idea: Instead of collapsing an A/B test into a single yes-or-no p-value, Bayesian testing keeps a whole distribution of belief about each variant's true rate. The Beta distribution is the perfect carrier for that belief: begin with a prior, add your observed successes and failures, and it updates in one algebraic step into a Beta posterior. From there you answer the question you actually care about — how probable is it that B beats A? — with a genuine number.
🎯 By the end, you'll be able to
  • Explain why the Beta distribution is the natural conjugate prior for an unknown proportion
  • Update a Beta prior to its Beta posterior from observed successes and failures
  • Compare two variants with P(rate_B > rate_A) instead of a single p-value
  • Read and interpret a Bayesian credible interval for a rate
📎 You should already know
  • Bayes' theorem and conditional probability
  • The binomial / Bernoulli success-failure process
  • Probability density functions and the normal curve

From one p-value to a distribution of belief

Suppose variant A of a landing page converted 40 of 200 visitors and variant B converted 55 of 200. The classical routine is to run a significance test, read off a single p-value, and declare a winner or a draw. That answer is binary and famously easy to misread.

The Bayesian route keeps more information. It treats each variant's true conversion rate \(p\) as an unknown quantity we hold a whole distribution of belief about. We start from a prior, let the data reshape it into a posterior, and then ask the question we actually care about: how probable is it that B's rate is higher than A's?

🔑 The conjugate-prior shortcut
If your prior belief about a rate \(p\) is a Beta distribution, then after observing \(s\) successes and \(f\) failures the posterior is again a Beta — you simply add the counts to the two parameters. A prior that stays in the same family after updating is called a conjugate prior, and Beta is the conjugate prior for any success-or-failure (Bernoulli) process. No integration required.

What the Beta distribution is

The Beta distribution is a family of curves living entirely on the interval \([0,1]\) — exactly the range a probability or proportion can occupy. It has two shape parameters, \(\alpha\) and \(\beta\), that you can read as pseudo-counts:

  • \(\alpha\) behaves like a tally of prior successes, \(\beta\) like a tally of prior failures.
  • The larger \(\alpha+\beta\), the more data-like the prior and the narrower the curve — more concentrated belief.
  • \(\text{Beta}(1,1)\) is perfectly flat: the uniform distribution, encoding the idea that every rate is equally plausible.
  • When \(\alpha>\beta\) the mass leans toward 1; when \(\beta>\alpha\) it leans toward 0.

That makes Beta the natural container for beliefs about a rate.

\[ f(p;\alpha,\beta)=\frac{p^{\alpha-1}(1-p)^{\beta-1}}{B(\alpha,\beta)},\qquad B(\alpha,\beta)=\frac{\Gamma(\alpha)\,\Gamma(\beta)}{\Gamma(\alpha+\beta)} \]
The Beta density for a rate p on [0,1]. The Beta function B(alpha, beta) merely rescales the area to 1; Gamma is the factorial-extending gamma function.
\[ \underbrace{\mathrm{Beta}(\alpha,\beta)}_{\text{prior}}\;\xrightarrow{\;s\ \text{successes},\ f\ \text{failures}\;}\;\underbrace{\mathrm{Beta}(\alpha+s,\ \beta+f)}_{\text{posterior}} \]
Conjugate updating in one step: add successes to alpha and failures to beta. The posterior mean E[p] = (alpha + s) / (alpha + beta + s + f) is your updated best single estimate of the rate.

Comparing two rates, the Bayesian way

Once each variant has its own posterior — call the densities \(f_A\) and \(f_B\) — comparing them becomes a direct probability question rather than a threshold ritual. We want \(P(p_A>p_B)\): the posterior probability that A's true rate beats B's. It is the joint posterior integrated over every pair of rates where A comes out ahead:

\[ P(p_A>p_B)=\int_{0}^{1}\!\!\int_{0}^{p_A} f_A(p_A)\,f_B(p_B)\;dp_B\,dp_A \]
The winning region p_A > p_B, integrated against both posteriors. This rarely has a tidy closed form, so it is estimated by Monte Carlo: sample many rate pairs, one from each posterior, and count how often A wins.
🎮 Interactive: two posteriors and P(A beats B) LIVE
Both Beta posteriors on the rate axis — A in blue, B in amber — with dashed lines at each posterior mean. More data for a variant tightens its curve. The P(A beats B) readout is a Monte Carlo estimate of P(rate_A > rate_B). The default 40/200 versus 55/200 gives A only about a 4 percent chance of being the better variant.
✨ Why a probability beats a lone p-value

Reporting \(P(p_B>p_A)\) alongside a credible interval answers the decision directly: how likely is B genuinely better, and by how much? Three practical wins follow:

  • It degrades gracefully. With little data the posteriors stay wide and \(P(p_B>p_A)\) drifts toward 50 percent — the method reports its own uncertainty instead of flipping a binary verdict.
  • It carries magnitude. The posterior of the difference tells you the plausible size of the lift, not merely its sign.
  • It can absorb real prior knowledge. If you genuinely know a rate sits near 10 percent, a suitable Beta prior encodes that from the start.

The honest caveat: this is a model-based probability. It is only as trustworthy as the prior you chose and the assumption that visits are independent draws at a stable rate.

Credible intervals: the interval you thought you were getting

A 95% credible interval is the central slice of the posterior that holds 95 percent of the belief mass. Because the posterior is a probability distribution over the rate, you may say plainly: there is a 95 percent probability the true rate lies inside this interval. That is precisely the reading people wrongly attach to a frequentist confidence interval.

You read a credible interval straight off the posterior's quantiles. When a variant has plenty of data its Beta posterior is close to normal, so a quick approximation is \(\text{mean}\pm 1.96\,\sigma_{\text{post}}\), where \(\sigma_{\text{post}}=\sqrt{\alpha\beta/[(\alpha+\beta)^2(\alpha+\beta+1)]}\) is the posterior standard deviation. Widen a variant's data in the sim and watch its posterior — and its interval — tighten.

📝 Worked example: Variant A converted 40 of 200 visitors; variant B converted 55 of 200. Starting from a uniform Beta(1,1) prior on each rate, find both posteriors, their mean rates, a 95% credible interval for B, and estimate P(A beats B).
  1. Update each prior by adding counts. A had 40 successes and 200 - 40 = 160 failures, so its posterior is Beta(1+40, 1+160) = Beta(41, 161). B had 55 successes and 145 failures, giving Beta(56, 146).
  2. Posterior mean rates E[p] = alpha/(alpha+beta): for A, 41/202 = 0.203 (20.3%); for B, 56/202 = 0.277 (27.7%). B looks better — but how sure are we?
  3. Each posterior's spread is sigma = sqrt(alpha*beta / [(alpha+beta)^2 (alpha+beta+1)]), giving sigma_A ~= 0.028 and sigma_B ~= 0.031. A 95% credible interval for B is 0.277 +/- 1.96 x 0.031 ~= (0.216, 0.339), roughly 21.6% to 33.9%.
  4. Both posteriors carry enough data to be nearly normal, so the difference p_A - p_B is roughly normal with mean 0.203 - 0.277 = -0.074 and standard deviation sqrt(0.028^2 + 0.031^2) ~= 0.042. Then P(A beats B) = P(difference > 0) = Phi(-0.074 / 0.042) = Phi(-1.76) ~= 0.04.
✓ The posteriors are Beta(41, 161) and Beta(56, 146), with mean rates 20.3% and 27.7% and a 95% credible interval for B of about (21.6%, 33.9%). There is only about a 4% posterior probability that A actually beats B — so roughly 96% that B is the better variant. Feeding 40/200 and 55/200 into the sim reproduces these numbers.

Check your understanding

1. Why is the Beta distribution the go-to prior for an unknown probability or rate?
Beta lives on [0,1], matching the range of a rate, and it is conjugate to the Bernoulli/binomial likelihood, so the posterior is again a Beta.
2. You hold a Beta(2, 2) prior for a click rate, then observe 8 clicks and 3 non-clicks. Your posterior is:
Add successes to alpha and failures to beta: alpha = 2 + 8 = 10, beta = 2 + 3 = 5, giving Beta(10, 5).
3. In a Bayesian A/B test, the number P(rate_B > rate_A) = 0.97 means:
It is a direct posterior probability about the hypothesis that B's rate exceeds A's — exactly the statement a p-value does not give you.
4. A 95% credible interval for B's rate is (0.22, 0.34). The correct Bayesian reading is:
A credible interval is a statement about the posterior for the fixed unknown rate: 95% of the belief mass sits inside it. The repeated-sampling option is the frequentist confidence-interval reading — a claim about many hypothetical repetitions, not about this single interval — which is a different idea.
✅ Key takeaways
  • The Beta(alpha, beta) family lives on [0,1], making it the natural distribution of belief about an unknown rate; alpha and beta act like counts of prior successes and failures.
  • Beta is the conjugate prior for a Bernoulli/binomial process: prior Beta(alpha, beta) + s successes + f failures gives posterior Beta(alpha+s, beta+f). You just add the counts.
  • The posterior mean alpha/(alpha+beta) is your updated point estimate; the whole posterior encodes how sure you are.
  • Compare variants with P(rate_B > rate_A), the posterior probability that B is better — a direct answer, estimated by sampling both posteriors.
  • A 95% credible interval is the central 95% of the posterior; you can honestly say the true rate lies inside it with 95% probability.
  • Bayesian A/B results are model-based: they depend on the prior and on visits being independent draws at a stable rate.