Resampling and Bayesian reasoning

Learning objective

Given an observed effect and a product decision, choose between a bootstrap distribution, a permutation or randomization distribution, and a Bayesian posterior; compute and interpret each one; identify the unit and exchangeability assumptions; and communicate an action without confusing an interval, a p-value, and a posterior probability.

1. One observed gap, three different questions

Suppose a randomized onboarding experiment has 16 users in treatment and 16 in control. Ten treatment users activate; seven control users activate. The observed difference in activation rates is:

10/16 − 7/16 = 3/16 = +18.75 percentage points

That arithmetic is fixed once the data arrive. The uncertainty is not. A good interview answer starts by saying what you want to learn from the gap.

• How variable would this estimator be across comparable samples? Keep the observed treatment and control roles, repeatedly resample independent units, recompute the effect, and summarize the resulting bootstrap distribution.
• How surprising would this effect be if assignment had no effect? Keep outcomes fixed, rearrange treatment labels only in ways licensed by the design, and count null statistics at least as extreme as the observed one. That is a permutation or randomization distribution.
• How probable is it that the effect clears a practical bar? State a prior and a likelihood, update to a posterior, and calculate something such as P(δ > 10 pp | data, model, prior).

These distributions can look similar in a large, well-behaved experiment, but they answer different counterfactual questions. Bootstrap is about the estimator’s sampling behavior. A null permutation distribution is about assignments under a null hypothesis. A posterior is conditional on a model and prior. Calling all three “simulations” hides the most important interview judgment: what is allowed to move?

2. Bootstrap: approximate repeated sampling from the data you have

The bootstrap treats the observed sample as an empirical stand-in for the population that could have generated it. For each replicate, sample independent units with replacement and keep the treatment structure that was observed. Recompute the estimator. After B replicates, the spread of estimates approximates the estimator’s sampling distribution.

For an activation-rate difference, the algorithm is:

1. Record the observed unit and the estimand: user-level treatment minus control activation.
2. Sample 16 treatment users with replacement from the 16 observed treatment users. Do the same within control.
3. Compute treatment rate minus control rate for this replicate.
4. Repeat B times. Use quantiles of the replicate effects for a percentile interval, or use a studentized or BCa method when the statistic and sample support it.

The important phrase is “independent units.” If treatment was assigned to a household, school, seller, or user, resampling individual sessions pretends there are more independent pieces of evidence than the design supplied. More rows can make a wrong-unit interval look impressively narrow. More bootstrap draws reduce Monte Carlo noise; they do not create independent information.

Worked calculation: two bootstrap replicates by hand

Use a tiny example so every number is inspectable. The control outcomes are [0, 0, 1, 0] and treatment outcomes are [1, 1, 0, 1]. The observed rates are 0.25 and 0.75, so the observed effect is 0.50, or 50 percentage points.

For replicate A, draw control [0, 1, 0, 0] and treatment [1, 1, 1, 0]. The replicate rates are:

control: 1/4 = 0.25; treatment: 3/4 = 0.75; difference: 0.75 − 0.25 = 0.50

For replicate B, draw control [0, 0, 0, 0] and treatment [1, 0, 1, 1]. Its effect is:

control: 0/4 = 0; treatment: 3/4 = 0.75; difference: 0.75 − 0 = 0.75

Two replicates are not an interval. They simply show the mechanics: units can repeat, some observed units can disappear from a replicate, and the statistic is recomputed from the resampled data rather than from a fixed standard error. With many replicates, the middle 95% of the effects would be a percentile interval. If the sample is tiny, the estimator is highly skewed, or the interval depends on a tail statistic, mention a studentized or BCa interval and run a sensitivity check rather than presenting “bootstrap” as a magic correction.

3. Permutation and randomization: build the null you actually mean

A permutation test begins with a null question. If assignment has no effect and units are exchangeable, which treatment labels could have been attached to these fixed outcomes? Pool the outcomes, shuffle the allowed labels, preserve the observed arm sizes, and recompute the statistic. The resulting null distribution is centered near zero for a difference-in-rates statistic.

In a hand-sized example with eight outcomes and four treatment labels, there are C(8,4) = 70 distinct assignments. An exact test can enumerate all 70, calculate each difference, and count how many are at least as extreme as the observed absolute difference. With a larger experiment, sample random assignments. A common Monte Carlo estimate uses:

p ≈ (1 + number of null statistics at least as extreme) / (1 + number of assignments)

The plus-one convention avoids reporting a simulated p-value of exactly zero. It does not solve an invalid permutation scheme. If treatment was blocked, stratified, paired, or cluster-randomized, the shuffle must respect that mechanism. For paired before/after data, swap treatment and control within pairs or randomly flip the sign of each pair difference. For cluster assignment, shuffle cluster labels, not individual rows.

Do not turn the null distribution into a confidence interval by changing its name. A p-value asks how surprising the observed statistic is under a specified null. A confidence interval asks which parameter values are compatible with a repeated-sampling procedure. They can be connected by test inversion, but a casual null histogram is not automatically an interval.

4. Bayesian updating: make prior information visible

Bayesian reasoning starts with a probability model, not with a request to “make the answer more intuitive.” For binary activation, a simple arm-level model is y_g \sim Binomial(n_g, p_g). A Beta prior is conjugate: if p_g \sim Beta(α_g, β_g), the posterior is:

p_g \mid data \sim Beta(α_g + successes,\; β_g + failures)

The posterior for the treatment-control difference δ = p_T − p_C is the distribution of the difference between two posterior draws. It is often easiest to explain with Monte Carlo draws, while labeling the result as conditional on the model and prior.

Worked calculation: the same experiment with a neutral prior

Use independent Beta(1, 1) priors for both rates. Treatment has 10 successes and 6 failures, so its posterior is Beta(11, 7) with mean 11/18 ≈ 0.611. Control has 7 successes and 9 failures, so its posterior is Beta(8, 10) with mean 8/18 ≈ 0.444. The posterior mean difference is therefore:

E[δ \mid data] ≈ 11/18 − 8/18 = 3/18 = +16.7 percentage points

The posterior is wider than the point estimate because 32 binary observations are not much evidence. A normal approximation to the difference has a standard deviation of about 0.16 here, so the probability of exceeding a +10 percentage-point launch bar is roughly 0.66. The exact interactive uses deterministic posterior draws for a reproducible visual; a production analysis should use a well-checked posterior computation and inspect diagnostics.

The sentence “there is a 66% probability that the true lift exceeds 10 points” is valid only as shorthand for the specified posterior probability: conditional on the Beta-Binomial model, the prior, the data, and the decision threshold. A 95% credible interval is not a 95% confidence interval with friendlier grammar. A confidence interval has a long-run coverage interpretation for a procedure; a credible interval assigns posterior probability under a model.

5. Worked product scenario: choose the method after defining the decision

A learning product tests a new onboarding checklist. The product manager asks, “Can we ship it?” The strongest answer does not start with a p-value. It starts by defining the estimand: the difference in seven-day retention for all eligible users if assigned the checklist versus if assigned the old flow, under a fixed exposure window. The independent unit is the user if assignment is user-level. The practical threshold is not necessarily zero; suppose the team needs at least +1 percentage point to justify engineering and support costs.

Suppose the analysis reports a +0.6 percentage-point lift, a bootstrap 95% interval of [−0.2, +1.4] points, a randomization p-value of 0.08, and an 18% posterior probability that the lift exceeds the +1-point bar. These outputs are not contradictory. They all say the data are compatible with a small positive lift, but the evidence for clearing the business threshold is weak.

My recommendation would be a cautious ramp or more data, not an unconditional launch and not a claim that the checklist “does nothing.” The interval includes a small harm and effects above the practical bar. The p-value says the observed statistic is not especially unusual under the sharp no-effect randomization distribution at a conventional cutoff; it does not say the null has an 8% probability. The posterior says only 18% of posterior effect draws clear the pre-specified bar under the chosen prior and model. The action depends on reversibility, guardrails, traffic cost, and how much harm a false launch would cause.

Before deciding, I would audit exposure and logging, sample-ratio mismatch, missing retention, repeated users, interference, and optional stopping. If users share a household or sellers influence one another, I would resample and analyze at that cluster or network-relevant unit. If the result was checked every hour until it crossed a threshold, I would use a sequential design or correct the analysis plan; simulating more ordinary bootstrap or permutation draws after peeking does not repair calibration. I would also report retention quality or support load as guardrails, because a small average gain can hide a harmful slice.

6. Failure modes: the distribution inherits the design

Wrong unit. Ten sessions from one user are not ten independent treatment assignments. If the experiment assigned users, resample users and keep their sessions together. If it assigned clusters, resample clusters. In the lab, switch from “user” to “cluster of two” and watch the distribution widen because fewer blocks carry the evidence. That widening is not a penalty; it is a more honest statement about dependence.

Post-treatment selection. Toggling “analyze only post-treatment clickers” can make the effect look cleaner. It also changes the population after treatment and can condition on a mediator or collider. A bootstrap of those selected rows faithfully describes the wrong estimand. Define intention-to-treat first; if the question is about treatment received or a mechanism, name the additional assumptions and estimand.

Peeking and multiple looks. Stopping at the first p < .05 changes the sampling rule. The usual p-value assumes a planned analysis or an appropriate sequential method, not a search through time. The lab’s warning state deliberately makes the tail look more favorable so you can practice saying what changed. The correction is a pre-declared stopping rule, alpha spending, sequential monitoring, or a holdout—not a larger simulation count.

Prior sensitivity and computation. A strong prior can move a small-sample posterior substantially. Check whether prior predictive data look plausible before updating, then compare reasonable prior choices. In a larger Bayesian model, inspect posterior and posterior-predictive checks, effective sample sizes, divergences, and convergence. A posterior probability from a broken sampler is not a trustworthy decision quantity.

Metric shape and missingness. Ratios, maxima, quantiles, heavy tails, censored outcomes, and incomplete follow-up need extra care. Resample the independent unit and recompute the full metric; do not average row-level ratios unless that is the estimand. For incomplete 30-day retention, decide whether the target is a fixed-horizon outcome, a survival quantity, or a prediction under censoring. Resampling observed survivors alone silently changes the population.

7. Nearest alternatives and the decision boundary

An analytic t, score, or model-based interval is the nearest alternative to a bootstrap. Use it when the statistic and standard-error approximation are credible, the design is simple, and speed or a transparent audit calculation matters. Bootstrap is attractive when the statistic is nonlinear, skewed, or awkward to derive, but it still depends on the observed sample and the correct unit. It does not make biased sampling representative.

A classical randomization or permutation test is close to a model-free test when the randomization or exchangeability assumptions are credible. It is usually the cleanest answer for “could this assignment gap arise under no effect?” A bootstrap is usually the cleanest answer for “how variable is my observed estimator under repeated samples?” If the assignment was not randomized, a permutation requires a defensible exchangeability story; it is not a universal replacement for confounding control.

Bayesian updating is preferable when prior information is substantive, hierarchical pooling is useful, or a decision naturally needs a probability about a threshold. Frequentist intervals are preferable when long-run calibration and a pre-specified procedure are the communication priority. In either case, state the model, target population, unit, threshold, and sensitivity. “Bayesian” does not mean assumption-free, and “frequentist” does not mean the result is automatically objective.

8. A common interview trap

Checkpoints

Interview practice bank

Timed synthesis: ten minutes

Incomplete: Chooses only by whether p < .05, misstates the p-value or credible interval, or ignores unit, threshold, and guardrails.

Competent: States the user-level estimand and assignment, interprets the interval, p-value, and posterior threshold probability correctly, and recommends more data or a cautious ramp based on cost and reversibility.

Strong: Also audits exposure, missingness, peeking, interference, and cluster structure; separates positive direction from evidence above the practical bar; checks prior sensitivity; and proposes a predeclared next action.

Interview-ready recap

• Define the estimand, population, independent or randomized unit, null, practical threshold, and stopping rule before simulating.
• Bootstrap: resample observed independent units with replacement, preserving treatment structure, then recompute the estimator for uncertainty.
• Permutation/randomization: hold outcomes fixed and rearrange only exchangeable or design-allowed assignments for a null comparison.
• Bayesian: prior × likelihood → posterior; report posterior probabilities about a threshold only with the prior, model, and sensitivity visible.
• A confidence interval is not a credible interval. A p-value is not the probability the null is true. More draws do not repair dependence, selection, peeking, interference, or a wrong estimand.
• Finish with the effect size, uncertainty, practical bar, limitations, guardrails, and next decision—not with a method name.

Sources and scope

Technical claims here follow Efron’s bootstrap paper, the NIST bootstrap reference, SciPy’s maintained bootstrap and permutation test documentation, and the Stan guidance on predictive checks. The ASA p-value statement supports the inferential language. The CZI 2025 interview guide calibrates emphasis on statistical reasoning and product communication; it is not a universal interview format. This lesson does not derive BCa intervals, cluster-bootstrap theory, sequential-testing corrections, or MCMC algorithms.