Power is a design decision, not a magic number

Objective

Given a proposed test or experiment, define the smallest effect that would change the decision, explain power as a conditional probability, sanity-check a sample-size calculation, and diagnose how noise, clustering, attrition, multiple hypotheses, and repeated looks change the evidence. In an interview, finish with a bounded recommendation: what the design can establish, what it cannot, and what you would do next.

1. Power asks a conditional question

Suppose the null hypothesis is H₀: θ = θ₀. A test has a rejection rule: after seeing the data, it rejects when the statistic lands in a pre-specified region. The significance level α is the probability of rejection when the null is true, assuming the stated procedure. A Type II error at a particular alternative θ₁ is:

β(θ₁) = P(fail to reject H₀ ∣ θ = θ₁)

Power is the complement:

Power(θ₁) = 1 − β(θ₁) = P(reject H₀ ∣ θ = θ₁)

The conditioning matters. “80% power” is incomplete until you name the effect, noise, sample size, alpha, outcome, analysis, and design. It means that if the true effect equals the planning alternative and the assumptions hold, repeated studies using that procedure would reject about 80% of the time. It is not an 80% probability that the alternative is true, an 80% guarantee that this study will be significant, or a posterior probability about the hypothesis.

Planning usually begins with a minimum effect of interest (often called an MDE). This is not a universal statistical threshold. It should come from the product value, scientific relevance, cost, safety margin, or non-inferiority boundary: the smallest effect that would actually change the decision. A study can be well powered for a 5-point lift and poorly powered for a 1-point lift. A non-significant result then says little about the smaller effect unless the interval is narrow enough to rule it out.

2. Why the sample-size formula works

Consider two equally sized, independent arms. Each arm has n observations, common standard deviation σ, and true mean difference δ = μ_T − μ_C. The difference in sample means has:

SE(x̄_T − x̄_C) = √(σ²/n + σ²/n) = σ√(2/n)

Under the null, the standardized statistic is centered at zero. Under the alternative, its center is shifted by:

signal = δ / [σ√(2/n)] = (δ/σ)√(n/2)

That shift is the distance between the null and alternative distributions in standard-error units. Larger effects move the alternative farther away. Less noise does the same. More independent units move it by the square root of n, not by n itself. For a two-sided alpha of .05, the null rejection cutoff is about 1.96. For 80% power, the alternative center needs roughly another 0.84 standard deviations beyond that cutoff:

n_{per arm} ≈ 2(1.96 + 0.84)²(σ/δ)²

So, holding everything else fixed, doubling the effect quarters the required independent-unit count; doubling the noise quadruples it. Lowering alpha pushes the rejection boundary outward and requires more information. Raising the desired power moves the alternative farther still. This z calculation is a transparent planning approximation. Unknown variance, binary outcomes, unequal allocation, finite samples, clusters, ratio metrics, missingness, and sequential rules may require a t or noncentral-t calculation, a binomial method, or simulation that reproduces the full design.

Worked calculation: from a useful effect to recruitment

Question. A product team compares two independent groups on a satisfaction score. A 4-point difference is the smallest improvement worth shipping. Historical within-arm standard deviation is 10 points. The team wants a two-sided α=.05 test with 80% power. How many independent users per arm are needed before attrition?

Step 1 — name the inputs. The practical effect is δ=4 points, noise is σ=10 points, the two-sided critical value is z_.975≈1.96, and the 80% power quantile is z_.80≈.84.

Step 2 — substitute.

n_{per arm} ≈ 2[(1.96 + .84)(10/4)]² = 2(2.80 × 2.50)² = 2(7.00)² = 98

Under this approximation, 98 independent users per arm gives about 80% rejection probability if the true difference is really 4 points, the users are independent, the analysis is fixed-horizon, and no additional confirmatory family changes alpha. It does not say the estimate will be exactly 4 points, that the comparison is causal without random assignment, or that effects smaller than 4 are reliably detectable.

Step 3 — add five co-equal confirmatory endpoints. If any one of five endpoints can authorize launch and the team chooses Bonferroni FWER control, the per-claim alpha becomes .05/5=.01. The two-sided critical value is now about 2.576:

n_{per arm} ≈ 2[(2.576 + .84)(10/4)]² ≈ 146

With 10% missing outcomes or non-exposure, recruit 146/.90=162.2, rounding up to 163 users per arm. This is not a blanket argument for Bonferroni. If there is one primary endpoint and descriptive guardrails, the family of confirmatory claims may be smaller. If the purpose is broad exploration, FDR may be a better fit. The point is that the error-control rule changes the power calculation and must be chosen before seeing the results.

Sanity checks expose the scaling. If the MDE falls from 4 to 2 points, the independent-unit requirement becomes roughly four times larger. If noise rises from 10 to 20 points, it also becomes roughly four times larger. If the “observations” are five events per user with intraclass correlation ρ=.20, the rough design effect is 1+(5−1)(.20)=1.8; 490 event rows are not equivalent to 490 independent users. And if the final p-value is .20, that does not prove the effect is absent: inspect the interval against the 4-point decision threshold.

3. Multiple testing: name the family before naming the winner

Every additional hypothesis, endpoint, subgroup, arm comparison, model variant, or interim look creates another opportunity to cross a rejection boundary. If all m null hypotheses are true, each test uses alpha α, and the tests are independent, the probability of at least one false rejection is:

FWER = P(V ≥ 1) = 1 − (1 − α)^m

For 20 tests at .05, this is 1−.95²⁰≈.642. Under an all-null world, about 64% of such families produce at least one nominally significant result under the independence calculation. The exact number depends on dependence, but correlation does not make the family concept disappear. The general interview move is to ask: which claims will authorize an action, and what error matters for that family?

• Bonferroni and Holm control FWER. Bonferroni compares each p-value with α/m; it is simple and valid under arbitrary dependence but can be conservative. Holm sorts p-values and uses step-down thresholds, retaining the same FWER guarantee while being at least as powerful as ordinary Bonferroni. A primary launch claim with a small number of high-consequence endpoints is a plausible use.
• Hierarchy and gatekeeping spend error according to priority. Test a primary claim first, then unlock pre-specified secondary claims, or assign different roles to a primary metric, guardrails, and descriptive diagnostics. A dashboard can contain many measurements; not every measurement has to be a confirmatory launch claim. It does have to be labeled honestly.
• Benjamini–Hochberg targets FDR. FDR is E[V/max(R,1)], the expected false proportion among discoveries. BH can preserve more discoveries for a broad exploratory screen under its dependence conditions. It does not guarantee that 5% of the findings in this one dataset are false, and it is not automatically appropriate for a single irreversible launch decision.

“Use Bonferroni everywhere” and “use FDR everywhere” are both weak answers. Pick the least conservative valid procedure that matches the decision: avoid even one false claim when that is the consequence, or tolerate a controlled expected fraction of false discoveries when discovery is the goal. Plan power for the adjusted procedure. Do not calculate 80% power for every unadjusted p-value and add a correction after the fact.

4. Repeated looks are testing over time

A fixed-horizon p-value assumes one planned final analysis at one planned sample size. If an analyst checks the dashboard every day and stops when p<.05, the stopping rule used information from earlier looks. Printing only the last p-value does not erase those opportunities. The same logic applies to trying many outcomes, segments, exclusions, transformations, and model variants and reporting only the favorable path.

Early stopping can be valuable and ethical, especially when a large benefit or safety problem should change exposure quickly. The defensible alternatives are a fixed final analysis, a group-sequential design with pre-specified efficacy and futility boundaries, an alpha-spending plan such as Lan–DeMets when timing can vary, or a simulation of the exact adaptive procedure. A sequential design is not “peeking with permission”; it is a different calibrated procedure with its own power, stopping probabilities, and interval interpretation. “We only peeked for safety” is not a general exemption: safety and efficacy decisions need separate rules.

Worked product scenario: onboarding activation

Question. An onboarding team wants to increase 7-day activation. Baseline activation is 25%, and the PM says a lift smaller than 2 percentage points will not repay engineering and support costs. Users are randomized 1:1, every eligible user has a fixed seven-day outcome window, and the team wants two-sided alpha .05 with 80% power. It also wants to watch invitations, revenue, and four country segments.

Frame the estimand and unit. The primary estimand is the intention-to-treat difference in user-level 7-day activation between assignment to the new flow and assignment to control, among eligible new users during the experiment window. The randomization unit is the user, even if activation produces several event rows. Exposure and logging checks protect interpretation; they do not replace randomization.

Do a first-pass binary calculation. For a rough equal-arm proportion calculation, the planning average is p̄=(.25+.27)/2=.26, so p̄(1−p̄)=.26(.74)=.1924. With δ=.02:

n ≈ 2(.1924)(1.96+.84)²/.02² ≈ 7,543 per arm

If 10% of randomized users are not exposed or do not complete the outcome window, a first inflation is 7,543/.90≈8,382, or about 8,382 eligible users per arm. A final calculation should use the chosen proportion test, anticipated allocation, missingness mechanism, calendar effects, and any covariate adjustment. The exact method can change the decimal; it cannot rescue a vague estimand or the wrong unit.

Decide how the extra claims work. Four countries do not automatically deserve four confirmatory launch claims. Each country receives less traffic, and an overall 2-point MDE may be undetectable within a country. Before launch, define whether country is a pre-specified heterogeneity or safety question, whether revenue is a key secondary claim, and whether invitations are a mechanism or descriptive metric. A guardrail can use a pre-specified harm margin rather than a symmetric “p < .05” win rule. If the team wants to screen a broad scorecard, BH/FDR can support discovery, but the findings should be labeled exploratory and replicated before becoming launch proof.

Edge case: a favorable country after daily checking. On day five, one country has p=.03. The team has checked every morning and wants to stop. This is not yet clean evidence of a country-specific win. The search crossed time and subgroup dimensions, while the original fixed-horizon p-value describes neither. Reconstruct the pre-specification and search history. Continue to the fixed endpoint, reanalyze under a valid sequential/multiplicity procedure, or label the result exploratory and run a fresh pre-specified test. Do not promote a discovered subgroup to the primary result because it is the only green cell.

Bound the decision. If the planned primary result is positive, its interval is above zero and the 2-point threshold is crossed, exposure and data-quality checks pass, and guardrails are acceptable, a staged rollout may be reasonable. If the interval crosses zero or the 2-point threshold, do not claim a proven worthwhile lift. Consider more pre-planned data, a lower-risk ramp with monitoring, a revised product hypothesis, or stopping. If only one country is positive after unplanned searching, do not generalize it to all users.

5. Assumptions, failure modes, and the nearest alternatives

The arithmetic only answers the design you actually specified. Name the assumptions at the point where they matter:

• Independent information. Ten events from one user are not ten independent users. With equal cluster size m and intraclass correlation ρ, a rough design effect is 1+(m−1)ρ. Unequal clusters, time dependence, covariate adjustment, and interference may need a better variance model or simulation.
• Stable outcome definition and follow-up. A conversion rate needs the right denominator and a complete, comparable outcome window. Selective missingness can reduce precision and change the estimand. “Ten thousand eligible users” is not ten thousand analyzable outcomes if only 80% are exposed and 90% are observed.
• Pre-specified direction and analysis. A one-sided test can have more power in its chosen direction, but direction must be justified before outcomes are seen and an opposite effect must not lead to the same action. If both directions matter, use two-sided inference.
• Practical threshold. A small p-value can accompany a trivial effect in a huge sample. A non-significant p-value can coexist with a useful effect in a small or noisy sample. Report the estimate and interval relative to the decision threshold.
• Multiplicity and workflow. The family includes the claims and looks that influenced the decision, not only the columns in the final table. Keep an analysis history and state what was planned, exploratory, or discovered after the fact.

The nearest plausible alternative to binary power planning is precision planning. Use power when the question is “Can this design detect at least δ with probability 1−β?” Use precision planning when the question is “Can the interval be narrow enough to distinguish the effects that matter?” Precision is often the better framing when no single null-versus-alternative pair represents the decision. Do not use post-hoc observed power as a substitute for the interval; it mostly repackages the observed estimate and p-value.

Use a closed-form z, t, or binomial calculation for a simple transparent design. Use simulation when there are clusters, repeated measures, interference, ratio metrics, correlated endpoints, adaptive allocation, complex missingness, or a nonstandard stopping rule. The simulation must reproduce the randomization, data-generating assumptions, analysis, and decision rule. Simulation can quantify operating characteristics; it cannot repair biased instrumentation or an ill-defined estimand.

6. Common interview trap

7. Three checkpoint disclosures

1. What does 80% power mean?

   At a specified true effect, under the planned test and assumptions, repeated studies would reject the null about 80% of the time. It is not an 80% probability that the alternative is true, and it is not a guarantee for every future study.

2. Why does halving the MDE require roughly four times the sample?

   For the simple equal-arm mean comparison, n is proportional to (σ/δ)². Replacing δ with δ/2 multiplies (σ/δ)² by four.

3. When is FDR a better fit than FWER?

   FWER is appropriate when even one false claim in a small confirmatory family is costly. FDR can be useful for broad exploratory screening when a controlled expected proportion of false discoveries is acceptable. Neither is a universal correction; define the family and decision first.

8. Interview practice bank

1. 1. Define statistical power without saying “the probability the hypothesis is true.”

   Strong-answer outline: Power at an alternative is P(reject H₀ ∣ θ is that alternative), or 1−β(θ). It depends on effect size, noise, independent sample size, alpha, the test, and the design. Name the target effect before quoting a power value. Likely follow-up: What does 80% power mean for half the MDE? Signal: gets the conditioning direction right.

2. 2. Derive the rough sample size for two equal independent mean comparisons.

   Strong-answer outline: Start with SE(diff)=σ√(2/n). Under the alternative the standardized shift is (δ/σ)√(n/2). Set it approximately to z_1−α/2+z_1−β and solve n≈2(z_1−α/2+z_1−β)²(σ/δ)² per arm. Likely follow-up: What changes for binary outcomes or clusters? Signal: shows mechanism and boundary conditions.

3. 3. A study powered for a 5% lift reports p = .20. What can you conclude?

   Strong-answer outline: It did not reject the stated null at the chosen alpha; it does not prove no effect. Report the estimate and interval, compare them with the 5% decision threshold, and inspect exposure, variance, attrition, dependence, and whether the planned effect was realistic. Likely follow-up: Would you report post-hoc power? Signal: separates non-detection from evidence of no practically important effect.

4. 4. Twenty metrics were tested and one has p = .04. What is wrong?

   Strong-answer outline: If any metric can declare success, the family-level false-positive risk exceeds .05; under independent all-null tests it is about 64%. Identify the pre-specified primary family, apply a planned FWER method or a clearly exploratory FDR analysis, and report the search rather than cherry-picking. Likely follow-up: Does correlation remove the problem? Signal: treats multiplicity as a decision-scope issue.

5. 5. Compare FWER and FDR for an experimentation team.

   Strong-answer outline: FWER controls the probability of at least one false rejection and suits a small, high-consequence confirmatory family. FDR controls the expected false proportion among discoveries and can preserve sensitivity for broad exploration. Bonferroni/Holm and BH answer different operating goals; choose based on consequence and dependence. Likely follow-up: Does BH guarantee that exactly 5% of this dataset’s discoveries are false? Signal: understands the error-rate definitions.

6. 6. The analyst checked the dashboard daily and stopped when p < .05. Is the result valid?

   Strong-answer outline: Not under an ordinary fixed-horizon p-value without more design information. Ask whether looks, boundaries, and stopping were pre-specified. Reanalyze under the valid sequential procedure, finish under a defensible fixed rule, or label the result exploratory and replicate. Likely follow-up: How can ethical early stopping be valid? Signal: distinguishes sequential design from unplanned peeking.

7. 7. Plan an A/B test for a 2-point activation lift.

   Strong-answer outline: Define the user-level ITT estimand, seven-day window, baseline rate, practical threshold, alpha, power, and randomization unit. Calculate or simulate independent-user count, inflate for exposure and missingness, pre-specify guardrails, metric family, subgroups, and interim rules, then analyze at the user unit and report lift plus interval. Likely follow-up: What if only 10% of users are exposed? Signal: carries the design from business question to evidence.

8. 8. You have one million event rows. Why might the study still be underpowered?

   Strong-answer outline: Rows may be repeated events from far fewer users and therefore correlated. Check the randomization unit, cluster size and ICC, exposure, missing outcomes, outcome rarity, and interference. Use cluster-aware variance or simulation rather than treating the row count as independent information. Likely follow-up: What does a design effect of 1.8 mean? Signal: recognizes the information unit as a first-order design choice.

9. Timed synthesis: eight minutes

Recap: the interview-ready cheat sheet

Technical definitions and the sample-size relationship follow the NIST/SEMATECH power reference and NIST sample-size guidance. The two-sample power calculation is cross-checked against the statsmodels power interface. FDR follows Benjamini and Hochberg; endpoint-family and sequential-planning boundaries are informed by FDA multiple-endpoint guidance and ICH E9. Product-experiment emphasis is calibrated with Microsoft’s pre-experiment guidance; it is not treated as the authority for statistical definitions. The examples are synthetic. This lesson does not catalog specialized survival, hierarchical Bayesian, mixed-effects, or adaptive-allocation power methods.