Confidence intervals, tests, and p-values

Objective

Given a point estimate, standard error, confidence interval, or hypothesis-test output, define the estimand and comparison; compute and interpret a test statistic and p-value; distinguish statistical evidence from effect size and practical significance; check the design and assumptions that make the inference credible; and finish with a bounded decision. In an interview, use this order: target → estimate and uncertainty → null and alternative → evidence → practical threshold → design check → action.

1. Name the estimand before reading the p-value

An estimand is the exact population quantity an analysis is trying to learn. Name the target population, unit, treatment or comparison, outcome definition, and time window. “Conversion improved” is incomplete: improved for which eligible users, after what exposure, measured over how many days, and compared with what counterfactual?

The sample is the observed data; the parameter is the fixed but unknown quantity; the estimator is the rule that would vary across repeated samples; and the estimate is the value produced by this sample. A standard error is the estimated standard deviation of that estimator across repeated samples. It is not the standard deviation of individual observations. This vocabulary keeps the uncertainty statement attached to the right random object.

For a one-sample mean with unknown population standard deviation, the estimate is x̄, sample standard deviation is s, sample size is n, and the standard error is s/√n. The usual two-sided interval is:

CI = x̄ ± t* · s/√n = estimate ± critical value × standard error

The critical value depends on the desired confidence level and degrees of freedom. When s estimates an unknown σ and the sample is small, Student’s t reference has heavier tails than a standard Normal reference. As degrees of freedom grow, the t critical value approaches the familiar 1.96 for a two-sided 95% interval.

2. What a confidence interval means

A 95% confidence interval is a property of a procedure. Imagine repeating the same sampling-and-interval procedure many times from the same target process. Under the assumptions, about 95% of the intervals would cover the fixed parameter. The parameter is not changing from repetition to repetition; the sample, estimate, and interval are.

For the interval you actually computed, it is useful to say: “The data and procedure are compatible with population means from the lower endpoint to the upper endpoint.” That is a decision-oriented description of the range, not a probability assignment to a fixed parameter. In a frequentist analysis, avoid saying “there is a 95% chance that the true mean is in this particular interval.” That sentence is tempting because uncertainty feels probabilistic, but it reverses the repeated-sampling calibration.

Confidence level, sample size, and noise have visible consequences:

• Higher confidence widens the interval. To cover more often, the procedure must use a larger critical value.
• Larger n narrows it roughly as 1/√n. Quadrupling the number of comparable independent observations roughly halves the standard error and margin.
• More observation noise widens it. More spread in individual measurements makes the mean less precise at the same sample size.
• Coverage is not the same as observation spread. An interval estimates a parameter; it is not meant to contain 95% of future individual observations. A prediction interval answers a different question.

3. From a null claim to a test statistic

A hypothesis test starts with a reference claim, not with a result. For a mean, write H₀: μ = μ₀. Choose the alternative before looking at the direction: Hₐ: μ ≠ μ₀ for a two-sided question, or a directional alternative if only one direction would change the decision. Choose a significance level α, often 0.05, as a tolerance for false rejection under the planned procedure.

The test statistic measures the observed distance from the null in standard-error units:

t = (x̄ − μ₀) / SE(x̄) = (x̄ − μ₀) / (s/√n)

The numerator is the observed difference from the null. The denominator expresses that difference in noise units. A large absolute statistic can come from a large effect, a small standard error, or both. For a two-sided test, the p-value is the probability—assuming H₀ and the test assumptions—that a reference statistic would be at least as extreme as the observed one:

p = P(|T_n−1| ≥ |t_obs| | H₀)

That conditional direction matters. A p-value is not the probability that the null is true, the probability the result will replicate, the effect size, or the probability the result is “due to chance.” A small p-value is evidence that the observed statistic is unusual under the stated null and reference model. A large p-value means the data did not provide strong evidence against that null; it does not prove no effect or validate the null.

When the test and interval use the same estimate, standard error, reference distribution, and two-sided convention, there is a useful correspondence: a level-α test rejects H₀: θ = θ₀ exactly when θ₀ falls outside the corresponding (1 − α) confidence interval. The interval is usually more informative because it shows direction and plausible magnitude rather than only a reject/fail-to-reject label.

4. Worked calculation: eight ticket wait times

Question. A support lead samples eight ticket wait times, in minutes: 8, 10, 11, 12, 13, 14, 16, 16. What can we infer about the population mean, and is it different from 10 minutes?

Step 1 — compute the estimate. The sum is 100, so x̄ = 100/8 = 12.5 minutes. The deviations from 12.5 are −4.5, −2.5, −1.5, −0.5, 0.5, 1.5, 3.5, 3.5. Squaring and adding gives 56.

The interval says that, under the sampling and measurement assumptions, the compatible range for the population mean is about 10.14 to 14.87 minutes. It does not say each ticket waits inside this range, nor that the fixed mean has a 95% probability of being there.

Step 2 — test the null. Let H₀: μ = 10 and Hₐ: μ ≠ 10. The observed statistic is t = (12.5 − 10)/1.000 = 2.500 with 7 degrees of freedom. The two-sided p-value is approximately 0.041. At α = .05, reject the null under this planned test. This matches the interval: 10 is just below the lower endpoint.

Step 3 — do not skip the decision threshold. Suppose the operational goal is a mean wait of no more than 11 minutes. The estimate is above that goal, but the interval includes values below and above 11. The test against 10 being significant does not certify that the process misses the 11-minute target. The analysis is inconclusive for that practical decision. More data, a pre-specified equivalence or non-inferiority analysis, or a process intervention may be appropriate.

Say the assumptions aloud: the eight tickets should represent the target ticket population; observations should be independent or analyzed with the right cluster/time structure; the small-sample t approximation should not be badly distorted by skew or outliers; and the wait-time measurement should be consistent. If the lead sampled only quiet hours, a small p-value cannot repair selection bias.

5. Worked scenario: an onboarding checklist and a launch bar

Question. A product team randomizes eligible new users to a control or an onboarding checklist. Activation is measured within 7 days. Treatment activates at 25.8%, control at 24.0%, so the estimated lift is +1.8 percentage points. The predeclared analysis reports a 95% CI of [+0.4, +3.2] points and a two-sided p = .012. The team’s minimum worthwhile lift is 1 point.

Frame the estimand first. One unit is one eligible randomized user. The intention-to-treat estimand is the difference in activation by day 7 under assignment to checklist versus assignment to control. It includes noncompliance and users who did not fully engage. The comparison is treatment minus control, in percentage points.

Interpret the evidence. The interval excludes zero and the p-value is below .05, so the planned analysis provides evidence against no average activation difference. Plausible lift ranges from 0.4 to 3.2 points. That is more useful than “significant,” but it does not establish that the lift clears the 1-point launch threshold: part of the interval is below 1.

Make a bounded decision. Check sample-ratio mismatch, assignment and exposure logging, missing outcomes, interference between users, experiment duration and novelty, repeated looks, and guardrails such as support contacts, retention, and revenue. If the checks pass and rollout is cheap to reverse, a cautious ramp with monitoring may be defensible. If a 1-point lift is a hard requirement, say “positive evidence, but not enough precision to certify the threshold,” then extend the preplanned experiment or use a decision analysis.

Show why design beats decimal precision. Suppose the same interval and p-value came from users who opted into the checklist, with a regression adjustment for observed device and country. That result could be precise for an adjusted association, but self-selection and unmeasured motivation could explain it. The conclusion would be “the observed groups differ after these adjustments,” not “the checklist caused a 1.8-point lift.” Precision quantifies uncertainty conditional on a design; it does not create causal identification.

Finally, if the team examined 40 metrics and subgroups and reported the one with p = .012, the nominal p-value understates the search process. Pre-specify the primary metric and account for multiplicity or label the finding exploratory. This lesson flags that workflow issue; power, multiple testing, and sequential monitoring are the next lesson’s deeper topics.

6. Assumptions, failure modes, and the nearest alternative

Interval validity is a property of the estimand, sampling or randomization design, standard-error calculation, and reference approximation together. A tidy output table cannot compensate for a wrong unit or a data-dependent analysis history.

• Small, skewed, or heavy-tailed data: a symmetric one-sample t interval can have distorted coverage at small n. Inspect the data-generating process and outliers; consider a larger sample, a robust or resampling method, or a better-defined estimand. Do not delete an inconvenient value just to restore symmetry.
• Dependence: repeated rows from one user, household, store, or time series are not independent just because the table has many rows. Match the standard error or resampling unit to the randomization and dependence structure.
• Selection and missingness: an interval can be narrow for a selected subset or observed outcomes while saying little about the target population. Audit eligibility, follow-up, opt-outs, duplicates, and whether missingness depends on treatment or outcome.
• Optional stopping and multiplicity: daily peeking, many subgroup cuts, and many outcomes turn a nominal 5% procedure into a different procedure. Reconstruct the analysis history; use a precommitted stopping rule, sequential method, multiplicity adjustment, or treat the result as exploratory.
• Association versus causation: a small p-value around an observational regression coefficient does not establish a causal effect. Randomization or a defensible identification strategy is required for causal language.

The nearest technical alternatives are bootstrap intervals, permutation or randomization tests, and Bayesian credible intervals. Use the classical t procedure when the estimand is simple, the design and standard error are credible, and a transparent analytic answer helps. Use bootstrap when the statistic has a difficult sampling distribution and the resampling unit can reproduce the design. Use permutation or randomization logic when treatment labels are exchangeable under a sharp null or were randomized. A Bayesian interval supports a different probability statement conditional on a likelihood and prior. None of these repairs selection bias, dependence ignored by the method, bad measurement, distribution shift, optional stopping, or a wrong estimand.

Also compare against the right null. Zero asks whether there is evidence of a difference. A practical threshold δ* asks whether the difference is large enough to matter. An interval such as [0.4, 3.2] can exclude zero while still failing to establish a 1-point minimum. The threshold is part of the decision, not a post-hoc adjective.

7. Common interview trap

8. Three checkpoint disclosures

1. What does a 95% confidence interval mean?

   Under repeated sampling and valid assumptions, about 95% of intervals produced by that procedure would cover the fixed parameter. It is not 95% of individual observations and not a frequentist 95% probability statement about this fixed parameter after the interval is observed.

2. What does p = .60 tell you?

   Under the stated null and assumptions, the data are not unusual enough to reject the null at common thresholds. It does not prove no effect or mean the null has a 60% probability. Inspect the interval to see which effect sizes remain plausible and whether the study was precise enough for the decision.

3. Why can an interval exclude zero but still fail a launch decision?

   Zero tests whether there is evidence of any difference. A launch threshold tests whether the effect is large enough to matter. If the interval is [0.4, 3.2] and the minimum worthwhile lift is 1 point, the direction is positive but the threshold is not certified.

9. Interview practice bank

1. 1. What does a 95% confidence interval mean?

   Strong answer: Define the parameter and procedure, then give the repeated-sampling coverage statement. For the observed interval, describe compatible effect sizes without assigning a frequentist probability to a fixed parameter. Likely follow-ups: What makes it wider? Does it contain 95% of observations? Signal: separates parameter uncertainty, observation variability, and procedure-level coverage.

2. 2. Given x̄ = 12.5, s = 2.828, n = 8, and t* = 2.365, compute the interval and test μ₀ = 10.

   Strong answer: SE = 2.828/√8 = 1; CI is 12.5 ± 2.365 = [10.135, 14.865]; t = 2.5 with 7 degrees of freedom; two-sided p ≈ .041. Reject at .05, then state assumptions and practical limits. Likely follow-ups: Why t rather than 1.96? What changes for a one-sided alternative? Signal: auditable arithmetic and correct degrees of freedom.

3. 3. A treatment effect is 1.8 points, CI [0.4, 3.2], p = .012. What do you tell the PM?

   Strong answer: State the randomized estimand and unit; report evidence against zero and the plausible lift range; compare that range with the minimum worthwhile lift, costs, and guardrails; recommend a ramp, more data, or hold according to the predeclared decision rule. Likely follow-up: Does it definitely meet a 1-point target? No—the interval includes values below 1. Signal: converts inference into a calibrated product decision.

4. 4. What is a p-value of .03?

   Strong answer: Under the specified null and assumptions, the chance of a test statistic at least as extreme as observed is .03. It is not the probability the null is true, the effect size, or replication probability. Likely follow-ups: What does .60 mean? Can you accept the null? Use “fail to reject” and inspect precision. Signal: gets the conditional direction right.

5. 5. Your significant experiment had daily peeking and 20 subgroup cuts. What is wrong?

   Strong answer: The nominal Type I error applies to the planned procedure, not unlimited looks and searches. Reconstruct the history, identify the primary estimand, adjust with a preplanned sequential or multiplicity method, or label the finding exploratory and seek replication. Likely follow-up: Can you choose a smaller alpha after seeing the result? Not as a clean repair. Signal: treats inference as a workflow property.

6. 6. When is a one-sample t interval questionable?

   Strong answer: Severe skew, heavy tails, or outliers at small n can damage the reference approximation; dependence, selection, nonrepresentative sampling, and bad standard errors damage coverage at any n. Consider a robust or resampling method only after matching it to the design. Likely follow-ups: Does the CLT fix everything? Would you delete an outlier? Signal: prioritizes design and avoids ritual assumption checking.

7. 7. Compare a classical interval, bootstrap interval, and Bayesian credible interval.

   Strong answer: Classical coverage is calibrated over repeated samples; bootstrap approximates a sampling distribution and depends on a correct resampling unit; a Bayesian credible interval is a posterior probability statement conditional on a likelihood and prior. Choose by estimand, design, statistic, sample size, and stakeholder interpretation. Likely follow-ups: Can bootstrap solve clustered data? When would a prior matter? Signal: explains alternatives without calling one assumption-free.

8. 8. A model metric rises from 0.71 to 0.73. Is that an improvement?

   Strong answer: Define the metric, deployment population, split, and paired or independent unit. Estimate uncertainty with a dependence-aware interval or paired resampling; check that the test set is representative and untouched; compare with a practical threshold, calibration, slices, cost, and drift. Likely follow-up: What if rows share users or time? Use the user/group/time unit, not naive rows. Signal: transfers inference to ML evaluation without blind t-testing.

10. Timed synthesis: eight minutes

Recap: the interview-ready cheat sheet

Technical framing follows the NIST/SEMATECH confidence-interval reference, its CI/test correspondence, and its critical-value and p-value guidance; assumptions and construction are cross-checked with Penn State STAT 500 on confidence intervals and hypothesis testing. The distinction between statistical significance, effect size, and practical meaning follows the American Statistical Association p-value statement. Interview emphasis is calibrated with the Chan Zuckerberg Initiative 2025 guide, an employer example rather than a universal interview loop.