Estimators, bias, and maximum likelihood

Objective

Given a sampling problem or a product/modeling scenario, define the target parameter and unit; distinguish an estimator from its realized estimate; compare candidates using bias, variance, mean squared error, and consistency; and derive or explain a maximum-likelihood estimator. In an interview, finish by stating the assumptions, the nearest alternative, and what the number does not establish.

1. Start with the target, not the algorithm

Suppose a population has an unknown parameter θ. A statistic is any function of observed data. An estimator, written T(X₁,…,Xₙ) or θ̂, is a statistic designed to estimate θ. Before the data arrive it is a random variable because the sample is random. Once the sample is observed, its realized value is an estimate. That small distinction matters in interviews: bias and variance describe the estimator across repeated samples, not whether one observed number feels high or low.

For a Bernoulli outcome, the target might be the seven-day retention probability p. If Xᵢ is 0 or 1 records retention for one eligible account, the sample proportion p̂ = (ΣXᵢ)/n is both a natural estimator and, under independent Bernoulli sampling, the MLE. But the formula is not the whole answer. The denominator says who is eligible, the unit says which observations are independent, and the outcome window says what “retained” means.

2. Compare estimators by repeated-sample behavior

Bias is the signed average displacement from the target:

Bias_θ(θ̂) = E_θ[θ̂] − θ

An estimator can be unbiased even when an individual estimate misses high or low; the average is the claim. Variance measures how much estimates spread around their own mean:

Var_θ(θ̂) = E_θ[(θ̂ − E_θ[θ̂])²]

Squared error combines both:

MSE_θ(θ̂) = Var_θ(θ̂) + Bias_θ(θ̂)²

The identity follows by adding and subtracting E[θ̂] inside (θ̂−θ)². The cross-term disappears because E[θ̂−E[θ̂]]=0. Its practical lesson is not “bias is good” or “variance is bad.” It is: name the loss and decision. A small deliberate shrinkage can lower total squared error, while an unbiased estimate can be preferable when systematic calibration matters more than squared error.

Consistency asks what happens as the sample grows: for every ε > 0, does P(|θ̂ₙ−θ| > ε) → 0? Unbiasedness is a finite-sample expectation statement; consistency is a large-sample concentration statement. Neither automatically implies the other without additional control. The Normal variance MLE below is biased at every finite n but consistent.

3. Likelihood turns a model into a comparison

Probability starts with a parameter and asks how likely data are. Likelihood fixes the observed data and treats the parameter as the argument:

L(θ; x) = ∏_i=1ⁿ f(xᵢ; θ)

The maximum-likelihood estimator is the parameter value that makes the observed sample most compatible with the stated model:

θ̂_MLE = arg max_θ L(θ; x)

For independent observations, the joint density factors into a product. The logarithm is strictly increasing, so maximizing L and maximizing the log-likelihood ℓ = log L produce the same maximizer. The log form is easier to differentiate, turns products into sums, and avoids numerical underflow when many probabilities are multiplied. A parameter-free constant can be dropped for optimization, though it still belongs in the actual likelihood value.

For Xᵢ ~ Bernoulli(p), let y = Σxᵢ. Then:

L(p) = p^y(1−p)^n−y;   ℓ(p) = y log p + (n−y) log(1−p)

For an interior solution, differentiate and set the score to zero:

y/p − (n−y)/(1−p) = 0 → y(1−p) = (n−y)p → p̂ = y/n

The second derivative is −y/p² − (n−y)/(1−p)², which is negative when 0 < y < n. If y=0, the maximum is at the boundary p̂=0; if y=n, it is at p̂=1. The derivative calculation alone does not cover those edge cases.

4. Worked calculation: two Normal variance estimators

Question. Four independent measurements are 2, 4, 5, 7, modeled as N(μ, σ²). Compute the Normal MLEs and compare the variance estimate with the usual unbiased estimator.

Step 1 — mean. x̄=(2+4+5+7)/4=18/4=4.5. The deviations are −2.5, −0.5, 0.5, 2.5, and their squared deviations are 6.25, 0.25, 0.25, 6.25. The sum is 13.

Why does the MLE use n? The Normal log-likelihood is −(n/2)log(2πσ²) − [Σ(xᵢ−μ)²]/(2σ²). Maximizing first over μ gives μ̂=x̄; differentiating over σ² then gives the average squared deviation with denominator n. Under the Normal model with unknown mean, E[σ̂²_MLE] = (n−1)σ²/n, so it is biased downward. The n−1 version is unbiased, but it is not the Normal MLE for this same problem.

For this model, the MLE’s bias is −σ²/n and its variance is 2(n−1)σ⁴/n². At n=4, its MSE is 7σ⁴/16≈0.438σ⁴; the unbiased estimator’s MSE is about 0.667σ⁴. The biased estimator can have lower MSE because its variance reduction outweighs its squared bias under this loss and model. That is a model-specific comparison, not a universal ranking.

5. Worked scenario: a retention baseline that does not prove lift

Question. A product team wants the baseline seven-day retention rate for 2,000 eligible new accounts before setting a notification threshold. Exactly 236 accounts are retained on day 7.

Frame the estimand. Let one row be one eligible new account, define eligibility before the outcome window, and let p be the probability that this account is retained on day 7 under the observed baseline process. Under an independent Bernoulli model, the MLE is p̂=236/2000=0.118, or 11.8%. The point estimate is easy; the defensibility is the work.

Audit the denominator. Confirm that accounts are unique, eligible, and observed for the same seven-day window. Check missing outcomes from tracking opt-outs, bot or test accounts, delayed events, and whether “new account” changed during the period. If the next month will be 80% returning users but the observed sample was 50% returning users, a pooled rate may not match the deployment population. Estimate stable pre-outcome segment rates and standardize them to the future mix; do not silently treat mix drift as a product effect.

Separate estimation from action. The 11.8% estimate can support a descriptive baseline or a forecast if the target population and sampling process are credible. It does not say a notification caused anyone to retain. For the ship decision, define the causal contrast—retention with notification versus without for comparable eligible accounts—and randomize assignment when feasible, while checking exposure logging, interference, missingness, and the analysis unit.

Bound the recommendation. “The observed baseline is 11.8% among these eligible accounts. Before setting a threshold, I would quantify uncertainty, audit opt-outs and duplicates, and standardize to the expected traffic mix. To claim incremental lift, I would run a randomized comparison.” That answer is stronger than a more precise decimal because it says which population and decision the estimate can support.

6. Assumptions, failure modes, and the nearest alternative

MLE’s useful large-sample story requires more than a converged optimizer. Under regularity conditions—an identifiable and sufficiently smooth model, a true parameter in the interior, and suitable independence—the MLE concentrates near the target and is approximately Normal after scaling. Curvature can motivate standard errors. It does not make a finite-sample MLE unbiased, prove the model is correct, or rescue an invalid sampling unit.

• Boundary solutions: all successes or all failures put the Bernoulli MLE at 1 or 0. A large peak does not imply certainty, especially at small n.
• Dependence: repeated events from the same user are not automatically independent rows. The proportion may still be a useful descriptive statistic, but the simple likelihood and standard error need a cluster or repeated-measures model.
• Selection and missingness: if outcomes are missing for people who opt out of tracking, the observed likelihood may identify only the observed subset. Audit the missingness mechanism and consider weighting or sensitivity analysis when defensible.
• Misspecification: if the assumed family is wrong, MLE converges toward the best-fitting member of that family under a population discrepancy; it is not recovering a literal true parameter that the family cannot represent.
• Optimization and identifiability: non-convergence, separation, weak identification, multiple maxima, or a parameter on a boundary are statistical signals as well as numerical issues. An optimizer meeting its stopping rule validates only the supplied objective.

The nearest plausible alternative is often method of moments: choose parameter values whose model moments match sample moments. It can be transparent, quick, and useful as an initializer, but it uses selected moments rather than the full likelihood and can be less efficient, unstable, or outside the valid parameter space. Choose MLE when the target, sampling process, model family, and dependence structure are credible and a model-based fit helps the decision. Choose moments when a simple, interpretable estimate or starting value is easier to defend. Choose a robust or design-based estimator when outliers, heavy tails, selection, clustering, or survey weights make the ideal IID likelihood implausible. Changing the optimizer does not repair a wrong data-generating story.

7. Common interview trap

8. Three checkpoint disclosures

1. What is the difference between bias and variance?

   Bias is the signed displacement E[θ̂]−θ across repeated samples. Variance is the spread around E[θ̂]. MSE adds variance and squared bias, so an unbiased estimate is not automatically best.

2. Why does the Normal MLE for σ² divide by n rather than n−1?

   After estimating μ, maximizing the Normal likelihood gives the average squared deviation with denominator n. The n−1 correction is unbiased for σ² under the same model, but it is not the likelihood maximizer.

3. Why optimize log-likelihood, and what does it not tell you?

   Log is strictly increasing, so it preserves the maximizing parameter while turning products into sums and improving numerical stability. It does not turn the parameter into a random variable, validate the model, or establish causality.

9. Interview practice bank

1. 1. Define estimator, estimate, bias, variance, MSE, and consistency.

   Strong answer: define the statistic before data versus its realized value after data; give the bias, variance, and MSE formulas; distinguish finite-sample expectation from convergence in probability. Likely follow-up: Can a biased estimator be preferable? Signal: precise vocabulary tied to repeated sampling.

2. 2. Derive the Bernoulli MLE from n trials and y successes.

   Strong answer: write L(p)=pʸ(1−p)ⁿ⁻ʸ, take logs, differentiate, check the negative second derivative, and handle y=0 or y=n as boundary cases. Likely follow-up: Why is the binomial coefficient irrelevant to the maximizer? Signal: can derive rather than quote.

3. 3. Interpret a 12% retention MLE for a product team.

   Strong answer: name the eligible unit, window, denominator, and target population; call it a descriptive baseline under assumptions; report uncertainty and avoid a causal claim. Likely follow-up: What if next month’s segment mix changes? Signal: connects a number to a decision without overclaiming.

4. 4. A solver returns a precise estimate from repeated users and missing opt-out outcomes. What do you diagnose?

   Strong answer: separate dependence from selection/missingness; audit the unit, duplication, exposure, and missingness mechanism; use cluster-aware or repeated-measures modeling and sensitivity/weighting when justified. Likely follow-up: Does a larger row count fix this? No. Signal: sees threats to identification and uncertainty.

5. 5. Compare MLE with method of moments.

   Strong answer: contrast full likelihood with matched moments; discuss efficiency, robustness, constraints, computation, initialization, and model credibility; choose by target and decision, not slogan. Likely follow-up: When would you use a robust estimator? Signal: articulates a real decision boundary.

6. 6. Explain how MLE appears in logistic regression.

   Strong answer: a Bernoulli conditional model makes maximizing likelihood equivalent to minimizing summed log loss; regularization changes it to penalized likelihood, so the result is not the unpenalized MLE. Check convergence, separation, scaling, calibration, and deployment-aligned validation. Likely follow-up: Is low log loss causal? No. Signal: connects statistical and ML objectives cleanly.

7. 7. All 20 trial outcomes are successes. What is the estimate and what is the trap?

   Strong answer: p̂=1, a boundary MLE; the interior derivative formula does not apply, and the point estimate does not mean future success is certain. Discuss sample size, uncertainty, and whether the trial process was correctly specified. Likely follow-up: What if the data are repeated measurements from one account? Signal: catches edge cases and overconfidence.

8. 8. A marketplace has 400 orders from 120 customers. Estimate cancellation probability and decide whether a reminder caused a fall.

   Strong answer: define whether the unit is order or customer, account for repeated customers and timing, estimate a baseline only under a defensible sampling model, then use randomization or a credible causal comparison for the reminder. Likely follow-up: What if reminder exposure was logged after cancellation? Signal: frames the estimand before calculating.

10. Timed synthesis: eight minutes

Recap: the compact cheat sheet

Technical framing follows the NIST/SEMATECH maximum-likelihood reference, Penn State STAT 504 on likelihood and discrete data, and Penn State STAT 415 on point estimation. The model-fitting boundary is cross-checked with the SciPy distribution-fitting documentation and scikit-learn’s logistic-regression objective notes. Interview emphasis is calibrated with the Chan Zuckerberg Initiative 2025 guide, an employer example rather than a universal interview loop.