Prediction, adjustment, and causality

Learning objective

Given a regression output or product scenario, state whether the goal is prediction, description, or intervention; interpret the model in that language; choose controls using time order and causal role; and finish with a validation or identification plan that matches the claim.

1. Name the question before naming the model

Suppose A means that a user received an onboarding checklist and Y means seven-day retention. The sentence “the checklist is associated with retention” is not yet a complete analysis. You need the unit, timing, population, and decision: one eligible user, checklist exposure at onboarding, retention seven days later, and a decision about targeting or shipping.

There are at least three different questions hiding inside the same table:

• Prediction: “At the end of day 1, what is this new user’s probability of retaining on day 7?” The target is a future outcome such as E[Y | X=x]. A feature may be useful because it predicts, even if changing it would not change retention.
• Description: “How did retention differ between users who did and did not receive the checklist?” The observed contrast E[Y | A=1] − E[Y | A=0] is a descriptive association, with its composition and selection made visible.
• Intervention: “What would seven-day retention be if this target population received the checklist versus if the same population did not?” This is a counterfactual question. It needs a defined treatment, target population, and design or assumptions that make the missing counterfactual recoverable.

That first sentence in an interview is diagnostic. If the decision is “who should receive a scarce outreach message?”, begin with a prediction problem and then evaluate whether the outreach changes an outcome. If the decision is “should we ship a new flow to everyone?”, begin with a causal estimand and prefer randomization when it is feasible. Do not choose OLS, logistic regression, matching, or a neural network before deciding which claim the analysis must support.

2. What “holding constant” actually means

In multiple regression, a coefficient is a comparison made inside the fitted model. If the model is E[Y | A,C] = β₀ + β₁A + β₂C, then β₁ is the model-implied difference between A=1 and A=0 at the same included value of C, under the model’s functional-form assumptions.

That is more precise than saying “all else equal,” but it is not yet an intervention. A real user who is assigned the checklist may also complete more onboarding steps, contact support differently, or be selected by a team that knows something not in the table. “Hold C constant” describes a comparison of model-implied conditional means. It becomes a causal sentence only when the study design and assumptions support that counterfactual interpretation.

For causal total effects, start with variables measured before treatment that help determine both treatment and outcome. Do not select controls because their p-values are small, because they increase R², or because a particular specification produces a desired coefficient. A control can be predictive but not a confounder, downstream of treatment, or a common effect that creates bias when conditioned on.

3. Worked calculation: the +30-point lift that disappears

Let C be a high- or low-intent label measured before assignment. The product team observes the following deliberately small table:

Step 1 — compute the observed contrast. Checklist users retain at 6/10 = 0.60. Non-checklist users retain at 3/10 = 0.30. The raw gap is:

0.60 − 0.30 = 0.30

That is a 30 percentage-point observed association. It is real arithmetic, but it compares an 80% high-intent checklist group with a 20% high-intent non-checklist group. Intent is already related to retention inside both treatment groups.

Step 2 — compare within intent. Among high-intent users, the checklist difference is 0.75 − 0.50 = +0.25. Among low-intent users, it is 0.00 − 0.25 = −0.25. The two stratum differences point in opposite directions. The raw +30 points did not reveal a stable effect; it reflected unequal composition plus these within-stratum contrasts.

Step 3 — standardize to a named target population. Choose a 50/50 high/low-intent target population. Under checklist assignment, the standardized retention is:

E[Y¹] = 0.5(0.75) + 0.5(0.00) = 0.375

Under no checklist, it is:

E[Y⁰] = 0.5(0.50) + 0.5(0.25) = 0.375

The standardized contrast is 0.375 − 0.375 = 0, or 0 percentage points for this deliberate target population. The calculation shows why the raw difference is insufficient; it does not prove that the true effect is zero. There are only 20 users, and the causal reading additionally requires credible measurement, treatment definition, overlap, and no important unmeasured confounding.

This is the regression connection. A saturated regression with checklist, intent, and a checklist-by-intent interaction can reproduce the four cell means. Standardization then predicts each person under both treatment options and averages those predictions over the target intent distribution. The hand calculation makes the estimand and weights visible before a coefficient hides them.

4. The minimum causal vocabulary

Causal inference starts with potential outcomes: Y¹ is the retention the same unit would have under the checklist, and Y⁰ is the retention that unit would have without it. The average treatment effect is E[Y¹ − Y⁰]. We cannot observe both for the same user, so design and assumptions do the work of making the missing comparison credible.

For a discrete pre-treatment adjustment set C, the standardization identity is:

E[Yᵃ] = Σᶜ E[Y | A=a,C=c] P(C=c)

Say the three conditions aloud:

• Conditional exchangeability: after conditioning on C, treatment groups are comparable for their potential outcomes. In observational data, this is an assumption about remaining unmeasured common causes, not something a significance test proves.
• Positivity: every target covariate pattern has some chance of receiving each treatment being compared. If every enterprise account gets the checklist and every free account does not, the data cannot learn a within-stratum contrast without extrapolation.
• Consistency: the observed treatment corresponds to the intervention in the estimand. “Received the checklist” must specify version, timing, dose, and exposure well enough that Y¹ means one coherent intervention.

Also name practical threats: selection into the observed data, missing outcomes, interference between users, treatment noncompliance, measurement error, time-varying confounding, and a misspecified outcome model. More rows and smaller standard errors cannot repair a causal effect that the design does not identify.

5. Controls are causal roles, not a shopping list

Use the timeline in the interaction. Intent measured before assignment can influence both assignment and retention, so it is a confounder candidate. Acquisition channel and plan type may play a similar role. Checklist completion is measured after assignment and may be a mediator: it is part of the pathway through which the checklist works. Controlling for it can remove part of the total effect and change the question to a direct-effect question.

Support contact after assignment is more subtle. Suppose checklist assignment and unmeasured user difficulty both affect whether a user contacts support. Support contact is then a common effect, or collider. Conditioning on it can create a path between treatment and difficulty that was not present in the unconditioned data. Do not add it because it is “important”; draw the causes and decide what estimand you want.

Prediction has a different rule. A downstream feature can be valuable for a later prediction task if it is available at that later scoring time and does not leak the target. A proxy can predict well without being a cause. But a feature computed from events through day 7 cannot be used to choose an intervention at day 1. The question is not only whether a column exists; it is whether it exists at the decision timestamp with the same definition it will have in production.

6. Worked applied scenario: one dataset, two defensible analyses

A learning product has acquisition channel, plan, device, prior activity, checklist exposure, checklist completion, support contact, and seven-day retention. The same dataset can support two useful analyses, but they require different answers.

Version A — predict who is likely to retain. At the end of day 1, score eligible new users for day-7 retention. Build point-in-time features from what is known by that cutoff: acquisition channel, plan, device, and events observed by day 1. Exclude day-7 retention and future support resolution. If users appear repeatedly, use a group-aware split; if the deployment is next month, a temporal holdout may be more honest than a random split. Choose the metric from the action: calibration for reliable probabilities, ranking when outreach capacity is fixed, or expected utility when contact costs differ.

Then inspect slices and compare with a simple business baseline. If prior activity predicts retention, say “users with more prior activity have higher predicted retention under this model.” Do not say “increasing prior activity will cause retention to rise.” The model can prioritize who receives a message; the message itself still needs an outcome evaluation.

Version B — estimate whether the checklist increases retention. Define the total-effect estimand: seven-day retention if the target population were assigned the checklist versus if it were not, under a specified exposure window. If assignment was preferentially shown to high-intent users, the raw gap is not enough. If feasible, randomize eligible users at the correct unit, define exposure and analysis windows, audit assignment and logging, and measure retention plus guardrails such as support load or downstream quality. Regression can improve precision or estimate pre-specified heterogeneity; it is not what creates comparability.

If randomization is unavailable, choose pre-treatment controls from the domain, check overlap, inspect missingness and selection, standardize to the target population, and report sensitivity to reasonable specifications. Do not control for completion for the total effect. Do not treat a high-accuracy retention model as evidence that the checklist works. A risk model says who is at risk; an intervention test says whether the intervention changes the risk.

7. Failure modes and the nearest alternative

The nearest plausible alternative to an adjusted observational regression is not always another algorithm. It is often a randomized experiment. Randomization makes treatment groups comparable in expectation, while regression adjustment can improve precision or describe pre-specified heterogeneity. An experiment can still fail through noncompliance, interference, exposure or logging errors, attrition, or a poorly defined outcome, so “randomized” is not a substitute for implementation checks.

Use observational adjustment when randomization is unavailable and the treatment, target population, timing, common causes, overlap, and measurement are sufficiently credible. If those conditions are not credible, label the result descriptive and recommend a design that would identify the effect. Use predictive modeling when the operational decision is about future risk, and validate on data that imitate deployment. These are different decision boundaries:

Other common failures are extrapolating beyond overlap, hiding effect heterogeneity behind one average coefficient, using a random row split when users repeat or time matters, and declaring an adjusted zero effect to mean “useless.” A null can reflect a genuinely small average effect, opposing segment effects, low support, measurement error, noncompliance, or model misspecification. Name which explanation you would investigate next.

Common interview trap

Three quick checks

1. Why is the raw difference not automatically causal?

   Treatment and control can differ in pre-treatment composition, selection, or measurement. The observed gap can mix an intervention effect with confounding, reverse causation, or selection. A design or defensible identifying assumptions are needed before causal verbs.

2. Name the three conditions in the basic observational-adjustment vocabulary.

   Conditional exchangeability, positivity, and consistency. They are substantive assumptions and design conditions, not outputs produced by running a regression.

3. Why not control for checklist completion when estimating the total effect?

   Completion occurs after assignment and may mediate the checklist’s effect. Controlling for it can remove part of the total effect and answer a different direct-effect question; it is not a neutral control just because it predicts retention.

Interview practice bank

1. What is the difference between predictive and causal regression?

   Strong answer: prediction estimates a rule for future outcomes and is judged on deployment-aligned held-out performance; causal analysis estimates a counterfactual intervention contrast and needs a design or assumptions that identify it. A coefficient is not causal merely because controls were included.

   Follow-up: Can a causal model predict? Can a predictive feature be a confounder? Signal: starts with the estimand, not a favorite algorithm.

2. Compute the raw and standardized contrasts in the onboarding table.

   Strong answer: raw is 6/10 − 3/10 = +30 points. The high/low mix is 80/20 in treatment and 20/80 in control. Within-stratum gaps are +25 and −25 points. At a 50/50 target, checklist is 0.5(.75)+0.5(0)=.375, control is 0.5(.50)+0.5(.25)=.375, so the standardized contrast is 0.

   Follow-up: Does this prove no effect? What if the target is 80% high intent? Signal: shows weights and refuses to overclaim from 20 users.

3. What does the checklist coefficient mean after controlling for plan and acquisition channel?

   Strong answer: it is the fitted conditional difference for checklist exposure at equal included plan and channel values, with units and functional-form assumptions. It is an adjusted association unless design and identifying assumptions support causality. Interactions may be needed if effects differ by segment.

   Follow-up: Is “holding constant” the same user with and without the checklist? Signal: uses conditional language without hiding behind “all else equal.”

4. Which would you control for: pre-exposure intent, completion after exposure, or support contact after exposure?

   Strong answer: pre-exposure intent is a confounder candidate if it affects assignment and outcome. Completion is a mediator candidate for the total effect. Support contact may be a collider if treatment and user difficulty cause it; draw the timeline and causes first. For prediction, a post-treatment field can be valid only for a later scoring time without leakage.

   Follow-up: What changes if the team wants a direct effect? Signal: changes the answer when the estimand or timeline changes.

5. A model has RMSE 0.12 on training rows and 0.16 on a random split. Is it ready to target users next month?

   Strong answer: not yet. Define deployment time and population, audit point-in-time availability, use a temporal or group-aware holdout when appropriate, check leakage and drift, compare with a business baseline, inspect calibration and slices, and choose a threshold or capacity rule.

   Follow-up: What if users contribute many rows? What if the temporal score drops? Signal: matches validation to the data-generating process.

6. When would you prefer a randomized experiment to adjusted regression?

   Strong answer: when the change can be assigned and the ethical and operational context permits it, because randomization establishes comparability in expectation. Regression can improve precision or model pre-specified heterogeneity. Still check exposure, logging, interference, attrition, and the outcome.

   Follow-up: Is regression unnecessary in an experiment? Can an experiment still be biased? Signal: separates design from analysis.

7. A retention model identifies high-risk users. Should the product automatically show them a new onboarding flow?

   Strong answer: separate risk prediction from intervention evaluation. Validate targeting at its scoring time, then randomize or otherwise evaluate the flow among eligible users with retention as the primary outcome and quality/support guardrails. Define exposure and keep a control group.

   Follow-up: How would you choose the targeting threshold? How would you measure incremental lift? Signal: connects model output to an intervention test.

8. The adjusted average effect is zero. Can we conclude the checklist is useless?

   Strong answer: no. The average may hide heterogeneous effects, lack support, or reflect low power, measurement error, noncompliance, or misspecification. Pre-specify meaningful segments, inspect overlap and uncertainty, use interactions or stratum-specific standardization, and avoid slicing until something wins.

   Follow-up: How would you communicate a heterogeneous result? Signal: balances null interpretation with multiplicity and decision relevance.

Timed synthesis

Interview-ready recap

Technical framing: Penn State STAT 501 on regression and association, multiple-regression interpretation, Hernán and Robins, Causal Inference: What If, and scikit-learn’s cross-validation guidance. Employer materials in the research packet calibrate the emphasis on applied reasoning; they do not establish a universal interview loop.