When the line is lying to you

Objective

Given a fitted regression and diagnostic evidence, identify the failure mechanism, explain its consequence for prediction or inference, and recommend a justified next check or remedy. The interview-ready habit is to connect every plot to the decision: forecasting, ranking, coefficient interpretation, uncertainty, or a possible intervention.

1. Ask what the residuals are saying

For observation i, the fitted value is ŷᵢ and the residual is observed minus predicted:

eᵢ = yᵢ − ŷᵢ

With an intercept, OLS makes the residuals sum to zero in the training sample. That fact is mechanical, not diagnostic: positive and negative misses can cancel while the misses trace a very clear curve or arrive in long time-ordered runs. The useful question is not “Are the residuals small?” but “Does the unexplained part still depend on a predictor, the fitted value, the collection order, the group, or a particular row?”

The ideal residual-versus-fitted plot is a roughly horizontal band around zero with approximately stable width. “Roughly” matters. Real data are not expected to form a perfect rectangle, and a small sample may conceal a problem. Look for structure large enough to matter for the decision, then investigate it with domain knowledge and a sensitivity analysis.

Scope and assumptions. The core model here is ordinary least squares with an intercept and a continuous outcome. A binary or count outcome, repeated or clustered outcome, or explicitly temporal process may call for a generalized linear, mixed-effects, or time-series model. The plots below diagnose a fitted sample and specification; they do not reveal unobserved errors, prove independence, or establish causality.

2. Five ways structure survives the line

Nonlinearity: the center is wrong

A U-shape, inverted U, curve, or repeated positive-negative pattern means the center of the residual distribution changes with the input. The line is systematically too low in one region and too high in another. This is a mean-function problem: the model has omitted curvature, a transformation, an interaction, a subgroup, or a feature that captures the changing baseline.

A respectable R² does not rescue a misspecified mean. A line can explain a large fraction of broad movement while still making predictable mistakes at the ends. Plot residuals against fitted values and each important predictor. In multiple regression, remember that a residual-versus-predictor plot is a useful first view, but the effect of one predictor is conditional on the others; partial or added-variable views may be more appropriate.

The remedy should match the mechanism. A quadratic is sensible when a domain story supports a bend and the new term survives validation. A spline or transformation may be safer when the shape is not known in advance. A segment or interaction may be the right answer when different groups have different baselines or slopes. Do not add polynomial terms merely to drive training SSE toward zero.

Heteroskedasticity: the scale is wrong

A funnel means that the residual center may be near zero while the vertical spread changes with fitted value. In symbols, the conditional mean may be adequate while Var(e ∣ X) is not constant. This matters because the usual OLS coefficient estimate can remain useful under exogeneity, but classical standard errors and interval calculations can be wrong; predictions can also be less precise in the wide part of the funnel.

Separate three goals. If you only need coefficient uncertainty for a retained mean model, heteroskedasticity-robust covariance can make standard errors more defensible. It does not change the fitted line, make the residuals homoskedastic, or narrow future prediction errors. If the outcome scale is the cause, a log or square-root transformation can change both units and spread. If the variance itself is operationally important, a variance model, weighted least squares, prediction interval, or quantile model may be better. A transformation is a model change; robust covariance is primarily an inference change.

Dependence: the index or group matters

Independence is not visible in an unordered cloud. Sort residuals by time, user, store, household, geography, or whatever unit may share shocks. Long same-sign runs, waves, or clusters suggest that nearby observations are not independent. A random row split can then place the same incident, user behavior, or merchant-day context in both training and validation, producing an optimistic estimate.

Preserve the unit that will receive the prediction or support the inference. Forecasting usually calls for a later-period or rolling split. Repeated users may require grouped splits, cluster-robust uncertainty, or a model that represents user-level variation. Serial dynamics may call for lag features or a time-series model. A large n does not automatically make dependence harmless; many correlated rows can contain much less independent information than their count suggests.

Collinearity: the predictors compete to explain the same direction

Suppose route distance and map ETA move almost together. The model may predict delivery delay well along the observed feature cloud, while it cannot tell which variable deserves an individual coefficient. If predictor j is regressed on the other predictors and that auxiliary regression has Rⱼ², the variance inflation factor is:

VIFⱼ = 1 / (1 − Rⱼ²)

For R² = 0.96, VIF = 1/(1−0.96) = 25. In the usual setup, the coefficient variance is inflated by 25 and the standard error by √25 = 5 relative to a comparable design without that linear redundancy. That is a warning about individual coefficient stability and extrapolation, not proof that predictions are poor or that a feature must be deleted. Pairwise correlations can miss redundancy involving several predictors, so inspect VIF, condition information, coefficient paths, and predictions across reasonable specifications.

Unusual observations: outlier, leverage, and influence are different

An outlier is unusual in the response relative to the fitted pattern. A high-leverage point is unusual in predictor space. Influence asks whether deleting the row materially changes fitted coefficients or predictions. They can overlap, but none is a synonym for “bad data.” A high-leverage observation close to the line may have a small residual because the line bends toward it. A large-residual point near the center of predictor space may have little effect on the fitted line.

Leverage is potential influence, not a verdict. The hat-matrix diagonal hᵢᵢ measures how unusual a predictor vector is relative to the design; with an intercept and p fitted parameters, average leverage is p/n. Cook’s distance combines residual size and leverage into a deletion-based view of how much the fit changes. Thresholds are screening heuristics. Audit the row's units, joins, eligibility, missingness, and feature timing; refit with and without it; then decide whether the case is an error, a valid rare regime, or evidence that the model needs segmentation.

Normality is narrower than the other checks

A Q–Q plot can reveal heavy tails or skew and is most relevant to small-sample normal-theory inference. It does not diagnose nonlinearity, independence, or collinearity, and residual normality is not a universal requirement for fitting OLS. Do not respond to every diagnostic prompt with “make the residuals normal.” Start with the decision and the failure mechanism.

3. A defensible diagnostic sequence

In an interview, a sequence is more convincing than a list of tests. Say what you would do and why:

1. Define the decision and unit. Is the model forecasting a future order, ranking users, estimating a coefficient, or supporting an intervention? One row should represent the unit to which the claim applies.
2. Audit before modeling. Check target construction, feature availability at prediction time, units, joins, missingness, duplicates, and whether unusual records are legitimate members of the target population.
3. Plot the residual center and spread. Use residuals versus fitted values and important predictors. Look separately for curvature and changing width; both can appear in one plot.
4. Respect order and groups. Plot residuals in collection order or by cluster. Change the validation split to match how new cases arrive.
5. Inspect predictor geometry and sensitivity. Use VIF or condition information for redundancy, leverage and studentized residuals for unusual cases, and Cook’s distance or deletion refits for influence.
6. Target the remedy. Change the mean model for curvature, the uncertainty method for a variance problem, the split or covariance for dependence, the feature representation for collinearity, and the data or regime treatment for an influential case.
7. Refit and report sensitivity. Compare the original and revised conclusions on validation data that matches deployment or inference. State what remains uncertain and what the diagnostic does not establish.

Worked calculation: a strong relationship hidden by a line

Consider the deliberately tiny dataset below. It is symmetric, so it is easy to audit by hand:

Step 1 — means. The average input is x̄ = 0. The average outcome is ȳ = (4+1+0+1+4)/5 = 2.

Step 2 — slope numerator. Because x̄ = 0, the centered-product sum is:

(−2)(2) + (−1)(−1) + 0(−2) + (1)(−1) + (2)(2) = 0

Step 3 — slope denominator. The centered input sum of squares is 4 + 1 + 0 + 1 + 4 = 10. Therefore β̂₁ = 0/10 = 0, and β̂₀ = ȳ − β̂₁x̄ = 2. The fitted line is ŷ = 2.

Step 4 — residuals. The fitted values are all 2. Observed minus fitted gives (2, −1, −2, −1, 2). The sum of squared errors is:

SSE = 2² + (−1)² + (−2)² + (−1)² + 2² = 14

The residuals are positive at both ends and negative in the middle: a U-shape. The zero slope does not mean that x and y are unrelated. In fact, the outcome is exactly y = x². Adding that candidate term produces fitted values (4,1,0,1,4) and zero training residuals. That perfect toy fit is a hypothesis to validate, not permission to add a quadratic to every dataset.

Worked case: a delivery-delay model under pressure

Question. A logistics team wants to estimate delivery delay in minutes before dispatch so it can staff support and set a customer-facing buffer. The first OLS model uses route distance, map ETA, restaurant prep time, hour of day, merchant, and weather. Many orders come from the same merchant and day.

Evidence. Aggregate R² looks acceptable, but the residual-versus-fitted plot is a funnel: short predicted delays have narrow misses and long predicted delays have wide misses. Timestamp-ordered residuals show positive runs during traffic incidents. Map ETA and route distance have a high VIF; the distance coefficient changes across reasonable specifications while predictions barely move. One test order has a 90 km route paired with a six-minute map ETA, giving it high leverage and a large deletion effect.

Reasoning. First confirm that one row is one delivered order, audit the test record, and verify that every feature was available before dispatch. Merchant and day repeatedness means ordinary random-row uncertainty is suspect. A future-week or rolling split, possibly grouped to keep incident context together, is more honest for the staffing use case.

For the funnel, ask whether the decision needs coefficient inference, average point predictions, or upper-tail protection. Robust covariance can improve inference for the retained mean model but does not narrow the wide prediction errors. A target transformation, variance model, weighted least squares, or quantile/prediction-interval approach may be more useful for staffing. For dependence, use a time-aware split and consider clustered or HAC uncertainty, lag/state features, or a time-aware model. For VIF, choose one operational representation of route difficulty, regularize if prediction is the goal, or admit that an individual distance effect is not stable under the current design.

The 90 km record is not automatically removable. If it is a logging or test artifact, document the rule and show the cleaned sensitivity fit. If it is a valid rare route, deleting it changes the target population; retain it and model or segment that regime. The bounded recommendation is to delay a customer promise or staffing threshold based only on aggregate fit. After audit, dependence-aware validation, targeted revision, tail and subgroup checks, the model might support an internal forecast with a conservative buffer and a review trigger. It still estimates association and prediction, not the causal effect of reducing distance or traffic.

Nearest alternatives: match the tool to the failure

Robust standard errors, robust regression, weighted least squares, regularization, clustered covariance, and time-aware validation are not interchangeable labels. They change different parts of the analysis. A strong answer says whether the proposed tool changes the fitted mean, the loss function, the uncertainty calculation, the validation design, or the target population.

Common interview trap

Interview practice bank

1. What are regression diagnostics, and where do you start?

   Strong answer: diagnostics ask whether residuals retain systematic information or whether predictors or individual rows make the fit unstable. Start by defining the decision and unit, auditing the data, then plotting residuals against fitted values, important predictors, and collection order before using summary measures.

   Follow-up: What would a random residual band establish? Signal: organizes evidence around mechanism and purpose rather than reciting tests.

2. Fit x = (−2, −1, 0, 1, 2), y = (4, 1, 0, 1, 4). What does the residual plot say?

   Strong answer: x̄=0, ȳ=2, centered-product sum is 0, denominator is 10, so ŷ=2. Residuals are (2,−1,−2,−1,2), a U-shape. The linear mean is misspecified even though the slope is zero.

   Follow-up: What would you do if x² fits perfectly in sample? Signal: combines arithmetic, sign convention, and an out-of-sample next step.

3. How do you distinguish nonlinearity from heteroskedasticity?

   Strong answer: nonlinearity changes the residual center across fitted values; heteroskedasticity changes its spread. A curve or alternating signs suggest a mean problem; a funnel suggests a variance problem. Both may coexist, so inspect center and width separately.

   Follow-up: Would a log transform always solve a funnel? Signal: matches remedies to mechanisms and does not promise a generic fix.

4. Residuals are correlated over time. What changes?

   Strong answer: investigate omitted events or state, preserve time order, avoid a random row split, validate on later periods, and use dependence-aware uncertainty or modeling as the goal requires. The relevant independent unit is not necessarily the row.

   Follow-up: When would clustered standard errors be enough, and when is a time-series model needed? Signal: understands dependence as both an inference and validation issue.

5. What does VIF tell you, and what does it not tell you?

   Strong answer: VIFⱼ=1/(1−Rⱼ²) measures how predictable a predictor is from the others and how that redundancy inflates coefficient variance. A high value warns about individual coefficient instability; it does not prove predictions are bad or dictate deletion.

   Follow-up: What if predictions are strong but coefficients are unstable? Signal: separates prediction from explanation and inference.

6. Distinguish an outlier, leverage, and influence.

   Strong answer: an outlier is unusual in response space, leverage is unusual in predictor space, and influence is a material change in coefficients or predictions after deletion. Use residuals, leverage, Cook’s distance or DFBETAs, and refits; audit before deleting.

   Follow-up: Can a high-leverage point have a small residual? Signal: sees the interaction of x, y, and fitted sensitivity.

7. When would you use robust covariance, a transformation, WLS, or a different model?

   Strong answer: robust covariance changes uncertainty for some variance misspecification; a transformation changes outcome or predictor scale and possibly the mean/variance structure; WLS changes the fitting weights and needs a defensible variance model; a different model is appropriate when outcome type, dependence, nonlinear mean, or the decision is not represented by OLS.

   Follow-up: Is robust standard error the same as robust regression? Signal: names what part of the analysis each tool changes.

8. A delivery-delay model has acceptable R², a funnel, time runs, VIF 18, and one extreme test row. What do you tell the team?

   Strong answer: defer a launch decision based only on aggregate fit. Audit the row and feature timing, use future or grouped validation, address the funnel for the operational target, resolve route/ETA redundancy, and report full-data versus sensitivity results including tail and subgroup error. Keep a valid rare route or document a real data error; do not turn the regression into a causal claim.

   Follow-up: What metric matters for staffing? Signal: integrates statistics, data quality, product consequences, and communication.

Timed synthesis

Three quick checks

1. What do a U-shape and a funnel mean in residuals?

   A U-shape suggests the conditional mean is misspecified; a funnel suggests that the conditional variance changes. They can occur together, so inspect center and spread separately.

2. Why is high leverage not the same as influence?

   Leverage measures unusual predictor position and therefore potential to affect the fit. Influence is observed sensitivity: how much coefficients or predictions change when the row is deleted.

3. What does VIF = 25 imply?

   With the usual model setup, the coefficient variance is inflated by 25 and the standard error by 5 relative to a comparable no-redundancy design. It is a screening signal about coefficient stability, not an automatic deletion rule or proof of poor prediction.

Keep this in mind

• Residual diagnostics ask whether the model left systematic information behind.
• A curve is a mean problem; a funnel is a scale problem; runs are a dependence problem.
• Collinearity can make explanations unstable while predictions stay useful.
• Outlier, leverage, and influence describe different evidence; audit before deleting.
• Choose transformations, robust or clustered covariance, WLS, regularization, and time-aware validation by the decision they support.
• Refit, validate under realistic conditions, report sensitivity, and separate prediction from causality.

Sources and scope

The diagnostic vocabulary follows Penn State STAT 501 on regression assumptions, model building, influential points, and multicollinearity and regression pitfalls. The residual-analysis framing is cross-checked against the NIST model-fit guidance and graphical residual analysis. Package-specific names such as statsmodels diagnostic interfaces are implementation references, not universal cutoffs. The examples are synthetic; the lesson does not teach formal diagnostic-test distributions, full time-series models, mixed effects, robust-regression algorithms, or causal adjustment criteria.