MS Module 20: Residuals and Standardized Residuals – practice problems

(The attached PDF file has better formatting.)

Exercise 20.1: Standardized residuals

An actuary regresses the five Y values on the five X values shown below.

X value    –2    –1    0    1    2
Y value    –4    0    2    2    0


A.    What is Sxx?
B.    What is Sxy?
C.    What is β1?
D.    What is β0?
E.    What is the fitted value at x = -2?
F.    What is the residual at x = -2?
G.    What is the error sum of squares (SSE)?
H.    What is s2, the least squares estimate of σ2?
I.    What is the variance of the residual at x = -2?
J.    What is the standard deviation of the residual at x = -2?
K.    What is the standardized residual at x = -2?

Part A: The mean of the X values, , is (-2 + -1 + 0 + 1 + 2) / 5 = 0.

Sxx (the sum of squared deviations) is (-2 - 0)2 + (-1 - 0)2 + (0 - 0)2 + (1 - 0)2 + (2 - 0)2 = 10

Part B: The mean of the Y values, , is (-4 + 0 + 2 + 2 + 0) / 5 = 0.

Sxy (the sum of cross deviations) is -2 × -4 + -1 × 0 + 0 × 2 + 1 × 2 + 2 × 0 = 10

Part C: β1 = Sxy / Sxx = 10 / 10 = 1

Part D: β0 = - β1 × = 0 - 1 × 0 = 0

Part E: The fitted values are the same as the x values (since β0 = 0 and β1 = 1), so the fitted value at x = -2 is -2.

Part F: The residual is the actual y value minus the fitted y value. At x = -2, this is -4 - -2 = -2.

Part G: The error sum of squares (SSE) is the sum of squared residuals (page 692).

●    We calculate the residuals for the five points as {-2, 1, 2, 1, -2}.
●    The sum of squared residuals is -22 + 12 + 22 + 12 + -22 = 14.

Part H: The value of s2 (the least squares estimate of σ2) = 14 / (N - 2) = 14 / (5 - 2) = 4.66667

Part I: The variance of the residual is σ2 × (1 - 1/n - (xi - )2 / Sxx).

For σ2 = 4.66667; n = 5; xi = -2; = 0; Sxx = 10, the variance of the residual is

    4.66667 × (1 - 1/5 - (-2)2 / 10) = 1.86667

Question: The formula for the width of the confidence interval uses (1 + 1/n + (xi - )2 / Sxx) instead of (1 - 1/n - (xi - )2 / Sxx). Why do the formulas have the opposite signs for 1/n and (xi - )2 / Sxx?

Answer: Regression analysis assumes that the variance is the same at all points. The variance is the expected value of the squared deviation of the actual value from the fitted value. The deviation has two parts:

●    The deviation caused by the fitted regression line, which may differ from the true regression line.
●    The deviation caused by the difference of the observed value minus the expected value (the error term).

The regression line passes through the mean x-value. The slope of the regression line depends on the observed y-values. A change in the slope has a larger effect on the fitted y-value if the x-value is farther from the mean x-value. The fitted value moves closer to the observed value, and the deviation of the residual (the observed value minus the fitted value) is smaller.

Question: How do random fluctuations affect the residual at points close to vs far from the mean x-value?

Answer: Points close to the mean of X have little influence on the slope of the regression line (β1). A random fluctuation in the observed Y value at x = causes the regression line to shift up or down but does not change its slope. The shift up or down is spread evenly over all points, with a variance of σ2/n. The change in the variance of the residual is σ2 – σ2 / n = σ2 × (1 – 1/n). [This explanation is heuristic (intuitive), not rigorous. The mathematics helps you remember the formulas; it does not prove the formulas.]

Points far from the mean of X affect the slope of the regression line (β1). A random fluctuation in the observed Y value at a point far from causes the regression line to change its slope (in addition to shifting up or down). The fitted value moves in the same direction as the observed value, so the residual has a smaller variance.

Part J: The standard deviation of the residual at x = -2 is 1.866670.5 = 1.36626

Part K: The standardized residual at x = -2 is -2 / 1.36626 = -1.46385

Question: I checked this practice problem with the Regression add-in Excel’s Analysis Pack. Excel gives the same regression coefficients and residuals, but different standardized residuals.

Answer: See the explanation at the web page:

https://stats.stackexchange.com/questions/166533/how-exactly-are-standardized-residuals-calculated

(The explanation is attached as a PDF file to this posting.)

Statisticians use several standardized residuals and studentized residuals. Excel does not explain its formula, and the comments on this web-site say it is not correct. You are responsible for the version in the textbook.


Exercise 20.2: Standardized residuals

A regression model has the independent variable X values {1, 2, …, 10, 11}.

●    At the point x = 2, the residual is 0.500 and the standardized residual is 0.300.
●    The residual at the point x = 11 is –0.750.

A.    What is , the mean X value?
B.    What is Sxx, the sum of squared deviations of the X values?
C.    What is the ratio of the residual to the standardized residual at the point x = 2?
D.    What is s, the least squares estimate for σ?
E.    What is the standardized residual at the point x = 11?

Part A: The mean X value is (1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11) / 11 = 6

Part B: Sxx, the sum of squared residuals for the X values, is

(1 – 6)2 + (2 – 6)2 + (3 – 6)2 + (4 – 6)2 + (5 – 6)2 + (6 – 6)2 + (7 – 6)2 + (8 – 6)2 + (9 – 6)2 + (10 – 6)2 + (11 – 6)2 = 110

Part C: The ratio of the residual to the standardized residual at the point x = 2 is 0.5 / 0.3 = 1.66667

Part D: The variance of the residual is σ2 × (1 - 1/n - (xi - )2 / Sxx) and the mean residual is zero, so

the standardized residual = (the residual – the mean residual) / the standard deviation of the residuals ➾
the standardized residual = the residual / [σ2 × (1 - 1/n - (xi - )2 / Sxx)]½ ➾
σ × [(1 - 1/n - (xi - )2 / Sxx)]½ = the residual / the standardized residual

At the point x = 2, [(1 - 1/n - (xi - )2 / Sxx)]½ = (1 - 1/11 - (2 - 6)2 / 110)0.5 = 0.87386 ➾

0.87386 × s = 5 / 3 ➾ s = (5 / 3) / 0.87386 = 1.90725

Part E: We derive the ratio of the residual to the standardized residual at the point x = 11.

At the point x = 11, [(1 - 1/n - (xi - )2 / Sxx)]½ = (1 - 1/11 - (11 - 6)2 / 110)0.5 = 0.82572 ➾

0.82572 s = 0.82572 × 1.90725 = 1.57485

The standardized residual at the point x = 11 is -0.750 / 1.57485 = -0.47624


Exercise 20.3: Variance of residuals

A linear regression uses the N points Xi = {1, 2, …, 11}

The error sum of squares (SSE) is 36.

A.    What is s2, the least squares estimator of σ2?
B.    What is Sxx, the sum of squared deviations for the X variable?
C.    What is the variance of the residual at x = 2?

Part A: The value of s2, the estimate of σ2, is SSE/(N-2) = 36 / (11-2) = 4.

Part B: The mean X value is (1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11) / 11 = (11 + 1) / 2 = 6

Sxx, the sum of squared residuals for the X values, is

(1-6)2 + (2-6)2 + (3-6)2 + (4-6)2 + (5-6)2 + (6-6)2 + (7-6)2 + (8-6)2 + (9-6)2 + (10-6)2 + (11-6)2 = 110

Part C: The variance of the residual is σ2 × (1 - 1/n - (xi - )2 / Sxx).

We use s2 = 4 as the estimate for σ2; n = 11; xi = 2; = 6; and Sxx = 110. The variance of the residual is

4 × (1 – 1/11 – (2 – 6)2 / 110) = 3.05455

Question: The variance of the width of the prediction interval is proportional to σ2 × (1 + 1/n + (xi - )2 / Sxx).

The width is narrowest at the mean X value and wider at points farther away from the mean. The formula for the variance of the residuals has minus signs instead of the plus signs. If the prediction interval is wider, shouldn’t the variance of the residual be greater?

Answer: This practice problem examines the variance of the residuals (yi – ŷi), not the variance of yi.

Points farther from the mean X value have more influence on the slope of the regression line.

●    If the Y value at the point x = is higher than expected by a random fluctuation of 1 unit, the regression line is shifted up by 1/n units, but its slope does not change. The residual is (1 - 1/n), and the variance of the residual is σ2 × (1 - 1/n).

●    If the Y value at the point x much greater than is higher than expected by a random fluctuation of 1 unit, the regression line is shifted up by 1/n units, and its slope increases, pulling the fitted value of y toward the observed value. The expected residual is (1 - 1/n - (xi - )2 / Sxx)., and the variance of the residual is σ2 × (1 - 1/n - (xi - )2 / Sxx).