Classical regression analysis is based on the statistical model yj = " + $1 xj1 + $2 xj2 + … + $k xjk + ,j.

Part A: The linearity assumption is that the expected error term is zero: E(,j) = 0

Jacob: Do you mean that the average y-value is a linear function of the average x-values? We show this by

E(yj) = E(" + $1 xj1 + $2 xj2 + … + $k xjk + ,j) A ¯y = " + $1 ¯x1 + $2 ¯x2 + … + $k ¯xk

Rachel: Your explanation shows that the means of the observed X values and the mean of the observed Y

values lie on the fitted regression line. The value of A (the least squares estimator of ") is chosen to force this

¯y = A + B1 ¯x1 + B2 ¯x2 + … + Bk ¯xk

The linearity assumption is that the expected error term is zero at each point: E(,j) = 0 for all j.

Jacob: Are you saying that the expected value of each observed residual is zero?

Rachel: Distinguish between the observed residual and the error term.

! The observed residual is the observed response variable Y minus the linear function of the observed

! The error term is the expected response variable Y at any point minus the linear function of the observed

explanatory variables, using the population regression parameters " and $.

The population regression parameters " and $ specify the relation between the explanatory variables and the

A and B are not the same as " and $. The linearity assumption says the expected value of the error term at

Jacob: Can you show how the linearity assumption relates to the expected value of the error term?

Rachel: Suppose the true relation is Y = 1 + X2 + ,, which is not linear. We fit a regression line through three

of " and $. They are linear functions of the observed y-values, so the expected values of A and B are fitted

values using the expected values of Y at each point. This fitted regression line is Y = 0.66667 + 2X + ,.

! X = 0: expected residual = 0.33333.

! X = 1: expected residual = –0.66667.

! X = 2: expected residual = 0.33333.

Intuition: The linearity assumption says that the expected error term at each X value is zero, not that the mean

Jacob: Is the expected error term over all X values zero even if the true relation is not linear?

Rachel: The expected residual over all X values is zero whether or not the true relation is linear.

, and Y.

Jacob: GLM stands for generalized linear model, implying that the relation is linear. But GLMs are used when

Rachel: The term generalized linear means that the response variable is a function of a linear combination

Part B: The variance of the error term is the same at each X value: var (,j) = F2

å = constant for all xj.

Illustration: If F2

å = 2 at x = 1, then F2

å = 2 at x = 3

Jacob: Another textbook says that the variance of the error term is the same at each expected Y value.

Rachel: E(Y) = " + $ × X, so the expected Y values map one to one into the X values. The variance of the

Intuition: Generalized linear models differ from classical regression analysis several ways, including the

! For generalized linear models, the variance of yj is often a function of íj.

! For classical regression analysis, the variance of yj is independent of íj.

Jacob: Is constant variance a reasonable assumption?

Rachel: The assumption of constant variance may not reflect actual conditions, but it underlies the formulas

Illustration: Suppose we regress annual income of actuaries on the number of exams they have passed, with

" = 30,000 and $ = 10,000.

! Students with no exams have average annual incomes of 30,000.

! Average income increases $10,000 an exam, so actuaries with nine exams have average incomes of

! Students with no exams get about the same starting salary. The actual range is narrow, perhaps 28,000

! Experienced actuaries vary: some are gifted statisticians or managers and earn 160,000, and some are

Jacob: This illustration shows that omitted variables may be more important for actuaries with more exams.

Rachel: GLMs show the relation of the variance of the response variable to its mean (fitted) value.

! General insurance claim counts have Poisson distributions or negative binomial distributions: the variance

! General insurance claim severities have lognormal or Gamma distributions: the standard deviation is

! Life insurance mortality rates have binomial distributions: the variance is proportional to B × (1 – B).

Jacob: If the variance of the response variable depends on its mean, why not use that as our assumption?

Rachel: In real statistical applications, we often do. We use weighted least squares estimators or GLMs (which

Part C: Normality: The error terms have a normal distribution with a mean of zero and a variance of F2

å:

,j - N(0, F2

å)

Jacob: Why not state these first three assumptions in the reverse order?

! First, the error terms have a normal distribution. This seems most important.

" If the error terms have a Poisson distribution, Gamma distribution, or binomial distribution, their

" If the error terms have a normal distribution, their variance may or may not be constant.

! Second, the variance of the error term is the same at all points. This assumption underlies the formulas

" If the error terms have a normal distribution with constant variance, the least squares estimators are

! Third, the mean of the error term is zero. If it equals k . 0, add k to " to get a mean error term of zero.

Rachel: On the contrary: the first three assumptions are listed in order of importance.

! The first assumption is critical if the expected error term is not zero at each point, the response variable

Jacob: Is the model biased only if it is structural, not if it is empirical?

Rachel: Fox says: Omission of explanatory variables causes bias only for structural relations, not for empirical

relations. Model specifiication error causes bias for both types of relation.

! The second assumption is also important: if the variance of the error term is not constant, we should give

for A and B (the ordinary least squares estimators for " and $) would not be correct.

! The third assumption is not that important. If the mean of the error term is zero and its variance is

Jacob: If the assumption about a normal distribution is not important, why do we include it?

Rachel: This assumption is underlies the formulas for the standard errors of the regression coefficients, the

t-values, the p-values, and the confidence intervals.

! If the error terms have a normal distribution with constant variance, we exact standard errors, t-values,

p-values, and confidence intervals.

! If the error terms do not have a normal distribution, these values are not exact. They are asymptotically

! The standard errors, t-values, p-values, and confidence intervals are not exact, but they are reasonably

good if the variance of the error terms is constant. If this variance is not constant, we use weighted least

Jacob: Is this assumption of normally distributed error terms generally true?

Rachel: The central limit theorem implies that it is a good approximation if the explanatory variables are the

Jacob: Another textbook says that the Y values have a conditional normal distribution.

Rachel: yj = " + $1 xj1 + $2 xj2 + … + $k xjk + ,j. The only random variable on the right side of the equation is ,j,

so if ,j has a normal distribution, yj has a normal distribution. Fox says: “Equivalently, the conditional

distribution of the response variable is normal: Yj - N(" + $xj, F2

å).”

Jacob: How does a conditional distribution differ from a distribution?

Rachel: The term distribution (or probability distribution) has two meanings. The regression analysis formulas

The sample distribution is the distribution of the observed values in the sample. The values of F2(x) and F2(y)

Jacob: How do these distributions differ for the error term?

Rachel: The error terms are independent and identically distributed. Each error term is normally distributed

with a mean of zero and a variance of F2

å, so the distribution of all error terms is also normally distributed with

a mean of zero and a variance of F2

å.

Take heed: This is the distribution of the population of error terms. The sample distribution of the observed

values does not have a mean of zero and a variance of F2

å (unless N ÿ 4). The sample distribution of the

residuals has a mean of zero (by construction) and a variance whose expected value is F2

å but whose actual

Jacob: How do these distributions differ for the response variable?

Rachel: The expected value of yj is " + $ × xj. The sample distribution of the y-values does not have a normal

distribution. Each yj is " + $ × xj + ,j, so each response variable has a normal distribution with a mean of its

expected value and a variance of F2

å.

Part D: Independence: the error terms are independent: ,j, ,k are independent for j . k

Jacob: Independence seems generally true. If we regress home prices on home sizes, why would the error

term for observation j+1 depend on error term for observation j?

Rachel: We have two statistics on-line courses: one for regression analysis and one for time series. In a time

Jacob: Why does a regression where the response variable is a time series create problems?

Rachel: If the response variable is a time series, the expected value of the error term is not zero at each point,

and the error term at point j+1 depends on the error term at point j. Suppose we regress the daily temperature

Jacob: A time series looks at values over time. What if all data points are from the same time, such as daily

Rachel: A time series can be temporal or spacial. The daily temperature is similar in adjacent cities and less

Illustration: Home prices may be a linear function of plot size and the number of rooms. But home prices have

! Homes in adjacent locations have more similar prices than homes in different states or countries.

! Home prices for all properties fluctuate over time with interest rates, recessions, and tax rates.

Illustration: Suppose we measure daily temperature in a city on the equator. The expected daily temperature

is 80E every day of the year, with on seasonal fluctuations, but random fluctuations cause the temperature to

vary from 60E to 100E. Daily temperature is a time series:.if the temperature is high one day, it will probably

Jacob: Another textbook says that any two Y values are independent: each yj is normally distributed and

independent. Does this mean

! that Yj - Yk is independent of j - k (that is, Yj+1 is independent of Yj), or

! that Yj - íj is independent of Yk - ík?

Rachel: yj = " + $1 xj1 + $2 xj2 + … + $k xjk + ,j. The only random variable on the right side of the equation is ,j,

and íj = " + $1 xj1 + $2 xj2 + … + $k xjk, so if ,j and ,k are independent, yj - íj and yk - ík are independent.

The Fox textbook says: “Any pair of errors ,i and ,j (or, equivalently, conditional response variables Yi and Yj)

are independent for i . j.

Part E: The explanatory variables (X values) are fixed.

! In experimental studies, the statistician picks explanatory variables and observes the response variable.

! In observational studies, the explanatory variables are measured without error.

Jacob: Are actuarial pricing studies experimental studies?

! To see how sex affects claim frequency, one picks 100 male drivers and 100 female drivers.

! To see how age affects claim frequency, one picks 100 youthful drivers and 100 adult drivers.

Rachel: Actuarial pricing studies are observational studies, not experimental studies. The actuary observes

Jacob: What is an intervention?

Rachel: In experimental studies, the researcher creates the explanatory variable. A clinical trial for a new drug

Jacob: Why is the difference between attributes and interventions important?

Rachel: Attributes are often correlated with other (omitted) explanatory variables. Attributes may create biases

Illustration: An actuary regresses motor insurance claim frequency on urban vs rural territories and concludes

Illustration: Duncan’s Canadian occupational prestige is an observational study, as are most social science

Jacob: Are the observed x-values a random sample?

Rachel: The term random sample has several meanings. An actuary comparing urban vs rural drivers may

The statistical term randomized means that an intervention is applied randomly to subjects. The subjects have

Illustration: Patients are randomly assigned the new vs the old medication. Even the doctors and nurses do

Illustration: A statistician regresses home values on area of the house or the lot. The regression depends on

Jacob: Is this assumption reasonable?

Rachel: It is often reasonable. Life insurance mortality rates and personal auto claim frequencies depend on

Part F: X is not invariant: the data points have at least two X values. If the X value is the same for all data

points, the least squares estimators are " = ¯y (the mean of the y-values).

Jacob: Do we infer that $ = 0 if all the x-values are the same?

Rachel: We can not infer anything about $, since no data show how a change in x affects y.

Jacob: Does invariance that mean that no X values should be the same?

Rachel: On the contrary, many X values are the same in regression analyses. Suppose we regress personal

xj is the value of the explanatory variable at point j.

,j is the value of the error term at point j.

yj is the observed value of the explanatory variable at point j.

íj is the fitted value of the explanatory variable at point j.

A. The correlation of xj with ,j is zero.

B. The correlation of íj with ,j is zero.

C. The correlation of yj with ,j is zero.

Part A: Classical regression analysis assumes the error term is independent of the explanatory variables.

Part B: í = " + $ × x, so the correlation of í with , is D(,, " + $ × x) = D(,, ") + $ × D (,, x) = 0 + $ × 0 = 0.

Part C: y = í + ,, so D(y, ,) . 0. When , > 0, y > í, and when , < 0, y < í. We state two implications:

! For any given observation j, ,j and yj are random variables that are perfectly correlated.

! For all observations combined, ,j and yj are random variables that are positively correlated.

xj is the value of the explanatory variable at point j.

,j is the value of the error term at point j.

yj is the observed value of the explanatory variable at point j.

íj is the fitted value of the explanatory variable at point j.

A. The expected value of xj × ,j is zero.

B. The expected value of yj × ,j is zero.

C. The expected value of íj × ,j is zero.

Part A: D(xj, ,j) = 0, so E(xj × ,j) = 0

Part B: D(yj, ,j) > 0, so E(yj × ,j) . 0

Part C: D(íj, ,j) = 0, so E(íj × ,j) = 0

yj = " + $1 xj1 + $2 xj2 + … + $k xjk + ,j.

Part A: The least squares coefficients are linear functions of the observed data.

Jacob: How are they linear functions? B = 3(xi – ¯x)(yi – ¯y) / 3(xi – ¯x)2. This has terms of xi yi in the numerator

and x2i in the denominator. A = ¯y – B × ¯x, so A is also a function of both the x and y values.

Rachel: The observed data are the response variable (the Y values), not the explanatory variables (X values).

! The explanatory variable is fixed, not a random variable.

! The error term is a random variable is a fixed (but usually unknown) variance F2

å.

! " and $ are unknown population regression parameters, not random variables which we estimate.

! The response variable yj is a random variable with

" a mean of " + $ × xj

" a variance of F2

The least squares estimator B is a linear function of the yj.

! If we repeat the regression analysis, B will differ from the first analysis, since the y-values differ.

! The x-values do not differ in the two experimental studies.

! If we repeat the regression analysis many times, the mean B values approaches $.

The xj are the coefficients of the linear function of the y-values.

Jacob: Must we know these coefficients for the on-line course?

Rachel: The fitted values íj are A + B xj. A and B are linear functions of the observed y-values, so each í value

Jacob: Most regression analysis in the social sciences (and in actuarial work) are observational studies, not

Rachel: Even for observational studies, where X values are observed (not fixed by the statistician), the values

Illustration: An actuary regresses life insurance mortality rates on sex and age. The observed mortality rate

Illustration: A real estate broker regresses home prices on the number of rooms and the size of the home. The

Jacob: What difference does measurement error make?

Rachel: Classical regression analysis assumes Yj = " + $ Xj + ,j. If the Xj have measurement error, we are

regressing Yj on (Xj + the measurement error). The inferences of the regression analysis, such as the variance

Part B: The least squares estimator are unbiased estimators of the population regression coefficients.

Jacob: $ is a fixed but unknown population regression parameter; B is a random variable.

! If B = $, then B is correct.

! If B > $, B is too high.

! If B < $, B is too low.

If B > $, doesn’t that mean that B is biased upward? If B < $, doesn’t that mean that B is biased downward?

Rachel: B is a random variable, with errors that are normally distributed. B is never exactly equal to $ because

of random fluctuations, but the expected value of B is $.

Don’t confuse an over-estimate or under-estimate with a biased estimator.

! An estimate may be too high or too low even if it is unbiased.

! An estimator can be biased or unbiased.

Illustration: Suppose the true linear relation is

Jacob: Aren’t most estimators unbiased? Why would anyone use a biased estimator?

Rachel: Some estimators are unbiased, some are biased. Actuaries use biased estimators for many studies,

Illustration: An actuary examines claim sizes, where the size-of-loss distribution is lognormal. Some actuaries

Part C: Suppose we have a random sample of ten numbers from a normal distribution with a mean of :. We

! Method #1: The average of the highest and lowest numbers.

! Method #2: The average of all numbers except the highest and lowest numbers.

! Method #3: The average of all the numbers.

Let kBj be estimate #j for method #k. For example, 2B1,000 is the 1,000th estimate for method #2. If all methods

are unbiased, 31Bj / 10,000 = 32Bj / 10,000 = 33Bj / 10,000 = :.

The squared error is (kBj – :)2, and the mean squared error is 3 (kBj – :)2 / 10,000. The three estimators do

3(1Bj – :)2 / 10,000 > 3(2Bj – :)2 / 10,000 > 3(3Bj – :)2 / 10,000.

An estimator with a lower mean squared error is more efficient. The least squares estimators are the most

efficient linear estimators.

Jacob: Perhaps one of the samples has an unusually high figure that occurs rarely. In this case, Method #2

Rachel: You are right. Instead of 10,000, we should say N estimates. In the limit, as N ÿ 4,

3(1Bj – :)2 / N > 3(2Bj – :)2 / N > 3(3Bj – :)2 / N.

Jacob: Why is the word linear in the line: the least squares estimators are the most efficient linear estimators?

Rachel: Other estimators, such as maximum likelihood estimators, may be more efficient, but they are not

Part D: The maximum likelihood estimators are the best estimators. Maximum likelihood is covered in later

modules, not here. If the classical regression assumptions are true, the least squares estimators are also the

Jacob: Part C says that non-linear estimators (like maximum likelihood estimators) may be more efficient than

Rachel: Part C assumes linearity, constant variance, and independence; it does not assume that the error

terms have a normal distribution. Part D assumes the error terms have a normal distribution. Fox says: when

Part E: The least squares estimators are normally distributed.

Jacob: An estimate is a scalar, like B = 1.183 or B = 25,000. B doesn’t have a distribution.

Rachel: Suppose the true $ is 100. We take 10,000 samples of 10 data points each, and for each sample we

compute B. The 10,000 B’s have a normal distribution with a mean of $.

Jacob: Why do we care about the distribution of the ordinary least squares estimators?

Rachel: The standard errors of the estimators, t-values, p-values, and confidence intervals assume a normal

formulas to calculate standard errors of the estimators, t-values, p-values, and confidence intervals are not

** Exercise 11.5: Empirical association vs causal relation

Fox distinguishes between empirical association and causal relation.

If an explanatory variable X2 is omitted from a regression equation, the incomplete regression equation (using

only X1) is biased if the following three conditions are true. Explain each of them.

B. The omitted explanatory variable X2 is a cause of the response variable Y.

C. The omitted explanatory variable X2 is correlated with the explanatory variable X1.

Part A: Suppose the response variable is the annual trend in workers’ compensation loss costs, and the two

explanatory variables are X1 = wage inflation and X2 = medical inflation. Workers’ compensation has two

The proper regression equation is Y = 0.500 X1 + 0.500 X2 + ,j

An actuary uses the regression equation Y = " + $1 X1 + ,j

The least squares estimators are A = 1% and B1 = 0.900.

Question: Is this regression equation biased?

Answer: If the regression equation represents an empirical association, it is correct. Wage inflation (X1) has

Jacob: The population regression parameter " is zero, so shouldn’t A be zero as well?

Rachel: Only 80% of medical inflation affects the regression equation with a $1 parameter but no $2 parameter.

Part B: The second and third conditions are that

! The omitted explanatory variable X2 is a cause of the response variable Y.

! The omitted explanatory variable X2 is correlated with another explanatory variable X1.

Question: Why does this say that X2 is a cause of the response variable Y but is just correlated with X1?

Answer: Suppose we regress workers’ compensation loss cost trends on inflation. Assume (for simplicity) that

causes a 1% higher trend and 1% higher nominal interest rate. In this example, X1 = inflation, X2 = interest

rate, and Y = loss cost trend. X2 is omitted from the regression equation, and it is correlated with Y and X1,

A statistician regresses Y on explanatory variable X1 but does not use a second explanatory variable X2. The

regression line is Y = " + $ X1 + ,

Under what combination of the following conditions is the estimate of $ biased?

A. D(Y, X1) = 0

B. D(Y, X2) = 0

C. D(Y, X1) . 0

D. D(Y, X2) . 0

E. D(X1, X2) = 0

F. D(X1, X2) . 0

G. X1 has a causal effect on Y

H. X2 has a causal effect on Y

I. X1 does not have a causal effect on Y

J. X2 does not have a causal effect on Y

Type of relation: An associative relation gives the empirical relation of X1 and Y. Ignoring other variables is

Causal effect: The nominal interest rate and the inflation rate are correlated. A structural regression of interest

Correlation: A structural regression of interest rates on the money supply growth rate is biased, since the

Jacob: What are observational studies vs experimental studies?

Rachel: Suppose we want to assess the effects of diet and exercise on weight gain or loss.

Jacob: What is the advantage of observational studies?

Rachel: It is relatively easy to record the diet and exercise of each person. Each person records all food and

Jacob: What are the drawbacks of observational studies?

Rachel: The drawbacks are measurement error, spread of the explanatory variables, and bias.

Measurement error: People tend to mis-estimate or ignore personal data. In a study of weight gain, a person

Spread: Most people have similar percentages of carbohydrates, fats, and protein. Observational studies with

Bias: People who eat more are often larger, so they may gain more weight. To hold constant other explanatory

Jacob: For experimental studies, are the X values random variables?

Rachel: For experimental studies, the X values are fixed by design.