LECTURE NOTE, DAY 19
ESTIMATION IN MULTIPLE REGRESSION
& GOODNESS OF FITS
ECO 3007, 2016 SPRING INSTRUCTOR : JUNGMO YOON
HANYANG UNIVERSITY
6.3. The OLS Estimator in Multiple Regression
Read pages 189 - 192. To present the materials in this section well, we will need matrix algebra. I will spend some time to review matrix notations.
6.4. Measures of Fit
Read pages 193 - 197. The standard error of the regression (SER) estimates the standard deviation of the error u.
SER = q
s2uˆ where
s2uˆ = 1 n − k − 1
n
X
i=1
ˆ
u2i = SSR n − k − 1.
Here k is the number of independent variables. The regression R2 is defined by
R2 = ESS
T SS = 1 − SSR T SS
where ESS =P( ˆYi− ¯Y )2, T SS =P(Yi− ¯Y )2, and SSR =P(Yi− ˆYi)2. A problem with R2
Whenever you add additional regressors, you always increase R2. It is easy to manipulate results. By adding (possibly meaningless) regressors, you can always
1
2 ECO 3007, 2016 SPRING INSTRUCTOR : JUNGMO YOON HANYANG UNIVERSITY
make your regression results look better. Many people do this, so be careful and don’t be fooled.
Example) Consider a minimisation problem,
minβ0,β1
n
X
i=1
(Yi− β0− β1X1i)2
Once we find solutions, ( ˆβ0, ˆβ1), the SSR of this problem is defined by SSR1 = Pn
i=1(Yi− ˆβ0− ˆβ1X1i)2.
Now suppose you add another regressor X2 and solve the following minimization problem,
β0min,β1,β2
n
X
i=1
(Yi− β0− β1X1i− β2X2i)2 then the SSR of the second problem is SSR2 =Pn
i=1(Yi− ˆβ0− ˆβ1X1i− ˆβ2X2i)2. My claim is that
SSR1 ≥ SSR2, all the time
that is, the sum of squared errors of the second problem is always smaller than the sum of squared errors of the first problem. Why?
Think about this. When you approach the second problem, the least thing you can do is to set β2 = 0, then the minimum value of the second problem is at least as small as that of the first problem. But we can do better than this by trying to find optimum values of β2.
Now think about the definition, R2 = 1 − SSRT SS. Note that T SS, the variation in Y , is fixed. But by adding Xs, you can always decrease SSR, therefore, R2 will be increased.
The adjusted R2
When we add one more X in a regression model, it will (i) decrease the prediction error (SSR) but (ii) increase the complexity of the model. We need to find the right balance between two competing objectives. The adjusted R2 try to achieve this. Its definition is
LECTURE NOTE, DAY 19 ESTIMATION IN MULTIPLE REGRESSION & GOODNESS OF FITS3
R¯2 = 1 − n − 1
n − k − 1 · SSR T SS
Note that as we add one more regressor, it decrease SSR, but increase n−k−1n−1 . There- fore, addition of extra regressors does not necessarily increase ¯R2.
Some observations.
(i) ¯R2 = 1 − s2uˆ/s2Y. It is because R¯2 = 1 − n − 1
n − k − 1 ·SSR
T SS = SSR/(n − k − 1)
T SS/(n − 1) = 1 − s2uˆ s2Y (ii) R2 > ¯R2 because 1 < n−k−1n−1 .
(iii) ¯R2 can be negative because
R¯2 = 1 − n − 1 n − k − 1
| {z }
A
·SSR T SS
| {z }
B
and 0 ≤ B ≤ 1, A > 1, so you never know if it will be positive or not.
