## Probability and Statistics - 古典線性迴歸模型：單變量線性回歸模型 與 高斯－馬爾可夫定理

### 一、迴歸分析簡介 ( Introduction to Regression Analysis)

#### 1. 迴歸分析目的與定義

Regression models involve the following variables:

• The unknown parameters, denoted as β, which may represent a scalar or a vector.
• The independent variables, X.
• The dependent variable, Y.

A regression model relates Y to a function of X and β : $Y \approx f (\mathbf {X}, \boldsymbol{\beta} )$

#### 2. 估計量(Estimator)

• for example, the parameters that describe the relationship between two or more explanatory variables.
• OLS is one choice that many people would consider a good one.
• trade-off between bias and variance in the choice of the estimator.

### 二、單變量線性回歸模型 : 最小平方法 (Ordinary least square estimation, OLSE)

$Y = \alpha + \beta X + \varepsilon$

$\sum_{i = 1}^n \varepsilon_i^2 = \sum_{i = 1}^n (y_i - \alpha - \beta x_i)^2$

$\left\{\begin{array}{lcl} n\ \alpha + \sum\limits_{i = 1}^n x_i\ \beta = \sum\limits_{i = 1}^n y_i \\ \sum\limits_{i = 1}^n x_i\ \alpha + \sum\limits_{i = 1}^n x_i^2\ \beta = \sum\limits_{i = 1}^n x_i y_i \end{array}\right.$

$\hat\beta = \frac {n \sum\limits_{i = 1}^n x_i y_i - \sum\limits_{i = 1}^n x_i \sum\limits_{i = 1}^n y_i} {n \sum\limits_{i = 1}^n x_i^2 - \left(\sum\limits_{i = 1}^n x_i\right)^2} =\frac{\sum\limits_{i = 1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sum\limits_{i = 1}^n(x_i-\bar{x})^2} \,$
$\hat\alpha = \frac {\sum\limits_{i = 1}^n x_i^2 \sum\limits_{i = 1}^n y_i - \sum\limits_{i = 1}^n x_i \sum\limits_{i = 1}^n x_iy_i} {n \sum\limits_{i = 1}^n x_i^2 - \left(\sum\limits_{i = 1}^n x_i\right)^2}= \bar y-\bar x \hat\beta$
$S = \sum\limits_{i = 1}^n (y_i - \hat{y}_i)^2 = \sum\limits_{i = 1}^n y_i^2 - \frac {n (\sum\limits_{i = 1}^n x_i y_i)^2 + (\sum\limits_{i = 1}^n y_i)^2 \sum\limits_{i = 1}^n x_i^2 - 2 \sum\limits_{i = 1}^n x_i \sum\limits_{i = 1}^n y_i \sum\limits_{i = 1}^n x_i y_i } {n \sum\limits_{i = 1}^n x_i^2 - \left(\sum\limits_{i = 1}^n x_i\right)^2}$
$\hat \sigma^2 = \frac {S} {n-2}.$

### 三、古典線性迴歸模型 (Classical Linear Regression Model)

#### 1. 古典線性迴歸模型假設

1. 隨機項的(條件)期望值為零 Zero mean assumption : $E(\varepsilon_i)= 0$
2. 隨機項的變異數皆相同 Homoscedasticity assumption : $Var(\varepsilon_i) = \sigma^2 < \infty$
3. 隨機項無自我相關 Non-autocorrelated assumption : $Cov(\varepsilon_i,\varepsilon_j) = 0$ for $i \neq j$
4. $x_t$ 不是隨機 The $x_t$ are non-stochastic : $Cov(\varepsilon_t, x_t) = 0$
5. 隨機項為常態分佈 The normality assumption : $\varepsilon_i\ \sim\ \mathcal{N}(0,\,\sigma^2)$

$\mbox{E}(Y_i \mid X_i = x_i) = \alpha + \beta x_i \,$

#### 2. 最佳線性無偏估計量 (Best Linear Unbiased Estimators，BLUE)

• 線性估計量 (Linear Estimator)：即這個估計量是隨機變量。
• 不偏估計量 (Unbiased Estimator)：即這個估計量的均值或者期望值 $E(a)$ 等於真實值 $a$。
• 有效估計量 (Efficient Estimators)：在二個不偏估計量中，具有較小變異數(即有較高的精確度與可靠度)者，稱為較有效的估計量。

[用心去感覺] 不偏性比一致性更嚴格

不偏性比一致性更嚴格，所以一般在BLUE只寫不偏性。一致估計量(Consistent Estimators)：若一不偏估計量隨著樣本數的增加而愈接近母體參數，則稱此不偏估計量具有一致性。

#### 3. 高斯－馬爾可夫定理 (Gauss–Markov theorem)

1. 隨機項的(條件)期望值為零 Zero mean assumption : $E(\varepsilon_i)= 0$
2. 隨機項的變異數皆相同 Homoscedasticity assumption : $Var(\varepsilon_i) = \sigma^2 < \infty$
3. 隨機項無自我相關 Non-autocorrelated assumption : $Cov(\varepsilon_i,\varepsilon_j) = 0$ for $i \neq j$
4. $x_t$ 不是隨機 The $x_t$ are non-stochastic : $Cov(\varepsilon_t, x_t) = 0$

[注意] 沒有常態分佈這個條件
The normality assumption : $\varepsilon_i\ \sim\ \mathcal{N}(0,\,\sigma^2)$

### References

wiki - Gauss–Markov theorem
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

wiki - 最小平方法
https://zh.wikipedia.org/wiki/%E6%9C%80%E5%B0%8F%E4%BA%8C%E4%B9%98%E6%B3%95