14.5 (Optional) Derivation of Linear Regression Line

Goal:

$\bullet$ Learn how to derive the formulas for finding the slope and intercept for linear regression; $\bullet$ Learn how calculus is involved in a statistical concept.

Note that this section requires knowledge of calculus.

Assume we have data $x_1,x_2,\mathrm{\dots },x_n$ from a random variable $X$ and $y_1,y_2,\mathrm{\dots },y_n$ from a random variable $Y$ such that $x_i$ is paired with $y_i$, for $i=1,\mathrm{\dots }n$. Writing these as pairs, we have

$$(x_1,y_1),(x_2,y_2),\mathrm{\dots },(x_n,y_n).$$

The goal is to find $a$ and $b$ in $Y=bX+a$. For each point $(x_i,y_i)$, for $i=1,\mathrm{\dots },n$, we can calculate the error obtained by using such a formula. That is, for each point, we have

$$\text{Error at }i\text{-th point}=e_i=(b{x}_i+a)-y_i.$$

Note that $e_i$ could be positive or negative based on whether the value of $(b{x}_i+a)$ is an overestimate or an underestimate. The line of best fit will be the line that incurs the least amount of error at each point. So we care more about the error being closer to zero than the sign. By squaring the error, we can then remedy this.⁵⁹⁵⁹Absolute value would also do this, but differentiability becomes an issue.

Our problem is now to find the slope and intercept, $b$ and $a$, that minimize the sum of squared errors. That is,

$\underset{(a,b)\in {ℝ}^2}{\min }{\displaystyle \sum _{i=1}^n}{e}_i^2$

$=\underset{(a,b)\in {ℝ}^2}{\min }{\displaystyle \sum _{i=1}^n}{(b{x}_i+a-y_i)}^2$

We do this by letting

$$E(a,b)=\sum _{i=1}^n{(b{x}_i+a-y_i)}^2.$$

Calculus techniques state that we want to find critical values first, then check to see if they satisfy conditions for local extrema. First, we find any critical values. Finding the partials with respect to $a$ and $b$, respectively, we have

$E_a(a,b)=2na+2b{\displaystyle \sum _{i=1}^n}{x}_i-2{\displaystyle \sum _{i=1}^n}{y}_i$

and

$E_b(a,b)=2a{\displaystyle \sum _{i=1}^n}{x}_i+2b{\displaystyle \sum _{i=1}^n}{x}_i^2-2{\displaystyle \sum _{i=1}^n}{x}_i{y}_i.$

Setting each of them equal to zero, we obtain

	$2n{a}^{*}$	$=2b{\displaystyle \sum _{i=1}^n}{x}_i-2{\displaystyle \sum _{i=1}^n}{y}_i$
	$a^{*}$	$={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{y}_i-{\displaystyle \frac{b}{n}}{\displaystyle \sum _{i=1}^n}{x}_i$
	$a^{*}$	$=\overline{y}-b\overline{x}$		(14.5)

and

	$(\overline{y}-b^{*}\overline{x}){\displaystyle \sum _{i=1}^n}{x}_i+b^{*}{\displaystyle \sum _{i=1}^n}{x}_i^2$	$={\displaystyle \sum _{i=1}^n}{x}_i{y}_i$
	$b^{*}\left({\displaystyle \sum _{i=1}^n}{x}_i^2-\overline{x}{\displaystyle \sum _{i=1}^n}{x}_i\right)$	$={\displaystyle \sum _{i=1}^n}{x}_i{y}_i-\overline{y}{\displaystyle \sum _{i=1}^n}{x}_i$
	$b^{*}$	$={\displaystyle \frac{\sum _{i=1}^n{x}_i{y}_i-\overline{y}\sum _{i=1}^n{x}_i}{\sum _{i=1}^n{x}_i^2-\overline{x}\sum _{i=1}^n{x}_i}}$
	$b^{*}$	$={\displaystyle \frac{n\sum _{i=1}^n{x}_i{y}_i-\sum _{i=1}^n{x}_i\sum _{i=1}^n{y}_i}{n\sum _{i=1}^n{x}_i^2-\left(\sum _{i=1}^n{x}_i\right)}}$		(14.6)

So, Eqs. (14.5) and (14.6) are the values of $a$ and $b$ that form a critical value $(a^{*},b^{*})$. Let’s check if this point is a relative minimum. In order for this to happen, we check to see if the following conditions, regarding the second partial derivatives, hold.

• •

  $E_{aa}(a^{*},b^{*})E_{bb}(a^{*},b^{*})-{\left[E_{ab}(a^{*},b^{*})\right]}^2>0$;

• •

  $E_{aa}(a^{*},b^{*})>0$.

Indeed, we have

		$E_{aa}(a,b)=2n$
		$E_{ab}(a,b)=2{\displaystyle \sum _{i=1}^n}{x}_i$
		$E_{bb}(a,b)=2{\displaystyle \sum _{i=1}^n}{x}_i^2.$

Clearly,

$$E_{aa}(a^{*},b^{*})=2n>0.$$

In addition,

	$E_{aa}(a^{*},b^{*})E_{bb}(a^{*},b^{*})-{\left[E_{ab}(a^{*},b^{*})\right]}^2$	$=\left(2n\right)\left(2{\displaystyle \sum _{i=1}^n}{x}_i^2\right)-{\left(2{\displaystyle \sum _{i=1}^n}{x}_i\right)}^2$
		$=4\left(n{\displaystyle \sum _{i=1}^n}{x}_i^2-{\left({\displaystyle \sum _{i=1}^n}{x}_i\right)}^2\right)>0.$

Hence, $(a^{*},b^{*})$ given by Eqs. (14.5) and (14.6), respectively, minimize the squared errors. These formulas are the intercept and slope as in Eqs. (14.3) and (14.2).⁶⁰⁶⁰We leave it to the reader to determine why the last line is positive.