Linear Regression

Statistics is the math of data. You collect numbers, describe what they show, and then decide how much of the pattern could just be chance.

Linear Regression: the central idea

A least-squares regression line predicts a response from a predictor by minimizing the sum of squared vertical residuals.

Words you need

Least-squares regression line

The least-squares regression line is the line that makes the sum of the squared vertical distances from the points to the line as small as it can possibly be.

Residual

A residual is the observed y value minus the value the line predicted for that same x, so a positive residual means the point sits above the line.

Slope

The slope of a regression line is the predicted change in y for each one-unit increase in x, and it carries the units of y divided by the units of x.

Correlation coefficient

The correlation coefficient, written r, is a number from -1 to 1 that measures how tightly the points hug a straight line and in which direction they tilt.

Coefficient of determination

The coefficient of determination, written r-squared, is the fraction of the up-and-down variation in y that the line accounts for, so 0.60 means the line explains 60% of it.

Extrapolation

Extrapolation is using the line to predict at x values outside the range you actually collected, and it is risky because nothing in the data promises the pattern continues out there.

What to know before this lesson

Know scatterplots, mean and standard deviation, slope, residuals, and the distinction between association and causation.

If one of those prerequisites is uncertain, use the Statistics subject guide to locate the earlier concept before memorizing a procedure.

Linear regression: a worked example

Every step, with the arithmetic

1. Step 1 - Write the paired datahours studied x = 1, 2, 3, 4, 5 and score y = 2, 4, 5, 4, 5
2. Step 2 - Find both meansx-bar = (1+2+3+4+5)/5 = 15/5 = 3 and y-bar = (2+4+5+4+5)/5 = 20/5 = 4
3. Step 3 - Multiply the paired deviations and add(-2)(-2) + (-1)(0) + (0)(1) + (1)(0) + (2)(1) = 4 + 0 + 0 + 0 + 2 = 6
4. Step 4 - Add the squared x deviations(-2)^2 + (-1)^2 + 0^2 + 1^2 + 2^2 = 4 + 1 + 0 + 1 + 4 = 10
5. Step 5 - Divide to get the slopeslope b = 6 / 10 = 0.6 points per hour of study
6. Step 6 - Use the means to get the intercepta = 4 - 0.6 x 3 = 4 - 1.8 = 2.2, so the line is y-hat = 2.2 + 0.6x
7. Step 7 - Predict and subtract to get a residualat x = 3, y-hat = 2.2 + 0.6(3) = 4.0, and the real score was 5, so the residual is 5 - 4.0 = 1.0
8. Step 8 - Measure how much of the variation the line explainssquared residuals 0.64 + 0.36 + 1.00 + 0.36 + 0.04 = 2.40, total squared spread in y = 6, so r-squared = 1 - 2.40/6 = 0.60

A slope of $2.4$ means the predicted response increases by $2.4$ response units for each one-unit increase in the predictor, on average.

Reading a correlation coefficient: what each value of r means for the scatterplot and for prediction

The step-by-step method for linear regression

1. Plot the paired data and check form, direction, unusual points, and whether a linear model is plausible.
2. Fit or calculate the line, then interpret slope and intercept with variables and units.
3. Inspect residuals for pattern and use the model only within a defensible range.

How to check your answer

For every observation compute residual $y-\hat y$; a useful linear fit has residuals scattered around zero without systematic curvature.

Verify that probabilities stay between 0 and 1, measures of spread are nonnegative, and numerical conclusions match the shape, units, and direction visible in the data.

A mistake that changes the mathematics

Correlation and regression do not establish causation, and extrapolation beyond observed predictor values can be unreliable.

Where you will actually use this

Farming

An agronomist regresses corn yield on inches of rainfall to predict harvest size, and reads the slope as bushels gained per extra inch of rain.

Retail planning

A shop fits a line of weekly sales on advertising dollars, then uses the slope to argue that each extra $100 of ads returns a predictable number of sales before deciding next quarter's budget.

Pediatric growth charts

Doctors compare a child's height to a fitted trend of height on age, and treat a large negative residual, not a low height by itself, as the signal worth investigating.

How linear regression connects to the rest of statistics

• Solving linear equations — A regression line is still just y = mx + b, so predicting a y from an x, or solving backward for the x that gives a target y, is ordinary linear-equation work.
• Mean, median, and mode — The regression line always passes through the point built from the mean of x and the mean of y, which is why both means show up in the first step.
• Standard deviation — The slope equals r times the standard deviation of y divided by the standard deviation of x, so spread in both variables sets the steepness.

Try a transfer problem

Create two data sets with the same positive slope but different residual spread, then compare their prediction reliability.

Show the worked answer

Use x = 1, 2, 3, 4, 5 for both sets. Set A: y = 3, 5, 7, 9, 11. Every point lies on y = 2x + 1, so every residual is 0, the sum of squared residuals is 0, and r = 1 with r-squared = 1.00. Set B: y = 5, 3, 5, 11, 11. Its mean y is 35/5 = 7, the paired deviation products add to (-2)(-2) + (-1)(-4) + (0)(-2) + (1)(4) + (2)(4) = 4 + 4 + 0 + 4 + 8 = 20, and the squared x deviations add to 10, so the slope is 20/10 = 2 and the intercept is 7 - 2(3) = 1. Set B fits the very same line y = 2x + 1. But its residuals are 2, -2, -2, 2, 0, giving a sum of squared residuals of 4 + 4 + 4 + 4 + 0 = 16 against a total squared spread of 56, so r-squared = 1 - 16/56 = 0.714 and r is about 0.845. Both models predict 13 when x = 6. Set A's prediction is worth trusting to the decimal, while Set B's typical miss is about 2 units, so its prediction should be quoted as roughly 13 give or take 2. Identical slopes, very different reliability.

Work without copying the example. When finished, use the relevant focused calculator or formula reference to check the setup and result, then correct the first line where your reasoning changed. When the method feels reliable, move to statistics practice questions.

Questions about linear regression

What is the difference between correlation and regression?

Correlation gives you one number, r, that measures how tightly the points follow a line, and it is the same whether you call x the predictor or y the predictor. Regression gives you an actual equation for predicting one specific variable from the other, and swapping which variable you predict generally changes the line.

What does r-squared = 0.60 mean in plain words?

It means the line accounts for 60% of the up-and-down variation in the y values, and the remaining 40% is scatter the line cannot explain. In the worked example the total squared spread in scores was 6 and the leftover squared residuals were 2.40, so 3.60 out of 6, or 60%, was explained.

Why is it called least squares?

Because among all possible lines, this one makes the sum of the squared vertical residuals smallest. Squaring is used so that a miss of +3 and a miss of -3 both count as bad, and so that one huge miss counts far more than several tiny ones.

Can I use the line to predict outside the range of my data?

That is extrapolation and it is unreliable. A line fit to 1 through 5 hours of study says a 20-hour student would score 2.2 + 0.6(20) = 14.2 on a test out of 10, which is impossible. Real relationships flatten out or bend, and your data has no evidence about what happens beyond where you looked.