Normal equations
Find a vector whose model prediction is closest in squared distance to data.
Normal equations is one of 1 least squares formula in the linear algebra section of this library, and it is used at university level.
Why normal equations works
When Ax = b has no solution, b sticks out of the space the columns of A can reach. The closest reachable point is the shadow of b on that space, and the error left over must be perpendicular to every column. Writing that perpendicularity as Aᵀ(b − Ax) = 0 and rearranging gives exactly these equations.
What each symbol means
$\widehat{\mathbf x}$ is the least-squares solution.
Normal equations: when it holds
A unique solution is guaranteed when columns of $A$ are linearly independent; QR is often numerically safer.
When it stops applying
If two columns of A are multiples of each other, AᵀA is singular and there is no unique answer, only a whole family of equally good fits. Even with independent columns the step of forming AᵀA squares the conditioning, so near-duplicate columns lose accuracy fast and a QR factorization is the safer route.
Normal equations: a worked example
The residual $\mathbf b-A\widehat{\mathbf x}$ is orthogonal to every column of $A$.
The mistake to avoid
What people do: Students see Aᵀ on both sides and cancel it, dropping back to Ax = b.
Why it goes wrong: You can only cancel a factor that has an inverse, and Aᵀ is usually rectangular with no inverse at all. Cancelling it asks for an exact solution, which is the thing that does not exist.
Do this instead: Multiply out both sides and solve the smaller square system instead. Fitting a line through (1, 1), (2, 2), and (3, 2) gives AᵀA = [[3, 6], [6, 14]] and Aᵀb = (5, 11), whose solution is intercept 0.667 and slope 0.5.
Normal equations: step by step
- Name the unknown, and the unit the answer has to come out in.
- Match the symbols to your values. $\widehat{\mathbf x}$ is the least-squares solution.
- Check the conditions before substituting. A unique solution is guaranteed when columns of $A$ are linearly independent; QR is often numerically safer.
- Substitute, keep exact values to the last line, then test the sign, size, and unit against a rough estimate — the check that catches most linear algebra slips.
Where this formula fits
- Subject
- Linear Algebra formulas — 17 entries in this library
- Topic
- Least squares
- Level
- University
Formulas are easiest to keep when they sit inside a method rather than on a list. Use the links below to see where normal equations comes from, to check a calculation against a tool, and to practise it until you can recall it without looking.
- Solving Linear Systems with Matrices — the lesson behind this formula: use row operations to reveal solutions and dependencies.
- Linear Algebra Calculator — check your substitution and the value it produces.
- Study linear algebra — the subject guide that explains the ideas these formulas compress.
- Linear Algebra Practice — questions that make you retrieve the formula instead of recognising it.
- All 17 linear algebra formulas — the full grouped reference, or the complete formula library.
Questions about normal equations
Why can I not just solve Ax = b directly?
Because with more data points than unknowns there is usually no line passing through every point. The system is inconsistent, so you switch the goal from exact to closest.
Should the residual come out as zero?
Only in the lucky case where the data really does fit exactly. Normally the residual is nonzero, and what makes the answer the best one is that this leftover is perpendicular to every column of A.
What does AᵀA being singular tell me about my data?
That your predictors carry duplicate information, such as one column being a constant multiple of another or a sum of two others. Drop or combine the redundant column and the fit becomes unique.
How big is the system I end up solving?
AᵀA is square with one row and column per unknown, no matter how many data points you have. Fitting a straight line to a million points still leaves you with just a 2 by 2 system.