3.2 Quadratic forms and principal axes

An expression like 5x² + 4xy + 2y² is a symmetric matrix in disguise. Rotating to the eigenvector axes removes the cross term, which classifies every conic and quadric, and decides whether a matrix is positive definite.

In JEE you met $ax^2 + 2hxy + by^2 = 1$ and memorised which signs make an ellipse or a hyperbola. The $xy$ term is the nuisance: it tilts the curve. This chapter removes it for good. Every such expression is $x^{\mathsf T}Ax$ for a symmetric matrix $A$ , and the spectral theorem (3.1) says that in the eigenvector coordinates $A$ is diagonal, so the cross term vanishes. The same move decides whether a critical point of a function is a minimum (5.6) and whether gradient descent behaves (5.10).

Before you start

Chapter 3.1: the spectral theorem, $A = Q\Lambda Q^{\mathsf T}$ .

Quick checkFor a symmetric matrix, the eigenvectors for different eigenvalues are:

Quick checkIf

Q

has orthonormal columns, what is

\lVert Qy \rVert

The picture

Look at what the symmetric map does to the unit square, and turn on the eigenlines: the perpendicular axes along $(1, 1)$ and $(1, -1)$ are the natural axes of the matrix. The ellipse $x^{\mathsf T}Ax = 1$ for this $A$ has its long and short axes along exactly these lines. In coordinates along them the matrix is diagonal, and the ellipse’s equation has no cross term.

Where this came from· Principal axes from Euler to Sylvester

Finding the axes of a tilted ellipse or ellipsoid is the oldest form of this problem. Euler met it for spinning bodies in the 1700s, and in the 1820s Cauchy showed that the same equation gives the principal axes of a quadric surface and the secular equation of planetary astronomy. The next question is what stays the same however you diagonalize. In 1852 James Joseph Sylvester published a proof that, for a real quadratic form, the number of positive, negative and zero coefficients never changes, and he named it the “law of inertia”, the name it still has. Carl Jacobi had found a proof earlier, but it appeared only in 1857, after his death.

Definitions and theorems

Definition A

Quadratic form

A quadratic form on $\mathbb R^n$ is $q(x) = x^{\mathsf T}Ax$ for a symmetric matrix $A$ . In two variables, $q(x, y) = ax^2 + 2bxy + cy^2$ corresponds to $A = \begin{bmatrix}a&b\\b&c\end{bmatrix}$ : the cross-term coefficient is split equally between the two off-diagonal entries.

In one lineA function $q(x) = x^{\mathsf T}Ax = \sum_{i,j} a_{ij}x_ix_j$ with $A$ symmetric: a sum of squares and cross terms.

Why it existsEvery second-degree expression without linear terms is one, and its behaviour is decided entirely by the eigenvalues of $A$ .

Worked example

Write

q(x, y) = 5x^2 + 4xy + 2y^2

x^{\mathsf T}Ax

Solution.

Diagonal entries. The $x^2$ and $y^2$ coefficients: $a_{11} = 5$ , $a_{22} = 2$ . $x^{\mathsf T}Ax$ contributes $a_{11}x^2 + a_{22}y^2$ .
Off-diagonal entries. The cross term is $(a_{12} + a_{21})xy = 4xy$ , and symmetry forces $a_{12} = a_{21} = 2$ . Splitting equally is what makes $A$ symmetric, so that 3.1 applies.

Answer. $A = \begin{bmatrix}5&2\\2&2\end{bmatrix}$ . Check: $\begin{bmatrix}x&y\end{bmatrix}\begin{bmatrix}5&2\\2&2\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix} = 5x^2 + 4xy + 2y^2$ .

Theorem A Apostol Thm 5.11

Principal axes theorem

Let $A = Q\Lambda Q^{\mathsf T}$ (spectral theorem). Substituting $x = Qy$ , that is $y = Q^{\mathsf T}x$ (the coordinates of $x$ in the eigenbasis), gives

x^{\mathsf T}Ax = \lambda_1 y_1^2 + \lambda_2 y_2^2 + \dots + \lambda_n y_n^2.

The change from $x$ to $y$ turns the axes without stretching anything, because $\lVert Qy \rVert = \lVert y \rVert$ . If $\det Q = -1$ the change also flips the picture; negating one eigenvector makes $\det Q = +1$ , a pure rotation.

In one lineRotating to an orthonormal eigenbasis of A turns $x^{\mathsf T}Ax$ into $\lambda_1 y_1^2 + \dots + \lambda_n y_n^2$ , with no cross terms.

Why it is trueSubstitute x = Qy with A = QΛQᵀ: the Q's cancel and only the diagonal Λ is left.

ProofApostol 5.11

The idea: Substitute and let Q^T Q = I cancel.

x^{\mathsf T}Ax = y^{\mathsf T}Q^{\mathsf T}AQ\,y

Substitute

x = Qy

, and

(Qy)^{\mathsf T} = y^{\mathsf T}Q^{\mathsf T}

Q^{\mathsf T}AQ = Q^{\mathsf T}Q\Lambda Q^{\mathsf T}Q = \Lambda

A = Q\Lambda Q^{\mathsf T}

and

Q^{\mathsf T}Q = I

on both sides.

y^{\mathsf T}\Lambda y = \sum_i \lambda_i y_i^2

\blacksquare

A diagonal matrix's quadratic form has no cross terms.

Definition A

Principal axes

The principal axes of $q(x) = x^{\mathsf T}Ax$ are the lines through an orthonormal eigenbasis $q_1, \dots, q_n$ of $A$ .

In one lineThe orthonormal eigenvectors of A: the perpendicular directions in which the quadratic form has no cross terms.

Why it existsThey are the natural axes of the ellipse, ellipsoid or saddle that the form describes.

Worked example

Classify the conic

5x^2 + 4xy + 2y^2 = 1

and find its axes.

Solution. Each step of the step-through below is one numbered stage of the method: matrix, eigenvalues, axes, rotate, read off.

Answer. An ellipse. Its nearest points to the origin lie along $\pm(2, 1)$ at distance $\frac1{\sqrt6} \approx 0.41$ , and its farthest points along $\pm(1, -2)$ at distance $1$ . In general, along a unit eigenvector with eigenvalue $\lambda > 0$ the curve $x^{\mathsf T}Ax = 1$ is at distance $\frac1{\sqrt\lambda}$ , so the biggest eigenvalue gives the nearest points and the smallest gives the farthest.

Quick check

x^2 - 6xy + y^2 = 1

: ellipse or hyperbola?

Worked example Apostol 5.15 style

Find the principal axes of

7x^2 + 6xy - y^2 = 1

and say what curve it is.

Solution.

Matrix. The cross term $6xy$ splits as $3 + 3$ , so $A = \begin{bmatrix}7&3\\3&-1\end{bmatrix}$ . Half the cross coefficient goes in each off-diagonal slot.
Eigenvalues. Trace $6$ , determinant $-7 - 9 = -16$ , so $\lambda^2 - 6\lambda - 16 = (\lambda - 8)(\lambda + 2) = 0$ and $\lambda = 8, -2$ . Characteristic polynomial from trace and determinant (2.8). Opposite signs, because the determinant is negative.
Eigenvectors. For $\lambda = 8$ : $A - 8I = \begin{bmatrix}-1&3\\3&-9\end{bmatrix}$ , null vector $(3, 1)$ . For $\lambda = -2$ : $A + 2I = \begin{bmatrix}9&3\\3&1\end{bmatrix}$ , null vector $(1, -3)$ . Their dot product is $3 - 3 = 0$ , perpendicular as 3.1 promises. Divide each by $\sqrt{10}$ to make them unit vectors.
Rotate. In coordinates $y_1, y_2$ along $(3, 1)/\sqrt{10}$ and $(1, -3)/\sqrt{10}$ the equation is $8y_1^2 - 2y_2^2 = 1$ . Principal axes theorem: the cross term is gone.
Read off. $\frac{y_1^2}{1/8} - \frac{y_2^2}{1/2} = 1$ is a hyperbola. It crosses the line along $(3, 1)$ at distance $\frac1{\sqrt8} \approx 0.35$ and never meets the line along $(1, -3)$ . On the $y_2$ -axis, $y_1 = 0$ would need $-2y_2^2 = 1$ , which has no real solution.

Answer. A hyperbola with axes along $(3, 1)$ and $(1, -3)$ .

Your answerWhat is the largest value of

7x^2 + 6xy - y^2

on the unit circle

x^2 + y^2 = 1

Your answerThe ellipse

3x^2 + 2xy + 3y^2 = 8

has eigenvalues 4 (along

(1,1)

) and 2 (along

(1,-1)

). How long is its half-axis along

(1,1)

Your answerThe curve

2x^2 + 6xy + 5y^2 = 1

has matrix

\begin{bmatrix}2&b\\b&5\end{bmatrix}

. What is

b

Quick checkWhat curve is

x^2 - 2xy + y^2 = 1

Positive definite matrices

Definition A

Positive definite

A symmetric matrix $A$ is positive definite if $x^{\mathsf T}Ax > 0$ for every $x \ne O$ , and positive semidefinite if $x^{\mathsf T}Ax \ge 0$ for every $x$ . Negative (semi)definite is defined with the inequalities reversed, and $A$ is indefinite if $x^{\mathsf T}Ax$ takes both signs.

In one lineA symmetric A is positive definite if $x^{\mathsf T}Ax > 0$ for every $x \ne O$ ; equivalently, all its eigenvalues are positive.

Why it existsIt is the matrix version of 'a positive number': a bowl shape, a minimum, a valid covariance, a convex loss.

Theorem A Apostol 5.12; Strang 6.5

Positive definite iff positive eigenvalues

A real symmetric matrix $A$ is positive definite if and only if all its eigenvalues are positive.

In one lineA symmetric matrix is positive definite exactly when every eigenvalue is positive.

Why it is trueIn principal axes the form is Σλᵢyᵢ², which is positive for every y ≠ 0 exactly when every λᵢ > 0.

ProofStrang 6.5

The idea: In principal-axis coordinates the form is Σ λᵢ yᵢ². (⇐) all positive coefficients give a positive sum when some yᵢ ≠ 0. (⇒) plug in an eigenvector.

x^{\mathsf T}Ax = \sum_i \lambda_i y_i^2

with

y = Q^{\mathsf T}x

Principal axes theorem;

y = Q^{\mathsf T}x

is non-zero exactly when

x

is, since

Q

is invertible.

If every

\lambda_i > 0

and

x \ne O

, the sum is positive.

(

\Leftarrow

) Each term is

\ge 0

and some

y_i \ne 0

gives a positive term.

A

is positive definite, then

0 < q_i^{\mathsf T}Aq_i = q_i^{\mathsf T}(\lambda_i q_i) = \lambda_i

\blacksquare

(

\Rightarrow

) Test the form on a unit eigenvector

q_i

Worked example

A = \begin{bmatrix}2&-1&0\\-1&2&-1\\0&-1&2\end{bmatrix}

positive definite?

Solution.

Use the definition directly. $x^{\mathsf T}Ax = 2x_1^2 + 2x_2^2 + 2x_3^2 - 2x_1x_2 - 2x_2x_3$ . Expand; each off-diagonal pair contributes twice.
Write it as a sum of squares. $= x_1^2 + (x_1 - x_2)^2 + (x_2 - x_3)^2 + x_3^2$ . Expand to check: $x_1^2 + x_1^2 - 2x_1x_2 + x_2^2 + x_2^2 - 2x_2x_3 + x_3^2 + x_3^2$ .
Positive unless $x = O$ . All four squares are 0 only if $x_1 = 0$ , $x_2 = x_1$ , $x_3 = x_2$ , so $x = O$ . A sum of squares is zero only when each square is.

Answer. Positive definite. (Its eigenvalues are $2 - \sqrt2, 2, 2 + \sqrt2$ , all positive.) This matrix is the discrete second derivative, and it appears in every finite-difference simulation of heat flow.

Beyond the course: for understanding· Quick tests: leading minors (Sylvester's criterion)

A symmetric matrix is positive definite iff all its leading principal minors (the determinants of the top-left $1\times1$ , $2\times2$ , …, $n\times n$ blocks) are positive. For $\begin{bmatrix}a&b\\b&c\end{bmatrix}$ : $a > 0$ and $ac - b^2 > 0$ . For the $3\times3$ example: $2$ , $3$ , $4$ , all positive. This is the test behind the 375 second-derivative test (5.6). For two variables you can prove it yourself with trace and determinant (worked example below). The general proof is in Strang 6.5, where the pivots of elimination turn out to be ratios of leading minors.

Worked example

For which

k

A = \begin{bmatrix}1&k\\k&4\end{bmatrix}

positive definite?

Solution.

Reduce to the eigenvalues. By the theorem, we need both eigenvalues positive. That is the equivalence proved above.
Use trace and determinant. For a $2 \times 2$ matrix, the trace is $\lambda_1 + \lambda_2$ and the determinant is $\lambda_1\lambda_2$ . Both eigenvalues are positive exactly when the product is positive and the sum is positive. A positive product means the same sign, and then a positive sum means both are positive. Conversely, two positive numbers have positive sum and product.
Compute. Trace $= 5 > 0$ always. Determinant $= 4 - k^2$ , which is positive when $k^2 < 4$ . The trace does not involve $k$ , so only the determinant restricts $k$ .
Solve. $-2 < k < 2$ . $k^2 < 4$ means $|k| < 2$ .

Answer. Positive definite exactly for $-2 < k < 2$ . At $k = \pm 2$ the determinant is 0 (positive semidefinite, eigenvalues $0$ and $5$ ), and beyond that the matrix is indefinite. This agrees with the leading-minor test: $1 > 0$ and $4 - k^2 > 0$ .

Your answerThe matrix

\begin{bmatrix}3&k\\k&12\end{bmatrix}

is positive definite exactly when

|k|

is less than what number?

Worked example

Prove: for any real matrix

B

, the matrix

B^{\mathsf T}B

is positive semidefinite, and positive definite when the columns of

B

are independent.

Solution.

Check it is symmetric. $(B^{\mathsf T}B)^{\mathsf T} = B^{\mathsf T}(B^{\mathsf T})^{\mathsf T} = B^{\mathsf T}B$ . The definition of positive (semi)definite is for symmetric matrices, so this comes first.
Write the quadratic form. $x^{\mathsf T}B^{\mathsf T}Bx = (Bx)^{\mathsf T}(Bx)$ . $(Bx)^{\mathsf T} = x^{\mathsf T}B^{\mathsf T}$ .
Recognize a length. $(Bx)^{\mathsf T}(Bx) = \lVert Bx\rVert^2 \ge 0$ . A vector dotted with itself is its squared length. This gives semidefinite.
When is it zero? $\lVert Bx\rVert^2 = 0$ only if $Bx = O$ . If the columns are independent, $Bx = O$ forces $x = O$ . That is the definition of independent columns (2.2).
Conclude. For $x \ne O$ the form is strictly positive.

Answer. $B^{\mathsf T}B$ is positive semidefinite always, and positive definite when $B$ has independent columns. $\blacksquare$ Test: $B = \begin{bmatrix}1&1\\1&1\end{bmatrix}$ has dependent columns, $B^{\mathsf T}B = \begin{bmatrix}2&2\\2&2\end{bmatrix}$ , and $x = (1, -1)$ gives $x^{\mathsf T}B^{\mathsf T}Bx = 0$ : semidefinite but not definite.

Worked example

Prove: if

A

is positive definite, then

A

is invertible and

A^{-1}

is positive definite. Test on

A = \begin{bmatrix}2&1\\1&2\end{bmatrix}

Solution.

Eigenvalues. $A = Q\Lambda Q^{\mathsf T}$ with every $\lambda_i > 0$ . The theorem above.
Invertible. $\det A = \lambda_1\cdots\lambda_n > 0$ , so $\det A \ne 0$ . The determinant is the product of the eigenvalues (2.8).
Write the inverse. $A^{-1} = (Q\Lambda Q^{\mathsf T})^{-1} = Q\Lambda^{-1}Q^{\mathsf T}$ . The inverse of a product reverses the order, and $Q^{-1} = Q^{\mathsf T}$ , $(Q^{\mathsf T})^{-1} = Q$ .
Read off its eigenvalues. $\Lambda^{-1} = \operatorname{diag}(\frac1{\lambda_1}, \dots, \frac1{\lambda_n})$ , all positive. The reciprocal of a positive number is positive. This is again a spectral decomposition, so these are the eigenvalues of $A^{-1}$ .
Symmetric. $(A^{-1})^{\mathsf T} = (A^{\mathsf T})^{-1} = A^{-1}$ , so by the theorem $A^{-1}$ is positive definite.
Test. $A^{-1} = \frac13\begin{bmatrix}2&-1\\-1&2\end{bmatrix}$ , with eigenvalues $\frac13$ and $1$ , the reciprocals of $3$ and $1$ .

Answer. $A^{-1}$ exists and is positive definite, with eigenvalues $\frac1{\lambda_i}$ . $\blacksquare$

Quick checkWhich matrix is positive definite?

Quick checkIf

A

is positive definite, what can you say about its diagonal entries

a_{ii}

Proof workshopRebuild: positive eigenvalues imply positive definite

Tap the lines in the order a proof would write them. Each line can only use what is already above it.

Where it is used

Used inside maths· Inside maths: maxima, minima and saddles

Near a critical point, Taylor’s theorem in two variables (5.6) says $f(a + h) \approx f(a) + \frac12 h^{\mathsf T}Hh$ , where $H$ is the symmetric Hessian matrix of second derivatives. Positive definite $H$ : a minimum. Negative definite: a maximum. Indefinite: a saddle. The 375 test ” $f_{xx} > 0$ and $f_{xx}f_{yy} - f_{xy}^2 > 0$ means a minimum” is Sylvester’s criterion for $2\times2$ .

Used in the real world· Engineering and data: energy, stiffness and covariance

In a structure of springs and beams, the stored energy is $\frac12 x^{\mathsf T}Kx$ where $K$ is the symmetric stiffness matrix; $K$ positive definite means every displacement costs energy, so the structure is stable. Finite-element software checks this before solving. In statistics, a covariance matrix is always positive semidefinite, because the variance of any combination $w^{\mathsf T}X$ of the variables is $w^{\mathsf T}\Sigma w \ge 0$ . Its principal axes are the principal components of 3.4.

Traps

Watch out· Split the cross term in half

$ax^2 + 2bxy + cy^2$ has matrix $\begin{bmatrix}a&b\\b&c\end{bmatrix}$ , not $\begin{bmatrix}a&2b\\0&c\end{bmatrix}$ . The non-symmetric version gives the same $q$ but wrong eigenvalues.

Watch out· Positive entries do not mean positive definite

$\begin{bmatrix}1&2\\2&1\end{bmatrix}$ has all positive entries and eigenvalues $3, -1$ : indefinite. Check eigenvalues or minors, never entries.

Problems

check

Write the matrix of $q(x, y, z) = x^2 + 3y^2 - z^2 + 4xz$ .

Solution

$\begin{bmatrix}1&0&2\\0&3&0\\2&0&-1\end{bmatrix}$ .

check

Classify by eigenvalue signs: (a) $\operatorname{diag}(3, 1)$ ; (b) $\operatorname{diag}(3, -1)$ ; (c) $\operatorname{diag}(3, 0)$ .

Solution

(a) positive definite, ellipse. (b) indefinite, hyperbola. (c) positive semidefinite; $3x^2 = 1$ is a pair of parallel lines.

useApostol 5.15 style

Find the principal axes of $3x^2 + 2xy + 3y^2 = 8$ and its half-axis lengths.

Solution

$A = \begin{bmatrix}3&1\\1&3\end{bmatrix}$ , eigenvalues $4$ along $(1,1)$ and $2$ along $(1,-1)$ . In those coordinates $4y_1^2 + 2y_2^2 = 8$ , i.e. $\frac{y_1^2}{2} + \frac{y_2^2}{4} = 1$ : half-axes $\sqrt2$ along $(1,1)$ and $2$ along $(1,-1)$ .

use

For which $k$ is $\begin{bmatrix}2&k\\k&8\end{bmatrix}$ positive definite?

Solution

Leading minors: $2 > 0$ and $16 - k^2 > 0$ , so $-4 < k < 4$ .

prove

Prove that $A^{\mathsf T}A$ is positive semidefinite for any matrix $A$ , and positive definite when $A$ has independent columns.

Solution

$x^{\mathsf T}A^{\mathsf T}Ax = (Ax)^{\mathsf T}(Ax) = \lVert Ax \rVert^2 \ge 0$ . If the columns are independent, $Ax = O$ only when $x = O$ , so the form is positive for $x \ne O$ . (This is why the normal equations of 2.10 have a unique solution.)

prove

Prove that a positive definite matrix is invertible, and that its inverse is positive definite.

Solution

Eigenvalues all positive, so none is 0 and $\det A = \prod \lambda_i \ne 0$ . $A^{-1} = Q\Lambda^{-1}Q^{\mathsf T}$ has eigenvalues $\frac1{\lambda_i} > 0$ , so it is positive definite.

stretch

Show that on the unit circle the quadratic form $q(x) = x^{\mathsf T}Ax$ has maximum $\lambda_{\max}$ and minimum $\lambda_{\min}$ . Use it to find the points of the ellipse $x^2 + xy + y^2 = 1$ closest to and farthest from the origin.

Solution

In principal axes $q = \sum \lambda_i y_i^2$ with $\sum y_i^2 = 1$ , so $\lambda_{\min} \le q \le \lambda_{\max}$ , attained at the eigenvectors. For the ellipse, $A = \begin{bmatrix}1&\frac12\\ \frac12&1\end{bmatrix}$ has eigenvalues $\frac32$ along $(1,1)$ and $\frac12$ along $(1,-1)$ . A point at distance $r$ in the unit direction $u$ has $r^2 q(u) = 1$ , so $r = 1/\sqrt{q(u)}$ . Closest: along $(1,1)$ at $r = \sqrt{2/3}$ , the points $\pm\left(\frac1{\sqrt3}, \frac1{\sqrt3}\right)$ . Farthest: along $(1,-1)$ at $r = \sqrt2$ , the points $\pm(1, -1)$ . Check: $1 - 1 + 1 = 1$ .

Summary

A quadratic form $x^{\mathsf T}Ax$ uses a symmetric $A$ ; the cross-term coefficient is split equally.
Principal axes: rotating to an orthonormal eigenbasis turns the form into $\sum \lambda_i y_i^2$ , with no cross terms.
Conics: same-sign eigenvalues give an ellipse, opposite signs a hyperbola. The JEE rule is $\det A > 0$ .
Positive definite means $x^{\mathsf T}Ax > 0$ for $x \ne O$ , equivalently all eigenvalues positive (or all leading minors positive).
This decides minima (Hessians), stability (stiffness matrices) and covariance structure.

Watch and read

The same ideas explained by someone else. Use these when an explanation here does not land.

3.2 Quadratic forms and principal axes

The picture

Definitions and theorems

Quadratic form

Principal axes theorem

Principal axes

Positive definite matrices

Positive definite

Positive definite iff positive eigenvalues

Where it is used

Traps

Problems

Summary

Watch and read

Where this sits

Comes from

Leads to

Same idea elsewhere