3.1 Symmetric matrices and the spectral theorem

A symmetric matrix has real eigenvalues and perpendicular eigenvectors, so it is nothing but a stretch along perpendicular axes. That is the spectral theorem, the most useful theorem in linear algebra, and Apostol's chapter 5 that MATH 375 only touched.

Chapter 2.8 showed that diagonalizing can fail: a shear has too few eigenvectors, and a rotation has no real ones. This chapter finds a large class of matrices for which nothing can go wrong. If a real matrix equals its own transpose, its eigenvalues are real, its eigenvectors can be chosen perpendicular, and it is diagonalized by a rotation of axes. Symmetric matrices appear everywhere: second derivatives, covariance in statistics, stress in materials, energy in physics. That is why this theorem is used more than any other in linear algebra.

Before you start

Chapter 2.8 (eigenvalues, diagonalization) and 2.9 (inner products, orthonormal bases, orthogonal complements).

Quick checkFor the dot product on

\mathbb R^n

, which identity is true for every matrix

A

Quick checkWhich matrix has no real eigenvectors?

The picture

The symmetric matrix $\begin{bmatrix}2&1\\1&2\end{bmatrix}$ stretches by 3 along $(1, 1)$ and by 1 along $(1, -1)$ , and these two directions are perpendicular. Compare with “Eigen example” (not symmetric): its eigenlines are not perpendicular. And “Rotate 90°” has none. A symmetric matrix is always the clean case: stretch along perpendicular axes, nothing else.

Where this came from· From the axes of a spinning body to Hilbert's 'spectrum'

Every rigid body, however lumpy, has three perpendicular axes about which it can spin steadily: the principal axes. Leonhard Euler worked them out in his mechanics of rigid bodies in the 1700s. In matrix language they are the eigenvectors of a symmetric matrix, the inertia tensor. Astronomers met the same mathematics from another side: to find out whether planetary orbits stay stable, they needed the roots of a “secular equation”, and in 1829 Augustin-Louis Cauchy showed that those roots are real when the underlying quadratic form is symmetric. That is our first theorem. Between about 1904 and 1910 David Hilbert, studying integral equations, called the set of eigenvalues the “spectrum”, with no physics in mind. When quantum mechanics arrived in the 1920s, the spectral lines of atoms turned out to be eigenvalues of Hermitian operators, so his word fitted better than he could have known.

Definitions

Chapter 2.9 defined inner products over the real numbers. The first theorem below needs eigenvalues that might be complex, so we need the complex version too. For complex vectors $x = (x_1, \dots, x_n)$ and $y = (y_1, \dots, y_n)$ define

(x, y) = x_1\overline{y_1} + \cdots + x_n\overline{y_n},

where the bar is the conjugate of 0.6. Three facts are all we use. It is linear in the first slot: $(cx, y) = c\,(x, y)$ . It is conjugate-linear in the second: $(x, cy) = \bar c\,(x, y)$ , because the bar acts on $c$ . And $(x, x) = |x_1|^2 + \cdots + |x_n|^2$ is a positive real number when $x \ne O$ . The bar is there for the third fact: without it, $(x, x)$ for $x = (i)$ would be $i^2 = -1$ , and lengths would break. For real vectors the bars do nothing and this is the dot product.

Definition A

Adjoint

Let $V$ be a finite-dimensional inner product space. The adjoint of a linear map $A: V \to V$ is the linear map $A^*$ with

(Ax, y) = (x, A^*y) \quad\text{for all } x, y \in V.

For $\mathbb R^n$ with the dot product, $A^* = A^{\mathsf T}$ . For $\mathbb C^n$ with $(x, y) = \sum x_i \overline{y_i}$ , $A^* = \overline{A}^{\mathsf T}$ , the conjugate transpose.

In one lineThe adjoint $A^*$ is the map with $(Ax, y) = (x, A^*y)$ for all $x, y$ ; for real matrices with the dot product it is the transpose.

Why it existsIt moves a map from one side of an inner product to the other, which is the step every proof in this chapter uses.

Spagnolie put it this way in Lecture 19: in general, “transpose” is whatever operation makes $(v, Aw) = (A^{\mathsf T}v, w)$ true. For real matrices the check is index bookkeeping: $(Ax) \cdot y = \sum_{i,j} a_{ij} x_j y_i = x \cdot (A^{\mathsf T}y)$ .

Definition T

Symmetric operator

A linear map $A$ on a real inner product space is symmetric (Apostol: Hermitian when complex, self-adjoint in general) if $A^* = A$ , that is, $(Ax, y) = (x, Ay)$ for all $x, y$ . A real matrix is symmetric when $A^{\mathsf T} = A$ : entry $a_{ij} = a_{ji}$ .

In one lineA real linear map with $A^* = A$ , that is $(Ax, y) = (x, Ay)$ ; for a matrix, $A^{\mathsf T} = A$ .

Why it existsThese are exactly the maps that can be diagonalized by perpendicular axes.

Definition A

Hermitian

A complex matrix is Hermitian if $\overline{A}^{\mathsf T} = A$ . A real symmetric matrix is Hermitian, since conjugating real numbers changes nothing.

In one lineA complex matrix equal to its conjugate transpose: $a_{ij} = \overline{a_{ji}}$ .

Why it existsIt is the complex version of symmetric, and the setting of quantum mechanics.

Definition A

Orthogonally diagonalizable

A real square matrix $A$ is orthogonally diagonalizable if $A = Q \Lambda Q^{\mathsf T}$ where $\Lambda$ is diagonal and $Q$ is an orthogonal matrix: its columns form an orthonormal basis, so $Q^{\mathsf T}Q = I$ and $Q^{-1} = Q^{\mathsf T}$ . Equivalently, $\mathbb R^n$ has an orthonormal basis of eigenvectors of $A$ (the columns of $Q$ ).

In one line $A = Q\Lambda Q^{\mathsf T}$ with $\Lambda$ diagonal and $Q$ having orthonormal columns: diagonalized by a rotation or reflection of axes.

Why it existsThen $Q^{-1} = Q^{\mathsf T}$ costs nothing to compute, and the eigen-directions are perpendicular.

The three facts

Theorem T Apostol Thm 5.2; Spagnolie Lecture 19

Symmetric matrices have real eigenvalues

If $A$ is Hermitian and $Ax = \lambda x$ with $x \ne O$ (allowing complex $\lambda$ and $x$ ), then $\lambda$ is real.

In one lineEvery eigenvalue of a Hermitian (in particular real symmetric) matrix is a real number.

Why it is true(Ax, x) = λ(x, x) and also (x, Ax) = conj(λ)(x, x); symmetry makes these equal, so λ equals its own conjugate.

ProofSpagnolie Lecture 19 (corrected), Apostol 5.2

The idea: Compute (Ax, x) in two ways: pull λ out of the first slot, and move A across with symmetry and pull λ out of the second slot, where it comes out conjugated. Compare.

(Ax, x) = (\lambda x, x) = \lambda (x, x)

Use

Ax = \lambda x

; an inner product is linear in its first slot.

(Ax, x) = (x, Ax) = (x, \lambda x)

A

is Hermitian, so

(Ax, x) = (x, Ax)

(x, \lambda x) = \bar\lambda (x, x)

A complex inner product is conjugate-linear in its second slot:

(x, cy) = \bar c(x, y)

.Spagnolie's notes write y where x is needed here; the argument uses the same vector x in both slots.

\lambda (x, x) = \bar\lambda (x, x)

The two computations equal the same number.

\lambda = \bar\lambda

, so

\lambda

is real.

\blacksquare

(x, x) = \lVert x \rVert^2 > 0

because

x \ne O

, so we may divide.

Theorem T Apostol Thm 5.4; Spagnolie Lecture 19

Eigenvectors of a symmetric matrix are orthogonal

If $A$ is symmetric, $Ax = \lambda x$ and $Ay = \mu y$ with $\lambda \ne \mu$ , then $(x, y) = 0$ .

In one lineEigenvectors of a symmetric matrix for different eigenvalues are perpendicular.

Why it is true(Ax, y) gives λ(x, y) and, moving A across, μ(x, y); with λ ≠ μ the only way out is (x, y) = 0.

ProofSpagnolie Lecture 19

The idea: Compute (Ax, y) two ways: as λ(x, y), and by moving A to y as μ(x, y). Subtract.

(Ax, y) = \lambda (x, y)

Ax = \lambda x

(Ax, y) = (x, Ay) = \mu (x, y)

Symmetry, then

Ay = \mu y

(and

\mu

is real by the previous theorem, so no conjugate appears).

(\lambda - \mu)(x, y) = 0

Subtract the two lines.

Hence

(x, y) = 0

\blacksquare

\lambda \ne \mu

, so the first factor is not zero.

Proof workshopGive the reason for each line: eigenvectors are orthogonal

Each line of the proof is shown in order. Choose the reason that justifies it, then press Check.

$(Ax, y) = \lambda (x, y)$
$(Ax, y) = (x, Ay)$
$(x, Ay) = \mu (x, y)$
$(\lambda - \mu)(x, y) = 0$ , so $(x, y) = 0$

Theorem A Apostol Thm 5.6 and 5.8; Axler 7.29

Spectral theorem

Let $A$ be a real symmetric $n \times n$ matrix. Then $\mathbb R^n$ has an orthonormal basis $q_1, \dots, q_n$ of eigenvectors of $A$ . Equivalently, $A = Q \Lambda Q^{\mathsf T}$ with $Q = [q_1 \cdots q_n]$ orthogonal and $\Lambda = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$ real. The theorem holds even when eigenvalues repeat.

In one lineEvery real symmetric matrix is orthogonally diagonalizable: $A = Q\Lambda Q^{\mathsf T}$ , with real eigenvalues and an orthonormal basis of eigenvectors.

Why it is trueFind one unit eigenvector; the space perpendicular to it is mapped into itself, so the same argument repeats there, one dimension down.

Why· The picture behind the proof

Find one eigenvector $q_1$ of length 1. Look at the plane (or hyperplane) $W$ perpendicular to it. A symmetric matrix maps $W$ into itself: if $w \perp q_1$ , then $Aw \perp q_1$ too. So inside $W$ you have a smaller symmetric problem, one dimension down, and the same argument finds the next eigenvector, perpendicular to the first. Repeating peels off one perpendicular axis at a time.

ProofApostol 5.6, Axler 7.29

The idea: Induction on n. A real eigenvalue exists (complex eigenvalues always exist and are real here). Take a unit eigenvector q₁. The orthogonal complement W = q₁⊥ is invariant under A, and A restricted to W is symmetric. By induction W has an orthonormal eigenbasis; add q₁.

Base case

n = 1

: true. Assume the theorem for symmetric maps on spaces of dimension

n - 1

Induction on the dimension

n

. For

n = 1

every unit vector is an eigenvector of the

1 \times 1

matrix.

A

has a real eigenvalue

\lambda_1

with a real eigenvector; scale it to length 1 and call it

q_1

The characteristic polynomial has a complex root (fundamental theorem of algebra, 0.6), and a real symmetric matrix is Hermitian, so by the first theorem that root is real. Then

A - \lambda_1 I

is a real matrix with determinant 0, so the real system

(A - \lambda_1 I)x = O

has a non-zero real solution (2.8).

Let

W = \{w : (w, q_1) = 0\}

The vectors perpendicular to

q_1

form a subspace of dimension

n - 1

(orthogonal complement, 2.9).

w \in W

, then

Aw \in W

W

is invariant under

A

Symmetry moves

A

across:

(Aw, q_1) = (w, Aq_1) = \lambda_1 (w, q_1) = 0

.This is the step that needs symmetry. For a non-symmetric matrix W need not be mapped into itself.

A

restricted to

W

is a symmetric map on the

(n - 1)

-dimensional inner product space

W

The identity

(Ax, y) = (x, Ay)

holds for all vectors, in particular those in

W

W

has an orthonormal basis

q_2, \dots, q_n

of eigenvectors of

A

Induction hypothesis.

q_1, q_2, \dots, q_n

is an orthonormal basis of eigenvectors of

A

\blacksquare

q_1

is a unit vector perpendicular to all of

W

, and

n

orthonormal vectors in

\mathbb R^n

form a basis (orthonormal sets are independent, 2.9).

Proof workshopRebuild the spectral theorem proof

Tap the lines in the order a proof would write them. Each line can only use what is already above it.

Same idea elsewhere· Why A = QΛQᵀ is the same statement

With $Q$ having columns $q_i$ : $AQ = [\lambda_1 q_1 \cdots \lambda_n q_n] = Q\Lambda$ . Orthonormal columns give $Q^{\mathsf T}Q = I$ , so $Q^{-1} = Q^{\mathsf T}$ , and $A = Q\Lambda Q^{\mathsf T}$ . This is the change of basis of 2.5 with the cheapest possible inverse, the transpose.

Quick checkIn the spectral theorem's proof, where is symmetry used?

Examples

Worked example

Orthogonally diagonalize

A = \begin{bmatrix}2&1\\1&2\end{bmatrix}

Solution.

Eigenvalues. $\det(A - \lambda I) = (2 - \lambda)^2 - 1 = (\lambda - 3)(\lambda - 1)$ . So $\lambda = 3, 1$ . Characteristic polynomial (2.8). Both real, as the theorem promises.
Eigenvectors. $\lambda = 3$ : $(A - 3I)x = O$ gives $x = (1, 1)$ . $\lambda = 1$ : $x = (1, -1)$ . Solve each null space.
Check perpendicular, then normalize. $(1,1) \cdot (1,-1) = 0$ . Unit vectors: $q_1 = \frac1{\sqrt2}(1, 1)$ , $q_2 = \frac1{\sqrt2}(1, -1)$ . Different eigenvalues, so perpendicular automatically; divide by the length to get an orthonormal basis.
Assemble. $Q = \frac1{\sqrt2}\begin{bmatrix}1&1\\1&-1\end{bmatrix}$ , $\Lambda = \begin{bmatrix}3&0\\0&1\end{bmatrix}$ , and $A = Q\Lambda Q^{\mathsf T}$ . Columns of $Q$ in the same order as the eigenvalues in $\Lambda$ .

Answer. $A$ stretches by 3 along the diagonal $y = x$ and leaves the perpendicular diagonal unchanged: the picture at the top.

Worked example

Spagnolie's Lecture 19 example: diagonalize

D = \begin{bmatrix}2/3&1&0\\1&2/3&0\\0&0&-4/3\end{bmatrix}

and explain its trace.

Solution.

Use structure. $D = A - \frac43 I$ with $A = \begin{bmatrix}2&1&0\\1&2&0\\0&0&0\end{bmatrix}$ , whose eigenvalues are $3, 1, 0$ (the $2\times2$ block from the previous example, and 0). Subtracting $cI$ shifts every eigenvalue by $c$ and keeps eigenvectors.
Eigenvalues of $D$ . $3 - \frac43 = \frac53$ , $1 - \frac43 = -\frac13$ , $0 - \frac43 = -\frac43$ . Shift each by $-\frac43$ .
Eigenvectors. $(1, 1, 0)$ , $(1, -1, 0)$ , $(0, 0, 1)$ , mutually perpendicular; divide the first two by $\sqrt 2$ . Same as for $A$ .
Trace. $\frac53 - \frac13 - \frac43 = 0 = \operatorname{tr} D$ . Trace = sum of eigenvalues (2.8). Spagnolie’s point: zero trace means the flow below preserves volume.

Answer. $D = Q\Lambda Q^{\mathsf T}$ with $\Lambda = \operatorname{diag}(\frac53, -\frac13, -\frac43)$ . In the fluid flow, a ball of fluid stretches fastest along $(1, 1, 0)$ .

Worked example

Show

B = \begin{bmatrix}1&2\\0&3\end{bmatrix}

is diagonalizable but NOT orthogonally diagonalizable.

Solution.

Eigenvalues. Triangular, so $1$ and $3$ , distinct, so $B$ is diagonalizable (2.8). Distinct eigenvalues give independent eigenvectors.
Eigenvectors. $\lambda = 1$ : $(1, 0)$ . $\lambda = 3$ : $(1, 1)$ . Solve $(B - \lambda I)x = O$ .
Angle. $(1,0) \cdot (1,1) = 1 \ne 0$ , not perpendicular, and each eigenspace is a single line, so no other choice exists. An orthonormal eigenbasis would need perpendicular eigenlines.

Answer. Diagonalizable, but not by perpendicular axes. In fact a real matrix is orthogonally diagonalizable exactly when it is symmetric (problem below), and $B \ne B^{\mathsf T}$ .

Python labSymmetric versus not symmetric, on a computer

import numpy as np

A = np.array([[2.0, 1.0], [1.0, 2.0]])
w, Q = np.linalg.eigh(A)          # eigh is for symmetric matrices
print("eigenvalues:", w)
print("Q^T Q =", np.round(Q.T @ Q, 12).tolist())
print("Q diag(w) Q^T =", np.round(Q @ np.diag(w) @ Q.T, 12).tolist())

B = np.array([[1.0, 2.0], [0.0, 3.0]])   # not symmetric
w2, V = np.linalg.eig(B)
print("non-symmetric eigenvectors dot product:", round(float(V[:, 0] @ V[:, 1]), 4))

Try this. Run it. For the symmetric matrix, eigh returns an orthogonal $Q$ (so $Q^{\mathsf T}Q = I$ ) and $Q\Lambda Q^{\mathsf T}$ rebuilds $A$ . For the non-symmetric matrix, the two eigenvectors have dot product $0.7071$ , not 0: they are not perpendicular. Change B to [[1.0, 2.0], [2.0, 3.0]] (symmetric) and the dot product becomes 0 up to rounding.

Your answerWhat is the larger eigenvalue of the symmetric matrix

\begin{bmatrix}5&2\\2&2\end{bmatrix}

Worked example

Compute

A^{10}

for

A = \begin{bmatrix}2&1\\1&2\end{bmatrix}

using

A = Q\Lambda Q^{\mathsf T}

Solution.

Use the decomposition from the first example. $A = Q\Lambda Q^{\mathsf T}$ with $\Lambda = \operatorname{diag}(3, 1)$ and $Q = \frac1{\sqrt2}\begin{bmatrix}1&1\\1&-1\end{bmatrix}$ . We already have the eigen-data, so no new eigenvalue work is needed.
Collapse the product. $A^2 = Q\Lambda Q^{\mathsf T}Q\Lambda Q^{\mathsf T} = Q\Lambda^2 Q^{\mathsf T}$ , because $Q^{\mathsf T}Q = I$ sits in the middle. The same cancellation repeats, so $A^{10} = Q\Lambda^{10}Q^{\mathsf T}$ . Each neighbouring pair $Q^{\mathsf T}Q$ becomes $I$ . This is why diagonalizing makes powers cheap.
Power the diagonal matrix entry by entry. $\Lambda^{10} = \operatorname{diag}(3^{10}, 1^{10}) = \operatorname{diag}(59049, 1)$ . A diagonal matrix multiplies each diagonal entry by itself.
Multiply back. $Q\Lambda^{10}Q^{\mathsf T} = \frac12\begin{bmatrix}1&1\\1&-1\end{bmatrix}\begin{bmatrix}59049&0\\0&1\end{bmatrix}\begin{bmatrix}1&1\\1&-1\end{bmatrix} = \frac12\begin{bmatrix}59050&59048\\59048&59050\end{bmatrix}$ . The factor $\frac12$ is $\left(\frac1{\sqrt2}\right)^2$ , from the two copies of $Q$ .

Answer. $A^{10} = \begin{bmatrix}29525&29524\\29524&29525\end{bmatrix}$ . Check with $A^2 = \begin{bmatrix}5&4\\4&5\end{bmatrix}$ : the formula $\frac12\begin{bmatrix}3^2+1&3^2-1\\3^2-1&3^2+1\end{bmatrix}$ gives the same.

Your answerFor

A = \begin{bmatrix}2&1\\1&2\end{bmatrix}

, what is the top-left entry of

A^{10}

Worked example Spagnolie Lecture 19

Split

A = \begin{bmatrix}1&3\\1&2\end{bmatrix}

into its symmetric and antisymmetric parts, and find the eigenvalues of each.

Solution.

Symmetric part. $E = \frac12(A + A^{\mathsf T}) = \frac12\begin{bmatrix}2&4\\4&4\end{bmatrix} = \begin{bmatrix}1&2\\2&2\end{bmatrix}$ . Adding $A$ and its transpose makes entry $(i,j)$ equal to entry $(j,i)$ .
Antisymmetric part. $R = \frac12(A - A^{\mathsf T}) = \frac12\begin{bmatrix}0&2\\-2&0\end{bmatrix} = \begin{bmatrix}0&1\\-1&0\end{bmatrix}$ . Subtracting makes entry $(i,j)$ the negative of entry $(j,i)$ .
Check they add up. $E + R = \begin{bmatrix}1&3\\1&2\end{bmatrix} = A$ . The halves of $A^{\mathsf T}$ cancel.
Eigenvalues of $E$ . $\lambda^2 - 3\lambda - 2 = 0$ , so $\lambda = \frac{3 \pm \sqrt{17}}2$ , about $3.56$ and $-0.56$ . Trace 3 and determinant $2 - 4 = -2$ . Real, as the spectral theorem promises for symmetric $E$ .
Eigenvalues of $R$ . $\lambda^2 + 1 = 0$ , so $\lambda = \pm i$ . Purely imaginary: $R$ is a quarter turn, so no real direction stays on its line.

Answer. $E = \begin{bmatrix}1&2\\2&2\end{bmatrix}$ (a stretch along perpendicular axes) and $R = \begin{bmatrix}0&1\\-1&0\end{bmatrix}$ (a rotation).

Worked example Apostol 5.5 style

Prove: if

A

is orthogonally diagonalizable, then

A

is symmetric.

Solution.

Write down the hypothesis. $A = Q\Lambda Q^{\mathsf T}$ with $Q^{\mathsf T}Q = I$ and $\Lambda$ diagonal. This is the definition of orthogonally diagonalizable.
Transpose both sides. $A^{\mathsf T} = (Q\Lambda Q^{\mathsf T})^{\mathsf T}$ . Goal: show $A^{\mathsf T} = A$ .
Reverse the order of the product. $(Q\Lambda Q^{\mathsf T})^{\mathsf T} = (Q^{\mathsf T})^{\mathsf T}\Lambda^{\mathsf T}Q^{\mathsf T}$ . The transpose of a product is the product of transposes in reverse order (2.5).
Simplify each factor. $(Q^{\mathsf T})^{\mathsf T} = Q$ and $\Lambda^{\mathsf T} = \Lambda$ . Transposing twice gives the original, and a diagonal matrix is unchanged by transposing.
Compare. $A^{\mathsf T} = Q\Lambda Q^{\mathsf T} = A$ .

Answer. $A$ is symmetric. $\blacksquare$ With the spectral theorem, a real matrix is orthogonally diagonalizable if and only if it is symmetric.

Worked example

Prove: every eigenvalue of a real antisymmetric matrix (

A^{\mathsf T} = -A

) is zero or purely imaginary. Test on

\begin{bmatrix}0&2\\-2&0\end{bmatrix}

Solution.

Set up. Let $Ax = \lambda x$ with $x \ne O$ , where $x$ and $\lambda$ may be complex. Eigenvalues of a real matrix can be complex, so we use the complex inner product defined above.
Compute $(Ax, x)$ two ways. First, $(Ax, x) = \lambda(x, x)$ . Second, moving $A$ across, $(Ax, x) = (x, A^{*}x) = (x, -Ax) = (x, -\lambda x) = -\bar\lambda\,(x, x)$ . For a real matrix $A^* = A^{\mathsf T} = -A$ . The conjugate appears because $\lambda$ comes out of the second slot.
Compare. $\lambda(x, x) = -\bar\lambda(x, x)$ , and $(x, x) > 0$ , so $\lambda = -\bar\lambda$ . Divide by the positive number $(x, x)$ .
Read off. Write $\lambda = a + bi$ . Then $a + bi = -a + bi$ , so $a = 0$ . The real parts must be opposites of each other and equal, so they are 0.
Test. $\begin{bmatrix}0&2\\-2&0\end{bmatrix}$ has $\lambda^2 + 4 = 0$ , so $\lambda = \pm 2i$ . Purely imaginary, as proved.

Answer. Every eigenvalue of a real antisymmetric matrix has real part 0. $\blacksquare$

Worked example

Write

A = \begin{bmatrix}2&1\\1&2\end{bmatrix}

as a sum of eigenvalue times projection, and show each piece is a projection.

Solution.

Take the unit eigenvectors. $q_1 = \frac1{\sqrt2}(1, 1)$ , $q_2 = \frac1{\sqrt2}(1, -1)$ , with eigenvalues $3$ and $1$ . From the first example.
Build each rank-one matrix. $q_1q_1^{\mathsf T} = \frac12\begin{bmatrix}1&1\\1&1\end{bmatrix}$ and $q_2q_2^{\mathsf T} = \frac12\begin{bmatrix}1&-1\\-1&1\end{bmatrix}$ . A column times a row is a matrix.
Show it is a projection. For any $x$ , $(q_1q_1^{\mathsf T})x = q_1(q_1^{\mathsf T}x) = (x \cdot q_1)\,q_1$ . This is the projection of $x$ onto the line through the unit vector $q_1$ (2.9).
Add up. $3q_1q_1^{\mathsf T} + 1\,q_2q_2^{\mathsf T} = \frac12\begin{bmatrix}3+1&3-1\\3-1&3+1\end{bmatrix} = \begin{bmatrix}2&1\\1&2\end{bmatrix}$ . Matches $A$ , so the decomposition is right.

Answer. $A = 3\,q_1q_1^{\mathsf T} + 1\,q_2q_2^{\mathsf T}$ : project onto each axis, then scale by that axis’s eigenvalue. In general, $A = \sum_i \lambda_i q_iq_i^{\mathsf T}$ , because the column-times-row expansion of $Q\Lambda Q^{\mathsf T}$ has exactly these terms.

Worked example Axler 7B

For

A = \begin{bmatrix}2&1\\1&2\end{bmatrix}

and

x = (1, 2)

, compute

x^{\mathsf T}Ax

in the eigenbasis and see why it lies between the eigenvalues.

Solution.

Compute directly. $Ax = (4, 5)$ , so $x^{\mathsf T}Ax = 4 + 10 = 14$ , and $\lVert x\rVert^2 = 5$ . The ratio $\frac{x^{\mathsf T}Ax}{x^{\mathsf T}x} = 2.8$ is what we explain next.
Find the coordinates in the eigenbasis. $c_1 = x \cdot q_1 = \frac3{\sqrt2}$ and $c_2 = x \cdot q_2 = -\frac1{\sqrt2}$ , so $x = c_1q_1 + c_2q_2$ . In an orthonormal basis the coordinates are dot products (2.9).
Expand. $x^{\mathsf T}Ax = \lambda_1c_1^2 + \lambda_2c_2^2 = 3 \cdot \frac92 + 1 \cdot \frac12 = 14$ . $Ax = \lambda_1c_1q_1 + \lambda_2c_2q_2$ , and the dot product with $x$ keeps only matching pairs, because $q_1 \cdot q_2 = 0$ .
Bound it. $c_1^2 + c_2^2 = 5 = \lVert x\rVert^2$ , so $14 = 3c_1^2 + c_2^2$ is a weighted average (weights $\frac{9/2}5$ and $\frac{1/2}5$ ) of the eigenvalues 3 and 1, times 5. An average of 3 and 1 lies between them. Here $2.8$ is between $1$ and $3$ .

Answer. $x^{\mathsf T}Ax = 14$ , and the ratio $2.8$ lies between the smallest and largest eigenvalue. Taking $x = q_1$ gives the top value 3, and any $x \ne O$ gives a positive value when both eigenvalues are positive.

Your answerFor

A = \begin{bmatrix}2&1\\1&2\end{bmatrix}

, what is the largest value of

x^{\mathsf T}Ax

over all unit vectors

x

Your answerWhat is the sum of the squares of the two eigenvalues of the symmetric matrix

\begin{bmatrix}1&4\\4&0\end{bmatrix}

Quick checkWhich real matrices can be orthogonally diagonalized (that is,

A = Q\Lambda Q^{\mathsf T}

with

Q

orthogonal)?

Quick checkA real

5 \times 5

symmetric matrix has eigenvalues

2, 2, 2, 7, 7

. How many independent eigenvectors does it have?

Where it is used

Used inside maths· Inside maths: quadratic forms, the Hessian, and PCA

Chapter 3.2 uses the spectral theorem to remove the cross term from $ax^2 + 2bxy + cy^2$ by rotating axes, which classifies every conic. Chapter 5.6 applies it to the Hessian (the symmetric matrix of second derivatives) to decide whether a critical point is a maximum, minimum or saddle. Chapter 3.4 applies it to $A^{\mathsf T}A$ to build the SVD.

Used in the real world· Fluid mechanics: how a blob of fluid deforms (Spagnolie, Lecture 19)

In a steady linear flow $\frac{dx}{dt} = xA$ , split $A = E + R$ into its symmetric part $E = \frac12(A + A^{\mathsf T})$ and antisymmetric part $R = \frac12(A - A^{\mathsf T})$ . $R$ rotates the fluid. $E$ is symmetric, so by the spectral theorem it is a pure stretch along three perpendicular axes $q_i$ : in those coordinates each component grows like $e^{\lambda_i t}$ , and a ball of fluid becomes an ellipsoid. Engineers call $E$ the strain-rate tensor, and its eigenvectors the principal strain directions. A shear flow, in which the layers slide past each other, is half rotation and half stretching along the diagonals (one of the problems below splits one).

Used in the real world· Physics and engineering: principal axes and real measurements

The inertia matrix of a rigid body is symmetric, so every body has three perpendicular principal axes; satellites and spinning tops are analysed in those coordinates. The stress tensor in a loaded beam is symmetric; its eigenvectors are the principal stress directions, along which materials crack. In quantum mechanics every measurable quantity is a Hermitian operator, and the first theorem of this chapter is why every measurement gives a real number.

Traps

Watch out· Repeated eigenvalues: choose perpendicular eigenvectors

Eigenvectors for DIFFERENT eigenvalues of a symmetric matrix are automatically perpendicular. Inside one eigenspace (a repeated eigenvalue) they need not be; run Gram–Schmidt inside that eigenspace, as in the step-through.

Watch out· Symmetric means equal to its transpose, not 'looks balanced'

$\begin{bmatrix}1&2\\2&1\end{bmatrix}$ is symmetric; $\begin{bmatrix}1&2\\-2&1\end{bmatrix}$ is not (it is a rotation times a scale, with complex eigenvalues).

Watch out· Complex inner products conjugate the second slot

The proof that eigenvalues are real works because $(x, \lambda x) = \bar\lambda(x, x)$ . Forgetting the conjugate “proves” every eigenvalue real, which is false for rotations.

Problems

check

Which are symmetric: $\begin{bmatrix}1&4\\4&0\end{bmatrix}$ , $\begin{bmatrix}0&1\\-1&0\end{bmatrix}$ , $\begin{bmatrix}5&0\\0&5\end{bmatrix}$ ?

Solution

The first and third. The second equals minus its transpose (antisymmetric); its eigenvalues are $\pm i$ .

check

A symmetric matrix has eigenvectors $(1, 2)$ for $\lambda = 4$ . Find an eigenvector for its other eigenvalue without knowing the matrix.

Solution

Eigenvectors of a symmetric $2 \times 2$ matrix for different eigenvalues are perpendicular, so $(2, -1)$ (or any multiple). (If the other eigenvalue were also 4, every vector would be an eigenvector, and $(2,-1)$ still works.)

check

If $Q$ is orthogonal, what is $Q^{-1}$ ? Why does that matter for computing $A = Q\Lambda Q^{-1}$ ?

Solution

$Q^{-1} = Q^{\mathsf T}$ . Inverting a matrix normally takes elimination; for an orthogonal matrix it is a transpose, which is free and numerically exact.

useApostol 5.5 style

Orthogonally diagonalize $\begin{bmatrix}5&2\\2&2\end{bmatrix}$ .

Solution

$\det = (5 - \lambda)(2 - \lambda) - 4 = \lambda^2 - 7\lambda + 6 = (\lambda - 6)(\lambda - 1)$ . $\lambda = 6$ : $(2, 1)$ ; $\lambda = 1$ : $(1, -2)$ . $Q = \frac1{\sqrt5}\begin{bmatrix}2&1\\1&-2\end{bmatrix}$ , $\Lambda = \operatorname{diag}(6, 1)$ .

use

Use $A = Q\Lambda Q^{\mathsf T}$ to compute $A^{10}$ for $A = \begin{bmatrix}2&1\\1&2\end{bmatrix}$ .

Solution

$A^{10} = Q\Lambda^{10}Q^{\mathsf T}$ with $\Lambda^{10} = \operatorname{diag}(3^{10}, 1)$ . Multiplying out: $A^{10} = \frac12\begin{bmatrix}3^{10} + 1 & 3^{10} - 1\\ 3^{10} - 1 & 3^{10} + 1\end{bmatrix} = \begin{bmatrix}29525 & 29524\\ 29524 & 29525\end{bmatrix}$ .

useSpagnolie Lecture 19

Split $A = \begin{bmatrix}0&1\\0&0\end{bmatrix}$ (a shear flow) into symmetric and antisymmetric parts, and find the principal stretching directions of the symmetric part.

Solution

$E = \frac12\begin{bmatrix}0&1\\1&0\end{bmatrix}$ , $R = \frac12\begin{bmatrix}0&1\\-1&0\end{bmatrix}$ . $E$ has eigenvalues $\pm\frac12$ with eigenvectors $(1, 1)$ (stretching) and $(1, -1)$ (compressing): the diagonals.

proveApostol 5.5 style

Prove that if $A$ is orthogonally diagonalizable, then $A$ is symmetric. (So for real matrices, orthogonally diagonalizable is the same as symmetric.)

Solution

$A = Q\Lambda Q^{\mathsf T}$ gives $A^{\mathsf T} = (Q^{\mathsf T})^{\mathsf T}\Lambda^{\mathsf T}Q^{\mathsf T} = Q\Lambda Q^{\mathsf T} = A$ , since $\Lambda$ is diagonal so $\Lambda^{\mathsf T} = \Lambda$ .

prove

Prove that every eigenvalue of an antisymmetric real matrix ( $A^{\mathsf T} = -A$ ) is purely imaginary or zero.

Solution

If $Ax = \lambda x$ , $x \ne O$ complex: $\lambda(x, x) = (Ax, x) = (x, A^*x) = (x, -Ax) = (x, -\lambda x) = -\bar\lambda(x, x)$ . So $\lambda = -\bar\lambda$ , meaning the real part of $\lambda$ is 0. (This is Spagnolie’s “antisymmetric part = pure rotation”.)

prove

Prove the spectral decomposition: if $A = Q\Lambda Q^{\mathsf T}$ then $A = \sum_{i=1}^n \lambda_i q_i q_i^{\mathsf T}$ , and each $q_i q_i^{\mathsf T}$ is the projection onto the line through $q_i$ .

Solution

$Q\Lambda Q^{\mathsf T} = [\lambda_1 q_1 \cdots \lambda_n q_n]\,Q^{\mathsf T} = \sum_i \lambda_i q_i q_i^{\mathsf T}$ (column times row expansion of a product). And $(q_i q_i^{\mathsf T})x = q_i(q_i^{\mathsf T}x) = (x \cdot q_i)q_i$ , the projection onto the unit vector $q_i$ (2.9).

proveAxler 7B

Prove that a symmetric $A$ with all eigenvalues positive satisfies $x^{\mathsf T}Ax > 0$ for every $x \ne O$ .

Solution

Write $x = \sum c_i q_i$ in the orthonormal eigenbasis. Then $Ax = \sum \lambda_i c_i q_i$ and $x^{\mathsf T}Ax = \sum_i \lambda_i c_i^2$ (orthonormality kills cross terms). Each term is $\ge 0$ and at least one $c_i \ne 0$ , so the sum is positive. (Such matrices are called positive definite; 3.2.)

stretch

Let $A$ be real symmetric with largest eigenvalue $\lambda_{\max}$ . Prove $\max_{\lVert x \rVert = 1} x^{\mathsf T}Ax = \lambda_{\max}$ (the Rayleigh quotient), and say where it is attained.

Solution

With $x = \sum c_i q_i$ , $\lVert x \rVert^2 = \sum c_i^2 = 1$ and $x^{\mathsf T}Ax = \sum \lambda_i c_i^2 \le \lambda_{\max}\sum c_i^2 = \lambda_{\max}$ . Equality at $x = q_{\max}$ , a unit eigenvector for $\lambda_{\max}$ . This is how PCA (3.4) finds the direction of greatest variance.

Summary

The adjoint moves a map across an inner product: $(Ax, y) = (x, A^*y)$ ; for real matrices it is the transpose.
A symmetric matrix ( $A^{\mathsf T} = A$ ) has real eigenvalues, and eigenvectors for different eigenvalues are perpendicular.
Spectral theorem: every real symmetric matrix is $A = Q\Lambda Q^{\mathsf T}$ , an orthonormal eigenbasis with real eigenvalues, even with repeated eigenvalues. Proof: one unit eigenvector, then its perpendicular space is invariant, and induct.
For real matrices, orthogonally diagonalizable is exactly the same as symmetric.
A symmetric matrix is a pure stretch along perpendicular axes: principal axes, principal strains, principal components.

Watch and read

The same ideas explained by someone else. Use these when an explanation here does not land.

3.1 Symmetric matrices and the spectral theorem

The picture

Definitions

Adjoint

Symmetric operator

Hermitian

Orthogonally diagonalizable

The three facts

Symmetric matrices have real eigenvalues

Eigenvectors of a symmetric matrix are orthogonal

Spectral theorem

Examples

Where it is used

Traps

Problems

Summary

Watch and read

Where this sits

Comes from

Leads to

Same idea elsewhere