2.9 Inner products, orthogonality, Gram–Schmidt, projection

An inner product gives any vector space lengths, angles and perpendicularity. With it, orthogonal bases make coordinates easy, Gram–Schmidt builds them, and projection onto a subspace gives the closest point.

So far a vector space can add and scale, but it has no length, no angle and no idea of “perpendicular”. In JEE you learned the dot product as $\vec a\cdot\vec b = |\vec a||\vec b|\cos\theta$ , which starts from the angle. This chapter goes the other way. We write down a few properties of the dot product as axioms (an inner product), and then we define length and angle from it. That works in any vector space, including spaces of functions, and it leads to the most useful idea in applied mathematics: the closest point to a vector, inside a subspace, is its shadow.

Before you start

You need bases and coordinates (2.2), matrix multiplication and the transpose (2.5), and one fact from 2.8 near the end. Some integration is used for function spaces. Three quick checks:

Quick checkIs

\{(1, 0), (1, 1)\}

a basis of

\mathbb R^2

Quick checkWhat are the coordinates of

(3, 5)

in the basis

\{(1, 0), (1, 1)\}

Quick checkFor column vectors

x = (1, 2)

and

y = (3, 4)

, what is the

1\times1

matrix product

x^{\mathsf T}y

The picture

Drag the blue vector $v$ and the grey direction $u$ . The green arrow $p$ is the shadow of $v$ on the line through $u$ . The red dashed arrow $e$ is what is left over. It meets the line at a right angle, and the little square marks that. Now use the slider under the picture to move the orange point along the line. It is never closer to $v$ than the shadow is. This one picture is the backbone of the chapter: Cauchy–Schwarz, Gram–Schmidt, best approximation, Fourier series and least squares are all this triangle in different spaces.

Where this came from· Cauchy, Bunyakovsky, Schwarz

The inequality that makes angles possible grew in steps, each for a bigger setting. Cauchy published it in 1821 for finite sums of numbers. Bunyakovsky (1859) proved the matching inequality for integrals, which is the version for functions that you use in this chapter. Schwarz (1888) gave the modern proof of the integral version. Some books, especially Russian ones, use all three names. The three versions are one theorem, because the proof below uses only the inner-product axioms and never what the vectors are. That is the point of defining an inner product by axioms: one proof then covers sums, integrals and every other case.

The idea, in words

The dot product $x\cdot y = x_1y_1 + \dots + x_ny_n$ does three jobs. It measures length ( $\sqrt{x\cdot x}$ ), it detects perpendicular vectors ( $x\cdot y = 0$ ), and it measures how much $x$ points along $y$ . All three come from four properties of the dot product. So we take those four properties as the rules, call anything that obeys them an inner product, and everything else follows by algebra. The payoff is that functions can be “perpendicular” too, and we can ask for the nearest function in a family.

Why· How much does x point along y?

Look at the picture. The shadow of $v$ on the line of $u$ is $p = c\,u$ with $c = \dfrac{v\cdot u}{u\cdot u}$ . That is the whole idea of projection: the number $c$ is the dot product, divided by the length of $u$ squared so that $u$ ‘s own size does not matter. When $v$ is perpendicular to $u$ , the shadow is zero. When $v$ is a multiple of $u$ , the shadow is $v$ itself. Every formula later in the chapter is this one repeated.

Definitions and theorems

Inner products

Definition T

Inner product

Let $V$ be a real vector space. An inner product assigns to every pair $x, y$ in $V$ a real number $(x, y)$ such that, for all $x, y, z$ in $V$ and every real $c$ :

$(x, y) = (y, x)$ (symmetry);
$(x + z, y) = (x, y) + (z, y)$ (additivity);
$(cx, y) = c\,(x, y)$ (homogeneity);
$(x, x) > 0$ whenever $x \ne O$ (positivity).

A real vector space with an inner product is a Euclidean space (Apostol and Spagnolie). Many books write $\langle x, y\rangle$ .

In one lineA rule $(x, y)$ giving a real number for every pair of vectors, symmetric, linear, and positive on non-zero vectors. The dot product is the model.

Why it existsIt lets any vector space have length, angle and perpendicularity, and these are what make 'closest point' mean something.

By symmetry, additivity and homogeneity also hold in the second slot, so $(x, y)$ is linear in each entry separately. Taking $c = 0$ in rule 3 gives $(O, y) = 0$ , so $(x, x) = 0$ exactly when $x = O$ . Axiom 4 is the one that fails most often.

Examples.

The dot product on $\mathbb R^n$ : $(x, y) = x_1y_1 + \dots + x_ny_n = x^{\mathsf T}y$ . Symmetry is clear, additivity and homogeneity are arithmetic, and $(x, x) = x_1^2 + \dots + x_n^2 > 0$ unless every $x_i = 0$ .
Functions: on $C[a, b]$ , the continuous functions on $[a, b]$ , put $(f, g) = \displaystyle\int_a^b f(t)g(t)\,dt$ . Rules 1 to 3 hold because integrals are symmetric and linear. For rule 4, if $f \ne 0$ then $f(t_0) \ne 0$ for some $t_0$ , so by continuity $f(t)^2 > c > 0$ on a small interval around $t_0$ , and the integral of $f^2$ is positive. Here continuity matters: a function that is $1$ at a single point and $0$ elsewhere has $\int f^2 = 0$ but is not the zero function.
A weighted inner product: $(f, g)_w = \int_a^b w(t)f(t)g(t)\,dt$ for any positive function $w$ . Spagnolie stressed “there is no unique inner product” on a space.
Another one on $\mathbb R^2$ : $(x, y) = x_1y_1 + 2x_2y_2$ . Positivity holds since $x_1^2 + 2x_2^2 > 0$ for $x \ne O$ .

Worked example Apostol 1.13 #1(c), MATH 375 HW3

Is (x, y) = (x1 + x2 + x3)(y1 + y2 + y3) an inner product on R³?

Solution.

Check symmetry. Swapping $x$ and $y$ gives the same product of two sums. Multiplication of numbers is commutative.
Check additivity and homogeneity. $(x + z, y) = \big(\sum(x_i + z_i)\big)\big(\sum y_i\big) = (x, y) + (z, y)$ and $(cx, y) = c\,(x, y)$ . The sum of coordinates is linear in $x$ , and multiplying by the fixed number $\sum y_i$ keeps that.
Check positivity. Here $(x, x) = (x_1 + x_2 + x_3)^2 \ge 0$ , but it equals $0$ whenever the coordinates add to zero. Take $x = (1, 0, -1)$ : $(x, x) = 0\cdot0 = 0$ although $x \ne O$ . Axiom 4 demands $(x, x) > 0$ for every non-zero $x$ . One counterexample is enough to break it.

Answer. No. It fails axiom 4. (Spagnolie’s remark: positivity is the usual failure, and one counterexample vector shows it.)

Quick checkIs

(x, y) = x_1y_1 - x_2y_2

an inner product on

\mathbb R^2

Your answerOn

\mathbb R^2

use the inner product

(x, y) = x_1y_1 + 2x_2y_2

. What is the norm

\lVert(2, 1)\rVert

Length and angle

Definition T

Norm

In a Euclidean space the norm of $x$ is $\lVert x\rVert = \sqrt{(x, x)}$ , which makes sense because $(x, x) \ge 0$ . The distance between $x$ and $y$ is $\lVert x - y\rVert$ . In $\mathbb R^n$ with the dot product this is the usual length $\sqrt{x_1^2 + \dots + x_n^2}$ . In $C[a, b]$ it is $\big(\int_a^b f^2\big)^{1/2}$ .

In one lineThe length of a vector, $\lVert x\rVert = \sqrt{(x, x)}$ , defined from the inner product.

Why it existsDistance and size are needed to say that one vector is closer to a target than another.

Now we want an angle. In JEE, $\cos\theta = \dfrac{x\cdot y}{|x||y|}$ was a consequence of the picture. Here we want to use that formula as the definition of the angle between non-zero vectors. There is a problem: $\cos\theta$ must lie in $[-1, 1]$ , or the inverse cosine does not exist. Why would $\dfrac{(x, y)}{\lVert x\rVert\lVert y\rVert}$ always be in that range, for functions, say? That is exactly what the next theorem proves.

Theorem T Apostol Thm 1.8; Spagnolie Lecture 4

Cauchy–Schwarz inequality

In any Euclidean space, for all $x$ and $y$ ,

|(x, y)| \le \lVert x\rVert\,\lVert y\rVert,

and equality holds if and only if $x$ and $y$ are linearly dependent.

In one line $|(x, y)| \le \lVert x\rVert\,\lVert y\rVert$ , with equality exactly when $x$ and $y$ are linearly dependent.

Why it is trueThe squared length $\lVert x + ty\rVert^2$ is a parabola in $t$ that never goes below zero. Evaluating it at its lowest point gives the inequality, and that lowest point is the projection of $x$ onto $y$ .

Why· The picture behind the proof

In the picture above, the point of the line closest to $v$ is the shadow. The proof walks along the line $x + ty$ and looks at its smallest distance from the origin. That smallest length cannot be negative, and writing it out gives the inequality. The special value $t_0 = -(x, y)/(y, y)$ that makes $x + t_0y$ shortest is exactly the leftover vector $e$ in the picture (with the roles $v = x$ , $u = y$ ): it is the part of $x$ that does not point along $y$ .

ProofSpagnolie Lecture 4 (the quadratic p(t)); Apostol 1.8

The idea: Look at the squared length of x + ty as a function of t. It is a quadratic that is never negative. Plug in the value of t where it is smallest. What is left is the inequality.

Assume

y \ne O

, so

(y, y) > 0

y = O

then

(x, y) = 0

and

\lVert y\rVert = 0

, so both sides are

0

and

x, O

are dependent.

For every real

t

0 \le (x + ty,\ x + ty)

Axiom 4 and

(O, O) = 0

: an inner product of a vector with itself is never negative. We apply it to the vector

x + ty

, for each real

t

(x + ty, x + ty) = (x, x) + 2t\,(x, y) + t^2(y, y)

Additivity and homogeneity in both slots, plus symmetry

(y, x) = (x, y)

Choose

t_0 = -\dfrac{(x, y)}{(y, y)}

This is the value of

t

at the bottom of the parabola: the derivative

2(x, y) + 2t(y, y)

is zero there. It is also the projection coefficient from the picture. We are free to choose any

t

because the inequality holds for all of them.

0 \le (x, x) - \dfrac{(x, y)^2}{(y, y)}

Substitute

t_0

into step 3 and simplify:

-2(x,y)^2/(y,y) + (x,y)^2/(y,y) = -(x,y)^2/(y,y)

(x, y)^2 \le \lVert x\rVert^2\lVert y\rVert^2

, so

|(x, y)| \le \lVert x\rVert\lVert y\rVert

Multiply both sides by

(y, y) > 0

, then take square roots (both sides are non-negative).

Equality implies

x = -t_0y

, so

x, y

are dependent.

If equality holds, the right side of step 5 is exactly

0

, so

(x + t_0y,\ x + t_0y) = 0

. Axiom 4 then forces

x + t_0y = O

Conversely, dependent vectors give equality.

x = cy

, then

|(cy, y)| = |c|\,(y, y)

by homogeneity. And

\lVert cy\rVert = \sqrt{(cy, cy)} = \sqrt{c^2(y, y)} = |c|\,\lVert y\rVert

, so

\lVert cy\rVert\lVert y\rVert = |c|\,(y, y)

too. (With

y \ne O

, dependent means

x = cy

Proof workshopRebuild the Cauchy–Schwarz proof

Tap the lines in the order a proof would write them. Each line can only use what is already above it.

So the angle is well defined.

Definition T

Orthogonal

For non-zero $x, y$ in a Euclidean space, the angle between them is $\theta = \arccos\dfrac{(x, y)}{\lVert x\rVert\lVert y\rVert} \in [0, \pi]$ . The vectors $x, y$ are orthogonal (written $x \perp y$ ) if $(x, y) = 0$ . This agrees with $\theta = \pi/2$ , and it also allows $x = O$ , which is orthogonal to everything. A set $S$ is orthogonal if $(x, y) = 0$ for all different $x, y$ in $S$ .

In one lineTwo vectors are orthogonal when their inner product is $0$ ; a set is orthogonal when every two different members are.

Why it existsIt is the abstract version of 'at right angles', and it is the property that makes coordinates and projections easy to compute.

Definition T

Orthonormal

A set is orthonormal if it is orthogonal and $\lVert x\rVert = 1$ for each member. Dividing any non-zero vector by its norm makes it have length $1$ : $\lVert x/\lVert x\rVert\,\rVert = 1$ .

In one lineAn orthogonal set in which every vector has length $1$ .

Why it existsUnit lengths remove the division in every formula, so the coefficient of a vector is an inner product, with no division.

Why the old and the new angle agree. In $\mathbb R^2$ or $\mathbb R^3$ , the cosine rule from the geometry you did in JEE says $|x - y|^2 = |x|^2 + |y|^2 - 2|x||y|\cos\theta$ . Expanding the left side with the dot product gives $|x|^2 + |y|^2 - 2\,x\cdot y$ . Comparing the two, $x\cdot y = |x||y|\cos\theta$ . So the JEE formula is a theorem about the dot product, and in higher dimensions and in function spaces we turn it around into a definition.

Worked example

Find the angle between (1, 1, 0) and (0, 1, 1) in R³.

Solution.

Compute the inner product. $(x, y) = 1\cdot0 + 1\cdot1 + 0\cdot1 = 1$ . The dot product is the inner product on $\mathbb R^3$ .
Compute the norms. $\lVert x\rVert = \sqrt{1 + 1 + 0} = \sqrt2$ and $\lVert y\rVert = \sqrt{0 + 1 + 1} = \sqrt2$ . The norm is $\sqrt{(x, x)}$ .
Form the cosine. $\cos\theta = \dfrac{1}{\sqrt2\cdot\sqrt2} = \dfrac12$ . This is the definition of the angle. Cauchy–Schwarz guarantees this number lies in $[-1, 1]$ .
Invert. $\theta = \arccos\tfrac12 = 60°$ . The inverse cosine of $\tfrac12$ is $\pi/3$ .

Answer. $\theta = 60°$ .

Your answerFind

\cos\theta

for the angle between

(1, 2, 2)

and

(3, 0, 4)

Quick checkWhen does equality

|(x, y)| = \lVert x\rVert\lVert y\rVert

hold in the Cauchy–Schwarz inequality?

Theorem T Apostol Thm 1.9; Spagnolie Lecture 4

Properties of the norm

In a Euclidean space: (a) $\lVert x\rVert \ge 0$ , and $\lVert x\rVert = 0$ iff $x = O$ . (b) $\lVert cx\rVert = |c|\,\lVert x\rVert$ for every real $c$ . (c) $\lVert x + y\rVert \le \lVert x\rVert + \lVert y\rVert$ .

In one line $\lVert x\rVert \ge 0$ with equality only for $O$ ; $\lVert cx\rVert = |c|\lVert x\rVert$ ; and the triangle inequality $\lVert x + y\rVert \le \lVert x\rVert + \lVert y\rVert$ .

Why it is trueExpand $\lVert x + y\rVert^2$ and bound the cross term with Cauchy–Schwarz.

ProofSpagnolie Lecture 4; Apostol 1.9

The idea: (a) and (b) follow directly from the axioms. For (c), square both sides so that no square roots remain, expand, and use Cauchy–Schwarz on the one cross term.

(a)

\lVert x\rVert = \sqrt{(x,x)} \ge 0

, with

0

only for

x = O

Axiom 4:

(x, x) > 0

for

x \ne O

, and

(O, O) = 0

(b)

\lVert cx\rVert = \sqrt{c^2(x, x)} = |c|\,\lVert x\rVert

Homogeneity in each slot gives

(cx, cx) = c^2(x, x)

, and

\sqrt{c^2} = |c|

, not

c

(c)

\lVert x + y\rVert^2 = (x, x) + 2(x, y) + (y, y) = \lVert x\rVert^2 + 2(x, y) + \lVert y\rVert^2

Expand by additivity and symmetry, as in the Cauchy–Schwarz proof.

\le \lVert x\rVert^2 + 2\lVert x\rVert\lVert y\rVert + \lVert y\rVert^2 = (\lVert x\rVert + \lVert y\rVert)^2

Cauchy–Schwarz:

(x, y) \le |(x, y)| \le \lVert x\rVert\lVert y\rVert

\lVert x + y\rVert \le \lVert x\rVert + \lVert y\rVert

Both sides are non-negative, so taking square roots keeps the inequality.

Worked example Apostol 1.13 #3; MATH 375 HW3

Prove that (x, y) = 0 if and only if the norm of x + y equals the norm of x − y. Then deduce the parallelogram law and the polarization identity.

Solution.

Expand the sum. $\lVert x + y\rVert^2 = (x + y, x + y) = \lVert x\rVert^2 + 2(x, y) + \lVert y\rVert^2$ . Additivity in both slots gives four terms, and symmetry makes the two cross terms equal.
Expand the difference. $\lVert x - y\rVert^2 = \lVert x\rVert^2 - 2(x, y) + \lVert y\rVert^2$ . Same computation with $-y$ in place of $y$ . Homogeneity with $c = -1$ flips the sign of the cross term.
Subtract. $\lVert x + y\rVert^2 - \lVert x - y\rVert^2 = 4(x, y)$ . The terms $\lVert x\rVert^2$ and $\lVert y\rVert^2$ cancel, and the cross terms add.
Both directions. If $(x, y) = 0$ the two squares are equal, so the norms are equal (norms are non-negative). If the norms are equal, the left side of step 3 is $0$ , so $(x, y) = 0$ . Squaring and taking square roots are reversible for non-negative numbers.
Add instead of subtracting. $\lVert x + y\rVert^2 + \lVert x - y\rVert^2 = 2\lVert x\rVert^2 + 2\lVert y\rVert^2$ , the parallelogram law. Now the cross terms cancel. The picture: the two diagonals of a parallelogram have squared lengths adding to twice the sum of the squared sides.
Divide step 3 by 4. $(x, y) = \tfrac14\big(\lVert x + y\rVert^2 - \lVert x - y\rVert^2\big)$ , the polarization identity. The inner product can be recovered from the norm alone.

Answer. $(x, y) = 0$ exactly when the two diagonals $x + y$ and $x - y$ have equal length, which for a parallelogram says it is a rectangle. The parallelogram law and polarization both follow from the same two expansions.

Beyond the course: for understanding· Norms without inner products

A norm in general is any function with properties (a), (b), (c), whether or not it comes from an inner product. Spagnolie lists $\lVert x\rVert_1 = \sum|x_i|$ , the $p$ -norms $\big(\sum|x_i|^p\big)^{1/p}$ , and $\lVert x\rVert_\infty = \max|x_i|$ . In the plane their unit “balls” ( $\lVert x\rVert = 1$ ) are a diamond, a circle and a square. Only the circle comes from an inner product; the others have no angle. Spagnolie’s non-example: $\lVert F\rVert = \int_0^1 F'(x)^2\,dx$ on differentiable functions is not a norm. It is $0$ for $F = 32$ , it scales like $c^2$ instead of $|c|$ , and $F(x) = g(x) = x$ gives $\lVert F + g\rVert = 4 > 2 = \lVert F\rVert + \lVert g\rVert$ , which breaks the triangle inequality. In machine learning the 1-norm gives sparse solutions (lasso), and the 2-norm used in this chapter gives least squares.

Worked example Apostol 1.13 #8, MATH 375 HW3

With weight log x on [1, e], find the norm of f(x) = √x, and a linear polynomial g = a + bx orthogonal to the constant 1.

Solution. The inner product is $(f, g) = \int_1^e (\log x)\,f(x)g(x)\,dx$ .

Squared norm of $f$ . $\lVert f\rVert^2 = \int_1^e x\log x\,dx$ . $f^2 = x$ , and the norm is the square root of $(f, f)$ .
Integrate by parts. $\int_1^e x\log x\,dx = \Big[\tfrac{x^2}{2}\log x\Big]_1^e - \int_1^e\tfrac x2\,dx = \tfrac{e^2}{2} - \tfrac{e^2 - 1}{4} = \tfrac{e^2 + 1}{4}$ . Take $u = \log x$ , $dv = x\,dx$ .
Take the square root. $\lVert f\rVert = \tfrac12\sqrt{e^2 + 1} \approx 1.448$ .
Orthogonality to $1$ . $(1, a + bx) = a\int_1^e\log x\,dx + b\int_1^e x\log x\,dx = a\cdot1 + b\cdot\tfrac{e^2 + 1}{4}$ . $\int_1^e\log x\,dx = [x\log x - x]_1^e = 1$ , and the second integral is from step 2.
Set it to zero. $a = -b\,\tfrac{e^2 + 1}{4}$ . Taking $b = 1$ : $g(x) = x - \tfrac{e^2 + 1}{4}$ . One linear equation in two unknowns, so there is a whole line of answers; any multiple works.

Answer. $\lVert\sqrt x\rVert = \tfrac12\sqrt{e^2 + 1}$ , and $g(x) = x - \tfrac{e^2 + 1}{4}$ (or any multiple) is orthogonal to $1$ .

Orthogonal sets and coordinates

Theorem T Apostol Thm 1.10; Spagnolie Lecture 5

Orthogonal sets are independent

If $x_1, \dots, x_k$ are non-zero and $(x_i, x_j) = 0$ for $i \ne j$ , then $x_1, \dots, x_k$ are linearly independent.

In one lineAn orthogonal set of non-zero vectors is linearly independent.

Why it is trueTake the inner product of a supposed dependence with one member. Every cross term vanishes and one coefficient is isolated.

ProofSpagnolie Lecture 5; Apostol 1.12

The idea: Suppose a combination is O. Take the inner product with x₁: all terms except the first vanish, leaving c₁‖x₁‖² = 0. Repeat for each vector.

Suppose

c_1x_1 + \dots + c_kx_k = O

The definition of dependence, to be refuted.

(x_1,\ c_1x_1 + \dots + c_kx_k) = 0

Take the inner product of both sides with

x_1

. The inner product with

O

0

c_1(x_1, x_1) + c_2(x_1, x_2) + \dots = c_1\lVert x_1\rVert^2 = 0

Additivity and homogeneity expand the left side. Orthogonality kills every

(x_1, x_j)

with

j \ne 1

c_1 = 0

x_1 \ne O

\lVert x_1\rVert^2 > 0

. Divide by it.

Likewise every

c_j = 0

, so the set is independent.

Repeat with

x_2, \dots, x_k

in place of

x_1

Quick checkTrue or false: every linearly independent set of non-zero vectors is orthogonal.

Here is why orthogonal bases are worth the trouble. Suppose $e_1, \dots, e_n$ is an orthogonal basis and $x = c_1e_1 + \dots + c_ne_n$ . The same trick isolates each coefficient.

Theorem T Apostol Thm 1.11; Spagnolie Lecture 5

Coordinates in an orthogonal basis

Let $e_1, \dots, e_n$ be an orthogonal basis of $V$ with all $e_i \ne O$ . Then for every $x$ ,

x = \sum_{i=1}^n \frac{(x, e_i)}{(e_i, e_i)}\,e_i, \qquad\text{and}\qquad \lVert x\rVert^2 = \sum_{i=1}^n \frac{(x, e_i)^2}{(e_i, e_i)}.

If the basis is orthonormal this reads $x = \sum(x, e_i)e_i$ and $\lVert x\rVert^2 = \sum(x, e_i)^2$ .

In one lineIn an orthogonal basis, the coordinate of $x$ on $e_i$ is $(x, e_i)/(e_i, e_i)$ ; in an orthonormal basis it is $(x, e_i)$ itself.

Why it is trueTaking the inner product with $e_i$ kills every term except the $e_i$ one.

ProofSpagnolie Lecture 5; Apostol 1.11 and 1.12

The idea: Write x in the basis with unknown coefficients, take the inner product with eᵢ, and solve. For the norm, expand (x, x) in the same way.

Write

x = c_1e_1 + \dots + c_ne_n

Coordinates exist and are unique (2.2).

(x, e_i) = c_i(e_i, e_i)

Take the inner product with

e_i

. Additivity, homogeneity, and

(e_i, e_j) = 0

for

j \ne i

c_i = (x, e_i)/(e_i, e_i)

(e_i, e_i) > 0

since

e_i \ne O

\lVert x\rVert^2 = \sum c_i(x, e_i) = \sum\dfrac{(x, e_i)^2}{(e_i, e_i)}

(x, x) = (x, \sum c_ie_i) = \sum c_i(x, e_i)

, and substitute

c_i

from step 3.

Spagnolie warns about an easy mistake. The denominator is $(e_i, e_i)$ , which is $\lVert e_i\rVert^2$ and not $\lVert e_i\rVert$ . His unit check: $x$ and $e_i$ have the same units (say metres), so the coefficient $c_i$ must have no units. $(x, e_i)/\lVert e_i\rVert^2$ is metre-squared over metre-squared, which is fine. $(x, e_i)/\lVert e_i\rVert$ would be metre-squared over metre, which has units of length, so it cannot be right.

Worked example Spagnolie Lecture 5 (these vectors are built in the Gram–Schmidt example below)

Write z = (1, 2, 3) as a combination of the orthogonal vectors y1 = (1, 0, 1), y2 = (1/2, 1, -1/2), y3 = (-1/3, 1/3, 1/3).

Solution.

Check they are orthogonal. $(y_1, y_2) = \tfrac12 + 0 - \tfrac12 = 0$ , $(y_1, y_3) = -\tfrac13 + 0 + \tfrac13 = 0$ , $(y_2, y_3) = -\tfrac16 + \tfrac13 - \tfrac16 = 0$ . The coordinate formula needs an orthogonal basis. Three non-zero orthogonal vectors in $\mathbb R^3$ are independent, hence a basis.
Compute the denominators. $(y_1, y_1) = 2$ , $(y_2, y_2) = \tfrac32$ , $(y_3, y_3) = \tfrac13$ .
Compute the numerators. $(z, y_1) = 1 + 3 = 4$ , $(z, y_2) = \tfrac12 + 2 - \tfrac32 = 1$ , $(z, y_3) = -\tfrac13 + \tfrac23 + 1 = \tfrac43$ .
Divide. $c_1 = \tfrac42 = 2$ , $c_2 = \tfrac{1}{3/2} = \tfrac23$ , $c_3 = \tfrac{4/3}{1/3} = 4$ . Coefficient $=$ (inner product with $y_i$ ) divided by $(y_i, y_i)$ . No equation system is solved.
Check. $2(1,0,1) + \tfrac23(\tfrac12,1,-\tfrac12) + 4(-\tfrac13,\tfrac13,\tfrac13) = (2 + \tfrac13 - \tfrac43,\ \tfrac23 + \tfrac43,\ 2 - \tfrac13 + \tfrac43) = (1, 2, 3)$ . Adding the pieces back must give $z$ .

Answer. $z = 2y_1 + \tfrac23y_2 + 4y_3$ . For comparison, Spagnolie’s non-orthogonal basis $x_1 = (1,0,0)$ , $x_2 = (1,1,0)$ , $x_3 = (1,1,1)$ and $z = (0,1,1)$ : taking inner products with each $x_i$ leaves a $3\times3$ system with matrix $\begin{bmatrix}1&1&1\\1&2&2\\1&2&3\end{bmatrix}$ , whose solution is $z = -x_1 + x_3$ . That is the “annoying” computation that an orthogonal basis avoids.

Your answerIn an orthonormal basis

q_1, q_2, q_3

\mathbb R^3

a vector

x

has

(x, q_1) = 2

(x, q_2) = -6

(x, q_3) = 3

. What is

\lVert x\rVert

Gram–Schmidt: making an orthogonal basis

Not every basis is orthogonal. The Gram–Schmidt process turns any basis into an orthogonal one with the same spans. Its idea is the picture at the top of the chapter, used repeatedly: to make a vector perpendicular to earlier ones, subtract its shadows on them.

Definition T

Gram–Schmidt process

Given independent $x_1, \dots, x_k$ , the Gram–Schmidt process defines

y_1 = x_1, \qquad y_m = x_m - \sum_{j=1}^{m-1}\frac{(y_j, x_m)}{(y_j, y_j)}\,y_j \quad (m = 2, \dots, k).

Dividing each $y_m$ by its norm then gives an orthonormal list.

In one lineFrom independent vectors $x_1, x_2, \dots$ build orthogonal vectors $y_1, y_2, \dots$ by subtracting from each $x_k$ its projections on the earlier $y_j$ .

Why it existsIt turns any basis into an orthogonal (then orthonormal) one without changing what each initial segment spans.

Theorem T Apostol Thm 1.13; Spagnolie Lecture 5

Gram–Schmidt theorem

If $x_1, \dots, x_k$ are linearly independent, the vectors $y_1, \dots, y_k$ above are non-zero, pairwise orthogonal, and $L(y_1, \dots, y_m) = L(x_1, \dots, x_m)$ for each $m$ . (Here $L(\cdot)$ is the span, Apostol’s notation.) Consequently every finite-dimensional Euclidean space has an orthonormal basis (Apostol Thm 1.14).

In one lineThe $y_m$ are non-zero and orthogonal, and $y_1, \dots, y_m$ span the same space as $x_1, \dots, x_m$ for every $m$ .

Why it is trueEach step subtracts exactly the part of $x_m$ along the earlier $y_j$ , so what is left is perpendicular to them, and the subtraction does not change the span.

ProofSpagnolie Lecture 5; Apostol 1.13

The idea: Induction on m. Assume y₁,…,y_{m−1} are orthogonal and span the same as x₁,…,x_{m−1}. Then y_m is x_m minus a combination of earlier y's, so it is in the same span as x₁…x_m, it is not zero because x_m is outside the earlier span, and it is perpendicular to each earlier y_i by a one-line computation.

Base case

m = 1

For

m = 1

y_1 = x_1 \ne O

(independent vectors are non-zero), and the spans are equal.

Assume

y_1, \dots, y_{m-1}

are non-zero, orthogonal, and

L(y_1, \dots, y_{m-1}) = L(x_1, \dots, x_{m-1})

The induction hypothesis: the claim holds for

m - 1

L(y_1, \dots, y_m) = L(x_1, \dots, x_m)

By the formula,

y_m

x_m

minus a combination of

y_1, \dots, y_{m-1}

, which lie in

L(x_1, \dots, x_{m-1})

. Conversely

x_m = y_m +

(a combination of

y_j

), so

x_m \in L(y_1, \dots, y_m)

y_m \ne O

y_m = O

, then

x_m

would be a combination of

y_1, \dots, y_{m-1}

, hence of

x_1, \dots, x_{m-1}

, contradicting independence of the

x

's. (When Gram–Schmidt is applied to dependent vectors, a zero

y

appears, and that is how it reveals dependence: Apostol 1.17 #2(b).)

(y_i, y_m) = (y_i, x_m) - \dfrac{(y_i, x_m)}{(y_i, y_i)}(y_i, y_i)

Take the inner product of the defining formula with

y_i

for some

i < m

. By the induction hypothesis

(y_i, y_j) = 0

for

j \ne i

, so only the

j = i

term of the sum survives.

(y_i, y_m) = 0

for every

i < m

. The claim holds for

m

, so by induction for all

m

The two terms are equal.

An orthonormal basis exists.

Start with any basis of the space, apply the process, divide each

y

by its norm.

Quick checkIn Gram–Schmidt, when we compute

y_3

, which vectors do we subtract projections onto?

Worked example Spagnolie Lecture 5 (with its arithmetic corrected)

Apply Gram–Schmidt to x1 = (1, 0, 1), x2 = (1, 1, 0), x3 = (0, 0, 1) in R³, then normalize.

Solution.

Keep the first vector. $y_1 = x_1 = (1, 0, 1)$ , with $(y_1, y_1) = 2$ . There is nothing earlier to be perpendicular to.
Remove the shadow of $x_2$ on $y_1$ . $(y_1, x_2) = 1$ , so $y_2 = x_2 - \tfrac12y_1 = (\tfrac12, 1, -\tfrac12)$ , and $(y_2, y_2) = \tfrac32$ . This is the red dashed leftover of the picture at the top of the chapter.
Check. $(y_1, y_2) = \tfrac12 + 0 - \tfrac12 = 0$ . Checking each new vector against the earlier ones catches arithmetic slips immediately.
Remove both shadows from $x_3$ . $(y_1, x_3) = 1$ and $(y_2, x_3) = -\tfrac12$ , so the coefficients are $\tfrac12$ and $\tfrac{-1/2}{3/2} = -\tfrac13$ . Then $y_3 = x_3 - \tfrac12y_1 + \tfrac13y_2 = (-\tfrac13, \tfrac13, \tfrac13)$ . We project on the $y$ ‘s, not on $x_1, x_2$ , because the $y$ ‘s are perpendicular, so the shadows do not interfere. The minus before $-\tfrac13$ turns into a plus.
Check again. $(y_1, y_3) = -\tfrac13 + \tfrac13 = 0$ and $(y_2, y_3) = -\tfrac16 + \tfrac13 - \tfrac16 = 0$ . Both are $0$ , so $y_1, y_2, y_3$ are an orthogonal basis.
Normalize. Divide each $y$ by its length $\sqrt2$ , $\sqrt{3/2}$ , $\sqrt{1/3}$ to get $\varphi_1, \varphi_2, \varphi_3$ . Unit lengths make the coefficient of each vector a plain inner product.

The step-through below shows the same computation with every product written out.

Answer. $y_1 = (1, 0, 1)$ , $y_2 = (\tfrac12, 1, -\tfrac12)$ , $y_3 = (-\tfrac13, \tfrac13, \tfrac13)$ are orthogonal, and $\varphi_1, \varphi_2, \varphi_3$ above are orthonormal. In the orthonormal basis the coordinates of $z = (0, 0, 1)$ are $(z, \varphi_i) = \tfrac1{\sqrt2}, -\tfrac1{\sqrt6}, \tfrac1{\sqrt3}$ , and the squares add up to $\tfrac12 + \tfrac16 + \tfrac13 = 1 = \lVert z\rVert^2$ , as the coordinate theorem says.

Your answerGram–Schmidt on

x_1 = (3, 4)

and

x_2 = (1, 0)

gives

y_1 = x_1

and

y_2 = x_2 - \frac{(y_1, x_2)}{(y_1, y_1)}y_1

. What is the length

\lVert y_2\rVert

Worked example Apostol 1.17 #2(b), new numbers

Apply Gram–Schmidt to x1 = (1, 1, 0), x2 = (0, 1, 1), x3 = (1, 2, 1) in R³. What does the result say about the three vectors?

Solution.

First vector. $y_1 = x_1 = (1, 1, 0)$ , and $(y_1, y_1) = 2$ . The first vector is kept.
Second vector. $(y_1, x_2) = 1$ , so $y_2 = x_2 - \tfrac12y_1 = (-\tfrac12, \tfrac12, 1)$ , and $(y_2, y_2) = \tfrac32$ . Subtract the shadow of $x_2$ on $y_1$ . Check: $(y_1, y_2) = -\tfrac12 + \tfrac12 = 0$ .
Third vector, the two coefficients. $(y_1, x_3) = 3$ gives $\tfrac32$ , and $(y_2, x_3) = -\tfrac12 + 1 + 1 = \tfrac32$ gives $\tfrac{3/2}{3/2} = 1$ . Divide by $(y_j, y_j)$ in each case, as in the formula.
Subtract. $y_3 = (1, 2, 1) - \tfrac32(1, 1, 0) - 1\cdot(-\tfrac12, \tfrac12, 1) = (1 - \tfrac32 + \tfrac12,\ 2 - \tfrac32 - \tfrac12,\ 1 - 0 - 1) = (0, 0, 0)$ . The shadows on $y_1$ and $y_2$ account for all of $x_3$ .
Read the zero. $y_3 = O$ means $x_3$ lies in the span of $x_1, x_2$ . If $y_3$ were non-zero, the Gram–Schmidt theorem says it would be a new direction. A zero output is how the process reports dependence.
Check the dependence directly. $x_3 = x_1 + x_2$ , since $(1, 1, 0) + (0, 1, 1) = (1, 2, 1)$ . The conclusion agrees with a direct look at the vectors.

Answer. $y_3 = O$ , so the three vectors are linearly dependent. The span is a plane with orthogonal basis $y_1, y_2$ . Discard $y_3$ and carry on with the next vector, if any.

Gram–Schmidt works in any Euclidean space, including a space of functions. Take polynomials on $[-1, 1]$ with $(f, g) = \int_{-1}^1 f(t)g(t)\,dt$ . The monomials $1, t, t^2, \dots$ are a basis but not orthogonal: $(1, t^2) = \tfrac23 \ne 0$ .

Worked example Spagnolie Lecture 5 (Legendre polynomials)

Apply Gram–Schmidt to 1, t, t², t³ on [-1, 1] with (f, g) = ∫ f g dt.

Solution.

Start with the constant. $y_0 = 1$ , and $(y_0, y_0) = \int_{-1}^1 1\,dt = 2$ . The first vector is kept.
Second vector. $(y_0, t) = \int_{-1}^1 t\,dt = 0$ , so nothing is subtracted and $y_1 = t$ , with $(y_1, y_1) = \tfrac23$ . $t$ is odd and $1$ is even; an odd function integrates to $0$ over a symmetric interval.
Third vector. $(y_0, t^2) = \tfrac23$ and $(y_1, t^2) = \int t^3 = 0$ . So $y_2 = t^2 - \dfrac{2/3}{2}\cdot1 - 0 = t^2 - \tfrac13$ . Coefficient of $y_0$ is $(y_0, t^2)/(y_0, y_0) = \tfrac13$ . By the parity argument the $y_1$ coefficient is $0$ .
Check. $(y_0, y_2) = \tfrac23 - \tfrac13\cdot2 = 0$ and $(y_1, y_2) = \int(t^3 - t/3)\,dt = 0$ . Always check orthogonality; it catches slips.
Fourth vector. $(y_1, t^3) = \int t^4 = \tfrac25$ , so the coefficient of $y_1$ is $\dfrac{2/5}{2/3} = \tfrac35$ . The coefficients of $y_0$ and $y_2$ are $0$ by parity (odd times even). So $y_3 = t^3 - \tfrac35t$ . Same formula; $t^3$ is odd, so it is orthogonal to the even $y_0$ and $y_2$ already.

Answer. $1,\ t,\ t^2 - \tfrac13,\ t^3 - \tfrac35t$ , the Legendre polynomials. They are usually scaled so that the value at $t = 1$ is $1$ : $P_2 = \tfrac12(3t^2 - 1)$ and $P_3 = \tfrac12(5t^3 - 3t)$ .

Orthogonal complement and projection

Definition T

Orthogonal complement

For a subspace $S$ of a Euclidean space $V$ , the orthogonal complement is

S^\perp = \{v \in V : (v, s) = 0 \text{ for every } s \in S\}.

In one lineAll vectors orthogonal to every vector of a subspace $S$ ; written $S^\perp$ ('S perp').

Why it existsIt collects exactly the leftover parts that projection throws away.

Theorem T Apostol 1.15

The orthogonal complement is a subspace meeting S only at O

For a subspace $S$ of $V$ , $S^\perp$ is a subspace of $V$ , and $S \cap S^\perp = \{O\}$ . If $S$ has a basis $e_1, \dots, e_k$ , then $v \in S^\perp$ if and only if $(v, e_i) = 0$ for each $i$ .

In one line $S^\perp$ is a subspace, and the only vector in both $S$ and $S^\perp$ is $O$ .

Why it is trueInner products are linear, so orthogonality passes through sums and multiples. A vector orthogonal to itself has zero length.

ProofApostol 1.15

The idea: Use the subspace test for the first part, then expand in the basis for the 'if and only if', and use positivity for the intersection.

O \in S^\perp

(O, s) = 0

for every

s

v, w \in S^\perp

then

v + w

and

cv

are in

S^\perp

. So

S^\perp

is a subspace.

Additivity and homogeneity in the first slot:

(v + w, s) = (v, s) + (w, s) = 0

and

(cv, s) = c(v, s) = 0

So if

(v, e_i) = 0

for all

i

, then

(v, s) = 0

for all

s \in S

. The converse is immediate.

Every

s \in S

\sum a_ie_i

, and

(v, \sum a_ie_i) = \sum a_i(v, e_i)

S \cap S^\perp = \{O\}

v \in S \cap S^\perp

then

v

can be used as the

s

(v, v) = 0

, and axiom 4 gives

v = O

In $\mathbb R^3$ , if $S$ is a plane through the origin then $S^\perp$ is the line through the origin perpendicular to it. Spagnolie’s picture: a plane $S$ , a vector $x$ above it, and a dotted perpendicular down to its foot in the plane.

Theorem T Apostol Thm 1.15; Spagnolie Lecture 6

Orthogonal decomposition theorem

Let $S$ be a subspace of a Euclidean space $V$ with a finite basis. Then every $x \in V$ can be written in exactly one way as $x = s + s^\perp$ , where $s \in S$ and $s^\perp \in S^\perp$ . Moreover $\lVert x\rVert^2 = \lVert s\rVert^2 + \lVert s^\perp\rVert^2$ (Pythagoras).

In one lineIf $S$ is a finite-dimensional subspace, every $x$ splits uniquely as $x = s + s^\perp$ with $s \in S$ and $s^\perp \in S^\perp$ , and $\lVert x\rVert^2 = \lVert s\rVert^2 + \lVert s^\perp\rVert^2$ .

Why it is trueTake $s$ to be the sum of the shadows of $x$ on an orthonormal basis of $S$ . What is left over is perpendicular to every basis vector.

ProofSpagnolie Lecture 6; Apostol 1.15

The idea: Choose an orthonormal basis of S by Gram–Schmidt. Define s by the shadow formula and s⊥ = x − s. Check that s⊥ is orthogonal to each basis vector. Uniqueness comes from S ∩ S⊥ = {O}, and Pythagoras from s ⟂ s⊥.

Let

q_1, \dots, q_k

be an orthonormal basis of

S

Gram–Schmidt applied to a basis of

S

, then normalizing.

Define

s = \sum_i (x, q_i)q_i

and

s^\perp = x - s

s

is a combination of the

q_i

, so it is in

S

. And

s + s^\perp = x

by definition of

s^\perp

(s^\perp, q_j) = (x, q_j) - (s, q_j) = (x, q_j) - (x, q_j) = 0

for each

j

Expand

(s, q_j)

: only the

i = j

term survives because

(q_i, q_j) = 0

for

i \ne j

and

(q_j, q_j) = 1

s^\perp \in S^\perp

. This proves existence.

Orthogonal to a basis of

S

means orthogonal to all of

S

(the previous theorem).

s - s'

lies in

S \cap S^\perp

Suppose

x = s + s^\perp = s' + s'^\perp

are two splits. Subtracting,

s - s' = s'^\perp - s^\perp

. The left side is in

S

(a subspace) and the right side is in

S^\perp

(a subspace).

s = s'

and

s^\perp = s'^\perp

. This proves uniqueness (and shows the projection does not depend on the chosen basis).

S \cap S^\perp = \{O\}

from the previous theorem.

\lVert x\rVert^2 = \lVert s\rVert^2 + \lVert s^\perp\rVert^2

(s, s^\perp) = 0

because

s \in S

and

s^\perp \in S^\perp

. Expanding

(s + s^\perp, s + s^\perp)

, the cross terms vanish.

The proof shows that the piece $s$ is well defined, so it gets a name.

Definition T

Projection

The vector $s$ in the decomposition is the projection of $x$ onto $S$ , written $\operatorname{proj}_S x$ . If $q_1, \dots, q_k$ is an orthonormal basis of $S$ , it is $s = \sum_{i=1}^k (x, q_i)\,q_i$ . If $e_1, \dots, e_k$ is an orthogonal basis, it is $\sum \dfrac{(x, e_i)}{(e_i, e_i)}e_i$ . For a line spanned by $u$ it is $\dfrac{(x, u)}{(u, u)}u$ , the shadow in the picture.

In one lineThe $S$ -part $s$ of $x = s + s^\perp$ : the shadow of $x$ on the subspace $S$ .

Why it existsIt is the point of $S$ that best stands in for $x$ , which the next theorem proves.

Worked example Spagnolie Lecture 5/6, continuing the example above

Project x = (0, 0, 1) onto the plane S spanned by x1 = (1, 0, 1) and x2 = (1, 1, 0), and find its distance from the plane.

Solution.

Get an orthogonal basis of $S$ . From the Gram–Schmidt example, $y_1 = (1, 0, 1)$ and $y_2 = (\tfrac12, 1, -\tfrac12)$ are orthogonal and span the same plane. The projection formula needs an orthogonal basis; the original $x_1, x_2$ are not orthogonal.
Coefficients. $\dfrac{(x, y_1)}{(y_1, y_1)} = \dfrac12$ and $\dfrac{(x, y_2)}{(y_2, y_2)} = \dfrac{-1/2}{3/2} = -\dfrac13$ .
The projection. $s = \tfrac12(1, 0, 1) - \tfrac13(\tfrac12, 1, -\tfrac12) = (\tfrac13, -\tfrac13, \tfrac23)$ .
The leftover. $s^\perp = x - s = (-\tfrac13, \tfrac13, \tfrac13)$ , which is exactly the $y_3$ of Gram–Schmidt. The last Gram–Schmidt vector is $x_3$ minus its projection on the span of the earlier ones. This is the same computation.
The distance. $\lVert s^\perp\rVert = \sqrt{\tfrac19 + \tfrac19 + \tfrac19} = \tfrac1{\sqrt3} \approx 0.577$ . The distance from a point to a plane is the length of the perpendicular.

Answer. $\operatorname{proj}_S x = (\tfrac13, -\tfrac13, \tfrac23)$ , at distance $1/\sqrt3$ from $x$ . Check with Pythagoras: $\lVert x\rVert^2 = 1 = \lVert s\rVert^2 + \lVert s^\perp\rVert^2 = \tfrac69 + \tfrac39$ .

Your answerThe projection of

x = (5, 5)

onto the line through

u = (1, 2)

c\,u

. Find

c

Worked example Axler 6C

Let S be a subspace of a finite-dimensional Euclidean space. Prove that the orthogonal complement of S⊥ is S.

Solution.

The easy inclusion. Take $s \in S$ . Every $v \in S^\perp$ has $(s, v) = 0$ , so $s$ is orthogonal to all of $S^\perp$ , that is, $s \in (S^\perp)^\perp$ . This is the definition of $S^\perp$ and of $(S^\perp)^\perp$ , unpacked once each.
The hard inclusion: split a vector. Take $x \in (S^\perp)^\perp$ and write $x = s + s^\perp$ with $s \in S$ and $s^\perp \in S^\perp$ . The orthogonal decomposition theorem gives this split for every vector. It needs $S$ finite-dimensional, which holds here.
Show the leftover is zero. $s^\perp = x - s$ . Now $x \in (S^\perp)^\perp$ is orthogonal to $s^\perp \in S^\perp$ , and $s \in S$ is orthogonal to $s^\perp$ . So $(s^\perp, s^\perp) = (x, s^\perp) - (s, s^\perp) = 0 - 0 = 0$ . Both inner products are $0$ for the reasons stated in this step.
Conclude. Axiom 4 gives $s^\perp = O$ , so $x = s \in S$ . A vector with zero squared length is $O$ .

Answer. $(S^\perp)^\perp = S$ . In $\mathbb R^3$ : the perpendicular line to the perpendicular line of a plane is the plane again.

Worked example MATH 375 Lecture 19, version 1

Let A be a real symmetric matrix and x, y eigenvectors with different eigenvalues λ and μ. Prove that x is perpendicular to y.

Solution.

Move $A$ across the dot product. $(Ax)\cdot y = (Ax)^{\mathsf T}y = x^{\mathsf T}A^{\mathsf T}y = x^{\mathsf T}Ay = x\cdot(Ay)$ . $(AB)^{\mathsf T} = B^{\mathsf T}A^{\mathsf T}$ , and $A^{\mathsf T} = A$ because $A$ is symmetric. This is the only place symmetry is used.
Evaluate the left side. $(Ax)\cdot y = \lambda\,(x\cdot y)$ . $Ax = \lambda x$ , and a scalar comes out of the dot product.
Evaluate the right side. $x\cdot(Ay) = \mu\,(x\cdot y)$ . $Ay = \mu y$ .
Compare. $(\lambda - \mu)(x\cdot y) = 0$ . Steps 1 to 3 say $\lambda(x\cdot y) = \mu(x\cdot y)$ .
Use that the eigenvalues differ. $\lambda - \mu \ne 0$ , so $x\cdot y = 0$ . A product of two numbers is $0$ only if one of them is $0$ .

Answer. $x \perp y$ . Check on $S = \begin{bmatrix}5&2\\2&5\end{bmatrix}$ from 2.8: the eigenvectors $(1, 1)$ for $7$ and $(1, -1)$ for $3$ have dot product $1 - 1 = 0$ .

Best approximation

Now the payoff. The closest point of a subspace to $x$ is its projection.

Theorem T Apostol Thm 1.16; Spagnolie Lecture 6

Best approximation theorem

Let $S$ be a finite-dimensional subspace of a Euclidean space $V$ , let $x \in V$ , and let $s = \operatorname{proj}_S x$ . Then for every $t \in S$ ,

\lVert x - t\rVert^2 = \lVert x - s\rVert^2 + \lVert s - t\rVert^2 \ \ge\ \lVert x - s\rVert^2,

with equality only when $t = s$ .

In one lineAmong all points of $S$ , the projection $s$ is the unique closest point to $x$ : $\lVert x - s\rVert \le \lVert x - t\rVert$ for every $t \in S$ .

Why it is trueFor any other point $t$ of the subspace, the three points $x$ , $s$ , $t$ form a right triangle with the right angle at $s$ , so the hypotenuse $x t$ is longer than the leg $x s$ .

Go back to the picture and move the slider so the orange point slides along the line. The green arrow $p$ is $s$ , the orange point is $t$ , the red dashed arrow is $x - s$ , and the right-angle mark is the fact $(x - s) \perp (s - t)$ that the proof uses. The “distance to the orange point” in the readout is always larger than the distance to the shadow, and its square exceeds the shadow’s by exactly the squared length from the shadow to the orange point.

ProofSpagnolie Lecture 6; Apostol 1.16

The idea: Write x − t as the sum of two pieces: x − s, which is perpendicular to S, and s − t, which lies in S. The pieces are perpendicular, so Pythagoras applies.

x - t = (x - s) + (s - t)

Add and subtract

s

. In the picture: the hypotenuse from

x

to the orange point

t

is the sum of the red leg

x - s

and the leg along the line,

s - t

.This is the red dashed arrow plus the segment from the shadow to the orange point.

(x - s,\ s - t) = 0

x - s = s^\perp \in S^\perp

by the decomposition theorem. And

s - t \in S

, since both

s

and

t

are in the subspace

S

\lVert x - t\rVert^2 = \lVert x - s\rVert^2 + \lVert s - t\rVert^2

Expand

(x - t, x - t)

using the two pieces. The cross terms are

2(x - s, s - t) = 0

.This is Pythagoras on the right angle marked in the picture.

\lVert x - t\rVert^2 \ge \lVert x - s\rVert^2

, so

\lVert x - t\rVert \ge \lVert x - s\rVert

A square is never negative.

Equality holds only for

t = s

Equality needs

\lVert s - t\rVert = 0

, and axiom 4 gives

s - t = O

Proof workshopGive the reason for each line: the projection is the closest point

Each line of the proof is shown in order. Choose the reason that justifies it, then press Check.

$x - t = (x - s) + (s - t)$
$(x - s,\ s - t) = 0$
$\lVert x - t\rVert^2 = \lVert x - s\rVert^2 + \lVert s - t\rVert^2$
$\lVert x - t\rVert \ge \lVert x - s\rVert$
$t = s$ if the two distances are equal

The squared error has a neat formula too. Since $x - s = s^\perp$ , Pythagoras from the decomposition theorem gives $\lVert x - s\rVert^2 = \lVert x\rVert^2 - \lVert s\rVert^2$ . With an orthonormal basis $q_i$ of $S$ , that is $\lVert x\rVert^2 - \sum(x, q_i)^2$ . In particular $\sum_i(x, q_i)^2 \le \lVert x\rVert^2$ (Bessel’s inequality), since the error cannot be negative.

Your answer

q_1, q_2

are orthonormal vectors and

x

has

\lVert x\rVert^2 = 30

(x, q_1) = 3

(x, q_2) = 4

. What is the squared distance from

x

to the plane spanned by

q_1

and

q_2

Worked example Apostol 1.17 #8, MATH 375 HW4

Find the linear polynomial g nearest to f(t) = e^t on [-1, 1], with (f, g) = ∫ f g dt, and the squared error.

Solution. The family of linear polynomials $S = \{a + bt\}$ is a 2-dimensional subspace of $C[-1, 1]$ .

Orthonormal basis of $S$ . From the Legendre example, $y_0 = 1$ and $y_1 = t$ are orthogonal, with $(y_0, y_0) = 2$ and $(y_1, y_1) = \tfrac23$ . So $\varphi_0 = \tfrac1{\sqrt2}$ and $\varphi_1 = \sqrt{\tfrac32}\,t$ . The projection formula in this simple form needs an orthonormal basis.
Two integrals. $I_0 = \int_{-1}^1e^t\,dt = e - e^{-1}$ and $I_1 = \int_{-1}^1te^t\,dt = [te^t - e^t]_{-1}^1 = 2e^{-1}$ . Integration by parts. These are the inner products of $f$ with $1$ and with $t$ .
Projection. $g = (f, \varphi_0)\varphi_0 + (f, \varphi_1)\varphi_1 = \dfrac{I_0}{2} + \dfrac{3}{2}I_1\,t = \dfrac{e - e^{-1}}{2} + \dfrac{3}{e}\,t \approx 1.1752 + 1.1036\,t$ . $(f, \varphi_0)\varphi_0 = \tfrac{I_0}{\sqrt2}\cdot\tfrac1{\sqrt2}$ and $(f, \varphi_1)\varphi_1 = \tfrac32I_1t$ .
Squared error. $\lVert f - g\rVert^2 = \lVert f\rVert^2 - \lVert g\rVert^2 = \dfrac{e^2 - e^{-2}}{2} - \Big(\dfrac{I_0^2}{2} + \dfrac32I_1^2\Big) = 1 - 7e^{-2} \approx 0.0527$ . The error formula above. Here $\dfrac{I_0^2}{2} = \dfrac{e^2 - 2 + e^{-2}}{2}$ and $\dfrac32I_1^2 = 6e^{-2}$ .
Check numerically. $\lVert f\rVert^2 = 3.6269$ , $\lVert g\rVert^2 = 2.7622 + 0.8120 = 3.5742$ , difference $0.0527$ . A second route to the same number.

Answer. $g(t) = \tfrac12(e - e^{-1}) + 3e^{-1}t$ , with squared error $1 - 7/e^2 \approx 0.0527$ . (Calculus gives the same by minimizing $\lVert f - a - bt\rVert^2$ over $a, b$ : setting the two partial derivatives to zero gives the same $a, b$ . Those equations are the “normal equations” of chapter 2.10.)

Quick checkIn the best approximation theorem, why is the projection

s

closer to

x

than any other point

t

S

Fourier series: best approximation by waves

Where this came from· Heat, waves and least squares

Fourier presented his memoir on how heat spreads through a solid body to the Paris Institute in December 1807 (the book Théorie analytique de la chaleur followed in 1822). He wrote the starting temperature as a sum of sine and cosine waves, because the heat equation treats each wave separately (each one fades at its own rate), and added the results. Lagrange, one of the committee that read the memoir, objected to expanding an arbitrary function in a trigonometric series. The inner-product view of this chapter explains the practical side: the coefficients are inner products, and a partial sum is a best approximation. Gram (1883) studied expanding functions in series by least squares and gave the orthogonalization process there. Erhard Schmidt (1907) used the process for integral equations, and acknowledged Gram’s earlier work. The process now carries both names, although Laplace was already familiar with the idea.

Take $V = C[0, 2\pi]$ with $(f, g) = \int_0^{2\pi}f(x)g(x)\,dx$ . The functions

\varphi_0 = \tfrac1{\sqrt{2\pi}}, \qquad \varphi_{2k-1} = \tfrac1{\sqrt\pi}\cos kx, \qquad \varphi_{2k} = \tfrac1{\sqrt\pi}\sin kx \qquad (k = 1, 2, \dots)

are orthonormal. (The check uses the product-to-sum identities. The next example does the cosine case on $[0, \pi]$ , and on $[0, 2\pi]$ the computation is the same.) Let $S_n$ be the span of the first $2n + 1$ of them. By the best approximation theorem, the closest function in $S_n$ to $f$ is the projection, whose coefficients are inner products:

s_n = \sum_{j=0}^{2n}(f, \varphi_j)\,\varphi_j, \qquad (f, \varphi_j) = \int_0^{2\pi}f(x)\varphi_j(x)\,dx.

This is the partial Fourier series of $f$ . “Closest” means smallest $\int_0^{2\pi}(f - s)^2$ , the root-mean-square error. The infinite sum is a different matter (the $\varphi_j$ are an orthonormal “basis” only in a limiting sense, which analysis in MATH 521 makes precise), and we use only partial sums and their errors.

Worked example Apostol 1.17 #3, MATH 375 HW3

Show that cos(mt) and cos(nt) are orthogonal on [0, π] for m ≠ n, and find the norm of cos(nt).

Solution.

Use the product-to-sum identity. $\cos mt\cos nt = \tfrac12\big[\cos(m + n)t + \cos(m - n)t\big]$ . Turns a product into a sum of two cosines, each easy to integrate.
Integrate for $m \ne n$ . $\int_0^\pi\cos mt\cos nt\,dt = \tfrac12\Big[\tfrac{\sin(m+n)t}{m+n} + \tfrac{\sin(m-n)t}{m-n}\Big]_0^\pi = 0$ . $\sin$ of an integer multiple of $\pi$ is $0$ , and both denominators are non-zero.
Integrate for $m = n \ge 1$ . $\int_0^\pi\cos^2nt\,dt = \tfrac12\int_0^\pi(\cos2nt + 1)\,dt = \tfrac\pi2$ . Now $m - n = 0$ , so the second cosine is the constant $1$ .
For $n = 0$ : $\int_0^\pi1\,dt = \pi$ .
Normalize. $y_0 = \tfrac1{\sqrt\pi}$ and $y_n = \sqrt{\tfrac2\pi}\cos nt$ are orthonormal. Divide each function by its norm, $\sqrt\pi$ and $\sqrt{\pi/2}$ .

Answer. The cosines are orthogonal, with $\lVert1\rVert^2 = \pi$ and $\lVert\cos nt\rVert^2 = \pi/2$ .

Worked example Spagnolie Lecture 6 (the Mathematica example), numbers computed here

Approximate the triangle wave f(x) = x on [0, π], 2π − x on [π, 2π] by constants, then by a constant plus a cosine, and measure the error.

Solution.

The squared length of $f$ . $\lVert f\rVert^2 = 2\int_0^\pi x^2\,dx = \tfrac{2\pi^3}{3} \approx 20.671$ . The two halves are mirror images.
Best constant. The coefficient on $\varphi_0$ is $(f, \varphi_0) = \tfrac1{\sqrt{2\pi}}\int_0^{2\pi}f = \tfrac{\pi^2}{\sqrt{2\pi}}$ , so the best constant is $\tfrac{\pi^2}{\sqrt{2\pi}}\cdot\tfrac1{\sqrt{2\pi}} = \tfrac\pi2$ , the average height of the wave. Its squared “energy” is $(f, \varphi_0)^2 = \tfrac{\pi^3}{2} \approx 15.503$ . The area under the triangle is $\pi^2$ . The projection onto the constants is the average value.
Error of the best constant. $\lVert f - \tfrac\pi2\rVert^2 = \tfrac{2\pi^3}{3} - \tfrac{\pi^3}{2} = \tfrac{\pi^3}{6} \approx 5.168$ . Error $=\lVert f\rVert^2 - \lVert s\rVert^2$ , from the best approximation theorem.
The cosine term. $(f, \varphi_1) = \tfrac1{\sqrt\pi}\int_0^{2\pi}f(x)\cos x\,dx = -\tfrac4{\sqrt\pi} \approx -2.257$ . (The sine coefficient is $0$ because $f$ is symmetric about $x = \pi$ .) Integration by parts on each half; the answer matches a numerical integration.
Add it. $s = \tfrac\pi2 - \tfrac4\pi\cos x$ . The squared energy added is $(f, \varphi_1)^2 = \tfrac{16}{\pi} \approx 5.093$ , so the new error is $\tfrac{\pi^3}{6} - \tfrac{16}\pi \approx 0.0748$ . Each new orthonormal function removes exactly its coefficient squared from the error.
Read the result. Two terms already remove $98.6\%$ of the error left by the constant. Adding $\cos3x$ (energy $\tfrac{16}{81\pi}$ ) brings the error to $0.0119$ , and $\cos5x$ to $0.0037$ . The coefficients fall off like $1/k^2$ for this wave because it has corners but no jumps. This is the shape of the three Mathematica plots in Spagnolie’s Lecture 6.

Answer. $f \approx \tfrac\pi2 - \tfrac4\pi\Big(\cos x + \tfrac{\cos3x}{9} + \tfrac{\cos5x}{25} + \cdots\Big)$ . At $x = 0$ the two-term version gives $\tfrac\pi2 - \tfrac4\pi = 0.30$ where the true value is $0$ .

Where it is used

Used inside maths· Inside maths: a sum of 1/k⁴ from Pythagoras

For the triangle wave, the squared length of $f$ is the sum of the squares of its coefficients (Parseval’s equality, which holds because the trigonometric system is complete; Bessel’s inequality above gives only " $\le$ ", and completeness, which is MATH 521 material, gives " $=$ "). The nonzero coefficients are the constant and the cosines of odd frequency $k$ , and the square of the coefficient of $\cos kx$ is $\dfrac{16}{\pi k^4}$ . So

\frac{2\pi^3}{3} = \frac{\pi^3}{2} + \frac{16}{\pi}\sum_{k\ \text{odd}}\frac1{k^4}, \qquad\text{hence}\qquad 1 + \frac1{3^4} + \frac1{5^4} + \cdots = \frac{\pi^4}{96}.

A numerical check: $\pi^4/96 = 1.014678\ldots$ , and the first terms $1 + 0.012346 + 0.0016 + 0.00042 + \cdots$ add to the same. An infinite sum, found by applying Pythagoras in a function space.

Used in the real world· Signal processing: a Fourier coefficient is an inner product

An audio signal on one second of time, $x(t) = 3\cos(2\pi\cdot5t) + \sin(2\pi\cdot12t)$ , is a mix of a 5 Hz cosine and a 12 Hz sine. To find how much of the 5 Hz cosine is present, take the inner product on $[0, 1]$ : $(x, \cos2\pi5t) = \tfrac32$ and $(\cos2\pi5t, \cos2\pi5t) = \tfrac12$ , so the amount is $\tfrac{3/2}{1/2} = 3$ , which is right. Likewise $(x, \sin2\pi12t) = \tfrac12$ gives amount $1$ . The inner product with any other frequency (say $\cos2\pi12t$ , $\sin2\pi5t$ or $\cos2\pi7t$ ) is $0$ , because the waves are orthogonal. This is exactly the coordinate theorem for an orthogonal basis. A JPEG file does the same with a cosine basis: each $8\times8$ block of pixels is treated as a vector with $64$ numbers and expanded in $64$ orthonormal cosine patterns; the small coefficients are thrown away. By the best approximation theorem, keeping the largest coefficients gives the closest picture that uses that many patterns. MP3 applies a related cosine transform to short frames of audio.

Used in the real world· Search and AI: cosine similarity

A text-embedding model turns each word or sentence into a vector (hundreds or thousands of numbers). Similar meanings should give similar directions, so the similarity is the cosine of the angle, defined above. Toy example with three numbers: $\text{king} = (0.9, 0.8, 0.1)$ , $\text{queen} = (0.85, 0.75, 0.2)$ , $\text{apple} = (0.1, 0.2, 0.95)$ . Then $(\text{king}, \text{queen}) = 1.385$ , with norms $1.208$ and $1.151$ , so $\cos\theta = 0.9958$ and $\theta \approx 5.3°$ . But $(\text{king}, \text{apple}) = 0.345$ with norms $1.208$ and $0.976$ gives $\cos\theta = 0.293$ , $\theta \approx 73°$ . Cauchy–Schwarz is what guarantees every such number lies in $[-1, 1]$ , whatever the dimension. Multiplying a vector by $10$ changes the inner product but not the cosine, so similarity ignores length. A transformer’s attention scores $q\cdot k$ are inner products of the same kind.

A preview· Preview: the closest point is least squares (chapter 2.10)

A GPS receiver measures its distance to four or more satellites. Each measurement is one equation in four unknowns (three position coordinates and the receiver’s clock error). With more than four equations and noisy measurements there is usually no exact solution. The fix is to find the vector of unknowns whose predicted measurements are closest to the actual ones. The predicted measurements lie in a subspace (the span of the columns of a matrix), so the closest point is the projection of the data onto that subspace, and the error vector is perpendicular to it. Chapter 2.10 turns this into a formula. Fitting a line to data points works the same way, as in the $e^t$ example above where the “data” is a whole function.

Beyond the course: for understanding· QR decomposition: Gram–Schmidt written as a matrix factorization

Let $A$ have the columns $x_1, x_2, x_3$ of the Gram–Schmidt example. Put the orthonormal vectors $\varphi_1, \varphi_2, \varphi_3$ as the columns of a matrix $Q$ . Since $x_m$ is $y_m$ plus a combination of earlier $y$ ‘s, each $x_m$ is a combination of $\varphi_1, \dots, \varphi_m$ with coefficients $r_{im} = (\varphi_i, x_m)$ , and so $A = QR$ with $R$ upper triangular. Here

R = \begin{bmatrix}\sqrt2 & \tfrac{\sqrt2}{2} & \tfrac{\sqrt2}{2}\\ 0 & \tfrac{\sqrt6}{2} & -\tfrac{\sqrt6}{6}\\ 0 & 0 & \tfrac{\sqrt3}{3}\end{bmatrix} = \begin{bmatrix}1.414&0.707&0.707\\0&1.225&-0.408\\0&0&0.577\end{bmatrix},

and one can check $QR = A$ entry by entry (for example the first column is $\sqrt2\varphi_1 = (1, 0, 1)$ ). This factorization is how numerical software solves least-squares problems, because $Q$ has perpendicular unit columns and so does not magnify errors, and $R$ is solved by back-substitution. In practice the classical process above loses perpendicularity to rounding errors on hard matrices, so libraries use Householder reflections or the “modified” Gram–Schmidt instead.

Beyond the course: for understanding· Orthogonal polynomials

Changing the interval or the weight in the inner product gives other families by the same process. On $[-1, 1]$ with weight $1$ , Gram–Schmidt on $1, t, t^2, \dots$ gives the Legendre polynomials (electrostatic potentials, Gaussian quadrature). With weight $1/\sqrt{1 - t^2}$ you get Chebyshev polynomials (used for stable polynomial approximation). On $[0, \infty)$ with weight $e^{-t}$ it gives the Laguerre polynomials: $1,\ t - 1,\ t^2 - 4t + 2,\ t^3 - 9t^2 + 18t - 6$ (Apostol 1.17 #5). With weight $e^{-t^2}$ on the whole line you get Hermite polynomials, which describe the quantum harmonic oscillator. All of them are one idea: the inner product decides what “orthogonal” means.

Traps

Watch out· Orthogonal depends on the inner product

With the dot product $(1, 1)$ and $(1, -1)$ are orthogonal. With $(x, y) = x_1y_1 + 2x_2y_2$ their inner product is $1 - 2 = -1$ , so they are not. In function spaces, whether $1$ and $t^2$ are orthogonal depends on the interval and the weight: on $[-1, 1]$ with weight $1$ they are not ( $\tfrac23$ ). Always say which inner product.

Watch out· Orthogonal implies independent, not the reverse; and O is excluded

An orthogonal set of non-zero vectors is independent. An independent set need not be orthogonal ( $(1,0)$ and $(1,1)$ ). And the zero vector is orthogonal to everything, so including it in a list breaks the theorem: $\{O, x\}$ is orthogonal but dependent.

Watch out· Gram–Schmidt: project onto the y's, divide by (y, y)

Two slips: projecting $x_3$ onto the original $x_1, x_2$ (they are not perpendicular, so you remove some directions twice), and dividing by $\lVert y\rVert$ instead of $(y, y) = \lVert y\rVert^2$ . A third: arithmetic. Spagnolie’s own class notes got $a = 1$ and $b = +\tfrac13$ for the example above, instead of $\tfrac12$ and $-\tfrac13$ , and the resulting $y_3$ was not orthogonal to $y_1$ . Always check $(y_i, y_j) = 0$ at the end. A zero vector appearing means the $x$ ‘s were dependent.

Watch out· Adding projections works only for an orthogonal basis

Let $S = \mathbb R^2$ with the basis $(1, 0), (1, 1)$ and $x = (0, 1)$ . Projecting $x$ onto each basis vector separately gives $(0, 0)$ and $(\tfrac12, \tfrac12)$ , and their sum $(\tfrac12, \tfrac12)$ is not $x$ , though $x$ is in $S$ . The sum-of-shadows formula needs the basis vectors to be perpendicular.

Watch out· The norm scales by |c|, and not every integral is a norm

$\lVert cx\rVert = |c|\,\lVert x\rVert$ . A student summary sheet wrote $c\,\lVert x\rVert$ , which would give negative lengths for negative $c$ . And $\int F'^2$ (the example in the beyond-box) fails all three norm rules. Check the rules before calling something a norm.

Watch out· Fourier 'basis' and cosine similarity are used loosely

A Fourier “basis” is infinite, so the expansion of $f$ is a limit, and a statement like ” $f$ equals its Fourier series” needs a notion of convergence (and at jumps it is false pointwise). Cosine similarity ignores length: two vectors pointing the same way are “identical” however different their sizes. Be sure that is what you want.

Problems

check

Is $(x, y) = x_1y_1 + 2x_2y_2$ an inner product on $\mathbb R^2$ ? Are $(1, 1)$ and $(1, -1)$ orthogonal for it?

Solution

Symmetry, additivity and homogeneity are clear (it is the dot product with the second coordinate weighted by $2$ ). Positivity: $(x, x) = x_1^2 + 2x_2^2 > 0$ unless $x = O$ . So yes, it is an inner product. $((1,1), (1,-1)) = 1 - 2 = -1 \ne 0$ , so they are not orthogonal for it, although they are for the dot product.

check

In $C[-1, 1]$ with $(f, g) = \int_{-1}^1fg\,dt$ , are $1$ and $t^2$ orthogonal? Are $t$ and $t^2 - \tfrac13$ ?

Solution

$(1, t^2) = \int_{-1}^1t^2\,dt = \tfrac23 \ne 0$ : not orthogonal (Spagnolie’s example showing the monomials are not orthogonal). $(t, t^2 - \tfrac13) = \int_{-1}^1(t^3 - \tfrac t3)\,dt = 0$ since the integrand is odd: orthogonal.

check

For $x = (3, 4)$ and $y = (4, 3)$ in $\mathbb R^2$ , compute $(x, y)$ , $\lVert x\rVert$ , $\lVert y\rVert$ and check Cauchy–Schwarz.

Solution

$(x, y) = 12 + 12 = 24$ . $\lVert x\rVert = \lVert y\rVert = 5$ . Then $|(x, y)| = 24 \le 25 = \lVert x\rVert\lVert y\rVert$ . The cosine of the angle is $24/25 = 0.96$ ( $\theta \approx 16.3°$ ). They are not dependent, so the inequality is strict.

check

An orthonormal basis $q_1, q_2$ of $\mathbb R^2$ is given, and $x$ has $(x, q_1) = 3$ and $(x, q_2) = -4$ . What is $\lVert x\rVert$ ?

Solution

$x = 3q_1 - 4q_2$ , so $\lVert x\rVert^2 = 3^2 + (-4)^2 = 25$ and $\lVert x\rVert = 5$ .

useApostol 1.17 #2(b), MATH 375 HW3

Apply Gram–Schmidt to $x_1 = (1, 1, 0, 1)$ , $x_2 = (1, 0, 2, 1)$ , $x_3 = (1, 2, -2, 1)$ in $\mathbb R^4$ and find an orthonormal basis of their span.

Solution

$y_1 = x_1$ , $(y_1, y_1) = 3$ . $(y_1, x_2) = 2$ , so $y_2 = x_2 - \tfrac23y_1 = (\tfrac13, -\tfrac23, 2, \tfrac13) = \tfrac13(1, -2, 6, 1)$ and $(y_2, y_2) = \tfrac{14}{3}$ . For $x_3$ : $(y_1, x_3) = 4$ and $(y_2, x_3) = \tfrac13(1 - 4 - 12 + 1) = -\tfrac{14}{3}$ . So $y_3 = x_3 - \tfrac43y_1 + 1\cdot y_2 = (1 - \tfrac43 + \tfrac13,\ 2 - \tfrac43 - \tfrac23,\ -2 + 2,\ 1 - \tfrac43 + \tfrac13) = (0, 0, 0, 0)$ . A zero vector means $x_3$ was dependent on $x_1, x_2$ ( $x_3 = \tfrac43x_1 - y_2$ ). Discard it. The span is 2-dimensional with orthonormal basis $\tfrac1{\sqrt3}(1, 1, 0, 1)$ and $\tfrac1{\sqrt{42}}(1, -2, 6, 1)$ (since $\lVert(1,-2,6,1)\rVert^2 = 42$ ).

useApostol 1.17 #6, MATH 375 HW4

In $C[1, 3]$ with $(f, g) = \int_1^3fg\,dx$ , find the constant function nearest to $f(x) = 1/x$ and the squared error.

Solution

The constants form a 1-dimensional subspace with orthonormal basis $\varphi_0 = 1/\sqrt2$ (since $\int_1^31 = 2$ ). The nearest constant is $(f, \varphi_0)\varphi_0 = \tfrac12\int_1^3\tfrac1x\,dx = \tfrac12\log3 \approx 0.549$ (the average value of $1/x$ ). The squared error is $\lVert f\rVert^2 - \lVert g\rVert^2 = \int_1^3x^{-2}\,dx - 2\cdot(\tfrac12\log3)^2 = \tfrac23 - \tfrac12(\log3)^2 \approx 0.0632$ , so the distance from $f$ to the nearest constant is $\sqrt{0.0632} \approx 0.251$ .

use

Project $x = (1, 2, 3)$ onto the plane $S$ spanned by $(1, 1, 0)$ and $(0, 1, 1)$ . Find the projection, the perpendicular part, and the distance from $x$ to the plane.

Solution

Make an orthogonal basis: $y_1 = (1, 1, 0)$ , and $y_2 = (0, 1, 1) - \tfrac12(1, 1, 0) = (-\tfrac12, \tfrac12, 1)$ (since $(y_1, (0,1,1)) = 1$ and $(y_1, y_1) = 2$ ). Then $(x, y_1) = 3$ , so the coefficient is $\tfrac32$ ; $(x, y_2) = -\tfrac12 + 1 + 3 = \tfrac72$ and $(y_2, y_2) = \tfrac32$ , so the coefficient is $\tfrac73$ . The projection is $\tfrac32(1,1,0) + \tfrac73(-\tfrac12,\tfrac12,1) = (\tfrac13, \tfrac83, \tfrac73)$ . The perpendicular part is $x - s = (\tfrac23, -\tfrac23, \tfrac23)$ . Check: its inner products with $(1,1,0)$ and $(0,1,1)$ are both $0$ . The distance is $\lVert(\tfrac23,-\tfrac23,\tfrac23)\rVert = \tfrac{2}{\sqrt3} \approx 1.155$ .

useApostol 1.17 #5, MATH 375 HW3

On the functions with $\int_0^\infty e^{-t}f^2\,dt < \infty$ , with $(f, g) = \int_0^\infty e^{-t}fg\,dt$ , apply Gram–Schmidt to $1, t, t^2$ . (Use $I_n = \int_0^\infty e^{-t}t^n\,dt = n!$ .)

Solution

$y_0 = 1$ , $(y_0, y_0) = I_0 = 1$ . For $y_1 = t - a\,y_0$ : $a = (y_0, t)/(y_0, y_0) = I_1 = 1$ , so $y_1 = t - 1$ , with $(y_1, y_1) = I_2 - 2I_1 + I_0 = 2 - 2 + 1 = 1$ . For $y_2 = t^2 - a\,y_0 - b\,y_1$ : $a = I_2 = 2$ and $b = (y_1, t^2)/(y_1, y_1) = (I_3 - I_2)/1 = 6 - 2 = 4$ . So $y_2 = t^2 - 2 - 4(t - 1) = t^2 - 4t + 2$ . (Check: $(y_0, y_2) = I_2 - 4I_1 + 2I_0 = 2 - 4 + 2 = 0$ .)

proveApostol 1.13 #3, MATH 375 HW3

Prove that $(x, y) = 0$ if and only if $\lVert x + y\rVert = \lVert x - y\rVert$ , in any Euclidean space.

Solution

Expand: $\lVert x + y\rVert^2 = \lVert x\rVert^2 + 2(x, y) + \lVert y\rVert^2$ and $\lVert x - y\rVert^2 = \lVert x\rVert^2 - 2(x, y) + \lVert y\rVert^2$ . Subtracting, $\lVert x + y\rVert^2 - \lVert x - y\rVert^2 = 4(x, y)$ . If $(x, y) = 0$ the squares are equal, and norms are non-negative, so $\lVert x + y\rVert = \lVert x - y\rVert$ . Conversely, equal norms give $4(x, y) = 0$ . (Geometrically: the two diagonals of a parallelogram have equal length exactly when it is a rectangle.)

proveApostol 1.13 exercises

Prove the parallelogram law $\lVert x + y\rVert^2 + \lVert x - y\rVert^2 = 2\lVert x\rVert^2 + 2\lVert y\rVert^2$ and the polarization identity $(x, y) = \tfrac14\big(\lVert x + y\rVert^2 - \lVert x - y\rVert^2\big)$ .

Solution

From the two expansions in the previous solution, adding gives $2\lVert x\rVert^2 + 2\lVert y\rVert^2$ , which is the parallelogram law, and subtracting gives $4(x, y)$ , which is polarization. So the inner product can be recovered from the norm alone.

provefollows Apostol 1.15 and 1.16

Prove Bessel’s inequality: if $q_1, \dots, q_k$ are orthonormal, then $\sum_{i=1}^k(x, q_i)^2 \le \lVert x\rVert^2$ for every $x$ .

Solution

Let $S = L(q_1, \dots, q_k)$ and $s = \operatorname{proj}_Sx = \sum(x, q_i)q_i$ . The $q_i$ are orthonormal, so $\lVert s\rVert^2 = \sum(x, q_i)^2$ (the coordinate theorem). By the decomposition theorem, $\lVert x\rVert^2 = \lVert s\rVert^2 + \lVert s^\perp\rVert^2 \ge \lVert s\rVert^2$ . So $\sum(x, q_i)^2 \le \lVert x\rVert^2$ , with equality exactly when $s^\perp = O$ , that is, when $x \in S$ .

proveAxler 6C

Let $V$ be finite-dimensional and $S$ a subspace. Prove $(S^\perp)^\perp = S$ .

Solution

( $\supseteq$ ) If $s \in S$ and $v \in S^\perp$ , then $(s, v) = 0$ , so $s$ is orthogonal to every vector of $S^\perp$ , that is, $s \in (S^\perp)^\perp$ . ( $\subseteq$ ) Let $x \in (S^\perp)^\perp$ and write $x = s + s^\perp$ with $s \in S$ , $s^\perp \in S^\perp$ (decomposition theorem). Then $s^\perp = x - s$ . Now $x \in (S^\perp)^\perp$ is orthogonal to $s^\perp$ , and $s \in S$ is also orthogonal to $s^\perp$ . So $(s^\perp, s^\perp) = (x - s, s^\perp) = (x, s^\perp) - (s, s^\perp) = 0 - 0 = 0$ , and axiom 4 gives $s^\perp = O$ . Hence $x = s \in S$ .

proveMATH 375 Lecture 19 (version 1); Apostol 5.x preview

Let $A$ be a real symmetric matrix ( $A^{\mathsf T} = A$ ) and let $x, y$ be eigenvectors of $A$ with different eigenvalues $\lambda \ne \mu$ (see 2.8). Prove $x \perp y$ for the dot product.

Solution

Use $(Ax)\cdot y = (Ax)^{\mathsf T}y = x^{\mathsf T}A^{\mathsf T}y = x^{\mathsf T}Ay = x\cdot(Ay)$ , which holds because $A^{\mathsf T} = A$ . Then $(Ax)\cdot y = \lambda\,(x\cdot y)$ and $x\cdot(Ay) = \mu\,(x\cdot y)$ . So $(\lambda - \mu)(x\cdot y) = 0$ , and since $\lambda \ne \mu$ , $x\cdot y = 0$ . This is why the ellipse $5x^2 + 4xy + 5y^2 = 1$ of 2.8 had perpendicular axes, and it is half of the spectral theorem (chapter 3.1).

stretch

Let $S$ be a finite-dimensional subspace of $V$ and $P(x) = \operatorname{proj}_Sx$ . Prove that $P$ is a linear map with $P\circ P = P$ , range $S$ , and null space $S^\perp$ . If $V$ is finite-dimensional, deduce $\dim V = \dim S + \dim S^\perp$ .

Solution

Linearity: with an orthonormal basis $q_i$ of $S$ , $P(x) = \sum(x, q_i)q_i$ , and $x \mapsto (x, q_i)$ is linear by additivity and homogeneity. For $s \in S$ , the decomposition of $s$ is $s + O$ , so $P(s) = s$ . Every $P(x)$ is in $S$ , so $P(P(x)) = P(x)$ , and the range is exactly $S$ . $P(x) = O$ means $x = O + s^\perp$ , so $x \in S^\perp$ ; conversely $x \in S^\perp$ has decomposition $O + x$ , so $P(x) = O$ . Thus $N(P) = S^\perp$ . By rank–nullity (2.3): $\dim V = \operatorname{rank}P + \operatorname{nullity}P = \dim S + \dim S^\perp$ .

stretch

Prove that $\lVert x + y\rVert = \lVert x\rVert + \lVert y\rVert$ holds if and only if one of $x, y$ is a non-negative multiple of the other.

Solution

Square both sides: $\lVert x\rVert^2 + 2(x, y) + \lVert y\rVert^2 = \lVert x\rVert^2 + 2\lVert x\rVert\lVert y\rVert + \lVert y\rVert^2$ , so equality holds iff $(x, y) = \lVert x\rVert\lVert y\rVert$ . If $y = O$ or $x = O$ this is true and one is $0$ times the other. Otherwise this is equality in Cauchy–Schwarz (with a plus sign), which requires $x = cy$ for some $c$ . Then $(cy, y) = c\lVert y\rVert^2$ and $\lVert cy\rVert\lVert y\rVert = |c|\lVert y\rVert^2$ , so $c = |c|$ , that is, $c \ge 0$ . Conversely, if $x = cy$ with $c \ge 0$ , both sides equal $(c + 1)\lVert y\rVert$ .

Summary

An inner product $(x, y)$ is symmetric, additive, homogeneous, and positive on non-zero vectors. The dot product is one; $\int fg$ is another. From it we define the norm $\lVert x\rVert = \sqrt{(x,x)}$ and the angle $\cos\theta = (x, y)/(\lVert x\rVert\lVert y\rVert)$ . JEE derived the dot product’s cosine formula; we define the angle by it.
Cauchy–Schwarz $|(x,y)| \le \lVert x\rVert\lVert y\rVert$ (proof: the parabola $\lVert x + ty\rVert^2 \ge 0$ at its lowest point) is what makes the angle defined. It gives the triangle inequality.
Orthogonal means $(x, y) = 0$ . Orthogonal non-zero vectors are independent. In an orthogonal basis the coordinate of $x$ on $e_i$ is $(x, e_i)/(e_i, e_i)$ , with no equations to solve.
Gram–Schmidt builds an orthogonal basis from any basis: subtract from each vector its projections on the earlier $y$ ‘s. A zero output means dependence. The Legendre polynomials come from doing it to $1, t, t^2, \dots$ .
Orthogonal complement $S^\perp$ ; every $x$ splits uniquely as $s + s^\perp$ (the projection plus a perpendicular leftover) and $\lVert x\rVert^2 = \lVert s\rVert^2 + \lVert s^\perp\rVert^2$ .
Best approximation: the projection is the unique closest point of $S$ to $x$ . Proof: Pythagoras on the right triangle $x, s, t$ . The error is $\lVert x\rVert^2 - \lVert s\rVert^2$ .
Fourier coefficients are inner products with orthonormal waves, and a partial Fourier sum is a projection. Cosine similarity, JPEG, the least squares of chapter 2.10 and QR decomposition are the same picture.

Watch and read

The same ideas explained by someone else. Use these when an explanation here does not land.

2.9 Inner products, orthogonality, Gram–Schmidt, projection

The picture

The idea, in words

Definitions and theorems

Inner products

Inner product

Length and angle

Norm

Cauchy–Schwarz inequality

Orthogonal

Orthonormal

Properties of the norm

Orthogonal sets and coordinates

Orthogonal sets are independent

Coordinates in an orthogonal basis

Gram–Schmidt: making an orthogonal basis

Gram–Schmidt process

Gram–Schmidt theorem

Orthogonal complement and projection

Orthogonal complement

The orthogonal complement is a subspace meeting S only at O

Orthogonal decomposition theorem

Projection

Best approximation

Best approximation theorem

Fourier series: best approximation by waves

Where it is used

Traps

Problems

Summary

Watch and read

Where this sits

Comes from

Leads to

Same idea elsewhere