5.6 Taylor's formula, the Hessian and extrema

Near a point, a C² field equals its value, plus its gradient dotted with the step, plus half a quadratic form in the step whose symmetric matrix is the Hessian. At a critical point the gradient term is zero, so the sign of the quadratic form, read off from the Hessian's eigenvalues by the spectral theorem, decides between maximum, minimum and saddle. Absolute extrema on a closed bounded set are found by comparing the critical points with the boundary.

In one variable, a point where $f' = 0$ is a maximum if $f'' < 0$ , a minimum if $f'' > 0$ , and the test says nothing if $f'' = 0$ . The reason is Taylor’s formula: $f(a + h) \approx f(a) + \frac12f''(a)h^2$ when $f'(a) = 0$ . In many variables the same argument works, but the second derivative is now a matrix, and “positive” has to mean “positive in every direction at once”. This chapter makes that precise. The matrix of second derivatives is symmetric (5.5), so the spectral theorem (3.1) turns the question into one about its eigenvalues. The last section does the other half of an optimisation problem: on a closed bounded set the extremum may sit on the boundary, where the gradient need not vanish.

Before you start

1.4: Taylor’s theorem with the Lagrange remainder, and the one-variable second-derivative test. 3.2: positive definite matrices and their eigenvalues. 5.2: the gradient is zero at an interior extremum. 5.5: the matrix of second partials is symmetric.

Quick checkFor

f(x) = x^4

we have

f'(0) = 0

and

f''(0) = 0

. What does the one-variable second-derivative test say about

x = 0

Quick checkA symmetric matrix

A

has eigenvalues

3

and

-1

. The quadratic form

v^{\mathsf T}Av

is:

Quick check

\nabla f(a) = O

. Must

a

be a local maximum or a local minimum?

The picture: bowl, dome, saddle

Move the two eigenvalue sliders. Both positive: a bowl, with level curves that are ellipses. Both negative: a dome. One of each: a saddle, whose level curves are hyperbolas and whose zero level set (the orange curve, through the critical point) is two crossing lines. The dashed coloured lines in the right-hand map are the eigenvector directions of the Hessian: the surface curves up along a green one and down along a red one. Rotate the axes with the angle slider: the surface does not change shape, only the matrix entries do, and the type stays the same. That is the whole content of the chapter: the type of a critical point is the sign pattern of the eigenvalues, and it does not depend on the coordinates. The same saddle, as a real surface with contours:

The idea, in words

Suppose $\nabla f(a) = O$ . The first-order Taylor formula of 5.3 says $f(a + v) - f(a)$ is smaller than any multiple of $\lVert v \rVert$ . To see which sign it has, go one order further. In one variable the next term is $\frac12f''(a)v^2$ . In $n$ variables it is $\frac12\sum_{i,j}D_{ij}f(a)\,v_iv_j = \frac12v^{\mathsf T}Hv$ : a quadratic form (3.2) in the step. Whatever the leftover is, it is small compared with $\lVert v \rVert^2$ . So if the quadratic form is positive for every nonzero $v$ , with room to spare, then $f(a + v) > f(a)$ for small $v$ : a minimum. If it takes both signs, then $f$ is above $f(a)$ in some directions and below it in others: a saddle. The eigenvalues are how one reads off the signs: rotate to the eigen-axes and the form becomes $\sum\lambda_iw_i^2$ .

Where this came from· Taylor, Hesse and a surface that looked like a minimum

Brook Taylor published the series that bears his name in his 1715 book Methodus incrementorum directa et inversa, as a formula with no careful treatment of convergence or of the size of the error. The remainder in the form used in 1.4 is credited to Lagrange, in his Théorie des fonctions analytiques of 1797. The determinant of the matrix of second derivatives was introduced by Otto Hesse in the early 1840s, in his work on the inflection points of cubic curves (he called it a functional determinant), and the matrix now carries his name. Serret’s calculus textbook of 1886 gave a criterion for maxima and minima in terms of the terms of order two, three and four. Peano’s example $(2x^2 - y)(y - x^2)$ , which he presented in 1899 in the German translation of Genocchi’s calculus textbook, defeats it: along every line through the origin the function has a strict maximum, and yet it takes positive values arbitrarily close to the origin, so the origin is a saddle. The one-variable habit of checking each direction fails because the positive region lies between two parabolas, which every straight line through the origin misses near the origin.

Definitions

Definition T

Hessian matrix

For a scalar field $f$ with second partials at $a$ , the Hessian matrix is $H_f(a) = \big[D_{ij}f(a)\big]_{i,j = 1}^n$ , where $D_{ij}f = D_jD_if$ means “differentiate in $x_i$ first, then in $x_j$ ” (this is entry $(i, j)$ of the Jacobian of $\nabla f$ , as in 5.5). Apostol (9.10) and Lecture 24 write $D_{ij}f = D_i(D_jf)$ , which puts the same numbers in the transposed position; for a $C^2$ field the two matrices are equal, so nothing below depends on the choice. In two variables

H_f(a) = \begin{bmatrix}f_{xx}(a) & f_{xy}(a)\\ f_{yx}(a) & f_{yy}(a)\end{bmatrix}.

It is the Jacobian matrix of the gradient field $\nabla f$ (5.5). If $f$ is $C^2$ it is symmetric, by Schwarz’s theorem (5.5), so its eigenvalues are real and it has an orthonormal basis of eigenvectors (3.1). Also written $\nabla^2f(a)$ .

In one lineThe $n \times n$ matrix $H_f(a)$ whose $(i, j)$ entry is the second partial derivative $D_jD_if(a)$ of a scalar field $f$ at $a$ .

Why it existsIt is the derivative of the gradient: it describes how the slope changes as you move, and its quadratic form is the second-order term of Taylor's formula.

Definition T

Critical point

A point $a$ in the interior of the domain of a differentiable scalar field $f$ is a critical point (stationary point) if $\nabla f(a) = O$ , that is, all partial derivatives of $f$ vanish at $a$ . The tangent plane of the graph there is horizontal.

In one lineA point $a$ where a differentiable scalar field has $\nabla f(a) = O$ ; Apostol calls it a stationary point.

Why it existsIt is the list of candidates for maxima and minima: by 5.2 an interior extremum of a differentiable field must be a critical point.

Definition T

Local maximum

A scalar field $f$ has a local maximum at $a$ if there is a ball $B(a, r)$ in its domain with $f(x) \le f(a)$ for all $x \in B(a, r)$ . It is a strict local maximum if $f(x) < f(a)$ for all $x \ne a$ in the ball.

In one lineA point $a$ such that $f(x) \le f(a)$ for all $x$ in some ball around $a$ (a relative maximum in Apostol).

Why it existsIt is the 'top of a hill' in the neighbourhood sense: nothing nearby is higher, though something far away might be.

Definition T

Local minimum

$f$ has a local minimum at $a$ if $f(x) \ge f(a)$ for all $x$ in some ball $B(a, r)$ , and a strict local minimum if $f(x) > f(a)$ for $x \ne a$ there. A local maximum or minimum is a local extremum.

In one lineA point $a$ such that $f(x) \ge f(a)$ for all $x$ in some ball around $a$ (a relative minimum in Apostol).

Why it existsIt is the 'bottom of a valley' in the neighbourhood sense, and the mirror image of a local maximum: $a$ is a local minimum of $f$ exactly when it is a local maximum of $-f$ .

Definition T

Saddle point

A critical point $a$ of a differentiable scalar field is a saddle point if every ball $B(a, r)$ contains points $x$ with $f(x) > f(a)$ and points $x$ with $f(x) < f(a)$ (Lecture 24, Apostol 9.9).

In one lineA critical point $a$ such that every ball around $a$ contains points where $f$ is larger than $f(a)$ and points where it is smaller.

Why it existsIt names the critical points that are neither a top nor a bottom, such as a mountain pass, which is a minimum along the path through it and a maximum across.

Definition T

Absolute maximum

$f: S \to \mathbb R$ has an absolute maximum at $a \in S$ if $f(x) \le f(a)$ for all $x \in S$ ; the number $f(a)$ is the absolute maximum value. An absolute maximum is a local maximum when $a$ is interior to $S$ . Absolute extrema may sit on the boundary of $S$ , where there is no ball inside $S$ and the gradient need not vanish.

In one lineA point $a$ of the domain $S$ with $f(x) \le f(a)$ for every $x \in S$ (a global maximum).

Why it existsIt is the answer to the question 'where is $f$ largest on this whole set?', which is what an optimisation problem asks.

Definition T

Absolute minimum

$f$ has an absolute minimum at $a \in S$ if $f(x) \ge f(a)$ for all $x \in S$ .

In one lineA point $a$ of the domain $S$ with $f(x) \ge f(a)$ for every $x \in S$ (a global minimum).

Why it existsIt is the mirror image of the absolute maximum, and the target of least squares and of training a model.

Worked example Apostol 9.9, Examples 1 to 6

Find and classify the critical points of

x^2 + y^2

-x^2 - y^2

xy

x^3 - 3xy^2

and

x^2y^2

at the origin, using only the definitions.

Solution.

$x^2 + y^2$ . Gradient $(2x, 2y) = O$ at the origin, and $f(x, y) \ge 0 = f(0, 0)$ for all $(x, y)$ : a strict local (indeed absolute) minimum. A sum of squares.
$-x^2 - y^2$ . Same critical point, $f \le 0$ : a strict absolute maximum. The negative of the first.
$xy$ . Gradient $(y, x) = O$ at the origin. In the first and third quadrants $xy > 0$ , in the second and fourth $xy < 0$ , so every ball has both signs: a saddle. The definition of saddle point, checked directly.
$x^3 - 3xy^2$ . Gradient $(3x^2 - 3y^2, -6xy) = O$ at the origin. Along the $x$ -axis $f = x^3$ takes both signs. A saddle (the ‘monkey saddle’: three hills and three valleys, a place for a monkey’s tail). A cubic changes sign through zero.
$x^2y^2$ . Gradient $(2xy^2, 2x^2y) = O$ at the origin, and $f \ge 0 = f(0, 0)$ everywhere: a local (absolute) minimum, not strict, since $f = 0$ on both axes. A product of squares is never negative.

Answer. Minimum, maximum, saddle, saddle, minimum (not strict). The last two have all second derivatives equal to $0$ at the origin, a warning for the test below.

Taylor’s formula in several variables

Theorem T Apostol Thm 9.4; Spagnolie Lecture 24

Second-order Taylor formula for scalar fields

Let $f$ be of class $C^2$ on the ball $B(a, r)$ . For every $v$ with $\lVert v \rVert < r$ there is $c \in (0, 1)$ with

f(a + v) = f(a) + \nabla f(a)\cdot v + \tfrac12\,v^{\mathsf T}H_f(a + cv)\,v.

Moreover $f(a + v) = f(a) + \nabla f(a)\cdot v + \tfrac12v^{\mathsf T}H_f(a)v + \lVert v \rVert^2E_2(a, v)$ , where $E_2(a, v) \to 0$ as $v \to O$ .

In one lineIf $f$ is $C^2$ on a ball $B(a, r)$ and $\lVert v \rVert < r$ , then $f(a + v) = f(a) + \nabla f(a)\cdot v + \frac12v^{\mathsf T}H_f(a + cv)\,v$ for some $c \in (0, 1)$ , and also $f(a + v) = f(a) + \nabla f(a)\cdot v + \frac12v^{\mathsf T}H_f(a)\,v + \lVert v \rVert^2E_2(a, v)$ with $E_2(a, v) \to 0$ as $v \to O$ .

Why it is trueIt is the one-variable Taylor formula of 1.4 along each line through $a$ , assembled into a single statement; it tells you the quadratic form that governs the shape of $f$ near $a$ .

ProofApostol 9.10

The idea: Restrict f to the line through a with direction v: a function g of one variable. Apply the one-variable Taylor theorem with Lagrange remainder to g, and compute g′ and g″ by the chain rule of 5.4. Then replace the Hessian at the intermediate point by the Hessian at a, paying an error that goes to zero by continuity of the second partials.

Fix

v

and put

g(u) = f(a + uv)

B(a, r)

is a ball, so the segment from

a

a + v

lies in it;

g

is defined on an open interval containing

[0, 1]

g'(u) = \nabla f(a + uv)\cdot v

and

g''(u) = v^{\mathsf T}H_f(a + uv)\,v

Chain rule along the path

r(u) = a + uv

, with

r' = v

and

r'' = O

(5.4:

g'' = r'^{\mathsf T}Hr' + \nabla f\cdot r''

g(1) = g(0) + g'(0) + \tfrac12g''(c)

for some

c \in (0, 1)

Taylor's theorem with the Lagrange remainder (1.4) for

g

with

n = 1

[0, 1]

f(a + v) = f(a) + \nabla f(a)\cdot v + \tfrac12v^{\mathsf T}H_f(a + cv)v

Substitute

g(1) = f(a + v)

g(0) = f(a)

and the formulas of step 2. This is the first claim.

So it remains to show

E_2 \to 0

Define

E_2

\lVert v \rVert^2E_2 = \frac12v^{\mathsf T}\big[H_f(a + cv) - H_f(a)\big]v

for

v \ne O

E_2(a, O) = 0

. Then the second formula holds by construction.The only work left is to show E₂ → 0. The number c depends on v, but the point a + cv is always within ‖v‖ of a, which is all we need.

|E_2(a, v)| \le \tfrac12\sum_{i,j}\big|D_{ij}f(a + cv) - D_{ij}f(a)\big| \to 0

\blacksquare

|v_iv_j| \le \lVert v \rVert^2

(each coordinate is at most the length). Each

D_{ij}f

is continuous at

a

, and

a + cv \to a

v \to O

Proof workshopGive the reason for each line of the second-order Taylor proofApostol 9.10

Each line of the proof is shown in order. Choose the reason that justifies it, then press Check.

Fix $v$ and set $g(u) = f(a + uv)$ , a function of one variable.
$g'(u) = \nabla f(a + uv)\cdot v$ and $g''(u) = v^{\mathsf T} H_f(a + uv)\, v$ .
$g(1) = g(0) + g'(0) + \tfrac12 g''(c)$ for some $c \in (0, 1)$ .
Hence $f(a + v) = f(a) + \nabla f(a)\cdot v + \tfrac12 v^{\mathsf T} H_f(a + cv)\, v$ .
$\lVert v \rVert^2 |E_2| \le \tfrac12 \sum_{i,j} |D_{ij} f(a + cv) - D_{ij} f(a)|\, |v_i v_j|$ , which is at most $\lVert v \rVert^2$ times a sum that tends to $0$ .

Worked example Practice Final 2013, problem 2

Practice Final 2013, problem 2.

f: \mathbb R^2 \to \mathbb R

is smooth with

f(1, 2) = 1

f_x(1, 2) = 2

f_y(1, 2) = -1

. (a) Estimate

f(\frac43, \frac53)

. (b) Given also

f_{xx}(1, 2) = 3

f_{xy}(1, 2) = -1

f_{yy}(1, 2) = -2

, improve the estimate.

Solution.

The step. $a = (1, 2)$ , $a + v = (\frac43, \frac53)$ , so $v = (\frac13, -\frac13)$ . Subtract.
(a) Linear estimate. $f(a + v) \approx f(a) + \nabla f(a)\cdot v = 1 + 2\cdot\frac13 + (-1)\cdot(-\frac13) = 1 + \frac23 + \frac13 = 2$ . The linearization of 5.3.
(b) The quadratic form. $H = \begin{bmatrix}3 & -1\\ -1 & -2\end{bmatrix}$ and $v^{\mathsf T}Hv = 3\cdot\frac19 + 2(-1)\cdot\frac13\cdot(-\frac13) + (-2)\cdot\frac19 = \frac39 + \frac29 - \frac29 = \frac13$ . $v^{\mathsf T}Hv = f_{xx}v_1^2 + 2f_{xy}v_1v_2 + f_{yy}v_2^2$ .
Add half of it. $f(a + v) \approx 2 + \frac12\cdot\frac13 = \frac{13}6$ . The second-order Taylor formula.

Answer. (a) $2$ , (b) $\dfrac{13}{6} \approx 2.1667$ .

Your answerPractice Final 2013, problem 2(b): what is the second-order estimate of

f(\frac43, \frac53)

? (Give a decimal or fraction.)

Worked example MATH 375 Final Review, 9.10

Final Review. Give the quadratic approximation of

F(x, y, z) = z\sin(xy)

about

a = (\pi/2, 1, 1)

Solution.

Value and gradient. $F(a) = \sin\frac\pi2 = 1$ . $\nabla F = (yz\cos xy,\ xz\cos xy,\ \sin xy)$ , which at $a$ is $(0, 0, 1)$ . $\cos\frac\pi2 = 0$ .
Hessian at $a$ . $H = \begin{bmatrix}-y^2z\sin xy & z\cos xy - xyz\sin xy & y\cos xy\\ \cdot & -x^2z\sin xy & x\cos xy\\ \cdot & \cdot & 0\end{bmatrix}$ (symmetric), which at $a$ is $\begin{bmatrix}-1 & -\pi/2 & 0\\ -\pi/2 & -\pi^2/4 & 0\\ 0 & 0 & 0\end{bmatrix}$ . At $a$ : $\sin xy = 1$ , $\cos xy = 0$ , so $z\cos xy - xyz\sin xy = -\pi/2$ .
Assemble. With $p = x - \frac\pi2$ , $q = y - 1$ : $F \approx 1 + (z - 1) + \frac12\big(-p^2 - \pi pq - \frac{\pi^2}4q^2\big) = z - \tfrac12\big(p + \tfrac\pi2q\big)^2$ . The bracket is the perfect square $\big(p + \frac\pi2q\big)^2$ , so the Hessian has rank 1: its eigenvalues are $0$ , $0$ and $-(1 + \pi^2/4)$ .
Check numerically. At $a + (0.1, -0.05, 0.08)$ the true value is $1.07985$ ; the linear approximation gives $1.08$ and the quadratic one $1.07977$ . The quadratic is closer, as the extra term is the next order.

Answer. $F(x, y, z) \approx z - \frac12\big(x - \frac\pi2 + \frac\pi2(y - 1)\big)^2$ .

Your answerUse the second-order Taylor polynomial of

f(x, y) = e^x\cos y

at the origin to estimate

f(0.1, 0.2)

. (Give a decimal.)

Worked example Lecture 24

Lecture 24. Find the quadratic approximation of

F(x) = \ln(1 + \lVert x \rVert^2)

near

x = O

\mathbb R^2

Solution.

Value and gradient at $O$ . $F(O) = 0$ and $\nabla F = \dfrac{2x}{1 + \lVert x \rVert^2}$ , which is $O$ at $O$ . The origin is a critical point.
Hessian. $F_{xx} = \dfrac{2(1 + x^2 + y^2) - 4x^2}{(1 + x^2 + y^2)^2}$ , which is $2$ at $O$ , likewise $F_{yy}(O) = 2$ , and $F_{xy} = \dfrac{-4xy}{(\cdots)^2}$ is $0$ at $O$ . So $H(O) = \begin{bmatrix}2 & 0\\ 0 & 2\end{bmatrix}$ . Quotient rule.
Quadratic approximation. $F(x) \approx 0 + 0 + \frac12\,(2x^2 + 2y^2) = x^2 + y^2 = \lVert x \rVert^2$ . This is $\ln(1 + u) \approx u$ with $u = \lVert x \rVert^2$ from 1.4.

Answer. $F(x) \approx \lVert x \rVert^2$ . The eigenvalues $2, 2$ are positive: the origin is a strict minimum, as $\ln(1 + u) \ge 0$ shows.

Beyond the course: for understanding· Higher orders, and the Taylor series in the lecture

If $f$ is $C^k$ the same proof (one variable along the line, then the chain rule $k$ times) gives $g^{(k)}(u) = \sum_{i_1, \dots, i_k}D_{i_1}\cdots D_{i_k}f(a + uv)\,v_{i_1}\cdots v_{i_k}$ , so

f(a + v) = \sum_{m = 0}^{k}\frac1{m!}\sum_{i_1, \dots, i_m}D_{i_1}\cdots D_{i_m}f(a)\,v_{i_1}\cdots v_{i_m} + o(\lVert v \rVert^k).

By 5.5 the partials may be grouped: the term $\frac1{3!}\sum D_iD_jD_kF\,v_iv_jv_k$ in two variables is $\frac1{3!}\big\{F_{xxx}x^3 + 3F_{xxy}x^2y + 3F_{xyy}xy^2 + F_{yyy}y^3\big\}$ , where the $3$ counts the three orderings $F_{xxy}, F_{xyx}, F_{yxx}$ (Lecture 24). In the ‘Taylor series’ version of the lecture one assumes $F = \sum c_{ijk}(x - a_1)^i(y - a_2)^j(z - a_3)^k$ and differentiates term by term: $c_{ijk} = \dfrac{\partial_x^i\partial_y^j\partial_z^kF(a)}{i!\,j!\,k!}$ , for example $c_{122} = F_{xyyzz}(a)/(1!\,2!\,2!)$ . Whether the series converges to $F$ is a separate question (it fails for $e^{-1/x^2}$ , whose Taylor series at $0$ is identically $0$ ); that is for analysis and complex analysis.

Quick checkTrue or false (a question of the Fall 2025 final): if the level sets of a scalar field

f

are not smooth curves, then

f

is not differentiable everywhere.

Quadratic forms and eigenvalues

The last tool is a bound on a quadratic form by the eigenvalues, a version of the fact in 3.2.

Lemma A Apostol 9.5; the Rayleigh bounds

A quadratic form is squeezed by the extreme eigenvalues

Let $A$ be a real symmetric $n \times n$ matrix with eigenvalues $\lambda_1, \dots, \lambda_n$ . Then for every $v \in \mathbb R^n$ , $\min_i\lambda_i\,\lVert v \rVert^2 \le v^{\mathsf T}Av \le \max_i\lambda_i\,\lVert v \rVert^2$ .

In one lineIf $A$ is a real symmetric matrix with eigenvalues $\lambda_{\min} \le \dots \le \lambda_{\max}$ , then $\lambda_{\min}\lVert v \rVert^2 \le v^{\mathsf T}Av \le \lambda_{\max}\lVert v \rVert^2$ for every $v$ , with equality at the corresponding eigenvectors.

Why it is trueIt converts the sign of the eigenvalues into a quantitative lower bound, $\lambda_{\min}\lVert v \rVert^2$ , that beats any error of smaller order than $\lVert v \rVert^2$ .

ProofApostol 9.5 via 3.1 and 3.2

The idea: Rotate to the eigenvector basis, where the form is a weighted sum of squares of the new coordinates with weights the eigenvalues, and compare each weight with the smallest or largest one.

Put

w = Q^{\mathsf T}v

. Then

v^{\mathsf T}Av = w^{\mathsf T}\Lambda w = \sum_i\lambda_iw_i^2

Spectral theorem (3.1):

A = Q\Lambda Q^{\mathsf T}

with

Q

orthogonal and

\Lambda = \operatorname{diag}(\lambda_i)

\lVert w \rVert = \lVert v \rVert

, so

\sum w_i^2 = \lVert v \rVert^2

An orthogonal matrix preserves length (3.3).

\lambda_{\min}\lVert v \rVert^2 \le \sum_i\lambda_iw_i^2 \le \lambda_{\max}\lVert v \rVert^2

Each

w_i^2 \ge 0

and

\lambda_{\min} \le \lambda_i \le \lambda_{\max}

, so

\lambda_{\min}w_i^2 \le \lambda_iw_i^2 \le \lambda_{\max}w_i^2

. Sum over

i

. Equality when

w

is a multiple of the corresponding unit vector.

\blacksquare

The second-derivative test

Theorem T Apostol Thm 9.6; Spagnolie Lecture 24

The second-derivative test

Let $f$ be of class $C^2$ on a ball around $a$ with $\nabla f(a) = O$ , and let $H = H_f(a)$ , a real symmetric matrix with real eigenvalues $\lambda_1, \dots, \lambda_n$ .

(a) If all $\lambda_i > 0$ , then $a$ is a strict local minimum.

(b) If all $\lambda_i < 0$ , then $a$ is a strict local maximum.

(d) If all $\lambda_i \ge 0$ with some $\lambda_i = 0$ , or all $\lambda_i \le 0$ with some $\lambda_i = 0$ , the test decides nothing: with eigenvalues $\ge 0$ the point can be a minimum or a saddle (it cannot be a maximum unless $H = O$ ), and with eigenvalues $\le 0$ a maximum or a saddle (a minimum only if $H = O$ ). If $H = O$ , all three types occur.

In one lineLet $f$ be $C^2$ near a critical point $a$ and $H = H_f(a)$ with eigenvalues $\lambda_1, \dots, \lambda_n$ . All $\lambda_i > 0$ : strict local minimum. All $\lambda_i < 0$ : strict local maximum. Some $\lambda_i > 0$ and some $\lambda_j < 0$ : saddle point. Otherwise (all $\ge 0$ or all $\le 0$ , with a zero) the test is inconclusive.

Why it is trueIt reduces the classification of a critical point to the signs of $n$ real numbers, which the spectral theorem makes available because $H$ is symmetric; the zero case is where the second-order information runs out.

ProofApostol 9.11; here the proof of (c) uses eigenvectors directly

The idea: At a critical point Taylor's formula says f(a + v) − f(a) is half the quadratic form plus an error of smaller order than ‖v‖². The quadratic form is at least λ_min‖v‖² when the eigenvalues are positive, which beats the error. For a saddle, walk along an eigenvector of each sign. For (d), give examples.

f(a + v) - f(a) = \tfrac12v^{\mathsf T}Hv + \lVert v \rVert^2E_2(a, v)

, with

E_2 \to 0

Second-order Taylor formula with

\nabla f(a) = O

(a)

\tfrac12v^{\mathsf T}Hv \ge \tfrac12\lambda_{\min}\lVert v \rVert^2

Lemma above, with all

\lambda_i > 0

so that

\lambda_{\min} > 0

.This is where positive eigenvalues, as opposed to 'nonnegative', are used: lambda_min must be a positive number to dominate the error.

For such

v

f(a + v) - f(a) \ge \big(\tfrac12\lambda_{\min} - \tfrac14\lambda_{\min}\big)\lVert v \rVert^2 > 0

. This proves (a).

Since

E_2 \to 0

, there is

\rho > 0

with

|E_2(a, v)| < \frac14\lambda_{\min}

for

0 < \lVert v \rVert < \rho

(b) follows.

Apply (a) to

-f

, whose Hessian is

-H

with eigenvalues

-\lambda_i > 0

(c)

f(a + tu_+) - f(a) = t^2\big(\tfrac12\lambda_+ + E_2(a, tu_+)\big)

, which is positive for small

t \ne 0

, since

E_2 \to 0

Let

u_+

be a unit eigenvector for an eigenvalue

\lambda_+ > 0

and

u_-

one for

\lambda_- < 0

; then

u^{\mathsf T}Hu = \lambda

for

u = u_\pm

. Take

v = tu_+

in step 1.

f(a + tu_-) - f(a) = t^2\big(\tfrac12\lambda_- + E_2(a, tu_-)\big) < 0

for small

t \ne 0

. So every ball around

a

has points above and below

f(a)

: a saddle.

Same computation with

\lambda_-

(d)

x^2 + y^4

and

x^2 - y^4

both have Hessian

\operatorname{diag}(2, 0)

at the origin; the first has a strict minimum and the second a saddle. Likewise

x^4 + y^4

(minimum) and

-x^4 - y^4

(maximum) both have the zero Hessian.

A single Hessian cannot decide between types, so no rule based on the Hessian alone can.

\blacksquare

Proof workshopRebuild the proof that positive eigenvalues give a minimumApostol 9.11

Tap the lines in the order a proof would write them. Each line can only use what is already above it.

Corollary A Apostol 9.13 #24 and the proof of Thm 9.6(c)

A local minimum needs a positive semidefinite Hessian

If $a$ is a critical point of a $C^2$ field and $H_f(a)$ has a negative eigenvalue, then $a$ is not a local minimum. If it has a positive eigenvalue, then $a$ is not a local maximum. Consequently, if two diagonal entries of $H_f(a)$ have opposite signs, then $a$ is a saddle point.

In one lineIf a $C^2$ field has a local minimum at a critical point $a$ , then every eigenvalue of $H_f(a)$ is at least $0$ ; if it has a local maximum, every eigenvalue is at most $0$ .

Why it is trueIt gives a necessary condition, the second-order analogue of 'the gradient is zero', and it explains why a negative eigenvalue rules out a minimum.

ProofApostol 9.13 #24

The idea: Walk along an eigenvector of a negative eigenvalue; the formula of the test shows f dips below f(a) there. For the diagonal entries, e_i^T H e_i is the entry H_ii, a value of the quadratic form.

\lambda < 0

, there are points arbitrarily close to

a

with

f < f(a)

, so

a

is not a local minimum.

Step (c) of the proof of the test: for a unit eigenvector

u

with eigenvalue

\lambda < 0

f(a + tu) - f(a) = t^2\big(\tfrac12\lambda + E_2(a, tu)\big) < 0

for small

t \ne 0

Likewise a positive eigenvalue rules out a local maximum.

Mirror image, with a positive eigenvalue.

H_{ii} > 0 > H_{jj}

the eigenvalues have both signs, so by (c) of the test

a

is a saddle.

\blacksquare

e_i^{\mathsf T}He_i = H_{ii}

, and by the Lemma

\lambda_{\min} \le H_{ii} \le \lambda_{\max}

. So

H_{ii} > 0

forces

\lambda_{\max} > 0

and

H_{jj} < 0

forces

\lambda_{\min} < 0

Worked example Practice Final 2013, problem 8 (the 'be sure to show' part)

Practice Final 2013, problem 8. Show that

I(a, b) = \int_0^1(ax + b - x^2)^2\,dx

is minimised at the critical point

(1, -\frac16)

Solution.

Second partials. From $I_a = 2\int_0^1x(ax + b - x^2)\,dx$ and $I_b = 2\int_0^1(ax + b - x^2)\,dx$ (found in 5.2): $I_{aa} = 2\int_0^1x^2\,dx = \frac23$ , $I_{ab} = I_{ba} = 2\int_0^1x\,dx = 1$ , $I_{bb} = 2\int_0^1dx = 2$ . Differentiate under the integral; the integrand is a polynomial in $a, b$ .
The Hessian is the same at every point. $H = \begin{bmatrix}\frac23 & 1\\ 1 & 2\end{bmatrix}$ . $I$ is a quadratic function of $(a, b)$ .
Its eigenvalues. Trace $\frac83$ , determinant $\frac43 - 1 = \frac13$ . Both are positive, so both eigenvalues are positive. The product $\lambda_1\lambda_2 = \det > 0$ means the same sign, and the sum $\lambda_1 + \lambda_2 = \operatorname{tr} > 0$ means positive.
Conclude. By the test, $(1, -\frac16)$ is a strict local minimum. It is the absolute minimum too: $I$ is a quadratic polynomial in $(a, b)$ , so its second-order Taylor formula has no error term, $I(p + v) = I(p) + \frac12v^{\mathsf T}Hv$ at the critical point $p$ , and $v^{\mathsf T}Hv > 0$ for every $v \ne O$ . Alternatively the geometry of 2.9: the minimum is the projection of $x^2$ onto $\operatorname{span}\{1, x\}$ .

Answer. $H$ is positive definite, so $(a, b) = (1, -\frac16)$ is a minimum, with $I(1, -\frac16) = \frac1{180}$ .

Worked example Apostol 9.13 #5

Classify the critical points of

f(x, y) = x^3 - 3xy + y^3

, and of Apostol 9.13 #5,

f(x, y) = 2x^2 - xy - 3y^2 - 3x + 7y

Solution.

First function: critical points. $f_x = 3x^2 - 3y = 0$ and $f_y = 3y^2 - 3x = 0$ give $y = x^2$ and $x = y^2$ , so $x = x^4$ : $x = 0$ or $x = 1$ . The points are $(0, 0)$ and $(1, 1)$ . Substitute $y = x^2$ into the second equation.
Hessian. $H = \begin{bmatrix}6x & -3\\ -3 & 6y\end{bmatrix}$ . At $(0, 0)$ : $\begin{bmatrix}0 & -3\\ -3 & 0\end{bmatrix}$ with eigenvalues $\pm3$ : a saddle. At $(1, 1)$ : $\begin{bmatrix}6 & -3\\ -3 & 6\end{bmatrix}$ with eigenvalues $9$ and $3$ : a strict local minimum, $f(1, 1) = -1$ . Eigenvalues of $\begin{bmatrix}p & q\\ q & p\end{bmatrix}$ are $p \pm q$ .
Second function. $f_x = 4x - y - 3 = 0$ , $f_y = -x - 6y + 7 = 0$ give $y = 4x - 3$ and $-x - 24x + 18 + 7 = 0$ , so $x = 1$ , $y = 1$ . $H = \begin{bmatrix}4 & -1\\ -1 & -6\end{bmatrix}$ has determinant $-25 < 0$ , so its eigenvalues $-1 \pm \sqrt{26}$ have opposite signs: a saddle, with $f(1, 1) = 2$ . A negative determinant means eigenvalues of opposite signs.

Answer. $x^3 - 3xy + y^3$ : saddle at $(0, 0)$ , minimum at $(1, 1)$ . The quadratic: a single saddle at $(1, 1)$ (a quadratic function of this kind has only one critical point when $\det H \ne 0$ ).

Your answerFor

f(x, y) = x^3 - 3xy + y^3

at the critical point

(1, 1)

, what is the larger eigenvalue of the Hessian?

Worked example Apostol 9.13 #25

Apostol 9.13 #25. Show that

f(x, y, z) = x^4 + y^4 + z^4 - 4xyz

has a critical point at

(1, 1, 1)

and classify it by the eigenvalues of the Hessian.

Solution.

Gradient. $\nabla f = (4x^3 - 4yz,\ 4y^3 - 4xz,\ 4z^3 - 4xy)$ , which at $(1, 1, 1)$ is $(0, 0, 0)$ . $4 - 4 = 0$ in each entry.
Hessian. $H = \begin{bmatrix}12x^2 & -4z & -4y\\ -4z & 12y^2 & -4x\\ -4y & -4x & 12z^2\end{bmatrix}$ , which at $(1, 1, 1)$ is $\begin{bmatrix}12 & -4 & -4\\ -4 & 12 & -4\\ -4 & -4 & 12\end{bmatrix} = 16I - 4J$ , where $J$ is the matrix of all ones. Diagonal $12 = 16 - 4$ and off-diagonal $-4$ .
Eigenvalues. $J(1, 1, 1)^{\mathsf T} = 3(1, 1, 1)^{\mathsf T}$ , so $H(1, 1, 1)^{\mathsf T} = (16 - 12)(1, 1, 1)^{\mathsf T}$ : eigenvalue $4$ . For $v$ with $v_1 + v_2 + v_3 = 0$ , $Jv = O$ and $Hv = 16v$ : eigenvalue $16$ with multiplicity $2$ . The eigenvectors of $J$ : the all-ones vector and the plane perpendicular to it.
Conclude. All three eigenvalues $4, 16, 16$ are positive: a strict local minimum, with $f(1, 1, 1) = -1$ .

Answer. Strict local minimum (value $-1$ ). The other critical points $(-1, -1, 1)$ , $(-1, 1, -1)$ , $(1, -1, -1)$ are minima too (by the symmetry $(x, y, z) \mapsto$ two sign changes), and $(0, 0, 0)$ has $H = 0$ : the test says nothing there.

Worked example Lecture 24

Lecture 24. Classify the critical point of

f(x, y) = x^2 - 2xy + 2y

and describe its Hessian.

Solution.

Critical point. $\nabla f = (2x - 2y,\ -2x + 2) = O$ gives $x = 1$ , $y = 1$ , with $f(1, 1) = 1$ . The second equation gives $x = 1$ , then the first gives $y = x$ .
Hessian. $H = \begin{bmatrix}2 & -2\\ -2 & 0\end{bmatrix}$ , with characteristic polynomial $\lambda^2 - 2\lambda - 4$ and eigenvalues $1 \pm \sqrt5$ , one positive ( $3.24$ ) and one negative ( $-1.24$ ). Trace $2$ , determinant $-4$ .

Answer. A saddle point. The surface rises along the eigenvector for $1 + \sqrt5$ and falls along the one for $1 - \sqrt5$ . The next sections show why this saddle is not the answer to the question of where $f$ is largest or smallest on a rectangle.

Quick checkAt a critical point the Hessian has eigenvalues

5

and

0

. What does the second-derivative test conclude?

Quick checkWhich result of this section lets you say that a point is a saddle even if one eigenvalue of the Hessian is

0

The 2 by 2 test

Theorem T Apostol Thm 9.7

The second-derivative test for two variables

Let $f$ be $C^2$ with $\nabla f(a) = O$ , $A = f_{xx}(a)$ , $B = f_{xy}(a)$ , $C = f_{yy}(a)$ , and $D = AC - B^2 = \det H_f(a)$ . (a) $D < 0$ : $a$ is a saddle. (b) $D > 0$ and $A > 0$ : a strict local minimum. (c) $D > 0$ and $A < 0$ : a strict local maximum. (d) $D = 0$ : the test is inconclusive.

In one lineLet $a$ be a critical point of a $C^2$ field $f(x, y)$ and $D = f_{xx}f_{yy} - f_{xy}^2$ at $a$ . If $D < 0$ : saddle. If $D > 0$ and $f_{xx} > 0$ : strict local minimum. If $D > 0$ and $f_{xx} < 0$ : strict local maximum. If $D = 0$ : inconclusive.

Why it is trueIt is the 375 version of the eigenvalue test for $2 \times 2$ Hessians, needing no eigenvalues: only a determinant and one second derivative, since the determinant is the product of the eigenvalues and the trace is their sum.

ProofApostol 9.12

The idea: The determinant is the product of the eigenvalues and the trace is their sum ([2.8](/vol2/2-8-eigenvalues/)). The sign of the product says whether the eigenvalues have the same sign, and the sign of the trace (or of A) says which sign.

The eigenvalues satisfy

\lambda_1\lambda_2 = D

and

\lambda_1 + \lambda_2 = A + C

Trace and determinant from the eigenvalues, 2.8:

\lambda_1\lambda_2 = \det H = D

and

\lambda_1 + \lambda_2 = A + C

(a) If

D < 0

, the eigenvalues have opposite signs: a saddle.

A negative product means one factor is positive and one negative. Part (c) of the eigenvalue test.

(b), (c) If

D > 0

, the eigenvalues have the same sign, equal to the sign of

A

. By (a) and (b) of the eigenvalue test, a minimum if

A > 0

and a maximum if

A < 0

A positive product means the same sign. Also

AC = D + B^2 > 0

, so

A

and

C

have the same sign, and the sign of

A + C

is that sign. It is also the sign of

\lambda_1 + \lambda_2

, hence of both eigenvalues.

(d) If

D = 0

, an eigenvalue is

0

and anything can happen.

D = 0

means an eigenvalue is

0

; the examples at the start of this section (Apostol's Examples 4 and 5:

x^3 - 3xy^2

a saddle,

x^2y^2

a minimum) have

D = 0

at the origin and different types.

\blacksquare

Worked example

For which

k

does

f(x, y) = x^2 + kxy + 4y^2

have a strict minimum at the origin? A saddle?

Solution.

The Hessian. $H = \begin{bmatrix}2 & k\\ k & 8\end{bmatrix}$ at the (only) critical point $O$ . $f_{xx} = 2$ , $f_{xy} = k$ , $f_{yy} = 8$ .
The determinant. $D = 16 - k^2$ . $AC - B^2$ .
Cases. $D > 0$ and $A = 2 > 0$ for $|k| < 4$ : strict minimum. $D < 0$ for $|k| > 4$ : saddle. $|k| = 4$ : $D = 0$ . There $f = (x \pm 2y)^2$ , a minimum that is not strict (it vanishes on the line $x = \mp2y$ ), which the test cannot see.

Answer. Minimum for $|k| < 4$ , saddle for $|k| > 4$ , and a degenerate (non-strict) minimum at $|k| = 4$ . The same answer comes from 3.2: $\begin{bmatrix}1 & k/2\\ k/2 & 4\end{bmatrix}$ is positive definite exactly for $|k| < 4$ .

Your answerThe origin is a strict minimum of

x^2 + kxy + 4y^2

exactly when

|k|

is less than what number?

When the test says nothing

Pick the functions with a zero Hessian eigenvalue. The Hessian is the same for $x^2 + y^4$ and $x^2 - y^4$ and $x^2 + y^3$ , yet the surfaces are a valley, a saddle and a saddle; the quadratic term misses all of that because it is zero in the $y$ direction, so the quartic or cubic term decides. The table below lists the standard cases, all with a critical point at the origin.

$f$	Hessian at $O$	eigenvalues	truth
$x^2 + y^2$	$\operatorname{diag}(2, 2)$	$2, 2$	strict minimum (test works)
$x^2 - y^2$	$\operatorname{diag}(2, -2)$	$2, -2$	saddle (test works)
$x^2 + y^4$	$\operatorname{diag}(2, 0)$	$2, 0$	strict minimum
$x^2 - y^4$	$\operatorname{diag}(2, 0)$	$2, 0$	saddle
$x^2 + y^3$	$\operatorname{diag}(2, 0)$	$2, 0$	saddle
$x^4 + y^4$	$0$	$0, 0$	strict minimum
$-x^4 - y^4$	$0$	$0, 0$	strict maximum
$x^2y^2$	$0$	$0, 0$	minimum, not strict
$x^3 - 3xy^2$	$0$	$0, 0$	saddle (monkey saddle)

Worked example compare Apostol 9.9 Examples 4 and 5

Settle the type of the critical point at the origin for

f = x^2 + y^3

and

g = x^2 - y^4

when the Hessian test fails.

Solution.

Both have Hessian $\operatorname{diag}(2, 0)$ at the origin. The test is inconclusive. One eigenvalue is $0$ .
Restrict to the flat direction. On the $y$ -axis $f(0, y) = y^3$ , which is negative for $y < 0$ and positive for $y > 0$ . On the $x$ -axis $f = x^2 > 0$ . So every ball has points above and below $f(0, 0) = 0$ : a saddle. The definition of saddle point, checked on two lines.
Second function. On the $y$ -axis $g(0, y) = -y^4 < 0$ and on the $x$ -axis $g(x, 0) = x^2 > 0$ : a saddle. Same method.
Contrast. $x^2 + y^4$ has $f \ge 0$ everywhere and $f = 0$ only at $O$ : a strict minimum. No sign change anywhere.

Answer. Saddle for $x^2 + y^3$ and for $x^2 - y^4$ . The method: find the direction of the null eigenvector and look at the next-order term of $f$ along it.

Worked example Apostol 9.13 #16

Apostol 9.13 #16. Show that

f(x, y) = 3x^4 - 4x^2y + y^2

has a minimum at the origin on every line through the origin, but no local minimum in the plane.

Solution.

Factor. $f = (x^2 - y)(3x^2 - y)$ . Roots of $y^2 - 4x^2y + 3x^4 = 0$ in $y$ are $x^2$ and $3x^2$ .
Along $y = mx$ with $m \ne 0$ . $f = m^2x^2 - 4mx^3 + 3x^4 = x^2\big(m^2 - 4mx + 3x^2\big)$ , which is positive for small $x \ne 0$ since $m^2 > 0$ . The leading term $m^2x^2$ dominates for small $x$ .
Along the axes. $f(x, 0) = 3x^4 > 0$ and $f(0, y) = y^2 > 0$ for nonzero values. The two lines that the formula $y = mx$ misses or specialises.
Along the parabola $y = 2x^2$ . $f = (x^2 - 2x^2)(3x^2 - 2x^2) = -x^4 < 0$ for $x \ne 0$ . Between the parabolas $y = x^2$ and $y = 3x^2$ the product is negative.
Conclude. Every ball contains points of the parabola $y = 2x^2$ near $O$ , where $f < 0 = f(O)$ . So $O$ is not a local minimum; as there are also points with $f > 0$ , it is a saddle.

Answer. Minimum along every line, saddle in the plane. The Hessian at $O$ is $\operatorname{diag}(0, 2)$ , so the test is inconclusive, and checking line by line is not a valid substitute: the curves along which $f$ is negative are parabolas, tangent to the $x$ -axis, which lines cannot detect. This is the shape of Peano’s surface of the history box.

Absolute extrema on a closed bounded set

The test above classifies interior critical points. The question “where is $f$ largest on this set?” is different. If the set is closed and bounded (compact, 4.3) and $f$ is continuous, the extreme value theorem (4.3) says a maximum and a minimum exist. By 5.2 they are critical points if they are interior. They may also lie on the boundary, where there is no ball inside the set and nothing forces $\nabla f = O$ . Lecture 24: “check the boundaries separately.”

Why· The method

To find the absolute maximum and minimum of a differentiable $f$ on a compact set $S$ : (1) find the critical points in the interior of $S$ and record $f$ there; (2) split the boundary into smooth pieces, restrict $f$ to each piece (a function of one variable, say by parametrising), find its critical points, and include the endpoints and corners; (3) compare all the values. The largest is the absolute maximum, the smallest the absolute minimum. The classification of step 1 is not needed to decide the answer; it only explains the picture. If the set is not compact, the maximum may not exist, and the same list of candidates tells you only where it could be.

Worked example Lecture 24; Apostol 9.9

Lecture 24. Find the absolute maximum and minimum of

f(x, y) = x^2 - 2xy + 2y

on the rectangle

S = [0, 3] \times [0, 2]

Solution.

Interior critical points. $\nabla f = (2x - 2y, -2x + 2) = O$ at $(1, 1)$ , which lies inside $S$ , with $f(1, 1) = 1$ . It is a saddle, so it is neither the maximum nor the minimum of the rectangle. Eigenvalues $1 \pm \sqrt5$ (previous section).
Bottom edge $y = 0$ . $f(x, 0) = x^2$ on $[0, 3]$ : values from $0$ at $(0, 0)$ to $9$ at $(3, 0)$ . An increasing function: extremes at the ends.
Right edge $x = 3$ . $f(3, y) = 9 - 4y$ on $[0, 2]$ : from $9$ at $(3, 0)$ down to $1$ at $(3, 2)$ . A line: extremes at the ends.
Top edge $y = 2$ . $f(x, 2) = x^2 - 4x + 4 = (x - 2)^2$ on $[0, 3]$ : critical point $x = 2$ with value $0$ at $(2, 2)$ ; ends $f(0, 2) = 4$ , $f(3, 2) = 1$ . The one-variable test of 1.4 on the edge, plus its endpoints (the corners).
Left edge $x = 0$ . $f(0, y) = 2y$ : from $0$ to $4$ . A line.
Compare. Candidate values: $1$ (saddle), $0$ at $(0, 0)$ , $9$ at $(3, 0)$ , $1$ at $(3, 2)$ , $0$ at $(2, 2)$ , $4$ at $(0, 2)$ .

Answer. Absolute maximum $9$ at $(3, 0)$ ; absolute minimum $0$ , attained at two points, $(0, 0)$ and $(2, 2)$ (the notes name only the first; $f(2, 2) = 4 - 8 + 4 = 0$ ). The interior saddle is irrelevant to both.

Your answerWhat is the absolute maximum of

f(x, y) = x^2 - 2xy + 2y

on the rectangle

[0, 3] \times [0, 2]

Worked example Apostol 9.13 #18

Apostol 9.13 #18. Find the absolute extreme values of

f(x, y) = xy(1 - x^2 - y^2)

on the unit square

[0, 1]^2

Solution.

Interior critical points. $f_x = y(1 - 3x^2 - y^2) = 0$ and $f_y = x(1 - x^2 - 3y^2) = 0$ ; with $x, y > 0$ : $3x^2 + y^2 = 1$ and $x^2 + 3y^2 = 1$ , so $x^2 = y^2 = \frac14$ , giving $(\frac12, \frac12)$ with $f = \frac14\cdot\frac12 = \frac18$ . Subtract the two equations to get $x^2 = y^2$ , then substitute.
Classify it (not needed, but instructive). $f_{xx} = -6xy = -\frac32$ , $f_{yy} = -\frac32$ , $f_{xy} = 1 - 3x^2 - 3y^2 = -\frac12$ . $D = \frac94 - \frac14 = 2 > 0$ and $f_{xx} < 0$ : a local maximum. The 2 by 2 test.
Edges $x = 0$ and $y = 0$ . $f = 0$ identically. The factor $xy$ is $0$ .
Edge $x = 1$ . $f(1, y) = y(1 - 1 - y^2) = -y^3$ on $[0, 1]$ : from $0$ down to $-1$ at the corner $(1, 1)$ . Edge $y = 1$ : $f = -x^3$ , the same. Monotone.
Compare. Values: $\frac18$ at $(\frac12, \frac12)$ , $0$ along two edges, $-1$ at $(1, 1)$ .

Answer. Absolute maximum $\frac18$ at $(\frac12, \frac12)$ ; absolute minimum $-1$ at $(1, 1)$ . The corner $(0, 0)$ is a critical point of the surface (Hessian with eigenvalues $\pm1$ : a saddle in the plane), but on the square $f = xy(\ldots) \ge 0$ near it, so it is a boundary minimum of the restriction with value $0$ , not an extremum overall.

Worked example

Find the absolute extreme values of

f(x, y) = x^2 + 2y^2 - x

on the closed unit disc

x^2 + y^2 \le 1

Solution.

Interior. $\nabla f = (2x - 1, 4y) = O$ at $(\frac12, 0)$ , inside the disc, with $f = \frac14 - \frac12 = -\frac14$ . Hessian $\operatorname{diag}(2, 4)$ : a minimum. Both eigenvalues positive.
Boundary: parametrise. $x = \cos t$ , $y = \sin t$ : $f = \cos^2t + 2\sin^2t - \cos t = 2 - \cos^2t - \cos t$ . With $c = \cos t \in [-1, 1]$ : $\varphi(c) = 2 - c^2 - c$ . $\sin^2t = 1 - \cos^2t$ .
One-variable problem. $\varphi'(c) = -2c - 1 = 0$ at $c = -\frac12$ : $\varphi = 2 - \frac14 + \frac12 = \frac94$ (a maximum, as $\varphi'' = -2$ ). At the ends: $\varphi(1) = 0$ , $\varphi(-1) = 2$ . Include the endpoints of the interval for $c$ .
Compare. $-\frac14$ (interior), $\frac94$ at $c = -\frac12$ (the points $(-\frac12, \pm\frac{\sqrt3}2)$ ), $0$ at $(1, 0)$ , $2$ at $(-1, 0)$ .

Answer. Absolute minimum $-\frac14$ at $(\frac12, 0)$ ; absolute maximum $\frac94$ at $(-\frac12, \pm\frac{\sqrt3}2)$ , on the boundary, where $\nabla f = (-2, \pm2\sqrt3) \ne O$ .

Your answerWhat is the absolute maximum of

f(x, y) = x^2 + 2y^2 - x

on the closed unit disc? (Give a decimal.)

Quick check

f

is continuous on a closed disc and has no critical points in the open disc. Where are its maximum and minimum?

Where it is used

Used inside maths· Inside maths: least squares, the centroid, and the next chapters

The least-squares line through points $(x_i, y_i)$ minimises $L(a, b) = \sum_i(ax_i + b - y_i)^2$ (Apostol 9.13 #21). Its gradient is zero exactly at the normal equations, and its Hessian is $2\begin{bmatrix}\sum x_i^2 & \sum x_i\\ \sum x_i & n\end{bmatrix}$ , with determinant $4\big(n\sum x_i^2 - (\sum x_i)^2\big) > 0$ unless all the $x_i$ are equal (Cauchy–Schwarz). So the critical point is a minimum, and since $L$ is a quadratic function, the only one and the absolute minimum. For the points $(0, 1), (1, 3), (2, 4)$ the equations are $5a + 3b = 11$ and $3a + 3b = 8$ , giving $a = \frac32$ , $b = \frac76$ , with Hessian $\begin{bmatrix}10 & 6\\ 6 & 6\end{bmatrix}$ of eigenvalues $1.68$ and $14.32$ . In the same way (Apostol 9.13 #23) the point $x$ minimising $\sum_k\lVert x - z_k\rVert^2$ has gradient $2\sum(x - z_k)$ , so it is the centroid, and the Hessian $2nI$ is positive definite. Constrained problems (Lagrange multipliers) use the same Hessian, restricted to the constraint surface.

Used in the real world· AI: loss landscapes, saddles, and what Newton's method does at a saddle

For the least-squares loss $L(w) = \lVert Xw - y\rVert^2$ the Hessian is $2X^{\mathsf T}X$ , positive semidefinite for any data (3.2), so the loss is a bowl (possibly with flat directions) and every critical point is a global minimum. In the line-fit example above, the eigenvalues $1.68$ and $14.32$ give a condition number $\approx 8.55$ , and gradient descent on $L$ converges only for a step size below $2/14.32 = 0.14$ , slowly along the direction of the small eigenvalue. Real networks are not convex. The simplest non-convex model has two scalar weights multiplied, $L(w_1, w_2) = (w_1w_2 - 1)^2$ , fitting the target $1$ . Its gradient is $2(w_1w_2 - 1)(w_2, w_1)$ , so $(0, 0)$ is a critical point, and its Hessian there is $\begin{bmatrix}0 & -2\\ -2 & 0\end{bmatrix}$ with eigenvalues $\pm2$ : a saddle, with loss $1$ . The minima form the hyperbola $w_1w_2 = 1$ (loss $0$ ), where the Hessian has eigenvalues $0$ (along the valley) and $2(w_1^2 + w_2^2)$ . Starting at $(0.01, 0.02)$ gradient descent with step $0.1$ slides off the saddle along the descending direction and reaches the valley, at about $(1, 1)$ with loss $\approx 0$ , in about 35 steps. Newton’s method, which solves $\nabla L = O$ with the Hessian, jumps from $(0.01, 0.02)$ to $(-4\times10^{-6}, -8\times10^{-6})$ in one step and stays there: it is attracted to any critical point, including the saddle at loss $1$ . That is why optimisers for neural networks use first-order methods with noise, or modify the Hessian to be positive definite, and why checking the sign of the Hessian’s eigenvalues, as in the lab below, distinguishes a minimum from a saddle that only looks flat.

Python labClassify critical points from a finite-difference Hessian, then watch gradient descent and Newton's method at a saddle

import numpy as np

def num_hessian(f, x, h=1e-4):
    n = len(x); E = np.eye(n) * h; H = np.zeros((n, n))
    for i in range(n):
        for j in range(n):
            H[i, j] = (f(x + E[i] + E[j]) - f(x + E[i] - E[j]) - f(x - E[i] + E[j]) + f(x - E[i] - E[j])) / (4 * h * h)
    return H

cases = [("x^2 + y^2 at 0", lambda p: p[0]**2 + p[1]**2, [0, 0]),
         ("x y at 0", lambda p: p[0] * p[1], [0, 0]),
         ("x^3 - 3xy + y^3 at (1,1)", lambda p: p[0]**3 - 3 * p[0] * p[1] + p[1]**3, [1, 1]),
         ("x^2 + y^4 at 0", lambda p: p[0]**2 + p[1]**4, [0, 0]),
         ("x^4 + y^4 at 0", lambda p: p[0]**4 + p[1]**4, [0, 0]),
         ("x^4+y^4+z^4-4xyz at (1,1,1)", lambda p: p[0]**4 + p[1]**4 + p[2]**4 - 4 * p[0] * p[1] * p[2], [1, 1, 1])]
for name, f, x in cases:
    H = num_hessian(f, np.array(x, float)); ev = np.linalg.eigvalsh((H + H.T) / 2)
    kind = "minimum" if ev.min() > 1e-3 else "maximum" if ev.max() < -1e-3 else "saddle" if (ev.max() > 1e-3 and ev.min() < -1e-3) else "test fails"
    print(f"{name:30s} eigenvalues {np.round(ev, 3)} -> {kind}")

L = lambda w: (w[0] * w[1] - 1)**2
g = lambda w: 2 * (w[0] * w[1] - 1) * np.array([w[1], w[0]])
Hs = lambda w: 2 * np.outer([w[1], w[0]], [w[1], w[0]]) + 2 * (w[0] * w[1] - 1) * np.array([[0, 1], [1, 0]])
start = np.array([0.01, 0.02])
w = start.copy()
for _ in range(60): w = w - 0.1 * g(w)
print("gradient descent ends at", w.round(4), "loss", round(L(w), 6))
w = start.copy()
for _ in range(5): w = w - np.linalg.solve(Hs(w), g(w))
print("Newton ends at         ", w.round(6), "loss", round(L(w), 6))

Try this. Run it. The first block classifies six critical points from numerically computed Hessians and agrees with the hand computations above; for x^2 + y^4 and x^4 + y^4 it prints 'test fails' because an eigenvalue is 0 (to within the finite-difference error). In the second block, gradient descent leaves the saddle of (w1 w2 - 1)^2 and ends in the valley w1 w2 = 1 with loss near 0, while Newton lands on the saddle with loss 1. Change the start to (0.0, 0.0): both methods stay at the saddle, because the gradient there is exactly zero. Change the start to (0.01, -0.02) and see where gradient descent goes (to the other branch of the valley, w1 w2 = 1 with both weights negative).

Traps

Watch out· Critical does not mean extremum

$\nabla f(a) = O$ is a necessary condition for an interior extremum and nothing more: $x^2 - y^2$ and $x^3 - 3xy^2$ have critical points that are neither maxima nor minima. The Hessian test or a direct look at the sign of $f(x) - f(a)$ is required.

Watch out· Zero eigenvalue: no conclusion

If the Hessian has an eigenvalue $0$ and none of the opposite sign, the test is silent: the types $x^2 + y^4$ (minimum), $x^2 - y^4$ (saddle), $-x^4 - y^4$ (maximum) all have a zero eigenvalue. Look at the next-order terms along the null direction. The Final Review states this case explicitly as ‘indeterminate’.

Watch out· Minimum on every line is not a minimum

For $3x^4 - 4x^2y + y^2$ the origin is a strict minimum along every straight line, and a saddle in the plane. A proof that restricts to lines through the point proves nothing about a neighbourhood. In one variable there is only one line through a point, which is why the habit of checking ‘both sides’ works there.

Watch out· Do not forget the boundary

On a closed bounded set the absolute extrema may be on the boundary with a nonzero gradient. The interior saddle $(1, 1)$ in Lecture 24’s example was neither the maximum nor the minimum; the answers were at $(3, 0)$ , $(0, 0)$ and $(2, 2)$ . Always list the corners, and do not stop at the first minimum found: Lecture 24’s notes name only $(0, 0)$ although $(2, 2)$ gives the same value.

Watch out· Strict versus non-strict, and local versus absolute

$x^2y^2$ has a minimum at the origin that is not strict (it is $0$ on both axes). A local minimum is not necessarily the absolute minimum: $x^3 - 3x$ has a local minimum at $x = 1$ and no absolute minimum. A positive definite Hessian proves only a strict local minimum, unless $f$ is known to be convex.

Watch out· The 2 by 2 test needs f_xx as well as the determinant

$D > 0$ says the eigenvalues have the same sign; it does not say which one. Use $f_{xx}$ (or $f_{yy}$ , or the trace) to decide minimum or maximum. Also note $D = f_{xx}f_{yy} - f_{xy}^2$ uses the square of the mixed partial; in the notation of 5.5, it is $f_{xy}f_{yx}$ only when they are equal.

Problems

checkApostol 9.13 #6

Locate and classify the critical points of $f(x, y) = x^2 - xy + y^2 - 2x + y$ .

Solution

$f_x = 2x - y - 2$ and $f_y = -x + 2y + 1$ . From the first, $y = 2x - 2$ ; the second gives $-x + 4x - 4 + 1 = 0$ , so $x = 1$ , $y = 0$ . $H = \begin{bmatrix}2 & -1\\ -1 & 2\end{bmatrix}$ has $D = 3 > 0$ and $f_{xx} = 2 > 0$ (eigenvalues $1, 3$ ): a strict local minimum, with $f(1, 0) = -1$ . Since $H$ is positive definite and constant, it is the absolute minimum.

check

At a critical point the Hessian is $\begin{bmatrix}1 & 2\\ 2 & 1\end{bmatrix}$ . What is the type? What if it is $\begin{bmatrix}2 & 2\\ 2 & 2\end{bmatrix}$ ?

Solution

First: $D = 1 - 4 = -3 < 0$ , eigenvalues $3$ and $-1$ : a saddle (positive entries do not make the matrix positive definite, 3.2). Second: $D = 0$ , eigenvalues $4$ and $0$ : the test is inconclusive.

check

Write the second-order Taylor polynomial of $f(x, y) = x^2y$ about $(1, 1)$ .

Solution

$f(1, 1) = 1$ , $\nabla f = (2xy, x^2) = (2, 1)$ . $H = \begin{bmatrix}2y & 2x\\ 2x & 0\end{bmatrix} = \begin{bmatrix}2 & 2\\ 2 & 0\end{bmatrix}$ . With $p = x - 1$ , $q = y - 1$ : $f \approx 1 + 2p + q + \frac12(2p^2 + 4pq) = 1 + 2p + q + p^2 + 2pq$ . (Exact: $f = (1 + p)^2(1 + q) = 1 + 2p + q + p^2 + 2pq + p^2q$ , so the error is exactly the cubic term $p^2q$ .)

checkApostol 9.13 #24 (a check)

At a critical point the Hessian is $\begin{bmatrix}3 & 1 & 0\\ 1 & -2 & 1\\ 0 & 1 & 5\end{bmatrix}$ . Is it a minimum, a maximum or a saddle?

Solution

The diagonal entries $3$ and $-2$ have opposite signs, so by the corollary the point is a saddle (a value of the quadratic form at $e_1$ is $3 > 0$ and at $e_2$ is $-2 < 0$ , so there are eigenvalues of both signs).

useApostol 9.13 #8 style

Find and classify the critical points of $f(x, y) = x^3 + y^3 - 3x - 12y + 20$ .

Solution

$f_x = 3x^2 - 3 = 0$ gives $x = \pm1$ ; $f_y = 3y^2 - 12 = 0$ gives $y = \pm2$ . Four points. $H = \begin{bmatrix}6x & 0\\ 0 & 6y\end{bmatrix}$ . At $(1, 2)$ : $\operatorname{diag}(6, 12)$ , minimum, $f = 1 + 8 - 3 - 24 + 20 = 2$ . At $(-1, -2)$ : $\operatorname{diag}(-6, -12)$ , maximum, $f = -1 - 8 + 3 + 24 + 20 = 38$ . At $(1, -2)$ and $(-1, 2)$ : eigenvalues of opposite signs, saddles.

useApostol 9.13 #21 (least squares)

Find the line $y = ax + b$ minimising $\sum_i(ax_i + b - y_i)^2$ for the points $(0, 1), (1, 3), (2, 4)$ and verify by the Hessian that it is a minimum.

Solution

$\partial_a = 2\sum x_i(ax_i + b - y_i) = 0$ and $\partial_b = 2\sum(ax_i + b - y_i) = 0$ give $5a + 3b = 11$ and $3a + 3b = 8$ . Subtracting, $2a = 3$ , $a = \frac32$ , $b = \frac76$ . $H = 2\begin{bmatrix}5 & 3\\ 3 & 3\end{bmatrix} = \begin{bmatrix}10 & 6\\ 6 & 6\end{bmatrix}$ , $D = 60 - 36 = 24 > 0$ , $L_{aa} = 10 > 0$ : a minimum, and since $L$ is quadratic, the absolute minimum. The residuals are $\frac76 - 1 = \frac16$ , $\frac32 + \frac76 - 3 = -\frac13$ , $3 + \frac76 - 4 = \frac16$ , with sum of squares $\frac1{36} + \frac19 + \frac1{36} = \frac16$ .

useLecture 24 style

Find the absolute maximum and minimum of $f(x, y) = x^2 + y^2 - 2x$ on the triangle with vertices $(0, 0)$ , $(2, 0)$ , $(0, 2)$ .

Solution

Interior: $\nabla f = (2x - 2, 2y) = O$ at $(1, 0)$ , which lies on the bottom edge, not in the interior. Edges. Bottom ( $y = 0$ , $0 \le x \le 2$ ): $f = x^2 - 2x$ , critical at $x = 1$ with $f = -1$ , ends $f(0, 0) = 0$ and $f(2, 0) = 0$ . Left ( $x = 0$ , $0 \le y \le 2$ ): $f = y^2$ , from $0$ to $4$ . Hypotenuse ( $y = 2 - x$ , $0 \le x \le 2$ ): $f = x^2 + (2 - x)^2 - 2x = 2x^2 - 6x + 4$ , critical at $x = \frac32$ with $f = \frac92 - 9 + 4 = -\frac12$ ; ends $f(0, 2) = 4$ and $f(2, 0) = 0$ . Candidates: $-1$ at $(1, 0)$ , $0$ , $4$ at $(0, 2)$ , $-\frac12$ . Absolute minimum $-1$ at $(1, 0)$ , absolute maximum $4$ at $(0, 2)$ .

use

Show that the Hessian test fails at the origin for $f(x, y) = x^2 - 2xy + y^2 + y^3$ , and decide the type of the critical point.

Solution

$f = (x - y)^2 + y^3$ . $\nabla f = (2x - 2y,\ -2x + 2y + 3y^2) = O$ at $(0, 0)$ . $H = \begin{bmatrix}2 & -2\\ -2 & 2\end{bmatrix}$ with eigenvalues $4$ and $0$ : inconclusive. On the line $x = y$ : $f = y^3$ , taking both signs. So every ball contains points with $f > 0$ and $f < 0$ : a saddle.

proveApostol 9.13 #23

Let $z_1, \dots, z_m$ be points of $\mathbb R^n$ and $f(x) = \sum_{k = 1}^m\lVert x - z_k\rVert^2$ . Prove that $f$ has an absolute minimum exactly at the centroid $\bar z = \frac1m\sum_kz_k$ .

Solution

Plan. Gradient and Hessian, then the fact that a positive definite quadratic has a unique critical point which is its absolute minimum.

$\nabla\lVert x - z_k\rVert^2 = 2(x - z_k)$ (5.2 or 3.8), so $\nabla f = 2\sum_k(x - z_k) = 2m(x - \bar z)$ , which is $O$ only at $x = \bar z$ . $H_f = 2mI$ , positive definite. By the second-derivative test $\bar z$ is a strict local minimum. For the absolute minimum: $f(x) = f(\bar z) + m\lVert x - \bar z\rVert^2$ (expand $x - z_k = (x - \bar z) + (\bar z - z_k)$ ; the cross term is $2(x - \bar z)\cdot\sum_k(\bar z - z_k) = 0$ ). So $f(x) \ge f(\bar z)$ with equality only at $\bar z$ . $\blacksquare$

proveApostol 9.13 #24

Let $a$ be a critical point of a $C^2$ field with Hessian $H$ . Prove that if two diagonal entries of $H$ have opposite signs, $a$ is a saddle point.

Solution

Plan. A diagonal entry is a value of the quadratic form, hence squeezed between the extreme eigenvalues; opposite signs force eigenvalues of both signs.

$H_{ii} = e_i^{\mathsf T}He_i$ with $\lVert e_i \rVert = 1$ . By the Lemma (the Rayleigh bounds), $\lambda_{\min} \le H_{ii} \le \lambda_{\max}$ for every $i$ . If $H_{ii} > 0 > H_{jj}$ then $\lambda_{\max} \ge H_{ii} > 0$ and $\lambda_{\min} \le H_{jj} < 0$ . Part (c) of the second-derivative test gives a saddle. $\blacksquare$ (Alternative without eigenvalues: $f(a + te_i) - f(a) = \frac12H_{ii}t^2 + t^2E_2 > 0$ and $f(a + te_j) - f(a) = \frac12H_{jj}t^2 + t^2E_2 < 0$ for small $t$ .)

prove

Prove that if $f$ is $C^2$ and its Hessian is positive definite at every point of an open convex set $S$ , then $f$ has at most one critical point in $S$ , and if it has one, it is the absolute minimum of $f$ on $S$ .

Solution

Plan. Restrict to the segment between two points; the restriction is a function of one variable with positive second derivative.

Let $p, q \in S$ be distinct and $g(t) = f(p + t(q - p))$ for $t \in [0, 1]$ (the segment lies in $S$ by convexity). By 5.4, $g'' = (q - p)^{\mathsf T}H(\cdot)(q - p) > 0$ , so $g'$ is strictly increasing. If $p$ is a critical point, $g'(0) = \nabla f(p)\cdot(q - p) = 0$ , so $g'(t) > 0$ for $t > 0$ and $g(1) > g(0)$ : $f(q) > f(p)$ . So a critical point $p$ is an absolute minimum, strictly (every other point has a larger value). If $q$ were also critical, $f(p) > f(q)$ would follow symmetrically, a contradiction. $\blacksquare$

provenew (uses 5.4)

Prove that the type of a nondegenerate critical point does not depend on the coordinates: if $\Phi$ is a $C^2$ change of variables with invertible Jacobian, $h = f \circ \Phi$ , and $f$ has a critical point at $b = \Phi(a)$ , then $a$ is a critical point of $h$ and $H_h(a) = D\Phi(a)^{\mathsf T}\,H_f(b)\,D\Phi(a)$ ; this matrix has as many positive and negative eigenvalues as $H_f(b)$ (Sylvester’s law of inertia).

Solution

Plan. Use the second-derivative chain rule of 5.4 and note that the term involving $\nabla f$ vanishes.

By 5.4, $\nabla h(a) = D\Phi(a)^{\mathsf T}\nabla f(b) = O$ , and $H_h(a) = D\Phi^{\mathsf T}H_fD\Phi + \sum_kD_kf(b)\,H_{\Phi_k}(a)$ ; the sum is $0$ since $\nabla f(b) = O$ . So $H_h(a) = P^{\mathsf T}H_f(b)P$ with $P = D\Phi(a)$ invertible. The quadratic forms satisfy $w^{\mathsf T}H_hw = (Pw)^{\mathsf T}H_f(Pw)$ : they take the same values on corresponding vectors. So the largest subspace on which $H_h$ is positive definite has the same dimension as the one for $H_f$ , and likewise for negative; hence the numbers of positive and negative eigenvalues agree (Sylvester’s law of inertia, which we use without proof). In particular a minimum, maximum or saddle in the $b$ coordinates is one in the $a$ coordinates. $\blacksquare$ (The eigenvalues themselves do change: only their signs are invariant.)

stretchApostol 9.13 #20

Let $f(x, y) = Ax^2 + 2Bxy + Cy^2 + 2Dx + 2Ey + F$ with $A > 0$ and $B^2 < AC$ . Prove that $f$ has an absolute minimum, at the solution $(x_1, y_1)$ of the linear system $Ax + By = -D$ , $Bx + Cy = -E$ .

Solution

Plan. A quadratic function with positive definite quadratic part: critical point from the gradient equations, then Taylor’s formula is exact.

$\nabla f = 2(Ax + By + D,\ Bx + Cy + E)$ , which vanishes exactly at the solution of the system; the coefficient matrix $M = \begin{bmatrix}A & B\\ B & C\end{bmatrix}$ has $\det = AC - B^2 > 0$ , so there is exactly one solution. $H = 2M$ , with $\det > 0$ and $A > 0$ : positive definite, so it is positive definite everywhere and, since $f$ equals its second-order Taylor polynomial, $f(p + v) = f(p) + \nabla f(p)\cdot v + v^{\mathsf T}Mv$ exactly. At $p = (x_1, y_1)$ the middle term is $0$ and $v^{\mathsf T}Mv > 0$ for $v \ne O$ , so $f(x_1 + v) > f(x_1, y_1)$ for every $v \ne O$ : an absolute minimum. $\blacksquare$

stretch

Let $f(x, y) = (y - x^2)(y - 2x^2)$ . Show that the origin is a critical point at which $f$ has a strict local minimum along every line through it, give the Hessian, and show that the origin is nevertheless a saddle. Check that $-f$ is the function $(2x^2 - y)(y - x^2)$ of the history box, so that function has a strict maximum along every line and is also a saddle.

Solution

Plan. Expand along lines, then find a curve on which the sign is the opposite.

$f = y^2 - 3x^2y + 2x^4$ , $\nabla f = (-6xy + 8x^3,\ 2y - 3x^2) = O$ at the origin, and $H = \begin{bmatrix}0 & 0\\ 0 & 2\end{bmatrix}$ there (eigenvalues $0$ and $2$ ): the test is inconclusive. Along $y = mx$ with $m \ne 0$ : $f = x^2(m^2 - 3mx + 2x^2) > 0$ for small $x \ne 0$ . Along the $x$ -axis $f = 2x^4 > 0$ and along the $y$ -axis $f = y^2 > 0$ . So $f$ has a strict local minimum along every line through $O$ . But on the parabola $y = \frac32x^2$ : $f = (\frac12x^2)(-\frac12x^2) = -\frac14x^4 < 0$ , and $f > 0$ on every line, so every ball around $O$ contains points with $f < 0 = f(O)$ and points with $f > 0$ : a saddle. Finally $-f = -(y - x^2)(y - 2x^2) = (2x^2 - y)(y - x^2)$ , and negating all the statements turns the strict minima along lines into strict maxima. $\blacksquare$

Summary

Taylor, order two. For $f$ of class $C^2$ , $f(a + v) = f(a) + \nabla f(a)\cdot v + \frac12v^{\mathsf T}H_f(a)v + \lVert v \rVert^2E_2$ with $E_2 \to 0$ ; the Hessian $H_f = [D_{ij}f]$ is symmetric (5.5). Proof: restrict to a line, apply the one-variable theorem, and use the chain rule of 5.4 twice.
A critical point has $\nabla f = O$ ; every interior local maximum or minimum is one (5.2), but a critical point can also be a saddle point (points above and below in every ball).
Second-derivative test. At a critical point of a $C^2$ field, with Hessian eigenvalues $\lambda_i$ (real, by the spectral theorem): all positive, strict local minimum; all negative, strict local maximum; both signs, saddle; a zero with no opposite sign, inconclusive. Proof: $\frac12v^{\mathsf T}Hv \ge \frac12\lambda_{\min}\lVert v \rVert^2$ beats the error, and eigenvectors detect saddles.
Two variables. With $D = f_{xx}f_{yy} - f_{xy}^2$ (product of eigenvalues): $D < 0$ saddle; $D > 0$ and $f_{xx} > 0$ minimum; $D > 0$ and $f_{xx} < 0$ maximum; $D = 0$ inconclusive.
Degenerate cases need the next-order terms along the null direction ( $x^2 + y^4$ minimum, $x^2 - y^4$ saddle). A minimum on every line is not a minimum ( $3x^4 - 4x^2y + y^2$ ). A local minimum needs a positive semidefinite Hessian.
Absolute extrema on a compact set exist (4.3); compare the interior critical points with the extrema on each boundary piece, including corners. Lecture 24’s rectangle: maximum $9$ at $(3, 0)$ , minimum $0$ at both $(0, 0)$ and $(2, 2)$ , the interior saddle irrelevant.
Applications. Least squares has Hessian $2X^{\mathsf T}X \succeq 0$ , so critical points are global minima; for $(w_1w_2 - 1)^2$ the origin is a saddle that gradient descent leaves and Newton’s method lands on.

Watch and read

The same ideas explained by someone else. Use these when an explanation here does not land.

5.6 Taylor's formula, the Hessian and extrema

The picture: bowl, dome, saddle

The idea, in words

Definitions

Hessian matrix

Critical point

Local maximum

Local minimum

Saddle point

Absolute maximum

Absolute minimum

Taylor’s formula in several variables

Second-order Taylor formula for scalar fields

Quadratic forms and eigenvalues

A quadratic form is squeezed by the extreme eigenvalues

The second-derivative test

The second-derivative test

A local minimum needs a positive semidefinite Hessian

The 2 by 2 test

The second-derivative test for two variables

When the test says nothing

Absolute extrema on a closed bounded set

Where it is used

Traps

Problems

Summary

Watch and read

Where this sits

Comes from

Leads to

Same idea elsewhere