5.2 Directional derivatives, partial derivatives and the gradient

To differentiate a function of many variables, pick a direction and ask the one-variable question along that line. The answers for the coordinate directions are the partial derivatives; collected into a vector they form the gradient, which points uphill and is perpendicular to the level sets. But having a derivative in every direction does not even make a function continuous.

In one variable the derivative answers: how fast does $f$ change when $x$ moves? In the plane or in space there are many ways to move, and the answer depends on which way. This chapter fixes one direction and asks the one-variable question along that line; then it collects the answers for all directions into a single vector, the gradient, which points the way uphill. The chapter ends with a warning that explains why the next one is needed: a function can have a derivative in every direction at a point and still fail to be continuous there.

Before you start

1.4: the derivative of a function of one variable. 5.1: scalar fields, level sets, and why a point must be interior to the domain (a ball fits around it). 2.9: the dot product and the Cauchy–Schwarz inequality.

Quick checkLet

g(t) = (3 + 2t)^2

. What is

g'(0)

Quick checkFor a vector

g

and a unit vector

u

, how large can

g \cdot u

be?

Quick checkWhat is the angle between two vectors with dot product

0

The picture: the slope of a slice

Left is the plane with the point $a$ (red) and a line through it in a chosen direction (purple). Right is the slice: the graph of $f$ above that line, a curve in one variable $t$ . Its slope at $t = 0$ is the number this chapter is about. Things to do:

Take $3x^2 + y^2$ and point the direction along the $x$ -axis (angle 0). The slope is $6x$ at your point: the ordinary derivative of $3x^2$ with $y$ held fixed. Now point along the $y$ -axis: the slope is $2y$ .
Turn the direction slowly. The slope changes smoothly, positive in some directions, negative in the opposite ones, and there are two directions in which it is $0$ (the direction of the level curve through the point).
Switch to the last function, with the point at the origin. Its slice has a slope in every direction, but near the vertical direction the slope jumps wildly. This is the warning of the last section.

The idea, in words

Fix a point $a$ and a vector $y$ . The line through $a$ with direction $y$ is $a + hy$ , where $h$ is a real number: at $h = 0$ we are at $a$ , at $h = 1$ at $a + y$ . Walk along it and record the value of $f$ : this is the one-variable function $g(h) = f(a + hy)$ , the slice. The directional derivative is the number $g'(0)$ , written as the limit of the difference quotient. Everything else in this chapter is a consequence of that sentence: we reuse the one-variable derivative of 1.4 along lines.

Why· Why the length of y matters

The distance from $a$ to $a + hy$ is $|h|\,\lVert y \rVert$ . If you want the rate of change per unit distance, you want $\lVert y \rVert = 1$ . If you double $y$ , you walk twice as fast along the same line, so the rate of change per unit of $h$ doubles. That is why Apostol allows any vector $y$ in the definition (“the derivative with respect to $y$ ”) and reserves the word “directional” for unit vectors. When $n = 1$ and $y = 1$ the definition is the derivative of 1.4.

Definitions

Definition T

Directional derivative

Let $f: S \to \mathbb R$ be a scalar field, let $a$ be an interior point of $S$ (some ball $B(a, r)$ lies in $S$ , so the points $a + hy$ with $|h|\,\lVert y \rVert < r$ are in $S$ ) and let $y \in \mathbb R^n$ . The derivative of $f$ at $a$ with respect to $y$ is

f'(a; y) = \lim_{h \to 0}\frac{f(a + hy) - f(a)}{h},

when the limit exists. If $\lVert y \rVert = 1$ it is called the directional derivative of $f$ at $a$ in the direction of $y$ . The fraction is the average rate of change over the segment from $a$ to $a + hy$ . (Apostol 8.6 and 8.7.)

The limit is two-sided: $h \to 0$ from both signs, so $h$ may be negative and then we walk backwards from $a$ . Some books use only $h \to 0^+$ (a one-sided derivative). Apostol, Spagnolie and this book use the two-sided one, and the difference matters: for $f(x) = \lVert x \rVert$ at $a = O$ we get $\frac{|h|\,\lVert y \rVert}{h} = \pm\lVert y \rVert$ , which has a one-sided limit but no two-sided limit.

In one lineFor a scalar field $f$ , a point $a$ and a vector $y$ : $f'(a; y) = \lim_{h \to 0} \frac{f(a + hy) - f(a)}{h}$ , the rate of change of $f$ as you walk from $a$ in the direction $y$ ; it is called the directional derivative when $\lVert y \rVert = 1$ .

Why it existsIt reduces 'how does f change here?' to the one-variable question, one direction at a time, and for a unit vector it is the change per unit distance.

Lemma T new (the idea of Apostol 8.9 #22)

Scaling a direction scales the derivative

If $f'(a; y)$ exists, then $f'(a; cy)$ exists and equals $c\,f'(a; y)$ for every real number $c$ .

In one lineIf $f'(a; y)$ exists then $f'(a; cy) = c\,f'(a; y)$ for every real $c$ .

Why it is trueWalking $c$ times as fast along the same line gives $c$ times the rate; in particular walking backwards ( $c = -1$ ) reverses the sign.

Proofnew

The idea: Change the variable in the difference quotient from h to ch.

c = 0

both sides are

0

For

c = 0

f(a + h \cdot 0) - f(a) = 0

for every

h

c \ne 0

\dfrac{f(a + h\,cy) - f(a)}{h} = c\cdot\dfrac{f(a + (hc)\,y) - f(a)}{hc}

Multiply and divide by

c

. This is allowed since

c \ne 0

h \to 0

the right side tends to

c\,f'(a; y)

\blacksquare

k = hc \to 0

exactly when

h \to 0

, and the quotient in

k

tends to

f'(a; y)

Theorem T Apostol Thm 8.3

The derivative of a slice

Let $g(t) = f(a + ty)$ . If one of $g'(t)$ and $f'(a + ty; y)$ exists, so does the other, and $g'(t) = f'(a + ty; y)$ . In particular $g'(0) = f'(a; y)$ .

In one lineFor $g(t) = f(a + ty)$ , the derivative $g'(t)$ exists exactly when $f'(a + ty; y)$ exists, and then $g'(t) = f'(a + ty; y)$ .

Why it is trueIt is the bridge to Unit 1: every directional derivative is an ordinary derivative of a one-variable function, so all of 1.4 (Fermat, the mean value theorem) applies to it.

ProofApostol 8.3

The idea: Write the two difference quotients and see that they are the same expression.

\dfrac{g(t + h) - g(t)}{h} = \dfrac{f(a + ty + hy) - f(a + ty)}{h}

Definition of

g

So the left side has a limit as

h \to 0

exactly when

f'(a + ty; y)

exists, and then the two are equal.

\blacksquare

The right side is the difference quotient defining

f'(b; y)

at the point

b = a + ty

Definition T

Partial derivative

The partial derivative of $f$ with respect to the $k$ -th variable at $a$ is the derivative with respect to the $k$ -th unit vector $e_k$ :

D_k f(a) = f'(a; e_k) = \lim_{h \to 0}\frac{f(a_1, \dots, a_k + h, \dots, a_n) - f(a_1, \dots, a_n)}{h}.

Other notations: $\dfrac{\partial f}{\partial x_k}(a)$ and $f_{x_k}(a)$ ; in the plane $\frac{\partial f}{\partial x}$ and $\frac{\partial f}{\partial y}$ , or $f_x$ and $f_y$ . In practice: hold all the other variables constant and differentiate with the rules you know. That is the JEE method, and this definition is why it is correct.

In one lineThe partial derivative $D_k f(a) = f'(a; e_k)$ is the derivative of $f$ along the $k$ -th coordinate axis: differentiate in $x_k$ with every other variable held fixed.

Why it existsIt is the cheapest directional derivative to compute, by the ordinary rules of 1.4, and the gradient is built from them.

Definition T

Gradient

If all the partial derivatives of $f$ exist at $a$ , the gradient of $f$ at $a$ is the vector

\nabla f(a) = \big(D_1 f(a), D_2 f(a), \dots, D_n f(a)\big).

Also written $\operatorname{grad} f(a)$ ; the symbol $\nabla$ is read “del” or “nabla”. Where it exists at every point of a set, $\nabla f$ is a vector field. In the plane, $\nabla f = \big(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\big)$ . The gradient lives in the domain space $\mathbb R^n$ (it has one entry per input variable), not in the space of outputs.

In one lineThe gradient $\nabla f(a) = (D_1 f(a), \dots, D_n f(a))$ is the vector of all the partial derivatives of $f$ at $a$ .

Why it existsIt packs the rates of change along all the axes into one vector, and the next sections show that this one vector knows the rate of change in every direction and the direction of steepest climb.

Where this came from· The steepness of temperature, and the harp

Joseph Fourier’s treatise on heat (1822) rests on a statement about the quantity we now write $\nabla T$ : heat flows from hot to cold, and the rate at which it crosses a surface is proportional to the steepness of the temperature, in the direction of steepest drop. He had no vector notation; the gradient was born as a physical idea first. The symbol $\nabla$ has its own history: William Rowan Hamilton introduced the operator in 1837 (he drew it as a triangle lying on its side), Peter Guthrie Tait used the upright upside-down triangle in his 1867 book on quaternions, and in 1870 William Robertson Smith suggested to Tait the name “nabla”, after a Greek word for a harp, which the symbol resembles. Spagnolie mentioned the harp in Lecture 21. The question both men were answering is the one of this chapter: how to speak, with one object, about the way a quantity varies in all directions at once.

Quick checkFor a scalar field

f: \mathbb R^3 \to \mathbb R

, in which space does the vector

\nabla f(a)

live?

Computing partial derivatives

Worked example Lecture 20

For

F(x, y, z) = xy\sin(z^2)

, find the three partial derivatives at

a = (1, 2, 3)

and the second partials

F_{xx}

F_{xy}

F_{zz}

Solution.

$x$ -partial. Hold $y, z$ fixed: $F_x = y\sin(z^2)$ . At $a$ : $2\sin 9$ . $xy\sin(z^2) = x \cdot (\text{constant})$ as a function of $x$ .
$y$ -partial. $F_y = x\sin(z^2)$ . At $a$ : $\sin 9$ . Same, now $y$ varies.
$z$ -partial. $F_z = xy\cos(z^2)\cdot 2z = 2xyz\cos(z^2)$ . At $a$ : $12\cos 9$ . Chain rule of 1.4 for $\sin(z^2)$ , the inner function $z^2$ having derivative $2z$ .
Second partials. $F_{xx} = 0$ , since $F_x = y\sin(z^2)$ does not contain $x$ . $F_{xy} = \sin(z^2)$ , the $y$ -derivative of $F_x$ . $F_{zz} = 2xy\big(\cos(z^2) - 2z^2\sin(z^2)\big)$ by the product rule on $2xyz\cos(z^2)$ . Each partial is again a scalar field, so we can differentiate it again.

Answer. $\nabla F(a) = (2\sin 9,\ \sin 9,\ 12\cos 9) \approx (0.824,\ 0.412,\ -10.93)$ , and the second partials are as above. (The values are numbers: the derivative at a fixed point has no $x$ , $y$ or $z$ left in it, as Spagnolie reminded the class.)

Your answerFor

f(x, y) = x^2 + y^2\sin(xy)

, find

\dfrac{\partial f}{\partial y}

(1, \pi/2)

Worked example MATH 375 HW 11 (Apostol 8.9 #1 and #8)

Compute

f'(x; y)

and all partial derivatives of

f(x) = a \cdot x

\mathbb R^n

, for a fixed vector

a

Solution.

Build the slice. $g(t) = f(x + ty) = a\cdot(x + ty) = a\cdot x + t\,(a\cdot y)$ . The dot product is linear in its second slot.
Differentiate it. $g'(0) = a\cdot y$ , so $f'(x; y) = a\cdot y$ for every $x$ and $y$ . The slice is a straight line in $t$ with slope $a\cdot y$ .
Partials. Take $y = e_i$ : $D_i f(x) = a\cdot e_i = a_i$ . A partial derivative is a derivative with respect to a unit coordinate vector.

Answer. $f'(x; y) = a\cdot y$ (independent of $x$ ), $\dfrac{\partial f}{\partial x_i} = a_i$ , so $\nabla f = a$ . For a linear field the gradient is a constant vector; compare 3.8, where $\nabla(a^{\mathsf T}x) = a$ .

Worked example MATH 375 HW 11 (Apostol 8.9 #2a, #2b)

Find

f'(a; y)

for

f(x) = \lVert x \rVert^4

\mathbb R^n

, and for

n = 2

find all

y = (p, q)

with

f'((2, 3); (p, q)) = 6

Solution.

Build the slice. $\lVert a + ty \rVert^2 = \lVert a \rVert^2 + 2t\,a\cdot y + t^2\lVert y \rVert^2$ , so $g(t) = f(a + ty) = \big(\lVert a \rVert^2 + 2t\,a\cdot y + t^2\lVert y \rVert^2\big)^2$ . Expand $(a + ty)\cdot(a + ty)$ using bilinearity.
Differentiate at $t = 0$ . Chain rule: $g'(t) = 2\big(\ldots\big)\cdot\big(2a\cdot y + 2t\lVert y \rVert^2\big)$ , so $g'(0) = 2\lVert a \rVert^2\cdot 2\,a\cdot y = 4\lVert a \rVert^2\,(a\cdot y)$ . The bracket at $t = 0$ is $\lVert a \rVert^2$ .
Read the gradient. $f'(a; y) = 4\lVert a \rVert^2\,(a\cdot y)$ means the partials are $4\lVert a \rVert^2 a_i$ , so $\nabla f(a) = 4\lVert a \rVert^2 a$ , a vector (as it should be). Take $y = e_i$ .
The case $n = 2$ . For $a = (2, 3)$ : $\lVert a \rVert^2 = 13$ , and $f'(a; (p, q)) = 4\cdot 13\,(2p + 3q) = 52(2p + 3q)$ . Set it equal to $6$ : $2p + 3q = \dfrac{6}{52} = \dfrac3{26}$ . One linear equation in $(p, q)$ .

Answer. $f'(a; y) = 4\lVert a \rVert^2(a\cdot y)$ ; the required vectors lie on the line $2p + 3q = \frac3{26}$ .

Worked example MATH 375 HW 11 (Apostol 8.9 #9)

Find all partial derivatives of the quadratic form

f(x) = \sum_{i,j} a_{ij}x_ix_j

when

a_{ij} = a_{ji}

Solution.

Differentiate a typical term. $\dfrac{\partial}{\partial x_k}(x_ix_j) = \delta_{ik}x_j + x_i\delta_{jk}$ , where $\delta_{ik}$ is $1$ if $i = k$ and $0$ otherwise. Product rule; $\frac{\partial x_i}{\partial x_k} = 1$ if $i = k$ and $0$ otherwise, since the variables are independent.
Sum over all $i, j$ . $\dfrac{\partial f}{\partial x_k} = \sum_j a_{kj}x_j + \sum_i a_{ik}x_i$ . Only the terms with $i = k$ (first sum) or $j = k$ (second sum) survive.
Use symmetry. $a_{ik} = a_{ki}$ , so both sums equal $\sum_j a_{kj}x_j = (Ax)_k$ . Rename the summation index in the second sum.

Answer. $\dfrac{\partial f}{\partial x_k} = 2(Ax)_k$ , that is $\nabla(x^{\mathsf T}Ax) = 2Ax$ for symmetric $A$ . This is the $2Ax$ of 3.8; the general rule $(A + A^{\mathsf T})x$ for non-symmetric $A$ is the same computation without step 3.

Worked example MATH 375 HW 11 (Apostol 8.9 #5); Lecture 20

Find all partial derivatives of

f(x, y) = \sqrt{x^2 + y^2}

, and decide whether they exist at the origin.

Solution.

Away from the origin. $f_x = \dfrac{x}{\sqrt{x^2 + y^2}}$ and $f_y = \dfrac{y}{\sqrt{x^2 + y^2}}$ . Chain rule: $\frac{d}{dx}\sqrt u = \frac{1}{2\sqrt u}\frac{du}{dx}$ with $u = x^2 + y^2$ and $\frac{du}{dx} = 2x$ .
In any dimension. For $f(x) = \lVert x \rVert$ , $\dfrac{\partial f}{\partial x_k} = \dfrac{x_k}{\lVert x \rVert}$ , so $\nabla\lVert x \rVert = \dfrac{x}{\lVert x \rVert}$ is the unit vector pointing away from the origin. Same computation with $n$ terms under the root.
At the origin. $f(h, 0) = \sqrt{h^2} = |h|$ , and $\dfrac{|h| - 0}{h}$ is $1$ for $h > 0$ and $-1$ for $h < 0$ . So $f_x(0, 0)$ does not exist (likewise $f_y$ ). This is the corner of $|x|$ from 1.4: the graph is a cone with its point at the origin.

Answer. The gradient is the radial unit vector $(x, y)/\sqrt{x^2 + y^2}$ at every point except the origin, where the partial derivatives do not exist.

Worked example MATH 375 HW 11 (Apostol 8.9 #11)

For

f(x, y) = \log(x^2 + y^2)

(x, y) \ne (0, 0)

, compute the first partials and check that the two mixed second partials agree.

Solution.

First partials. $D_1 f = \dfrac{2x}{x^2 + y^2}$ , $D_2 f = \dfrac{2y}{x^2 + y^2}$ . Chain rule for $\log u$ , derivative $\frac{1}{u}\frac{du}{dx}$ .
Differentiate $D_2 f$ in $x$ . $D_1(D_2 f) = \dfrac{-2y\cdot 2x}{(x^2 + y^2)^2} = \dfrac{-4xy}{(x^2 + y^2)^2}$ . Quotient rule on $2y\,(x^2 + y^2)^{-1}$ with $y$ constant.
Differentiate $D_1 f$ in $y$ . $D_2(D_1 f) = \dfrac{-2x\cdot 2y}{(x^2 + y^2)^2} = \dfrac{-4xy}{(x^2 + y^2)^2}$ . Same computation with the roles swapped.

Answer. $D_1(D_2 f) = D_2(D_1 f) = \dfrac{-4xy}{(x^2 + y^2)^2}$ . The mixed partials are equal here; the next box says when to expect that.

Watch out· Mixed partials: notation, and when the order does not matter

$D_1(D_2 f)$ means: differentiate in the second variable first, then the first. Books write it $\frac{\partial^2 f}{\partial x\,\partial y}$ (Apostol) or $f_{yx}$ , and Spagnolie’s notes write $F_{xy}$ for ” $x$ first, then $y$ ”. The conventions differ, so read the definition in the source. Fortunately it rarely matters: if the second partials are continuous near the point, the two mixed partials are equal (Apostol 8.23; proved in a later chapter of this unit). Continuity is needed: see the stretch problem for a function whose two mixed partials at the origin are $+1$ and $-1$ .

Worked example MATH 375 HW 11 (Apostol 8.9 #18)

Find the constant

n

for which

v(r, t) = t^n e^{-r^2/(4t)}

satisfies

\dfrac{\partial v}{\partial t} = \dfrac{1}{r^2}\dfrac{\partial}{\partial r}\Big(r^2\dfrac{\partial v}{\partial r}\Big)

Solution.

$t$ -derivative. $v_t = nt^{n-1}e^{-r^2/4t} + t^n e^{-r^2/4t}\cdot\dfrac{r^2}{4t^2} = e^{-r^2/4t}\Big(nt^{n-1} + \tfrac14r^2t^{n-2}\Big)$ . Product rule; $\frac{d}{dt}\big(-\frac{r^2}{4t}\big) = \frac{r^2}{4t^2}$ .
$r$ -derivative. $v_r = -\dfrac{r}{2t}\,t^ne^{-r^2/4t}$ , so $r^2v_r = -\tfrac12r^3t^{n-1}e^{-r^2/4t}$ . Chain rule on the exponent, then multiply by $r^2$ .
Differentiate again. $\dfrac{\partial}{\partial r}(r^2v_r) = -\tfrac32r^2t^{n-1}e^{-r^2/4t} + \tfrac14r^4t^{n-2}e^{-r^2/4t}$ . Divide by $r^2$ : $-\tfrac32t^{n-1}e^{-r^2/4t} + \tfrac14r^2t^{n-2}e^{-r^2/4t}$ . Product rule again: $\frac{d}{dr}\big(-\frac12r^3\big) = -\frac32r^2$ , and $\frac{d}{dr}e^{-r^2/4t} = -\frac{r}{2t}e^{-r^2/4t}$ .
Compare coefficients. The $r^2t^{n-2}$ terms agree ( $\frac14$ on both sides). The $t^{n-1}$ terms need $n = -\frac32$ . Both sides are $e^{-r^2/4t}$ times a combination of $t^{n-1}$ and $r^2t^{n-2}$ .

Answer. $n = -\tfrac32$ . This $v$ is the temperature spreading from a point source: the right side is the heat equation for a spherically symmetric temperature in three dimensions (a diffusivity of 1). The numbers $t^{-3/2}$ and $e^{-r^2/4t}$ say that the heat cloud has width about $\sqrt t$ and height about $t^{-3/2}$ , the total heat being constant.

The gradient points uphill

So far the gradient is only a bookkeeping device: a list of partials. It becomes geometry through one fact, proved in the next chapter.

A preview· What we borrow from 5.3

If $f$ is differentiable at $a$ (the total derivative of 5.3: $f$ is well approximated by a linear function near $a$ ), then for every vector $y$

f'(a; y) = \nabla f(a)\cdot y.

This is Theorem “Directional derivative as a dot product with the gradient”. It holds for every field in this chapter’s examples, because their partials are continuous (a sufficient condition proved in 5.3). The last section of this chapter shows that it can fail without differentiability. Everything in this section assumes it, and says so.

Theorem T Apostol 8.12

The gradient is the direction of steepest ascent

Suppose $f'(a; y) = g\cdot y$ for every $y$ , where $g = \nabla f(a) \ne O$ . For unit vectors $u$ :

(i) $-\lVert g \rVert \le f'(a; u) \le \lVert g \rVert$ .

(ii) $f'(a; u) = \lVert g \rVert$ if and only if $u = g/\lVert g \rVert$ ; $f'(a; u) = -\lVert g \rVert$ if and only if $u = -g/\lVert g \rVert$ .

(iii) $f'(a; u) = 0$ if and only if $u \perp g$ .

In one lineIf $f'(a; y) = \nabla f(a)\cdot y$ for all $y$ and $\nabla f(a) \ne O$ , then among unit vectors $u$ the rate $f'(a; u)$ is largest, equal to $\lVert\nabla f(a)\rVert$ , for $u = \nabla f(a)/\lVert\nabla f(a)\rVert$ , smallest for $u = -\nabla f(a)/\lVert\nabla f(a)\rVert$ , and $0$ for $u$ perpendicular to $\nabla f(a)$ .

Why it is trueIt turns a list of partials into a compass: the gradient tells you which way is uphill and how steep it is, in one vector.

ProofApostol 8.12 (via Cauchy–Schwarz); this proof avoids angles

The idea: Compare u with the unit vector ĝ along g by expanding ‖u − ĝ‖², which is never negative.

Let

\hat g = g/\lVert g \rVert

, so

\lVert\hat g\rVert = 1

and

g\cdot u = \lVert g \rVert\,(\hat g\cdot u)

Name the unit vector along

g

\lVert u - \hat g \rVert^2 = 1 - 2\,\hat g\cdot u + 1 = 2 - 2\,\hat g\cdot u

Expand

(u - \hat g)\cdot(u - \hat g)

using

\lVert u \rVert = \lVert\hat g\rVert = 1

\hat g\cdot u \le 1

, with equality if and only if

u = \hat g

A squared length is

\ge 0

, and it is

0

exactly when

u = \hat g

Likewise

-\hat g\cdot u \le 1

, with equality if and only if

u = -\hat g

. Together

|\hat g\cdot u| \le 1

Replace

u

-u

in the previous step.

This proves (i) and (ii).

Multiply by

\lVert g \rVert > 0

and use

f'(a; u) = \lVert g \rVert\,(\hat g\cdot u)

(iii) follows.

\blacksquare

f'(a; u) = g\cdot u

, which is

0

exactly when

u \perp g

(definition of orthogonal, 2.9).

In the picture: the red arrow is $\nabla f$ at your dot, the orange curve is the level curve through it, and the dashed line is the tangent to that level curve. Slide the direction $u$ (green) around: the number $\nabla f\cdot u$ is largest when $u$ lies along the red arrow, zero when $u$ lies along the dashed line, and most negative on the opposite side. The red arrow always crosses the level curve at a right angle. That is the next theorem.

Quick checkYou walk along a contour line (a level curve) of a hill. What is your rate of climb, and how does the gradient sit relative to your path?

Proof workshopGive the reason for each line of the steepest-ascent proofApostol 8.12

Each line of the proof is shown in order. Choose the reason that justifies it, then press Check.

Let $\hat g = g/\lVert g \rVert$ , a unit vector, so $g \cdot u = \lVert g \rVert\,(\hat g \cdot u)$ .
$\lVert u - \hat g \rVert^2 = \lVert u \rVert^2 - 2\,\hat g \cdot u + \lVert \hat g \rVert^2 = 2 - 2\,\hat g \cdot u$ .
Hence $\hat g \cdot u \le 1$ , with equality if and only if $u = \hat g$ .
Replacing $u$ by $-u$ : $-\,\hat g \cdot u \le 1$ , with equality if and only if $u = -\hat g$ .
So $-\lVert g \rVert \le f'(a; u) \le \lVert g \rVert$ , with the extremes at $u = \pm\hat g$ .

Why· The perpendicular to level sets (proved in 5.3)

Walk along the level curve through $a$ : $f$ does not change, so the rate of change along the curve is $0$ . The rate along a direction $u$ is $\nabla f(a)\cdot u$ , so the direction of the curve is perpendicular to $\nabla f(a)$ . That is the theorem of 5.3, which needs the chain rule along a curve (not only along straight lines) to be stated correctly. Check it now on an example: $f = x^2 + y^2$ , level curve the circle through $(1, 0)$ . Its tangent there is vertical, direction $(0, 1)$ , and $\nabla f(1, 0) = (2, 0)$ ; the dot product is $0$ .

Worked example MATH 375 HW 12 (Apostol 8.14 #3)

At which points

(x, y)

of the circle

x^2 + y^2 = 1

, and in which directions, is the directional derivative of

f(x, y) = 3x^2 + y^2

largest?

Solution.

The gradient. $\nabla f = (6x, 2y)$ . Partial derivatives.
Fix a point; choose the best direction. At a point $p$ the largest directional derivative is $\lVert\nabla f(p)\rVert$ , reached in the direction $\nabla f(p)/\lVert\nabla f(p)\rVert$ . Steepest ascent theorem (we may use it: the partials are polynomials, so $f$ is differentiable by 5.3).
Now choose the best point on the circle. $\lVert\nabla f\rVert^2 = 36x^2 + 4y^2 = 4 + 32x^2$ when $x^2 + y^2 = 1$ . Replace $y^2$ by $1 - x^2$ .
Maximise. This is largest when $x^2 = 1$ (the biggest $x^2$ can be), giving $\lVert\nabla f\rVert^2 = 36$ and $\lVert\nabla f\rVert = 6$ . $x^2 \le 1$ on the circle.
The direction. At $(1, 0)$ : $\nabla f = (6, 0)$ , direction $(1, 0)$ . At $(-1, 0)$ : $\nabla f = (-6, 0)$ , direction $(-1, 0)$ . Normalise the gradient.

Answer. The largest directional derivative on the circle is $6$ , attained at $(1, 0)$ in the direction $\mathbf i$ and at $(-1, 0)$ in the direction $-\mathbf i$ . The level curves of $f$ are the ellipses $3x^2 + y^2 = c$ ; they are closest together along the $x$ -axis, where $f$ climbs fastest.

Your answerAt a point where

\nabla f = (3, 4)

, what is the largest possible value of the directional derivative

f'(a; u)

over unit vectors

u

Worked example MATH 375 HW 12 (Apostol 8.14 #1b, #1e)

Find the gradients of

f = e^x\cos y

and of

f = \log(x^2 + 2y^2 - 3z^2)

Solution.

First field. $f_x = e^x\cos y$ and $f_y = -e^x\sin y$ . Hold $y$ fixed, then $x$ fixed.
Second field. With $u = x^2 + 2y^2 - 3z^2$ : $f_x = \dfrac{2x}{u}$ , $f_y = \dfrac{4y}{u}$ , $f_z = \dfrac{-6z}{u}$ . Chain rule for $\log u$ .

Answer. $\nabla(e^x\cos y) = (e^x\cos y,\ -e^x\sin y)$ and $\nabla\log(x^2 + 2y^2 - 3z^2) = \dfrac{(2x,\ 4y,\ -6z)}{x^2 + 2y^2 - 3z^2}$ , where the argument of $\log$ is positive. Both match Apostol’s answers.

Worked example Apostol 8.14 #2

Evaluate the directional derivatives of

f(x, y, z) = x^2 + 2y^2 + 3z^2

(1, 1, 0)

in the direction of

\mathbf i - \mathbf j + 2\mathbf k

, and of

f = (x/y)^z

(1, 1, 1)

in the direction of

2\mathbf i + \mathbf j - \mathbf k

Solution.

Gradient of the first. $\nabla f = (2x, 4y, 6z)$ , which is $(2, 4, 0)$ at $(1, 1, 0)$ . Partials.
Make the direction a unit vector. $(1, -1, 2)$ has length $\sqrt6$ , so $u = \dfrac{(1, -1, 2)}{\sqrt6}$ . A directional derivative uses a unit vector; otherwise the answer is scaled by $\lVert y \rVert$ .
Dot. $f' = (2, 4, 0)\cdot u = \dfrac{2 - 4 + 0}{\sqrt6} = -\dfrac{2}{\sqrt6}$ .
Second field. $\nabla f = \big(z\,x^{z-1}y^{-z},\ -z\,x^zy^{-z-1},\ (x/y)^z\ln(x/y)\big)$ , which is $(1, -1, 0)$ at $(1, 1, 1)$ . $\ln(1) = 0$ kills the last entry.
Direction and dot. $u = (2, 1, -1)/\sqrt6$ , so $f' = \dfrac{2 - 1 + 0}{\sqrt6} = \dfrac{1}{\sqrt6}$ .

Answer. $-\dfrac{2}{\sqrt6}$ and $\dfrac{1}{\sqrt6}$ , matching Apostol’s answers. Forgetting to divide by $\sqrt6$ is the usual slip.

Worked example Apostol 8.14 #4

A differentiable scalar field

f

has, at

(1, 2)

, directional derivative

+2

toward

(2, 2)

and

-2

toward

(1, 1)

. Find

\nabla f(1, 2)

and the directional derivative toward

(4, 6)

Solution.

Directions as unit vectors. Toward $(2, 2)$ from $(1, 2)$ is $(1, 0) = e_1$ . Toward $(1, 1)$ is $(0, -1) = -e_2$ . Subtract the points and normalise; both vectors already have length 1.
Translate into partials. $f'(a; e_1) = f_x = 2$ . $f'(a; -e_2) = -f_y = -2$ , so $f_y = 2$ . Scaling lemma with $c = -1$ .
The gradient. $\nabla f(1, 2) = (2, 2)$ .
The new direction. Toward $(4, 6)$ is $(3, 4)$ , of length $5$ , so $u = (3, 4)/5$ . Unit vector.
Use the formula. $f'(a; u) = (2, 2)\cdot\frac{(3, 4)}{5} = \frac{14}{5}$ . Directional derivative equals gradient dotted with a unit vector (differentiability assumed).

Answer. $\nabla f(1, 2) = 2\mathbf i + 2\mathbf j$ and the derivative toward $(4, 6)$ is $\frac{14}{5}$ .

Worked example Apostol 8.14 #5, with new numbers

Find constants

a, b, c

so that the directional derivative of

f(x, y, z) = ax^2y + byz + cz^3

(1, 1, 1)

has the largest possible value

8

in a direction parallel to the

x

-axis.

Solution.

The gradient at the point. $\nabla f = (2axy,\ ax^2 + bz,\ by + 3cz^2)$ , which at $(1, 1, 1)$ is $(2a,\ a + b,\ b + 3c)$ . Partial derivatives, then substitute the point.
Translate the wording. “Largest possible value” means $\lVert\nabla f\rVert = 8$ , and the direction where it is reached is $\nabla f/\lVert\nabla f\rVert$ . “Parallel to the $x$ -axis” means that direction is $\pm e_1$ , so $\nabla f$ is a multiple of $e_1$ : its last two entries are $0$ . Steepest ascent theorem: the largest rate is the length of the gradient, in the direction of the gradient.
Write the equations. $a + b = 0$ and $b + 3c = 0$ , and the first entry is $\pm8$ , so $2a = \pm8$ . The length of $(2a, 0, 0)$ is $|2a|$ .
Solve. With $2a = 8$ : $a = 4$ , $b = -4$ , $c = \frac43$ (direction $+e_1$ ). With $2a = -8$ : $a = -4$ , $b = 4$ , $c = -\frac43$ (direction $-e_1$ ). Back-substitute: $b = -a$ , then $c = -\frac b3$ .

Answer. $(a, b, c) = (4, -4, \frac43)$ or $(-4, 4, -\frac43)$ . Both give a maximum rate of $8$ , one pointing along $+e_1$ and the other along $-e_1$ . The same method, with other numbers, gives Apostol’s answer $(6, 24, -8)$ or its negative.

Where the derivative must vanish

Theorem T Apostol 8.14 #10(b); Lecture 24

Partial derivatives vanish at an interior extremum

Let $f$ be defined on a ball around $a$ , with $f(x) \le f(a)$ for all $x$ in the ball (or $f(x) \ge f(a)$ for all $x$ ). If $D_kf(a)$ exists then $D_kf(a) = 0$ . Consequently, if $\nabla f(a)$ exists then $\nabla f(a) = O$ .

In one lineIf $f$ has a local maximum or minimum at an interior point $a$ and a partial derivative $D_kf(a)$ exists, then $D_kf(a) = 0$ ; so $\nabla f(a) = O$ when all partials exist.

Why it is trueIt is the many-variable Fermat theorem: it finds the only places where an extremum can hide, and it is how the normal equations of least squares come out of calculus.

ProofApostol 8.14 #10(b)

The idea: Restrict to the k-th coordinate line. The restriction is a one-variable function with an interior maximum at t = 0, so its derivative there is 0 by Fermat's theorem of 1.4.

Let

g(t) = f(a + te_k)

for small

|t|

The slice along

e_k

g

has a maximum (or minimum) at the interior point

t = 0

g(t) = f(a + te_k) \le f(a) = g(0)

, as

a + te_k

lies in the ball.

g'(0)

exists.

Derivative of a slice:

g'(0) = f'(a; e_k) = D_kf(a)

g'(0) = 0

, that is

D_kf(a) = 0

\blacksquare

Fermat's theorem (1.4).

Watch out· Necessary, not sufficient

$f(x, y) = x^2 - y^2$ has $\nabla f(0, 0) = (0, 0)$ , but the origin is not an extremum: $f > 0$ along the $x$ -axis and $f < 0$ along the $y$ -axis. A point where $\nabla f = O$ is a candidate; telling a maximum from a minimum from a saddle takes the second-derivative test of 5.6 (eigenvalues of the Hessian, using 3.2). And an extremum on a boundary need not have $\nabla f = O$ at all: “check the boundary separately” (Lecture 24).

Worked example Practice Final 2013, problem 8

Find the constants

a, b

minimising

I(a, b) = \int_0^1 (ax + b - x^2)^2\,dx

Solution.

Differentiate under the integral. $I_a = 2\int_0^1 x(ax + b - x^2)\,dx = 2\big(\tfrac a3 + \tfrac b2 - \tfrac14\big)$ and $I_b = 2\int_0^1(ax + b - x^2)\,dx = 2\big(\tfrac a2 + b - \tfrac13\big)$ . Chain rule on the square: the derivative of $(ax + b - x^2)^2$ in $a$ is $2(\ldots)\cdot x$ , in $b$ is $2(\ldots)\cdot 1$ . We may differentiate inside the integral since the integrand is a polynomial in $a, b$ .
Set both to zero. $\frac a3 + \frac b2 = \frac14$ and $\frac a2 + b = \frac13$ . At a minimum (an interior point of the plane) the gradient is zero, by the theorem.
Solve. From the second, $b = \frac13 - \frac a2$ . Substituting: $\frac a3 + \frac16 - \frac a4 = \frac14$ , so $\frac a{12} = \frac1{12}$ , $a = 1$ , $b = -\frac16$ . Linear elimination.
Show it is a minimum. $\int_0^1(ax + b - x^2)^2dx = \lVert(ax + b) - x^2\rVert^2$ is the squared distance from $x^2$ to the line-polynomial $ax + b$ . By the best approximation theorem of 2.9, the closest polynomial in $\operatorname{span}\{1, x\}$ is the orthogonal projection of $x^2$ , and it is unique. Our critical point is the only candidate, so it is that projection. The theorem says a critical point is a candidate only; here geometry tells us the minimum exists and is unique.

Answer. $a = 1$ , $b = -\frac16$ : the best straight-line fit to $x^2$ on $[0, 1]$ in the least-squares sense is $x - \frac16$ . Setting two partial derivatives to zero here is the same as the normal equations of 2.10.

A warning: derivatives in every direction are not enough

In one variable, a derivative at $a$ forces continuity at $a$ . One might hope the same holds for directional derivatives: if they exist in every direction, is $f$ continuous? It is not.

Worked example Lecture 20; Piazza @105; Apostol 8.10

Let

f(x, y) = \dfrac{xy^2}{x^2 + y^4}

for

x \ne 0

and

f(0, y) = 0

. Show that

f'(0; y)

exists for every direction

y

, that

f

is not continuous at the origin, and that

f'(0; y) \ne \nabla f(0)\cdot y

Solution.

A direction $y = (p, q)$ with $p \ne 0$ . $f(hp, hq) = \dfrac{hp\cdot h^2q^2}{h^2p^2 + h^4q^4} = \dfrac{hpq^2}{p^2 + h^2q^4}$ . So $\dfrac{f(hp, hq) - f(0)}{h} = \dfrac{pq^2}{p^2 + h^2q^4} \to \dfrac{q^2}{p}$ as $h \to 0$ . Cancel $h^2$ from numerator and denominator, then divide by $h$ .
A direction $y = (0, q)$ . $f(0, hq) = 0$ for every $h$ , since $f(0, y) = 0$ by definition. So $f'(0; (0, q)) = 0$ . The line is the $y$ -axis, where $f$ is defined to be $0$ .
Conclusion for directions. $f'(0; y)$ exists for every $y$ , equal to $q^2/p$ if $p \ne 0$ and $0$ if $p = 0$ . Piazza @105: “the directional derivative exists for every direction”.
The gradient. $\nabla f(0) = (f'(0; e_1), f'(0; e_2)) = (0, 0)$ , using $q = 0$ , $p = 1$ and then $p = 0$ , $q = 1$ . So $\nabla f(0)\cdot y = 0$ for all $y$ . But $f'(0; (1, 1)) = 1 \ne 0$ . The formula $f'(a; y) = \nabla f(a)\cdot y$ fails here; also $y \mapsto f'(0; y)$ is not even a linear function of $y$ .
Discontinuity. Along the parabola $x = y^2$ (with $y \ne 0$ ): $f(y^2, y) = \dfrac{y^2\cdot y^2}{y^4 + y^4} = \dfrac12$ . Along $x = -y^2$ : $-\frac12$ . But along every straight line through the origin $f \to 0$ (step 1 shows $f(hp, hq) = O(h)$ ). Points with $f = \frac12$ lie arbitrarily close to the origin while $f(0, 0) = 0$ .

Answer. All directional derivatives exist at the origin, yet $f$ is not continuous there. Spagnolie’s moral (Piazza @105): what goes wrong is that the directional derivative $q^2/p$ is not itself a continuous function of the direction: it explodes as $p \to 0$ , while at $p = 0$ it is $0$ .

Your answerFor the function above, what is the directional derivative

f'(0; y)

in the direction

y = (2, 3)

Quick checkSuppose

f'(a; y)

exists for every direction

y

. Which conclusion is justified?

Select the last function above (the point starts at the origin). The direction is $(p, q) = (\cos\theta, \sin\theta)$ , so the slope is $\frac{q^2}{p} = \frac{\sin^2\theta}{\cos\theta}$ . Turn the direction towards the vertical from the right and watch the slope grow without bound; from the left it becomes very negative. Press the button “straight up” to see the slope at exactly the vertical, where the formula does not apply and the slope is $0$ .

Python labFinite-difference gradients, and the field that fools them

import numpy as np

def f(p):
    x, y, z = p
    return x * y * np.sin(z**2)

def grad_formula(p):
    x, y, z = p
    return np.array([y * np.sin(z**2), x * np.sin(z**2), 2 * x * y * z * np.cos(z**2)])

a = np.array([1.0, 2.0, 3.0])
exact = grad_formula(a)
print("formula gradient:", exact.round(6))
print("   h        forward error   central error")
for k in range(1, 14):
    h = 10.0**-k
    fw = np.array([(f(a + h * e) - f(a)) / h for e in np.eye(3)])
    ce = np.array([(f(a + h * e) - f(a - h * e)) / (2 * h) for e in np.eye(3)])
    print(f"{h:8.0e}   {abs(fw - exact).max():12.2e}   {abs(ce - exact).max():12.2e}")

# A field that is not differentiable at the origin, but has a "gradient" there
def g(x, y):
    return 0.0 if x == 0 else x * y**2 / (x**2 + y**4)
h = 1e-6
print("partials of g at 0:", (g(h, 0) - g(0, 0)) / h, (g(0, h) - g(0, 0)) / h)
print("g'(0; (1,1)) =", (g(h, h) - g(0, 0)) / h, "but grad . (1,1) = 0")

Try this. Run it. The forward difference is most accurate near h = 1e-8, and the central difference near h = 1e-5 or 1e-6: below that, rounding error in f(a + h) - f(a) takes over (the subtraction of two nearly equal numbers loses digits). Change f to np.sin(x) * np.exp(y) * z and update grad_formula to match. In the second block, g has both partials 0 at the origin, so a finite-difference 'gradient check' there would pass, yet the derivative along (1,1) is 1: gradient checking is only meaningful for differentiable fields.

Where it is used

Used inside maths· Inside maths: calculus on a segment, and optimisation

The mean value theorem of 1.4 extends to scalar fields by applying it to a slice (the proof is in the problems): if $f'(a + ty; y)$ exists for $0 \le t \le 1$ then $f(a + y) - f(a) = f'(z; y)$ for some $z$ on the segment (Apostol 8.4; Lecture 20). From it follows that a field whose derivative vanishes in every direction on a ball is constant there. The vanishing theorem above is the first step of all many-variable optimisation: critical points (5.6), constrained optimisation with Lagrange multipliers (5.8), and the gradient descent of 5.10 all start from $\nabla f$ .

Used in the real world· Physics and AI: heat flow, and checking a gradient

Fourier’s law says the heat flux through a material is $\mathbf q = -k\nabla T$ , with $k$ the thermal conductivity. Pure copper has $k \approx 400\ \text{W/(m K)}$ . Take a copper rod 0.2 m long with its ends held at 100 °C and 20 °C. The temperature drops uniformly, so $\nabla T$ has length $\frac{80}{0.2} = 400\ \text{K/m}$ and the flux is $400 \times 400 = 160{,}000\ \text{W/m}^2$ , flowing from hot to cold, that is against $\nabla T$ (a rod of cross-section $1\ \text{cm}^2$ would carry 16 W). The negative sign in the law is the steepest-descent direction of the theorem above.

In machine learning, a hand-written or automatic gradient is routinely checked by comparing it with central differences, $\dfrac{f(w + he_k) - f(w - he_k)}{2h}$ for each weight $w_k$ , with $h$ around $10^{-5}$ (the lab above shows why). This costs two evaluations of the loss per weight, which for a model with a million weights is two million evaluations. That is why it is used only to test, and training uses the backward pass of 3.8, which delivers all million partials for roughly the price of one evaluation.

Traps

Watch out· Derivative with respect to y depends on the length of y

$f'(a; 2y) = 2f'(a; y)$ . For “the rate of change per unit distance” use a unit vector. Forgetting to divide by $\lVert y \rVert$ was the usual slip in Apostol 8.14 #2.

Watch out· Existing partials do not make a field differentiable, nor continuous

All the directional derivatives of $\frac{xy^2}{x^2+y^4}$ exist at the origin, and it is not continuous there. Do not conclude continuity (or $f'(a; y) = \nabla f\cdot y$ ) from the existence of derivatives. Chapter 5.3 gives the condition that works.

Watch out· The formula f′(a; y) = ∇f(a) · y is a theorem with a hypothesis

It needs differentiability at $a$ . For the example above, $\nabla f(0) = O$ yet $f'(0; (1, 1)) = 1$ .

Watch out· The gradient lives in the input space

For $f: \mathbb R^2 \to \mathbb R$ , $\nabla f$ is a vector in the $(x, y)$ plane, drawn on the contour map, pointing across the level curves. It is not a vector pointing out of the surface. “Steepest ascent” means: steepest climb of the height, expressed as a direction in the plane below.

Watch out· Partials of a cone at its tip

$\sqrt{x^2 + y^2}$ has partials everywhere except the origin: the cone’s tip is a corner. A field can have a gradient formula that looks fine and still fail to have a partial derivative at one point. Check the point separately with the definition.

Watch out· The quotient rule has a sign

$\nabla(f/g) = \dfrac{g\nabla f - f\nabla g}{g^2}$ . The printed homework key for Apostol 8.14 #7(e) has it upside down. Compare with the one-variable rule $(f/g)' = \frac{f'g - fg'}{g^2}$ .

Problems

checkMATH 375 HW 12 (Apostol 8.14 #1)

Compute the gradient of $f(x, y, z) = x^2y^3z^4$ .

Solution

$\nabla f = (2xy^3z^4,\ 3x^2y^2z^4,\ 4x^2y^3z^3)$ .

checkMATH 375 HW 11 (Apostol 8.9 #10)

For $f(x, y) = x^4 + y^4 - 4x^2y^2$ find $\nabla f$ and verify $D_1(D_2f) = D_2(D_1f)$ .

Solution

$D_1f = 4x^3 - 8xy^2$ and $D_2f = 4y^3 - 8x^2y$ . Then $D_1(D_2f) = -16xy$ and $D_2(D_1f) = -16xy$ .

check

Does $\dfrac{\partial}{\partial x}\sqrt{x^2 + y^2}$ exist at $(0, 0)$ ? At $(3, 4)$ , and what is the gradient there?

Solution

At $(0, 0)$ : no, $f(h, 0) = |h|$ has no derivative at $0$ . At $(3, 4)$ : $\nabla f = \frac{(3, 4)}{5}$ , a unit vector pointing away from the origin.

check

If $f'(a; y) = 7$ for $y = (1, 2)$ , what is $f'(a; (-3, -6))$ ?

Solution

By the scaling lemma with $c = -3$ ( $(-3, -6) = -3\,(1, 2)$ ): $f'(a; (-3, -6)) = -21$ .

useApostol 8.14 #6; MATH 375 HW 12 style

A scalar field $f$ is differentiable at a point $a$ in $\mathbb R^2$ , with $f'(a; y) = 1$ and $f'(a; z) = 2$ where $y = 2\mathbf i + 3\mathbf j$ and $z = \mathbf i + \mathbf j$ . Find $\nabla f(a)$ and describe all $(x, y)$ with $f'(a; x\mathbf i + y\mathbf j) = 6$ .

Solution

Write $\nabla f(a) = (p, q)$ . Then $2p + 3q = 1$ and $p + q = 2$ . From the second $q = 2 - p$ , so $2p + 6 - 3p = 1$ , $p = 5$ , $q = -3$ . So $\nabla f(a) = 5\mathbf i - 3\mathbf j$ , and $f'(a; x\mathbf i + y\mathbf j) = 5x - 3y = 6$ is the line through $(\frac65, 0)$ and $(0, -2)$ .

useApostol 8.14 #5

Find constants $a, b, c$ such that the directional derivative of $f(x, y, z) = axy^2 + byz + cz^2x^3$ at $(1, 2, -1)$ has the maximum value $64$ in a direction parallel to the $z$ -axis.

Solution

$\nabla f = (ay^2 + 3cz^2x^2,\ 2axy + bz,\ by + 2czx^3)$ , which at $(1, 2, -1)$ is $(4a + 3c,\ 4a - b,\ 2b - 2c)$ . The maximum rate is $\lVert\nabla f\rVert$ , attained in the direction of $\nabla f$ . For that direction to be parallel to the $z$ -axis, the first two entries vanish: $4a + 3c = 0$ and $4a = b$ . The third entry is $\pm 64$ . Taking $+64$ (direction $+\mathbf k$ ): $b = 4a$ , $c = -\frac{4a}{3}$ , $2b - 2c = 8a + \frac{8a}{3} = \frac{32a}{3} = 64$ , so $a = 6$ , $b = 24$ , $c = -8$ . Taking $-64$ gives $(a, b, c) = (-6, -24, 8)$ (maximum 64 in the direction $-\mathbf k$ ). Apostol lists both.

usePractice Final 2013, problem 2(a)

$f$ is differentiable with $f(1, 2) = 1$ , $f_x(1, 2) = 2$ , $f_y(1, 2) = -1$ . Estimate $f(\frac43, \frac53)$ by walking along the straight line from $(1, 2)$ .

Solution

$y = (\frac13, -\frac13)$ and $f(a + y) \approx f(a) + \nabla f(a)\cdot y = 1 + \frac23 + \frac13 = 2$ . The reason: the slice $g(t) = f(a + ty)$ has $g(0) = f(a)$ and slope $g'(0) = \nabla f(a)\cdot y$ (differentiability), and $g(1) \approx g(0) + g'(0)$ is the tangent line of the slice. This is the linearization of 5.3, worked there as an example; the improvement in part (b) of that exam uses the second derivatives of 5.6.

use

Find the maximum and minimum of $f(x, y) = x^2 + y^2 - 2x$ over all of $\mathbb R^2$ using the vanishing theorem, and explain why the critical point is a minimum.

Solution

$\nabla f = (2x - 2, 2y) = O$ at $(1, 0)$ , with $f(1, 0) = -1$ . Completing the square, $f = (x - 1)^2 + y^2 - 1 \ge -1$ , so $(1, 0)$ is the global minimum. There is no maximum: $f \to \infty$ as $\lVert(x, y)\rVert \to \infty$ .

proveMATH 375 HW 12 (Apostol 8.14 #7d, #7e)

If $f, g$ are scalar fields with gradients, prove $\nabla(fg) = f\nabla g + g\nabla f$ and, where $g \ne 0$ , $\nabla(f/g) = \dfrac{g\nabla f - f\nabla g}{g^2}$ .

Solution

Plan. Work one coordinate at a time with the one-variable rules.

The $k$ -th partial of $fg$ is a one-variable derivative with the other variables held fixed, so the product rule of 1.4 gives $D_k(fg) = (D_kf)g + f(D_kg)$ . Collecting the components: $\nabla(fg) = g\nabla f + f\nabla g$ . Likewise the quotient rule of 1.4 gives $D_k(f/g) = \dfrac{g\,D_kf - f\,D_kg}{g^2}$ , so $\nabla(f/g) = \dfrac{g\nabla f - f\nabla g}{g^2}$ . $\blacksquare$ (The printed key had the numerator reversed.)

proveApostol 8.14 #8

Let $\mathbf r = (x, y, z)$ and $r = \lVert\mathbf r\rVert$ . Show $\nabla r = \mathbf r/r$ for $\mathbf r \ne O$ , and $\nabla(r^n) = nr^{n-2}\mathbf r$ . Is the second formula valid for negative or zero $n$ ? Find a scalar field $f$ with $\nabla f = \mathbf r$ .

Solution

Plan. Differentiate $r^n = (x^2 + y^2 + z^2)^{n/2}$ by the chain rule.

$\dfrac{\partial r}{\partial x} = \dfrac{x}{r}$ , and similarly for $y, z$ : $\nabla r = \mathbf r/r$ (a unit vector). For $r^n = (x^2 + y^2 + z^2)^{n/2}$ : $\dfrac{\partial}{\partial x}r^n = \frac n2(x^2 + y^2 + z^2)^{n/2 - 1}\cdot 2x = nr^{n-2}x$ , so $\nabla(r^n) = nr^{n-2}\mathbf r$ . The computation is valid for every real $n$ at points where $r \ne 0$ (for $n \le 0$ the origin is excluded; for $n = 0$ the gradient of the constant $1$ is $O$ , which the formula gives). For $\nabla f = \mathbf r$ take $n = 2$ and divide by 2: $f = \frac12r^2 = \frac12(x^2 + y^2 + z^2)$ . $\blacksquare$

proveApostol 8.4 (Theorem) and 8.9 #20(a)

(a) Prove the mean value theorem for scalar fields: if $f'(a + ty; y)$ exists for every $t \in [0, 1]$ , then $f(a + y) - f(a) = f'(a + \theta y; y)$ for some $\theta \in (0, 1)$ . (b) Deduce: if $f'(x; y) = 0$ for every $x$ in a ball $B(a, r)$ and every vector $y$ , then $f$ is constant on the ball.

Solution

Plan. Apply the one-variable mean value theorem to the slice.

(a) Let $g(t) = f(a + ty)$ . By the derivative-of-a-slice theorem, $g'(t) = f'(a + ty; y)$ exists on $[0, 1]$ . Then $g$ is differentiable on $[0, 1]$ hence continuous, and the mean value theorem of 1.4 gives $\theta \in (0, 1)$ with $g(1) - g(0) = g'(\theta)$ , that is $f(a + y) - f(a) = f'(a + \theta y; y)$ . (b) Let $x \in B(a, r)$ and $y = x - a$ . The segment from $a$ to $x$ lies in the ball (each point $a + ty$ has $\lVert ty \rVert \le \lVert y \rVert < r$ ). By (a), $f(x) - f(a) = f'(z; y) = 0$ for some $z$ on the segment. So $f(x) = f(a)$ for every $x$ in the ball. $\blacksquare$

proveApostol 8.9 #22

(a) Prove there is no scalar field $f$ such that $f'(a; y) > 0$ for a fixed $a$ and every nonzero $y$ . (b) Give a scalar field with $f'(x; y) > 0$ for a fixed nonzero $y$ and every $x$ .

Solution

(a) If $f'(a; y)$ exists and is positive, the scaling lemma with $c = -1$ gives $f'(a; -y) = -f'(a; y) < 0$ , which contradicts positivity in the direction $-y$ . (b) $f(x) = x\cdot y$ : by the worked example, $f'(x; y) = y\cdot y = \lVert y \rVert^2 > 0$ for all $x$ . $\blacksquare$

proveMATH 375 Final Review, 8.8 to 8.13

Prove that for the field $f(x, y) = \dfrac{xy}{x^2 + y^2}$ , $f(0, 0) = 0$ , both partial derivatives exist at the origin and are $0$ , but the directional derivative in the direction $(1, 1)$ does not exist.

Solution

Along the axes, $f = 0$ , so $D_1f(0) = D_2f(0) = 0$ . For $y = (1, 1)$ : $\dfrac{f(h, h) - f(0)}{h} = \dfrac{\frac12 - 0}{h} = \dfrac1{2h}$ , which has no limit as $h \to 0$ (it tends to $+\infty$ from the right and $-\infty$ from the left). So $f'(0; (1, 1))$ does not exist, although $\nabla f(0) = O$ does. $\blacksquare$

stretchApostol 8.23; Schwarz and Peano

Let $f(x, y) = \dfrac{xy(x^2 - y^2)}{x^2 + y^2}$ for $(x, y) \ne (0, 0)$ and $f(0, 0) = 0$ . Compute $D_1f(0, y)$ and $D_2f(x, 0)$ , and deduce that $D_2(D_1f)(0, 0)$ and $D_1(D_2f)(0, 0)$ are different.

Solution

For $y \ne 0$ : $\dfrac{f(h, y) - f(0, y)}{h} = \dfrac{y(h^2 - y^2)}{h^2 + y^2} \to -y$ as $h \to 0$ , so $D_1f(0, y) = -y$ (also true at $y = 0$ ). For $x \ne 0$ : $\dfrac{f(x, h) - f(x, 0)}{h} = \dfrac{x(x^2 - h^2)}{x^2 + h^2} \to x$ , so $D_2f(x, 0) = x$ . Then $D_2(D_1f)(0, 0) = \frac{d}{dy}(-y)\big|_0 = -1$ and $D_1(D_2f)(0, 0) = \frac{d}{dx}(x)\big|_0 = +1$ . The mixed partials differ, so the second partials of this $f$ cannot be continuous at the origin: the equality theorem needs continuity.

stretchApostol 8.17 #6, previewed

Let $f(x, y) = \sqrt{|xy|}$ . Show that both partial derivatives are $0$ at the origin, but that $f'(0; (1, 1))$ does not exist. What does this say about a tangent plane to the surface $z = f(x, y)$ at the origin?

Solution

On the axes $f = 0$ , so both partials are $0$ . Along $(1, 1)$ : $\dfrac{f(h, h) - 0}{h} = \dfrac{|h|}{h}$ , which is $1$ for $h > 0$ and $-1$ for $h < 0$ : no limit. Cutting the surface by the plane $x = y$ gives $z = |x|$ , a V with a corner at the origin. A tangent plane would have to contain the tangent line to every smooth curve on the surface through the origin; no line fits a corner. So the surface has no tangent plane at the origin, although both partials exist (and are zero). Chapter 5.3 explains: the partials alone do not determine a tangent plane.

Summary

The derivative of $f$ at $a$ with respect to $y$ is $f'(a; y) = \lim_{h \to 0}\frac{f(a + hy) - f(a)}{h} = g'(0)$ for the slice $g(t) = f(a + ty)$ . For a unit vector $y$ it is the directional derivative: the rate of change per unit distance. Scaling $y$ by $c$ scales it by $c$ .
A partial derivative $D_kf = f'(\cdot; e_k)$ is computed by holding the other variables fixed. The gradient $\nabla f = (D_1f, \dots, D_nf)$ lists them; it lives in the input space.
If $f$ is differentiable (5.3), then $f'(a; y) = \nabla f(a)\cdot y$ for all $y$ . Then $\nabla f$ points in the direction of steepest ascent, its length is the steepest rate, and it is perpendicular to the level set through the point.
At an interior maximum or minimum every partial derivative that exists is $0$ (Fermat in many variables). This is a necessary condition only ( $x^2 - y^2$ at the origin), and the boundary must be checked separately.
Warning: a field can have a directional derivative in every direction at a point and still be discontinuous there ( $\frac{xy^2}{x^2+y^4}$ ). The map $y \mapsto f'(a; y)$ need not be linear, and $f'(a; y) = \nabla f\cdot y$ can fail. That gap is closed by the total derivative of 5.3.
Mixed partials are equal when the second partials are continuous; without continuity they can differ ( $+1$ against $-1$ for Schwarz and Peano’s example).

Watch and read

The same ideas explained by someone else. Use these when an explanation here does not land.

5.2 Directional derivatives, partial derivatives and the gradient

The picture: the slope of a slice

The idea, in words

Definitions

Directional derivative

Scaling a direction scales the derivative

The derivative of a slice

Partial derivative

Gradient

Computing partial derivatives

The gradient points uphill

The gradient is the direction of steepest ascent

Where the derivative must vanish

Partial derivatives vanish at an interior extremum

A warning: derivatives in every direction are not enough

Where it is used

Traps

Problems

Summary

Watch and read

Where this sits

Comes from

Leads to

Same idea elsewhere