Unit 5

Differential calculus in many variables

The derivative as the best linear approximation, in any dimension.

10 chapters written · 46 definitions · 60 theorems · builds on Unit 1, Unit 2, Unit 4

The story

The derivative of a function of many variables is a linear map: the best linear approximation near a point. Unit 2 made that idea precise; this unit uses it. Partial derivatives and the gradient (5.2) are pieces of the derivative, the Jacobian (5.3) is its matrix, and the chain rule (5.4) is matrix multiplication.

From there: Taylor's theorem and the Hessian decide maxima and minima using eigenvalues (5.6), the inverse and implicit function theorems say when equations can be solved locally (5.7), and Lagrange multipliers handle constraints (5.8). The unit ends with partial differential equations (5.9) and gradient descent (5.10).

Chapters

The theorem chain

Every theorem in the unit, in the order it is proved. Each one is used by the ones after it. The label says whether it is on the 375/376 exams (T), an Apostol extra (A), or beyond both (B).

5.1 Scalar and vector fields, limits and continuity

Theorem · T Limit laws for fields If $F \to b$ and $G \to c$ as $x \to a$ , then $F + G \to b + c$ , $\lambda F \to \lambda b$ , $F \cdot G \to b \cdot c$ and $\lVert F \rVert \to \lVert b \rVert$ .
Theorem · T A field has a limit exactly when its components do $F(x) \to b$ as $x \to a$ if and only if $F_k(x) \to b_k$ for every component $k$ .
Theorem · T A limit forces the same value along every path If $F(x) \to b$ as $x \to a$ and a path $r(t) \to a$ with $r(t) \ne a$ , then $F(r(t)) \to b$ along the path.
Proposition · T Squeeze for fields If $\lVert F(x) - b \rVert \le \varphi(\lVert x - a \rVert)$ near $a$ and $\varphi(r) \to 0$ as $r \to 0^+$ , then $F(x) \to b$ .
Theorem · T Building continuous fields Sums, scalar multiples, dot products, norms and compositions of continuous fields are continuous; so are the coordinate functions, linear maps, polynomials, and quotients where the denominator is not zero.
Theorem · T A strict inequality between continuous fields defines an open set

5.2 Directional derivatives, partial derivatives and the gradient

Lemma · T Scaling a direction scales the derivative If $f'(a; y)$ exists then $f'(a; cy) = c\,f'(a; y)$ for every real $c$ .
Theorem · T The derivative of a slice For $g(t) = f(a + ty)$ , the derivative $g'(t)$ exists exactly when $f'(a + ty; y)$ exists, and then $g'(t) = f'(a + ty; y)$ .
Theorem · T The gradient is the direction of steepest ascent If $f'(a; y) = \nabla f(a)\cdot y$ for all $y$ and $\nabla f(a) \ne O$ , then among unit vectors $u$ the rate $f'(a; u)$ is largest, equal to $\lVert\nabla f(a)\rVert$ , for $u = \nabla f(a)/\lVert\nabla f(a)\rVert$ , smallest for $u = -\nabla f(a)/\lVert\nabla f(a)\rVert$ , and $0$ for $u$ perpendicular to $\nabla f(a)$ .
Theorem · T Partial derivatives vanish at an interior extremum If $f$ has a local maximum or minimum at an interior point $a$ and a partial derivative $D_kf(a)$ exists, then $D_kf(a) = 0$ ; so $\nabla f(a) = O$ when all partials exist.

5.3 The total derivative, the Jacobian matrix and tangent planes

Theorem · T Directional derivative as a dot product with the gradient If a scalar field $f$ is differentiable at $a$ , then $f'(a; y)$ exists for every $y$ and equals $T_a(y) = \nabla f(a)\cdot y$ ; in particular $T_a$ is unique.
Theorem · T A vector field is differentiable exactly when its components are $F = (F_1, \dots, F_m)$ is differentiable at $a$ if and only if every $F_i$ is, and then $F'(a)(y) = (\nabla F_1(a)\cdot y, \dots, \nabla F_m(a)\cdot y)$ , whose matrix is the Jacobian.
Theorem · T Differentiable fields are continuous If $F$ is differentiable at $a$ , then $F$ is continuous at $a$ .
Theorem · A Continuous partial derivatives imply differentiable If $f$ is continuously differentiable at $a$ , then $f$ is differentiable at $a$ , with $T_a(v) = \nabla f(a)\cdot v$ .
Theorem · T Rate of change along a curve If $r$ is differentiable at $t$ and $f$ is differentiable at $r(t)$ , then $g(t) = f(r(t))$ has $g'(t) = \nabla f(r(t))\cdot r'(t)$ .
Theorem · T The gradient is perpendicular to level sets If $f$ is differentiable at $a$ and $r$ is a curve in the level set $f = c$ through $a$ , then $\nabla f(a)\cdot r'(t_0) = 0$ : the gradient is perpendicular to the tangent vector of every such curve.

5.4 The chain rule for fields

Theorem · T The chain rule for fields If $g$ is differentiable at $a$ and $f$ is differentiable at $b = g(a)$ , then $h = f \circ g$ is differentiable at $a$ and $h'(a) = f'(b) \circ g'(a)$ ; in matrices, $Dh(a) = Df(b)\,Dg(a)$ .
Corollary · A The gradient of a composite If $f$ is a scalar field and $g = (g_1, \dots, g_n)$ a vector field, then $\nabla(f \circ g)(a) = \sum_{k=1}^n D_kf(g(a))\,\nabla g_k(a) = Dg(a)^{\mathsf T}\,\nabla f(g(a))$ .
Corollary · T Jacobian determinants multiply If $f$ and $g$ both map $\mathbb R^n$ to $\mathbb R^n$ , then $\det D(f \circ g)(a) = \det Df(g(a))\cdot\det Dg(a)$ .
Theorem · T The derivative of an inverse field If $F: U \to V$ is a diffeomorphism, then $DF(x)$ is an invertible matrix for every $x$ , and $D(F^{-1})(F(x)) = \big(DF(x)\big)^{-1}$ .
Theorem · T Second derivative along a path If $r$ is twice differentiable and $f$ has continuous second partials, then $\dfrac{d^2}{dt^2}f(r(t)) = \sum_{i,j} D_jD_if(r(t))\,r_i'(t)\,r_j'(t) + \nabla f(r(t))\cdot r''(t)$ .
Theorem · B Second derivatives of a composite field If $h = f \circ g$ with $f$ a scalar field and $g: \mathbb R^p \to \mathbb R^n$ , all $C^2$ , then the matrix of second partials of $h$ is $Dg(a)^{\mathsf T}\,[D_lD_kf(b)]\,Dg(a) + \sum_{k=1}^n D_kf(b)\,[D_jD_ig_k(a)]$ .

5.5 Higher partial derivatives and the equality of mixed partials

Theorem · B Differentiability from n minus one continuous partials If $D_1f, \dots, D_nf$ exist in a ball around $a$ , and $D_2f, \dots, D_nf$ are continuous at $a$ (while $D_1f$ merely exists at $a$ ), then $f$ is differentiable at $a$ with total derivative $v \mapsto \nabla f(a)\cdot v$ .
Lemma · A The rectangle lemma If $f_x$ and $f_{xy}$ exist on an open set containing the rectangle with corners $(a, b)$ and $(a + h, b + k)$ , then $\Delta(h, k) = hk\,f_{xy}(\xi, \eta)$ for some point $(\xi, \eta)$ strictly inside the rectangle.
Theorem · A Equality of mixed partials (Schwarz, Clairaut) If $f_x$ , $f_y$ , $f_{xy}$ , $f_{yx}$ exist on an open set containing $(a, b)$ and $f_{xy}$ and $f_{yx}$ are both continuous at $(a, b)$ , then $f_{xy}(a, b) = f_{yx}(a, b)$ .
Theorem · A Peano's refinement of Schwarz's theorem If $f_x$ , $f_y$ and $f_{xy}$ exist on an open set containing $(a, b)$ and $f_{xy}$ is continuous at $(a, b)$ , then $f_{yx}(a, b)$ exists and equals $f_{xy}(a, b)$ .
Corollary · A Order does not matter for C^k fields If $f$ is of class $C^k$ on an open set, then any two partial derivatives of order at most $k$ that differentiate the same number of times in each variable are equal; in particular the matrix of second partials of a $C^2$ field is symmetric.

5.6 Taylor's formula, the Hessian and extrema

Theorem · T Second-order Taylor formula for scalar fields If $f$ is $C^2$ on a ball $B(a, r)$ and $\lVert v \rVert < r$ , then $f(a + v) = f(a) + \nabla f(a)\cdot v + \frac12v^{\mathsf T}H_f(a + cv)\,v$ for some $c \in (0, 1)$ , and also $f(a + v) = f(a) + \nabla f(a)\cdot v + \frac12v^{\mathsf T}H_f(a)\,v + \lVert v \rVert^2E_2(a, v)$ with $E_2(a, v) \to 0$ as $v \to O$ .
Lemma · A A quadratic form is squeezed by the extreme eigenvalues If $A$ is a real symmetric matrix with eigenvalues $\lambda_{\min} \le \dots \le \lambda_{\max}$ , then $\lambda_{\min}\lVert v \rVert^2 \le v^{\mathsf T}Av \le \lambda_{\max}\lVert v \rVert^2$ for every $v$ , with equality at the corresponding eigenvectors.
Theorem · T The second-derivative test
Corollary · A A local minimum needs a positive semidefinite Hessian If a $C^2$ field has a local minimum at a critical point $a$ , then every eigenvalue of $H_f(a)$ is at least $0$ ; if it has a local maximum, every eigenvalue is at most $0$ .
Theorem · T The second-derivative test for two variables

5.7 Inverse and implicit functions

Lemma · B Mean value inequality for fields If $H$ is differentiable on a convex set $K$ and $\lVert DH(z)\rVert \le M$ for all $z \in K$ , then $\lVert H(x_1) - H(x_2)\rVert \le M\lVert x_1 - x_2\rVert$ for $x_1, x_2 \in K$ .
Theorem · B Inverse function theorem If $F: U \to \mathbb R^n$ is continuously differentiable on an open set $U$ and $\det DF(a) \ne 0$ , then $F$ has a continuously differentiable local inverse $G$ near $a$ , with $DG(F(x)) = DF(x)^{-1}$ .
Theorem · T Implicit function theorem If $F(x, y) = O$ at $(a, b)$ , $F$ is continuously differentiable and the $m \times m$ matrix $D_yF(a, b)$ is invertible, then near $(a, b)$ the equation defines exactly one $y = g(x)$ , it is continuously differentiable, and $Dg(x) = -D_yF^{-1}D_xF$ .
Corollary · T Implicit differentiation formulas Under the hypotheses of the implicit function theorem: $\dfrac{dy}{dx} = -\dfrac{F_x}{F_y}$ ; for $F(x, y, z) = 0$ , $z_x = -\dfrac{F_x}{F_z}$ and $z_y = -\dfrac{F_y}{F_z}$ ; for two equations in $u, v$ , $u_x = -\dfrac{\partial(F, G)/\partial(x, v)}{\partial(F, G)/\partial(u, v)}$ .
Corollary · B Level sets near a regular point are graphs If $a$ is a regular point of $F: \mathbb R^N \to \mathbb R^m$ with $F(a) = c$ , then near $a$ the level set $\{F = c\}$ is the graph of a $C^1$ function of $N - m$ of the coordinates, and its tangent vectors at $a$ are exactly the vectors in the null space of $DF(a)$ .

5.8 Constrained extrema and Lagrange multipliers

Theorem · T Lagrange multiplier theorem If $f, g_1, \dots, g_m$ are $C^1$ , $f$ has a local constrained extremum at $a$ , and $a$ is a regular point of $g$ (the gradients $\nabla g_i(a)$ are linearly independent), then $\nabla f(a) = \sum\lambda_i\nabla g_i(a)$ for unique numbers $\lambda_i$ .
Theorem · B Second-order test for constrained extrema At a regular point with multipliers $\lambda_i$ : if $v^{\mathsf T}Hv \gt 0$ for all nonzero $v$ in the tangent space then $a$ is a strict local constrained minimum; if $\lt 0$ , a strict local maximum; a local minimum forces $v^{\mathsf T}Hv \ge 0$ on the tangent space.
Corollary · B Bordered determinant test in two variables For $f(x, y)$ with one constraint $g$ at a regular point: $\det B \gt 0$ means a strict local constrained maximum, $\det B \lt 0$ a strict local constrained minimum, where $B$ is the $3 \times 3$ bordered Hessian.
Theorem · B The multiplier is the price of the constraint If the constrained optimum $x^*(c)$ depends differentiably on $c$ and $V(c) = f(x^*(c))$ , then $V'(c) = \lambda(c)$ : the multiplier is the rate at which the best value changes when the constraint $g = c$ is moved.
Theorem · B Gradient inequality for convex functions A differentiable convex function lies above each of its tangent planes: $f(y) \ge f(x) + \nabla f(x)\cdot(y - x)$ .
Theorem · B Positive semidefinite Hessian implies convex If $f$ is $C^2$ on an open convex set and its Hessian is positive semidefinite everywhere, then $f$ is convex; in particular $\tfrac12x^{\mathsf T}Ax - b^{\mathsf T}x$ is convex when $A$ is symmetric positive semidefinite.
Corollary · B A convex Lagrangian gives a global minimum If $x \mapsto L(x, \lambda) = f(x) - \sum\lambda_i(g_i(x) - c_i)$ is convex and $a$ satisfies $\nabla f(a) = \sum\lambda_i\nabla g_i(a)$ and $g(a) = c$ , then $a$ is an absolute minimum of $f$ on $\{g = c\}$ .

5.9 Partial differential equations

Proposition · A Superposition for linear homogeneous equations The solutions of a linear homogeneous PDE form a vector space: sums and multiples of solutions are solutions.
Theorem · A First-order equation with constant coefficients If $a, b$ are constants, not both $0$ , then $f$ satisfies $af_x + bf_y = 0$ if and only if $f(x, y) = g(bx - ay)$ for some differentiable function $g$ of one variable.
Theorem · A d'Alembert's solution of the wave equation If $F$ is twice differentiable and $G$ is differentiable, then $u(x, t) = \tfrac12\big[F(x + ct) + F(x - ct)\big] + \dfrac1{2c}\displaystyle\int_{x - ct}^{x + ct}G(s)\,ds$ solves $u_{tt} = c^2u_{xx}$ with $u(x, 0) = F(x)$ and $u_t(x, 0) = G(x)$ ; conversely every $C^2$ solution on $\mathbb R^2$ has this form.
Lemma · A Orthogonality of the sine modes $\displaystyle\int_0^L\sin\frac{n\pi x}L\sin\frac{m\pi x}L\,dx = 0$ for $n \ne m$ and $= \frac L2$ for $n = m$ ; hence if $f = \sum b_m\sin\frac{m\pi x}L$ (finite sum) then $b_n = \frac2L\int_0^Lf\sin\frac{n\pi x}L$ .
Theorem · B Uniqueness for the heat equation A $C^2$ solution of $u_t = \alpha u_{xx}$ on $[0, L]\times[0, T]$ with given initial temperature and given end temperatures is unique.
Lemma · A Polar form of the Laplacian If $U(r, \theta) = u(r\cos\theta, r\sin\theta)$ with $u$ of class $C^2$ , then for $r \gt 0$ : $u_{xx} + u_{yy} = U_{rr} + \dfrac1rU_r + \dfrac1{r^2}U_{\theta\theta}$ .
Theorem · B Mean value property of harmonic functions If $u$ is harmonic on an open set containing the closed disc of radius $R$ about $a$ , then $u(a) = \dfrac1{2\pi}\displaystyle\int_0^{2\pi}u\big(a_1 + R\cos\theta,\ a_2 + R\sin\theta\big)\,d\theta$ : the value at the centre is the average over the circle.
Corollary · B Maximum principle for harmonic functions If $u$ is harmonic on a connected open set $\Omega$ and has a maximum (or a minimum) at a point of $\Omega$ , then $u$ is constant. So a non-constant harmonic function on a bounded region, continuous up to the boundary, attains its largest and smallest values on the boundary only.

5.10 Gradient descent, momentum and Newton's method

Lemma · B Descent lemma If $\nabla f$ is $L$ -Lipschitz, then $f(y) \le f(x) + \nabla f(x)\cdot(y - x) + \frac L2\lVert y - x\rVert^2$ ; consequently the step $y = x - \eta\nabla f(x)$ with $\eta \le \frac1L$ gives $f(y) \le f(x) - \frac\eta2\lVert\nabla f(x)\rVert^2$ .
Theorem · B Gradient descent on a quadratic For $f(x) = \frac12x^{\mathsf T}Ax - b^{\mathsf T}x$ with $A$ symmetric positive definite with eigenvalues in $[\mu, L]$ , gradient descent converges to $x^* = A^{-1}b$ for every start if and only if $0 \lt \eta \lt 2/L$ ; the error is multiplied by $\max(|1 - \eta\mu|, |1 - \eta L|)$ per step, which is smallest, equal to $\frac{\kappa - 1}{\kappa + 1}$ with $\kappa = L/\mu$ , at $\eta = \frac2{L + \mu}$ .
Corollary · B Gradient descent is a contraction If $f$ is $C^2$ with $\mu I \preceq H_f(x) \preceq LI$ everywhere on $\mathbb R^n$ and $0 \lt \eta \lt 2/L$ , then $T(x) = x - \eta\nabla f(x)$ is a contraction with constant $\rho = \max(|1 - \eta\mu|, |1 - \eta L|) \lt 1$ , so gradient descent converges to the unique minimiser $x^*$ and $\lVert x_k - x^*\rVert \le \dfrac{\rho^k}{1 - \rho}\lVert x_1 - x_0\rVert$ .
Theorem · B Gradient descent on a convex smooth function If $f$ is convex and $L$ -smooth with a minimiser $x^*$ , then gradient descent with $\eta = 1/L$ satisfies $f(x_k) - f(x^*) \le \dfrac{L\lVert x_0 - x^*\rVert^2}{2k}$ .
Theorem · B Momentum on a quadratic For a quadratic with eigenvalues in $[\mu, L]$ , the heavy-ball method with $\eta = \dfrac4{(\sqrt L + \sqrt\mu)^2}$ and $\beta = \Big(\dfrac{\sqrt\kappa - 1}{\sqrt\kappa + 1}\Big)^2$ makes the error decay at the rate $\dfrac{\sqrt\kappa - 1}{\sqrt\kappa + 1}$ (up to a factor polynomial in $k$ ).
Theorem · B Noise floor of stochastic gradient descent For the loss $\frac1N\sum\frac12(w - a_i)^2$ , whose minimiser is the mean $\bar a$ , SGD with a fixed step $\eta \in (0, 2)$ has $\mathrm{E}[w_k] - \bar a \to 0$ but $\mathrm{E}[(w_k - \bar a)^2] \to \dfrac{\eta\sigma^2}{2 - \eta}$ , where $\sigma^2$ is the variance of the $a_i$ ; with steps $\eta_k = \frac1{k + 1}$ the iterate is the running average and the error tends to $0$ .
Theorem · B Quadratic convergence of Newton's method If $F(x^*) = O$ , $J_F(x^*)$ is invertible with $\lVert J_F(x^*)^{-1}\rVert \le \beta_0$ and $J_F$ is $M$ -Lipschitz near $x^*$ , then for $\lVert x_0 - x^*\rVert \le \dfrac1{2\beta_0M}$ the Newton iterates satisfy $\lVert x_{k+1} - x^*\rVert \le \beta_0M\lVert x_k - x^*\rVert^2$ : the number of correct digits doubles at each step.
Proposition · B Gradient descent near a saddle For $f = \frac12(ax^2 - \gamma y^2)$ with $a, \gamma \gt 0$ , gradient descent with step $\eta$ has $y_k = (1 + \eta\gamma)^ky_0$ : it leaves the saddle exponentially fast but with a time of order $\dfrac1{\eta\gamma}\ln\dfrac1{|y_0|}$ , which is long when the negative curvature $\gamma$ is small.

Terms this unit defines

Unit review

A mixed set of 12 quick checks drawn at random from the 77 in this unit's chapters. Each one says which chapter it comes from, so a miss tells you what to reread. Press "New set" for another mix.

Where it leads

Units that build on this one: Unit 6, Integration in many variables, Unit 7, Differential equations, Unit 8, Fourier analysis, Unit 9, Complex analysis, Unit 13, Geometry and topology, Unit 15, Optimisation, dynamics and computation.