Lagrange Multipliers

In science and mathematics it is very common for the need to determine the local extrema of a function, $f(x,y,z,\cdots)$ subject to some constraint. If the constraint (function), which depends on same set of independent variables, can be expressed as $h(x,y,z,\cdots) = 0$, then one can rewrite a new constrained function as:

$$ g\left(x,y,z,\cdots,\lambda \right) = f\left(x,y,z,\cdots\right) + \lambda h\left(x,y,z,\cdots\right).$$

The variable $\lambda$ is known as the Lagrange multiplier. The condition for an extrema (i.e., minima or maxima) occurs when the partial derivatives are zero, more explicitly:

$$\begin{align}\frac{\partial g}{\partial x} &= \frac{\partial f}{\partial x} - \lambda \frac{\partial h}{\partial x} = 0 \\\frac{\partial g}{\partial y} &= \frac{\partial f}{\partial y} - \lambda \frac{\partial h}{\partial y} = 0  \\ \vdots \\\frac{\partial g}{\partial \lambda} &= \frac{\partial f}{\partial \lambda} - \lambda \frac{\partial h}{\partial \lambda} = 0,\end{align}$$

with the last term $\frac{\partial g}{\partial \lambda} = h(x,y,z) =  0$. This sets up our system of equations that can be solved algebraically. To provide some additional context, this approach is used in areas such as thermodynamics and machine learning (more commonly referred to as regularization). The more general form of the function $g$ when several constraints are used is given by the Lagrangian function:

$$ \mathcal{L}\left(x,y,z,\cdots,\{\lambda\}\right) = f\left(x,y,z,\cdots\right) + \sum_i \lambda_i \, h_i\left(x,y,z,\cdots\right) $$

Lets see the use of Langrange multiplier in action with a simple example, a parabola constrained by a circle. Our function, $g(x,y) = 10x^2 - 5y$, and our constraint is $h(x,y) = x^2 + y^2 -1$.  Writing the new function $h(x,y,\lambda)$,

$$ h\left(x,y,\lambda\right) = 10x^2 - 5y + \lambda(x^2+y^2-1) $$

Now working through the partial derivatives of $h(x,y,\lambda)$:

$$\frac{\partial g}{\partial x} = 10x+\lambda(2x) = 0$$
$$\frac{\partial g}{\partial y} = -5 +\lambda(2y) = 0$$
$$\frac{\partial g}{\partial \lambda} = x^2+y^2-1 = 0$$

We now find the Lagrange multiplier, $\lambda$, using the first equation,

$$\begin{align}
2x(10+\lambda) &= 0 \\
\lambda &= -10
\end{align}
$$

we can calculate $y$, with the second equation,

$$\begin{align}
-5-10*(2y) &= 0 \\
y &= -0.25
\end{align}
$$

and with the constraint equation we can find $x = 0.9862$. This example is very simple and does not demosntrate the full power of Lagrange multipliers, which is extremely powerful in multivariate calculus. The quote for this blog post is by Joseph-Louis Lagrange himself:

"I regard as quite useless the reading of large treatises of pure analysis: too large a number of methods pass at once before the eyes. It is in the works of applications that one must study them; one judges their ability there and one apprises the manner of making use of them"
- Joseph-Louis Lagrange

References & Additional Reading

[1] Advanced Engineering Mathematics, Wiley, 8th ed., E. Kreyszig.

[2] The Mathematics Companion, IoP, 1st ed., A.C. Fischer-Cripps.

[3] Thermodynamics in Materials Science, Robert DeHoff, 2nd ed.

Reuse and Attribution