Optimization Made Trivial: the Lagrange Multiplier

A few years back, I was asked about my preference between pure and applied mathematics. Being the purist I was, I had no doubt that pure mathematics merited far more consideration than applied mathematics. However, recently, upon the request of my research supervisor, I’ve began treating applied mathematics with some more respect. And I’ve found, quite ironically, that both facets of mathematics are beautiful in their own way. In this article, I am going to share my revelation by displaying the beauty of one of the most elegant optimization methods known to man — the Lagrange multiplier.

Now, it is well-known that the derivative test satisfies most unconstrained optimization problems. But what if there are constraints that need consideration as well? Enter: the Lagrange multiplier. The basic idea of this technique is to reduce the constrained problem to an unconstrained problem and apply the derivative test.

While the rigorous formulation allows even for multiple constraints, this article will only lend itself to a single constraint. I direct the interested reader here to learn about the Lagrange multiplier with multiple constraints.

The Single Constraint Affair

Before we begin, we need to introduce a concept that finds itself at the core of this technique: the Lagrangian function (referred to as the Lagrangian henceforth). In the general case, the Lagrangian is defined as:

where 〈∙,∙〉 is the inner product (or the analog of the dot product) and reduces simply to the algebraic product in the case of scalars. A summarization of the method is as follows:

In order to optimize a function, f(x), subjected to a constraint g(x)=0, find the stationary points of ℒ (considered as a function of x and λ). That is, all the partial derivatives should be zero:

The solution corresponding to the original constrained optimization is always a saddle point of the Lagrangian function, which can be identified among the stationary points from the definiteness of the bordered Hessian matrix. And our work is complete. While this seems all convoluted and difficult to understand right now, it will become incomparably easier and practically deployable after we treat an example.

The Lagrange Multiplier in Action

Suppose we wish to maximize f(x,y)=x+y, subjected to the constraint x²+y²=1. We will first graph this function, along with the constraint:

It is very evident from the graph that the minimum is -√2 and the maximum is √2. However, if we were to approach this analytically using the method of Lagrange multipliers, we would do so in the following way.

First, we must represent the constraint as a single function that can be equated to 0. Rearranging the constraint and substituting into the Lagrangian yields:

The Lagrangian is equivalent to f(x,y) if we set the constraint to 0. So, as a sanity check, we see that our work isn’t built on a mathematical fallacy.

Next, we must calculate the grad of the Lagrangian:

Upon equating the grad to the 0 vector, we see our work becomes nothing more than some trivial linear algebra:

It is important to note here that the last equation is simply nothing but the original constraint. Upon solving the first two simultaneous equations for x and y, we see that:

substituting this result into the original constraint, we see that

This implies that the stationary points of ℒ are

Evaluating f(x,y) at these points yields the two values √2 and -√2. Thus, we see that the function is maximized at √2 and minimized at -√2 under the constraint above. This is in accordance with our results from graphing the optimization problem and our mathematics is both right and legitimate.

With this, we reach this article’s conclusion. The interested reader can find elegant analytical and geometric proofs for the method of Lagrange multipliers here.

Thank you for reading and I hope you have a great day!