Column generation

Column generation or delayed column generation is an efficient algorithm for solving large linear programs.

The overarching idea is that many linear programs are too large to consider all the variables explicitly. The idea is thus to start by solving the considered program with only a subset of its variables. Then iteratively, variables that have the potential to improve the objective function are added to the program. Once it is possible to demonstrate that adding new variables would no longer improve the value of the objective function, the procedure stops. The hope when applying a column generation algorithm is that only a very small fraction of the variables will be generated. This hope is supported by the fact that in the optimal solution, most variables will be non-basic and assume a value of zero, so the optimal solution can be found without them.

In many cases, this method allows to solve large linear programs that would otherwise be intractable. The classical example of a problem where it is successfully used is the cutting stock problem. One particular technique in linear programming which uses this kind of approach is the Dantzig–Wolfe decomposition algorithm. Additionally, column generation has been applied to many problems such as crew scheduling, vehicle routing, and the capacitated p-median problem.

Algorithm

The algorithm considers two problems: the master problem and the subproblem. The master problem is the original problem with only a subset of variables being considered. The subproblem is a new problem created to identify an improving variable (i.e. which can improve the objective function of the master problem).

The algorithm then proceeds as follow:

1. Initialise the master problem and the subproblem
2. Solve the master problem
3. Search for an improving variable with the subproblem
4. If an improving variable is found: add it to the master problem then go to step 2
5. Else: The solution of the master problem is optimal. Stop.

Finding an improving variable

The most difficult part of this procedure is how to find a variable that can improve the objective function of the master problem. This can be done by finding the variable with the most negative reduced cost (assuming without loss of generality that the problem is a minimization problem). If no variable has a negative reduced cost, then the current solution of the master problem is optimal.

When the number of variables is very large, it is not possible to find an improving variable by calculating all the reduced cost and choosing a variable with a negative reduced cost. Thus, the idea is to compute only the variable having the minimum reduced cost. This can be done using an optimization problem called the pricing subproblem which strongly depends on the structure of the original problem. The objective function of the subproblem is the reduced cost of the searched variable with respect to the current dual variables, and the constraints require that the variable obeys the naturally occurring constraints. The column generation method is particularly efficient when this structure makes it possible to solve the sub-problem with an efficient algorithm, typically a dedicated combinatorial algorithm.

We now detail how and why to compute the reduced cost of the variables. Consider the following linear program in standard form:

${\begin{aligned}&\min _{x}c^{T}x\\&{\text{subject to}}\\&Ax=b\\&x\in \mathbb {R} ^{+}\end{aligned}}$

which we will call the primal problem as well as its dual linear program:

${\begin{aligned}&\max _{u}u^{T}b\\&{\text{subject to}}\\&u^{T}A\leq c\\&u\in \mathbb {R} \end{aligned}}$

Moreover, let $x^{*}$ and $u^{*}$ be optimal solutions for these two problems which can be provided by any linear solver. These solutions verify the constraints of their linear program and, by duality, have the same value of objective function ($c^{T}x^{*}=u^{*T}b$) which we will call $z^{*}$. This optimal value is a function of the different coefficients of the primal problem: $z^{*}=z^{*}(c,A,b)$. Note that there exists a dual variable $u_{i}^{*}$ for each constraint of the primal linear model. It is possible to show that an optimal dual variable $u_{i}^{*}$ can be interpreted as the partial derivative of the optimal value $z^{*}$ of the objective function with respect to the coefficient $b_{i}$ of the right-hand side of the constraints:$u_{i}^{*}={\frac {\partial z^{*}}{\partial b_{i}}}$ or otherwise $u^{*}={\frac {\partial z^{*}}{\partial b}}$. More simply put, $u_{i}^{*}$ indicates by how much increases locally the optimal value of the objective function when the coefficient $b_{i}$ increases by one unit.

Consider now that a variable $y$ was not considered until then in the primal problem. Note that this is equivalent to saying that the variable $y$ was present in the model but took a zero value. We will now observe the impact on the primal problem of changing the value of $y$ from $0$ to ${\hat {y}}$. If $c_{y}$ and $A_{y}$ are respectively the coefficients associated with the variable $y$ in the objective function and in the constraints then the linear program is modified as follows:

${\begin{aligned}&\min _{x}c^{T}x+c_{y}{\hat {y}}\\&{\text{subject to}}\\&Ax=b-A_{y}{\hat {y}}\\&x\in \mathbb {R} ^{+}\end{aligned}}$

In order to know if it is interesting to add the variable $y$ to the problem (i.e to let it take a non-zero value), we want to know if the value $z_{\hat {y}}^{*}$ of the objective function of this new problem decreases as the value ${\hat {y}}$ of the variable $y$ increases. In other words, we want to know ${\frac {\partial z_{\hat {y}}^{*}}{\partial {\hat {y}}}}$. To do this, note that $z_{\hat {y}}^{*}$ can be expressed according to the value of the objective function of the initial primal problem: $z_{\hat {y}}^{*}=c_{y}{\hat {y}}+z^{*}(c,A,b-A_{y}{\hat {y}})$. We can then compute the derivative that interests us:

${\begin{aligned}{\frac {\partial z_{\hat {y}}^{*}}{\partial {\hat {y}}}}&~=~&&c_{y}+{\frac {\partial z^{*}}{\partial {\hat {y}}}}\\&~=~&&c_{y}+{\frac {\partial z^{*}}{\partial c}}{\frac {dc}{d{\hat {y}}}}+{\frac {\partial z^{*}}{\partial A}}{\frac {dA}{d{\hat {y}}}}+{\frac {\partial z^{*}}{\partial b}}{\frac {db}{d{\hat {y}}}}\\&~=~&&c_{y}+{\frac {\partial z^{*}}{\partial b}}{\frac {db}{d{\hat {y}}}}\\&~=~&&c_{y}+u^{*}(-A_{y})\\&~=~&&c_{y}-u^{*}A_{y}\end{aligned}}$

In other words, the impact of changing the value ${\hat {y}}$ on the value $z_{\hat {y}}^{*}$ translates into two terms. First, this change directly impacts the objective function and second, the right-hand side of the constraints is modified which has an impact on the optimal variables $x^{*}$ whose magnitude is measured using the dual variables $u^{*}$. The derivative ${\frac {\partial z_{\hat {y}}^{*}}{\partial {\hat {y}}}}$ is generally called the reduced cost of the variable $y$ and will be denoted by $cr_{y}$ in the following.