An active-set algorithm for solving large-scale nonsmooth optimization models with box constraints

Yong Li; Gonglin Yuan; Zhou Sheng

doi:10.1371/journal.pone.0189290

Abstract

It is well known that the active set algorithm is very effective for smooth box constrained optimization. Many achievements have been obtained in this field. We extend the active set method to nonsmooth box constrained optimization problems, using the Moreau-Yosida regularization technique to make the objective function smooth. A limited memory BFGS method is introduced to decrease the workload of the computer. The presented algorithm has these properties: (1) all iterates are feasible and the sequence of objective functions is decreasing; (2) rapid changes in the active set are allowed; (3) the subproblem is a lower dimensional system of linear equations. The global convergence of the new method is established under suitable conditions and numerical results show that the method is effective for large-scale nonsmooth problems (5,000 variables).

Citation: Li Y, Yuan G, Sheng Z (2018) An active-set algorithm for solving large-scale nonsmooth optimization models with box constraints. PLoS ONE 13(1): e0189290. https://doi.org/10.1371/journal.pone.0189290

Editor: Marzio Alfio Pennisi, Universita degli Studi di Catania, ITALY

Received: December 18, 2016; Accepted: November 23, 2017; Published: January 2, 2018

Copyright: © 2018 Li et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper.

Funding: This work is supported by the National Natural Science Foundation of China (Grant No. 11661009 and 11261006), the Guangxi Science Fund for Distinguished Young Scholars (Grant No. 2015GXNSFGA139001), the Guangxi Fund of Young and Middle-aged Teachers for the Basic Ability Promotion Project(No. 2017KY0019) and the Guangxi Natural Science Key Fund (No. 2017GXNSFDA198046).

Competing interests: The authors have declared that no competing interests exist.

Introduction

Consider (1) where f: ℜⁿ → ℜ is a possibly nonsmooth convex function, K = {x ∣ l ≤ x ≤ u}, the vectors l and u represent lower and upper bounds on the variables, and n is the number of variables. Similar problems are discussed by Fukushima [1, 2], in which equality constraints are considered and a penalty strategy is used. The form of problem (1) can be viewed as an extension of the linearly constrained convex nonsmooth problem considered in, e.g., [3, 4] from linear to possibly nonlinear. In fact, many fields including finance, engineering, management, biology, and medicine can convert to the optimization models (1) (see [5–9] in detail).

Generally, nonsmooth problems are very difficult to solve even when they are unconstrained. Derivative-free methods, like Powell’s method [10] or genetic algorithms [11], may be unreliable and become inefficient whenever the dimension of the problem increases. The direct application of smooth gradient-based methods to nonsmooth problems may lead to a failure in optimality conditions, in convergence, or in gradient approximation [12]. Wolfe [13] and Lemaréchal [14] initiated a giant stride forward in nonsmooth optimization by the bundle concept. Kiwiel [15] proposed a bundle variant that is close to the bundle trust iteration method [16]. Some good results about the bundle technique can be found in [17–19] etc. At the moment, various versions of bundle methods are regarded as the most effective and reliable methods for nonsmooth optimization. Bundle methods are efficient for small- and medium-scale problems. This is explained by the fact that bundle methods need relatively large bundles to be capable of solving the problems efficiently [17]. Therefore, special tools for solving nonsmooth optimization problems are needed. At present, Haarala et al. (see [20, 21] etc.) introduce the limited memory bundle methods for large scale nonsmooth unconstrained and constrained minimization, which are a hybrid of the variable metric bundle methods and the limited memory variable metric methods and some good results are obtained. More related literature can be found in [22–26]. The test problems can have thousands of decision variables. Yuan et al. [27–31] make some studies where nonsmooth problems with the largest dimension 100,000 were solved in the unconstrained cases [28].

The active-set method can be generalized easily when the objective function is nonsmooth. For example, Sreedharan [32] extends the method developed in [33] to solve nonsmooth problems with a special objective function and inequality constraint. Also, it is quite easy to generalize the ε-active set method to the nondifferentiable case (see, e.g., [34]). In this paper we use the active-set method to solve (1) when the objective function f is convex but not necessarily differentiable. Convexity, which is not essential for our study, is assumed only for simplicity. For the objective function, we first use the Moreau-Yosida regularization technique to make it smooth. Then the active-set limited memory BFGS (L-BFGS) technique is proposed to solve it. Global convergence is established under suitable conditions. The main features of the proposed method are as follows.

The iterates are feasible; large changes are allowed in the active set; the subproblem has lower dimension; and the objective function sequence {f^MY (x_k, ε_k)} is decreasing.
The L-BFGS method uses function and gradient values.
Global convergence is established under suitable conditions.
Numerical results show that the method is effective for large-scale problems (up to 5,000 variables).

The paper is organized as follows. In the next section, we briefly review some nonsmooth analysis, a BFGS method and the L-BFGS method for unconstrained optimization, and the motivation for using these techniques. In Section 3, we describe the active-set algorithm with L-BFGS update for (1). In Section 4, global convergence is established under suitable conditions. Numerical results are reported in Section 5, and conclusions are given in the last section.

Nonsmooth analysis and the L-BFGS update

This section states some results on nonsmooth analysis, a modified BFGS formula, and a L-BFGS formula for unconstrained optimization problems.

Some results of convex analysis and nonsmooth analysis

Let f^MY: ℜⁿ → ℜ be the so-called Moreau-Yosida regularization of f defined by (2) where ‖⋅‖ denotes the Euclidean norm of vectors and λ is a positive parameter. Then it is not difficult to see that problem (1) is equivalent to the problem (3) The function f^MY is a differentiable convex function and has a Lipschitz continuous gradient even when f is nondifferentiable. Under some reasonable conditions, using the following properties of f^MY (x) and assuming ∇f^MY (x) is globally Lipschitz continuous, the gradient ∇f^MY (x) is semismooth (see [35, 36] etc.). By these properties, many algorithms have been given for solving (3) (see [37] etc.) when K = ℜⁿ. Some features of f^MY (x) can be seen in [38–40] et al. Set and denote p(x) = argmin θ(z). Since θ(z) is strongly convex, it is easy to deduce that p(x) is well-defined and unique. Then f^MY (x) in (2) can be rewritten as

The generalized Jacobian of f^MY (x) and the property of BD-regular can be found in [41, 42], respectively. Here some properties are listed without proof.

(i) The function f^MY is finite-valued, convex, and everywhere differentiable. If g(x) = ∇f^MY (x), then g: ℜⁿ → ℜⁿ is globally Lipschitz continuous: (4) where (5)

(ii) g is BD-regular at x means that all matrices V ∈ ∂_Bg(x) are nonsingular. Then there exist constants μ₁ > 0, μ₂ > 0 and a neighborhood Ω of x satisfying It is easy to find that p(x) of the minimizer for θ(z) is difficult or even impossible to solve exactly. Fortunately, for each x ∈ ℜⁿ and any ε > 0, there exists a vector p(x, ε) ∈ ℜⁿ satisfying (6) Thus, we can use p(x, ε) to define approximations of f^MY (x) and g(x) by (7) and (8) respectively. Some implementable algorithms to find such p(x, ε) for a nondifferentiable convex function are introduced in [43]. A remarkable feature of f^MY (x, ε) and g(x, ε) given by [35] is introduced, which show that, by choosing parameter ε small enough, we can compute approximations f^MY (x, ε) and g(x, ε) closing to f^MY (x) and g(x) respectively.

Proposition 1. Suppose that f^MY (x, ε) and g(x, ε) are defined by (7) and (8), respectively. Let p(x, ε) be a vector satisfying (6). Then (9) (10) and (11) hold.

A modified BFGS formula and the L-BFGS formula

The BFGS method is one of the most effective quasi-Newton methods for unconstrained optimization problems (UNP) min_x∈ℜⁿ h(x), where h(x): ℜⁿ → ℜ is continuously differentiable. The famous BFGS quasi-Newton formula is (12) where s_k = x_k+1 − x_k, y_k = ∇h(x_k+1) − ∇h(x_k), and it is easy to see that the quasi-Newton equation (13) holds. If H_k is the inverse of B_k, we get the inverse update formula of (12): (14) which is the dual form of the DFP update formula in the sense that H^k ↔ B^k, H^k+1 ↔ B^k+1, and s^k ↔ y^k. The L-BFGS method is an adaptation of the BFGS method to large-scale problems (see [44–46] in detail). Instead of storing the matrices H^k, at every iteration x_k the method stores a small number, say m, of correction pairs {s_i, y_i}, i = k − 1, …, k − m. Let and . The L-BFGS update has the form (15) which can provide a fast rate of linear convergence and requires minimal storage. From the BFGS formula (14) and the L-BFGS update (15), it is not difficult to find that both of these formulas contain only the gradient information of the objective function, while the function values available are neglected. Some modified quasi-Newton formulas using both gradient and function information are presented (e.g. [47, 48]). Wei et al. [49] also gave a new quasi-Newton equation where and the corresponding BFGS update formula is defined by (16) where . The quasi-Newton formula (16) contains both gradient and function information; moreover the modified BFGS update formula possesses a higher order approximation of ∇²h(x) than that of the standard BFGS update (see [47, 49] in detail).

Global convergence and superlinear convergence of the quasi-Newton method with (16) have been established for uniformly convex functions [47, 49], but fails for general convex functions. One of the main reasons lies in the condition that may not hold for general convex functions. To overcome the weaknesses, Yuan and Wei [50] presented a modified named . The idea of paper [50] is based on the following two cases:

Case i: It follows from A^k > 0 that (17)
Case ii: On the other hand, if A^k < 0, it is easy to get (18) where the second inequality follows the definition of the convexity of h(x), which means that holds. This modified BFGS formula with possesses global convergence and superlinear convergence for general convex functions. However, its applications in L-BFGS and nonsmooth optimization have not been widely studied.

This article will attempt to do this. The following gives the modified L-BFGS formula for (3) with form (19) where , , δ_k = g(x_k+1, ε_k+1) − g(x_k, ε_k), and . It is clear that the modified L-BFGS formula (19) contains both function and gradient information at the current and previous step if . In the following, the matrix H_k is generated by (19). This is very costly for even moderately large nonsmooth problems with box constraints, since the limited memory update is used to store , update it as a full matrix, reduce in the free subspace, and the set of active constraints changes at the first finite steps.

Inspired by the Moreau-Yosida regularization and the modified method of [50], we combine them with the limited memory technique, and use them to solve box constrained optimization with nonsmooth objective function. This paper can be regarded as an improvement of the method in [51] with extension to nonsmooth objective functions. Comparing with [51], at each step of our method, a lower-dimensional system of nonlinear equations and nonsmooth objective function needs to be solved. The method is also similar to the algorithm in [44], but, at each iteration, we use an identification technique and solve nonsmooth optimization problems.

L-BFGS active-set algorithm

The following assumptions are needed to obtain convergence.

Assumption A The level set ϕ = {x ∈ ℜⁿ ∣ f^MY (x) ≤ f^MY (x₀)} ⋂ K is compact.

Assumption B f^MY is bounded from below and the sequence {ε_k} converges to zero. We first solve (3) and adapt its solution to problem (1). With the feasible region K = {x ∈ ℜⁿ: l_i ≤ x_i ≤ u_i, i = 1, …, n}, a vector is said to be a stationary point for problem (3) if the relations (20) hold. Consider the relations (21) where the scalar tends to zero. By the definition of and , we have By (10), if we get Thus, if (21) holds, it is easy to deduce that (20) holds. In the following, without special note, we concentrate on the relation (21) and regard it as the stationary point condition. In the following, we always suppose that the point x_k is consistent with ε_k and is consistent with without special remark.

Similar to normal numerical optimization methods, the iteration formula is (22) where {x_k} ⊆ K = {x ∈ ℜⁿ: l_i ≤ xⁱ ≤ u_i, i = 1, …, n}, xⁱ is the ith element of x, d_k is a descent direction of f^MY at x_k, and α_k is a step length determined by the Armijo line search technique (23) where , α^k = 2⁻ⁱ with the smallest integer i = 0, 1, 2, …, and the sequence {ε_k} satisfies ε_k > ε_k+1 > 0. Before we give the direction definition, we introduce the procedure that estimates the active bounds. Suppose that is a stationary point of problem (1). Let the associated active constraint set be (24) and the set of free variables be Then condition (21) can be stated in the form (25) where is the ith element of . Let a_i and b_i be nonnegative continuous bounded from above on K, satisfying, if xⁱ = l_i or xⁱ = u_i then a_i(x) > 0 or b_i(x) > 0, respectively. Define the following approximation Υ(x), Γ(x) and Λ(x) to and , respectively: (26)

Theorem 1. For any feasible x, Υ(x) ∩ Λ(x) = ∅. Furthermore, if is a stationary point of problem (3) where strict complementarity holds, then there exists a neighborhood Ψ of such that for every feasible point x in this neighborhood we have (27)

Proof. For any feasible x, if k ∈ Υ(x), it is obvious that g_k(x, ε) ≥ 0 holds. Suppose that k ∈ Λ(x); then we have u_k ≥ x_k ≥ u_k + b_k(x)g_k(x, ε) ≥ u_k. This implies that l_k = x_k = u_k and g_k(x, ε) = 0, which is a contradiction. Thus Υ(x) ∩ Λ(x) = ∅.

Now we prove the second conclusion. If , then by the definition of , . Since a_i is nonnegative, then . Since both a_i and g_i are continuous in , we deduce that i ∈ Υ(x). Thus we have .

Otherwise if i ∈ Υ(x), then by the definition of Υ(x), a_i(x)g_i(x, ε) ≥ x_i − l_i ≥ 0. Since a_i is nonnegative, g_i(x, ε) ≥ 0. Since g_i is continuous in , we deduce that . Thus we get .

Therefore, we obtain . Analogously, we can conclude that and . □

This theorem proved that Υ(x), Γ(x) and Λ(x) are “good” estimates of and . The proof can also be found in [52].

The search direction is chosen as (28) where (29) with (30) and (31)Υ_k = Υ(x_k), Γ_k = Γ(x_k), Λ_k = Λ(x_k), is an approximation to the reduced inverse Hessian matrix, H_k is an approximation of the full space inverse Hessian matrix, Z is the matrix whose columns are {e_i ∣ i ∈ Γ_k}, and e_i is the ith column of the identity matrix in ℜ^n×n. If the strict complementarity condition holds, is a strict interior point of , and is always positive (see [53] in detail).

Based on the above discussions, we state our algorithm as follows.

Algorithm 1. (Act-L-BFGS-Alt-Non)

Step 0: Given x₀ ∈ Ψ, ε₀ ∈ (0, 1), and positive integer m, the “basic matrix” θI, set k = 0.

Step 1: Use (26) to determine Υ_k = Υ(x_k), Λ_k = Λ(x_k), and Γ_k = Γ(x_k).

Step 2: Compute d_k by (28).

Step 3: If d_k = 0, stop.

Step 4: Choose 0 < ε_k+1 < ε_k and α_k = 2⁻ⁱ, where i is the smallest integer of {0, 1, 2, …} such that the line search rule (23) holds.

Step 5: Let x_k+1 = x_k + α_kd_k and update H_k by (19).

Step 6: Set k ≔ k + 1 and go to Step 1.

Global convergence

In order to prove global convergence of Algorithm 1, the following further assumption is needed.

Assumption C. There exist positive scalars ς₁, ς₂ such that any sequence of matrices satisfy

The following lemma shows that d_k ≠ 0 is determined by (28) satisfies the sufficiently descent property.

Lemma 1. Suppose that d_k ≠ 0 is determined by (28) and x_k ∈ Ψ. Then the inequality (32) holds with a constant ω > 0.

Proof. We prove this result by three cases.

Case 1. i ∈ Υ_k. By x_k ∈ Ψ and we get implying a_i(x_k) > 0 and where A_i is an upper bound on a_i(x) in Ψ.

Case ii. i ∈ Λ_k. As in Case i, it is easy to get which means that b_i(x_k) > 0 and where B_i is an lower bound on b_i(x) in Ψ.

Case iii. i ∈ Γ_k. By (29), (30), and with symmetric positive definite, we obtain Then By Assumption C, we have . So we have Letting completes the proof. □

The following lemma is similar to [52], so we state it without proof.

Lemma 2. If the conditions in Lemma 1 hold, then x^k is a stationary point of (3) if and only if d^k = 0. Moreover, is a stationary point of problem (3) when the subsequences and {d_k}_K → 0 as k → ∞.

Now we establish the global convergence theorem of Algorithm 1.

Theorem 2. Let the sequence {x_k} be generated by Algorithm 1 under Assumptions A, B, and C. Then the sequence {x_k} at least has a limit point, and every limit point is a stationary point for problem (3). □

Proof. If d_k = 0, by Lemma 2, the theorem obviously holds. Suppose that d^k ≠ 0. By Lemma 1, (23), and Assumption B, we obtain which shows that the sequence {f^MY (x_k, ε_k)} is descending. So {x^k} has at least a limit point. Suppose that is a limit point of {x^k}. It is sufficient to prove that is a stationary point for problem (3). Lemma 2 means that we only show that the sequence {d_k} → 0. Without loss of generality, we suppose that and . By the property of limit, it is clear that is feasible.

By the feasibility of , Theorem 1, and the positive functions a_i(x) and b_i(x) with any possible choice, we have It follows from (27) that By (30), we get Thus . By Lemma 2, we deduce that is a stationary point for problem (3).

Remark. If the condition holds, by (11) it is not difficult to deduce that as . By the convexity of f^MY (x), the point is the optimal solution.

Numerical results

In this section, we test the numerical behavior of Algorithm 1. All codes were written in MATLAB 7.6.0 and run on a PC Core 2 Duo CPU, E7500 @2.93GHz with 2GB memory and Windows XP operating system.

Initialization

Our experiments are performed on a set of the nonlinear box-constrained nonsmooth problems from Karmitsa [54] which have given initial points. We choose σ = 0.1, a_i(x) = b_i(x) = 10⁻⁵ in (26), θ = 1 and the “basic matrix” to be the identity matrix I in the limited memory BFGS method, and m = 5. ε_k = 1/(NF + 1)²(NF is the function number). For subproblem (2), we use the PRP conjugate gradient algorithm, where the iteration number and the function number are added to the main program. Since the line search cannot always ensure the descent condition uphill search direction may occur in the numerical experiments. In this case, the line search rule may fail. In order to avoid it, the stepsize α_k is accepted if the search number is more than six in the line search. The following Himmelblau stopping rule isp used: If ∣ f^MY (x_k, ε_k) ∣ > 10⁻⁴, let ; Otherwise, let stop1 = ∣ f^MY (x_k, ε_k) − f^MY (x_k+1, ε_k+1) ∣. If stop1 < 10⁻⁴, the program stop. We also stop the program if the iteration number is more than 5000, and the corresponding method is considered to have failed.

Results

In this section, the test results of our algorithm for some box-constrained nonsmooth problems are reported. The columns of Table 1 have the following meaning:

Dim: the dimension of the problem; NI: the total number of iterations;

NF: the total number of function values; cpu: the cpu time in second;

: denotes the function value at the point when the program is stopped.

Download:

Table 1. Numerical results.

https://doi.org/10.1371/journal.pone.0189290.t001

The numerical results indicate that our algorithm is effective for these box constrained nonsmooth problems. The iteration number and function number do not change obviously with the increasing dimension. Problems Chained CB3 I and Chained CB3 II, and Chained Crescent I and Chained Crescent II, have many similar properties and have the same optimal values. From Table 1, we see that the final function value is close to the optimal value, especially for Problems Chained CB3 I and Chained CB3 II, and Chained Crescent I and Chained Crescent II, whose final function values are the same respectively, which shows that the presented method is stable. The cpu time is acceptable for this algorithm, although the iteration number is large for some problems. In the experiments, we find that different stopping rules influence the iteration number and the function number, but not the final function value.

To show the sequence of function values, we give the line chart graph (Figs 1, 2, 3 and 4) for problems Generalization of MAXQ (Fig 1), Chained LQ (Fig 2), Generalization of Brown function 2 (Fig 3), and Chained Crescent I (Fig 4) withp 5,000 variables. We see that the functions are descending. The descent property of the first two steps is very obvious, and these two steps make the function value close to the optimal value. However, the descent is not obvious for other steps. In our opinion, the reason is that the stopping rules are not ideal. Overall, the numerical performance of the proposed algorithm is reasonable for these large-scale nonsmooth problems. We conclude that the method provides a valid approach for solving large-scale box-constrained nonsmooth problems.

Download:

Fig 1. Generalization of MAXQ with 5,000 variables.

https://doi.org/10.1371/journal.pone.0189290.g001

Download:

Fig 2. Chained LQ with 5,000 variables.

https://doi.org/10.1371/journal.pone.0189290.g002

Download:

Fig 3. Generalization of Brown function with 5,000 variables.

https://doi.org/10.1371/journal.pone.0189290.g003

Download:

Fig 4. Chained Crescent I with 5,000 variables.

https://doi.org/10.1371/journal.pone.0189290.g004

Conclusion

In this paper, a modified L-BFGS method was presented for solving box constrained nonsmooth optimization problems. This method uses both gradient information and function values in the L-BFGS update formula. The proposed algorithm possesses global convergence.

(i) It is well known that nonsmooth problems are difficult to solve even when the objective function is unconstrained, especially for large-scale nonsmooth problems. To overcome this drawback, the Moreau-Yosida regularization technique is proposed to make the objective function smooth. Moreover, the L-BFGS method is introduced to reduce the computation and make the active-set algorithm suitable for solving large-scale nonsmooth problems.

(ii) The bundle method is one of the most effective methods for nonsmooth problems. However, its efficiencies are applied to small- and medium-scale problems. In order to find more effective methods for large-scale nonsmooth problems, the bundle L-BFGS algorithms are presented by many scholars, where the dimension can be 1,000 variables. In this paper, the given algorithm can successfully solve 1,000-5,000 variables nonsmooth problems with bound constraints.

(iii) In experiments, we find the different stopping rules influence the iteration numbers and the function numbers but not the final functions. Moreover, from Figs 1–4, we see that the first two iteration steps are the most effective, which shows that the proposed algorithm is effective for large-scale nonsmooth box constrained problems. In our opinion, the reason lies in the stopping criteria. Better rules should be found.

(iv) Considering the above discussions, we think there are at least four issues that could lead to improvements. The first that should be considered is the choice of the parameters in the active-set identification technique. The parameters used are not the only choice. Another important point that should be further investigated is the adoption of the gradient projection technique. The third is adjustment of the constant m in the L-BFGS update formula. The last is the most important one, from the numerical experiments, namely whether are there other optimality conditions and convergence conditions in the nonsmooth problems? We will study these aspects in our future works.

Although the proposed method does not obtain significant development that we expected, we feel that its performance is noticeable.

Acknowledgments

The authors are very grateful to the anonymous referees for their suggestions, which have helped improve the numerical results and presentation of the paper. This work is supported by the National Natural Science Foundation of China (Grant No. 11661009 and 11261006), the Guangxi Science Fund for Distinguished Young Scholars (Grant No. 2015GXNSFGA139001), the Guangxi Fund of Young and Middle-aged Teachers for the Basic Ability Promotion Project(No. 2017KY0019) and the Guangxi Natural Science Key Fund (No. 2017GXNSFDA198046).

References

1. Fukushima M. A successive quadratic programming method for a class of constrained nonsmooth optimization problems, Mathematical Programming, 49, 231–251 (1991)
- View Article
- Google Scholar
2. Yuan G., Wei Z., Zhang M. An active-set projected trust region algorithm for box constrained optimization problems. Journal of Systems Science and Complexity, 28, 1128–1147 (2015)
- View Article
- Google Scholar
3. Kiwiel K. C. An algorithm for linearly constrained convex nondifferentiable minimization problems, Journal of Mathematical Analysis and Applications, 105, 452–465 (1985)
- View Article
- Google Scholar
4. Panier E. R. An active set method for solving linearly constrained nonsmooth optimization problems, Mathematical Programming, 37, 269–292 (1987)
- View Article
- Google Scholar
5. Yuan G., Wei Z., Lu X. Global convergence of BFGS and PRP methods under a modified weak Wolfe-Powell line search, Applied Mathematical Modelling, 47, 811–825 (2017)
- View Article
- Google Scholar
6. Yuan G., Sheng Z., Wang B. et al. The global convergence of a modified BFGS method for nonconvex functions, Journal of Computational and Applied Mathematics, 327, 274–294 (2018)
- View Article
- Google Scholar
7. Xu C, Zhang J. A survey of quasi-Newton equations and quasi-Newton methods for optimization. Annals of Operations Research, 103(1–4), 213–234 (2001)
- View Article
- Google Scholar
8. Li G, Tang C, Wei Z. New conjugacy condition and related new conjugate gradient methods for unconstrained optimization. Journal of Computational and Applied Mathematics, 202(2), 523–539 (2007)
- View Article
- Google Scholar
9. Sheng Z., Yuan G., Cui Z. et al. An adaptive trust region algorithm for large residual nonsmooth least squares problems. Journal of Industrial and Management Optimization, (2017)
- View Article
- Google Scholar
10. Fletcher R. Practical Methods of Optimization, 2nd ed. John Wiley and Sons, Chichester, (1987)
11. Goldberg D. E. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading, MA, (1998)
12. Lemaréchal C. Nondifferentiable optimization. In Optimization, Nemhauser G. L., Rinnooy Kan A. H. G., and Todd M. J., Eds. Elsevier North-Holland, Inc., New York, 529–572, (1989)
13. Wolfe P. A method of conjugate subgradients for minimizing nondifferentiable convex functions, Mathematical Programming Study, 3, 145–173 (1975)
- View Article
- Google Scholar
14. Lemaréchal, C. Extensions diverses des méthodes de gradient et applications, Thèse d’Etat, Paris, (1980)
15. Kiwiel K. C. Proximity control in bundle methods for convex nondifferentiable optimization, Mathematical Programming, 46, 105–122 (1990)
- View Article
- Google Scholar
16. Schramm H., Zowe J. A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results, SIAM Journal on Optimization, 2, 121–152 (1992)
- View Article
- Google Scholar
17. Kiwiel K. C. Methods of descent for nondifferentiable optimization, lecture notes in Mathematics 1133, Springer-Verlag, Berlin, New York, (1985)
18. Kiwiel K. C. Proximal level bundle methods for convex nondifferentiable optimization, saddle-point problems and variational inequalities, Mathematical Programming, 69, 89–109 (1995)
- View Article
- Google Scholar
19. Schramm H. Eine kombination yon bundle-und trust-region-verfahren zur Lösung nicht- differenzierbare optimierungsprobleme, Bayreuther Mathematische Schriften, Heft 30, Bayreuth, Germany, (1989)
20. Haarala M., Mäkelä M. M. Limited memory bundle algorithm for large bound constrained nonsmooth minimization problems, Reports of the Department of Mathematical Information Technology, Series b. Scientific Computing, No. B. 1/2006, University of Jyväskylä, Finland, (2006)
21. Haarala M., Miettinen K., Mäkelä M. M. New limited memory bundle method for large-scale nonsmooth optimization, Optimization Methods and Software, 19, 673–692 (2004)
- View Article
- Google Scholar
22. Floudas C. A., Pardalos P. M. Encyclopedia of optimization, Springer Science & Business Media, (2001)
23. Karmitsa N. Limited memory bundle method for large bound constrained nonsmooth optimization, Proceedings of the International Conference on Engineering Optimization, Rio de Janeiro, (2008)
24. Karmitsa N., Mäkelä M. M. Limited memory bundle method for large bound constrained nonsmooth optimization: convergence analysis, Optimization Methods & Software, 25(6), 895–916 (2010)
- View Article
- Google Scholar
25. Karmitsa N., Mäkelä M. M. Adaptive limited memory bundle method for bound constrained large-scale nonsmooth optimization, Optimization, 59(6), 945–962 (2010)
- View Article
- Google Scholar
26. Demyanov V. F. Constructive Nonsmooth Analysis and Related Topics, New York: Springer, (2014)
27. Yuan G., Meng Z., Li Y. A modified Hestenes and Stiefel conjugate gradient algorithm for large-scale nonsmooth minimizations and nonlinear equations, Journal Optimization Theory and Applications, 168, 129–152 (2016)
- View Article
- Google Scholar
28. Yuan G., Sheng Z., Liu W., The modified HZ conjugate gradient algorithm for large-scale nonsmooth optimization, PLoS ONE, 11, 1–15 (2016)
- View Article
- Google Scholar
29. Yuan G., Wei Z. The Barzilai and Borwein gradient method with nonmonotone line search for nonsmooth convex optimization problems, Mathematical Modelling and Analysis, 17, 203–216 (2012)
- View Article
- Google Scholar
30. Yuan G., Wei Z., Li G. A modified Polak-Ribière-Polyak conjugate gradient algorithm with nonmonotone line search for nonsmooth convex minimization, Journal of Computational and Applied Mathematics, 255, 86–96 (2014)
- View Article
- Google Scholar
31. Yuan G., Wei Z., Wang Z. Gradient trust region algorithm with limited memory BFGS update for nonsmooth convex minimization, Computational Optimization and Applications, 54,45–64 (2013)
- View Article
- Google Scholar
32. Sreedharan V.P. A subgradient projection algorithm, Journal of Approximation Theory, 35, 111–126 (1982)
- View Article
- Google Scholar
33. Schultz H. K. A Kuhn-Tucker algorithm, SIAM Journal on Control and Optimization, 11, 438–445 (1973)
- View Article
- Google Scholar
34. Nguyen V. H., Strodiot J. J. A linearly constrained algorithm not requiring derivative continuity, Engineering Structures, 6, 7–11 (1984)
- View Article
- Google Scholar
35. Fukushima M., Qi L. A global and superlinearly convergent algorithm for nonsmooth convex minimization, SIAM Journal on Optimization, 6, 1106–1120 (1996)
- View Article
- Google Scholar
36. Qi L., Sun J. A nonsmooth version of Newton’s method, Mathematical Programming, 58, 353–367 (1993)
- View Article
- Google Scholar
37. Birge J. R., Qi L. and Wei Z. A general approach to convergence properties of some methods for nonsmooth convex optimization, Applied Mathematics & Optimization, 38, 141–158 (1998)
- View Article
- Google Scholar
38. Bonnans J. F., Gilbert J. C., Lemaréchal C., and Sagastizábal C. A. A family of veriable metric proximal methods, Mathematical Programming, 68, 15–47 (1995)
- View Article
- Google Scholar
39. Correa R. and Lemaréchal C. Convergence of some algorithms for convex minimization, Mathematical Programming, 62, 261–273 (1993)
- View Article
- Google Scholar
40. Hiriart-Urruty J. B., Lemmaréchal C. Convex analysis and minimization algorithms II, Spring-Verlay, Berlin, Heidelberg, (1983)
41. Calamai P., Moré J. J. Projected gradient for linearly constrained programms, Mathematical Programming, 39, 93–116 (1987)
- View Article
- Google Scholar
42. Qi L. Convergence analysis of some algorithms for solving nonsmooth equations, Mathematics of Operations Research, 18, 227–245 (1993)
- View Article
- Google Scholar
43. Fukushima M. A descent algorithm for nonsmooth convex optimization. Mathematical Programming, 30(2), 163–175 (1984)
- View Article
- Google Scholar
44. Ryrd R. H., Lu P. H., Nocedal J., Zhu C. Y. A limited memory algorithm for bound constrained optimization, SIAM Journal on Scientific Computing, 16, 1190–1208 (1995)
- View Article
- Google Scholar
45. Byrd R. H., Nocedal J., Schnabel R. B., Representations of quasi-Newton matrices and their use in limited memory methods, Mathematical Programming, 63, 129–156 (1994)
- View Article
- Google Scholar
46. Powell M.J.D., A fast algorithm for nonlinearly constrained optimization calculations, Numer. Ana., 155–157 (1978)
47. Wei Z., Li G., Qi L. New Quasi-Newton Methods for unconstrained optimization problems, Applied Mathematics and Computation, 175, 1156–1188 (2006)
- View Article
- Google Scholar
48. Zhang J. Z., Deng N. Y., Chen L. H. New quasi-Newton equation and related methods for unconstrained optimization, Journal Optimization Theory and Applications, 102, 147–167 (1999)
- View Article
- Google Scholar
49. Wei Z., Yu G., Yuan G., Lian Z. The superlinear convergence of a modified BFGS-type method for unconstrained optimization, Computational Optimization and Applications, 29, 315–332 (2004)
- View Article
- Google Scholar
50. Yuan G., Wei Z. Convergence analysis of a modified BFGS method on convex minimizations, Computational Optimization and Applications, 47, 237–255 (2010)
- View Article
- Google Scholar
51. Facchinei F., Júdice J., An active set Newton algorithm for large-scale nonlinear programs with box constraints, SIAM Journal on Optimization, 8, 158–186 (1998)
- View Article
- Google Scholar
52. Yuan G., Lu X. An active set limited memory BFGS algorithm for bound constrained optimization, Applied Mathematical Modelling, 35, 3561–3573 (2011)
- View Article
- Google Scholar
53. Xiao Y., Wei Z. A new subspace limited memory BFGS algorithm for large-scale bound constrained optimization, Applied Mathematics and Computation, 185, 350–359 (2007)
- View Article
- Google Scholar
54. Karmitsa N. Test problems for large-scale nonsmooth minimization, Reports of the Department of Mathematical Information Technology, Series B. Scientific Computing, No. B. 4/2007, University of Jyväskylä, Finland, (2007)

[ref1] 1. Fukushima M. A successive quadratic programming method for a class of constrained nonsmooth optimization problems, Mathematical Programming, 49, 231–251 (1991)
View Article
Google Scholar

[2] View Article

[3] Google Scholar

[ref2] 2. Yuan G., Wei Z., Zhang M. An active-set projected trust region algorithm for box constrained optimization problems. Journal of Systems Science and Complexity, 28, 1128–1147 (2015)
View Article
Google Scholar

[5] View Article

[6] Google Scholar

[ref3] 3. Kiwiel K. C. An algorithm for linearly constrained convex nondifferentiable minimization problems, Journal of Mathematical Analysis and Applications, 105, 452–465 (1985)
View Article
Google Scholar

[8] View Article

[9] Google Scholar

[ref4] 4. Panier E. R. An active set method for solving linearly constrained nonsmooth optimization problems, Mathematical Programming, 37, 269–292 (1987)
View Article
Google Scholar

[11] View Article

[12] Google Scholar

[ref5] 5. Yuan G., Wei Z., Lu X. Global convergence of BFGS and PRP methods under a modified weak Wolfe-Powell line search, Applied Mathematical Modelling, 47, 811–825 (2017)
View Article
Google Scholar

[14] View Article

[15] Google Scholar

[ref6] 6. Yuan G., Sheng Z., Wang B. et al. The global convergence of a modified BFGS method for nonconvex functions, Journal of Computational and Applied Mathematics, 327, 274–294 (2018)
View Article
Google Scholar

[17] View Article

[18] Google Scholar

[ref7] 7. Xu C, Zhang J. A survey of quasi-Newton equations and quasi-Newton methods for optimization. Annals of Operations Research, 103(1–4), 213–234 (2001)
View Article
Google Scholar

[20] View Article

[21] Google Scholar

[ref8] 8. Li G, Tang C, Wei Z. New conjugacy condition and related new conjugate gradient methods for unconstrained optimization. Journal of Computational and Applied Mathematics, 202(2), 523–539 (2007)
View Article
Google Scholar

[23] View Article

[24] Google Scholar

[ref9] 9. Sheng Z., Yuan G., Cui Z. et al. An adaptive trust region algorithm for large residual nonsmooth least squares problems. Journal of Industrial and Management Optimization, (2017)
View Article
Google Scholar

[26] View Article

[27] Google Scholar

[ref10] 10. Fletcher R. Practical Methods of Optimization, 2nd ed. John Wiley and Sons, Chichester, (1987)

[ref11] 11. Goldberg D. E. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading, MA, (1998)

[ref12] 12. Lemaréchal C. Nondifferentiable optimization. In Optimization, Nemhauser G. L., Rinnooy Kan A. H. G., and Todd M. J., Eds. Elsevier North-Holland, Inc., New York, 529–572, (1989)

[ref13] 13. Wolfe P. A method of conjugate subgradients for minimizing nondifferentiable convex functions, Mathematical Programming Study, 3, 145–173 (1975)
View Article
Google Scholar

[32] View Article

[33] Google Scholar

[ref14] 14. Lemaréchal, C. Extensions diverses des méthodes de gradient et applications, Thèse d’Etat, Paris, (1980)

[ref15] 15. Kiwiel K. C. Proximity control in bundle methods for convex nondifferentiable optimization, Mathematical Programming, 46, 105–122 (1990)
View Article
Google Scholar

[36] View Article

[37] Google Scholar

[ref16] 16. Schramm H., Zowe J. A version of the bundle idea for minimizing a nonsmooth function: conceptual idea, convergence analysis, numerical results, SIAM Journal on Optimization, 2, 121–152 (1992)
View Article
Google Scholar

[39] View Article

[40] Google Scholar

[ref17] 17. Kiwiel K. C. Methods of descent for nondifferentiable optimization, lecture notes in Mathematics 1133, Springer-Verlag, Berlin, New York, (1985)

[ref18] 18. Kiwiel K. C. Proximal level bundle methods for convex nondifferentiable optimization, saddle-point problems and variational inequalities, Mathematical Programming, 69, 89–109 (1995)
View Article
Google Scholar

[43] View Article

[44] Google Scholar

[ref19] 19. Schramm H. Eine kombination yon bundle-und trust-region-verfahren zur Lösung nicht- differenzierbare optimierungsprobleme, Bayreuther Mathematische Schriften, Heft 30, Bayreuth, Germany, (1989)

[ref20] 20. Haarala M., Mäkelä M. M. Limited memory bundle algorithm for large bound constrained nonsmooth minimization problems, Reports of the Department of Mathematical Information Technology, Series b. Scientific Computing, No. B. 1/2006, University of Jyväskylä, Finland, (2006)

[ref21] 21. Haarala M., Miettinen K., Mäkelä M. M. New limited memory bundle method for large-scale nonsmooth optimization, Optimization Methods and Software, 19, 673–692 (2004)
View Article
Google Scholar

[48] View Article

[49] Google Scholar

[ref22] 22. Floudas C. A., Pardalos P. M. Encyclopedia of optimization, Springer Science & Business Media, (2001)

[ref23] 23. Karmitsa N. Limited memory bundle method for large bound constrained nonsmooth optimization, Proceedings of the International Conference on Engineering Optimization, Rio de Janeiro, (2008)

[ref24] 24. Karmitsa N., Mäkelä M. M. Limited memory bundle method for large bound constrained nonsmooth optimization: convergence analysis, Optimization Methods & Software, 25(6), 895–916 (2010)
View Article
Google Scholar

[53] View Article

[54] Google Scholar

[ref25] 25. Karmitsa N., Mäkelä M. M. Adaptive limited memory bundle method for bound constrained large-scale nonsmooth optimization, Optimization, 59(6), 945–962 (2010)
View Article
Google Scholar

[56] View Article

[57] Google Scholar

[ref26] 26. Demyanov V. F. Constructive Nonsmooth Analysis and Related Topics, New York: Springer, (2014)

[ref27] 27. Yuan G., Meng Z., Li Y. A modified Hestenes and Stiefel conjugate gradient algorithm for large-scale nonsmooth minimizations and nonlinear equations, Journal Optimization Theory and Applications, 168, 129–152 (2016)
View Article
Google Scholar

[60] View Article

[61] Google Scholar

[ref28] 28. Yuan G., Sheng Z., Liu W., The modified HZ conjugate gradient algorithm for large-scale nonsmooth optimization, PLoS ONE, 11, 1–15 (2016)
View Article
Google Scholar

[63] View Article

[64] Google Scholar

[ref29] 29. Yuan G., Wei Z. The Barzilai and Borwein gradient method with nonmonotone line search for nonsmooth convex optimization problems, Mathematical Modelling and Analysis, 17, 203–216 (2012)
View Article
Google Scholar

[66] View Article

[67] Google Scholar

[ref30] 30. Yuan G., Wei Z., Li G. A modified Polak-Ribière-Polyak conjugate gradient algorithm with nonmonotone line search for nonsmooth convex minimization, Journal of Computational and Applied Mathematics, 255, 86–96 (2014)
View Article
Google Scholar

[69] View Article

[70] Google Scholar

[ref31] 31. Yuan G., Wei Z., Wang Z. Gradient trust region algorithm with limited memory BFGS update for nonsmooth convex minimization, Computational Optimization and Applications, 54,45–64 (2013)
View Article
Google Scholar

[72] View Article

[73] Google Scholar

[ref32] 32. Sreedharan V.P. A subgradient projection algorithm, Journal of Approximation Theory, 35, 111–126 (1982)
View Article
Google Scholar

[75] View Article

[76] Google Scholar

[ref33] 33. Schultz H. K. A Kuhn-Tucker algorithm, SIAM Journal on Control and Optimization, 11, 438–445 (1973)
View Article
Google Scholar

[78] View Article

[79] Google Scholar

[ref34] 34. Nguyen V. H., Strodiot J. J. A linearly constrained algorithm not requiring derivative continuity, Engineering Structures, 6, 7–11 (1984)
View Article
Google Scholar

[81] View Article

[82] Google Scholar

[ref35] 35. Fukushima M., Qi L. A global and superlinearly convergent algorithm for nonsmooth convex minimization, SIAM Journal on Optimization, 6, 1106–1120 (1996)
View Article
Google Scholar

[84] View Article

[85] Google Scholar

[ref36] 36. Qi L., Sun J. A nonsmooth version of Newton’s method, Mathematical Programming, 58, 353–367 (1993)
View Article
Google Scholar

[87] View Article

[88] Google Scholar

[ref37] 37. Birge J. R., Qi L. and Wei Z. A general approach to convergence properties of some methods for nonsmooth convex optimization, Applied Mathematics & Optimization, 38, 141–158 (1998)
View Article
Google Scholar

[90] View Article

[91] Google Scholar

[ref38] 38. Bonnans J. F., Gilbert J. C., Lemaréchal C., and Sagastizábal C. A. A family of veriable metric proximal methods, Mathematical Programming, 68, 15–47 (1995)
View Article
Google Scholar

[93] View Article

[94] Google Scholar

[ref39] 39. Correa R. and Lemaréchal C. Convergence of some algorithms for convex minimization, Mathematical Programming, 62, 261–273 (1993)
View Article
Google Scholar

[96] View Article

[97] Google Scholar

[ref40] 40. Hiriart-Urruty J. B., Lemmaréchal C. Convex analysis and minimization algorithms II, Spring-Verlay, Berlin, Heidelberg, (1983)

[ref41] 41. Calamai P., Moré J. J. Projected gradient for linearly constrained programms, Mathematical Programming, 39, 93–116 (1987)
View Article
Google Scholar

[100] View Article

[101] Google Scholar

[ref42] 42. Qi L. Convergence analysis of some algorithms for solving nonsmooth equations, Mathematics of Operations Research, 18, 227–245 (1993)
View Article
Google Scholar

[103] View Article

[104] Google Scholar

[ref43] 43. Fukushima M. A descent algorithm for nonsmooth convex optimization. Mathematical Programming, 30(2), 163–175 (1984)
View Article
Google Scholar

[106] View Article

[107] Google Scholar

[ref44] 44. Ryrd R. H., Lu P. H., Nocedal J., Zhu C. Y. A limited memory algorithm for bound constrained optimization, SIAM Journal on Scientific Computing, 16, 1190–1208 (1995)
View Article
Google Scholar

[109] View Article

[110] Google Scholar

[ref45] 45. Byrd R. H., Nocedal J., Schnabel R. B., Representations of quasi-Newton matrices and their use in limited memory methods, Mathematical Programming, 63, 129–156 (1994)
View Article
Google Scholar

[112] View Article

[113] Google Scholar

[ref46] 46. Powell M.J.D., A fast algorithm for nonlinearly constrained optimization calculations, Numer. Ana., 155–157 (1978)

[ref47] 47. Wei Z., Li G., Qi L. New Quasi-Newton Methods for unconstrained optimization problems, Applied Mathematics and Computation, 175, 1156–1188 (2006)
View Article
Google Scholar

[116] View Article

[117] Google Scholar

[ref48] 48. Zhang J. Z., Deng N. Y., Chen L. H. New quasi-Newton equation and related methods for unconstrained optimization, Journal Optimization Theory and Applications, 102, 147–167 (1999)
View Article
Google Scholar

[119] View Article

[120] Google Scholar

[ref49] 49. Wei Z., Yu G., Yuan G., Lian Z. The superlinear convergence of a modified BFGS-type method for unconstrained optimization, Computational Optimization and Applications, 29, 315–332 (2004)
View Article
Google Scholar

[122] View Article

[123] Google Scholar

[ref50] 50. Yuan G., Wei Z. Convergence analysis of a modified BFGS method on convex minimizations, Computational Optimization and Applications, 47, 237–255 (2010)
View Article
Google Scholar

[125] View Article

[126] Google Scholar

[ref51] 51. Facchinei F., Júdice J., An active set Newton algorithm for large-scale nonlinear programs with box constraints, SIAM Journal on Optimization, 8, 158–186 (1998)
View Article
Google Scholar

[128] View Article

[129] Google Scholar

[ref52] 52. Yuan G., Lu X. An active set limited memory BFGS algorithm for bound constrained optimization, Applied Mathematical Modelling, 35, 3561–3573 (2011)
View Article
Google Scholar

[131] View Article

[132] Google Scholar

[ref53] 53. Xiao Y., Wei Z. A new subspace limited memory BFGS algorithm for large-scale bound constrained optimization, Applied Mathematics and Computation, 185, 350–359 (2007)
View Article
Google Scholar

[134] View Article

[135] Google Scholar

[ref54] 54. Karmitsa N. Test problems for large-scale nonsmooth minimization, Reports of the Department of Mathematical Information Technology, Series B. Scientific Computing, No. B. 4/2007, University of Jyväskylä, Finland, (2007)

Figures

Abstract

Introduction

Nonsmooth analysis and the L-BFGS update

Some results of convex analysis and nonsmooth analysis

A modified BFGS formula and the L-BFGS formula

L-BFGS active-set algorithm

Global convergence

Numerical results

Initialization

Results

Conclusion

Acknowledgments

References