1 Introduction

The interest in this paper is solution of the constrained system of nonlinear equations

(1)

where is a nonempty closed convex set and F is a continuous vector-valued mapping, which is also monotone, namely, it satisfies

(2)

Real life applications of (1) can be found in studies such as general equilibrium problems in [1,2], the control systems analyzed in [3], networking and communications [4], modelling and data estimation [5], and in compressed sensing problems [6–14]. Some of the methods employed in solving (1) include adaptive versions of Newton’s method and its enhanced version, the quasi-Newton (QN) schemes [15–18], the derivative-free methods [19–24], Levenberg-Marquardt method [25], trust-region method [26] as well as spectral projection methods [27,28]. Despite their fast convergence properties, adaptations of Newton and QN methods have not been widely studied and few of the schemes for solving (1) have been developed. A viable reason for this is the huge memory required at each iteration to store matrices generated by the methods. The classical Newton and QN methods were developed to solve the optimization problem

(3)

with f being a smooth real-valued nonlinear function, with gradient . The sequence of iterates generated by these methods are obtained via the formula

where represents a step-size that is usually calculated using a line search procedure, and is search direction that is computed via

in Newton’s method, or

(4)

in QN method, where is a matrix known as the true Hessian or its approximation B_k in the QN method. The first QN update was developed by Davidon [29] with subsequent analysis by Fletcher and Powell [30], hence the name DFP after the three authors. The update is defined by

(5)

in which , . By setting H_k to be the identity matrix, the memoryless version of (6) is obtained, namely

A DFP scheme with the update (6) exhibit some nice properties, which includes quadratic termination, hereditary property, and generating conjugate directions for a quadratic objective function and an exact line search procedure. It retains positive-definiteness for general nonlinear functions, when the curvature condition is satisfied. Iterations of the scheme require and it has super-linear rate of convergence. Also, for strictly convex functions under the exact line search rule, the DFP scheme converges globally [31]. Notwithstanding these attributes, numerical studies and computations conducted have shown that the DFP update is usually unstable. This is due to the loss of positive-definiteness along some iterative points as well as the distribution of its eigenvalues. The Broyden [32], Fletcher [33], Goldfarb [34], and Shanno [35] (BFGS) update is the most prominent QN scheme available. Being a member of the Broyden family, the BFGS update is given by

(6)

where s_k and y_k remain as defined in (6). For the purpose of practical computations, search directions of the QN methods (5) are usually expressed in closed form as

(7)

where denotes the inverse approximation to the Hessian of the BFGS method. By applying the popular Woodbury formula [36] twice on (8), the inverse is obtained as

(8)

In order to avoid computing the matrix H_k, which involves huge storage at each iteration, it is replaced with the identity matrix I and the so called memoryless update is obtained, i.e.,

(9)

Further analysis of (8) by Andrei [37] shows that efficiency of the BFGS method depends so much on structure of its eigenvalues. It was also shown by Powell [38] and Byrd et al. [39] that the BFGS method suffers more from large eigenvalues of (8) than small ones. By conducting some numerical experiments on the BFGS method, Gill and Leonard [40] discovered that large number of iterations or functions and gradient evaluations may be required on some problems for the scheme to converge. These shortcomings of the scheme were shown in [40] to mostly emanate from poor initial Hessian approximations or ill-conditioning along the iterations. To address these problems, some scaling procedures were applied on the update in (8). By scaling the third term in (8) with a positive parameter , Biggs [41] gave the modification

Oren and Luenberger [42] scaled the first two terms of the update in (8) to obtain

where is a positive parameter. By adopting the technique of changing structure of eigenvalues Andrei [43] presented a two-parameter BFGS method, with B_k+1 given as

with and being positive parameters. Here, is computed to ensure that eigenvalues of B_k+1 are clustered, while is obtained so that large eigenvalues of B_k+1 are reduced, i.e., the eigenvalues are shifted to the left. This strategy brings about a better distribution of the eigenvalues. A modification of (11) was proposed in [44], namely

(10)

where and are parameters that are determined by employing Byrd and Nocedal’s measure function in [45]. The method proves to be computationally cheap and also has superlinear rate of convergence on practical problems.

In order to address some of the shortfalls of the DFP (6) and BFGS (8) updates and address some of their shortfalls, some variants for solving problem (3) have been proposed in the literature. Based on the matrix norm and an upper bound for condition number of the scaled version of (11), Babaie-Kafaki [46] proposed a BFGS-type method for the unconstrained optimization problem (3). The author proved that the method satisfy the descent condition and conducted numerical experiments to show its effectiveness. Also, Babaie-Kafaki and Aminifard [47] proposed two parameter scaled memoryless BFGS update in (10) for the problem (3) with a monotone choice for the initial step length. The authors proved the sufficient descent condition of the scheme for uniformly convex functions. Aminifard and Babaie-Kafaki [48] developed a DFP-type method for solving (3) by analyzing eigenvalues of a scaled memoryless variant of the DFP update (6), where the scaled parameter is obtained by skipping the direction of maximum magnification. By exploiting a spectral secant condition for the classical DFP update (6), Dargahi et al. [49] proposed a DFP-type method for (3), where the updating matrix is obtained such that its condition number is minimized. Bakhshinejad and Babaie-Kafaki [50] also proposed a DFP-type algorithm as an extension of the search direction obtained by applying the DFP update (6). The method was shown to satisfy the much required sufficient descent condition.

In order to exploit some of the advantages of the DFP and BFGS updates and address their shortcomings, some adaptive versions for solving (1) and its unconstrained form have been proposed in recent years. In line with this, Mamat et al. [51] proposed a DFP-type algorithm for solving the symmetric form of problem (1) by employing a scaled memoryless version of the DFP update (6). The authors proved global convergence of the scheme by employing some basic assumptions. As an enhancement of the classical DFP update (6), Dauda [52] proposed a three-term DFP-type method for solving the symmetric form of (1). The authors proved its global convergence using some mild conditions. Based on a scaled memoryless version of the DFP update (6), Awwal et al. [53] proposed a derivative-free method for solving (1). The authors proved global convergence of the method and compared its performance with some methods for solving (1). Inspired by a scaled version of the memoryless DFP update (6), another derivative-free method was proposed by Rao and Huang [54] for the unconstrained form of (1). Global convergence of the method was shown by employing some mild conditions. By applying a scaling technique and the measure function of Byrd and Nocedal [45] on a scaled memoryless DFP update (6), Ullah et al. [55] proposed a derivative-free DFP-type method for the constrained problem (1). Both global convergence and numerical efficiency were shown by the authors. In a recent development, Rehman et al. [56] proposed another version of a derivative-free DFP method for solving (1). The method converges globally and its effectiveness was illustrated by comparison with some recent methods in the literature. Recently, by computing an optimal value for the scaled memoryless BFGS update (11), Shah et al. [57] proposed a scaled derivative-free BFGS method for solving (1). The authors proved global convergence of the scheme and demonstrated its application to motion control. For more recent methods for solving (1), the reader can refer to works in [58–65]. The objectives of this paper are as follows:

• To develop a QN-type algorithm close to the BFGS method, for solving problem (1), which avoids the explicit computation of any matrix in the algorithm.
• To develop a scheme that exhibits some nice attributes of the classical BFGS update, addresses some of its shortcomings, and which possess the vital property for analyzing the convergence for problem (1).
• To develop a globally convergent algorithm without the Lipschitz condition and to analyze its numerical performance in solving problem (1).

The paper is structured as follows: Motivation and details of the method are given in the following section. Analysis of its global convergence is given in Section 3. Results of some numerical experiments of the scheme and some recent methods in the literature are presented in Section 4. Conclusions are given in Section 5.

2 An adaptive QN-type method

An adaptive QN-type scheme will be derived in this section. To achieve that, the following lemma is imperative:

Lemma 2.1. Cauchy Schwarz inequality

Let be an inner product space over the field of complex numbers with inner product . Then, for every pair of vectors , the following inequality holds.

(11)

As stated in the introduction section, adaptations of the iterative methods for the problem in (3) have been developed over the years. However, to the best of our knowledge few QN-type schemes exists in the literature. The ones available include the derivative-free DFP-type method proposed by Awwal et al. [53], where the authors scaled the first term of the classical DFP update (6) with a positive parameter to present

(12)

where represents a positive parameter that is computed such that the condition for analyzing global convergence holds for the scheme. Setting in (14) yields

(13)

where , is a scaling parameter. By substituting (15) into (9), the authors obtained the search direction

with , . The choice of suggested by the authors only guarantees the condition for analyzing global convergence holds, it does not ensure stability of the proposed method. Following the work by Awwal et al. [53], Ullah et al. [55] scaled the third term of the memoryless version of the DFP update (6) with a positive parameter to obtain

(14)

where ,which is derived by employing the measure function of Nocedal and Byrd [45] is obtained as . Using this value of , the authors presented search direction of the scheme as

(15)

The value of the scaling parameter obtained reduces the chances of the DFP updating matrix from being ill-conditioned. Still, it does not ensure that the search direction (18) satisfies the condition for analyzing global convergence. Following this, Shah et. al. [57] proposed another QN-type method for solving (1) by scaling the first and second term of the memoryless version of the Hessian approximation of the BFGS update [66], namely

(16)

The authors minimized the condition number of (19) to obtain the choice of as resulting in the search direction

In (19), the parameter does not always ensure stability of the BFGS method as the third term on the right may grow larger producing a shift of the eigenvalues to the right. In a recent attempt to address the shortfalls of the DFP update (6), Rehman et. al. [56] scaled the first and second terms of the memoryless DFP update (6), namely

(17)

with being a positive parameter whose optimal value was obtained as the minimizer of the condition number of H_k+1 which is given as

(18)

By substituting (22) into the memoryless DFP update (21), the authors obtained

(19)

and the search direction

Now, trace of the matrix H_k+1 in (23) can be computed as

(20)

It can be seen that the third-term on the right of (25) will shift the eigenvalues of H_k+1 towards the right making the matrix to have large eigenvalues. On the other hand, the second term being negative tends to shift the eigenvalues to the left. Even though this effect is countered by the first term, which is positive, it does not ensure proper distribution of the eigenvalues of H_k+1, and by extension stability of the scheme is not guaranteed.

Remark 2.2 As explained above, while the choices of the parameters and in (15), (17), (19), and (21) may enable the scheme generated with the corresponding matrix H_k+1 to satisfy the vital condition for analyzing global convergence, they do not ensure its stability – a condition attributed to uneven distribution of eigenvalues of H_k+1. In addition to this, like most of the published works in the literature for the constrained problem (1), the methods in [53–56] depend on the Lipschitz condition for proving their global convergence. And since not all problems satisfy the Lipschitz condition, the number of problems analyzed tend to be limited.

Now, by replacing s_k with , y_k with F_k−1, B_k+1 with Q_k in (8), and motivated by the scaling strategy in (12) and the above remark, we present the following two parameter scaled symmetric rank-two quasi-Newton type update,

(21)

in which and

(22)

Setting Q_k−1 as the identity matrix yields the following scaled memoryless version of (26):

(23)

Now, in order to define search direction of the proposed scheme, we require the inverse of the update Q_k and appropriate values for the positive parameters and . To achieve that goal, we first show that Q_k is positive-definite. Assuming that , with , then we have that and . This further implies that a set of mutually orthonormal vectors exists such that , which determines that

Thus, Q_k contains n − 2 eigenvalues equal to each. We label the two eigenvalues left as and .

From (28), the trace of Q_k is obtained as

(24)

which then reveals that

(25)

Applying the Sherman-Morrison formula [36], the inverse of Q_k is computed as

(26)

Since has n − 2 repeated eigenvalues equal to each, labelling the remaining two as and , we obtain from (31) that

(27)

Also, expressing as the product of two matrices yields

Let and , then

(28)

By applying the formula for determinant of an arbitrary matrix of rank two, namely

and setting , , , and , we obtain

(29)

Therefore, from (33) and (34), we obtain

(30)

Now, combining (32), (35) with some simplification, we get

(31)

Using (30) and (36), we obtain

which can also be expressed as

(32)

Now, by applying the inequality in (13) to (37), we obtain

(33)

Also, by employing the Cauchy Schwarz inequality in (13) and applying similar argument, we have that , which applying (38) consequently yields

(34)

Following (39), and since is a positive parameter, we conclude that the matrix Q_k is positive-definite, which implies that is also positive-definite. We are now set to employ the measure function of Byrd and Nocedal [45] to compute approximations of and that will ensure well-conditioning of the matrix Q_k as well as better distribution of its eigenvalues. The measure function is formulated as

(35)

From (28) and applying the same approach employed to compute the determinant of earlier, we obtain the determinant of Q_k as

(36)

Now, from (29), (40) and (41), we have

(37)

In order to obtain approximations for and , we have to minimize the function with respect to both and . So, from (42), we get

Setting this result to zero, yields

which implies that

or

(38)

Also,

Setting this result to zero yields

which implies that

or

(39)

Following the above analysis and the fact that restarting search directions tend to improve convergence and efficiency of algorithms [67], we substitute (43) and (44) into (31), and propose the following auxiliary search direction:

(40)

Remark 2.3 It should be noted that the inequality , which triggers the restart process was required for two reasons: one, it is essential for analyzing eigenvalues of the matrix Q_k, and two, it helps in proving boundedness of the search direction.

Lemma 2.4 The search direction d_k in (45) satisfies the inequality

(41)

Proof: First, from (45) for k = 0, we have

Now, we proceed to analyze each of the two cases in (45) for k ≥ 1.

1^st Case: If , we have

2^nd Case: If , then from (45) and Cauchy Schwarz inequality, we get

By setting , we established the proof.■

For the line search strategy, we employ the adaptive procedure proposed by Yin et. al. [68], where the step-size is defined as , with i_k being the smallest nonnegative integer i for which

(42)

for each , is the projection operator defined as

where , and u > 0.

Now, we described algorithm of the method as follows:

Algorithm 1 (MKAS)

1: Initialization. Select , , , , r > 1, , .

Set , k=0.

2: For , stop. Else, move to 3.

3: Compute as in (27), with as defined earlier such that (48) holds.

4: If and , stop . Otherwise, compute

(43)

is the projection operator, which is defined in [69] with the property

(44)

with

(45)

5: Testing , and . Compute , and .

6: Set if , where , otherwise compute , where r > 1.

7: Set k = k+1. Goto 2.

3 Analysis of convergence

We now proceed to analyze global convergence of the MKAS algorithm: First, we give the following assumptions:

Assumption 3.1 The mapping F is monotone.

Assumption 3.2 There exists such that .

Lemma 3.3 The search direction d_k generated in step 6 of the MKAS algorithm satisfy the inequality

(46)

Proof. By applying Cauchy-Schwarz inequality and (46), the first inequality is satisfied. Now, from (45), and for k = 0, we have that , which shows that . For k ≥ 1, we consider the case of . From (45) and Cauchy-Schwarz inequality, we have

Setting , we get

For the second case, we have from (45) and Cauchy Schwarz inequality, that

Setting , we get

Therefore, since N1 is the max of (1,N1,N2), we obtain that

(47)

which proves that the second inequality of (52) holds.■

Lemma 3.4 Given that F in (1) is continuous on , then for each k ≥ 0, there exists nonnegative integer i_k such that (48) is satisfied.

Proof. Suppose by contradiction that k ≥ 0 exists for which (48) does not hold for any nonnegative integer i, i.e.,

(48)

The second inequality in (54) was obtained from the fact that . By continuity of F, and letting with , we get

which contradicts

Lemma 3.5 Suppose Assumptions hold and the sequences and are generated by Algorithm 1. Then for a solution , the sequence is convergent and and are bounded.

Furthermore

(49)

Proof: Using (48) and definition of , we have

(50)

By (2) and for all , we have

(51)

Combining the inequality (57) above, (49), (50) and (51), we obtain

(52)

where the last inequality indicates that holds. The implication of this is that is a monotone decreasing sequence that is bounded below. So, is convergent and is bounded. Now, since is bounded and F is continuous, we see that there exists a constant such that for all k ≥ 0,

(53)

Combining (59) with (53) shows that is bounded. Furthermore, from (2), Cauchy Schwarz inequality, and (56), we have

which further yields

which clearly shows the boundedness of , which by continuity of F, indicates boundedness of , namely, a constant exists for which

Combining this result with the last inequality of (58) yields

which further implies that . Hence the proof is obtained. ■

Theorem 3.6 The sequence generated by MKAS Algorithm converges globally to a solution of (1).

Proof. Two cases are analyzed for the proof

1^st case: Assuming that , then the boundedness of and continuity of F implies that some cluster points exists for which . Also, since is convergent, it implies that converges to .

2^nd case: . In this case we have that a constant exists such that

(54)

From first inequality in (52), we have

Combining this result with (55) yields

(55)

By the boundedness of the sequences and , we see that there exists subsequences and for which

with being an infinite indexing set. From (52), we get

(56)

Letting in (63) with continuity of F and (61), we obtain

(57)

On the other hand, from (48), if , then will not satisfy (48), i.e.,

(58)

By taking limit as , with in (65), (62) and continuity of F, we obtain

which shows a contradiction with (64). Hence the proof is established. ■

4 Numerical experiments

To test the MKAS method’s effectiveness, it is compared with the QN-type methods in [55] labelled SMDFP, [57] labelled SMBFGS and [56] labelled PBDFP. To code the four algorithms, MATLAB R2014a was employed with a PC configured as (2.30ghz cpu, 4GB RAM). The criteria for stopping the iterations is or or iterations exceed 1000 without attaining a solution. Parameters values for MKAS are set as , , , , , m = 0.01, r = 1.6, and u = 0.8. Parameter values for the other algorithms are set according to how they are used in the papers.

The following test examples with dimensions 1000, 5000, and 10000, where F is given as: .

Example 1 This is the gradient of the strictly convex test example in [70] where added to yield

, .

Example 2 This is a modified version of the test example in [71] with added to yield

,

, .

Example 3 [72] with

, .

Example 4 [73].

,

,

,

with .

Example 5 A modified version of the test example in [72] with

, .

Example 6 A modification of the test example 3 with

, .

Example 7 A modification of the test example 3 with added to yield

,

,

,.

Example 8 A modification of the test example 3 with added to yield

,

, .

Example 9 A modification of the test example 2 with added to yield

,

,.

The following initial guesses were used:

x₁, x₂, x₃ , x₄ ,

x₅ , x₆ . The numerical result of the experiments for the four algorithms is displayed in three tables, and is accessible from the link https://github.com/hungugida/hungugida/blob/main/Tables1-3MKAS.pdf, where “N” and “Pdim” denotes example of the operator F and dimension, “SP” and “Nit” represents for initial guess and number of iterations. Also, “Fval,” “Ptime(s)” and “Norm” stands for function evaluations, processing time and the residual at stopping point, while “***” denotes failure to obtain a solution.

In order describe the values shown in the above link, we use the evaluation too by Dolan and Moré [74]. Figs 1–3 were plotted for iterations, function evaluations, and CPU time performance metrics. From Fig 1, we see that MKAS solved about 95% of the test examples with less number of iterations. This values however, include 9% solved by the algorithm together with PBDFP, SMDFP, and SMBFGS. It also showed that SMDFP solved 15% of the test examples with less iterations. Here also, this value includes 12% solved with the other algorithms. Fig 2 showed that MKAS solved 93% of the test examples with less function evaluations. It also showed that PBDFP solved 8%, which includes 2% solved with SMDFP, and SMBFGS algorithms. For least CPU time metric, Fig 3 showed that MKAS perform better than PBDFP, SMDFP, and SMBFGS because it solved 89% while the latter recorded 8%, 1% and 2%. To further validate the results of the experiments, we present the average residual obtained for each of the four algorithms as computed from tables in the above link as follows: MKAS (1.21 × 10⁻¹¹), PBDFP (4.41 × 10⁻¹¹), SMDFP (1.83 × 10⁻¹⁰) and SMBFGS (1.75 × 10⁻¹⁰). Hence, the above analysis indicates that MKAS method is better suited for solving (1) than the other schemes.

5 Concluding remarks and future research

We have presented a new quasi-Newton-type method for solving constrained system of monotone nonlinear equations in this article. The method is an adaptive modification of the BFGS method for unconstrained optimization. The scheme possesses the vital property for analyzing global convergence as well as the trust region property irrespective of the line search procedure; what’s more, the global convergence was proven without the Lipschitz condition, thereby broadening its applicability to a wider class of problems. Another important finding was the impact of the restart strategy in improving convergence of the scheme, as evident in the best average residual it recorded. Furthermore, numerical experiments with the method and some of the available quasi-Newton type methods for solving constrained system of monotone nonlinear equations showed that it is effective. The focus of a future research will be to develop a variant of the proposed method, where the global convergence will be analyzed without the monotonicity and Lipschitz assumptions. We also hope to include real-life applications of the method in the study.

Acknowledgments

The authors thank Rabiu Bashir Yunus for his initial involvement and preliminary input during the early stages of the study.