https://orcid.org/0009-0003-4894-9215
Figures
Abstract
This article proposes a continuous-time optimization approch instead of tranditional optimiztion methods to address the nuclear norm minimization (NNM) problem. Refomulating the NNM into a matrix form, we propose a Lagrangian programming neural network (LPNN) to solve the NNM. Moreover, the convergence condtions of LPNN are presented by the Lyapunov method. Convergence experiments are presented to demonstrate the convergence of LPNN. Compared with tranditional algorithms of NNM, the proposed algorithm outperforms in terms of image recovery.
Citation: Dai X, Qiu J, Wan C, Dai F (2024) A lagrange programming neural network approach for nuclear norm optimization. PLoS ONE 19(2): e0292380. https://doi.org/10.1371/journal.pone.0292380
Editor: Ji-Hoon Yun, Seoul National University of Science & Technology, KOREA, REPUBLIC OF
Received: April 24, 2023; Accepted: September 19, 2023; Published: February 8, 2024
Copyright: © 2024 Dai et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper.
Funding: The work described in this paper was supported by the Science and Technology Research Program of Chongqing Municipal Education Commission (Grant No. KJZD-M202201204, KJZD-K202201205), the Opening fund of Chongqing Engineering Research Center of Internet of Things and Intelligent Control Technology (Grant No. zhlv-20221007, zhlv-20221002), Science and Technology Innovation Smart Agriculture Project of Science and Technology Department, Wanzhou District of Chongqing (Grant No. 2022-17), and the Opening Project of Sichuan Province University Key Laboratory of Bridge Non-destruction Detecting and Engineering Computing (Grant No. 2022QYY04). All grants were received by XD.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Recently, many researchers were increasingly interested in collecting useful messages from very limited information. Most real data matrices restoring information are low-rank or approximate low-rank. In other words, a fundamental assumption of the problem is that the target matrix has a low-rank structure. Thus, we call similar issues of a low-rank structure to be the Rank Minimization Problem (RMP). Some applications (e.g. control [1], machine learning [2], and computer vision [3]) can be formulated as different versions of RMPs. In the view of applications, RMP can be commonly reformulated as a problem of reconstructing data matrix from partial matrix sampling. Suppose that is a target matrix with partial observations. Ω is a set of known elements in matrix M. X is a low-rank matrix after recovered and rank(X) is the rank of X. RMP can be formulated into the following optimization problem
(1)
Due to the nonconvexity and combinatorial nature properties of rank functions, problem (1) is an NP-hard problem. Candès et al. [4] proved that the majority missing elements of a matrix with the rank r can be properly restored with a high probability by using a simple convex optimization programme in case of
(2)
where C is a certain positive numerical constant. Commonly, the above-mentioned problem can be reformulated as a nuclear norm minimization problem (NNM). The task of fully recovering the majority missing elements M with rank r can be accomplished by NNM. Problem (1) can be rewritten as
(3)
where
is the nuclear norm of X, σ(X) is the singular value of the matrix X and k = min(n1, n2).
Many optimization approaches [5–12] were proposed to solve problem (3). Cai et al. [7] proposed singular value thresholding (SVT) to address the dual of a regularized approximation of (3) by the linearized Bremgan iterations [11]. Toh et al. [12] proposed an accelerated proximal gradient with linesearch algorithm (APGL) by [6]. Ma et al. [9] proposed fixed point continuation with approximate (FPCA) singular value decomposition (SVD) using a fast Monte Carlo algorithm for SVD calculations. In addition, there are other approaches [5, 8, 10]. Notably, above referenced approaches can generate continuous real-time solutions. Much more room is needed to develop new optimization algorithms of problem (3).
Neural networks were used to address nonlinear equality constraint problems. Many researchers aimed to reformulate the optimization problem (3) and solve it by neural networks [13, 14]. The main advantage of neural networks is to generate real-time solutions. Zhang and Constantinides [15] firstly created a Lagrange programming neural network (LPNN) to address a variety of nonlinear constrained optimization problems. After that, the global convergence conditions of LPNN were attracted the attention [16, 17]. Recently, a large number of variants of LPNNs were proposed to address different application problems. Feng et al. [18] proposed a new LPNN approach to recovere sparse signals based on the locally competitive algorithm (LCA). Liang et al. [19] and Shi et al. [20] applied LPNN to address the radar location problem. Xiong et al. [21, 22] applied LPNN to Robust TDOA source localization problem and Elliptic target positioning problem. Based on above mentioned researches, we try to solve problem (3) by using LPNN. The contributions are as follows:
- The nuclear norm minimization problem (NNM) is reformulated as a optimization problem of the matrix form, and we solve NNM by using a Lagrange programming neural network method (NNM-LPNN).
- By constructing an appropriate Lyapunov function, we propose the convergence conditions to prove the stability of NNM-LPNN.
2 Preliminaries
2.1 Notations
Suppose that is m × n-dimensional real matrices. The capital A is a matrix and a(i, j) is the the i, j-th entry of A. A⊤, rank(A) and Tr(A) are the transposition, the rank and the trace of A, respectively. ∥A∥F and ∥A∥* represent the Frobenius norm and the nuclear norm of A.
, and
is the largest singular value of A. 〈A, B〉F and A ⊙ B are the Frobenius inner product and the Hadamard product of A and B.
2.2 Subdifferential
The definition of subdifferential is put forward in [23].
Definition 1. Let the function be convex and Ω be convex. For any
, we have
so that Δ ∈ ∂f(X), where ∂f(X) serves as the subdifferential of f(X). The partial differential function of f(X, Y) with regard to X is denoted by the symbol ∇Xf(X, Y).
Proof. Let X, Y ∈ Rm×n, according to Definition 1, we have
(5a)
(5b)
Add (5a) and (5b), we obtain
(6)
A matrix X can be precisely represented by its singular value decomposition (SVD)
(7)
where the elements ρ1, ρ2, …, ρr ≥ 0 are the singular values, the singular vectors are u1, u2, …, ur ∈ Rm and v1, v2, …, vr ∈ Rn.
In [24], Y has the form
(8)
where W ∈ {Z|PUZ = 0, ZPV = 0, ∥Z∥F ≤ 1}. Obviously, Y is a subgradient of ∥X∥*.
In this article, the subgradient of the nuclear norm is represented as follows:
(9)
Remark 1. The computation of the subgradient of the nuclear norm can be derived using the formula in (8). In [24], minimum-norm subgradient can be used to ensure convergence. Since E is SVD decomposition from X, it is only necessary to minimize Y by minimizing ∥W∥F. Due to the positive definiteness of Frobenius norm, we can easily get W = 0.
2.3 Lagrange programming neural network
Problems involving general nonlinear programming with equality requirements can be solved using the following LPNN method
(10)
where
is the objective function,
describes the m × n constraint on equality. Suppose that both functions f and h are quadratically differentiable. A Lagrangian function of problem (10) is constructed as follows:
(11)Λ is a matrix form of Lagrange multipliers. In the view of the structure of neural networks, variable and Lagrange neurons cannot belong to the similar types of neurons. In other words, the entries of X and Λ are stored in variable neurons and Lagrange neurons, respectively. The dynamics of neurons determined by
(12)
where τ0 is the circuit’s time constant.
3 Lagrange programming neural network for the nuclear norm optimization
3.1 NNM properties
NNM can be formulated as
(13)
According to problem (13), we have the following proposition.
Proposition 1. Let X⋆ an optimal solution of problem (13) if and only if there exists a Λ⋆ and the following conditions
(14a)
(14b) satisfied.
The conditions for the Karush-Kuhn-Tucker (KKT) equation are outlined in Proposition 1 by (14) in its entirety. The KKT requirements are both necessary and sufficient because the problem is convex.
3.2 NNM-LPNN dynamics
To avoid stability issues around equilibrium, we provide an enhanced term in the Lagrange function as follows:
(15)
The objective value at an equilibrium X* is unaffected by the addition of the augmented term, because W ⊙ (X* − M) = 0. The gradients of
are given by
(16a)
(16b)
where Y ∈ ∂ ∥ X∥*.
A neurodynamic centralized method for solving(13) is
(17)
Theorem 1. Suppose that {X⋆, Λ⋆} serves as the LPNN dynamics of the equilibrium point of system (17). The KKT criterion of problem (15) is satisfied at the equilibrium point. Thus, the KKT criterion of NNM is necessary and sufficient. The stabilization point of (17) is consistent with the desired solution of problem (15).
Proof. By the definition of stabilization points
(18)
Based on (17) and (18), we can conclude that
(19a)
(19b)
With Y⋆ ∈ ∂ ∥ X⋆∥*, we have
(21)
Therefore, (16a) is also satisfied. Similarly, we can prove that (14b) leads to (19b). The proof is finished.
Theorem 2. The neurodynamic method (17) globally converges to the optimal solution of the problem (13).
Proof. Construct a Lyapunov function:
(22)
By definition, the above function V1(X, Λ) is semi-positive definite and it is radially unbounded in the domain of definition. The derivative of V1 can be expressed as
(23)
From (17) and (19), we have
(24)
Substituting (24) into (23), we obtain
(25)
From Lemma 1 and the definition of W, we could get
(26)
According to (26) and
, we have
. In other word, for arbitrary initial value
, V1(X, Λ) is nonincreasing when t → +∞. Hence, the proof is completed.
4 Experiments
We carry out experimental simulations of the following aspects to verify the performance of the proposed approach (NNM-LPNN in short):
- Optimality and convergence.
- Mean Square Error (MSE), Normalized Mean Square Error (NMSE), Recovery Error (RE) and Peak signal-to-noise ratio (PSNR).
- Recovery ability of low-rank images at different sampling rates (SRs) and comparison with classical Singular Value Thresholding (SVT) approach and alternating direction multiplier method (ADMM) [25] and fixed-point continuation(FPC) [9].
4.1 Low-rank numerical matrix recovery
Example 1: is synthesized obeying distributed standard Gaussian distribution [4]. Giving a rank r, we generate two matrices Mm×r and Mr×n using NNM-LPNN. Suppose that m = n = 50 and r = 5. The matrix reconstruction result is shown in Fig 1. The recovery results with SR = 50% are shown. This phenomenon indicates that NNM-LPNN can recover the target M accurately. Moreover, the optimal solution of problem (13) is shown in Fig 2. Its reconstruction error and relative error are shown in Figs 3 and 4. In summary, the NNM-LPNN can reach the equilibrium point and achieve the optimal solution.
4.2 Application of image reconstruction
Example 2: We select a real image as shown in Fig 5 and its first 50 singular values are shown in Fig 5. This image can be considered to be a low-rank image. As shown in Fig 7, we sampled the image at SR = {10%, 50%, 90%} and the reconstruction results of the three approaches are shown in Fig 7. In addition, we have compared the Mean Square Error (MSE), the Normalised Mean Square Error (NMSE) and the Peak Signal-to-Noise Ratio (PSNR) of the proposed methods with the classical approach SVT, where ,
and
.
Process the original image Fig 6. As shown in the Fig 7, NNM-LPNN can reconstruct the approximate low-rank real image. Since the reconstruction effects of the four approaches are similar visually, we analyze the reconstruction from the indicators data. For MSE and NMSE, the smaller the value is, the better. PSNR is opposite. In Tables 1–4, we can find that the effect of NNM-LPNN is always better than the other two approaches, and occasionally FPC can be as good as NNM-LPNN. In a word, the NNM-LPNN proposed in this paper are better than the traditional SVT approach in reconstructing real images. Their results are shown in Figs 8–10.
5 Conclusion
In this article, a Lagrange programming neural network was proposed to solve the nuclear norm minimization. The stability and optimality of the proposed approaches were proved theoretically. Low-rank matrix recovery and approximate low-rank image recovery experiments demonstrated that the presented neural network is effective.
References
- 1. Mesbahi M, Papavassilopoulos G P. On the rank minimization problem over a positive semidefinite linear matrix inequality[J]. IEEE Transactions on Automatic Control, 1997, 42(2): 239–243.
- 2. Luo X, Zhou M C, Li S, et al. A nonnegative latent factor model for large-scale sparse matrices in recommender systems via alternating direction method[J]. IEEE transactions on neural networks and learning systems, 2015, 27(3): 579–592. pmid:26011893
- 3. Tomasi C, Kanade T. Shape and motion from image streams under orthography: a factorization method[J]. International journal of computer vision, 1992, 9: 137–154.
- 4. Candes E, Recht B. Exact matrix completion via convex optimization[J]. Communications of the ACM, 2012, 55(6): 111–119.
- 5.
Wright J, Ganesh A, Rao S, et al. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization[J]. Advances in neural information processing systems, 2009, 22.
- 6. Beck A, Teboulle M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems[J]. SIAM journal on imaging sciences, 2009, 2(1): 183–202.
- 7. Cai J F, Candès E J, Shen Z. A singular value thresholding algorithm for matrix completion[J]. SIAM Journal on optimization, 2010, 20(4): 1956–1982.
- 8.
Lin Z, Chen M, Ma Y. The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices[J]. arXiv preprint arXiv:1009.5055, 2010.
- 9. Ma S, Goldfarb D, Chen L. Fixed point and Bregman iterative methods for matrix rank minimization[J]. Mathematical Programming, 2011, 128(1-2): 321–353.
- 10. Yang J, Yuan X. Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization[J]. Mathematics of computation, 2013, 82(281): 301–329.
- 11. Yin W, Osher S, Goldfarb D, et al. Bregman iterative algorithms for l1-minimization with applications to compressed sensing[J]. SIAM Journal on Imaging sciences, 2008, 1(1): 143–168.
- 12. Toh K C, Yun S. An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems[J]. Pacific Journal of optimization, 2010, 6(615-640): 15.
- 13. Hopfield J J. Neural networks and physical systems with emergent collective computational abilities[J]. Proceedings of the national academy of sciences, 1982, 79(8): 2554–2558. pmid:6953413
- 14. Chua L, Lin G N. Nonlinear programming without computation[J]. IEEE Transactions on Circuits and Systems, 1984, 31(2): 182–188.
- 15. Zhang S, Constantinides A G. Lagrange programming neural networks[J]. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 1992, 39(7): 441–452.
- 16. Lou X, Suykens J. Stability of coupled local minimizers within the Lagrange programming network framework[J]. IEEE Transactions on Circuits and Systems, 2013, 60(2): 377–388.
- 17. Xia Y. Global convergence analysis of Lagrangian networks[J]. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 2003, 50(6): 818–822.
- 18. Feng R, Leung C S, Constantinides A G, et al. Lagrange programming neural network for nondifferentiable optimization problems in sparse approximation[J]. IEEE transactions on neural networks and learning systems, 2016, 28(10): 2395–2407. pmid:27479978
- 19. Liang J, So H C, Leung C S, et al. Waveform design with unit modulus and spectral shape constraints via Lagrange programming neural network[J]. IEEE Journal of Selected Topics in Signal Processing, 2015, 9(8): 1377–1386.
- 20. Shi Z, Wang H, Leung C S, et al. Robust MIMO radar target localization based on Lagrange programming neural network[J]. Signal Processing, 2020, 174: 107574.
- 21. Xiong W, Schindelhauer C, So H C, et al. Robust TDOA source localization based on Lagrange programming neural network[J]. IEEE Signal Processing Letters, 2021, 28: 1090–1094.
- 22. Xiong W, Liang J, Wang Z, et al. Elliptic target positioning based on balancing parameter estimation and augmented Lagrange programming neural network[J]. Digital Signal Processing, 2023, 136: 104004.
- 23.
Guenin B, Könemann J, Tuncel L. A gentle introduction to optimization[M]. Cambridge University Press, 2014.
- 24. Recht B, Fazel M, Parrilo P A. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization[J]. SIAM review, 2010, 52(3): 471–501.
- 25. Chen C, He B, Yuan X. Matrix completion via an alternating direction method[J]. IMA Journal of Numerical Analysis, 2012, 32(1): 227–245.