Introduction

Digital image inpainting is an important part of digital image processing. The final presentation result of digital image inpainting technology must meet the connectivity principle of the human and the fluency of the overall image. On the basis of guaranteeing inpainting of cracks or removal of contaminants, the person or landscape in the inpainting image looks as natural as before.

Image inpainting is the process of restoring missing pixels in digital images in a plausible way. The challenge is that the recovery processes themselves introduce noticeable artifacts within and around the restored image regions. The curvature-driven diffusion model repairs the lost region by diffusing the data near the lost region during image transmission, and uses the edge information of the region to be repaired to diffuse the information outside the region to be repaired into the region to be repaired along the vertical direction of the gradient. The curvature-driven diffusion model introduces a curvature term to make its partial differential equation of higher order, and applies the fast marching method to the boundary of the region to be repaired to gradually advance it from outside to inside, and while advancing it uses the correlation between the pixel to be repaired and its neighboring pixels around it to repair the various discrete pixel points on the boundary. On this basis, the P-Laplace operator is introduced to fill the damaged region by using the performance of the nonlinear anisotropic diffusion of the P-Laplace operator, combining the P-Laplace operator with the diffusion term of the curvature-driven diffusion model to discretize the improved curvature-driven diffusion model model, and dividing the known information around the broken region into two weighted average iterations to obtain the final repair image.

Current restoration methods:

Deep learning-based image restoration methods

With the gradual improvement and development of deep learning algorithms, researchers have started to apply deep learning algorithms to the field of image restoration. Researchers have successively proposed Convolutional Neural Networks such as AlexNet, VGG-NET, and ResNet [1–4], which are used to train data to restore images and predict the structure of images efficiently [5–7]. Goodfellow proposed generative adversarial networks [8] in 2014, which consist of a generator and a discriminator, where the generator synthesizes data from a given noise data, and the discriminator discriminates the similarity between the synthesized data and the real data, and if the computationally generated sample image is similar enough to the image of the area to be restored, image restoration can be achieved. With the work of deep learning in the field of image restoration, recurrent neural networks have been applied in the automatic generation of unmarked images, and further research results have been achieved by combining them with convolutional neural networks for image restoration. However, there are the following shortcomings: CNN is widely researched, but there is a shortage for texture restoration; GAN can be applied to image restoration with a large amount of missing data, but the instability problem in the training phase of GAN needs more in-depth research to solve; RNN has excellent performance in processing sequential data, but it is not so good for processing large sample data. Since then, many excellent networks have been proposed. The literature [9] proposed a novel two-stream network for image restoration is proposed, which models structure-constrained texture synthesis and texture-guided structure reconstruction in a coupled manner so that they can better utilize each other for more rational generation. In addition, to enhance global consistency, a bidirectional gated feature fusion (Bi-GFF) module is designed to exchange and combine structural and texture information, and a contextual feature aggregation (CFA) module is developed to refine the generated content affinity learning and multi-scale feature aggregation by region.

In 2023, the literature [10] proposed a U-Net-based latent diffusion model that performs diffusion in a low-resolution latent space while retaining high-resolution information from the original input for the decoding process. Compared to previous latent diffusion models that train VAE-GAN to compress images, the proposed U-Net compression strategy is more stable and can recover highly accurate images without relying on adversarial optimization, aiming to improve the applicability of diffusion models in real-world image recovery. the literature [11] proposed a lightweight single-image super-resolution network that fuses multi-level features. The components are mainly two-level nested residual blocks. To better extract features and reduce the number of parameters, each residual block adopts an asymmetric structure. Moreover, the selection of the appropriate loss function of the neural network requires high hardware requirements, and the network training is time-consuming and other problems. Partial differential equation based image restoration algorithm can avoid these shortcomings.

Partial differential equation based image restoration algorithm

The image restoration method of repairing lost areas through known areas originated from the restoration techniques used by craftsmen for damaged artworks, i.e., diffusion restoration based on local information, which mainly revolved around anisotropic filtering until 2000. Malik [12] proposed various anisotropic diffusion equations. These methods can achieve good results in small scratches, small object removal, etc. However, for applications such as restoration of larger areas of broken images, removal of large objects from images, etc., it is difficult to achieve the desired restoration results in these applications.

And essentially, the first consideration in large region restoration is definitely the high-level semantic information of the restored object, and then the repair is based on the large amount of a prior information it has accumulated. For large region restoration, there are mainly two types of effective methods: one is based on texture synthesis techniques [13–17], which can achieve good results in texture detail restoration, but it is difficult to capture the global structure of the image, and the semantics of the image; the other is based on external database search methods [18], which assume that regions surrounded by similar contexts may have similar content, which is very effective when sample images with sufficient visual similarity to the image to be restored can be found, but when the restored image is not well represented in the sample database, false restoration occurs, making the final restoration unsatisfactory.

In 2000, researchers thus proposed early image restoration algorithms based on partial differential equations, which use the idea of diffusion, where the data near the lost region during image transmission arrives to repair the lost region by diffusion, and in 2000 Bertalmio, Sapiro, Caselles, and BallesterIl [19] first proposed a BSCB repair model, which uses the edge information of the region to be repaired to diffuse the information outside the region to be repaired into the region to be repaired along the gradient vertical direction. This process is an iso-illumination line extending along the tangent direction. The algorithm does not require the topology inside the repaired region and is well adapted, but defects such as too slow repair speed and easy blurring after repair hinder its application prospects. In 2001, Chan et al. improved a total variation model(TV) by applying it to image restoration and proposed a new repair model, namely Curvature-Driven Diffusion (CDD) [14], which can overcome the “deficiencies” of TV. It is an extension of the total variance model and the main improvement is the introduction of curvature information in the process of image information diffusion. In 2007, the literature [20] proposed a fast image restoration algorithm for the traditional classical restoration method, which has the disadvantages of slow restoration speed and does not maintain strong edges, by applying a fast marching method to the boundary of the region to be restored. The fast marching method gradually advances from the outside to the inside, and the advancement uses the correlation between the pixel to be repaired and its surrounding neighboring pixels to repair each discrete pixel point on the boundary until the end of the restoration. In the literature [21], the P-Laplace operator is applied to image repair to establish the P-Laplace model for image repair. The model uses the performance of nonlinear anisotropic diffusion of the P-Laplace operator to fill the damaged area, and controls the diffusion direction and time by taking a P value between 1 and 2 to achieve improved repair results and shorter repair time. In 2013, the paper [22] proposed a new tensor diffusion-based wavelet repair model(TDWI) to recover lost or damaged wavelet coefficients. A hybrid model was developed by combining structural adaptive anisotropy regularization with wavelet representation. The shape of the diffusion kernel adaptively changes according to the features of the image, including sharp edges, corner points, and homogeneous regions. Compared with existing wavelet restoration models, this model can more adaptively and accurately control the geometric regularity in the image and has better robustness to noise. In 2016, the literature [23] established a new class of image restoration models based on fractional-order nonlinear anisotropic diffusion equations that employ the P-Laplace norm of the fractional-order gradient of the image intensity function. This model can effectively enhance the texture details of the image and eliminate the step and speckle effect, in addition to effectively removing noise and non-linearly maintaining the high-frequency edges of the image. In 2017, the literature [24] proposed a new TV-Stokes model for image deblurring with a better geometric interpretation. In the image reconstruction, we first calculate the smoothed part of the image from the smoothed tangential field, and then use the anisotropic TV model to obtain the “textured” part of the deblurred image. Solvable properties of the two-step minimization problem are established, and a fast algorithm is given. The new deblurring model is able to capture image details hidden in blurred and noisy images, and the fast algorithm is efficient and robust. In 2018, the literature [25] proposed a wavelet repair model using fractional-order full-variance regularization method. The computational efficiency is improved and the convergence of the new algorithm is ensured. In 2023, the literature [26] proposed a fine inpainting method of incomplete image based on features fusion and two-steps inpainting (FFTI), Firstly, the dynamic memory networks (DMN+) are used to fuse the external features and internal features of the incomplete image to generate the incomplete image optimization map. Secondly, a generation countermeasure generative network with gradient penalty constraints is constructed to guide the generator to rough repair the optimized incomplete image and obtain the rough repair map of the target to be repaired. Finally, the coarse repair graph is further optimized by the idea of coherence of relevant features to obtain the final fine repair graph.

Image repair methods based on partial differential equations can repair small-scale broken images well and can also repair multiple broken fields simultaneously, but the repair results are not satisfactory for images with a large range of missing information, and with the application of image repair in new fields such as biomedicine and face image repair, the quality requirements of the repaired images are getting higher and higher, and the researchers proposed a dynamic weighted matching image restoration algorithm, which better utilizes the known information of the image, improves the image restoration quality, and uses the local average gray-scale entropy fast image restoration algorithm to speed up the computer to perform the restoration of structural information, connect the edges, and later repair the texture components, which can improve the consistency of the image structure, texture, etc.

Embodiment of the advantages of the algorithm in this paper: The algorithm in this paper mainly repairs old photos with scratches and images covered by text. Theoretical analysis and experimental results show that the model has shorter repair time than the CDD model and can obtain visually more natural repaired images.

CDD inpainting model

CDD inpainting model is built on TV model. As practiced in the variational methodology, it is very convenient to solve the TV inpainting problem. Fig 1 indicates that E (Extended Area), D (inpainting Area), etc. (1) Where plays the role of the Lagrange multiplier for the constrained variational problem. The Euler-Lagrange equation for the energy functional is (2) for all, adding Neumann boundary conditions. Here the extended Lagrange multiplier is given by The infinitesimal steepest equation for is therefore given by (3) Since takes two different values. (2) or (3) is a two-phase problem, and the interface is the boundary P of the inpainting domain. From the numerical point of view, in all of the above differential equations, we replace the curvature term.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Inpainting domain.

https://doi.org/10.1371/journal.pone.0305470.g001

Let D be an inpainting domain with piecewise smooth boundary, and E any fixed closed domain in the complement D^c, so that lies in the interior of E ∪ D. To satisfy the connectivity principle of human, Chan and Shen [15] proposed a new PDE model based on curvature driven diffusions(CDD). Which is closely inspired by the TV inpainting model. The CDD inpainting model is governed by the following PDE: (4) where k is the scalar curvature . The new ingredient of the CDD model, compared with the TV inpainting model. The choice of a coefficient value of 1 outside the inpainting domain indicates that the model carries out the regular TV denoising task outside D. Meanwhile, g(s) can be any appropriate function that penalizes large curvatures and stabilizes small curvatures inside the inpainting domain. In Chan and Shen [15], it is argued that g(s) must satisfy. Thus, for example, one can choose for some g(s) = s^p for some p ≥ 1. Under this condition, the model stretchesout bent level lines inside the inpainting domain, outputs connected objects, and therefore realizes the connectivity principle. For better repair effect, we replace the diffusion function g(s) with Δ_pu where Δ_pu = div (|∇u|^p−2∇u), 1 ≤ p < ∞. Using P-Laplace operator to fill the damaged domain can make the image edge transition more natural and improve the inpainting effect. Then, the CDD model with P-Laplace operator is (5) Using the variational method to solve the extremum question, the following equations are solved (6)

The inpainting algorithm of this paper

Various image inpainting algorithms based on CDD model only make use of the reference information of four neighborhood pixels. Therefore, they cannot keep shape edges and their inpainting precisions high enough. To conquer these difficulties, the image inpainting algorithm based on improved CDD was presented in which the reference information for damaged pixel was extended from 4 into 16 neighborhood pixels. The improved CDD algorithm can effectively improve the inpainting precision and keep shape edeges. Extending the method of repairing damaged points with information from known pixel points in the damaged point domain was inspired by the literature [27], where the CDD model was used to repair damaged points using information from 8 points on P = {N, S, W, E, NW, NE, SW, SE} only. In this paper, we first introduce the P-Laplace operator to modify the diffusion coefficient of the CDD, and then use the improved CDD model to repair the damaged points using the known information of a total of 16 points on p and p′ = {N, S, W, E, N₁, S₁, W₁, E₁}.

An improved CDD inpainting algorithm is proposed to inpaint the image twice. Experiments show that the algorithm proposed in this paper not only has good quality of image inpainting, but also the inpainting time takes less, so it is a better algorithm for fast image inpainting. Fig 2 let E, N, W, S denote four adjacent pixels of the pixel O, and e, n, w, s the corresponding four midway points. Write Usually, the noise is ignored in the missing or damaged areas. then Eq (3) can be modified as (7) Eq (7) can be simplified as , where , the curvature . We can also use the half-point center difference method to discretize formula (7), as shown in Fig 2.

Then the divergence is first discretized by central differencing: (8) where h denotes the grid size, which is always taken to be 1 in image processing. Next, we generate further approximations at the midway points, where image information is not directly available. Take the midpoint e, for example: (9) (10) (11) (12) (13) (14)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. A pixel O and its neighbors.

https://doi.org/10.1371/journal.pone.0305470.g002

Similar discussion applies to the other three directions N, W, S and the other half-pixel points n, w, s. Therefore, at a pixel O, the Eq (2) is discretized to: (15) where the function represents the original defective image. For any arbitrary point , the function w_d,ε be defined: (16) where the parameter ε > 0. When ε = 0, the function w_d is simplified as: (17) Similar, the gradient norma |∇u_w|, |∇u_n|, |∇u_s| of other half-pixel points can be calculated. We solve Eq (15) to obtain (18) The formula represents the diffusion coefficient corresponding to the four pixels e, n, w, s. The function u_p represents the gray value of the four points:E, N, W, S. Let (19) Using formula (19) to simplify formula (18), a new inpainting formula is obtained: (20) With Eq (20) is in the form of a low pass filter, which is of course a system of nonlinear equations since the filter coefficients all depend on u. Freezing the filter coefficients (to linearize the equations), and adopting the Causs-Jacobi iteration scheme for linear systems, at each step n, we update u⁽ⁿ⁻¹⁾ to u⁽ⁿ⁾ by (21) Where h^m−1 = h(u^m−1). Since h is a low pass filter, the iterative algorithm is stable and satisfies the maximum principle [16]. In particular, the gray value interval [0, 1] is always preserved during the iterating process.

Useful variations of the algorithm can be obtained by altering the definition w_d in (17). Experiments show that such variations sometimes work better for inpainting sharp edges in the digital setting. In implementation, the weights w_d are (22) For some small number ε, to avoid a zero divisor in smooth regions. Notice that choosing a large ε brings the TV model closer to the harmonic inpainting(especially computationally, since the spatial step size h is set to 1, and u takes values from the finite gray-scale interval [0.1]). In addition, as ε gets bigger, the convergence the iteration scheme speeds up.

Although the pixels in area E are known, the iterative operation is still performed. The new value of the point O is calculated from the old value and the adjacent pixels. The iterative operation is performed for each pixel in the area E, so that the known information is gradually advaced to the damaged area, which inpaints the image.

The above analysis shows that the restoration value of the point O is only related to eight pixels in its area on the surface, but in fact, every pixel in the area participates in the iterative operation, and the restoration value of the point O is related to the pixel value in the area. However, because each operation only considers eight pixels in the field of point O, the information obtained is limited.

In order to reduce the inpainting time, our algorithm uses a weighted average kernel with the diffusion coefficient to inpaint the highest bit layer of image.

At the highest level of the image, we use the new inpainting formula (20) to calculate the inpainting point O by eight pixels neighborhood of the point O, and use the information of the eight points to predict the point O. When |∇u_d| = 0, the weighting coefficient tends to infinity, So the adjustment parameter ε is introduced. The following inpainting formula is obtained: (23) As can be seen from the above equation, the inpainting formula is closely related to the curvature term and gradient. When the gradient is large, the weighting coefficient is small, and when the gradient is small, the weighting coefficient is large, so the inpainting algorithm is also anisotropic. Due to the addition of gradient and curvature terms, the “connectivity criterion” can be satisfied.

The essence of the CDD model is to use the weighted average of four points in the field of points to be repaired as the change variable in the repair iterative process. This model can achieve both denoising and preserving image edge in the process of restoration. However, it only uses 8 points{N, S, W, E, NE, NW, SW, SE}in the field of points to be repaired. If four more points in the repair area are added to provide effective information for the points to be repaired, the accuracy of the repaired image will be greatly improved. Following this idea, on the basis of the CDD model, the solution information of the points to be repaired is extended to 12 points in the neighborhood, so that the accuracy of the repaired images can be improved under the condition of increasing the time complexity. Then 8 points in the repair area are selected again, and the repair points are repaired by using the CDD model, and a formula (24) was obtained, (24) where, Λ₀ = {E₁, N₁, W₁, S₁}, .

Formula (24) is obtained by rotating the field points of Formula (18) in the following way: In the above process, the repair points have been repaired twice at the same time and the results obtained are respectively u₀ and . Below, the weighted average of these two different results is carried out to obtain the final repair effect.

Since two different fields with different distances of the point O to be repaired are used respectively in function u₀ and .

Therefore, the difference step size used to solve O is different, which results in different influence on O. Therefore, according to the distance ratio and weighted average, the final repair value of the points to be repaired is as follows: (25)

Specific details and methods:

1. Firstly, based on the information of the eight points {N, E, S, W, NE, SE, WS, NW} of the inner solid square in Fig 2, the point O to be repaired is repaired using Formula (18) to obtain u₀.
2. Then, based on the information of the eight points {E₁, N₁, W₁, S₁, NE, SE, WS, NW} of the external solid square in Fig 2, the point O to be repaired is repaired using Formula (24) to obtain .
3. Finally, the repair values u₀ and are superimposed by taking the weights according to the degree of influence of the distance, as in Formula (25), to obtain the value of the point to be repaired O, u.

Reasons for empowerment:

1. The closer it is to the point to be repaired, the greater its correlation with the point to be repaired, and the greater its influence on the point to be repaired;
2. The image connectivity criterion is satisfied. For a certain pixel related to the repair point, when the distance of unit length is increased, the increased pixel value is equal. Guass-Jacobi iterative method is adopted: (26)

Experimental results

We have implemented the algorithm with MatlabR2016a on the PC with i5-1035G1, 1.00GHz, 16G memory. The size of the image used is 256 * 256. Take the parameters λ = 1, P = 1.6 and carry out experiments on large spots and scratches respectively.

In order to verify the experimental results, we use TV model, CDD model and the improved algorithm to carry out restoration experiments. The algorithm parameter ε = 1. The signal-to-noise ratio is calculated as follows: (27) Considering the importance of inpainting application, several metrics have been proposed in the literature [28] specially dedicated to the image inpainting quality assessment. Then, we apply Structure Similarity Index Measure(SSIM), Feature Similarity Index Mersure (FSIM) and Normalized Mean Square Error(NMSE)to evaluate the quality of the images: (28) (29) (30) where M = N = 255. The above images are all taken by ourselves. For Fig 3, for the original image, TV model and CDD model inpainting algorithm are basically consistent with the inpainting results of the algorithm in this paper. In Fig 3, the number of iterations is 150,1000,100 respectively, CDD model and TV model can clearly show up the inpainting traces in the inpainting results, but the inpainting traces are almost invisible with the algorithm in this paper. The algorithm in this paper is least iterations, and the restoration effect of the restoration algorithm in this paper is best.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Image inpainting of dog, flower and buildings.

https://doi.org/10.1371/journal.pone.0305470.g003

The above three images of dogs, flowers and buildings have been studied by TV model, CDD model and the algorithm proposed in this paper. From Tables 1–3 and Fig 4, it can be seen that there are obvious traces of restoration in TV and CDD restored images, and the restoration effect of the restoration algorithm proposed in this paper is better. In the image inpainting experiments, compared with the TV model, the algorithm proposed in this paper consumes the least time and has the best effect, so the algorithm proposed in this paper is better than the TV model and CDD model. Fig 4 shows graphically that the iteration time growth rate of the proposed algorithm and TV model is much larger than that of the CDD model with the growth of time.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Iteration times and time curve.

https://doi.org/10.1371/journal.pone.0305470.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of algorithm time (s) and signal-to-noise ratio (PSNR).

https://doi.org/10.1371/journal.pone.0305470.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of Structure Similarity Index Mersure(SSIM) and Feature Similarity Index Mersure (FSIM).

https://doi.org/10.1371/journal.pone.0305470.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of Normalized Mean Square Error(NMSE).

https://doi.org/10.1371/journal.pone.0305470.t003

The image restoration model based on partial differential equations utilizes useful information around the area to be repaired to diffuse into the area. The advantage of this method is its simplicity of use. However, this method cannot repair large damaged areas, while neural network image repair methods can repair large damaged areas. In reference [29], the occlusion of facial images reached 25%, and the PSNR value after repairing using CE network reached 30.75. Although neural networks require high software and hardware conditions, the method of repairing images using neural networks has a better effect on repairing areas with large damage. In the future, neural network methods will be the mainstream method for image processing.

Conclusions

On the basis of studying TV model and CDD model inpainting algorithm proposed by Chan et al., the improved algorithm which uses an improved CDD model to perform on the highest level of image. The algorithm uses the structural information such as image gradient and curvature to inpaint, and achieves good results. In the case of fewer iterations, the improved algorithm is better than TV model, CDD model, and the image texture features can also be inpainted. It is worth improving that because the algorithm in this paper targets grayscale images, when there is a brighter area in the image, it will default to blank for repair, causing the area to become blurred and destroy the original connectivity. Moreover, when there are a large number of blank areas, the repair effect is not satisfactory. The above problems deserve more in-depth research.

Overall, the design of deep learning network and the selection of loss function during training are to be explored, and the selection of appropriate loss function will improve the image repair quality while also speeding up the training speed of deep learning. The improvement of image restoration quality can also be solved by reducing the depth of the noise model. Therefore, more in-depth research is needed on how to design a more perfect denoising model and how to design a restoration network with universal applicability to improve the accuracy of the restoration results.