Fig 1.
Φ is the source region/image, Ω is the target region and ∂Ω represents the boundary between the source region and the target region.
ψp is the patch that will be filled, stands for a bigger patch having the same center p with ψp.
Fig 2.
(a) The original image; (b) The green target region; (c) and (d) are the inpainted images by the method in [23] and our method, respectively. (e)-(h) are the process of Criminisi’s method when 5th step, 50th step, 72th step and 100th step, respectively. (i)-(l) describe the process of the proposed method when 5th step, 50th step, 72th step and 270th step, respectively.
Fig 3.
An inpainting image I with structure lines (i.e., black solid lines).
Fig 4.
(a) The edge image of “plane” by “canny” edge detector; (b) a structure bar ϕs (orange part), blue frames are the filling patches which have a central black structure line (note that we regard edges as structure lines in our work). In particular, AΦs is computed by the sum of all structure bars in Φ (color images are better visualized in the pdf file).
Fig 5.
Test images from top to bottom: “zebra” (with the size of 115 × 138), “windmill” (311 × 380) and “plane” (277 × 302).
(a) The original images; (b) The original images with the green target regions; (c) and (d) are the inpainted images by the method in [23] and our method, respectively. The first phase of the proposed method executes 16, 45 and 10 steps, respectively. Readers are recommended to zoom in all images for better vision.
Fig 6.
An open dataset (237 images) to test the assumption in Eq (7).
Fig 7.
The rates of (blue points) and
(red points) for Eq (7) on the open dataset.
Note that Eq (7) is reasonable since the red points approach to the blue points. The average rate of the 237 images is 0.132 for the blue points and 0.097 for the red points. The mask is randomly with unknown region.
Fig 8.
The rates of (blue points) and
(red points) with the mask of randomly
unknown region.
The average rate of the 237 images is 0.116 for the blue points and 0.095 for the red points.
Fig 9.
Test images from top to bottom: “ErieLake” (321 × 455) and “ErieLake2” (321 × 455).
(a) The original images; (b) The original images with different green target regions; (c) and (d) are the inpainted images by the method in [23] and our method, respectively. The results demonstrate that the proposed method is robust to various target regions.
Fig 10.
Test images from top to bottom: “kicking” (213 × 343) and “sky” (210 × 407).
(a) The original images; (b) The original images with the red target regions; (c) Criminisi’s method “04’TIP” [23]; (d) The method “07’TPAMI” by Wexler et al. [18]; (e) Photoshop CS5 [16]; (f) The method “13’TIP” by Liu et al. [19]; (g) The proposed method. Note that the man in the first example is one of authors, he consents that the image is freely available. The individual in this manuscript has given written informed consent (as outlined in PLOS consent form) to publish these case details.
Table 1.
Computation time of Criminisi’s method [23] and our patch-in-patch method.
For simplicity, “Ready phase”(RP), “Compute gradients” (CG), “Get priorities” (GP), “Get exemplars” (GE), “Copy&Update” (CU). In particular, the phase “RP” of the proposed method includes the procedure of estimating steps. “ErieLake2” and “Curveline2” represent the computation time in the second row of Fig 9 and the second row of Fig 11, respectively. (Time unit: second.)
Table 2.
Computation time of different methods: Criminisi’s [23], “07’TPAMI” [18], Photoshop CS5 [16], “13’TIP” [19] and the proposed method.
Note that we can not measure the time of Photoshop CS5 accurately, but it is the fastest method. (Time unit: second.)
Fig 11.
Test images from top to bottom: “curveball” (105 × 233), “curveball2” (105 × 233) and “circle” (454 × 547).
(a) The original images; (b) The original images with the red target regions; (c) Criminisi’s method “04’TIP” [23]; (d) The method “07’TPAMI” by Wexler et al. [18]; (e) Photoshop CS5 [16]; (f) The method “13’TIP” by Liu et al. [19]; (g) The proposed method.
Fig 12.
(a) The inpainted image when the first phase uses C(p) and the second phase uses D(p) (first phase executes 120 steps); (b) The inpainted image when the first phase uses C(p)D(p) and the second phase uses D(p) (first phase executes 120 steps); (c) The inpainted image when the priority is determined only by confidence term C(p).
Fig 13.
Test images from top to bottom: “cross-shape” (110 × 317) and “triangle” (182 × 184).
(a) The original images; (b) The original images with the red target regions; (c) Criminisi’s method “04’TIP” [23]; (d) The method “07’TPAMI” by Wexler et al. [18]; (e) Photoshop CS5 [16]; (f) The method “13’TIP” by Liu et al. [19]; (g) The proposed method.