The answers to each point raised by reviewers and the changing details are as follows:

Response to Reviewer 1

General Comments:

“The problem of low infrared imaging resolution limits the application of UAVs in target recognition and other fields to some extent. To this end, the combination of high-resolution image acquisition and low-resolution image processing, and the design of UAV flight optimization path schemes, the preservation of image edge features and the improvement of spatial resolution, as well as the application of the SR method in the field of UAV target recognition play an active role. At the same time, it has promoted the application of UAVs in target detection. But before the paper is published, there are still some shortcomings that need further improvement. The author is suggested to read the full text carefully and make appropriate adjustments to the structure of the paper.”

Response:

We really appreciate the reviewer’s positive comments and recognition of the contribution of the manuscript. We revise the manuscript again based on the reviewers’comments and believe the revised manuscript is substantially improved.

1.Introduction, P1~P3 (answer to Reviewer #1, suggestion #1)

Comment:

“The introduction of the paper is relatively tedious for the relevant expressions of the basic theories. It is recommended that the author reduce the content of this part appropriately, or start a paragraph to explain the research basis of the paper. In the introduction of the paper, the content related to the research progress worldwide should be highlighted.”

Response:

Thanks for pointing out our shortcomings! In this revision, we reduce some relatively unnecessary content of expressions of the basic theories. Also we replace some explanations of the research basis with shorter and more concise expressions to highlight the content related to the research progress.

In the revised version:

The first three paragraphs are reduced as follows:

Imaging devices have limited achievable resolution due to several theoretical and practical restrictions. Different theories reveal that an optical imaging system acts as a low-band pass filter, sparing the low spatial frequencies of an object’s spectrum but cutting off high frequency information [1]. Some of the high-spatial frequency signal information will be lost almost completely, due to the finite size of the lens apertures. Knowing how to obtain a high-resolution (HR) image is still an important and fundamental research topic.

Paragraph 4, “A possible method to reduce the time cost is to maximize the height, which increases the viewing perspective. However, this approach will reduce the optical detail of image.” is adjusted to:

A possible method to reduce the time cost is to maximize the height, however, this will reduce the optical detail of image.

Add a short transitional paragraph on the top of Page.3:

Considering that the IR light has a longer wavelength than visible light, the IR cameras are able to detect targets beyond the visually distinguishable range. Besides, IR cameras provide excellent spatial correlation among neighborhood pixels. These characteristics make IR cameras widely popular when equipped by UAVs. For these reasons, we consider the reconstruction problem for IR cameras in this study.

2.the last paragraph of the Introduction, P3 (answer to Reviewer #1, suggestion #2)

Comment:

“In the last paragraph of the introduction of the paper, the author is suggested to add the research purpose and content, as well as express the research structure in detail.”

Response:

Thanks for your suggestion. We have introduced our research purpose in the penultimate paragraph (The aim of this study is to develop... (4)A geometric shaping of the reconstructed image can be systematically performed with the designed sparseness constraint condition.). In this revision, we add some explanations for the structure organization of this study.

In the last manuscript:

The rest of this paper is organized as follows. Section 2 presents the problem formulation. In Section 3, a three-stage SR method is proposed. Simulation and experimental results are shown in Section 4, followed by the conclusion in Section 5.

In the revised version:

The rest of this paper is organized as follows. The problem formulation is presented in Section 2, where we make a further explanation for the problem to be solved through simple mathematical models. In Section 3, a three-stage SR method is proposed and the workflow of our method is presented from the overall aspects and the detail respectively. Section 4 shows the simulation and experimental results to demonstrate the performance of the proposed method. Section 5 draws the conclusion of our study.

3.P2, Section 1, Paragraph 8 (answer to Reviewer #1, suggestion #3)

Comment:

“In the introduction of the paper, the expression ‘In our work, a lp norm-based regularization function is proposed against weak edge preservation of IR images, which fits the sparse characters of IR images and achieve fast-convergence performance’ appears. It is recommended that the author add the citation of references related to regularization functions here to support the research work of the paper.”

Response:

Thanks for this suggestion! In this revision, we add a reference, of which regularization was firstly applied in machine learning and has drawn wide concerns ever since, to the beginning of this paragraph.

In the last manuscript:

Different forms of regularization terms have been designed on the basis of certain requirements to solve the above ill-posed problem...

In the revised version:

Evgeniou [27] applied regularization to machine learning for the first time. After that different forms of regularization terms have been designed on the basis of certain requirements to solve the above ill-posed problem...

....

Reference

...

...

27. EvgeniouT, Pontil M, Poggio T. Regularization networks and support vector machines. Advances in Computational Mathematics. 2000; 13(1): 1-50

4.The full text (answer to Reviewer #1, suggestion #4)

Comment:

“For all the letter symbols appearing in the paper, the meanings of them should be explained, such as "Vk" and "Wk" in equation (1). Please check the full text and modify it.”

Response:

Thank you for your careful reading. We go through the entire manuscript once again to correct all the issues. We highlight all the changes in the manuscript.

In the revised version:

(1) P4, Section 2, Paragraph 3

① where denotes the blur matrix, represents the warp matrix, is the additive noise present in each image process.

② with the sensor and the sampling operator .

③ and is reduced to a small translation

(2) P8, Section 3.2, Paragraph 4

where and represent the Fourier Transform of g (x, y) and h (x, y) respectively, ∗ indicates the complex conjugate.

(3) P11, Section 3.3, Paragraph 4

where denotes downsampling, denotes the blur process, represents the warp matrix, is the additive noise.

5.P8, Section 3.2, Paragraph 9 (answer to Reviewer #1, suggestion #5)

Comment:

In the establishment of the three-stage SR method of the paper, the expression "Considering that singular value decomposition (SVD) is advanced in data simplification as well as noise elimination" appears. The author is suggested to add the citation of references in the corresponding field here to provide support for the subsequent research of the paper.

Response:

Thanks for your suggestion. We cite relative paper in our revision this time.

In the last manuscript:

Considering that singular value decomposition (SVD) is advanced in data simplification as well as noise elimination

In the revised version:

Considering that singular value decomposition (SVD) [35] is advanced in data simplification as well as noise elimination

....

Reference

...

...

35.De L, L DMB, Vandewalle J. A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications. 2000; 21(4): 1253-1278.

....

Response to Reviewer 2

General Comments:

“This paper proposes a method combining high-level image acquisition and low-level SR processing. During the integration process, UAV path optimization, sub-pixel image registration and sparse constraints are combined into the computational imaging framework of the region of interest (ROI). At the same time, the optimal flight control scheme is designed to optimize ROI complementary feathers and obtain sufficient image sequences from multiple angles. In particular, the phase correlation method is used to achieve reliable sub-pixel image feature matching. On this basis, an effective sparse regularization model is established to enhance the fine structure of infrared images. Experimental results of actual infrared sequences show that the algorithm has good performance in edge preservation and detail enhancement, and provides a very promising computational imaging method for obtaining high-resolution images. The paper should make the following amendments before publication:”

Response:

Thank you very much for your comprehensive comments and constructive suggestions. We read and consider each comment very carefully, and thoroughly revise the manuscript according to your comments and suggestions. We hope that the manuscript reads more convincingly after the revision.

1.Abstract, P1 (answer to Reviewer #2, suggestion #1)

Comment:

“The abstract is a relatively independent part. Readers only need to read the abstract section to understand all the research contents. Therefore, the abstract is a general description of the full text. The writing of the abstract of the paper lacks logic. The author needs to express the research purpose, methods, results and conclusions logically. Most content of the abstract in the paper is about the background of the research. But the description of the results of the study only mentions "Experimental results with real-life IR sequences indicate encouraging improvements by using our method." It is very simple. In addition, the abstract lacks relevant descriptions of the research conclusions. It is recommended that the author rewrite the abstract to reflect the substantive content of the paper.”

Response:

Thanks for your suggestion. According to the problems you point out, we rewrite the abstract to highlight the substantive content of our work.

In the last manuscript:

Abstract—Despite the rapid development of infrared (IR) imaging technology, IR images still generally suffer from low spatial resolution, weak contrast, and low signal-to-noise ratio. Super- resolution (SR) technology provides a far promising computational imaging approach in obtaining a high-resolution image (or image sequences) from observed multiple low-resolution (LR) images by incorporating complementary information. Traditional multiple-frame SR algorithms usually focus on signal processing and achieve good performances when using standard test datasets. However, practical LR IR image sequences collected by unmanned aerial vehicles (UAVs) can hardly provide enough complementary information of the same scene without any flight control strategy of the UAV. In this paper, a method is proposed to integrate a high-level image capturing process and a low-level SR process. In the integrated process, we incorporate UAV path optimization, sub-pixel image registration, and sparseness constraint into a computational imaging framework of a region of interest (ROI). To refine ROI complementary feathers, we design an optimal flight control scheme to acquire adequate image sequences from multi-angles. In particular, a phase correlation approach achieving reliable sub- pixel image feature matching is adapted, on the basis of which an effective sparseness regularization model is built to enhance the fine structures of the IR image. Experimental results with real-life IR sequences indicate encouraging improvements by using our method.

In the revised version:

Abstract—Super-resolution (SR) technology provides a far promising computational imaging approach in obtaining a high-resolution (HR) image (or image sequences) from observed multiple low-resolution (LR) images by incorporating complementary information. In this paper, a three-stage SR method is proposed to generate a HR image from infrared (IR) LR Images acquired with Unmanned Aerial Vehicle (UAV). The proposed method integrates a high-level image capturing process and a low-level SR process. In this integrated process, we incorporate UAV path optimization, sub-pixel image registration, and sparseness constraint into a computational imaging framework of a region of interest (ROI). To refine ROI complementary feathers, we design an optimal flight control scheme to acquire adequate image sequences from multi-angles. In particular, a phase correlation approach achieving reliable sub-pixel image feature matching is adapted, on the basis of which an effective sparseness regularization model is built to enhance the fine structures of the IR image. Unlike most traditional multiple-frame SR algorithms that mainly focus on signal processing and achieve good performances when using standard test datasets, the performed experiments with real-life IR sequences indicate the three-stage SR method can also deal with practical LR IR image sequences collected by UAVs. The experimental results demonstrate that the proposed method is capable of generating HR images with good performance in terms of edge preservation and detail enhancement.

2.Introduction, P1~P3 (answer to Reviewer #2, suggestion #2)

Comment:

“In the introduction, the author should explain the research significance and innovation more clearly.”

Response:

Thank you for your comment. We rewrite the abstract introduction in our revision. To be specific, we reduce some relatively unnecessary content of expressions of the basic theories. Also we replace some explanations of the research basis with shorter and more concise expressions to highlight the research significance and innovation more clearly

In the revised version:

(1)The first three paragraphs are reduced as follows:

Imaging devices have limited achievable resolution due to several theoretical and practical restrictions. Different theories reveal that an optical imaging system acts as a low-band pass filter, sparing the low spatial frequencies of an object’s spectrum but cutting off high frequency information [1]. Some of the high-spatial frequency signal information will be lost almost completely, due to the finite size of the lens apertures. Knowing how to obtain a high-resolution (HR) image is still an important and fundamental research topic.

(2)Paragraph 4, “ A possible method to reduce the time cost is to maximize the height, which increases the viewing perspective. However, this approach will reduce the optical detail of image. ” is adjusted to:

A possible method to reduce the time cost is to maximize the height, however, this will reduce the optical detail of image.

(3)Add a short transitional paragraph on the top of Page.3:

Considering that the IR light has a longer wavelength than visible light, the IR cameras are able to detect targets beyond the visually distinguishable range. Besides, IR cameras provide excellent spatial correlation among neighborhood pixels. These characteristics make IR cameras widely popular when equipped by UAVs. For these reasons, we consider the reconstruction problem for IR cameras in this study.

(4)We add some explanations for the structure organization of this study:

the last paragraph of Introduction is adjusted to:

The rest of this paper is organized as follows. The problem formulation is presented in Section 2, where we make a further explanation for the problem to be solved through simple mathematical models. In Section 3, a three-stage SR method is proposed and the workflow of our method is presented from the overall aspects and the detail respectively. Section 4 shows the simulation and experimental results to demonstrate the performance of the proposed method. Sections 5 draws the conclusion of our study.

3.Conclusion, P18 (answer to Reviewer #2, suggestion #3)

Comment:

“The conclusion should include a summary of the research content, deficiencies and limitations, as well as put forward a future outlook or research direction. However, the conclusion in this paper only summarizes the research results, and does not reflect the deficiencies, limitations, the future prospects or the research direction. Please confirm and modify accordingly.”

Response:

Thank you for your suggestion. According to the problems you point out, we modify the conclusion and add some necessary content.

In the last manuscript:

A sparse representation based super-resolution framework for on-board high resolution IR imaging is proposed in this study to integrate a high-level image capturing and a low-level super-resolution process. A path-optimal UAV flight control method is designed to acquire sufficient multi-angle image sequences, serving for small targets detection. Subsequently, a sub-pixel image registration algorithm is developed to eliminate pixel deviation. In particular, a sparseness constraint mechanism is established in accordance with the textural features of the images. The results demonstrate that the algorithm is capable of generating HR images with good performance in terms of edge preservation and detail enhancement. To our knowledge, our proposed method is one of the first methods used for simultaneous UAV control and resolution enhancement.

In the revised version:

A sparse representation based super-resolution framework for on-board high resolution IR imaging is proposed in this study to integrate a high-level image capturing and a low-level super-resolution process. A path-optimal UAV flight control method is designed to acquire sufficient multi-angle image sequences, serving for small targets detection. Subsequently, a sub-pixel image registration algorithm is developed to eliminate pixel deviation. In particular, a sparseness constraint mechanism is established in accordance with the textural features of the images. The results demonstrate that the algorithm is capable of generating HR images with good performance in terms of edge preservation and detail enhancement. To our knowledge, our proposed method is one of the first methods used for simultaneous UAV control and resolution enhancement.

Although the proposed method has achieved satisfactory results, this work is just the beginning to implement this automated approach in IR image SR reconstruction. In this work, the UAV is assumed to achieve steady attitude by proposed optimal control methods. However, external disturbances (e.g., wind field) would inevitably induce displacement, jitter and blurring of images. In further works, we will develop a motion deblurring algorithm incorporated into our research, and we are also interested in evaluating the effect of AOV (angle of view) on SR results.

4.REFERENCES, P19~P20 (answer to Reviewer #2, suggestion #4)

Comment:

“Reference information should be complete (author, title, periodical name, year of publication, volume, period, page number). It is noted that the three basic information of publication year, volume and page number are required. The information of some references in the paper are incomplete and the format is relatively confusing, especially the author information. Some surnames are first, some names are first, some are abbreviated, and some are not abbreviated. Please check and modify it. In addition, there are many references that are relatively old. The author should cite more references for the last 3 to 5 years.”

Response:

Thank you for your careful reading. We go through the entire manuscript once again to correct and unify all the reference information according to the criterion of PLOS ONE. Moreover, we add more references for the last 3 to 5 years in the manuscript according to your valuable suggestion.

In the last manuscript:

REFERENCES

1.Sun, T., Liu, J., Yan, H., “Super-resolution reconstruction based on incoherent optical aperture synthesis,” Optics Letters, Vol. 38, No. 17, 3471–3474, 2013.

….

….

….

30.X. Wang, Y. Ke, D. Chao, and C. L. Chen, “Recovering realistic texture in image super-resolution by deep spatial feature transform,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

In the revised version:

1.Sun T, Liu J, Yan H. Super-resolution reconstruction based on incoherent optical aperture synthesis. Optics Letters. 2013; 38(17): 3471–3474.

2.Kok KY, Rajendran P. Differential-evolution control parameter optimization for unmanned aerial vehicle path planning. PLOS ONE. 2016; 11(3): e0150558.

3.Luo H, Liang Z, Zhu M, Hu X, Wang G. Integrated optimization of unmanned aerial vehicle task allocation and path planning under steady wind. PLOS ONE. 2018; 13(3): e0194690.

4.Lai HS, Wang F, Li Y, Jia B, Liu L. Super-resolution real imaging in microsphere-assisted microscopy. PLOS ONE. 2016; 11(10): e0165194.

5.Obuchi T, Ikeda S, Akiyama K, Kabashima Y. Accelerating cross-validation with total variation and its application to super-resolution imaging. PLOS ONE. 2017; 12(12): e0188012.

6.Shi X, Garcia G III, Wang Y, Reiter JF, Huang B. Deformed alignment of super-resolution images for semi-flexible structures. PLOS ONE. 2019; 14(3): e0212735.

7.Clark JJ, Palmer M, Lawrence PD. A transformation method for the reconstruction of functions from non-uniformly spaced samples. IEEE Transactions on Acoustics, Speech, and Signal Processing. 1985; 33(5): 1151-1165.

8.Kim SP, Bose NK. Reconstruction of 2-d band limited discrete signals from non-uniform samples. Radar and Signal Processing, IEE Proceedings F. 1990;137(3): 197-204.

9.Huang TS, Tsai RY. Multiple-frame image restoration and registration. Advances in Computer Vision and Image Processing. 1984; 24(10): 317–339.

10.Kim SP, Bose NK. Recursive reconstruction of high resolution image from noisy undersampled multiframes. IEEE Transactions on Acoustics, Speech, and Signal Processing. 1990; 38(10): 1013–1027.

11.Kim SP, Su WY. Recursive high-resolution reconstruction of blurred multiframe images. IEEE Transactions on Image Processing. 1993; 2(10): 534–539.

12.Zhao S, Han H, Peng S. Wavelet-domain HMT-based image super-resolution. Proceedings of IEEE International Conference on Image Processing. 2003; 2: 983-956.

13.Chappalli MB, Bose NK. Simultaneous noise filtering and super-resolution with second-generation wavelets. IEEE Signal Processing Letters. 2005; 12(11): 772-775.

14.Demirel H, Anbarjafari G. Image resolution en hancement by using discrete and stationary wavelet de composition. IEEE Transactions on Image Processing. 2011; 20(5): 1458-1460.

15.Hong M, Kang MG, Katsaggelos AK. An iterative weighted regularized algorithm for improving the resolution of video sequences. International conference on image processing. 1997.

16.K Kang MG. Generalized multichannel image deconvolution approach and its applications. Optical Engineering. 1998; 37(11): 2953-2964.

17.Hardie RC, Barnard KJ, Bognar JG, Armstrong EE, Watson EA. High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system. Optical Engineering, 1998; 37(1): 247-260.

18.Bose NK, Lertrattanapanich S, Koo J. Advances in super-resolution using L-curve. international symposium on circuits and systems. 2001.

19.Lin Z, Shum H. Fundamental limits of reconstruction-based super-resolution algorithms under local translation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2004; 26(1): 83-97.

20.Schultz RR, Stevenson RL. Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing. 1996; 5(6): 996-1011.

21.Hardie RC, Barnard KJ, Armstrong EE. Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Transactions on Image Processing.1997; 6(12): 1621-1633.

22.Tian J, Ma KK. Stochastic super-resolution image reconstruction. Journal of Visual Communication and Image Representation. 2010; 3(21): 232-244.

23.Chainais P, Koenig E, Delouille V, Hochedez JF. Virtual super resolution of scale invariant textured images using multi-fractal stochastic processes. Journal of Mathematical Imaging and Vision. 2011; 39(1): 28-44.

24.Sroubek F, Cristobal G, Flusser J. A unified approach to superresolution and multichannel blind deconvolution. IEEE Transactions on Image Processing. 2007; 16(9): 2322-2332.

25.Shen H, Zhang L, Huang B, Li P. A MAP Approach for Joint Motion Estimation, Segmentation, and Super Resolution. IEEE Transactions on Image Processing. 2007; 16(2): 479-490.

26.Hu Y, Lam K, Qiu G, Shen T. From Local Pixel Structure to Global Image Super-Resolution: A New Face Hallucination Framework. IEEE Transactions on Image Processing. 2011; 20(2): 433-445.

27.Evgeniou T, Pontil M, Poggio T. Regularization networks and support vector machines. Advances in Computational Mathematics. 2000; 13(1): 1-50.

28.Henningsyeomans PH, Baker S, Kumar BV. Simultaneous super-resolution and feature extraction for recognition of low-resolution faces. Computer vision and pattern recognition. 2008.

29.Ng MK, ShenH, Lam EY, & Zhang L. A total variation regularization based super-resolution reconstruction algorithm for digital video. Journal on Advances in Signal Processing. 2007; 2(2007): 074585.

30.Chan TF, Ng MK, Yau AC, Yip AM. Superresolution image reconstruction using fast inpainting algorithms. Applied & Computational Harmonic Analysis. 2007; 23(1): 3-24.

31.Aziz SA, Steward BL, Tang L, Karkee M. Utilizing repeated GPS surveys from field operations for development of agricultural field dems. Transactions of the Asabe. 2019; 52(4): 1057-1067.

32.Dong C, Loy CC, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis & Machine Intelligence. 2016; 38(2): 295-307.

33.Solomon O, Mutzafi M, Segev M, Eldar YC. Sparsity-based super-resolution microscopy from correlation information. Optics Express. 2018; 26(14): 18238.

34.Wang X, Yu K, Dong C, Loy CC. Recovering realistic texture in image super-resolution by deep spatial feature transform. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

35.De L, L DMB, Vandewalle J. A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications. 2000; 21(4): 1253-1278.

Attachments

Attachment

Submitted filename: Response to Reviewer.docx