https://orcid.org/0000-0002-2313-5309
Figures
Abstract
Complex steerable pyramid (CSP) performs well when applied to magnify subtle motions of structures for observing the dynamic characteristics of facilities. However, the impact of the types and parameters of CSP filters upon the performance of phase-based optical flow (PBOF) in measuring motion parameters has not been systematically studied. The purpose of this study is to comprehensively evaluate the impact of different CSP filter types (Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave) and parameters on the performance of PBOF in measuring motion parameters. Firstly, by measuring simulated translational motion, the influence of the CSP’s down-sampling rates on the displacement measurement accuracy of PBOF is analyzed to determine appropriate settings. Subsequently, the effective displacement measurement interval and accuracy of PBOF using the CSP are studied through simulated and experimental translational motion measurements. Further, the vibration parameter’s accuracy is analyzed through simulated periodic vibration measurements. Finally, the characteristics of PBOF using the four kinds of CSP and practical considerations are discussed. Simulation and experimental results demonstrate that when using middle-level filters within the effective level range of HalfOctave, PBOF achieves the best overall displacement measurement performance. Additionally, this method can easily integrate with signal processing techniques in analyzing structural dynamic characteristics under field conditions.
Citation: Peng Z, Wang X, Wang Z, Liu W, Liu M (2024) Evaluation of the impact of filter types and parameters upon the accuracy of phase-based optical flow method with a complex steerable pyramid. PLoS ONE 19(9): e0308943. https://doi.org/10.1371/journal.pone.0308943
Editor: Parsa Ghannadi, Islamic Azad University, ISLAMIC REPUBLIC OF IRAN
Received: February 28, 2024; Accepted: August 1, 2024; Published: September 4, 2024
Copyright: © 2024 Peng et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are available on Figshare: https://doi.org/10.6084/m9.figshare.25971616.v1.
Funding: This research was funded by the National Key Research and Development Program of China (Grant No. 2022YFF0608700), the Major Landmark Innovation Demonstration Project in Hunan Province (Grant No. 2022XK2301), the National Natural Science Foundation of Hunan Province (Grant No. 2024JJ8273 and 2018JJ3167), the Open Research Project of the National Key Laboratory of Offshore Wind Power Generation Equipment and Efficient Wind Energy Utilization (Grant No. HFQZS2024-14), and the Visiting Scholar Program of China Scholarship Council (Grant No. 202008430103). All funds were received by Xian Wang. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
The displacement signals, such as vibration, dynamic deflection, and dynamic deformation, generated during the operational processes of equipment and structures contain rich inherent structural features and system state information. These signals form the foundational data for condition monitoring, safety assessment, life prediction, and structural optimization. Consequently, structural displacement measurement technology has consistently remained a focal point of attention [1].
In recent years, when compared with the contact measurement method [2–4] and the non-contact measurement method [5–7] based on the theory of light wave interference and laser Doppler effect, the rapidly growing visual displacement measurement technology [8–11] has emerged as the most prominent technology in the field of structural displacement measurement due to its significant advantages. These advantages include high efficiency, low equipment costs and site layout costs, and high spatial resolution in measurement.
Visual displacement measurement methods can be divided into two categories: methods based on image grayscale and methods based on image phase [12]. Image grayscale is obtained through the camera sensor’s perception of the object’s illumination intensity, while phase is acquired by convolving the image with a pair of orthogonal complex-valued filters. The methods based on image grayscale have been widely used in structural displacement measurement [13, 14], such as the feature point tracking [15, 16], the digital image correlation (DIC) [17, 18], and the gradient-based optical flow (GBOF) [19, 20]. For example, Lydon et al. [21] and Zhu et al. [22] proposed a combination of feature points and GBOF to measure the dynamic displacement of structures. Wei et al. [23] proposed a 3D-DIC method to measure the full-field dynamic displacement of the cassette structure.
In contrast to the GBOF, the phase-based approach demonstrates enhanced resistance to interference and greater adaptability to changes in illumination and viewing angles [24, 25]. Since Wadhwa et al. [26] proposed the phase-based motion magnification technique using the complex steerable pyramid (CSP) filter decomposition image algorithm [27, 28] to observe subtle motions, the image phase-based displacement measurement method has gained increasing attention. The CSP filter decomposition image algorithm was proposed by Wadhwa et al. when studying video motion magnification. This method decomposes images to obtain the spatial phase information of different frequency domain sub-bands and ensures that there is no aliasing among the frequency domain sub-bands, so that high-quality motion magnified images can be reconstructed. The core principle of video motion magnification is to use the CSP filter decomposition image algorithm to obtain the motion phases of the video, magnify the phases and reconstruct the original video to visualize small displacements. In order to further increase the magnification factor of motion magnified videos and improve image quality, Wadhwa et al., based on Octave CSP, successively proposed HalfOctave, SmoothHalfOctave and QuarterOctave CSP. Among these CSPs, which use narrower filter bandwidths to achieve higher magnification, SmoothHalfOctave and QuarterOctave CSP use smoother filters to improve the quality of motion magnified images. Chen et al. [29] first proposed combining phase-based optical flow (PBOF) with CSP to measure the vibration of a cantilever beam. In the PBOF using CSP filter, the CSP filter decomposition image algorithm is used to obtain the motion phases of images. Based on the phase constancy assumption, PBOF calculates the displacement of the object from the obtained motion phase. This process does not involve the magnification or reconstruction of any motion phase of a video. Therefore, unlike motion magnification of a video, although PBOF also uses the CSP filters decomposition image algorithm to obtain the motion phases of images, it does so to accurately measure the displacement, not just visualize small displacements. Then, various techniques have been developed to utilize the image phase obtained through CSP decomposition for motion estimation, structural health monitoring, and defect detection of structures. Cosco et al. [30] proposed a defect detection method for vibrating plates based on PBOF using CSP. Jana et al. [31] realized the practical estimation of the real-time vibration frequency of the bridge by combining phase-based motion estimation using CSP with the signal blind source separation method.
A deep understanding of the performance characteristics of various visual displacement measurement methods is the basis for successfully using these methods in engineering practice. Diamond et al. [32] evaluated the micro-displacement measurement accuracy of PBOF and analyzed the impact of the measured pixel position and image down-sampling ratio upon the PBOF measurement accuracy. Based on the above work, Collier et al. [33] studied the interval of effective displacement measurement of PBOF and analyzed the impact of image contrast and bit depth upon the range of effective displacement measurement. The above fruitful studies provide a better understanding of the characteristics of the PBOF algorithm to a certain extent and help promote the application of the PBOF algorithm. However, to the best of the author’s knowledge, there is currently no systematic study on the impact of CSP filter types and parameters on the performance of PBOF in measuring motion parameters, and such an absence leads to a lack of reliable basis in engineering practice for selecting CSP filter types and parameters according to specific circumstances.
The purpose of this study is to comprehensively evaluate the impact of different CSP filter types (Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave) and parameters on the performance of PBOF in measuring motion parameters. This evaluation is conducted through simulations of translational motion, periodic vibration, and experimental translational motion. The evaluation metrics of the measurement performance of motion parameters include the effective displacement measurement interval and accuracy, and the accuracy of measuring the vibration frequency. In addition, the characteristics of PBOF using four kinds of CSP and the problems that need attention in practical applications are discussed.
2 Theories
2.1 Overall framework
The implementation of the PBOF using CSP in this study is based on reference [29], and the overall framework of this method is shown in Fig 1. First, the spatial domain amplitude and phase information of each frame of the input image sequence in the x and y directions are extracted by using the complex filter Gθ + iHθ with rotation angles θ being 0 and π/2 respectively in the CSP at different image scales:
(1)
In the Eq (1) , I(x,y,tj) is the grayscale of the pixel when the coordinates are (x,y) at time tj; * is convolution computation; R(x,y,tj) is the spatial domain response of the image to the complex filter Gθ + iHθ in the θ direction at time tj, and Aθ (x,y,tj) and ϕθ (x,y,tj) are the spatial domain amplitude and phase response respectively.
The complex filter rotation angle θ = 0.
According to the assumption of constant phase, the phase of the same pixel remains unchanged in adjacent images:
(2)
The Taylor series expansion of is carried out at (x,y,t), and the first-order derivative approximation is taken. Eq (2) is transformed into:
(3)
The gradient of ϕθ (x,y,t) at the spatial coordinates (x,y),
(4)
where R*(x,y,t) is the conjugate of R(x,y,t);
is the gradient of R(x,y,t) at the spatial coordinates (x,y) and the operator Im[•] is the imaginary part of the complex number. The partial derivative
of ϕθ to the time variable t can be obtained by using forward difference:
(5)
Let the instantaneous velocity vector of the pixel at the spatial coordinates (x,y) be v = (vx,vy)Τ and then Eq (3) can be transformed into:
(6)
When θ = 0, the partial derivative of ϕ0 (x,y,t) to the coordinate variable y is . When θ = π/2, the partial derivative of ϕπ/2(x,y,t) to the coordinate viriable x is
. The following formula can be obtained by substituting the above conditions into Eq (6) respectively:
(7)
By multiplying the velocity vector v with the time interval between the corresponding two frames Δt, the displacement vector d of the pixel at (x,y) can be obtained:
(8)
where dx and dy are the displacement of the spatial coordinates (x,y) along the x and y directions.
According to the literature [34, 35], when using the PBOF to obtain the full-field measurement displacement of all frames, the pixels I(x,y,t) in the image may randomly exhibit a non-linear phase response, leading to significant measurement errors at that point. To address this issue, literature [34, 35] proposes using non-linear phase response confidence to filter pixels for motion estimation. This method is implemented when an adequate number of frames are available. Therefore, the above method cannot select pixel positions that can be accurately measured for PBOF in a single frame measurement. In order to eliminate the potential interference of pixel position selection on the measurement accuracy of PBOF, this study adopts the method from reference [33] to obtain the final measurement result of PBOF in a single frame image. When using PBOF with a CSP filter, for each frame image, n measurement values d will be obtained, where n represents the total number of pixels in the single frame image. According to the three-sigma rule, values within the μ ± 3σ interval are selected as effective measurement values from the n measurement values, and the average value m of the effective measurement values is taken as the final measurement result of PBOF for the single-frame image.
2.2 Complex steerable pyramid
In the overall framework of PBOF using CSP, the function of the CSP is to decompose the image so that the multi-scale spatial domain phase information in the directions of θ = 0 and θ = π/2 can be obtained (corresponding to step 2 of Fig 1). There are four types of CSP including Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave [26]. The processes of decomposing images of these four types of CSP are similar, while the difference lies in the high-pass filter, low-pass filter, and oriented band-pass filter used in the image decomposition process. In the CSP filter decomposition image algorithm, the oriented band-pass filters play a core role. The oriented band-pass filters of the four different types of CSP at level 1 and rotation angle 0 are shown in Fig 2.
(a) Octave. (b) HalfOctave. (c) SmoothHalfOctave. (d) QuarterOctave.
2.2.1 Octave CSP.
Similar to the original Octave pyramid filter image decomposition algorithm [36], the principle of the Octave CSP filter image decomposition algorithm is to recursively multiply the high-pass filter H0 (r,θ), low-pass filter L0 (r,θ), and oriented band-pass filter Bk (r,θ) at different frequency domain scales with the image frequency domain. The corresponding time domain operation is shown in Eq (1), and the overall process is shown in Fig 3.
When Octave CSP is used for PBOF, its down-sampling rate can be set to 0 or 2. However, when Octave CSP is used for video motion amplification, as shown in Fig 3, its down-sampling rate can only be set to 2. In Fig 3, the number of orientations of the Octave CSP’s oriented band-pass filter K = 4 and the number of pre-decomposition scales P1 are calculated as:
(9)
where min(•) represents the minimum function; [•] is the integral function while H and W represent the length and width of images respectively. The calculation formulas of H0 (r,θ) and L0 (r,θ) are as follows:
(10)
(11)
where r and θ are the polar coordinates of the image frequency domain (r,θ), r∈R,θ∈(-π,π]. The calculation formulas of H(r,θ) and L(r,θ) are as follows:
(12)
(13)
The calculation formula of Bk (r,θ) is:
(14)
(15)
(16)
In Eq (14), H(r,θ) and Gk (θ) respectively control the amplitude-frequency characteristics and rotation direction of the Octave CSP oriented band-pass filter.
2.2.2 HalfOctave CSP.
When HalfOctave CSP is used for PBOF, its down-sampling rate can be set to 0 or 1.41. In the HalfOctave CSP, the number of orientations K = 8; the number of pre-decomposition scales P2 = P1×2, and the radial bandwidths of the high-pass filter H1 (r,θ) and the low-pass filter L1 (r,θ) are 1/2 of Octave CSP’s H0 (r,θ) and L0 (r,θ). The calculation formulas are as follows:
(17)
(18)
Among them, the calculation formulas of H11 (r,θ) and L11 (r,θ) are as follows:
(19)
(20)
The differences in oriented band-pass filters between the HalfOctave CSP and the Octave CSP are shown in Fig 2(A) and 2(B). Specifically, the bandwidth and the number of orientations of the oriented band-pass filter Bk1 (r,θ)of HalfOctave CSP are 1/2 and 2 times those of Bk (r,θ) of Octave CSP, respectively. The calculation formula of Bk1 (r,θ) is:
(21)
Comparing Eqs (14) and (21), it can be seen that compared with the oriented band-pass filter of Octave CSP, the HalfOctave CSP oriented band-pass filter has a narrower bandwidth under the control of H11 (r,θ) and has more rotatable directions under the setting of the number of rotation directions K.
2.2.3 SmoothHalfOctave CSP.
When SmoothHalfOctave CSP is used for PBOF, its down-sampling rate can be set to 0 or 1.41. In the SmoothHalfOctave CSP, the number of orientations K = 8; the number of pre-decomposition scales P3 = P1×2, and the radial bandwidths of the high-pass filter H2 (r,θ) and the low-pass filter L2 (r,θ) are 1/2 of Octave CSP’s H0 (r,θ) and L0 (r,θ) with smoother amplitude-frequency characteristic. The calculation formulas are as follows:
(22)
(23)
The calculation formulas of H21 (r,θ) and L21 (r,θ) are as follows:
(24)
(25)
The differences in oriented band-pass filters between the SmoothHalfOctave CSP and the Octave CSP are shown in Fig 2(A) and 2(C). Specifically, the bandwidth and the number of orientations of the oriented band-pass filter Bk2 (r,θ) of SmoothHalfOctave CSP are 1/2 and 2 times those of Bk (r,θ) of Octave CSP, respectively, and the amplitude-frequency characteristic of Bk2 (r,θ) is smoother. The calculation formula of Bk2 (r,θ) is:
(26)
Comparing Eqs (21) and (26), it can be seen that the oriented band-pass filter of the SmoothHalfOctave CSP has a smoother amplitude-frequency characteristic under the control of H21 (r,θ) compared to the oriented band-pass filter of the HalfOctave CSP.
2.2.4 QuarterOctave CSP.
When QuarterOctave CSP is used for PBOF, its down-sampling rate can be set to 0 or 1.19. In the QuarterOctave CSP, the number of orientations K = 8; the number of pre-decomposition scales P4 = P1×4, and the radial bandwidths of the high-pass filter H3 (r,θ) and the low-pass filter H3 (r,θ) are 1/4 of Octave CSP’s H0 (r,θ) and L0 (r,θ) with smoother amplitude-frequency characteristic. The calculation formulas are as follows:
(27)
(28)
where the calculation formulas of H31 (r,θ) and L31 (r,θ) are as follows:
(29)
(30)
The differences in oriented band-pass filters between the QuarterOctave CSP and the Octave CSP are shown in Fig 2(A) and 2(D). Specifically, the bandwidth and the number of orientations of the oriented band-pass filter Bk3 (r,θ) of QuarterOctave CSP are 1/4 and 2 times those of Bk (r,θ) of Octave CSP, respectively, and the amplitude-frequency characteristic of Bk3 (r,θ) is smoother. The calculation formula of Bk3 (r,θ) is:
(31)
Comparing Eqs (26) and (31), it can be seen that the oriented band-pass filter of QuarterOctave CSP has a narrower bandwidth under the control of H31 (r,θ) compared to the oriented band-pass filter of SmoothHalfOctave CSP.
3 Methodology
In this study, the impact of different CSP filter types (Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave) and parameters on the performance of PBOF in measuring motion parameters is evaluated using simulations and experiments. The process is shown in Fig 4. Firstly, through measuring the simulated translational motion, the influence of CSP’s down-sampling rates on the displacement measurement accuracy of the PBOF is analyzed to select the appropriate settings for subsequent research. Next, the effective displacement measurement interval and accuracy of PBOF using the CSP are studied by measuring the simulated and experimental translational motion. Then, the vibration parameter measurement accuracy of PBOF using the CSP is analyzed by measuring the simulated periodic vibration. Finally, the comprehensive performance and characteristics of PBOF using the CSP are discussed.
3.1 Performance evaluation metrics
In the simulation and experiment of translational motion measurement (section 3.2), the relative error (RE) and absolute error (AE) are employed to assess the accuracy of the final measured displacement m obtained by PBOF using a CSP filter on a per-frame basis. The calculation formulas for RE and AE are as follows:
(32)
(33)
In the formula, r represents the true displacement of a single frame image. The RE metric can simultaneously reflect the errors at different scales of displacement, while the AE metric intuitively reflects the magnitude of measurement errors. The effective displacement measurement interval described in this paper pertains to the interval of displacements where the RE is less than 10%.
Time-domain and frequency-domain evaluation metrics are used to assess the overall accuracy of the measured vibration displacement curves (section 3.3). The time-domain evaluation metrics include the correlation coefficient R2 between measured and actual vibration displacement, as well as the vibration amplitude absolute error (VAAE) and the root mean square error (RMSE). The frequency-domain evaluation metric uses the maximum frequency amplitude absolute error (MFABE) to quantify the absolute error between the measured and actual vibration frequencies. The calculation formulas for RMSE, VAAE, R2, and MFABE are as follows:
(34)
In the formula, mi is the displacement measurement of the i-th frame; ri is the true displacement of the i-th frame; N is the number of all measurement frames.
In the formula, mmax is the maximum measured displacement over N frames; rmax is the maximum true displacement over N frames.
In the formula, um is the mean of the measured displacements over N frames, σm is the standard deviation of the measured displacements over N frames; ur is the mean of the true displacements over N frames, and σr is the standard deviation of the true displacements over N frames.
In the formula, |P(fM)| and |P(fR)| respectively represent the frequency magnitude of the measured displacement and the frequency magnitude of the true displacement.
3.2 Measurement of translational motion of simulation and experiment
3.2.1 Analysis of the influence of down-sampling rates of the CSP.
In this section, a simulation speckle reference image is generated with a resolution of 140×140 pixels, as depicted in Fig 5, following the recommended settings of the Glare algorithm [37]. Then, the Binning method [38] is applied with a step size of 0.1 pixel, consecutively shifting the speckle reference image along the positive x direction ten times. This process creates displacements ranging from 0 to 1 pixel.
The down-sampling rate of the CSP, which affects the grayscale noise intensity of images, the spatial reach of the CSP filter, and the dimensions of the output spatial domain response map at each CSP filter level, significantly influences the measurement accuracy of PBOF. Thus, applying optimal CSP filters with varying down-sampling rates is employed to measure the displacement of speckle image sequences through PBOF. The influence of down-sampling rates on PBOF’s displacement measurement accuracy is analyzed to determine appropriate settings for subsequent investigations by comparing the average RE and average AE.
3.2.2 Analysis of effective displacement measurement interval and accuracy.
The data used in this section comes from two parts: the simulated data generated in this section and the experimental data shared in reference [39]; they provide displacements as depicted in Fig 6. The experimental data contains 45 consecutive sub-pixel rigid body translation speckle images with an image size of 1600×1200 pixels and an exposure time of 1000us, its reference speckle image is depicted in Fig 7. Since the experimental data can provide a maximum displacement of 1.1 pixel, assessing the maximum effective measurement displacement of PBOF using the CSP is impossible. Therefore, the maximum displacement in the simulated data is set to 9 pixels, more significant than the upper displacement limit in most applications. Additionally, the size of the simulated speckle reference image is consistent with the experimental data, and the remaining configurations are the same as those in section 3.2.1.
Firstly, at the optimal down-sampling rate, the displacement of image sequences in the simulation is measured by PBOF using CSP, and RE and AE are calculated for each frame. According to the evaluation criteria in Section 3.1, the effective displacement measurement interval is analyzed. Then, based on the average value of RE within both the micro-displacement interval (0~1 pixels) and the effective measurement interval, the measurement accuracy of PBOF using CSP within these two ranges is analyzed.
Subsequently, applying the same methodology as described above, an analysis is performed on the minimum effective measurement displacement and the accuracy within the range of 0 to 1.1 pixels using experimental data. Finally, the experimental results are compared with the simulation results in this section.
3.3 Measurement of parameters of simulation vibration
In applying vibration displacement measurement and structural modal analysis, more attention is paid to the variation trend of displacement measurement and the accuracy of frequency domain parameter measurement instead of the accuracy of displacement measurement itself. Therefore, the purpose of the simulation in this section is to analyze the accuracy of PBOF using the CSP to measure vibration parameters.
In this section, the simulated speckle reference image in Section 3.2.2 undergoes sinusoidal vibration displacement with an amplitude of 1 pixel along the x direction, a vibration frequency of 5 Hz, a sampling frequency of 120 Hz, and a sampling time of 1 second, respectively. One hundred twenty frames of vibration image sequences are generated, and the corresponding provided displacements are illustrated in Fig 8.
After the vibration image sequence is generated, the displacement of the vibration image sequence is measured by PBOF using the CSP at the optimal down-sampling rate. Then, the calculations for R2, VAAE, RMSE, and MFABE between the measured vibration displacement and the actual vibration displacement are conducted, and the accuracy of the PBOF the CSP to measure the vibration parameters is analyzed.
4 Results
4.1 Hardware and software platform and pyramid decomposition scale
The hardware platform configuration that runs the simulation and experimental program in this section is dual Intel (R) Xeon (R) Gold 6139M CPU, dual Nvidia GTX3080 GPU, and 128GB memory. Under the Windows 10 operating system, the software algorithm is implemented using the Matlab 2021 (a) platform.
When measuring the displacement of the simulated and experimental image sequence with PBOF using four types of CSP, each CSP has multiple pre-decomposition scales, and different pre-decomposition scales can independently obtain a motion parameter measurement result in different directions. Since the displacement of the simulated and experimental image sequences in this paper is along the x direction, the oriented band-pass filter with a rotation angle of 0 when pyramids are on different pre-decomposition scales is selected to measure the displacement. Therefore, P1, P2, P3, and P4 different measurement results can be obtained respectively with PBOF using Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave. According to the introduction of the calculation method of the decomposition layers of CSP in Section 2.2, for the image data provided in Section 3.2.1, the decomposition layers of each type of CSP are: P1 = 5, P2 = P3 = 10, and P4 = 20; for the image data provided in Sections 3.2.2 and 3.3, the decomposition layers of each type of CSP are: P1 = 8, P2 = P3 = 16, and P4 = 32, respectively.
According to the introduction in Section 2.2 regarding the down-sampling rates that can be set for CSP when applied to PBOF, the optional down-sampling rates d for Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave CSP are 2 and 0, 1.41 and 0, 1.41 and 0, 1.19 and 0, respectively. When d is not equal to 0, the spatial domain response image size of the oriented band-pass filter at the lth level is 1/d as large as that of the oriented band-pass filter at the l-1th level. The higher the level of the oriented band-pass filter is, the lower the spatial resolution of the spatial domain response image is. When d is equal to 0, the spatial resolution of the spatial domain response image of the oriented band-pass filter at each level is always the same as that of the original image.
4.2 Analysis of the influence of down-sampling rates of CSP
When employing the simulated data from Section 3.2.1, the AE and the average value of RE across all frames obtained by PBOF using the most accurate filters from each CSP under various down-sampling rates are illustrated in Fig 9. According to the introduction of the optional down-sampling rate of CSP in Section 4.1, the non-zero down-sampling rates of Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave CSP in Fig 9 are 2, 1.41, 1.41, and 1.19, respectively.
(a) The value of AE across all frames. (b) The average value of RE across all frames. HO, SHO, and QO respectively represent HalfOctave, SmoothHalfOctave, and QuarterOctave.
As shown in Fig 9(A), when the down-sampling rate of CSP is zero, the value of AE is reduced by over five times compared to when the down-sampling rate is non-zero. Simultaneously, in Fig 9(B), the average value of RE is reduced by over seven times. It is evident that with a down-sampling rate of zero, the displacement measurement accuracy of PBOF is significantly higher than when the rate is non-zero. This is because, according to the principle of CSP filter decomposition of images, when the down-sampling ratio of CSP is not zero, as the filter level increases, the resolution of the measured image will decrease, which will have an adverse effect on the displacement measurement performance of PBOF [32], whereas when the down-sampling ratio of CSP is zero, the increase in the filter level will not affect the edge resolution, in which case, the displacement measurement performance of PBOF is maximized [24].
In summary, because each category of CSP has only one non-zero down-sampling rate, and the measurement accuracy of PBOF is significantly higher when the down-sampling rate is zero compared to when it is non-zero, the down-sampling rates for CSP used in subsequent analyses are all set to zero.
4.3 Analysis of effective displacement measurement interval and accuracy
4.3.1 Simulation results.
When employing the simulated data described in Section 3.2.2 and applying the evaluation criteria for measurement validity described in Section 3.1, the minimum value of effective measurement displacement (EMD) results obtained by PBOF using the effective level filters of each CSP are presented in Fig 10. Most of each CSP’s effective level filters can achieve effective measurements of the 0.1-pixel micro-displacement simulated in the simulation. However, the micro-displacement sensing capability is limited when using the 13th and 14th level filters of the SmoothHalfOctave CSP, as well as the 9th, 23rd, and 24th level filters of the QuarterOctave CSP.
When using the effective level filters of CSP, the maximum value of EMD obtained by PBOF is shown in Fig 11. Within the micro-displacement interval (MDI) and the effective displacement measurement interval (EDMI), the average value of AE and the average value of RE of PBOF across all measurement frames using the effective level filters of CSP are shown in Fig 12. In Fig 11, it can be seen that no matter which CSP is used, the maximum value of EMD first increases with the increase of the filter level, and the displacement measurement ability is the strongest at about 1/2 of the effective filter level range of the CSP, which can realize the effective measurement of the maximum displacement (9 pixel) that can be simulated in the simulation. After that, the maximum value of EMD decreases with the increase of filter level.
(a) Octave. (b) HalfOctave. (c) SmoothHalfOctave. (d) QuarterOctave.
It can also be seen from Fig 12 that the variation trend of the average value of RE corresponding to each CSP is almost the same as that of the maximum value of EMD. In other words, when employing the relatively middle-level filters of each CSP, PBOF demonstrates higher accuracy. Hence, when using the middle-level filter of each CSP, PBOF exhibits the most considerable displacement measurement interval, the highest measurement accuracy, and the best performance.
The premise for accurate measurement by PBOF with a CSP filter is to ensure that the motion phases obtained are not distorted [34]. As shown in Fig 10, when the level 13 and 14 filters of SmoothHalfOctave CSP and the level 9, 23, and 24 filters of QuarterOctave CSP are used, their micro-displacement perception ability is poor. The reason is that when the displacement of the image is smaller than the minimum effectively measurable displacement perceivable by the above filters, they are not able to obtain the accurate motion phase. Fig 11 shows that when the level 5 filter of Octave CSP, the level 8 filter of HalfOctave CSP, the level 10 filter of SmoothHalfOctave CSP, and the level 16–18 filters of QuarterOctave CSP are used, the maximum effectively measurable displacement of PBOF is 9 pixels. Analogously, this is because when the displacement of the image is 9 pixels, the above filters are still able to accurately obtain the motion phase. It is shown in Fig 12 that in the case of various CSPs, when filters located about 1/2 of the range of effective levels are used, PBOF is able to take measurements with higher accuracy in both the range of effectively measurable displacement and the range of micro-displacement measurement. This is because the filters at about 1/2 of the range of effective levels of various CSPs are able to obtain the motion phases more accurately in both ranges of measurable displacement.
When using the most accurate filters of Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave CSP respectively, the RE of each frame within the effective measurement range obtained by PBOF is shown in Fig 13. The values on the vertical axis of Fig 13 represent the relative error of PBOF when measuring a single frame image. In the 0~5 pixels displacement interval, the values of RE corresponding to the four types of CSP are less than 2%, and the single-frame measurement accuracy is relatively high. However, in the 5~9 pixels displacement interval, the single-frame measurement accuracy corresponding to each CSP decreases rapidly with the displacement increase, with a total decrease of about 6%. It can be seen from Fig 13 that no matter which CSP filter is used, under the simulation conditions herein, the displacement accuracy of PBOF will be significantly reduced when the displacement is greater than 5 pixels. This is because an important premise for accurate measurement by PBOF is that the small motion assumption can be established, allowing Eq (2) to be approximately expanded using the first-order Taylor formula [40]. When the displacement is larger, the small motion assumption can no longer be established, in which case it is impossible to guarantee the PBOF accuracy.
Furthermore, as indicated by Fig 13, the variation trend of RE of the single-frame displacement measurement obtained by the most accurate filter of each CSP is almost the same. However, in different displacement intervals, according to the average value of RE corresponding to each CSP, the difference in accuracy between the CSP with the highest accuracy and that with the lowest accuracy is relatively significant. Therefore, when the displacement interval changes, the characteristics of each CSP can be fully utilized to select the appropriate one according to the displacement interval to improve the accuracy.
4.3.2 Experimental results.
When utilizing the experimental data described in Section 3.2.2, the minimum value of EMD obtained by PBOF using the effective level filters of the four types of CSP is shown in Fig 14. It can be observed that PBOF using the most level filters of the four types of CSP can achieve effective measurements of the minimum displacement (0.026 pixel) provided by the experiment. However, for the 2nd and 6th level filters of the Octave CSP, the 4th, 11th, and 12th level filters of the HalfOctave CSP, the 14th and 15th level filters of the SmoothHalfOctave CSP as well as the 9th~10th, 22nd, and 24th~27th level filters of the QuarterOctave CSP, the micro-displacement sensing ability is limited.
In the displacement interval provided by the experiment, the average value of RE obtained by PBOF using the effective level filters of the four types of CSP is shown in Fig 15. It can be seen from Fig 15 that no matter which CSP is used, the average value of RE decreases sharply with the increase of the filter level. The micro-displacement measurement accuracy is the highest at about 1/2 of each CSP’s effective filter level range. After that, the average value of RE gradually increases with the increment of filter levels. The filter level has a significant effect on the accuracy of micro-displacement measurement. When using the middle-level filter of each CSP, PBOF has the highest measurement accuracy.
The simulation results in Section 4.3.1 are highly consistent with the experimental results in this section. It can be concluded that: (I) PBOF using effective level filters of each CSP demonstrates a strong capability for perceiving micro-displacements. When using the QuarterOctave CSP, the largest number of filters with strong displacement perception ability can be obtained. (II) When PBOF uses the different effective level filters of the same CSP, the measurement accuracy will change significantly, and the highest measurement accuracy can be obtained when the middle-level filter is used.
4.4 Vibration parameter measurement results
When employing the effective level filters of each CSP, the error metric results for VAAE, RMSE, and MFABE obtained from PBOF measurements of displacement in the simulated periodic vibration motion data of Section 3.3 are presented in Table 1. Notably, in the MFABE metric, measured and actual vibrations’ maximum frequency amplitude values are at 5 Hz. Additionally, all effectively measured displacements exhibit a correlation coefficient (R2) of 1 with the actual displacements. It can be seen from Table 1 that no matter which CSP is used, the three kinds of measurement errors first decrease with the increase of the filter level. The measurement accuracy of the vibration parameters is the highest at about 1/2 of the effective filter level range of the CSP. Then, the three kinds of measurement errors increase with the increase of filter level. These rules are almost consistent with those manifested in the simulation and experiment of displacement measurement in Section 4.3.
5 Discussion
5.1 The characteristics of the four types of CSP
The simulation and experimental results of this paper clearly show the characteristics of PBOF using the four types of CSP, which are as follows:
- Whether when measuring the micro-displacement (0~1 pixels) and large-scale displacement (5~9 pixels) or in the simulated entire displacement interval (0~9 pixels), the accuracy that can be achieved by using the earliest proposed Octave CSP is at the lowest level among the four types of CSP.
- The comprehensive performance of PBOF using the HalfOctave CSP is the best. The HalfOctave CSP is the highest in the micro-displacement measurement accuracy and the average accuracy of the entire displacement interval.
- When measuring micro-displacement, the accuracy obtained using the SmoothHalfOctave CSP is relatively low. The accuracy achieved using the SmoothHalfOctave CSP is medium when measuring large-scale displacement. The overall measurement accuracy obtained using the SmoothHalfOctave CSP is low in the simulated entire displacement interval.
- When measuring micro-displacement or in the simulated full displacement interval, the accuracy obtained using the QuarterOctave CSP is at a medium level. However, a higher accuracy can be obtained when measuring large-scale displacement using the QuarterOctave CSP.
5.2 Problems that need attention in practical applications
No matter which pyramid is used, PBOF using the optimal level filter is high in accuracy and has the most considerable effective displacement measurement interval. Therefore, finding and using the optimal level filter when PBOF using the four types of CSP is applied in engineering practice is essential. In addition, the law of translation motion displacement measurement in Section 4.3 is almost the same as that of vibration parameter measurement in Section 4.4. Therefore, for the vibration parameter measurement occasions that do not require a direct measure of displacement, the selection of CSP types and filter levels should be based on their capability of accurately measuring the displacement caused by vibration.
Although PBOF using all CSPs can achieve effective measurement (with RE being less than 10%) in the displacement interval of the simulation, the RE corresponding to the four pyramids is less than 2% with higher accuracy when the displacement is less than 5 pixel (Fig 13). Therefore, the displacement measurement interval can be controlled below 5 pixels as far as possible through system design in practical applications.
According to the characteristics of PBOF using different CSPs, the HalfOctave pyramid can be given priority in the cases of considerable displacement variation and micro-displacement measurement. PBOF using this CSP has the highest overall accuracy in the whole effective displacement measurement interval simulated in this study, and the accuracy is also the highest when measuring the micro-displacement of less than 1 pixel. In large-scale displacement measurement occasions, the QuarterOctave CSP can be given priority, and PBOF using this CSP has the highest accuracy when measuring large-scale displacement.
6 Conclusions
The CSP performs well when applied to phase-based Euler motion amplification, but the impact of its filter types and parameters on motion parameter measurement performance hasn’t been systematically studied. In this paper, the performance of the PBOF using Octave, HalfOctave, SmoothHalfOctave, and QuarterOctave CSP to measure motion parameters is systematically evaluated from three aspects: effective displacement measurement interval, accuracy within effective displacement measurement interval, and vibration parameter measurement accuracy. The main conclusions are as follows:
- The optimal level filter of each of the four types of CSP is at about 1/2 of the effective level range. When using the optimal level filter, PBOF has the highest accuracy and the most considerable effective displacement measurement interval.
- Among the four types of CSP, PBOF using the HalfOctave CSP has the best comprehensive performance and can be the first choice for most applications.
- As for the vibration parameter measurement occasions that do not require directly measuring the displacement, the selection of CSP types and filter levels should also be based on their capability to measure the displacement generated by the vibration accurately.
PBOF using the CSP is a relatively new visual displacement measurement method. At present, the optimization of this method lags behind that of other mainstream visual displacement measurement methods, and there is still much room for improvement in terms of overall framework design, sub-pixel estimation, etc. Meanwhile, as mentioned in the literature [31], this type of phase-based motion estimation using the CSP could be easily integrated with signal processing methods, such as blind source separation, to realize the analysis of structural dynamic characteristics under field conditions. The displacement measurement accuracy of PBOF is not only affected by the types and parameters of CSP filters, but also by factors such as visual marker type, image noise, and image contrast. In-depth study of the impact of these factors upon the displacement measurement accuracy of PBOF will be also the direction of the next step.
References
- 1. Sony S, Laventure S, Sadhu A. A literature review of next‐generation smart sensing technology in structural health monitoring. Structural Control and Health Monitoring. 2019;26(3):e2321.
- 2. Ghaffar A, Li Q, Haider SA, Sun A, Leal-Junior AG, Xu L, et al. A simple and high-resolution POF displacement sensor based on face-coupling method. Measurement. 2022;187:110285.
- 3. Li S, Liang Z, Guo P. A FBG pull-wire vertical displacement sensor for health monitoring of medium-small span bridges. Measurement. 2023:112613.
- 4. Zheng Y, Li Z, Yu J, Xiao J. Experimental research on a novel spring-shaped fiber-optic displacement sensor for settlement monitoring. Measurement. 2022;191:110754.
- 5. Sun B, Zheng G, Zhang X. Application of contact laser interferometry in precise displacement measurement. Measurement. 2021;174:108959.
- 6. Chen D-M, Zhu W. Investigation of three-dimensional vibration measurement by three scanning laser Doppler vibrometers in a continuously and synchronously scanning mode. Journal of Sound and Vibration. 2021;498:115950.
- 7. Shi H, Shi J, Yang Y, Tang S, Su C, Liu L. Spatial Vibration and Displacement Measurement Through Single-Point Continuous and Interval Scanning Laser Doppler Vibrometer. IEEE Transactions on Instrumentation and Measurement. 2022.
- 8. Song Q, Wu J, Wang H, An Y, Tang G. Computer vision-based illumination-robust and multi-point simultaneous structural displacement measuring method. Mechanical Systems and Signal Processing. 2022;170:108822.
- 9. Wang M, Ao WK, Bownjohn J, Xu F. A novel gradient-based matching via voting technique for vision-based structural displacement measurement. Mechanical Systems and Signal Processing. 2022;171:108951.
- 10. Zhu Q, Cui D, Zhang Q, Du Y. A robust structural vibration recognition system based on computer vision. Journal of Sound and Vibration. 2022;541:117321.
- 11. Lu B, Bai B, Zhao X. Vision-based structural displacement measurement under ambient-light changes via deep learning and digital image processing. Measurement. 2023:112480.
- 12. Miao Y, Kong Y, Nam H, Lee S, Park G. Phase-based vibration imaging for structural dynamics applications: Marker-free full-field displacement measurements with confidence measures. Mechanical Systems and Signal Processing. 2023;198:110418.
- 13. Dong C-Z, Catbas FN. A review of computer vision–based structural health monitoring at local and global levels. Structural Health Monitoring. 2021;20(2):692–743.
- 14. Feng D, Feng MQ. Computer vision for SHM of civil infrastructure: From dynamic response measurement to damage detection–A review. Engineering Structures. 2018;156:105–17.
- 15. Jiang X, Ma J, Xiao G, Shao Z, Guo X. A review of multimodal image matching: Methods and applications. Information Fusion. 2021;73:22–71.
- 16. Ma J, Jiang X, Fan A, Jiang J, Yan J. Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision. 2021;129:23–79.
- 17. Pan B. Digital image correlation for surface deformation measurement: historical developments, recent advances and future goals. Measurement Science and Technology. 2018;29(8):082001.
- 18. Zhao J, Sang Y, Duan F. The state of the art of two‐dimensional digital image correlation computational method. Engineering reports. 2019;1(2):e12038.
- 19. Horn BK, Schunck BG. Determining optical flow. Artificial intelligence. 1981;17(1–3):185–203.
- 20. Lucas BD, Kanade T, editors. An iterative image registration technique with an application to stereo vision. IJCAI’81: 7th international joint conference on Artificial intelligence; 1981.
- 21. Lydon D, Lydon M, Taylor S, Del Rincon JM, Hester D, Brownjohn J. Development and field testing of a vision-based displacement system using a low cost wireless action camera. Mechanical Systems and Signal Processing. 2019;121:343–58.
- 22. Zhu J, Lu Z, Zhang C. A marker-free method for structural dynamic displacement measurement based on optical flow. Structure and Infrastructure Engineering. 2021;18(1):84–96.
- 23. Wei K, Yuan F, Shao X, Chen Z, Wu G, He X. High-speed multi-camera 3D DIC measurement of the deformation of cassette structure with large shaking table. Mechanical Systems and Signal Processing. 2022;177:109273.
- 24. Fleet DJ, Jepson AD. Computation of component image velocity from local phase information. International journal of computer vision. 1990;5(1):77–104.
- 25. Tomasi M, Barranco F, Vanegas M, Díaz J, Ros E. Fine grain pipeline architecture for high performance phase-based optical flow computation. Journal of Systems Architecture. 2010;56(11):577–87.
- 26. Wadhwa N, Rubinstein M, Durand F, Freeman WT. Phase-based video motion processing. ACM Transactions on Graphics (TOG). 2013;32(4):1–10.
- 27. Heeger DJ, Bergen JR, editors. Pyramid-based texture analysis/synthesis. Proceedings of the 22nd annual conference on Computer graphics and interactive techniques; 1995.
- 28. Portilla J, Simoncelli EP. A parametric texture model based on joint statistics of complex wavelet coefficients. International journal of computer vision. 2000;40(1):49–70.
- 29. Chen JG, Wadhwa N, Cha Y-J, Durand F, Freeman WT, Buyukozturk O. Modal identification of simple structures with high-speed video using motion magnification. Journal of Sound and Vibration. 2015;345:58–71.
- 30. Cosco F, Cuenca J, Desmet W, Janssens K, Mundo D. Towards phase-based defect detection: A feasibility study in vibrating panels. Journal of Sound and Vibration. 2022;537:117196.
- 31. Jana D, Nagarajaiah S, Yang Y. Computer vision‐based real‐time cable tension estimation algorithm using complexity pursuit from video and its application in Fred‐Hartman cable‐stayed bridge. Structural Control and Health Monitoring. 2022;29(9):e2985.
- 32. Diamond D, Heyns P, Oberholster A. Accuracy evaluation of sub-pixel structural vibration measurements through optical flow analysis of a video sequence. Measurement. 2017;95:166–72.
- 33. Collier S, Dare T. Accuracy of phase-based optical flow for vibration extraction. Journal of Sound and Vibration. 2022;535:117112.
- 34. Miao Y, Jeon JY, Kong Y, Park G. Phase-based displacement measurement on a straight edge using an optimal complex Gabor filter. Mechanical Systems and Signal Processing. 2022;164:108224.
- 35. Gautama T, Van Hulle M. A phase-based approach to the estimation of the optical flow field using spatial filtering. IEEE transactions on neural networks. 2002;13(5):1127–36. pmid:18244509
- 36. Briand T, Vacher J, Galerne B, Rabin J. The Heeger & Bergen pyramid based texture synthesis algorithm. Image processing on line. 2014;4:276–99.
- 37. Su Y, Zhang Q. Glare: A free and open-source software for generation and assessment of digital speckle pattern. Optics and Lasers in Engineering. 2022;148:106766.
- 38. Reu PL, Toussaint E, Jones E, Bruck HA, Iadicola M, Balcaen R, et al. DIC challenge: developing images and guidelines for evaluating accuracy and resolution of 2D analyses. Experimental Mechanics. 2018;58(7):1067–99.
- 39. Baldi A, Santucci PM, Bertolino F. Experimental image dataset for validation of the noise-induced bias that affects Digital Image Correlation. Data in Brief. 2022;42:108156. pmid:35496488
- 40. Bruhn A, Weickert J, Schnörr C. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International journal of computer vision. 2005;61:211–31.