Fig 1.
Our proposed method consists of two distinct steps: upper line (A to E), deriving sequential gait feature data from movies via body joint coordinate extraction using OpenPose; lower line, estimating cadence from the sequential data using the short-time autocorrelation function and the subsequent analysis (F to I).
Table 1.
Video data and subjects’ characteristics.
Fig 2.
Detailed feature extraction process using the OpenPose application.
The detailed process corresponds to the upper line in Fig 1. CASIA Dataset-B contains gait sequences of the same gait recorded from different angles simultaneously: (A) laterally viewed angle; and (B) frontally viewed angle. Flow (A-1) to (A-4) shows the process for laterally viewed movies, while flow (B-1) to (B-4) shows that for frontally viewed movies. After estimating keypoints (A-2, B-2), joint coordinate data of each frame are converted to gait features: leg angle (A-3) and difference in bilateral leg-length ratio (B-3). Then sequential waveform data are obtained (A-4, B-4) to calculate gait cycle frequency in the subsequent steps. Note that 1 cycle in the sequential LRdiff (B-3) corresponds to 2 walked steps (= 1 gait cycle), while 1 cycle in the sequential LRang (A-3) corresponds to 1 walked step (= half of 1 gait cycle), so the raw frequency of sequential LRang in laterally viewed movies is multiplied by 0.5 before being converted to a gait frequency.
Fig 3.
Short-time ACF used to calculate cadence of a normal gait sequence from CASIA Dataset-B.
(A) The filtered LRdiff sequence of control gait movies from CASIA Dataset-B. For these normal gait movies, we applied ST-ACF with a window length of 2 seconds and a shift-length of 0.01 second (A). In ST-ACF (D), the reciprocal of the lag at the peak of the second positive phase after the initial negative peak (white arrow in D) was selected as the representative frequency. When the sequential ST-ACF matrix is plotted on a heatmap (E), the selected lag is seen as the second from the top and most red horizontal line (white arrow in E). STFT (B, C) is shown as a corresponding example to the ST-ACF: sequential STFT is plotted as a heatmap (C), where the minimum value is white and the maximum color is most-red. The selected frequency just corresponds to the height of the horizontal most-red line (filled arrow in C).
Fig 4.
Applied result of cadence calculation for gait sequences of a mildly affected PD patient.
(A) and (B) show the graphical summary of the sequential ST-ACF obtained from the gait sequences of the same PD patient (Pt. 1) recorded in our hospital’s hallway before (A, 5.5 sec) and after (C, 3.8 sec) DBS treatment (frontally viewed movies only). Representative gait frequency is calculated as the reciprocal of the median lag at the second positive peak in each moment ST-ACF (= most red color on the heatmaps) (arrowheads in A, B). The calculated gait cadence was 140.5 steps/min before DBS and 134.1 steps/min after DBS.
Fig 5.
Example of cadence calculation for a significantly abnormal gait sequence.
(A) Filtered sequential ‘LRdiff’ of gait sequence of our PD patient’s (Pt. 2), whose gait mixed with significant FOG and small steps. The movie is 14 seconds long, the time she took to walk less than 2 m. Many instances of freezing and involuntary oscillations were observed during the gait. The sequential ST-ACF of 11 seconds’ duration (subtracting window length of 3 seconds from the raw movie length) obtained from (A) is shown as (B). (C)–(F) show the ST-ACF at the corresponding tC − tF timings in (A, B). (C)–(F) demonstrate examples in which accurate gait frequency detection is confirmed (D, E) or doubtful (C, F) when applying the same procedure used for the normal gait movies as shown in Fig 3.
Fig 6.
Example of identifying periodic gait steps among the mixed FOG and metrics comparison between different gait sequences.
(A) Data obtained from the same patient (Pt. 2) as in Fig 5. In reference to the ST-ACF of control gait that is randomly selected from CASIA Dataset-B sequences, the similarity of ST-ACF at each moment to that of normal gait is measured in terms of DTW-based distance (second from bottom row in A) or kNN-based anomaly (bottom row in A). As long as the same reference and preprocessing configurations are used, the DTW- or kNN-based distance are comparable between different gait sequences as summarized in boxplots (B) for DTW and in boxplots (C) for kNN. In (B) and (C), there was a significant difference in statistical distance/anomaly value among the 3 subgroups: control gait, affected gait from Pt. 1 (Fig 4), and severely affected gait from Pt. 2 (one-way analysis of variance; p < 2.2e-16). Regarding the difference between control gait sequence and gait from Pt. 1 (Fig 6D), discrimination performance was better for DTW (area under the curve [AUC], 0.957) than for kNN (AUC, 0.754) (p < 2.2e-16 for DeLong’s test). Meanwhile, when comparing gait sequences before versus after DBS treatment from Pt. 1 (Fig 4A and 4C), discrimination performance was better for kNN-based distance (AUC, 0.980) than for DTW-based distance (AUC, 0.572) (p < 2.2e-16 on DeLong’s test) (Fig 6E).