Overall pipeline.

Our analysis pipeline (Fig 2) followed a standard human activity recognition framework [26]. We used Mediapipe [20] for on-device pose estimation [13], preprocessed the poses, segmented them into overlapping 1-second analysis frames, extracted statistical, frequency time series features, and trained Support Vector Machine (SVM) [27], Random Forest (RF) [28], and Extreme Gradient Boosting (XGBoost) [29] classifiers to make predictions for each analysis frame. Video-level gait class predictions were made by aggregating frame-level predictions using majority voting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Overall video-based gait analysis pipeline and evaluation approach.

https://doi.org/10.1371/journal.pdig.0001004.g002

Preprocessing.

Pose estimation was conducted using MediaPipe [13,20], extracting the position of 33 keypoints across time from each video(Fig 2; 1. MediaPipe Pose Estimation). Each keypoint within a frame was represented by three channels: its x and y pixel coordinates, along with the estimated depth or z-axis distance from the camera. The pose estimator was run on-device, a setup allowing for only the detected poses to be uploaded to the server, ensuring that identifiable or sensitive data, such as the face or any nudity, would be safeguarded from potential data breaches. Frames where the participant was not visible or where no pose was detected were excluded from the analysis. For poses with partially missing keypoints due to occlusions (e.g., the arm facing away from the camera being obstructed by the torso), missing keypoints were linearly interpolated over time. All the pose sequences were downsampled to 30 frames per second (fps), as the majority of low-spec mobile phone cameras support 30 fps, improving the scalability of the developed pipeline.

Next, all poses were projected to a hip-center coordinate. This process removed potential biases in the model, arising from variations in the detected pose’s relative position to the camera. To preserve the natural sway of the hips during movement, we centered the keypoints using the median of the hip location from a 2-second (60 frames) window centered at the corresponding frame. For the frontal view, we re-scaled poses to the same height, removing perspective biases caused by changes in size due to the individual’s distance from the camera. Participants further away from the camera appeared smaller, which necessitated this re-scaling. For the sagittal view, we skipped this re-scaling process, as the participant’s perspective size and distance from the camera remained nearly constant throughout the video sequence. The centering and re-scaling processes were illustrated in Fig 2 (2. Preprocessing). Finally, the preprocessed pose sequences were segmented into 1-second windows (analysis frames) using a sliding window with 30 frames (1 second) and 50% overlap Fig 2 (3. Windowing). We chose a 1-second window following previous work in human activity recognition for gait classification [30].

Gait impairment video classification.

Extraction and selection of features.

For each 1-second analysis, we extracted time series features from each keypoint to characterize the gait window. Specifically, we used Time Series FeatuRe Extraction based on Scalable Hypothesis tests (TSFRESH) [31] Python package, a widely used time series feature extraction pipeline, to generate 783 features per channel [31]. In total, we extracted 77,517 features from 99 keypoint time series. These features were then reduced using the Fresh (FeatuRe Extraction and Scalable Hypothesis testing) algorithm [32], which eliminates time series features that are statistically insignificant to classification tasks. The Fresh algorithm utilizes the Benjamini-Yekutieli (BY) method [33], which identifies high dependencies or autocorrelation in time series features. Feature selection reduced the set of feature extraction methods used in the sagittal view to 31 and in the frontal view to 37, as included in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Comparison of selected feature types across sagittal and frontal views.

Legend:     Not relevant in Sagittal View as per TSFresh       Not relevant in Frontal View as per TSFresh

https://doi.org/10.1371/journal.pdig.0001004.t001

Data augmentation.

The different walking speeds observed across simulated gait classes and participants led to an imbalance in the number of segmented windows per gait class, as displayed in Fig 2 (5.2 Window-level Class Balancing). Class imbalance could potentially result in suboptimal performance of machine learning models [34]. To address this, we applied the Synthetic Minority Over-sampling Technique (SMOTE) [35], which increased the sample sizes of the minority gait classes (VAU, NOR, CIR, TRE, ANT, and CRO) to match those of the PAR gait class. SMOTE oversampled existing minority class examples by generating new synthetic samples by interpolating across their features, helping to mitigate class imbalance and improve model performance.

Video-level classification.

The augmented window-level features were input into several machine learning models previously used in gait research [36–38], which were SVM [27], RF [28], and XGBoost [29], to classify each window to a specific gait class. For single view video-level classification, we aggregated the labels inferred from each window in the video using a majority voting scheme. For multi-view video-level classification, majority voting was applied across all window-level labels from both frontal and sagittal views of the participant simulating the gait. The video-level aggregation was illustrated in Fig 2 (6. Video-level Aggregation).

Experiment and evaluation.

Cross validation and evaluation metrics.

To validate the proposed gait classification pipeline, we used a user-independent nested cross-validation approach following the standard methods in machine learning research [39,40]. This was to validate the generalizability of the trained model when tested on unseen participants, separate from the training set. First, we split the participant data using a Leave-One-Subject-Out (LOSO) cross-validation scheme, where each fold used data from one participant as the test set and the remaining participants’ as the training set (Outer Loop). Within each fold, the training set was further divided using a user-independent 5-fold cross-validation scheme, where 20% of the participants were used as the validation set and the remainder as the training set (Inner Loop). The inner loop was used to tune hyperparameters of each model (SVM, RF, and XGBoost), listed in Table 2, using an open-source package named Optuna [41]. The outer loop’s test set was used to derive the final model evaluation. The Fresh feature selection and SMOTE techniques were applied exclusively to the training set to prevent information leakage from the validation set (in the inner loop) and the test set (in the outer loop) during nested cross-validation. Model performance was evaluated using accuracy, F1 score, precision, and recall. To evaluate the statistical significance of the model performances, we applied ten trials of nested cross-validation and reported the average test scores across all splits, accompanied by 95% confidence intervals based on Z-type or normal approximation type confidence intervals [42].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Hyperparameters tuned for SVM, RF, and XGBoost models.

https://doi.org/10.1371/journal.pdig.0001004.t002

Multi-class and per-gait classification tasks.

We first studied the overall multi-class classification performance for all seven gait classes. Then, we studied the per-gait classification performance when considering one gait class as positive samples while considering all other six gait samples as negative samples. This was to understand which gait class was specifically challenging to identify when presented with other gait impairments. We only used XGBoost for the per-gait classification evaluation, which showed the best performance in the overall classification performance. Our evaluations were done for frontal view-only, sagittal view-only, and frontal and sagittal combined view models. The overall cross-validation scheme was illustrated in Fig 2 (5. LOSO Cross-validation, 5.1 TSFRESH Feature Selection, and 5.3 Model Training).

Feature importance analysis.

We applied permutation feature importance analysis to analyze the relevance of each feature for classifying the seven gait classes (NOR, CIR, TRE, ANT, CRO, PAR, and VAU). Permutation importance assessed the impact of each feature on the model’s performance by randomly shuffling the feature values and observing the change in the model’s performance [43]. A positive value indicates the model performance dropped and the feature is therefore important to the model. A negative value indicates the model performance increased and the feature maybe confusing the model. A zero value indicates the model performance is unaffected by the feature.

Concerning the statistical significance of feature importance, we also reported a 95% confidence interval for each feature importance score. We further analyzed the keypoint importance by summing up the feature importance score of all features belonging to each keypoint. This was to understand the overall importance of a particular joint movement for distinguishing different gait classes, agnostic to feature classes.

Model performance with only frontal view videos as model input.

Overall performance.

When given only frontal view input data, the best performing model was XGBoost. Conversely, the SVM model showed the lowest performance.

Per-class performance.

The XGBoost model with frontal video views only as inputs showed the highest F1 score for circumduction, F1 score ≥ 0.7 for CIR, CRO, PAR and NOR gaits, F1 score ≥ 0.6 for ANT and TRE gaits, and lowest F1 score for VAU gait.

Model performance with only sagittal view videos as input.

Overall performance.

XGBoost achieved the highest overall performance when only sagittal view data was given. Similar to frontal view, SVM showed the lowest performance.

Per-class performance.

XGBoost showed an F1 score of ≥ 0.90 for CRO and PAR gaits, an F1 score of ≥ 0.85 for NOR and VAU gaits, and lowest F1 score for ANT and TRE gaits.

Model performance with combined video view (both frontal and sagittal views) as input.

Overall performance.

XGBoost was the best performing model when combined frontal and sagittal views were used for input data. SVM demonstrated the lowest performance. Interestingly, the combined view model in SVM was the only one that performed worse than its single view sagittal counterpart.

Per-class performance.

XGBoost showed an F1 score of ≥ 0.9 for CIR, NOR, and PAR gaits, an F1 score of ≥ 0.8 for ANT, CRO, and TRE gaits, and the lowest F1 score for VAU gait. For XGBoost, the per-class performance in combined view models showed either improved or statistically similar performance (within confidence intervals) for six out of seven gait classes compared to the best single view model. The only class that performed worse in the combined view model was VAU, with a decrease in F1-score compared to the sagittal view.

Feature importance.

In Fig 3, the feature importance analysis for (a) frontal and (b) sagittal view is shown when using XGBoost for multi-class gait classification. The heatmap represents the overall importance of features across keypoint_channels (x-axis) and feature types (y-axis). The heatmap represents the top 20 keypoint_channels and top 20 feature types, ranked by cumulative permutation importance in XGBoost’s performance. The color gradient from white to red in the heatmap indicates the level of importance, with red representing higher significance, while white indicates insignificance to the model classification. Notably, in both heatmaps, most features and keypoint combinations show zero importance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Keypoint feature importance analysis for (A) frontal and (B) sagittal view when classifying seven gait types using XGBoost.

The x-axis shows the keypoint and channel (x, y, or z) and the y-axis shows the feature type. Darker color implies higher feature importance in the classification.

https://doi.org/10.1371/journal.pdig.0001004.g003

Frontal view videos.

The most relevant features when analyzing frontal view videos were ‘mean_n_absolute_max’ in the x-axis of the right_foot_index_toe; and the ‘cwt_coefficients’ and ‘quantile’ across several keypoints belonging to the hand (fingers), and face (eyes and nose). See Table 1 for definitions of frontal view feature types.

Sagittal view videos.

Among the features, ‘fft_coefficient’ for the right_ankle, left_knee and right_knee in the x-axis, which represents the anterio-posterior axis, and ‘sample_entropy’ for the right_ankle in the x-axis stand out as particularly influential, demonstrating high feature importance values. The x-axis of the left_knee and right_knee shows importance across numerous feature types, including: ‘linear_trend’, ‘change_quantiles’, ‘number_crossing_m’ and the earlier mentioned ‘fft_coefficient’. For definitions and the full list of sagittal view feature types see Table 1.

Keypoint importance.

Fig 4 shows the top 20 keypoint importance, based on permutation analysis, aggregating all feature importance values for the corresponding keypoints when using (a) frontal view and (b) sagittal view.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Anatomical keypoint importance analysis in (A) frontal view and (B) sagittal view for classifying seven gait types using XGBoost.

https://doi.org/10.1371/journal.pdig.0001004.g004

Frontal view videos.

Along the frontal view, we find that the x-axis of the right_foot_index is by far the most important, with more than the importance of the next most important keypoint, which is the frontal view y-axis of the left_index. We also notice a greater number of upper limb/body features being important. With the y-axis of left_index, left_pinky, left_wrist, and the x-axis of the right_shoulder all being in the top five of feature importance and scoring more than 0.025.

Sagittal view videos.

We find that only the top thirteen keypoints are important to the model from Fig 1(b). Lower limb keypoints like the x-axis of the left_knee, right_knee, right_ankle, and the y-axis of the left_heel are the top four in keypoint importance. The remaining seven keypoints in the top 20 are of negative importance and include only upper body and facial keypoints.

Machine learning models.

The results from Table 3 demonstrate that machine learning model performance varied significantly across different gait video views (frontal, sagittal, and combined frontal + sagittal) and machine learning model types.

XGBoost’s consistent outperformance of both SVM and RF likely stems from its ability to learn nonlinear decision boundaries through iteratively training ensemble classifiers to correct errors made by one classifier to another, making XGBoost robust at handling missing data and outliers [29]. Capturing discriminative patterns across seven gait classes is likely not linearly separable and requires learning non-linear decision boundaries, which is underscored by the poor convergence of linear-kernel SVMs during our experiments.

We applied the Radial Basis Function (RBF) kernel to the SVM after the failure of linear separation, but this kernel was insufficient to project the data into a linearly separable space. As a result, SVM exhibited significantly lower performance compared to XGBoost and RF [44]. Notably, while RF also uses ensemble classifiers, it lacks XGBoost’s iterative process that compensates for errors across classifiers, resulting in lower overall accuracy, a result consistent with prior research showing XGBoost generally outperforming RF in many classification tasks [44].

Frontal vs Sagittal vs Combined gait video data as model inputs.

Model performance varied significantly depending on the camera view used (frontal, sagittal, or combined). Models trained on sagittal videos outperformed frontal view models, agreeing with clinical practice, where the sagittal view of a patient’s gait provides greater information to a clinician when diagnosing gait impairments, especially for gait pathologies such as ANT, PAR, and CRO gait. Two gait classes (CIR and ANT) deviated from this trend, performing marginally better in frontal view, likely because their hallmark deviations are more readily visible from the frontal view.

The accuracy of markerless pose estimation relies heavily on detecting and tracking keypoints on the human body [13,14]. These keypoint landmarks serve as reference points that help the algorithm infer the positions of other body parts through skeletal modeling [45]. The mobile phone-based pose estimation model [13] is a lightweight alternative to cloud-server models [14,46], designed to fit the limited computing resources of a phone. Blazepose had a higher error rate in measuring keypoint locations and depth from the camera (z-axis values) compared to the models requiring heavy GPU resources available in cloud servers. We suspect that the lower performance of frontal view video models is due to the inaccuracy of depth information obtained from monocular cameras, which is crucial for evaluating gait in a frontal view. Supporting this, previous studies have shown that sagittal plane kinematics derived from pose estimation algorithms exhibit lower average errors compared to frontal plane counterparts in post-stroke and PAR gaits [12,15].

Inaccuracies in keypoint depth estimation may also contribute to the per-class performance differences observed between frontal and sagittal views. Most notably, detecting CIR requires precise measurement of the semi-circular movement of the lower limb, which is easily captured along the medio-lateral axis and difficult to capture in the sagittal view due to the distinguishing direction of the movement being perpendicular to the camera plane. In line with this, the per-class performance of CIR in the frontal view had a 0.229 higher per-class F1-score compared to the sagittal view model. Similarly, CRO gait is better analyzed by understanding the posture in the sagittal plane, which is poorly captured in the frontal view. This explains the superior performance of XGBoost’s sagittal view in comparison to the frontal view when detecting CRO. As discussed above, the reliability of captured keypoint position data varies between the frontal and sagittal views. By using majority voting on predictions from both the frontal and sagittal view models, the limitations of each view are compensated, leading to improved overall predictions. However, the combined view can be less effective for poorly performing models, such as the frontal view SVM, which has a < 0.40 F1-score, leading to significantly decreased overall prediction quality. This can also be seen in the poor combined view per-class performance of VAU within the XGBoost model, likely caused by the low per-class VAU F1-score of the frontal view model.

Features in frontal vs Sagittal views.

The notable features that demonstrated high significance in feature importance analysis were few, as shown in Fig 3. ‘mean_n_absolute_max’ measures the maximum deviation of the right foot from the body center, along the frontal view horizontal axis, during walking. This deviation is an important measure for the gaits captured from the frontal view, like CIR, which are characterized by abnormal deviation of the lower leg away from the body midline in the frontal plane. ‘cwt coefficients’ measure the frequency characteristics of a channel, enabling the identification of subtle changes or periodicities in gait cycle data that distinguish different gait classes. ‘quantiles’ is a statistical measure used to infer the nature of the distribution of points in any time series. ‘approximate_entropy’ quantifies the regularity and complexity of the gait cycle, enabling classifiers to differentiate between steady, predictable walking and irregular behaviors that may discriminate between specific gait impairments. ‘agg_linear_trend’ captures the overall directional movement of keypoint time series, such as forward hip progression or vertical knee oscillation, helping to identify consistent gait characteristics crucial for classification. Finally, ‘absolute_maximum’ measures the peak values of keypoint time series during the gait cycle, which are critical for distinguishing different walking speeds or intensities.

Along the sagittal plane video data, the features like ‘fft_coefficient’ and ‘sample_entropy’ were particularly influential in the x axes of the left knee, right knee, and right ankle. The importance of the ‘sample_entropy’ of the right ankle’s x-axis was the highest and ≥ 0.03. The ‘fft_coefficient’ and ‘sample_entropy’ provide detailed information on the frequency and regularity of movements, which may be vital for understanding motor control, and inform future research and the design of data-driven clinical decision-making algorithms for gait diagnosis. The standout features–such as ‘fft_coefficients’, ‘sample_entropy’, and ‘linear_trends’– demonstrate the importance of capturing both the frequency and complexity of movements across specific body parts. For example, ‘fft_coefficients’ are highly relevant for joints like the ankle and knee, indicating that repetitive or cyclic patterns in lower body motion play a significant role in the model’s predictions. These movements could correspond to gait frequency, or other repeated actions, all of which are important in gait analysis [47,48]. The entropy-based features, such as ‘sample_entropy’ and ‘binned_entropy’, highlight the need to capture the unpredictability and complexity in motion, suggesting that irregular or erratic movements may be key indicators of certain classes [49,50]. Overall, the model performances appear to leverage feature types that capture the frequency, complexity, and directional trends of movement, most of which have been shown to be meaningful in gait analysis and align with clinical intuition. Our results show that a model trained on features extracted without gait expert input also focuses on and prioritizes aspects of gait characteristics considered important in clinical practice.

Keypoints in Frontal vs Sagittal views.

Other than the right foot index, the frontal view had a very high permutation importance in the y-axis of the left index finger, the left pinky, and the left wrist, and the x-axis of the right shoulder.

As mentioned above, the high importance of the right foot keypoint can be attributed to its relevance in classifying gaits like CIR, where quantifying deviation of the lower limb from the midline of the body is very helpful [51,52]. Furthermore, upperlimb movement has also been shown to be important for balance, posture, and gait coordination in NOR gait [53]. The high importance of the contralateral y-axis upperlimb keypoints indicate their utility in capturing compensatory movements and imbalance in the gaits. The keypoint importance analysis revealed that the top five features in the sagittal view included lower limb keypoints, in order of importance, they are the x-axis of the left knee, right knee, and right ankle, followed by the y-axis of the left heel. It also includes the right ear’s x-axis with an importance > 0.02. The importance of keypoints in the lower body (e.g., ankles and knees) along with facial keypoints (as is the ear) suggests that the model is sensitive to not only lowerlimb positions but also the relative posture of the head. Lower limb movements often reflect changes in balance, gait, and physical effort, while keypoints of the head, along with the similarly important hip, are indicative of postural height or the height as measured during the subjects gait, which is closely tied to gait classes like CRO and PAR.