Data collection and calibration

High quality datasets play a crucial role in training high-performance and highly generalizable machine learning models. In this study, we used the data described in reference [8]: The First Affiliated Hospital of Nanjing Medical University and Nanjing University of Technology Hospital collaboratively conducted raw data collection and calibration work from Dec 29, 2016 to Dec 28, 2017: a total of 85 volunteers were recruited, including 69 COPD patients and 16 normal individuals.

The 6-min walk test (6MWT) conducted in this study is a routine clinical examination item, and the pulse oxygen detector (pulse oximeter wristwatch) is also a non-invasive detection instrument used in clinical practice. We fully informed the volunteers of the testing process, precautions, and related risks during the trial, and obtained their informed verbal consents. We also assured volunteers that all collected data will be encrypted and stored to ensure the security of medical data and the protection of personal privacy. In addition, all collected data must undergo desensitization before being used for research, including this study. After review by the ethics committee of The First Affiliated Hospital of Nanjing Medical University (ethical review number: 2015-SR-207.A1), this study is exempt from obtaining informed written consent of volunteers.

By wearing a pulse oximeter wristwatch (model: ORANGER, CMS50S) on the wrists of each volunteer, we collected their six-minutes’ walk data mainly including two attributes: saturation of pulse oxygen (SpO2) and pulse rate (PR). The original raw data of each volunteer is a matrix which consists of six columns, i.e., Timestamp (timestamp), SpO2 value (blood oxygen saturation index, scalar), PR value (pulse rate, scalar), IsValid (indicating whether the current data is valid, True or False), Quality (indicating the quality of the current data, True or False), IsEvent (indicating whether the corresponding data point is a candidate EIOD point, True or False), and IsEIOD (indicating whether the current data point is a true EIOD point, True or False). A data point (being labeled as IsEIOD = 1) is valid and make sense if and only if IsValid = True, Quality = True, and IsEvent = True, while a data point (being labeled as IsEIOD = 0) is valid and make sense if and only if IsValid = True, Quality = True, and IsEvent = False. For the collected raw data, it is first manually calibrated by senior medical staff to filter and remove abnormal data points. After the screening and calibration steps, the six-minutes’ walk data of each volunteer can be simplified into a four-column matrix, denoted as, with each column having the following meanings: Timestamp, SpO2 (saturation of pulse oxygen) value, PR (pulse rate) value, and IsEIOD (indicating whether the corresponding data point is in an exercise-induced oxygen desaturation (EIOD) status: 1 indicates in EIOD status; 0 indicates in non-EIOD status).

Each volunteer completes the 6-minute walking test in a 30 meters long corridor and the detailed 6MWT data collection protocol is as follows:

1. Step 1: Make sure that the volunteer has no contraindications to the walking test (i.e., ruling out the possibility of danger to the volunteer from the walking test);
2. Step 2: The volunteer wears a pulse oximeter wristwatch (model: ORANGER, CMS50S) with a finger cuff on the middle finger, and ensures that the wristwatch functions properly.
3. Step 3: Measure the blood pressure and the blood oxygen saturation of the volunteer;
4. Step 4: The volunteer makes effort to complete the 6-minute walking test. The wristwatch will automatically records data and related symptoms. Note that in this phase, the volunteer is allowed to slow down when chest tightness and asthma symptoms appear;
5. Step 5: Observe the data from the volunteer’s wristwatch and remove the wristwatch until the blood oxygen saturation returns to the value of 95 or the value before the 6-minute walking test.

Feature encoding

As described in the Section of Data Collection and Calibration, the six-minutes’ walk data of each participant is a four-column matrix. To facilitate the subsequent description, we remove the first column (Timestamp) and obtain a three-column matrix as shown in Eq (1), where s_i, p_i and l_i in the i-th line of M are the SpO2 value, the PR value and the EIOD label of the i-th data point: (1)

For the i-th data point, the most simple and intuitive feature vector can be formulated as: (2)

However, previous studies have shown that whether a data point in EIOD status depends not only on the SpO2 and PR values at that time, but also on the SpO2 and PR values of its neighboring data points. In view of this, Hu [8] extended f_i by adding the widow feature () and gradient feature () of SpO2 as shown in Eq (3): (3) where is the window feature of the i-th data point obtained by averaging the SpO2 values in the window centered at the i-th data point; , as defined in Eq (4), is the gradient feature of SpO2 of the i-th data point: (4)

Further, Wu et al. [12] further improved the discriminability of the feature vector by incorporating a new feature component (correlation coefficient) and obtained the new feature vector as defined in Eq (5): (5) where is the correlation coefficient feature component as defined in Eq (6): (6) where , 1≤g≤G, w_size is the size of the window centered at the i-th data point, and G is a prescribed integer.

Considering the good discriminability of the feature , in this study, we also use this feature to represent each data point in the dataset.

Dealing with imbalance by using classifier ensemble

By carefully analyzing the data points in the calibrated dataset, it is found that the number of positive data points (in EIOD) is far less than that of the negative data points (in non-EIOD). In other words, the machine learning based EIOD status identification is a typical imbalanced problem, where the number of samples in majority class (non-EIOD) is significantly larger that of samples in minority class (EIOD). Previous studies have demonstrated that directly applying the machine learning algorithms, which assume that samples in different classes are balanced, to imbalanced problems tends to a poor performance [17]. To deal with this problem, we adopted the workflow, as shown in Fig 1, which ensembles multiple base classifiers trained on balanced training subsets with random under-sampling techniques [18]. More specifically, we sample L different majority training subsets, each of which consists of the same number of samples to that of the minority class, by randomly under-sampling the majority class L times; then, we train a base classifier on each of the sampled majority training subset plus the minority training set; finally, the L trained based classifiers are ensembled by using AdaBoost [19, 20] to form the final ensembled classifier.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Workflow of the AdaBoost classifier ensemble based on random under-sampling technique for exercise-induced oxygen desaturation (EIOD) identification.

https://doi.org/10.1371/journal.pone.0318837.g001

Let x be the feature vector encoded by using Eq (6) of an input sample (i.e., data point in this study), Pool = {S₁(x),S₂(x),⋯,S_i(x),⋯,S_L(x)} be the set of trained base classifier pool, the AdaBoost ensemble is to develop the classifier team , K≤L and , by incrementally selecting one base classifier each time from the pool S. The ensembled classifier is , where α_k is the weight of the selected base classifier . Finally, the ensembled classifier S(x) can be used to perform identification for those unseen samples. Details for AdaBoost algorithm could be found in [19] or [20].

Training and testing procedures

We conducted all the experiments by taking leave-one-out cross-validation (LOOCV) strategy on the calibrated dataset: we take 1 people’s data to construct the testing set and the remaining 84 peoples’ data were used to construct the training set; this practice continued until all the 85 peoples’ data were traversed over. In each round of the LOOCV, we obtain L balanced training subsets by using random under-sampling technique from the training set; then, for each data point in all the L training subsets, we encoded it in to a feature vector as defined in Eq (5); after that, L learning models can be trained on the L training subsets, respectively, with extracted feature vectors; finally, the L trained models are ensembled by using AdaBoost [19, 20] to form the final ensembled classifier. For those data points in the testing set, each of which will be first encoded into a feature vector as defined in Eq (5) and then fed into the final ensembled classifier. The outputs of the final ensembled classifier for all the data points will be used for calculating the evaluation indices.

Evaluation indices

The trained ensembled classifier S(x) can then be used for the identification of EIOD for those unseen data points as follows: First, the feature vector (i.e.,x) of an unseen data point is extracted and fed to the trained ensembled classifier; Then, the corresponding output of the trained ensembled classifier, denoted as p, is the probability of being in EIOD, the range of which is 0~1; Finally, we determine whether the unseen data point is of being in EIOD by using a prescribed threshold T: the unseen data point is in EIOD if the predicted probability p is greater than T; otherwise, the unseen data point is in non-EIOD. By varying the value of T from 0 to 1, we can obtain the Receiver Operating Characteristic (ROC) Curve of the trained ensembled classifier. The area under the ROC Curve, denoted as AUC, is a global performance index of the trained ensembled classifier [17]. The larger the value of AUC, the higher the global performance of the trained ensembled classifier. In view of this, we will use AUC as the global performance index to evaluate the performance the proposed and the compared classifiers.

Experimental configuration and parameter setting

To objectively evaluate the performance of the proposed method and fairly compare it with other existing EIOD status identification methods, we performed comparative experiments of all considered methods on the same experimental configuration as follows:

1. Each experiment was performed by using leave-one-out cross-validation (LOOCV) strategy as described in the Section of Training and Testing Procedures;
2. Each data point was encoded into a feature vector as defined in Eq (5);
3. Three types of base classifiers, i.e., MLP [21], SVM [22] and LSTM [23] were considered to perform classifier ensemble for EIOD status identification.

It has not escaped from our notice that the discriminative capability of the extracted feature is closely related to the parameter setting, i.e., the window size (w_size) and the correlation coefficient (G). Nevertheless, the optimal values of w_size and G depends not only on the dataset itself, but also on the base classifier used. In view of this, for the given calibrated dataset and a given base classifier (e.g., MLP, SVM and LSTM in this study), we obtained the optimal values of the two parameters by using a grid search strategy. More specifically, we vary the value of w_size from 5 to 15 and the value of G from 1 to 9, both with a step size of 2. For each parameter pair (w_size,G), we tested the performance of the identification model trained with the given base classifier by taking leave-one-out cross-validation (LOOCV) strategy on the calibrated dataset; the parameter pair which achieves the highest identification performance measured by AUC is selected as the best parameter set. By using this grid search strategy, for example, we found the optimal values of w_size and G are 11 and 5, respectively, under taking LSTM as the base classifier.

In the classifier ensemble stage, we set the size of the base classifier pool to be 11. The parameter K, i.e., the number of base classifiers used in the ensembled classifier, which depends on the type of base classifiers (MLP, SVM, and LSTM) used and is automatically determined by AdaBoost algorithm. In this study, it is found that the optimal value for parameter K is 7 under taking LSTM as the base classifier.

Classifier ensemble helps to improve EIOD status identification

In this section, three popular classifiers, i.e., MLP [21], SVM [22] and LSTM [23] were used as base classifiers to demonstrate the efficacy of classifier ensemble for EIOD status identification. More specifically, for each type of base classifier (e.g., MLP), we first trained a single MLP identification model on the EIOD training set; then, we tested it and evaluated its performance on the EIOD testing set; after that, we further trained a corresponding ensembled classifier (e.g., Ensembled MLP) using the ensemble method as described in the Section of Dealing with Imbalance by Using Classifier Ensemble. Table 1 summarized the performance comparisons between different base classifiers and the corresponding ensembled classifiers for EIOD status identification. Note that for each type of base classifier and its corresponding ensembled classifier, both and were used as input feature vectors.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Performance comparisons between different base classifiers and the corresponding ensembled classifiers for EIOD status identification.

https://doi.org/10.1371/journal.pone.0318837.t001

From Table 1, several observations can be made as follows: First, the discriminative capability of the feature vector is superior to that of the feature vector f_i. As shown in Table 1, the AUCs of Ensembled MLP, Ensembled SVM and Ensembled LSTM under feature vector f_i are 0.7982, 0.8186 and 0.8403, which are improved to 0.8152, 0.8357 and 0.8532 under feature vector . Similar phenomenon was also observed for the three base classifiers. This observation is consistent with conclusion in reference [12]. Second, for all the three base classifiers, their corresponding ensembled classifiers perform much better under both f_i and feature vectors. For example, compared with MLP, SVM and LSTM, the improvements of 2.7%, 3.1% and 2.6% were obtained by ensembled MLP, ensembled SVM and ensembled LSTM under feature vector . Also, the improvements of 2.0%, 2.0% and 1.9% for MLP, SVM and LSTM, respectively, were observed under feature vector f_i. This observation demonstrates that the ensembled classifier can well deal with the severe class imbalance in EIOD status identification problem thus can obtain better performance.

Ablation experiments

In this section, we will investigate the contributions of different features to the EIOD status identification under ensembled LSTM model using LOOCV strategy. More specifically, we use (s_i,p_i) as baseline feature vector; then we gradually added the widow feature (), the gradient feature () and the correlation coefficient feature () into the baseline feature vector. Table 2 summarizes the EIOD status identification performances with different feature combinations under ensembled LSTM model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Performance comparisons between different features under ensembled LSTM model using LOOCV strategy.

https://doi.org/10.1371/journal.pone.0318837.t002

Several observations can be made by carefully analyzing Table 2. First, it is found that baseline feature, i.e., (s_i,p_i) achieved the AUC of 0.7713, demonstrating the possibility of performing EIOD status identification by directly using SpO2 and PR values; Second, improvements of 5.9% and 6.9% were observed by adding the widow feature () and the gradient feature (), respectively, into the baseline feature vector. This observation demonstrates that the widow feature and the gradient feature have positive impacts to the identification of EIOD; Third, we found that when both and were added into the baseline feature vector, the value of AUC was further improved to 0.8403; Finally, it is found that the value of AUC can be further improved to the maximum value of 0.8532 after further adding the correlation coefficient feature component (). The results in ablation experiments shown in Table 2 indicate that the gradient feature, the window feature, and the correlation coefficient feature are effective auxiliary features for identifying EIOD; By combining these three features with the baseline feature, the model’s capability for the identification of EIOD can be further significantly improved.

Comparisons with existing EIOD status identification models

In this section, we compare the proposed ensembled LSTM with several state-of-the-art EIOD status identification models under the same feature representation method and on the same dataset. The first is the AttLSTM [7], which performs EIOD status identification by using an attention-based Long Short-Term Memory network; the second is SSOM [12], which identifies EIOD using a supervised self-organizing map neural network. Table 3 summarizes the performance comparisons of the proposed ensembled LSTM with AttLSTM and SSOM.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Performance comparisons of the proposed ensembled LSTM with AttLSTM and SSOM.

https://doi.org/10.1371/journal.pone.0318837.t003

By observing Table 3, it is found that the proposed ensembled LSTM achieved the best performance on EIOD status identification with AUCs of 0.8403 and 0.8532 under f_i and as input feature vectors, respectively. Compared with AttLSTM and SSOM, improvements of 2.7% and 1.8%, respectively, on AUC were obtained by ensembled LSTM under input vector. Although not significant, the ensembled LSTM still achieved competitive and even better performance if compared with AttLSTM and SSOM under f_i. These superior performances under both f_i and over the two state-of-the-art EIOD status identification models further demonstrate the efficacy of the proposed ensembled LSTM.