Fig 1.
Process for generating patient case scenarios and collecting algorithm training and validation data.
Fig 2.
Training, validation, and optimization procedure for building COPD exacerbation and triage prediction algorithms.
Table 1.
List of patient profile, comorbidity, vital sign, and symptom factors, with respective measures, used in the COPD triage and exacerbation algorithms.
Fig 3.
Comparison between ML classifiers at matching consensus decision in the validation set.
SVMP, SVML, and SVMG are all support vector machine algorithms with polynomial, linear, and Gaussian kernels respectively. RF = Random Forest, NB = Naïve Bayes, LR = Logistic Regression, KNN = K-Nearest Neighbors, GB = Gradient Boosted Random Forest, and ET = Extra Decision Tree Classifier.
Fig 4.
Performance comparison when the algorithm and all of the physicians got a vote in the consensus opinion.
Comparison of the algorithm and individual physicians at predicting the consensus triage and exacerbation (y/n) in the validation set: (a) triage identification, (b) exacerbation identification. A comparison of the algorithm with the average physician in accuracy, sensitivity, specificity, ppv, and npv for: (c) triage identification, (d) exacerbation identification. Triage statistics were computed as defined in Eqs 1–7.
Table 2.
Statistical measures (Eqs 1–7) of triage and exacerbation identification ability for the top 2 performing algorithms and top physician.
Fig 5.
Performance comparison of algorithm and individual physicians at predicting the consensus of the validation sets.
(a) triage performance, algorithm was not included in consensus, (b) exacerbation performance, algorithm was not included in consensus. (c) triage performance, no member votes when assessing their accuracy, (d) exacerbation performance, no member votes when assessing their accuracy.
Fig 6.
Confusion matrices comparing assessment performance of the GB algorithm to the top physician.
(a) triage, (b) exacerbation. Note: top physician = the physician with highest classification accuracy.
Table 3.
Statistical measures of performance of the top 2 algorithms (highest classification accuracy) and top performing physician when classifying the need for medical attention.
Fig 7.
Comparing the performance of the GB algorithm to the top physician in assessing the need for medical attention.
Note: top physician = physician with highest classification accuracy.
Fig 8.
Distributions for each physician in the validation set (left) and the averaged distributions (right).
(a) triage distribution, (b) averaged triage distribution, (c) exacerbation distribution, (d) averaged exacerbation distribution. Note: error bars indicate 1 standard deviation about the mean.
Fig 9.
Plot of % change in consensus triage answers as additional doctors are added to the validation panel (plus algo).
The average change when the panel reaches 10 members (from 9) is 5.5%.
Table 4.
Feature importance for the top two performing algorithms for both triage and exacerbation models.