https://orcid.org/0000-0001-5616-6649
Figures
Abstract
Background
Postoperative nausea and vomiting (PONV) is a still highly relevant problem and is known to be a distressing side effect in patients. The aim of this study was to develop a machine learning model to predict PONV up to 24 h with fentanyl-based intravenous patient-controlled analgesia (IV-PCA).
Methods
From July 2019 and July 2020, data from 2,149 patients who received fentanyl-based IV-PCA for analgesia after non-cardiac surgery under general anesthesia were applied to develop predictive models. The rates of PONV at 1 day after surgery were measured according to patient characteristics as well as anesthetic, surgical, or PCA-related factors. All statistical analyses and computations were performed using the R software.
Results
A total of 2,149 patients were enrolled in this study, 337 of whom (15.7%) experienced PONV. After applying the machine-learning algorithm and Apfel model to the test dataset to predict PONV, we found that the area under the receiver operating characteristic curve using logistic regression was 0.576 (95% confidence interval [CI], 0.520–0.633), k-nearest neighbor was 0.597 (95% CI, 0.537–0.656), decision tree was 0.561 (95% CI, 0.498–0.625), random forest was 0.610 (95% CI, 0.552–0.668), gradient boosting machine was 0.580 (95% CI, 0.520–0.639), support vector machine was 0.649 (95% CI, 0.592–0.707), artificial neural network was 0.686 (95% CI, 0.630–0.742), and Apfel model was 0.643 (95% CI, 0.596–0.690).
Citation: Shim J-G, Ryu K-H, Cho E-A, Ahn JH, Cha YB, Lim G, et al. (2022) Machine learning for prediction of postoperative nausea and vomiting in patients with intravenous patient-controlled analgesia. PLoS ONE 17(12): e0277957. https://doi.org/10.1371/journal.pone.0277957
Editor: Sathishkumar V. E., Hanyang University, REPUBLIC OF KOREA
Received: June 28, 2022; Accepted: November 7, 2022; Published: December 22, 2022
Copyright: © 2022 Shim et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: De-identified and pre-processed data were deposited to a public data repository (https://github.com/jgshim/IV-PCA). All data underlying the findings described in our paper are fully available without restriction.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: ANN, Artificial neural network; AUROC, Area under the receiver operating characteristic curve; CI, Confidence interval; DT, Decision tree; GBM, Gradient boosting machine; KNN, K-nearest neighbors; PCA, Patient-controlled analgesia; PONV, Postoperative nausea and vomiting; RF, Random forest; SVM, Support vector machine; TIVA, Total intravenous anesthesia
Background
Postoperative nausea and vomiting (PONV) is a common condition and is known to be a distressing side effect in patients [1]. The incidence of PONV is 30% and can be as high as 80% in high-risk patients [2, 3]. Although the mechanism of PONV is not clear, the use of perioperative opioids is known to be associated with it [4]. Nonetheless, opioid-based intravenous patient-controlled analgesia (IV-PCA) currently plays an important role in routine postoperative analgesic therapy [5–7]. Therefore, by accurately predicting PONV, patients can be warned of the risk of developing PONV, and clinicians can be assisted in making decisions about preventive treatment.
Apfel’s risk score is a simple assessment tool derived to predict the 24-h rates of PONV [8, 9]. However, the Apfel model does not guarantee the accurate prediction of the risk of PONV, with limited discrimination and calibration properties [10, 11]. Recently, studies have used dynamic predictive models or machine learning to improve the predictive performance of PONV [12–15].
Machine learning is the application of artificial intelligence, whereby a computer algorithm automatically learns and improves from prior experience [16]. The machine learning algorithm produces an inferred function that can be used as the predictor of new data after sufficient training with known input and output values [17]. It may be used for prediction in the medical field. Recently, machine learning algorithms have shown high performance in various fields of medicine, such as diagnosis, prognosis, and clinical decision support [18–21].
To the best of our knowledge, no previous study has compared the performance of Apfel and machine learning methods in predicting PONV 24 h after surgery. We expect our research results to improve the prediction of PONV and quality of patient care.
Methods and methods
Study population
We collected data from patients (>19 years) after non-cardiac surgery under general anesthesia who received fentanyl-based IV-PCA at Kangbuk Samsung Hospital between July 2019 and July 2020. The exclusion criteria for this study were refusal to receive PCA and admission to the intensive care unit. This study was reviewed and approved by the Institutional Review Board (IRB No. 2020-08-001) of Kangbuk Samsung Hospital (Seoul, Korea). This study was conducted in accordance with the principles of the Declaration of Helsinki of the World Medical Association. The need for written informed consent was waived as this was a retrospective study of electronic medical records.
Data collection
The rates of PONV at 1 day after surgery were measured with information on postoperative pain scores and other complications by the PCA team in our hospital. We also included patient characteristics as well as anesthetic, surgical, or PCA-related factors in the predictive models. The continuous variables were age, body mass index (BMI), duration of anesthesia, and dosage of fentanyl in IV-PCA. The categorical variables were sex, history of motion sickness or PONV, American Society of Anesthesiologists (ASA) physical status, diabetes mellitus, hypertension, premedication, use of preintubation opioids, anesthetic agents (sevoflurane, desflurane, or TIVA), intraoperative remifentanil infusion, the use of intraoperative opioids (fentanyl or meperidine), emergency operation, laparoscopic surgery, type of surgery, adjuvant nefopam, and antiemetic (ramosetron) in IV-PCA. Continuous variables were transformed to values between 0 and 1 by minimum-maximum normalization, implemented in the caret package in the R software.
Feature selection
Feature selection is the process of selecting features that contribute the most to our prediction variable, leading to improved performance. In this process, recursive feature elimination was used as a method that fits the random forest function in the core of the model and removes the weakest feature until the specified number of features is reached. Features are ranked by the model’s feature importance by iteratively eliminating a small number of features per loop. To enable the machine learning algorithms to run efficiently, we only used the data features resulting from recursive feature elimination to train our machine learning models.
Model assessment
To determine the goodness of the prediction ability, model performance was evaluated by comparing machine learning approaches to the Apfel model in terms of the area under the receiver operating characteristic curve (AUROC). The AUROC was plotted using the test dataset to understand the tradeoff in performance for different threshold values in imbalanced classification problems. We also compared the accuracy, sensitivity, and specificity.
The confusion matrix is used for summarizing the performance of a classification problem as shown in Table 1. Accuracy, sensitivity, and specificity are described in terms of true positive (TP), true negative (TN), false negative (FN) and false positive (FP).
The accuracy of model is the ratio of correct predictions to total predictions made and is defined as
Accuracy = (TN + TP) / (TN+TP+FN+FP)
The sensitivity of model is the proportion of actual positive cases that are correctly identified and is defined as
Sensitivity = TP / (TP + FN)
The specificity of model is the proportion of actual negative cases that are correctly identified and is defined as
Specificity = TN / (TN + FP)
Statistical analysis
All statistical analyses and computations were performed using the R software version 3.6.3 (R Development Core Team, Vienna, Austria). The machine learning algorithm was implemented using the following packages: Caret (https://CRAN.R-project.org/package=caret), Xgboost (https://CRAN.R-project.org/package=xgboost), and Keras (https://CRAN.R-project.org/package=keras). The entire code of our study (https://github.com/jgshim/PONV) is provided.
Before applying the machine learning models, our data set was randomly divided into 70/30 training and test sets, as we did not want our models to overfit and generalize well. Specifically, 70% of the data was used for training prediction models, and 30% was used as the testing set for verification. A 10-fold cross-validation repeated three times was used to assess how the predictive model generalizes to an independent dataset. The missing data were imputed using the nearest neighbor imputation algorithms, where each missing value is replaced by a value obtained from related cases in the entire data set [22]. The synthetic minority oversampling technique method, addressing imbalanced classification problems, was used to oversample the minority class and balance the low incidence of PONV in the training set [23].
Results
Patient’s characteristics
The sample group included 2,680 patients who received fentanyl-based IV-PCA for analgesia after non-cardiac surgery under general anesthesia at Kangbuk Samsung Hospital between July 2019 and July 2020. A total of 23 patients aged ≤18 years were excluded. In addition, 508 patients were excluded because they were subjected to regional anesthesia. As a result, a total of 2,149 patients satisfying all inclusion criteria were enrolled in the study. During the 24 h after surgery, 337 patients (15.7%) experienced PONV. The patient characteristics as well as anesthetic, surgical, or PCA-related variables are summarized in Table 2. The correlation analysis showed a weak positive correlation between motion sickness, laparoscopy, desflurane, and gynecology surgery and PONV, as shown in Fig 1. However, male sex and smoking status showed a weak negative correlation with PONV.
Feature selection
We identified 21 variables, including patient characteristics as well as anesthetic, surgical, or PCA-related factors, from previous studies conducted to identify features that may contribute to PONV. Among these variables, anesthetics and type of surgery were categorical variables with more than two levels. As an input for our models, categorical variables with n levels were transformed into n variables, each with two levels. As a result, 34 variables were initially considered as input variables for the model.
The recursive feature elimination algorithm resulted in the final 13 factors contributing to PONV. Fig 2 shows the process of feature selection after the step of recursive feature elimination. On final feature selection, only 13 features were used as input variables in training the machine learning models for predicting PONV.
Model performance
The predictive performance of various machine learning and Apfel models is shown in Table 3. After applying the test dataset for all machine learning techniques and the Apfel score to predict PONV, we found that the AUROC using logistic regression was 0.576 (95% confidence interval [CI], 0.520–0.633), k-nearest neighbor was 0.597 (95% CI, 0.537–0.656), decision tree was 0.561 (95% CI, 0.498–0.625), random forest was 0.610 (95% CI, 0.552–0.668), gradient boosting machine was 0.580 (95% CI, 0.520–0.639), support vector machine (SVM) was 0.649 (95% CI, 0.592–0.707), artificial neural network (ANN) was 0.686 (95% CI, 0.630–0.742), and Apfel score was 0.643 (95% CI, 0.596–0.690). The ANN showed the largest AUROC (0.686, 95% CI, 0.630–0.742), as shown in Fig 3. The SVM showed the highest accuracy (0.717, 95% CI, 0.681–0.752).
The entire code used in this study is available online without restrictions (https://github.com/jgshim/IV-PCA). The detailed hyperparameters of the machine learning model used in this study can be found in the S1 Table.
Discussion
We analyzed and compared the predictive ability of seven machine learning approaches and the Apfel model to predict PONV during 24 h after surgery. The results showed that the ANN method had the largest AUROC for identifying PONV using clinical data. The key findings were as follows: (1) machine learning models such as ANN and SVM showed better performance than the Apfel model and (2) feature selection using recursive feature elimination improved human insight into complex and non-linear models associated with PONV. To our knowledge, this is the first study to predict the occurrence of PONV by comparing various classification machine learning approaches with the Apfel model.
Conventional machine learning approaches generally work efficiently with traditional datasets and allow for nonlinear relationships between predictors but may deteriorate with high-dimensional problems [24]. We explored the number of selected features using the wrapped algorithm used in the recursive feature elimination procedure. By means of dimensionality reduction, the dependencies and collinearity that may exist in the model can be eliminated to improve performance.
Although volatile anesthetics was an important factor of PONV in a previous study [25], in this study, the type of volatile anesthetics or intravenous anesthesia was not helpful in predicting PONV. An increase in the duration of anesthesia was associated with a reduction in PONV, which is inconsistent with previous results [26]. The increase in PONV with the use of preintubation opioids or laparoscopic surgery is evidenced in previous studies [27, 28]. Hypotension occurring during shoulder surgery may be a major factor of PONV and was used as one of the features in our study [12, 29].
We believe that anesthesiologists in the operating room can help manage PONV. For example, considering the possibility of PONV, the ANN or SVM model could be useful in deciding whether to take preemptive measures, such as preparing an antiemetic in advance or continuing follow-up and observation. Furthermore, cost-effective management will be possible because the models require only 13 clinical variables to identify patients at a high risk of PONV.
The patients with chemotherapy history are at high risk for opioid induced PONV [30]. However, the number of cases of postchemotherapy patients were very small. Even if they had a history of cancer, it was not clear whether they had received chemotherapy. Thus, we were not able to include the postchemotherapy patient group as the input variable. Including chemotherapy as an input factor in further studies may improve the performance of the PONV prediction model.
It is clear that PONV is a distressing side effect in patients. A suitable screening test for PONV should include adequate sensitivity and specificity, and be acceptable to both patient and medical practitioners. Having high sensitivity but low specificity may lead to inappropriate preemptive measures. For instance, a patient with a low risk of PONV might be given anti-emetics. Therefore, careful attention should be paid when used as a screening tool.
There are some limitations to our study. First, we are not sure that the amount of data we used was enough to work on machine-learning problems, considering the complexity of the problem and nonlinear algorithms. Further data about PONV should be collected to improve the predictive power. Second, one of the PCA teams in the anesthesiology department visited only once during the day after surgery and asked about the effects and complications of PCA. As a result, because of recall bias, the PONV occurrence rate may have been underestimated. Third, because our study analyzed data from a single center, it might not be possible to apply our model to a wider population. Further studies are needed, with large heterogeneous samples, to improve generalizability.
Conclusions
In summary, we developed and compared various machine learning models and the Apfel model to predict the occurrence of PONV using IV-PCA. We expect our results to help reduce PONV by helping clinicians predict it and take preemptive actions.
Supporting information
S1 Table. Optimal hyperparameters of all machine learning models.
https://doi.org/10.1371/journal.pone.0277957.s001
(DOCX)
References
- 1. Choi DH, Ko JS, Ahn HJ, Kim JA. A korean predictive model for postoperative nausea and vomiting. J Korean Med Sci. 2005;20(5):811–5. Epub 2005/10/15. pmid:16224155; PubMed Central PMCID: PMC2779278.
- 2. Gan TJ, Belani KG, Bergese S, Chung F, Diemunsch P, Habib AS, et al. Fourth Consensus Guidelines for the Management of Postoperative Nausea and Vomiting. Anesth Analg. 2020;131(2):411–48. Epub 2020/05/30. pmid:32467512.
- 3. Lee MJ, Lee KC, Kim HY, Lee WS, Seo WJ, Lee C. Comparison of ramosetron plus dexamethasone with ramosetron alone on postoperative nausea, vomiting, shivering and pain after thyroid surgery. Korean J Pain. 2015;28(1):39–44. Epub 2015/01/16. pmid:25589945; PubMed Central PMCID: PMC4293505.
- 4. de Boer HD, Detriche O, Forget P. Opioid-related side effects: Postoperative ileus, urinary retention, nausea and vomiting, and shivering. A review of the literature. Best Pract Res Clin Anaesthesiol. 2017;31(4):499–504. Epub 2018/05/10. pmid:29739538.
- 5. Choi JB, Shim YH, Lee YW, Lee JS, Choi JR, Chang CH. Incidence and risk factors of postoperative nausea and vomiting in patients with fentanyl-based intravenous patient-controlled analgesia and single antiemetic prophylaxis. Yonsei Med J. 2014;55(5):1430–5. Epub 2014/07/23. pmid:25048507; PubMed Central PMCID: PMC4108834.
- 6. Grass JA. Patient-controlled analgesia. Anesth Analg. 2005;101(5 Suppl):S44–61. Epub 2005/12/13. pmid:16334492.
- 7. McNicol ED, Ferguson MC, Hudcova J. Patient controlled opioid analgesia versus non-patient controlled opioid analgesia for postoperative pain. Cochrane Database Syst Rev. 2015;2015(6):Cd003348. Epub 2015/06/03. pmid:26035341; PubMed Central PMCID: PMC7387354 interest to declare JH has no relevant conflicts of interest to declare.
- 8. Apfel CC, Läärä E, Koivuranta M, Greim CA, Roewer N. A simplified risk score for predicting postoperative nausea and vomiting: conclusions from cross-validations between two centers. Anesthesiology. 1999;91(3):693–700. Epub 1999/09/15. pmid:10485781.
- 9. Weilbach C, Rahe-meyer N, Raymondos K, Weissig A, Scheinichen D, Piepenbrock S. Postoperative nausea and vomiting (PONV): usefulness of the Apfel-score for identification of high risk patients for PONV. Acta Anaesthesiol Belg. 2006;57(4):361–3. Epub 2007/01/24. pmid:17236637.
- 10. Engel JM, Junger A, Hartmann B, Little S, Schnöbel R, Mann V, et al. Performance and customization of 4 prognostic models for postoperative onset of nausea and vomiting in ear, nose, and throat surgery. J Clin Anesth. 2006;18(4):256–63. Epub 2006/06/27. pmid:16797426.
- 11. van den Bosch JE, Kalkman CJ, Vergouwe Y, Van Klei WA, Bonsel GJ, Grobbee DE, et al. Assessing the applicability of scoring systems for predicting postoperative nausea and vomiting. Anaesthesia. 2005;60(4):323–31. Epub 2005/03/16. pmid:15766334.
- 12. Chae D, Kim SY, Song Y, Baek W, Shin H, Park K, et al. Dynamic predictive model for postoperative nausea and vomiting for intravenous fentanyl patient-controlled analgesia. Anaesthesia. 2020;75(2):218–26. Epub 2019/09/19. pmid:31531854.
- 13. Wu HY, Gong CA, Lin SP, Chang KY, Tsou MY, Ting CK. Predicting postoperative vomiting among orthopedic patients receiving patient-controlled epidural analgesia using SVM and LR. Sci Rep. 2016;6:27041. Epub 2016/06/02. pmid:27247165; PubMed Central PMCID: PMC4887988.
- 14. Gong CS, Yu L, Ting CK, Tsou MY, Chang KY, Shen CL, et al. Predicting postoperative vomiting for orthopedic patients receiving patient-controlled epidural analgesia with the application of an artificial neural network. Biomed Res Int. 2014;2014:786418. Epub 2014/08/28. pmid:25162027; PubMed Central PMCID: PMC4138736.
- 15. Yuh-Jyh H, Jia-Ying S, Tien-Hsiung K. Predicting Postoperative Nausea and Vomiting Under Patient-Controlled Analgesia Medication: A Study of Machine Learning Approaches. Prim Health Care. 2017;7(272):2167–1079.1000272.
- 16. Lötsch J, Ultsch A. Machine learning in pain research. Pain. 2018;159(4):623–30. Epub 2017/12/02. pmid:29194126; PubMed Central PMCID: PMC5895117 the end of this article.
- 17. Obermeyer Z, Emanuel EJ. Predicting the Future—Big Data, Machine Learning, and Clinical Medicine. N Engl J Med. 2016;375(13):1216–9. Epub 2016/09/30. pmid:27682033; PubMed Central PMCID: PMC5070532.
- 18. Wang Y, Lei L, Ji M, Tong J, Zhou CM, Yang JJ. Predicting postoperative delirium after microvascular decompression surgery with machine learning. J Clin Anesth. 2020;66:109896. Epub 2020/06/07. pmid:32504969.
- 19. Panesar SS, D’Souza RN, Yeh FC, Fernandez-Miranda JC. Machine Learning Versus Logistic Regression Methods for 2-Year Mortality Prognostication in a Small, Heterogeneous Glioma Database. World Neurosurg X. 2019;2:100012. Epub 2019/06/21. pmid:31218287; PubMed Central PMCID: PMC6581022.
- 20. Kendale S, Kulkarni P, Rosenberg AD, Wang J. Supervised Machine-learning Predictive Analytics for Prediction of Postinduction Hypotension. Anesthesiology. 2018;129(4):675–88. Epub 2018/08/04. pmid:30074930.
- 21. Lee HC, Yoon SB, Yang SM, Kim WH, Ryu HG, Jung CW, et al. Prediction of Acute Kidney Injury after Liver Transplantation: Machine Learning Approaches vs. Logistic Regression Model. J Clin Med. 2018;7(11). Epub 2018/11/11. pmid:30413107; PubMed Central PMCID: PMC6262324.
- 22. Beretta L, Santaniello A. Nearest neighbor imputation algorithms: a critical evaluation. BMC Med Inform Decis Mak. 2016;16 Suppl 3(Suppl 3):74. Epub 2016/07/28. pmid:27454392; PubMed Central PMCID: PMC4959387.
- 23. Wang Q, Luo Z, Huang J, Feng Y, Liu Z. A Novel Ensemble Method for Imbalanced Data Learning: Bagging of Extrapolation-SMOTE SVM. Comput Intell Neurosci. 2017;2017:1827016. Epub 2017/03/03. pmid:28250765; PubMed Central PMCID: PMC5304315 of this paper.
- 24. Darst BF, Malecki KC, Engelman CD. Using recursive feature elimination in random forest to account for correlated variables in high dimensional data. BMC Genet. 2018;19(Suppl 1):65. Epub 2018/09/27. pmid:30255764; PubMed Central PMCID: PMC6157185.
- 25. Apfel CC, Kranke P, Katz MH, Goepfert C, Papenfuss T, Rauch S, et al. Volatile anaesthetics may be the main cause of early but not delayed postoperative vomiting: a randomized controlled trial of factorial design. Br J Anaesth. 2002;88(5):659–68. Epub 2002/06/18. pmid:12067003.
- 26. SINCLAIR CHUNG DR F, MEZEI G. Can Postoperative Nausea and Vomiting Be Predicted? Survey of Anesthesiology. 2000;44(1):3–4.
- 27. Roberts GW, Bekker TB, Carlsen HH, Moffatt CH, Slattery PJ, McClure AF. Postoperative nausea and vomiting are strongly influenced by postoperative opioid use in a dose-related manner. Anesth Analg. 2005;101(5):1343–8. Epub 2005/10/26. pmid:16243992.
- 28. Cao X, White PF, Ma H. An update on the management of postoperative nausea and vomiting. J Anesth. 2017;31(4):617–26. Epub 2017/04/30. pmid:28455599.
- 29. Pusch F, Berger A, Wildling E, Tiefenthaler W, Krafft P. The effects of systolic arterial blood pressure variations on postoperative nausea and vomiting. Anesth Analg. 2002;94(6):1652–5, table of contents. Epub 2002/05/29. pmid:12032046.
- 30. Kumar A, Solanki SL, Gangakhedkar GR, Shylasree TS, Sharma KS. Comparison of palonosetron and dexamethasone with ondansetron and dexamethasone for postoperative nausea and vomiting in postchemotherapy ovarian cancer surgeries requiring opioid-based patient-controlled analgesia: A randomised, double-blind, active controlled study. Indian J Anaesth. 2018;62(10):773–9. Epub 2018/11/18. pmid:30443060; PubMed Central PMCID: PMC6190431.