Machine learning for prediction of postoperative nausea and vomiting in patients with intravenous patient-controlled analgesia

Jae-Geum Shim; Kyoung-Ho Ryu; Eun-Ah Cho; Jin Hee Ahn; Yun Byeong Cha; Goeun Lim; Sung Hyun Lee

doi:10.1371/journal.pone.0277957

Peer Review History

Original SubmissionJune 28, 2022
11 Sep 2022 Decision Letter - Jianhong Zhou, Editor PONE-D-22-18403Machine learning for prediction of postoperative nausea and vomiting in patients with intravenous patient-controlled analgesiaPLOS ONE Dear Dr. Lee, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Oct 24 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Jianhong Zhou Staff Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, all author-generated code must be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse. 3. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well. Additional Editor Comments: Specifically, PLOS ONE requires that all experiments, statistics and other analyses are performed to a high technical standard, described in sufficient detail and adhere to appropriate reporting guidelines and community standards. After review of your manuscript, we had several concerns with the analyses, conclusions and quality of the scientific reporting. Please provide the following information: -Why Apfel was chosen as the comparison model? Are there any other existing approaches? -Data collection rationale and criteria, e.g. why pick the time range July 2019 and July 2020? Was sample size calculated before data collection to determine if the final data size is appropriate in light of the study design and your conclusion? -Feature selection strategies, i.e. how were those 21 and 34 variables identified? For the 13 features that were used for training, how did they correlate with each other? -Definitions of the model performance measurements such as accuracy, sensitivity and specificity - Machine learning model details, i.e. any specific parameters used or they are default in the R packages? How are the data normalized? In addition, PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction at the time of publication (https://journals.plos.org/plosone/s/data-availability>). This policy is aimed to ensure that other researchers can reproduce the analysis. We understand that there are privacy and ethical restrictions of your raw data underlying the prediction, if possible, please deposit your de-identified data to a public data repository or include it in the Supporting Information files and update your data availability statement accordingly. Minor comments: -In Table 2, please highlight the models that performed the best for each measurement -Please provide figure legends -Please discuss the likely underlying reasons for why ANN and SVM were better than other models Finally, please present the conclusions that are supported by your results. Given that the accuracy and specificity of the models are not that high, we suggest modifying your conclusions to avoid overstating the implications. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: Dear Author. It is a well conducted study and nicely written manuscript. My points are: 1. Did you take the postoperative pain into consideration while applying machine learning for PONV? The postoperative pain is also a major cause of PONV. 2. Opioid induced PONV (PCA based) in patients with high risk of PONV like post chemotherapy patients or previous history of PONV? This should be considered also during the ML data processing or should include in discussion and limitations. Comparison of palonosetron and dexamethasone with ondansetron and dexamethasone for postoperative nausea and vomiting in postchemotherapy ovarian cancer surgeries requiring opioid-based patient-controlled analgesia: A randomised, double-blind, active controlled study. Indian J Anaesth. 2018 Oct;62(10):773-779. doi: 10.4103/ija.IJA_437_18. PMID: 30443060; PMCID: PMC6190431. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0277957.r001
Revision 1
26 Sep 2022 Author Response Response to Reviewers Academic editor 1. Why Apfel was chosen as the comparison model? Are there any other existing approaches? Response: Apfel’s simplified risk score is the most widely used tool for risk stratification of postoperative nausea and vomiting (PONV). It needs only 4 independent risk factors including female sex, nonsmoking status, history of PONV or motion sickness, and postoperative opioid use. Therefore, we chose Apfel as the comparison model. However, the Apfel model does not guarantee an accurate prediction for the risk of PONV.[1, 2] Recent studies using various logistic regression or machine learning models have shown poor results.[3-5] Also, there was a study that used the Apfel model as a comparison model in predicting PONV.[1] References 1. Chae D, Kim SY, Song Y, Baek W, Shin H, Park K, et al. Dynamic predictive model for postoperative nausea and vomiting for intravenous fentanyl patient-controlled analgesia. Anaesthesia. 2020;75(2):218-26. Epub 2019/09/19. doi: 10.1111/anae.14849. PubMed PMID: 31531854. 2. van den Bosch JE, Kalkman CJ, Vergouwe Y, Van Klei WA, Bonsel GJ, Grobbee DE, et al. Assessing the applicability of scoring systems for predicting postoperative nausea and vomiting. Anaesthesia. 2005;60(4):323-31. Epub 2005/03/16. doi: 10.1111/j.1365-2044.2005.04121.x. PubMed PMID: 15766334. 3. Hu YJ, Ku TH. Pattern discovery from patient controlled analgesia demand behavior. Comput Biol Med. 2012;42(10):1005-11. Epub 2012/09/11. doi: 10.1016/j.compbiomed.2012.08.002. PubMed PMID: 22959278. 4. Palazzo M, Evans R. Logistic regression analysis of fixed patient factors for postoperative sickness: a model for risk assessment. Br J Anaesth. 1993;70(2):135-40. Epub 1993/02/01. doi: 10.1093/bja/70.2.135. PubMed PMID: 8435254. 5. Yuh-Jyh H, Jia-Ying S, Tien-Hsiung K. Predicting Postoperative Nausea and Vomiting Under Patient-Controlled Analgesia Medication: A Study of Machine Learning Approaches. Primary Health Care: Open Access. 2017;7(3):1-6. 2. Data collection rationale and criteria, e.g. why pick the time range July 2019 and July 2020? Was sample size calculated before data collection to determine if the final data size is appropriate in light of the study design and your conclusion? Response: In this study, the sample size of machine learning models was not determined prior to the data collection. The PCA regimen frequently changed in the hospital where the author works. The time between July 2019 and July 2020 was selected as the data collection period as the fentanyl-based IV-PCA regimen remained constant. Unfortunately, an accurate calculation method for machine learning sample size is lacking at this time. Therefore, considering the rule of thumb, we needed at least 1,000 samples per class. The sample size of the non-PONV group was suitable in this study, but the number of samples of PONV group was insufficient. As a result, we oversampled the minority class and balanced the low incidence of PONV in this study. 3. Feature selection strategies, i.e. how were those 21 and 34 variables identified? For the 13 features that were used for training, how did they correlate with each other? Response: First, 21 variables were selected based on the previous literature and our clinical experience. Then categorical variables with n levels were transformed into n variables, each with two levels. As a result, 34 variables were considered as input variables for the model. Then, 13 variables with the best prediction performance were selected by the Recursive Feature Elimination (RFE) algorithm. The correlation of 13 variables that were used for the training is shown in the figure below. There was a weak association between PONV and previous motion sickness. We also found a very weak association between PONV and female, laparoscopy surgery, and background dose of fentanyl. 4. Definitions of the model performance measurements such as accuracy, sensitivity and specificity Response: Thank you for suggesting important points. We have added a table of confusion matrix and revised manuscript. “The confusion matrix is used for summarizing the performance of a classification problem as shown in Table x. Accuracy, sensitivity, and specificity are described in terms of true positive (TP), true negative (TN), false negative (FN) and false positive (FP). The accuracy of model is the ratio of correct predictions to total predictions made and is defined as Accuracy = (TN + TP) / (TN+TP+FN+FP) The sensitivity of model is the proportion of actual positive cases that are correctly identified and is defined as Sensitivity = TP / (TP + FN) The specificity of model is the proportion of actual negative cases that are correctly identified and is defined as Specificity = TN / (TN + FP)” Table 1. Confusion matrix Actual Positive Negative Predicted Positive TP TN Negative FP FN TP, true positive; TN, true negative; FN, false negative; FP, false positive 5. Machine learning model details, i.e. any specific parameters used or they are default in the R packages? How are the data normalized? Response: The S1 Table contains the details of the hyperparameters that we used for our machine learning model. In our study, continuous variables were transformed into values between 0 and 1 by minimum-maximum normalization, implemented in the caret package in the R software. We added the S1 Table in manuscript. “The detailed hyperparameters of the machine learning model used in this study can be found in the S1 table.” S1 Table. Optimal hyperparameters of all machine learning models Model Optimal hyperparameters LR nIter* = 31 KNN k = 8 DT Maximum depth = 5 Criterion = Gini index RF mtry** = 3 GBM Maximum depth = 3 Number of estimators = 50, Gamma = 0 SVM degree = 3, scale = 0.1 and C = 1.0 ANN Number of hidden layers = 2 Number of nodes in a layer = 32, 16 LR, logistic regression; KNN, k-nearest neighbors; DT, decision tree; RF, random forest; GBM, gradient boosting machine; SVM, support vector machine; ANN, artificial neural networks nIter indicates an integer, describing the number of iterations for which boosting should be run. mtry indicates the number of variables available for splitting at each tree node. 6. In addition, PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction at the time of publication (https://journals.plos.org/plosone/s/data-availability>). This policy is aimed to ensure that other researchers can reproduce the analysis. We understand that there are privacy and ethical restrictions of your raw data underlying the prediction, if possible, please deposit your de-identified data to a public data repository or include it in the Supporting Information files and update your data availability statement accordingly. Response: We understand the guidelines of PLOS journals. There were ethical restrictions of raw data, and thus we deposited de-identified and preprocessed data to a public data repository. (https://github.com/jgshim/IV-PCA) 7. Minor comments: In Table 2, please highlight the models that performed the best for each measurement Response: We highlighted the models in Table 2. Please provide figure legends Response: We provided figure legends. Please discuss the likely underlying reasons for why ANN and SVM were better than other models Response: Our primary analysis goal was to compare the prediction ability of machine learning approaches to Apfel model in terms of the area under the receiver-operating characteristics curve (AUROC). As shown in previous literature, the analysis of AUROC is a valuable tool to evaluate predictive models.[6] Table x shows the comparison of test AUROC to predict PONV according to the model. The AUROC of ANN and SVM were larger and therefore better than all other models. Reference 6. Zou KH, O'Malley AJ, Mauri L. Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation. 2007;115(5):654-7. Epub 2007/02/07. doi: 10.1161/circulationaha.105.594929. PubMed PMID: 17283280. 8. Finally, please present the conclusions that are supported by your results. Given that the accuracy and specificity of the models are not that high, we suggest modifying your conclusions to avoid overstating the implications. Response: We fully agree with your opinion. We revised our manuscript. “It is clear that PONV is a distressing side effect in patients. A suitable screening test for PONV should include adequate sensitivity and specificity, and be acceptable to both patient and medical practitioners. Having high sensitivity but low specificity may lead to inappropriate preemptive measures. For instance, a patient with a low risk of PONV might be given anti-emetics. Therefore, careful attention should be paid when used as a screening tool. Response to peer reviewers Reviewer #1: 1. Did you take the postoperative pain into consideration while applying machine learning for PONV? The postoperative pain is also a major cause of PONV. Response: We fully agree with your opinion. The postoperative pain is a major cause of PONV. Unfortunately, postoperative pain was not used as an input variable in our predictive model for the following two reasons. Initially the postoperative pain score was obtained from EMR in raw data. However, we concluded that the postoperative pain score recorded was inconsistent due to the following reasons. Firstly, the frequency and dosage of the rescue analgesia were inconsistent. Secondly, the pain scores fluctuated depending on the time of the day, situation (i.e. ambulation, during sleep), and environment. 2. Opioid induced PONV (PCA based) in patients with high risk of PONV like post chemotherapy patients or previous history of PONV? This should be considered also during the ML data processing or should include in discussion and limitations. Comparison of palonosetron and dexamethasone with ondansetron and dexamethasone for postoperative nausea and vomiting in postchemotherapy ovarian cancer surgeries requiring opioid-based patient-controlled analgesia: A randomised, double-blind, active controlled study. Indian J Anaesth. 2018 Oct;62(10):773-779. doi: 10.4103/ija.IJA_437_18. PMID: 30443060; PMCID: PMC6190431 Response: As the reviewer pointed out, postchemotherapy patients or those with a previous history of PONV are at high risk for PONV. The previous history of PONV was used as an input to our model. However, the number of cases of postchemotherapy patients were very small. Even if they had a history of cancer, it was not clear whether they had received chemotherapy. Thus, we were not able to include the postchemotherapy patient group as the input variable. Including chemotherapy as an input factor in further studies may improve the performance of the PONV prediction model. We revised manuscript in “Discussion” section. “The patients with chemotherapy history are at high risk for opioid induced PONV.[7] However, the number of cases of postchemotherapy patients were very small. Even if they had a history of cancer, it was not clear whether they had received chemotherapy. Thus, we were not able to include the postchemotherapy patient group as the input variable. Including chemotherapy as an input factor in further studies may improve the performance of the PONV prediction model. Reference 7. Kumar A, Solanki SL, Gangakhedkar GR, Shylasree TS, Sharma KS. Comparison of palonosetron and dexamethasone with ondansetron and dexamethasone for postoperative nausea and vomiting in postchemotherapy ovarian cancer surgeries requiring opioid-based patient-controlled analgesia: A randomised, double-blind, active controlled study. Indian J Anaesth. 2018;62(10):773-9. Epub 2018/11/18. doi: 10.4103/ija.IJA_437_18. PubMed PMID: 30443060; PubMed Central PMCID: PMCPMC6190431. Attachments Attachment Submitted filename: Response to Reviewers.docx* https://doi.org/10.1371/journal.pone.0277957.r002
8 Nov 2022 Decision Letter - Sathishkumar V E, Editor Machine learning for prediction of postoperative nausea and vomiting in patients with intravenous patient-controlled analgesia PONE-D-22-18403R1 Dear Dr. Lee, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Sathishkumar V E Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: Dear Authors, Thank you for the revision. All the comment has been addressed by the authors. The article is good to go for the publications ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: Yes: Sohan Lal Solanki ******** https://doi.org/10.1371/journal.pone.0277957.r003
Formally Accepted
14 Dec 2022 Acceptance Letter - Sathishkumar V E, Editor PONE-D-22-18403R1 Machine learning for prediction of postoperative nausea and vomiting in patients with intravenous patient-controlled analgesia Dear Dr. Lee: I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. If we can help with anything else, please email us at plosone@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Sathishkumar V E Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0277957.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .