Comparing machine learning with case-control models to identify confirmed dengue cases

In recent decades, the global incidence of dengue has increased. Affected countries have responded with more effective surveillance strategies to detect outbreaks early, monitor the trends, and implement prevention and control measures. We have applied newly developed machine learning approaches to identify laboratory-confirmed dengue cases from 4,894 emergency department patients with dengue-like illness (DLI) who received laboratory tests. Among them, 60.11% (2942 cases) were confirmed to have dengue. Using just four input variables [age, body temperature, white blood cells counts (WBCs) and platelets], not only the state-of-the-art deep neural network (DNN) prediction models but also the conventional decision tree (DT) and logistic regression (LR) models delivered performances with receiver operating characteristic (ROC) curves areas under curves (AUCs) of the ranging from 83.75% to 85.87% [for DT, DNN and LR: 84.60% ± 0.03%, 85.87% ± 0.54%, 83.75% ± 0.17%, respectively]. Subgroup analyses found all the models were very sensitive particularly in the pre-epidemic period. Pre-peak sensitivities (<35 weeks) were 92.6%, 92.9%, and 93.1% in DT, DNN, and LR respectively. Adjusted odds ratios examined with LR for low WBCs [≤ 3.2 (x103/μL)], fever (≥38°C), low platelet counts [< 100 (x103/μL)], and elderly (≥ 65 years) were 5.17 [95% confidence interval (CI): 3.96–6.76], 3.17 [95%CI: 2.74–3.66], 3.10 [95%CI: 2.44–3.94], and 1.77 [95%CI: 1.50–2.10], respectively. Our prediction models can readily be used in resource-poor countries where viral/serologic tests are inconvenient and can also be applied for real-time syndromic surveillance to monitor trends of dengue cases and even be integrated with mosquito/environment surveillance for early warning and immediate prevention/control measures. In other words, a local community hospital/clinic with an instrument of complete blood counts (including platelets) can provide a sentinel screening during outbreaks. In conclusion, the machine learning approach can facilitate medical and public health efforts to minimize the health threat of dengue epidemics. However, laboratory confirmation remains the primary goal of surveillance and outbreak investigation.


4)
shows statistically significant differences between included and excluded groups. The authors should address the potential implications of these differences for the results. Did the authors review whether these subjects could be included in validation of the final models? [Response] We have added the following paragraph in the section Discussion to address the potential implications of these differences. "Fifth, we conjecture that those thrombocytopenia cases excluded from our study population should be dengue cases and should have dengue confirmatory test before visiting NCKUH ED…If these cases were included in the final models, the positive predictive values of our models could be improved."

5) Line 188-Why were these cutoffs selected? Would other cutoff values have had
greater discriminatory value? [Response] The cutoff values represented the reference (i.e. normal ranges) of the tests which have been routinely used in the NCKU hospital with greater discrimination. We then employed these cutoff values, representing which represented reference ranges, shown in S2 Table [Page 43], to stratify as "normal" vs "abnormal" (high or low) values these variables and computed the crude odds ratios of the 18 variables. Figure 3) should be described in Methods. [Response] We have described the WHO definitions in section Methods as below. "The clinical diagnosis of dengue-like illness in Taiwan was usually made according to the 1997 or 2009 WHO clinical definitions … The reported sensitivity and specificity of the 1997 and 2009 WHO definitions in predicting dengue [26] were also presented in the Figure   3 for better comparison." [Pages 14-15, Lines 307-317] 7) Table 3-The clinical relevance of showing results for subgroups based on laboratory data included in the models (e.g., WBC, platelet) is unclear. These subgroup analyses should be deleted. [Response] Leukopenia (low White Blood Cell (WBC)) has been widely used for global clinicians and the WHO as first laboratory clue of clinical suspicion of dengue disease. Rapidly decreasing platelet counts was also listed as warning signs in 2009 WHO classification.

6) The WHO definitions (noted in
Therefore the accuracy of clinical diagnosis of dengue will be affected by these laboratory results. We moved the Table 3 [Response] We have added the following paragraph in the section Discussion to address this important issue. "Since we did not don't have data on day of illness (fever day) at presentation, it's impossible to know which phases of dengue natural course the patient is at…We are trying to include this information into the entry in our electronic medical record (EMR) system in the near future." [Pages 27-28, Lines 599-608]

9) The timing of the epidemic in Taiwan in 2015 should be noted under Study
Population. [Response] The timing of the epidemic in Taiwan  10) Line 147-The authors should note the number of visits excluded for incomplete records, cancellation, or re-admission. [Response] We have discussed the numbers of cases excluded due to cancellation and re-admission in the following paragraph in subsection "Study population" of section Methods. "In total, there were 100,491 visits to the ED of NCKUH (NCKUH-ED) during 2015.
Among them, 3698 patients canceled the emergency consultation and therefore were excluded… In other words, the numbers of excluded cases and merged cases were not affected by the dengue endemic." [Page 7, Lines 170-176] As shown in the following table, the numbers of cases excluded due to cancellation and re-admission essentially stayed steady throughout the entire year.  Table 1 in the text. [Response] All of the repeated text was deleted, according to your kind suggestions. Figure S1 and S2-The labels on each branch should be reformatted to avoid overwriting the graphical representation of the decision tree.

12)
[Response] Figures S1 and S2 (S2 and S3 in the revised manuscript) were reformatted to avoid overwriting the graphical representation of the decision tree. [Pages 38-39] 13) References 54 and 57 are identical. [Response] Reference 57 was removed accordingly. And what is the total number of samples (rows) in the data set? [Response] (1) The dataset contained 2,942(~60%) positive subjects and 1,952(~40%) negative subjects. Since the dataset was not highly unbalanced, we did not employ any procedure to address this issue. In this respect, we have added the following paragraph in subsection "Prediction models" of section Methods as below. "The last issue with respect to building the prediction models was how the distributions of the dataset should be handled…This issue is of concern only if the numbers of subjects in different groups, e.g. positive and negative, are highly unbalanced."

[Page 12, lines 258-263]
(2) There was no repeated variable in data set. The total number of samples in the dataset was 4894.
3) Correct statistical analysis was used to support conclusion, and there are no concerns about ethical or regulatory requirements being met. About data validation, internal validation with 2 time-repeats was used. As a suggestion, whether it is better if the models are validated with new data set (external validation)? [Response] We have not had large-scale outbreaks in Taiwan to do external validation using the data in different years since 2016 because the annual numbers of indigenous lab.-confirmed dengue cases from 2016 as of May 16, 2020 were 10, 0, 1, 31, 0 in Tainan, respectively.
Therefore, we are planning to start international collaboration with those S. E. Asia countries for external validity in the future.
Reviewer #2: 1) Objectives were clearly formulated, the study design is appropriate. [Response] Thank you very much for the kind encouragement. Therefore, we will not be able to provide the algorithm of the DNN model. In this respect, we have revised a paragraph in subsection "Prediction models" of the section Methods

Reviewer #2
1) Results are well presented. [Response] Thank you very much for the kind encouragement.

[Conclusions]
Reviewer #1 1) The authors' conclusions are supported by the data presented, and all limitations of the study were clearly reported. Discussion of the manuscript is interesting, and the authors discussed how these data can be helpful to advance our understanding of the topic under study. Results of the study reveal that machine-learning based models can be developed to identify dengue cases with four commonly available key features. This implies that the prediction models can be widely deployed to all levels of medical facilities, including hospitals and local clinics. It would be clearer if the authors emphasize more on how these models will be applied in practice. [Response] For The second dimension has to consider computer facilities that we wrote into different parts of this manuscript. For example, a local community hospital/clinic with an instrument of complete blood counts (including platelets) can provide a sentinel screening during outbreaks. Epidemic sites with adequate computer facilities containing a graphic processing unit (GPU) can carry out the training efficiently, the DNN models can be applied to achieve the highest prediction performance. In contrast, at sites with very limited or even no computer facilities, the DT models or the explicit prediction logic regression model alone can be used with a typical personal computers or lap-tops to obtain reasonable prediction performance. Once the training process is completed, the DNN model can be executed on a typical personal computer efficiently.
Alternatively, the training process of a DNN model can be executed in a centralized computer facility and then the model can be distributed to local clinics equipped with minimal computer hardware. In other words, once the prediction model is built at central lab, we may still utilize the models in remote areas, with efficient execution of the prediction software. In conclusion, the machine learning approach can facilitate medical and public health efforts to minimize the health threat of dengue epidemics

[Editorial and Data Presentation Modifications]
Reviewer #1 1) The discussion is sometimes rather long for the readers to focus, and should be more concise. [Response] The revised section Discussion is more concise with focusing on our results and further

[Summary and General Comments]
Reviewer #1 The study group used novel machine learning -based prediction models to identify dengue confirmed cases with the rationale of applying these models in health facilities where dengue confirmed tests are not available. As mentioned above, the prediction models to identify dengue cases with four commonly available key features, and can be widely implemented in all levels of medical facilities, and serve as a key component in an integrated dengue surveillance system. Overall, the manuscript is well prepared and organized. I recommend it is considered to be published with minor revision. [Response] Thank you for the kind encouragement and we hope this revised manuscript will be acceptable.

Reviewer #2
The authors extract a small number of parameters, the combination of which they conclude are predictive of dengue during an outbreak in Taiwan Artificial intelligence and other parameters may aid when laboratories are overwhelmed, but should never replace laboratory confirmation. in fact, the call is for more enhanced laboratory dengue surveillance in all countries, including low to middle income countries. [Response] We agreed with the reviewer that laboratory surveillance is crucially important and our study using machine learning methods can serve as an assisting role when surging capacity of laboratories have difficulties during large-scale outbreaks. We also include two paragraphs in the section Discussion to highlight your insightful comments:  to the UPLOAD tab, where you will find detailed instructions on how to use the tool.
If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. [Response] Dr. TS Ho representing our study group had registered as a user of the PACE and then double-checked our figure files before final uploading.

[Data Requirements]
Please note that, as a condition of publication, PLOS' data policy requires that you make available all data used to draw the conclusions outlined in your manuscript. Data must be deposited in an appropriate repository, included within the body of the manuscript, or uploaded as supporting information. This includes all numerical values that were used to generate graphs, histograms etc.. For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908 #s5. [Response] We will provide one set of raw data to generate main figures and tables.

[Reproducibility]
To enhance the reproducibility of your results, PLOS recommends that you deposit laboratory protocols in protocols.io, where a protocol can be assigned its own identifier