3.2.1 Tree based algorithms: Decision Tree, Random Forest, AdaBoost and Gradient Boost.

A Decision Tree is a non-parametric method that can be applied in problems with categorical variables (classification tree, the focus of this work) and also with continuous variables (regression tree). A tree is composed of a root node, internal nodes, and leaf nodes, and it is built successively dividing data according to one of the predictor variables [58]. To build a Decision Tree, it is necessary to define the node-splitting algorithm to minimise the impurity of the node. If the split achieves the maximum reduction of impurity, then the node is defined as a leaf [59]. The most common splitting algorithms are the Information Gain (used by Classification and Regression Tree (CART)) and the Gini index (used by Iterative Dichotomiser 3 (ID3) and C4.5 algorithms). The main advantage of using Decision Tree algorithms is the implicit feature selection during the model building process and the interpretability of results. Decision Trees are also able to handle missing values, which are commonly encountered in clinical studies [42]. On the other hand, over-complex trees do not generalise the data well, often presenting overfitting (or underfitting), and are prone to errors with a relatively small number of samples for training. The SLR sample includes eleven Decision Tree models—[42, 44–47, 50–53, 60, 61].

In contrast to Decision Trees (sometimes referred to as ‘strong learner’ approaches) optimised to solve a specific problem by looking for the best possible solution, ensemble learning techniques are based on a set of “weak learners”. Ensemble learning techniques can be categorised into three classes: (1) bagging (or bootstrapping), (2) boosting, and (3) stacking [62]. Random Forest is an ensemble technique based on bagging that combines several Decision Trees. It is built randomly from a set of possible trees with K characteristics in each node. Random in this context means that in the set of trees, each tree has an equal chance of being sampled. Multiple classification trees are obtained from bootstrap samples in order to calculate the final majority classification. The SLR sample includes three Random Forest models—[48, 50, 52]. As Random Forest models combine different Decision Trees, their results are not as easy to understand as a Decision Tree and are also more expensive computationally. Notwithstanding this, Random Forests typically outperform Decision Trees and handle balancing errors better when working with an imbalanced data set [63].

Boosting is an ensemble technique that combines k low performance models (M1, M2…, Mk) in order to improve the final model, M* [64]. The k classifiers are learned iteratively and after a Mi is learned, the weights are updated in order to generate the next classifier, Mi + 1. Performance is improved by training tuples that were misclassified by Mi. The final boosted model, M*, combines the results of each k classifier. The Adaptive Boosting (AdaBoost) [65] is the first stepping stone in boosting techniques and it uses Decision Trees with a single split (one node and two leaves), also named Decision Stumps, as “weak learners”. Gradient Boost uses a technique named forward stage-wise additive modelling that adds a new Decision Tree at each step to minimise a global cost function using the Steepest Gradient Descent method [62]. The main advantages of boosting algorithms in general, including AdaBoost and Gradient Boost, are intrinsic automated variable selection, and flexibility regarding the type of predictors and stability when handling high-dimensional data [66]. AdaBoost, in particular, is also known to be quite resistant to overfitting. While these advantages have attracted the attention of biomedical researchers [66], only one paper in the SLR sample proposed an AdaBoost model (Fahmi et al. [52]) and another, Veiga et al. [50], proposed a Gradient Boost model.

3.2.2 Support Vector Machine (SVM).

SVM is a classifier based on Vapnik’s statistical learning theory [67]. To perform classification, SVM builds hyperplanes in a multidimensional space in order to separate instances of different classes. The goal is to find the optimal separating hyperplane and, at the same time, maximise the distance between the support vectors (which are the extreme delimiters) [62, 67].

Robustness is one of the main advantages of SVM models. Data with outliers do not impact negatively in SVM model performance. While Decision Tree models benefit from interpretability, lack of transparency is a drawback of SVM models, especially when dealing with high-dimensional data sets. SVM models can also be quite memory-intensive and therefore processing large and complex data sets can be slow [62]. SVM models feature in five studies in the SLR sample—[43, 46, 48, 49, 52].

3.2.3 Neural Networks (NN).

Inspired by the human nerve system, an NN is composed of groups of nodes (or units) simulating layers of neurons. Each neuron is multiplied by weights to simulate synapses and the result is passed to the neuron of the next layer. All results received by a neuron are summed up and a mathematical formula, called an activation function, is used to convert the received value to be transported to the next layer of neurons, simulating the activation of a human neuron. This type of architecture allows the training of a NN model by adjusting the weights that connect neurons through a learning experience. During the NN training, different sets of neurons in the NN are “activated” and at the end, the goal is that the model can generalise past patterns.

The most basic NN model is a perceptron [68]. It is composed of only two layers of neurons: (1) the input layer to receive the data, and (2) the output layer to perform the prediction. Once it has a simple activation function, it is possible to solve linearly separable problems. The Multilayer Perceptron (MLP) is an evolved form of perceptron with additional layers of neurons in the middle, the hidden layers, and a more complex activation function in the neurons. These additions make the MLP very useful to solve complex problems for both classification and regression. MLPs are often referred to as NN and the terms are used interchangeably. In general, NNs present many advantages including high capacity to learn and generalise, and the ability to deal with imprecise, fuzzy, noisy, and probabilistic information [69, 70]. As such, they are widely used in health research [71–73].

MLP was a popular ML solution in the 1980s with applications in various fields. Recently the interest in this type of model was renewed due to the success of DL. Several authors classify MLP as a traditional model of ML [74, 75], but with the advent of DL, concepts of MLP were improved and it can also be classified as a DL [76]. In this paper, we classify MLP as a traditional model of ML due to the context observed in the selected proposals. Five papers employed NN models in the SLR sample: [44, 45, 49, 52, 77].

3.2.4 Naive Bayes.

Naive Bayes is a probabilistic classifier that performs classification based on the Bayes’ Theorem, selecting the most likely class according to its independent variables [78]. The term naive is due to the way the model calculates the probabilities of each event i.e. all attributes of the data set are equally important and independent.

In general, Naive Bayes models are simple, fast and effective, and perform well when a data set contains outliers or is missing data [62], common features in many health data sets. However, Naive Bayes models are not without drawbacks. They assume that all attributes of a data set have the same importance which is often not true. If a data set has large numbers of attributes, the reliability of the results may be limited. Four works in the SLR sample applied Naive Bayes models—[43, 45, 48, 52].

3.2.5 Logistic regression.

Logistic Regression is a classification technique based on idea of modelling the odds of belonging to Class 1 using an exponential function [62]. In this technique, the dependent variable, Y, is binary and the independent variables, X = {x₁, x₂, …, x_n}, are used to estimate the value of Y by using a logistic function. The goal is to find an optimal hyperplane, that separates the two target classes (binary classification). In case of multi-class problems, the one-vs-all strategy can be used to address the problem.

Some advantages of this model include dealing with categorical independent variables and a high degree of reliability. As disadvantages, this type of model does not generalise well when using a large number of features, it is vulnerable to overfitting, and cannot solve non-linear problems, requiring a transformation of non-linear resources [79]. Two works employed Logistic Regression in the SLR sample—[47] and [52].

3.2.6 k-Nearest Neighbours (kNN).

kNN is a classification technique that defines a minimum number of neighbours, k, and calculates the distance, the similarity, of each data element with respect to its k neighbours. There are many measures of similarity, and the most commons approaches are the Euclidean distance and the Minkowski metric [62]. The most frequent class among the neighbouring k is determined as the class of the target instance [80].

Simplicity is the main advantage of kNN. kNN is a good choice when working with a small low-dimensional data set however it can be extremely inefficient when dealing with large data sets because it is computes all pairwise distances [62]. Only one paper in the SLR sample proposed a kNN model, Fahmi et al. [52], using Euclidean distance with k = 5.

3.2.7 Convolutional Neural Networks (CNN).

A CNN is a DL technique. CNNs are designed to process input data in the form of multiple arrays [81]. A basic CNN comprises convolutional layers, pooling layers, nonlinear function (generally ReLU), and fully connected layers. Units in a convolutional layer are organised into feature maps and each unit is connected to local patches in the feature maps of the previous layer. It is done by using a set of weights called filters. The result of this local weighted sum is passed through a nonlinear activation function. All units in a feature map share the same filter bank, and different feature maps in one layer can use different filter banks. The result of this whole process feeds fully connected layers resulting in a final classification. As discussed earlier, DL models, including CNN, often outperform traditional ML models however their adoption in health settings has suffered due to an inherent lack of transparency.

Although the methodology of training and testing models is clearly well-defined, the resultant models themselves can be often unexplainable to humans [82]. Even when techniques are used to select attributes resulting in good model performance, the relationships between those attributes and the output classification may not directly track causal relationships in the real world [82]. One paper in the SLR sample, Ho et al. [47], uses a CNN, DenseNet. DenseNet is a CNN architecture in which each layer is connected to all others within a dense block [83]. In this case, all layers can access feature maps from their preceding layers enabling heavy feature reuse. As a direct consequence, the model is more compact and less prone to overfitting. Furthermore, each individual layer receives direct supervision from the loss function through the shortcut paths, which provides implicit deep supervision [84].

3.3.1 Dengue.

Of the 15 relevant papers in the SLR sample, 12 included the diagnosis of Dengue in their studies including binary classification and multi-class classification:

• Binary classification
  • Dengue or not Dengue: Tanner et al. [42], Fathima and Hundewale [43], Sajana et al. [44], Gambhir et al. [45], Sanjudevi and Savitha [46], Ho et al. [47];
  • Severity of Dengue: Tanner et al. [42], Potts et al. [60], Phakhounthong et al. [61];
  • Risk of Dengue: Faisal et al. [77]; and
  • DHF or not: Arafiyah et al. [48]
  
  Tanner et al. appeared twice because they presented two distinct classification problems.
• Multi-class classification
  • DF, DHF or DSS: Fahmi et al. [52]; and
  • DHF1, DHF2 or DHF3: Thitiprayoonwongse et al. [51]

Tanner et al. [42] proposed two Decision Tree models. First, the Dengue Diagnostic Model (DDM) sought to classify if a patient had Dengue or not using 1,200 records of patients with acute febrile illness. The second, the Dengue Severity Prediction Model (DSPM), sought to classify the severity of Dengue in adults using data from 161 patients. A C4.5 Decision Tree classifier was built using the Inforsense software. A k-fold cross-validation approach (k = 10) was used to avoid model over-fitting. Both models presented good performance however the DDM performed better across all metrics, i.e., sensitivity (71.2%), specificity (90.1%), overall error rate (15.7%) and Area Under the Curve (AUC) (0.88). This is unsurprising given the larger data set available to the DDM.

Fathima and Hundewale [43] compared two classification models, SVM and Naive Bayes, to classify if a patient had Dengue or not. To determine the best SVM hyperparameters, a Grid Search was performed changing the gamma parameter and the cost (c). Despite executing the Grid Search, neither the best configuration nor the configuration of the Naive Bayes model was detailed. In general, the SVM model presented the best performance, despite its low sensitivity (47%). The Naive Bayes model presented high sensitivity and very low specificity, accuracy and risk rates, all above 18%.

Sajana et al. [44] proposed three models for binary classification of Dengue using clinical and laboratory data: an MLP, and two Decision Trees (C4.5 and CART). Like Fathima and Hundewale [43], the configurations of the models were not detailed. The CART model presented the best results, achieving 100% in all metrics (accuracy, sensitivity, precision and F-Measure). There is no mention regarding the use of feature selection and hyperparameter optimisation. Given the results, it is possible that the models were overfitting due to the small amount of data available, i.e., only 20 records.

Gambhir et al. [45] proposed three models (NN, Decision Tree and Naive Bayes) to classify whether a patient had Dengue or not. The configuration of the models were described in the paper and are summarised in Table 1. K-fold cross-validation (k = 10) was used to validate and test the models. The NN presented the best results—79.09% accuracy, 55.55% sensitivity, and 88.5% specificity. However, the other models achieved similar performance. Gambhir et al. [45] did not describe whether hyperparameter optimisation or feature selection was applied in the data set.

Sanjudevi and Savitha [46] compared Decision Tree and SVM models to classify whether a patient had Dengue or not. Model configurations were not described. The WEKA tool was employed to run the experiments and calculate the metrics. The SVM model obtained the best results achieving 100% sensitivity, 100% specificity, 100% precision, and AUC of 99%. The extremely high performance results suggest model overfitting.

Ho et al. [47] proposed three models to classify Dengue using clinical data—a Decision Tree, a Logistic Regression, and a CNN. Models were validated and tested using k-fold cross-validation (k = 10). Feature selection was performed using crude odds ratio and adjusted odds ratio analysis. From 18 available attributes, four were initially selected—age, body temperature, White blood cells (WBC) count, and Platelet count (PLT). Three experiments were performed with more attributes: (1) six attributes, the four previously mentioned plus gender and haemoglobin count; (2) 11 attributes, the previous six and five more vital signals; and (3) the entire data set with 18 attributes. Results suggested that when using only four attributes, the AUC in all experiments were close to 84%. The CNN performed marginally better than Decision Tree and Logistic Regression models.

Potts et al. [60] proposed Decision Trees to classify pediatric patients into “severe” or “non-severe” Dengue cases. The stopping rules used to create the trees were described in the paper and are summarised in Table 1. Five scenarios were evaluated with different definitions of Severe Dengue: (1) Tree 1 considered Severe Dengue as DSS, and used four of 11 available attributes (WBC, Hematocrit (HTC), Monocyte percent (MONOP), PLT); (2) Tree 2 defined Severe Dengue as DHF Grade 3 or 4 or pleural effusion index (PEI) > 15, and used five attributes (Age, WBC, PLT, Neutrophil percent (NEUTP), Aspartate aminotransferase (AST)); (3) Tree 3 defined Severe Dengue as DSS or required intravenous fluid; (4)Tree 4 defined Severe Dengue as DSS or PLT less than 50,000; and (5) Tree 5 defined Severe Dengue as DSS or received fluid intervention. Trees 3, 4 and 5 were not described in the work because, according to Potts et al. [60], they did not obtain any significant improvement in relation to Trees 1 and 2. Both Decision Trees 1 and 2 have the same initial splitting variable, WBC, reinforcing the utility of this variable in distinguishing Severe Dengue. Models were validated and tested using k-fold cross-validation (k = 5) and results (the work did not explicitly present any metrics for evaluating the models, such as accuracy or sensitivity. However, in the results, a table was presented with values that can be interpreted as the sensitivity of the “Severe” and “non-Severe” classes), suggested that Trees 1 and 2 achieved a sensitivity metric in excess of 90% for the “Severe” class, while the sensitivity for the “non-Severe” class was below 50%.

Phhakhounthong et al. [61] proposed a CART Decision Tree model to classify Dengue severity based on clinical and laboratory attributes. They performed a Logistic Regression analysis to determine the significance of each attribute to compose the tree. In their case, the most significant factor in predicting severe dengue was haematocrit. The results obtained from k-fold cross-validation (k = 10) for binary classification of Severe Dengue were 60.5% sensitivity, 65% specificity, and 64.1% accuracy. Phhakhounthong et al. [61] state that tree pruning and tuning parameters were applied to optimise the model but did not describe the settings used for the experiment. Despite having performed a feature selection with Logistic Regression, the results did not exceed 65% in the metrics evaluated.

Faisal et al. [77] sought a binary classification of Dengue risk, differentiating patients as “high risk” or “low risk”. An MLP model was proposed for the classification task, and a Grid Search technique was performed to optimise the model configuration, changing four parameters: number of neurons, momentum, learning rate, and number of iterations (it is noteworthy that in the Grid Search technique process, each attribute was tested individually). Seven attributes were selected using Self Organising Map (SOM) and the model achieved 70% accuracy.

Thitiprayoonwongse et al. [51] classified a patient as DF, DHF1, DHF2 and DHF3 using a Decision Tree. Two data sets were used, one from the Srinagarindra Hospital and another from the Songklanagarind Hospital. Three experiments were performed: (1) using only data from the Srinagarindra Hospital; (2) using only Songklanagarind Hospital data; and (3) using data from both hospitals. The attributes of both two data sets were not described, but the attributes selected to compose the Decision Tree for each experiment were presented. In Experiment (1), six attributes were selected: shock, leakage, bleeding, platelet, liver size and je-vaccine. In Experiment (2), nine attributes were selected: shock, leakage, bleeding, platelet, abdominal pain, rash, uri, HTC, AST. In Experiment (3) eight attributes were selected: shock, leakage, bleeding, platelet, Alanine transaminase (ALT), lymp, WBC (count and minimal count). The configuration of the Decision Tree was changed only in the degree of confidence parameter in Experiments (1) and (2); no detail was provided for Experiment (3). Experiment (1) presented the best overall results. It is interesting that even with the addition of one more data set, Experiment (3) largely did not achieve superior results. The class posing the greatest classification challenge, DHF1, reported the lowest values in all three experiments although all were greater than 80%.

Arafiyah et al. [48] proposed and evaluated three models to classify DHF or not DHF—Random Forest, SVM and Naive Bayes. Unfortunately, there is no information about the model configurations. The models were trained and the metrics were calculated using the Orange tool. Random Forest achieved better results than SVM and Naive Bayes models: 79.6% accuracy, 84.1% precision, 82.2% sensitivity, 83.1% F1-Score and 89.8% AUC. No details on hyperparameter optimisation or feature selection were provided.

Fahmi et al. [52] evaluated eight models for classifying Dengue into three categories: DF, DHF and DSS. The models included NN, SVM, kNN, Decision Tree, Random Forest, Naive Bayes, AdaBoost, and Logistic Regression. The configuration of all models were described and are summarised in Table 1. Experiments were carried out in two different scenarios: (1) without feature selection, and (2) with feature selection using the ReliefF technique (is an algorithm developed by Kira and Rendell in 1992 [95] that takes a filter-method approach to feature selection that is notably sensitive to feature interactions). In both scenarios, the best result was obtained by the NN model with 71.3% accuracy, 70.8% precision, and 71.3% sensitivity in Scenario (1) and with 72% accuracy, 71.5% precision, and 72% sensitivity in Scenario (2). Results showed that the feature selection did provide significant improvements.

In summary, those studies in our sample addressing the diagnosis of Dengue primarily focused on binary classification (11); only two studies performed multi-class classification. Multi-class classification studies sought to classify Dengue subtypes [51] or different levels of disease severity [52]. The prevalence of binary classification reflects its simpler nature. Multi-class classification is both more complex to perform and interpret, and consequently results are often inferior to simpler models. This is reflected in our SLR [51, 52]. Tree based models (Decision Tree and derivatives) were the most common technique used in Dengue classification (10); nine of which used simple Decision Trees, often obtaining better results than other benchmark models. It is important to highlight that despite tree-based learning algorithms being broadly used for classification problems due to their simplicity for implementation and interpretability of results, the usage of imbalanced data sets can skew the performance of such models, exacerbating inadequacies inherent in the tree splitting criterion [96]. It was noted that a number of studies likely suffered model overfitting [44, 46]. Further analysis is not possible due to the lack of detail in their publications, although it should be noted that each of these studies used the smallest data sets (see Section 3.4). The usage of different data sets and lack of detail regarding both model configuration, feature selection, and hyperparameter optimisations made comparisons of these studies difficult. For example, six of the 12 studies [43, 44, 46, 48, 53, 61] did not present any description of their proposed models thus adversely impacting future reproducibility.

3.3.2 Chikungunya.

Only one study in the SLR sample sought to classify Chikungunya. Hossain et al. [49] proposed a Specialised Belief Rules System (BRBES) to classify Chikungunya using clinical data containing vital signs and symptoms, and considering severity classes as output (very high, high, medium and low). The BRBES system was compared with a NN, an SVM and a Fuzzy Logic Based Expert System (FLBES) as well as expert opinions. Due to the scope of our work, we consider only the ML models for analysis, i.e., NN and SVM. The NN outperformed the SVM model, obtaining an AUC of 81.1% vs 80.8%. Notwithstanding the small difference in performance between these two ML models, neither models outperformed the BRBES system. A significant limitation of this study is the lack of detail on model configurations thereby adversely impacting comparability and reproducibility.

3.3.3 Zika.

In relation to the Zika classification, only one study was identified. Veiga et al. [50] sought to classify suspected cases of CZS using clinical and non-clinical data. The authors compared five algorithms: kNN, CART Trees, Random Forest, AdaBoost and Gradient Boost. After performing a Grid Search, only two of the candidate models were selected and described in the final publication—(1) the Random Forest model was used with a data set without textual data to address a binary classification (“Discarded cases” and “Somewhat probable”); and (2) the Gradient Boost model was used with a data set with supplementary textual data to handle a multi-class output (“Discarded cases”, “Somewhat probable”, “Moderately probable” and “Highly probable”). For the binary classification, the Random Forest model obtained 91% sensitivity and and F1-Score of 83% for the “Discarded cases” class however exhibited significantly poorer results for the “Somewhat probable” class with 50% sensitivity and an F1-Score of 61%,. During the execution of Grid Search, the tree-based models obtained a similar performance and all were superior to kNN. The small but better performance of Random Forest in relation to other tree-based models is probably due to its bootstrapping process that helps to avoid overfitting when using small data sets as per Group 1 (272 samples).

For the multi-class problem, the Gradient Boost model presented good performance mainly for the “Discarded cases” class with 91% in all metrics (precision, sensitivity and F1-score). As the amount of data used in this experiment is greater (1109 samples) than the binary classification, the Gradient Boost obtained a better performance. However, Veiga et al. [50] did not provide details on the proportion of data in each class. As such, it is not possible to analyse whether any data imbalance impacted the models performance. This is the only study where the code of the final models are available for download (https://github.com/rafael-veiga/Classification-algorithm-of-Congenital-Zika-Syndrome-characterizations-diagnosis-and-validation).

3.3.4 Differential arboviral diagnosis.

Given the difficulties in the differential diagnosis of arboviruses discussed in Section 1, it was surprising that only one study was identified that sought to distinguish between two different arboviral diseases, in this case Dengue and Chikungunya. Lee et al. [53] proposed models to differentiate between DF, DHF and Chikungunya cases. Four experiments were presented—(1) DF and Chikungunya using only clinical data; (2) DF and Chikungunya using clinical and laboratory data; (3) DHF and Chikungunya using clinical data only; and (4) DHF and Chikungunya using clinical and laboratory data. For each classification, a Decision Tree model was developed using R software. Details on model configuration were not provided. Results suggested that Decision Trees using clinical and laboratory data outperformed the models using only clinical data.

3.3.5 Cross-validation.

It is worth noting that a number of studies used cross-validation techniques to validate and test their models. [42, 45, 47, 50, 52, 61] used k-fold cross-validation with k = 10; and [60] also used k-fold but with k = 5. Lee et al. [53] was the only study that applied the leave-one-out (LOO) cross-validation. Commonly, cross-validations are recommended when handling with small data sets, and in an attempt to minimise the learning bias. LOO cross-validation is a type of k-fold cross-validation in which k is the number of samples in the data set. Therefore, despite taking advantage of each data point, LOO cross-validation can be computationally expensive, especially if the data set is large. While Lee et al. had a data set composed of 1,034 samples, they did not mention anything about the computational effort needed to execute the experiments.

3.4.1 Summary of data sets and attributes.

Table 2 summarises the data sets used by the included studies in this SLR by number of records, number of attributes, input for models, period of the data, and location. While the number of attributes presented in this table were described as available in a focal data set, in some cases, the studies did not use all of them for training and testing their proposed models (see Sub-section 3.4.2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of the data sets used to evaluate Machine Learning and Deep Learning models for arboviral diseases classification.

https://doi.org/10.1371/journal.pntd.0010061.t002

Tanner et al. [42] used a data set comprising 1,200 patient records from Singapore and Vietnam with acute febrile illness. The data set is composed of 15 clinical (symptoms and vital signs) and 26 laboratory attributes. The selected attributes for the Dengue classification model were PLT, WBC and Lymphocyte count (LYMPH), body temperature, haematocrit count, and Neutrophil count (NEUT). For the classification of Dengue severity, the selected attributes were PLT, the crossover value (Ct) of the real-time Reverse transcription polymerase chain reaction (RT-PCR) for Dengue viral RNA, and the presence of anti-Dengue Immunoglobulin G (IgG) antibodies.

Fathima and Hundewale [43] used a data set comprising 5,000 records of patients with Dengue from Chennai and Tirunelveli, India. The data set includes details on 29 patient symptoms. The structure of the data set is not provided in sufficient detail to infer the extent to which the data set is balanced or unbalanced; most of the data would appear to be related to non-Dengue patients.

Sajana et al. [44] used the data collected from various medical wards of hospitals in Vijayawada, India; it comprises only 20 records with 12 attributes. As no feature selection technique is referenced in the paper, we assumed that all attributes were used for model training.

Gambhir et al. [45] used clinical and non-clinical data acquired from patients in Delhi from 2015 and 2016. The data set contains 110 records—85 positive Dengue cases and 25 negative Dengue cases. Each record has 16 attributes, of which nine are clinical data (age, gender, vomit, abdomen pain, chills, bodyache, headache, weakness, and fever) and the remainder physical examination/laboratory data (temperature, heart rate, PLT, dengue antigen NS1 or serology (Immunoglobulin M (IgM), IgG)). In the original work, these data were mistakenly classified. Gambhir et al. [45] considered age, gender, vomit, abdomen pain, chills, bodyache, headache, weakness, and fever as non-clinical data; and PLT, temperature, heart rate, dengue antigen NS1, IgM, IgG, dengue NS1 antigen as clinical.

Sanjudevi and Savitha [46] used a data set composed of 108 records with 17 attributes. The attributes were not detailed in the paper and no feature selection technique was performed.

Ho et al. [47] used data from the National University Hospital Cheng Kung (NCKUH) in Tainan City, Taiwan. The data set comprised 4,894 records of clinical and laboratory data including 2,942 cases of laboratory-confirmed Dengue cases and 1,952 non-Dengue cases. Ho et al. [47] analysed odds ratios to select four attributes and create a subset of data; two additional subsets were created with six and 11 attributes based on the initial subset. In the experiments, the data set with all attributes was also used for comparison purposes, however there was no evidence that more attributes contributed to improved model performance.

Potts et al. [60] used data from 1,384 pediatric patients with Dengue and DHF, with 11 attributes. After initial screening, 1,230 records were included in the analysis—208 cases of DHF, 374 of DF, and 648 of Other febrile illness (OFI). Data was collected in Thailand for the periods 1994 to 1997, 1999 to 2002, and 2004 to 2007.

Phakhounthong et al. [61] used a data set comprising 1,225 records related to febrile episodes in children from Angkor Hospital for Children, Cambodia. From those 1,225 records, 198 were confirmed cases of Dengue; only 38 were Severe Dengue cases. The data set included information about demographic, clinical and laboratory data. Logistic Regression was used and these five attributes were selected for model training.

Faisal et al. [77] used a data set with records of 210 patients with 40 attributes, divided into demographic, clinical, laboratory and Bioelectrical Impedance Analysis (BIA) parameters measurements to classify risk of Dengue. Laboratory data were used to classify baseline patient risk and thus create the output attribute for model training. This procedure was performed using an unsupervised model, SOM. After that, another SOM model was used to perform feature selection to define the attributes to be used as input for the proposed model; ultimately seven attributes were selected.

Thitiprayoonwongse et al. [51] used two data sets: one composed of information from 524 patients from Srinagarindra Hospital, Thailand and another with 477 patients from Songklanagarind Hospital, Thailand to classify DF, DHF1, DHF2 or DHF3. The Decision Tree model selected different attributes for each experiment.

Arafiyah et al. [48] used a medical data set comprising 213 records using only four clinical data sources—temperature, presence of spotting, presence of bleeding, and tornikuet test. The complete list of the attributes present in the data set was not described by the authors, so there is no way to know if or how feature selection was applied.

Fahmi et al. [52] used a data set that was provided by the Disease Prevention and Control Division in Central Java, Indonesia to classify DF, DHF or DSS. After the verification of the missing values, selection of relevant attributes, and data normalisation, the final data set comprised 14,019 records with 16 attributes including demographic, epidemiological, clinical and laboratory (hematological) information. Despite having 16 attributes available, after the application of the feature selection procedure, only ten attributes remained for model training and testing based on their importance.

As discussed, only one study addressed Chikungunya classification. Hossain et al. trained and tested their model using a data set comprising 250 records collected from various hospitals in Dhaka and Chittagong, Bangladesh. The data set had five attributes indicative of patient symptoms i.e. fever, muscle pain, joint pain, headache and swelling in the joints, each classified as high, medium or low intensity.

In relation to Zika classification, Veiga et al. [50] sought to classify suspected cases of CZS using clinical and non-clinical data. A data set with 1,501 records of live newborns suspected of microcephaly reported in the Public Health Event Registry (RESP) and the National Birth Registration System (SINASC) from Brazil was considered. This data set contains information about demographic, epidemiological, clinical (signs), laboratory (serological and others). From 13 attributes, seven were used as input for the model. Additionally, there is also textual data provided by the health professional when registering the newborns’ information in the system such as reports, descriptions and other possible observations. Veiga et al. [50] separated the records into two groups where Group 1 contained only clinical and non-clinical data (272 records), and Group 2 containing clinical, non-clinical and complementary textual notes (1,109 records). The most frequent terms presented in the notes were used to assist the classification. Group 2, which considered these textual notes, obtained better results compared to Group 1.

Lee et al. [53] used demographic, epidemiological, clinical and laboratory data with 1,034 records to train and test their model to distinguish between two different arboviral diseases: Dengue and Chikungunya. While 36 attributes are identified in the study, only five were used for training and testing the model i.e. period of symptoms, fever, fever (duration), bleeding and PLT. Of the 1,034 records, 917 were related to adult Dengue patients confirmed by Polymerase chain reaction (PCR) test, including 55 records related to DHF. 117 were records related to Chikungunya patients confirmed by RT-PCR. The Chikungunya data were collected in August 2008, while the Dengue data were collected during the large 2004 Dengue outbreak, both in Singapore.

It is important to note that none of the included studies explicitly describe or discuss how they handle imbalanced data (between-class imbalance). Given how the data sets were reported in the papers, none of the models were trained with a data set with similar number of records per class, as presented in Table 3). The data set used by Tanner et al. [42] presents this imbalancing issue in which the DSS class represents only 0.016 of the entire data set while the non -DF class presents 0.696. A similar situation is found in the data set used by Lee et al. [53] in which DHF is 0.053 and DF is 0.833. Although not explicitly mentioned, Lee at al. [53] applied the LOO cross-validation, as described in sub-section 3.3.4, that can be considered an alternative when evaluating models with imbalanced data sets. According to He et al. [96], when presented with complex imbalanced data sets, most standard learning algorithms “fail to properly represent the distributive characteristics of the data and resultantly provide unfavorable accuracies across the classes of the data”. It happens essentially because ML models learn by reducing the error and do not take into consideration the class proportion. In health, where the minority class is commonly the positive case for the target disease (or the rare case), it is desirable that a classifier provides high accuracy for the minority class, without severely impacting on the performance of the majority class [96]. Three exceptions were identified in our sample. Gambhir et al. [45] and Ho et al. [47] used data sets in which the number of Dengue cases is larger than non-Dengue; and Fahmi et al. [52] used a large data set, in which there were more DHF cases than DF.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Distribution of samples per classes.

https://doi.org/10.1371/journal.pntd.0010061.t003

Additionally, the combination of imbalanced data and small sample size issue was also found in this SLR. For example, Gambhir et al. [45] used a data set composed of 110 records with 85 positive cases of Dengue and 25 negative cases. Other works also presented a small data set, such as [44, 46, 48, 49, 77], having less than 250 records each, however none of them described the distribution of the classes. In such cases, traditional learning algorithms may fail to use inductive rules over the sample space [96]. When samples are limited, the rules formed can become too specific, leading to overfitting [96]. This is likely to be the case in Sajana et al. [44] and Sanjudevi and Savitha [46]. To address these issues, some methods, such as sampling, cost-sensitivity, kernel-based and active learning [96] are available in the literature.

3.4.2 Attributes of the data sets.

Fig 3 presents the types of attributes found in the data sets described previously. Demographic, epidemiological and clinical (symptoms, signs and co-morbidities) data were grouped as resource-limited attributes following the terminology presented by Lee et al. [53]; specific equipment is not specified for these data as they were collected at the time of the appointment. Laboratory attributes (hematological, biochemical and serological) and others are grouped as well-resourced attributes because they require specific equipment to be performed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Attributes found in the data sets.

https://doi.org/10.1371/journal.pntd.0010061.g003

Table 4 presents a summary of all demographic, epidemiological and clinical data present in the data set used by the 15 included studies. Despite the focus on studies that used clinical data as input for the classifiers as per [48, 49], we also found cases in which clinical data was used together with other types of data, e.g. [42, 44, 45, 47, 50–53, 60, 61, 77]. Fathima and Hundewale [43] and Sanjudev et al. [46] neither provided details about the data set nor the attributes used in their studies. Thitiprayoonwongse et al. [51] only described the final selected attributes in the data set.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Summary of all demographic, epidemiological and clinical data presented in data set used by the primary studies.

https://doi.org/10.1371/journal.pntd.0010061.t004

Age, gender, weight and residence (state) were the demographic information present in the data set described in [45, 47, 50, 52, 53, 60, 77]. The most common clinical data used to classify arboviral diseases were: abdominal pain, fever, temperature, and bleeding.

Table 5 presents the summary of all non-clinical (laboratory and others) data found in this SLR As expected, none of the 15 primary studies used only non-clinical data (since these works were excluded from our SLR). The non-clinical data presented in the Table 5 is quite diverse. The most common attribute used as input for models was the PLT used in nine studies—[42, 44, 45, 47, 51–53, 60, 61]. For model training, most non-clinical data hematological in nature e.g. PLT, WBC and HTC. Dengue IgM, Dengue IgG and Dengue NS1 antigen were used by Gamhbir et al. [45]; Zika Virus (ZIKV) RT-PCR, Toxoplasmosis, Rubella, Cytomegalovirus, Herpes Symplex, and Syphilis infections (TORCHS) serology (others except Zika) and neuroimaging reports (US, CT, MRI) were used by Veiga et al. [50]. The biochemical data used in the models were ALT, creatinine and liver size.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Summary of all non-clinical data (laboratory and others) presented in the data set used by the primary studies.

https://doi.org/10.1371/journal.pntd.0010061.t005

Lee et al. [53] compared two cases in relation to the attributes present in their data set: (1) a resource-limited case in which only data available at the time of hospital presentation was used (clinical data), and (2) a well-resourced case in which clinical and laboratory data were used for classification. As per Sub-section 3.3.4, the majority of the best results of the classification of DF, DHF or Chikungunya was obtained using a set of clinical and laboratory data. These results demonstrate that the restricted usage of clinical data for multi-classification may not be as satisfactory as when clinical and non-clinical data is combined. Based on their results, we also highlight that the use of few attributes (they considered only five attributes) is feasible for the classification of DF, DHF and Chikungunya with good performance. Regarding the number of attributes, a similar conclusion was found by Ho et al. [47]. They stated that the addition of more attributes did not provide any significant improvement in the results in any of their models, so the subset with only four attributes was able to provide as much essential information as possible and can be easily collected with minimal cost. Ho et al. [47] highlight two major findings: (1) their “high-sensitivity models can be an effective surveillance tool in the pre-epidemic period” to complement clinical diagnosis, and (2) high-specificity models, as in their proposal, can be exploited to identify laboratory-confirmed dengue cases at outbreak sites for real-time monitoring of epidemic trends.

It is interesting to note that the data sets used are quite different with regard to the number of samples and attributes. In addition, the included studies did not use similar attributes for training and testing their proposed models. In general, the included studies did not describe clinical and non-clinical data in a standardised way, making it difficult to summarise these data without a health professional. Another challenge when analysing the data sets is related to the lack of detailed description in the included studies. Data description and experiment methodology are fundamental for replicability of studies; in half of the cases, there is no information about the model configuration. Additionally, although all studies used data sets, none of these are available for usage further adversely impacting reproducibility.

3.5.1 Sensitivity.

Sensitivity, also known as recall, was used by seven studies—[42–48, 50–53, 61]. It defines how well a model correctly predicted TP cases. It is calculated as the number of TP divided by the sum of TP and FN, as shown in Eq 1. (1)

3.5.2 Accuracy.

Accuracy was the most common metric used among the studies in this SLR—[42–48, 50–52, 77]. It is used to find out how much a model is right. It is calculated as the sum of TP and TN divided by the total of samples, as shown in Eq 2. (2)

3.5.3 Specificity.

Specificity was used by five studies included in this SLR—[42, 43, 45–47, 51, 53, 61]. This metric determines how well the model correctly predicted TN cases. It is calculated by the number of TN divided by the sum of TN and FP as per Eq 3. (3)

3.5.4 Precision.

Precision was used in [44, 48, 50, 52, 61]. This metric defines how many cases classified as TP actually are TP, and is calculated as the number of TP divided by the sum of TP and FP, as shown in Eq 4. (4)

3.5.5 Receiver Operating Characteristic (ROC) and Area Under the Curve (AUC).

ROC and AUC were used in four studies—[46, 48, 49, 53]. The ROC curve is a graph to analyse the discriminating ability of the model, that is, how well the model is able to divide between two classes. It is a graph with the True Positive Rate (TPR), the sensitivity, in the x axis, and the False Positive Rate (FPR), the complement of the specificity, in the y axis. Based on ROC, it is possible to calculate the AUC. The AUC summarises the ROC curve in a single value, aggregating all the ROC thresholds. Its result varies between 0 and 1; an AUC of 0.5 represents a test without discriminating ability, while an AUC of 1.0 represents a test with perfect discrimination [98].

3.5.6 F1-Score.

The F1-score is the harmonic mean between two metrics: precision and sensitivity. It is used when the objective is to seek a balance between these two metrics, being calculated as presented in Eq 5. This metric was used in [44, 48, 50]. (5)

To address the imbalanced data issues mentioned earlier, more informative assessment metrics can be used to evaluate models. These include AUC, precision-recall curves and cost curves. Notwithstanding suspected imbalances, most of the included studies employed traditional metrics such as accuracy, sensitivity and specificity. Relatively few (3) used AUC [42, 46, 49].