https://orcid.org/0000-0002-7543-1273
Correction
6 Dec 2023: Terabe ML, Massago M, Iora PH, Rocha TAH, de Souza JVP, et al. (2023) Correction: Applicability of machine learning technique in the screening of patients with mild traumatic brain injury. PLOS ONE 18(12): e0295717. https://doi.org/10.1371/journal.pone.0295717 View correction
Figures
Abstract
Even though the demand of head computed tomography (CT) in patients with mild traumatic brain injury (TBI) has progressively increased worldwide, only a small number of individuals have intracranial lesions that require neurosurgical intervention. As such, this study aims to evaluate the applicability of a machine learning (ML) technique in the screening of patients with mild TBI in the Regional University Hospital of Maringá, Paraná state, Brazil. This is an observational, descriptive, cross-sectional, and retrospective study using ML technique to develop a protocol that predicts which patients with an initial diagnosis of mild TBI should be recommended for a head CT. Among the tested models, he linear extreme gradient boosting was the best algorithm, with the highest sensitivity (0.70 ± 0.06). Our predictive model can assist in the screening of mild TBI patients, assisting health professionals to manage the resource utilization, and improve the quality and safety of patient care.
Citation: Terabe ML, Massago M, Iora PH, Hernandes Rocha TA, de Souza JVP, Huo L, et al. (2023) Applicability of machine learning technique in the screening of patients with mild traumatic brain injury. PLoS ONE 18(8): e0290721. https://doi.org/10.1371/journal.pone.0290721
Editor: Md Asaduzzaman, Staffordshire University, UNITED KINGDOM
Received: December 14, 2022; Accepted: August 14, 2023; Published: August 24, 2023
Copyright: © 2023 Terabe et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are available here: https://figshare.com/s/252ab41f8c317206c4a7.
Funding: The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The increase in the demand even in mild cases of illness associated with low capacity of health care services leads to overcrowding of emergency health services in different countries of the world [1, 2]. Among the diseases with low severity which is on expansion, there is the mild traumatic brain injury (TBI), which corresponds to 81.01% of all registered 69 million cases annually or approximately 740 cases per 100,000 people worldwide [3, 4]. Previous studies have shown the overuse of the head computed tomography (CT) in patients with suspected mild TBI. [5, 6]. However, only 5.2 to 9.4% of patients have intracranial lesions and 0.2 to 3.5% require neurosurgical intervention [7–9].
An example of this overcrowding can be seen in Brazil, where the probability of hospitalization due to TBI has tripled between 2001 and 2017 [10]. In 2019 alone, the country witnessed over 100,000 cases of TBI-related hospitalizations, resulting in an estimated cost of 18,489,452.36 Brazilian reals (equivalent to U$4,587,953.44). On average, each patient required approximately 6.3 days of hospitalizations [10].
Early recognition of high-risk clinical factors can help identify a subset of patients who are likely to have intracranial lesions and increase the survival of individuals [11–13]. However, performing CT in all patients is not feasible, and time and resources should be focused on those most likely to have lesions that require neurosurgical intervention [14].
To solve this problem several studies have been carried out, and guidelines have been developed and validated bringing a set of prediction rules to help in making appropriate decisions to determine which patients with mild TBI are indicated to undergo CT of the head. Among them, the most cited are the Canadian CT Head Rule and New Orleans Criteria [15, 16].
Although these guidelines aid in screening patients with mild TBI in their countries of origin, the application of these guidelines in Brazil leads to external validity issues, which is limited to patients who had loss of consciousness, post-traumatic amnesia or witnessed disorientation [8, 11, 17–20]. Furthermore, these guidelines performed differently in countries with distinct socioeconomic situations, due to the difference in the target audience profile [9], mechanism of trauma [21, 22] and availability of financial resources [8, 23].
These gaps demonstrate that protocols should be validated and applied according to the intrinsic characteristics of the target audience, including culture, sociodemographic patterns, epidemiological profile, and the availability of medical resources and equipment in each location [11, 17]. Thus, finding and improving the most contextually relevant and location-specific clinical decision protocol has been the aim of study of Fournier et al., which adjusted the Canadian CT head rule to people 75 years old or over [24]. Vedin et al., in their turns, studied the applicability history of patients in the final decision of the people under 59 years old [13].
Previous research has shown that machine learning techniques, specifically grouping and analyzing variables that indicate an inclination to certain pathology, can provide a global view of the patients’ clinical status [25, 26]. This global view can then assist in complex clinical decision making, and reduce patients’ time spent in health centers by automating several functions [27]. In addition, machine learning models have been shown useful in predicting hypertension in Qatar [25], diabetes in Brazil [28] and TBI in Tanzania [26].
Despite these advancements, there are still no studies focused on the development of risk predictor tools for intracranial injuries in adults with mild TBI in Brazil. Hence, this study aims to evaluate the applicability of machine learning to screen patients with mild TBI evaluated at the Regional University Hospital of Maringa using a supervised classification model.
Methodology
Study design
This is an observational, descriptive, cross-sectional, and retrospective study, using machine learning (ML) developed computational models to predict which patients with an initial diagnosis of mild TBI are recommended to undergo head CT. We evaluated the need for imaging as lesions identified on CT which required evaluation by neurosurgeons and presented the following outcomes: requiring hospitalization for neurological observation or neurosurgical intervention, requiring transfer to advanced neurosurgical care, and patients who died with TBI as the main cause.
Data sources
This project was approved by the Ethics Committee on Research Involving Humans at the State University of Maringá (Registration No. 3.952.659/20). We used secondary data from electronic medical records of patients with mild TBI evaluated at the Regional University Hospital of Maringá (HUM) in 2018 available in the Hospital and Outpatient Management System of the Unified Health System. For this reason, the resolution number 466/2016 of the Brazilian Ministry of Health allows us to work with this data without an informed consent form.
The Regional University Hospital of Maringa (HUM) is one of the referral hospitals in the Northwest macroregion of the state of Paraná and is responsible for the care of individuals with complex traumatic injuries. Patients suffering injuries who are suspected or confirmed to have TBI are attended first by surgical clinic physicians who decide to do a CT and requests neurosurgeon evaluation when it is necessary. The HUM is equipped with, among other equipment, a computerized tomography (Lightning Aquilion 80-row multi-detector—Canon Medical System), which works with 80-slices, 24 hours a day seven days per week.
Data selection
Patients’ selection.
In compliance with data protection regulations, for safeguarding individuals’ personal data, the hospital provided the data after strictly followed after strictly following the principles about the principles regarding the treatment and processing of data for this study. Data anonymization techniques were employed to protect privacy and confidentiality.
The initial dataset was composed of data of 2,360 patients 14 years old or older with TBI who presented at the HUM emergency Department, between January 1st and December 31st of 2018. Of all these reports, 96.91% or 2,287 patients had Glasgow Coma Scale (GCS) scores between 13 and 15.
Previous literature cites people over 65 years old [29] and/or with GCS scores equal or less than 13 have an increased risk of being diagnosed with intracranial lesions [30]. As this information can induce the predictive model to provide a positive output, it is recommended to perform head CT for such individuals regardless of other clinical conditions [29, 30]. Furthermore, to avoid the possibility of duplicated data, when a patient presents at the HUM Emergency Department less than one week after their prior visit with the same mechanism of trauma, the second visit was excluded from the dataset, resulting in 1851 remaining observations.
Variables’ selection.
Initially, twenty-eight variables were available; we excluded time of attendance, variables with more than 25% missing data, as well as those that generated doubts in the interpretation, to avoid underestimation of the data. Other variables such as change in cognitive responses (ie, pupil dilatation, neurological deficit and loss of consciousness) since there was almost none with visual or neurological alteration during clinical analysis were also excluded.
To mitigate overfitting, lesions suggestive of a potential brain injury, including those confirmed by imaging tests like Skull x-ray and cranial tomography performance were excluded. Additionally, elements displaying a Pearson correlation of over 70% (38) were also removed from the data.
Hence, after evaluating the reliability of the variables and their association with the occurrence of intracranial lesions, only 10 variables, which were not directly correlated among them, were selected for machine learning prediction model development (Box 1).
Box 1. Variables selected for this study
*The following parameters were considered as high impact trauma: traffic crashes involving pedestrians, ejection of the driver and/or passenger from the vehicle in movement, vehicle speed above 60 km/h, physical aggression with a blunt object, and fall from at least one meter or five steps, according to Canadian protocol [18]*.
**For the evaluation of post-traumatic amnesia, loss of consciousness, and dizziness, the signs and symptoms reported by the patient were analyzed along with those described by third parties and/or observed during the medical examination.
Outcome
We evaluated the performance of six machine learning models to correctly identify individuals with head CT findings among patients with mild TBI attended by the HUM ED based on physiological characteristics as predictor variables.
Statistical test
The chi-square statistical test through the software RStudio version 4.1.0 [31] was used for comparisons between patient data with and without head CT finding, adopting a significance level of 5% (p≤0.05).
Development of prediction models and predictors
The TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) protocol [32] was adopted to standardize the development of supervised and classificatory machine learning algorithms. The package caret [33] was used to train and create the models.
The model development process was as follows: data selection by exclusion and inclusion criteria, imputation of missing data and cross-validation, testing of six computational algorithms to assess their effectiveness, and selection of the best model based on sensitivity. Based on the performances of the algorithms, a prototype of a risk equation to predict changes in head CT, called predict CT-Calculator, was built using the best models (Fig 1).
The data selected in this research was divided into a 70/30 (train/test) ratio and the missing values were imputed 10 times using the package MICE (multivariate imputation by chained equations) [34] for 3 tree-based algorithms (Random Forest, Gradient boosting machine and C5.0) and 3 non-tree-based algorithms (K-nearest neighbors, Linear extreme gradient boosting and Naive bayes generalized linear models), using caret package [33] as described below.
- Random forest (RF): a combination of decision trees that makes the analysis more complex, increasing the efficiency of prediction [35, 36];
- K-Nearest Neighbors (KNN): a method used to classify a given parameter based on the results obtained from its closest neighbors, that is, it uses likelihood to classify data [37, 38].
- Gradient boosting machine (GBM): a strong model that combines several weak models (e.g., decision tree), optimizing the predictions through a boost gradient [39].
- Linear extreme gradient boosting (XGB): a linear highly efficient gradient boosting model [40].
- C5.0: a model based on a decision tree or collection of rules. It preserves factors and other classes, prevents the automatic creation of false variables, and facilitates their implementation and understanding compared to other models, such as Supporting Vector Machine and neural networks [41, 42].
- Naive Bayes generalized linear models (NBglm): a classificatory Bayesian model based on the occurrence of a given event for data with a normal distribution [43, 44].
Since more than 90% of patients did not have a finding on imaging, the proportion of negative and positive imaging results were corrected using the package Random Over-Sampling Examples (ROSE) to lead with binary class imbalance [45]. The cross-validations were performed 100 times. The test and train datasets were divided in 10 parts/each (for each of 10 validations, nine parts were used to train and one for internal validation) and this process was repeated 10 times, according to cross-validation methodology described by Bergamini et al. [46], adapted by the authors.
We chose the performance metrics that would show the accuracy of the model in predicting a binary classifier of mild brain injury. We chose the standard general performance metrics of accuracy (ACCU), sensitivity (SENS), specificity (SPEC), and area under the receiver operating characteristic curve (AUROCC) [28, 47–51].
We also report the balanced accuracy (BALACCU) as a measure of a classification performance that takes into account both positive and negative. It is calculated as (SENS + SPEC) / 2. Balanced accuracy can be useful when the class distribution is imbalanced, as it considers both classes equally [28, 47–51]. Finally, we decided to report the probability of false negatives (PFN) because we wanted to know the probability of misclassification of a positive instance as negative [25, 28, 47–51], taking in account the variation of results for each imputation.
Based on sensitivity, the probability of the true positive results among all positive samples, we chose the best model and plotted its mean receiver operating characteristic curve for each imputation using the package pROC [52]. The algorithm with highest sensitivity was also used to develop the prototype of the risk equation [26], to help set the priority of CT scans. This algorithm was built into a website where health professionals can input patient data. The software then calculates the probability of the individuals to have head CT findings, as described in the discussion of this paper.
Results
The mean age of participants was 43±20 years and 1556 (67.60%) were male. In addition, 1609 (70.35%) patients were referred to the hospital by the prehospital care service, Mobile Emergency Care Service (SAMU) or the Integrated Emergency Trauma Care Service (SIATE), 173 (7.56%) were transferred from another hospital, and 504 (22.04%) self-presented to HUM ED.
The most common mechanism of trauma was road traffic injuries (42.59%), followed by a fall from standing height (29.56%), physical aggression (14.25%), and a fall from height (8.61%). Of these, 407 (17.80%) were classified as dangerous mechanisms. In the initial evaluation of the patient with mild TBI, there was a predominance of GCS equal to 15 (86.84%), followed by GCS equal to 14 (10.10%) and GCS equal to 13 (2.17%).
It was also observed that 19.14% (438/2287) of patient encounters occurred in July and December 2018, but 20% (15/74) of the positive head CT findings occurred in May. Furthermore, although head CT was performed in 70.09% (1603/2287) of the patients, only 4.62% of them (74/1603) showed positive imaging results.
When analyzing only the patients chosen for the prediction (1851), it was observed that there were statistical differences between patients with and without changes in the head CT in the following parameters: dangerous mechanism, GCS, amnesia, dizziness, headache, vomiting and/or nausea, and convulsion (Table 1).
The mean values and standard deviation of the computational models were 0.84±0.04 for accuracy, 0.51±0.13 for sensitivity, 0.84±0.04 for specificity, 0.68±0.05 for balanced accuracy, 0.78±0.03 for AUCROC (Fig 2).
Among the tested models, the Extreme Gradient Boosting was considered the best computational model because it showed higher sensitivity (Table 2).
The chosen computational model (linear extreme gradient boosting) also presents a similar ROC curve profile for each data imputation (Fig 3).
Imp01 = Result of first imputation; Imp02 = Result of second imputation; Imp03 = Result of third imputation; Imp04 = Result of fourth imputation; Imp05 = Result of fifth imputation; Imp06 = Result of sixth imputation; Imp07 = Result of seventh imputation; Imp08 = Result of eighth imputation; Imp09 = Result of ninth imputation; Imp10 = Result of tenth imputation; Agreg = Mean value of all imputation.
Discussion
Research focused on developing better performing prognostic methods has increased due to scientific advancements and the emerging use of machine learning (ML) algorithms. However, gaps in literature indicate that ML has not been used in screening patients with mild TBI specifically in developing countries. Hence, we aimed to fill that gap by examining the performance of six classification supervised ML algorithms.
All of the algorithms tested showed good accuracy (80 to 86%) and specificity (80 to 87%) but low sensitivity (43 to 70%). To determinate the best model, we adopted sensitivity as a reference, since it identifies the positive outcome among the samples (show the probability of true positive outcome among all the results predicted as positive) [25] and observed that XGB had the best performance agreeing with the results obtained to predict the clinical outcome of patients undergoing surgery for lumbar spinal stenosis in the Netherlands [53].
Even though the sensitivity is lower than the predictions the risk of recovery in patients with TBI in general [26] or optimizing the sensitivity with a very low specificity [54], the methodology developed in this study can help to set priority of CT scans in developing countries, but it cannot yet be used to make the final decision.
Finding the patients with higher probability of positive imaging results can aid in prioritizing attendance, reducing the patient exposure to ionizing radiation, and decreasing financial costs associated with Emergency Department examinations for traumatic brain injury [5, 29, 55]. However, some adjustments must be made to increase the sensitivity for better applicability in health centers.
In the United States of America, when the electroencephalogram was associated with the clinical symptoms of the patients, such as loss of consciousness, headache, nausea and/or vomiting, light and/or sound sensitivity, confusion and memory dysfunction, for the brain injury prediction in people affected by mild TBI, the researchers obtained an accuracy of 91% using the gradient boosting model [5] and sensitivity of 86% and specificity of 71% with the genetic algorithm model based on linear discriminant functions [56].
Since the main injury mechanism in developed countries is the fall of elderly people [21], while for low- and middle-income countries it is traffic accidents and violence involving young people [22], usually, it is not possible to use the same algorithms in the countries with different socioeconomic conditions, since it can generate divergent results.
The divergence in the performance from place to place and from mild TBI to general TBI, turn the comparison and use of the same algorithms in different environments, where the available variables are different, to be difficult [8]. For these reasons, it is still a challenge to find a tool that accurately detects all brain injuries in patients with mild TBI and safely discharges them from hospital, while avoiding unnecessary head CT. This obstacle is due to the fact that the chosen model must have a high sensitivity while keeping the specificity within satisfactory values [57], since sensitivity and specificity are inversely proportional [58].
This way, our study has some limitations. First, the use of the retrospective secondary data can lead to insufficient data and record failures. In addition, the evaluation of data from a single emergency department reflects a restricted scenario. So, these models must be tested and validated in other health centers as well. Considering that occurrence of mild TBI usually is not an isolated event, other variables like sociodemographic and clinical aspects can influence the outcome of the patients with mild TBI, so they should be analyzed to strengthen our study.
Another limitation observed in this study is the low amount of positive imaging results, since the effectiveness of a computational model can be reduced if there is a class imbalance, since most of the time, the majority group tends to overcome the minority, increasing the probability of false-negative results [59].
These limitations show us that although the computational models currently in practice allow researchers to work with easily obtained clinical data, it is important to conduct more robust studies and validate methodology in other health centers. Moreover, future refinements of the algorithm may incorporate other non-invasive measures of traumatic brain injury. For this reason, it is important to develop an integrated and systematic tool to objectively and quantitatively identify patients with mild TBI, with high precision for sensitivity and specificity.
Moreover, to transform the risk equation into a user-friendly tool that can help physicians to decide whether to perform a brain CT by inserting data in real time during the clinical evaluation of the individuals, a software prototype was developed to calculate the probability of patients having a brain injury (Fig 4).
Conclusion
Best model (XGB) correctly identified around 70% of all patients with mild TBI, indicating that after the improvement in its performance, this model has a high potential to be used in screening patients with mild TBI. The methodology developed in our study also uses the variables usually collected in Brazilian healthcare services, so it can be used to assist health professionals to manage the financial resources, and improve the quality and safety of patient care in Brazil and other countries with similar conditions, reducing the overcrowding of emergency healthcare services and time to diagnostic of whose show lesions which require neurological intervention in these geographical regions.
References
- 1. Oliveira GN, Campanharo CRV, Lopes MCBT, Barbosa DA, Okino MFP, Batista REA. Correlation between classification in risk categories and clinical aspects and outcomes. Rev Latin Am. Enfermagem. 2016; 24: 1–9. pmid:27982310
- 2. Morley C, Unwin M, Peterson GM, Stankovich J, Kinsman L. Emergency department crowding: a systematic review of causes, consequences and solutions. PLoS One. 2018; 13(8): e0203316. pmid:30161242
- 3. Vadan I, Ilut S.Estimating the global incidence of TBI. 2022. [Cited 2023 May 31]. Available from: https://brain-amn.org/global-incidence-of-tbi/#:~:text=The%20global%20incidence%20of%20all,suffer%20a%20severe%20TBI%20annually.
- 4. Dewan MC, Rattani A, Gupta S, Baticulon RE, Hung YC, Punchak M, et al. Estimating the global incidence of traumatic brain injury. J Neurosurg. 2018 Apr 1:1–18. pmid:29701556
- 5. Sharp AL, Nagaraj G, Rippberger EJ, Shen E, Swap CJ, Silver MA, et al. Computed tomography use for adults with head injury: describing likely avoidable Emergency Department imaging based on the Canadian CT head rules. Acad Emerg Med. 2017 Jan; 24(1): 22–30. pmid:27473552
- 6. Al Omran B, Patil JD, Anala A, Menezes P, Ahmed N, Cheffi I, et al. Prevalence of Computed Tomography Overuse for Mild Head Injury in Adults. Cureus. 2023 Feb 27;15(2): e35551. pmid:37007404
- 7. Shobeirian F, Ghomi Z, Soleimani R, Mirshahi R, Taheri MS. Overuse of brain CT scan for evaluating mild head trauma in adults. Emerg Radiol. 2021; 28(1): 251–257. pmid:32844320
- 8. Svensson S, Vedin T, Clausen L, Larsson PA, Edelhamre M. Application of NIC or SNC guidelines may reduce the need for computerized tomographies in patients with mild traumatic brain injury: a retrospective chart review and theoretical application of five guidelines. Scand J Trauma Resusc Emerg Med. 2019; 27(1): 1–10. pmid:31684991
- 9. Marincowitz C, Lecky FE, Townend W, Borakati A, Fabbri A, Sheldon TA. The risk of deterioration in GCS 13–15 patients with traumatic brain injury identified by computer tomography imaging: a systematic review and meta-analysis. J. Neurotrauma. 2018; 35(5): 703–718. pmid:29324173
- 10.
Brasil. Ministério da Saúde. Banco de dados do Sistema Único de Saúde—DATASUS. [Accessed 2022 Mar 09]. Available from: https://datasus.saude.gov.br/informacoes-de-saude-tabnet/.
- 11. Foks KA, Van der Brand CL, Lingsma HF, Van der Naalt J, Jacob B, Jong E, et al. External validation of computed tomography decision rules for minor head injury: prospective, multicentre cohort study in the Netherlands. BMJ. 2018; 362: 1–33. pmid:30143521
- 12. Molaei-Langroudi R, Alizadeh A, Kazemnejad-Leili E, Monsef-Kasmaie V, Moshirian SY. Evaluation of clinical criteria for performing brain CT-scan in patients with mild traumatic brain injury: A new diagnostic probe. Bull Emerg Trauma. 2019; 7(3): 269–277. pmid:31392227
- 13. Vedin T, Svensson S, Edelhamre M, Karlsson M, Bergenheim M, Larsson PA. Management of mild traumatic brain injury-trauma energy level and medical history as possible predictors for intracranial hemorrhage. Eur J Trauma Emerg Surg. 2019; 45(5): 901–907. pmid:29550926
- 14. Ghandour EK, Hould LL, Fortier FA, Gélinas V, Melnick ER, Hess EP, et al. Adapting two American decision aids for mild traumatic brain injury to the Canadian context using the nominal group technique. Patient. 2020; 13(6): 729–743. pmid:33078377
- 15. Haydel MJ, Preston CA, Mills TJ, Luber S, Blaudeau E, DeBliex PM. Indications for computed tomography in patients with minor head injury. N. Eng. J. Med. 2000 Jul 13; 343(2):100–105. pmid:10891517
- 16. Stiell IG, Wells GA, Vandemheen K, Clement C, Lesiuk H, Laupacis A., et al. The Canadian CT Head Rule for patients with minor head injury. Lancet. 2001 May 5; 357(9266): 1391–1396. pmid:11356436
- 17. DeAngelis J, Lou V, Li T, Tran H, Bremjit P, McCann M, et al. Head CT for minor head injury presenting to the emergency department in the era of choosing wisely. West J Emerg Med. 2017 Aug; 18(5): 821–829. pmid:28874933
- 18. Jagoda AS, Bazarian JJ, Bruns JJ Jr., Cantrill SV, Gean AD, Howard PK, et al. Clinical policy: neuroimaging and decision making in adult mild traumatic brain injury in the acute setting. Ann Emerg Med. 2008 Dec; 52(6): 714–748. pmid:19027497
- 19. Smits M, Dippel DW, Steyerberg EW, de Haan GG, Dekker HM, Vos PE, et al. Predicting intracranial traumatic findings on computed tomography in patients with minor brain injury: the CHIP prediction rule: Ann Intern Med. 2007 Mar 20; 146(6): 397–405. pmid:17371884
- 20. Vaniyapong T, Patumanond J, Ratanalert S, Limpastan K. Clinical indicators for traumatic intracranial findings in mild traumatic brain injury patients. Surg Neurol Int. 2019 Apr 24; 10–64. pmid:31528402
- 21. Quaglio G, Gallucci M, Brand H, Dawood A, Cobello F. Traumatic brain injury: a priority for public health policy. Lancet Neurol. 2017 Dec; 16(12): 951–952. pmid:29122519
- 22. Saatian M, Ahmadpoor J, Mohammadi Y, Mazloumi E. Epidemiology and patterns of traumatic brain injury in a developing country regional trauma center. Bull Emerg Trauma. 2018 Jan; 6(1): 45–53. pmid:29379809
- 23. Barbosa RR, Jawa R, Watters JM, Knight JC, Kerwin AJ, Winston ES, et al. Evaluation and management of mild traumatic brain injury: an Eastern Association for the Surgery of Trauma practice management guideline. J Trauma Acute Care Surg. 2012 Nov; 73 (Suppl 4): S307–S314. pmid:23114486
- 24. Fournier N, Gariepy C, Prévost JF, Belhumeur V, Fortier É, Carmichael PH, et al. Adapting the Canadian CT head rule age criteria for mild traumatic brain injury. Emerg Med J. 2019 Oct; 36(10): 617–619. pmid:31326953
- 25. AIKaabi LA, Ahmed LS, Attiyah MFA, Abdel-Rahman ME. Predicting hypertension using machine learning: findings from Qatar biobank study. PLoS One. 2020; 15(10). pmid:33064740
- 26. Rocha TAH, Elahi C, Silva NC, Sakita FM, Fuller A, Mmbaga BT, et al. A traumatic brain injury prognostic model to support in-hospital triage in a low-income country: a machine learning-based approach. J Neurosurg. 2019 May 10; 132(6): 1961–1958. pmid:31075779
- 27. Verma AA, Murray J, Greiner R, Cohen JP, Shojania KG, Ghassemi M, et al. Implementing machine learning in medicine. CMAJ. 2021 Aug 30; 193(34): E1351–e1357. pmid:35213323
- 28. Oliveira AR, Roesler V, Iochpe C, Schmidt MI, Vigo A, Barreto SM, et al. Comparison of machine learning algorithms to build a predictive model for detecting undiagnosed diabetes–ELSA–Brasil: accuracy study. Sao Paulo Med J. 2017 May-June; 135(3): 234–246. pmid:28746659
- 29. Freire-Aragón MD, Rodríguez-Rodríguez A, Egea-Guerrero JJ. Update in mild traumatic brain injury. Med Clin (Barc). 2017 Aug 10; 149(3): 122–127. pmid:28571951
- 30. Mena JH, Sanchez AI, Rubiano AM, Peitzman AB, Sperry JL, Gutierrez MI, et al. Effect of the modified Glasgow Coma Scale score criteria for mild traumatic brain injury on mortality prediction: comparing classic and modified Glasgow Coma Scale score model scores of 13. J Trauma. 2011 Nov;71(5):1185–92. pmid:22071923
- 31. Alberti G. Chi-Square and G-Square test of independence, residual analysis, and measures of categorical association. 2023 Apr 21 [Accessed 2023 May 31]. Available from: https://cran.r-project.org/web/packages/chisquare/index.html.
- 32. Collins GS, Reitsma JB, Altman DG, Moon KG. Transparent reporting of a multivariate prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. Br J Surg. 2015 Feb; 102(3): 148–158. pmid:25627261
- 33. Kuhn M, Wing J, Weston S, Willian A, Keefer C, Engelhardt A, et al. Classification and regression training. 2022 Apr 19 [Accessed 2022 Apr 19]. Available from: cran.r-project.org/web/packages/caret/index.html.
- 34. Buuren SV, Karin GO, Vink G, Schouten R, Robitzsch A, Rockenschaub P, et al. Multivariate Imputation by Chained Equations. 2021 Nov 24 [Accessed 2021 Nov 30]. Available from: https://cran.r-project.org/web/packages/mice/mice.pdf.
- 35. Liaw A, Wiener M. Classification and regression by Random Forest. R News. 2002; 2(3): 18–22. [Accessed 2021 Nov 30]. Available from: https://cogns.northwestern.edu/cbmg/LiawAndWiener2002.pdf
- 36. Denisko D, Hoffman MM. Classification and interaction in Random Forests. Proc Natl Acad Sci USA. 2018 Feb 20; 115(8): 1690–1693. pmid:29440440
- 37. Jawaharlal V. KNN using caret R package. 2014 Apr 29 [Accessed 2021 Aug 16]. Available from: https://rpubs.com/njvijay/16444.
- 38. Guo G, Wang H, Bell D, Bi Y, Greer K. KNN model-based approach in classification. Proc. ODBASE. 2003: 986–996.
- 39. Friedman JH. Greedy function approximation: a Gradient Boosting Machine. Ann. Statist. 2001; 29(5).
- 40. Chen T, He T, Benesty M, Khotilovich V, Tang Y, Cho H, et al. Extreme Gradient Boosting. 2022 Apr 16 [Accessed 2022 Apr 16]. Available from: https://cran.r-project.org/web/packages/xgboost/index.html.
- 41. Kuhn M, Weston S, Culp M, Coulter N, Quinlan R. C5.0 Decision Trees and rule-based models. 2022 Feb 5 [Accessed 2022 Feb 22]. Available from: https://cran.r-project.org/web/packages/C50/C50.pdf.
- 42. Yobero C. Determining creditworthiness for loan applications using C5.0 Decision Trees. 2019 Sept. 7 [Accessed 2021 Jul 15]. Available from: https://rpubs.com/cyobero/C50.
- 43. Gelman A, Jakulin A, Pittau MG, Su YS. A weakly informative default prior distribution for logistic and other regression models. Ann Appl Stat. 2008; 2(4): 1360–1383.
- 44.
Lindsey JK. Applying generalized linear models. New York (NY): Springer, 1997. 256 p.
- 45. Lunardon N, Menardi G, Torelli N. ROSE: a package for binary imbalanced learning. The R Journal. 2014; 6: 79–89.
- 46. Bergamini M, Iora PH, Rocha TAH, Tchuisseu YP, Dutra AC, Scheidt JFH, et al. Mapping risk of ischemic heart disease using machine learning in a Brazilian state. PLoS One. 2020; 15(12): 1–15.
- 47. Kawamura T. Interpretação de um teste sob a visão epidemiológica: eficácia de um teste. Arq Bras Cardiol. 2002; 79(4): 437–441. pmid:12426656
- 48. Trullols E, Ruisánchez I, Rius FX. Validation of qualitative analytical methods. Trend Anal Chem. 2004; 23(2): 137–145.
- 49. Ferreira JC, Patino CM. Entendendo os testes diagnósticos. Parte 3. J Bras Pneumol. 2018; 44(1): 4. pmid:29538535
- 50. Parikh R, Mathai A, Parikh S, Chandra Sekhar G, Thomas R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol. 2008;56(1):45–50. pmid:18158403
- 51. Coughlin LN, Tegge AN, Sheffer CE, Bickel WK. A Machine-Learning Approach to Predicting Smoking Cessation Treatment Outcomes. Nicotine Tob Res. 2020 Mar 16;22(3):415–422. pmid:30508122
- 52. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. Display and analyze ROC curves. 2021 Sep 3 [Accessed 2021 Aug 24]. Available from: https://cran.r-project.org/web/packages/pROC/pROC.pdf.
- 53. Siccoli A, Wispelaere MP, Schroder ML, Staartjes VE. Machine learning-based preoperative predictive analytics for lumbar spinal stenosis. Neurosurg Focus. 2019 May 1, 46(5). pmid:31042660
- 54. Marincowitz C, Paton L, Lecky F, Tiffin P. Predicting need for hospital admission in patients with traumatic brain injury or skull fractures identified on CT imaging: a machine learning approach. Emerg Med J. 2022 May; 39(5): 394–401. pmid:33832924
- 55. Sharp AL, Huang BZ, Tang T, Shen E, Melnick ER, Venkatesh AK, et al. Implementation of the Canadian CT head rule and its association with use of computed tomography among patients with head injury. Ann Emerg Med. 2018 Jan; 71(1): 54–63. pmid:28739290
- 56. Bazarian JJ, Elbin RJ, Casa DJ, Hotz GA, Neville C, Lopez RM, et al. Validation of a machine learning brain electrical activity-based index to aid in diagnosing concussion among athletes. JAM Netw Open. 2021 Feb 1; 4(2). pmid:33587137
- 57. Abujaber A, Fadlalla A, Gammoh D, Abdelrahman H, Mollazehi M, El-Menyar A. Prediction of in-hospital mortality in patients with post traumatic brain injury using National Trauma Registry and machine learning approach. Scand J Trauma Resusc Emerg Med. 2020, 28(1):44. pmid:32460867
- 58. Polo TCF, Miot HA. Aplicações da curva ROC em estudos clínicos e experimentais. J Vasc Bras. 2020, 19: 23–26. pmid:34211533
- 59. Aldraimli M, Soria D, Parkinson J, Thomas EL, Bell JD, Dwek MV, et al. Machine learning prediction of susceptibility to visceral fat associated diseases. Health Technol. 2020, 10: 925–944.