Introduction

Headache is a common nervous system disorder, affecting approximately 50% of the general population [1, 2]. It can be classified as a primary headache disorder, typically migraine, tension, or cluster headaches [1], or a secondary headache disorder, which includes: giant cell arteritis, meningitis/encephalitis, subarachnoid hemorrhage (SAH), cerebral venous thrombosis, idiopathic intracranial hypertension, brain tumor, and ischemic stroke [1, 3, 4]. Although secondary headaches are rare compared with primary headaches, they are extremely important to recognize as they can require immediate intervention [1, 3], with further and rapid diagnostic evaluation including neuroimaging and lumbar puncture often needed [3, 5].

In primary care, headache is one of the most common presenting symptoms, with most due to primary causes, although many headache sufferers do not receive a specific diagnosis [6]. Headaches also account for up to 4.5% of all visits to the emergency department (ED) [2, 3, 5, 7, 8], with one observational study revealing an equal prevalence of primary versus secondary headache etiologies presenting to the ED (48 vs. 52%, respectively) [2]. Thus, accurate diagnosis of the underlying cause of headache and treatment initiation can be critical.

Headache has a historic reputation as being one of the most poorly classified neurologic disorders, and the International Headache Society classification system was developed to provide a hierarchy of criteria for diagnosis [9]. In the United Kingdom (UK) primary care setting, recognition of the importance of effective headache diagnosis, and management has prompted the additional training of general practitioners (GPs) with special interest in headache, and the establishment of headache clinics in general practice [10, 11]. Despite this, a varied degree of confidence in headache diagnosis and treatment still exists amongst emergency and primary clinicians, due in part to a lack of specific experience in neurology and poor use of the International Headache Society classification system [2, 6, 10]. A study conducted in Canada reported that misdiagnosis or diagnostic uncertainty occurred in more than one-third (35.7%) of cases of neurologic complaints in the ED, when comparing the initial diagnosis to the diagnosis made by the consulting neurologist [12]. In the UK, where there is a concerning shortfall of neurologists [10], the fear of clinical neurology by doctors outside the discipline has been described to amount to ‘neurophobia’ [10]. This can, together with patient anxiety and legal concerns, result in multiple appointments and unnecessary investigations, which may subsequently increase diagnosis times, healthcare costs, and economic burden [1, 4, 13].

The effective triaging of patients presenting with primary versus secondary headache is an important and currently unmet need [1, 5, 14]. Consideration of patients’ medical history and physical examination are currently the most important aspects of headache assessment, and clinicians must be vigilant for “red flag” symptoms that are characteristic of serious secondary causes [1, 3]. This qualitative approach could be complimented by a quantitative methodologic tool, which could potentially streamline diagnosis, facilitate clinical decision-making, and reduce unnecessary investigations [2, 5, 14].

A complete blood count (CBC) is one of the most commonly requested blood tests [15] and its results may provide important diagnostic and prognostic biomarkers indicating underlying causes of headache. Several CBC parameters have been investigated for their ability to distinguish between primary and secondary headaches. A retrospective study of patients presenting to the ED with headache reported that leukocytosis or an increase in the percentage of polymorphonuclear leukocytes (PMNs) had a sensitivity of 89.8%, a specificity of 46.7%, a positive predictive value of 82.1%, and a negative predictive value of 62.8% for diagnosing SAH within 6 or 12 hours of ED admission [16]. The ratio of neutrophils to lymphocytes (NLR) has also received increasing attention as a diagnostic and prognostic marker of inflammation and can be easily calculated from standard measurements of CBC tests. A retrospective study found NLR to be higher in people presenting to the ED with a migraine attack versus people without a headache [17]. A further retrospective, single-center analysis in ED patients presenting with headache accompanied by nausea and vomiting, found that NLR could distinguish between those with migraine and those with SAH [16]. Although this retrospective study involved a limited number of clinical cases, median NLR values were found to be significantly higher in patients with SAH compared with those with migraine and other headaches (both p < 0.001) [18].

Measurements derived from CBC tests have also been evaluated in the prediction of secondary headache severity. In one retrospective, single-center review, patients with SAH whose leukocyte count was >15 x 10⁹/L during admission were more than three times more likely to develop vasospasm [19]. In another study of patients with SAH, the authors reported that patients admitted with spontaneous SAH with a leukocyte count of >20,000 had a mortality rate of 50% [20]. Furthermore, mean platelet volume and platelet distribution width has been shown to be increased in patients with cerebral venous thrombosis and brain parenchymal lesions compared with patients with cerebral venous thrombosis without lesions [21].

Aligning with the concept of using routine blood test results as a triaging tool to assist physicians when deciding whether to perform neuroimaging on patients presenting with severe headache, this study was designed to evaluate a machine learning (ML)-based approach to classify primary versus secondary headache using real-world data (RWD) derived from a large patient group.

Materials and methods

Results

Cohort description

A total of 121,241 patients satisfied the inclusion/exclusion criteria and formed the analytical cohort, comprising 108,906 patients with primary headache and 12,335 patients with secondary headache (Fig 1). Baseline demographic and clinical characteristics of the analytical cohort, stratified into primary and secondary headache groups, are presented in Table 1. Overall, patients presenting with primary headache were significantly younger than those presenting with secondary headache (mean 44 vs. 70 years; p < 0.001). The majority of primary headache patients were between 21 and 50 years of age (n = 64,004; 58.7%), whereas the majority of secondary headache patients were between 61 and 80 years of age (n = 6,454; 52.3%). In both headache groups, there were considerably more female than male patients (78.1% vs. 21.9% and 65.7% vs. 34.3%, for primary and secondary headache groups, respectively).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Patient flow and determination of the analytical cohort.

*Patient data flagged as “research acceptable” by CPRD. Two consultation types ("follow-up/routine visit" and "mail from patient") occurring on the index date were removed from the data set. CPRD, Clinical Practice Research Datalink.

https://doi.org/10.1371/journal.pone.0282237.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Baseline demographic and clinical characteristics of the analytical cohort (n = 121,241).

https://doi.org/10.1371/journal.pone.0282237.t001

Descriptive statistical analyses

Results from descriptive statistical analyses revealed statistically significant differences (p < 0.001) between the population means for almost all of the ten parameters from the CBC tests and the 19 ratio variables measured in patients between headache groups (Tables 2 and 3). However, there were substantial overlaps in the range of values for these variables with broadly similar mean values (S1 and S2 Figs). Further analysis using the Hotelling T²-test showed that the two headache groups could not be properly differentiated (Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive statistics for variables from the CBC test for the analytical cohort (n = 121,241).

https://doi.org/10.1371/journal.pone.0282237.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Descriptive statistics for blood cell ratios for the analytical cohort (n = 67,974).

https://doi.org/10.1371/journal.pone.0282237.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Hypothesis testing results using the Hotelling T²-test (n = 67,974).

https://doi.org/10.1371/journal.pone.0282237.t004

Predicting primary headaches versus secondary headaches

Table 5 shows five-fold cross-validated performance metrics of the prediction models, with and without the ratio variables of blood cell count parameters before varying the probability threshold, using logistic regression and random forest methods separately. The performance metrics suggested that the random forest method without the blood cell count ratio variables had an overall better predictive performance for predicting primary headaches versus secondary headaches. Spearman’s correlation matrix and feature weight analysis suggested that age group, sex, total WBC count, neutrophil, and monocyte count correlated strongly with headache type (S3 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Performance metrics of the prediction models using logistic regression and random forest methods with five-fold cross-validation.

https://doi.org/10.1371/journal.pone.0282237.t005

After changing the probability threshold, the final prediction model showed accurate and robust performance with a balanced accuracy at 0.7405 (Table 6), reflecting an ability to achieve a false negative rate (incorrect prediction of secondary headache as primary headache) of 10% while maintaining a false positive rate (incorrect prediction of primary headache as secondary headache) of 42% (Fig 2). The sensitivity (correct prediction of secondary headache as secondary headache) and specificity (correct prediction of primary headache as primary headache) of the final model were 58% and 90%, respectively (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Confusion matrix (normalized to ratio instead of patient count) of the final prediction model on the test set using the random forest method.

Features included in the model are: age group, sex, and laboratory blood test results of ten CBC parameters. CBC, complete blood count.

https://doi.org/10.1371/journal.pone.0282237.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Performance metrics of the final prediction model on the test set using the random forest method.

https://doi.org/10.1371/journal.pone.0282237.t006

Discussion

In summary, descriptive statistical analysis revealed substantial overlap in values for the ten CBC parameters and 19 ratio variables measured in patients in both headache groups, and patients with primary and secondary headaches could not be properly differentiated based on results from the Hotelling T²-test. However, we have developed a ML-based prediction model with a modest predictive accuracy to differentiate between primary and secondary headaches on the basis of readily available patient characteristics and routine blood test results.

Diagnostic procedures and acute treatment for headaches may vary across different countries, depending on factors such as catchment area, structure of the care facility, in-house protocols, and local medical staff [3]. In current clinical practice, an excess of patients presenting to the ED with a severe headache are referred for neuroimaging, despite current guidelines recommending against routine neuroimaging for headaches [33]. In one European study, neurologic examination was performed in 72.5% of patients presenting to the ED with headache; 60.9% subsequently underwent technical investigation and 53.2% had non-contrast cranial computed tomography [3]. However, unnecessary investigations should be avoided [1]; it is not appropriate, for example, to routinely use computed tomography in headache patients, because of the high cost and radiation exposure [16].

Our study suggests that a CBC-based ML algorithm could mitigate this problem, by simplifying the triage of patients who require such diagnostic procedures. Our findings indicate that age group, sex, and ten parameters that are usually collected during CBC tests represent convenient, measurable variables for use in a ML prediction model, to differentiate patients with primary and secondary headache. We also developed a prediction model with modest performance, predicting almost 60% of secondary headache patients and 90% of primary headache patients. Future studies may look at including certain clinical characteristics (such as a history of brain trauma, hypertension, epilepsy, or stroke) in the prediction model to assess whether their addition could improve its performance further.

In a similar study, the sensitivity of leukocytosis or increase in the percentage of PMNs in cases of patients with SAH was investigated, with a view to developing a non-invasive blood test to facilitate diagnosis [16]. Investigators concluded that CBC had an excellent sensitivity (89.8%) in the exclusion of SAH in non-traumatic headache. Specificity, however, was poor (46.7%), as leukocytosis can result from other headache etiologies such as migraine, temporal arteritis (giant cell arteritis), and hypertension, suggesting CBC levels could only be used to rule out, rather than confirm, SAH [16].

Our ML model has potentially important clinical implications. The reasonably low error rate (10%) of misclassifying secondary headache as primary headache could aid clinicians’ decision-making on which patients to refer for further examination (e.g. neuroimaging and/or lumbar puncture), thereby avoiding unnecessary procedures and reducing the drain on healthcare resources. It is envisaged that our model could be used alongside taking a detailed headache history and, if indicated, a thorough neurologic examination. In cases of abnormal findings, neuroimaging should be performed to rule-out secondary headaches. Currently, if there is a clinical suspicion of SAH, computed tomography is undertaken, followed by lumbar puncture if the scan is inconclusive [34]. As our model is able to distinguish between primary and secondary headaches with a sensitivity of 58%, a specificity of 90%, a false negative rate of 10% and false positive rate of 42%, it is hoped that it may help reduce the number of unnecessary procedures. Furthermore, the model will be of particular use in countries where neuroimaging is not readily available, but should be used with caution, particularly when headache history is sparse.

As a point to consider, because primary headache accounted for approximately 90% of all headache cases in this study, a default prediction model with good prediction accuracy may have been informed by an imbalanced data set (i.e. a model that predicts primary headache for all patients will be correct in 90% of cases). However, the model developed herein accounted for such imbalanced data and focused on the accurate prediction of secondary headaches, resulting in a modest predictive model with a balanced accuracy of 0.7405. Furthermore, as healthcare progresses, in the future our model could be expanded to include other relevant parameters to further improve the model performance.

When comparing random forest and logistic regression methods, the former marginally outperformed the latter in nearly all prediction performance metrics, particularly those suitable for imbalanced data sets (balanced accuracy, average precision, and F1-score). This is likely due to the fact that the random forest model is fundamentally a large number of uncorrelated individual decision trees operating as an ensemble/committee and is hence better suited to capture interactions between variables. The logistic regression model on the other hand, has a linear function and cannot capture such interactions.

Correlated features have an influence on the prediction performance of the model, which was suggested by performance metrics to be slightly compromised by the features of blood cell ratios. Although this is counterintuitive, according to the premise that “more data is better”, data are required to be independent and identically distributed, and it follows that such correlations are detrimental for the performance of ML techniques. The features of blood cell ratios are highly inter-dependent, being strongly correlated both to each other and to the original features of blood cell counts, thus violating this data premise.

We attempted to simplify the model, whilst maintaining good performance, by using feature selection techniques, including lasso regularization and recursive feature elimination. As correlation and feature weight analyses indicated that age group, total WBC count, monocyte count, and neutrophil count were of the greatest significance for the model, only these features were included. However, none of the ML feature selection techniques yielded a better result compared with the final model that included age group, sex, and laboratory results of ten parameters usually collected during a CBC test. This is probably due to the final prediction model demonstrating only a marginal improvement when compared with the ML selection techniques focusing on a certain metric, e.g. accuracy. In addition, many simpler models with fewer features resort to the baseline “guess” model of predicting every patient as primary headache regardless of the data.

The key strength of this study is the large sample size of >120,000 patients and the use of RWD from the UK. Awareness in the UK of the difficulties around headache diagnosis and treatment has prompted the training of GPs with special interest in headache [10] and the establishment of a network of headache centers [11]; our findings could potentially inform these endeavors.

The use of primary care data rather than data directly from ED settings is a limitation of the study, due to the extrapolation performed, which makes it difficult to assess the real-life advantages of the ML approach to differentiating headache types in the ED environment. Further studies using ED data are warranted to validate the algorithm described here for this differentiation. This study considered primary and secondary headaches as two groups of heterogeneous conditions; future work could evaluate diagnostic accuracy of measurements from CBC tests in different forms of primary and secondary headache. As this was a retrospective study of anonymized data some patients included in the study may, for example, have comorbidities, such as an underlying infection or inflammatory condition, or may be using medications that alter certain CBC markers. Further research should be carried out in specific patient populations, such as immunocompromised individuals, to elucidate the potential prognostic value of CBC and CBC-derived ratio parameters in differentiating primary and secondary headaches.

In conclusion, this study demonstrated the use of a ML approach to create a prediction model with a modest level of performance to differentiate patients with primary and secondary headache in clinical settings.

Supporting information

Acknowledgments

The authors gratefully thank Marie Stobbe (Roche Diagnostics) for project coordination and support, Iori Namekawa (F. Hoffmann-La Roche Ltd.) for data management, Martine Kallemeijn (Roche Diagnostics) for critical review of the manuscript, and Simon John Davidson (Freelance Hemostasis Consultant, London, UK) for his input into study design and conception. Third-party medical writing assistance, under the direction of the authors, was provided by Anna King, PhD and Elizabeth Hilsley, BSc, of Ashfield MedComms (Macclesfield, UK), an Inizio company.