Data driven models

Machine Learning (ML) in digital health is a growing area. The rise of data-rich platforms is enabling researchers to follow a data-driven approach to modelling diseases, rather than an epidemiological approach (for a review, see [5]). This has improved all areas of health, from prevention through to post-treatment care. For example, ML is able to provide disease diagnosis at significantly earlier stages than typical methods. It was found that ensemble predictors such as Random Forests were able to predict diabetes to 93% accuracy [6]. Another study applied a gradient boosting machine to predict the risk of sepsis in patients, significantly improving on the state-of-the-art (from 49.2% to 67.7%) [7]. Furthermore researchers have used genetic algorithms to predict heart disease in its early stages [8] and lastly, a combination of principal component analysis and neural networks have been used to diagnose diabetic retinopath in patients [9].

ML has also aided the analysis of patient data improving treatment. For example, support vector machines and Random Forests are able to localise brain tumours in magnetic resonance images [10]. Finally, machine learning is aiding post-care treatment with algorithms now able to predict which patients are at a high risk of readmission, a significant cost to healthcare systems [11], and algorithms that are able to monitor public opinion on how healthcare systems manage certain crises. For example, [12] used social media to scrape public opinions of how people reacted to COVID-19. It is clear from the breadth of examples that machine learning can advance healthcare in a multitude of ways.

Malaria specific data-driven models are rare given the lack of structured data-sets. [13] carried out a study looking at a variety of different machine-learning techniques to predict Malaria cases in healthcare centres in the district of Visakhapatnam, India. Using 6 years worth of data they found that Gradient Boosting was a good predictor of cases. However, this was limited to just one small region of India, with no inter-healthcare variance.

Despite the opportunity that machine-learning presents, risks exist. Biases in poorly selected training and tests samples (for example through insufficient representation of a given class in the sample [9]), can result in biased decision making once the algorithm has been implemented [14]. It is clear that going forward, if machine-learning is to be used to improve the delivery of healthcare, it must be done so with caution and transparency [15].

In this paper we attempt to directly model the case rate of malaria in Burkina Faso over the last three years. In this way we can provide in-situ predictions of future case rates immediately, without the need to run complicated numerical simulations of malaria transmission in the country. Moreover, by directly modelling the case rate we encompass not only the transmission but other factors, including social, to provide extremely localised predictions.

IeDA

The Integrated e-Diagnostic Approach (IeDA, https://www.tdh.ch/en/ieda) is an implementation of the Integrated Management of Childhood Illness, IMCI [16], on Android-based tablets. The application allows primary healthcare workers to conduct efficient and informed consultations for children younger than 5 years. The primary aim of the IMCI protocol is to combat child mortality, which in the case of Burkina Faso fluctuates around 8% [17]. The IeDA program started in 2010 in 39 primary healthcare facilities (PHC) in Tougan, Burkina Faso. Since then (at time of writing), it has risen to cover more than 67% of the entire country, taking around 200, 000 consultations each month. This has already dramatically improved the delivery of healthcare to Burkina Faso, particularly in rural areas. As a result of the IeDA project there has been a reduction in antibiotic usage by ∼ 6 − 15% [18] and the potential savings of millions of Swiss Francs.

For those PHCs in the program, the tablet is used in 92% of children’s consultations, such that we have a good coverage of this population. The database is fully pseudonymised to retain the privacy of the patients and contains over 9 million consultations. Throughout this study we use the confirmed diagnosis of malaria as the target feature. This may have its own biases, which we discuss later. The objective of this paper is two fold:

1. Predict the trajectory of the case numbers of malaria for a specific primary healthcare facility;
2. Train and validate an early warning system for the onset of an epidemic up to three months in advance.

Epidemic detection

The definition of an epidemic is defined as the point at which the instantaneous case rate rises above the five-year mean for that period of year plus two standard deviations. The disadvantage of this is that it is an instantaneous definition, such that a region does not know it is in an epidemic until it arrives. Early warning of the onset of an epidemic will allow local and federal governments to prepare and react before the onset of an emergency. It is the aim of this work to develop an early warning system for malarial epidemics in Burkina Faso up to three months in advance.

Patient and public involvement

We utilise the IeDA database that consists of consultations of infants less than an age of five years. All data is fully anonymised. Patients agree to be part of the database through a verbal agreement. Data has been provided by the Burkina Faso government under strict licensing uniquely to Terres des hommes for the purpose of improving healthcare delivery throughout Burkina Faso.

Time series

The base framework of the algorithm is to fit a library of models to some observed data that will then predict the subsequent time period required. For example, we will use models which span 26 weeks. This will allow us to fit to 13 weeks of observed data and then for the model to predict 13 weeks ahead in time.

This library of models will be agnostic in time and geographical location, that is, to make a single prediction we will compare a single observed time series to the entire library of time series that we have and find the best fitting ones.

We therefore first manipulate the historical data vectors into series of time windows of 26 weeks. With ∼ 1000 PHCs and ∼ 2.5 years of data we have in total ∼ 100, 000 separate time series that we can use to formulate a library of models.

Learning each time series: Gaussian Processes

A naive method would be to use the raw data as our library of models, comparing historical pasts directly with the current data. However, the historical past suffers from large variance due to not just random noise but also noise generated from social factors such as strikes and routine monthly cycles of medicine distribution. If we are to generate a library of reliable models that look beyond weekly variations and extract the seasonal cycles of malaria and how these correlate with other observables then we must use a more sophisticated model. To do this we choose a Gaussian Process.

A Gaussian Process (GP from now on), is defined as a collection of finite random variables that are Gaussian distributed. Each variable can be fully described by their mean, m and their co-variance, k, where we assume a exponential, sine squared kernel, (1) where d is the distance between observed values, x_i and x_j, p is the periodicity and l is the length scale, both free parameters to be fit. Indeed we test these free parameters and find that the impact on these data-sets is small. We choose a periodicity of p = 52, corresponding to the annual cycle of 52 weeks, and the length scale, l = 1.

Following these choices we then fit the GP to our data. During the fitting we define how well we expect the GP to do at describing the data, i.e. to avoid over-fitting to each time series we give it some buffer. This acts to choose functions that are smoother since they are statistically a better fit. Not only this, however but co-variances between observables, builds a model that jointly fits each observable equally. As such the advantages of a GP is that it takes in to account correlations not only in time but between observables.

We use the freely available Gaussian Process Regressor (GPR) package from scikit-learn [19] to model our time series. We jointly fit each GPR to each chosen time-dependent feature, assuming some level of noise. This noise must be tuned empirically to strike a balance between fitting to signal in the data and over-fitting to noise. The top panel of Fig 1 shows the impact of a varying noise parameter, α, on the modelled time series of absolute malarial cases. We see that in the low α limit the GPR finds models that fit the data exactly, greatly over-fitting, whereas a very high noise value results in the GPR fitting only the mean of the time series.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1.

Top: How the fit of the Gaussian Processor Regressor to the time series data depends on the noise value α. Assuming a sine, exponential squared kernel, we see that for small α values the regressor over-fits the malaria (top), fitting every point in each curve. Additionally, at high values of alpha, the regressor tends to the mean of each time series. Bottom: We optimise the noise value through a simple grid-based search for the best fitting value of α. For each time-series we derive a log-likelihood of the model given the data. We then find the alpha parameter that has the best mean log-likelihood over the entire data-set. We find that the log(alpha) = −2.75 returns the highest mean likelihood.

https://doi.org/10.1371/journal.pone.0253302.g001

To tune α we calculate the log marginal likelihood of each model given the data-set and find the noise value that returns the maximum. The log marginal likelihood of a Gaussian Process takes in to account how well the data fits the model, and how complex the model is. From Bayes Theorem, the marginal likelihood, or evidence, of the target y given the input data X, is the product of likelihood of the target values given the function f, p(y|f, X), and the prior over f, p(f|X), integrated over the parameters of the function f, i.e. (2) For a Gaussian Process, the prior is a Gaussian with a mean of zero and co-variance, k, and the likelihood of the function given the data is also a Gaussian with a mean of the input function f and the input variance, α. [19] show by carrying out the integral in Eq (2), the log marginal likelihood is, (3) where I is an identity matrix and n is the number of observations. We see that the first term of the marginal likelihood is how well the data is fit by the Gaussian Process, the second term penalises complexity in the algorithm. To find the best fitting α parameter we carry out a grid based search, testing different levels of noise and finding the mean marginal log-likelihood. The bottom panel of Fig 1 shows the result of this search. We find that a value of log(α) = −2.75 best fits the data.

Model selection

Now with a library of models based on historical data, we want to be able predict the case rate of malaria for a given PHC. To do this, we take the observed data from the previous 13 weeks and compare this to the first half of every model in our library. We construct a joint likelihood of each model and its component features and find the best fitting one. From this we can take the second half of these models to predict the following 13 weeks.

We construct the total log-likelihood for a given observed data-set, , as the linear addition of each observable’s log-likelihood, , i.e. (4) where we assume either a Gaussian log-likelihood, (5) where D and M are the data and model respectively and σ is the total error or Poisson, (6) where this is the log-likelihood of observing D events for a Poisson distribution with a mean of M events. Fig 2 shows an example of the likelihood function. Each panel shows the distribution of the likelihood of a model given a single observed data set for the absolute case numbers of malaria (top) and the rain fall (middle). The bottom panel shows the total likelihood given the two observables. We see that the final distribution of likelihoods has a small number of models that fit well and the rest that do not.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. An example of how we select our models.

We compare the observed data to each GP and generate a likelihood according to the type of data (whether it be Poisson or Gaussian). In this case the top panel showing the absolute number of cases of malaria is Poisson distributed, whereas the rain-fall is Gaussian. We then combined these likelihoods in to a final likelihood and select only those models that have an overall good fit.

https://doi.org/10.1371/journal.pone.0253302.g002

We then select all models that lie within some threshold, λ, relative to the best fitting model’s log-likelihood. We define this as the Δχ² for the given number of degrees of freedom, where the Δχ² is given by, (7) Despite the assumption of Gaussian statistics, PHCs have extremely irregular time series that result in models that are equally irregular. This nuance with the data lead us to two decisions: (1) we need to tune the value of Δχ² which selects models that fit a time-series at hand, and (2) from the selected models we need to derive what the predictions are at various levels of confidence.

This is not a trivial operation since a model may have a very low Δχ² (fitting the time-series very well), but still give a prediction which is extremely rare when compared to other models with a same low Δχ². In order to make a prediction we therefore carry out an iterative process: We first choose an initial threshold, find the prediction and estimate the error bars. The second iteration will increase the threshold slightly and make another prediction. The iterative approach continues until the error bars are stable to within 5%. This ensures that the estimated error bars are stable and not sensitive to the addition of a couple of models, moreover, the adaptive threshold allows the algorithm to account for time series with different noise properties. How this initial threshold and the sub-sample selection is carried out depends on the desired confidence region (see section).

Estimating the amplitude: Random Forest Regressor

In addition to the standard observables we can also introduce meta-observables to use during the fitting. For example, a key variable in making the prediction is understanding the overall normalisation of the time series (i.e. the mean case number). This way we can remove many models that rise too early, or do not rise at all.

In order to introduce this meta-variable in to our feature set we set-up a classic machine-learning problem: using 13 weeks prior information can we estimate the 26 week mean case rate. To do this we initialise a random-forest regressor (RFR) [22]. We set up feature base where we calculate the mean and standard deviation of each observable over the 13 week time period. The target feature is the 26 week mean of the absolute malaria case. Then using an Extra Trees Random Forest Regressor from the scikit-learn package we train a random forest.

We first split the historical data in to a test and training set. We take all data before the 1st May 2020 as the training set and all data after as the test set. For the training data we compile our feature set of the 13 week mean and variance for all available features (a total of 14 features) and the target feature as the 26 week mean of the absolute malaria numbers. We then carry out a simple grid-based search we determine the optimum number of trees and minimal split sample, which turn out to be 1000 and 2 respectively.

We show the results of our test sample in Fig 3. The left hand panel shows the true versus the estimated normalisation for the test set with the solid blue line showing the 1–1 correlation and the red markers the mean and variance in the estimate. We see that there is a mild bias in our estimate, often over-estimating the true normalisation. We find that the RFR has an error of ∼30%. The right hand panel shows the estimated importance of each feature when estimating the normalisation, with the caption giving the legend. We find that the absolute number of confirmed malaria cases, absolute number of consultations and the absolute number of tests needed (and their standard deviations) are strong predictors, with the others adding only a small amount of information.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The performance of the Random Forest Regressor at estimating the mean malarial case rate over a 26 week period using data from the first 13 weeks.

We set up a feature base and train the regressor. The top panel shows the estimated mean case rate as a function of the true case rate for the test set. The red markers show the mean and variance in binned true case rates. We find that the estimated case rate has a slight bias. The bottom panels shows feature importance from the Random Forest. The legend is as follows, the first seven as the mean 13 value for (1):absolute number of malaria cases, (2):the absolute number of consultations, (3): the absolute number of tests needed, (4): the Gaussian smooth case rate of malaria over a 30km region, (5): the Gaussian smooth case rate of malaria over a 100km region, (6):the rain-fall in mm over a 15km Gaussian smoothed area and 15day time period, (7): the amount of surface-water, and the variance in each feature over the same 13 week window.

https://doi.org/10.1371/journal.pone.0253302.g003

We find that introducing this normalisation constraint improves our model by a factor of two in its precision.

Calibrating the errors: One and two-tailed uncertainties

The threshold (Δχ²) in which we cut the number of models we include in our estimate is somewhat arbitrary. As such we must calibrate it in order to ensure that the Δχ² corresponds directly to the confidence region we state.

The algorithm that we have developed will have two key objectives. The first is a warning system; whereby we alert the user that the number of malaria cases are expected to rise above the threshold for an epidemic in the coming three months. This constitutes a lower bound and hence a one-tailed probability distribution. The second objective is a forecast such that the algorithm sets out a lower and upper bound between which it believes the case rates of the malaria will follow; this constitutes a two tailed probability distribution. As such we calibrate the error to four distinct uncertainties: the two tailed 1σ and 2σ and the one tailed 1σ and 2σ upper and lower bound.

Since the lower the threshold the smaller the uncertainties, we search for the lowest possible threshold that remains consistent with the desired confidence region. We use a Monte Carlo Markov Chain to find this lowest threshold.

Precision and recall

We now estimate the precision and recall of the algorithm at estimating the occurrence of epidemics. It is important to distinguish between ‘claimed epidemics’ (the algorithm raises an alert, regardless of what eventually happens), and ‘actual epidemics’. Precision is defined as the fraction of claimed epidemics that are actual epidemics, and recall is the fraction of actual epidemics which were claimed. Formally, they are defined as (8) where tp = the number of true positives, fp = the number of false positives and fn = the number of false negatives. In order to do this we must create a test set that has known epidemics in them. The World Health Organisation’s definition of an epidemics is when the case numbers rise above the five year average plus two standard deviations. Unfortunately the database only has three years worth of historical information. We therefore create a threshold around the three year mean plus two standard deviations. We create a test set of 131 situations where epidemics have occurred and a test set of 300 scenarios where no epidemic has occurred, again with a minimum date of the 1st May 2020 to ensure not future data exists in the training set.

In order to test the precision and recall of our algorithm we must now make clear what and when our algorithm defines an epidemic. Given the confidence intervals of the prediction, there will be different percentages of confidence in different situations. To avoid confusion, let us now use percentiles rather than confidence intervals. The xth percentile is the value below which x% of the predictions fall. So if the 5th percentile curve crosses the WHO definition of an epidemic, this means that 95% of our models predict an epidemic. Conversely, if only the 95th percentile exceeds the WHO definition, this means that 5% of the models predict an epidemic. It should be straightforward to see now, how choosing the percentile at which we trigger an epidemics directly translates into a degree of confidence at which the epidemic is going to happen.

In brief: when the 5th percentile equals the WHO epidemic definition, we are 95% confident that the epidemic will happen, and vice versa. As a direct consequence, using a low percentile as the trigger guarantees a high precision: the alert is only raised when we are highly confident that an epidemic will happen, and as a consequence only true epidemics are claimed, but also many true epidemics are missed. There is a high number of false negatives. Using a high percentile for the trigger, guarantees a high recall: all true epidemics are claimed, together with a high number of false-positives and few false-negatives.

The top panel of Fig 6 shows how we define our epidemic alert system. We show the probability distribution for the number of cases for a given week and PHC. Should the WHO definition of an epidemic cross a certain percentile, the system will trigger an alert. The bottom panel shows the corresponding precision and recall as a function of these trigger percentiles (that must exceed the threshold before an epidemic is claimed). For example the left-most points show the recall (red, star) and precision (blue, balls) when the 95th percentile goes above the threshold. The blue line gives the precision and the red the recall. We find that when we claim an epidemic as soon as the 95th percentile exceeds the threshold, we return 100% of the epidemics, however with only 30% precision. I.e 70% of the time we claim an epidemic when in-fact there isn’t one. When using higher percentiles, we find that our precision improves to 100% with a recall of 4.5%. Table 1 gives the results of these tests. We note here that the total number of positives in the test set was 131 and negatives in the test set 300 therefore we have a limit of 0.76%, therefore should our algorithm get all correct then this results in a > 99% (since 100% is clearly impossible). Finally we also would like to note that these percentiles are how we classify as our tier grading system in the alert system (see section).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6.

Top: How we trigger an alert for an epidemic. Here we show the predicted number of cases for a given week (of thirteen) and PHC. Each colour shows a different tier of alert and the associated percentile trigger. Should the WHO definition of an epidemic cross one of those triggers, an alert is made, with the associated precision and recall. Bottom: The precision and recall of the algorithm corresponding to these trigger levels. The precision (blue) is the fraction of true positives the algorithm makes from the total claimed positives and the recall (red) is the fraction of true positives from the total number of positives (or epidemics) there are. Here we show these as a function of the proportion of models predicting a case rate above the threshold. For those bounds that include more models that predict the case rate to go above the threshold we get a higher precision, however a lower recall.

https://doi.org/10.1371/journal.pone.0253302.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The precision (fraction of detected epidemics are indeed epidemics) and recall (the rate at which we claim an epidemics in the case of an epidemic) as a function of the percentile (i.e. the proportion of models predicting a case rate below the threshold).

For example, when we raise an alert when the 95th percentile goes above the threshold, we detect all epidemics, however, 68% of the time the claimed epidemics do not result in an epidemic; simply put we claim epidemics all the time.

https://doi.org/10.1371/journal.pone.0253302.t001