Study population.

A total of 25,902 infants born between 23⁺⁰ and 27⁺⁶ weeks’ gestation were included in the study cohorts to develop and internally validate the prediction tool. Of these, 5,550 (21%) infants died before neonatal discharge. 1,642 (6%) infants with missing data for the nine predictors used in the modelling were excluded from the analysis (S1 Table). The proportion of deaths in infants in the ‘test’ cohort was lower than that in the ‘training and validation’ cohort (20% vs 22%). The infants in the ‘test’ cohort were also slightly more preterm and had a lower median birth weight z score. Mothers of infants in the ‘test’ cohort were more likely to have received a complete course of antenatal corticosteroids (ANC) and delivered in a centre with a co-located neonatal intensive care unit (NICU) (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of infants in the cohorts used to develop, internally and externally validate the prediction model.

https://doi.org/10.1371/journal.pdig.0000955.t001

Algorithm development.

The nine machine learning approaches are described in S1 File. Across all machine learning approaches used, gestational age and birth weight z-score were consistently the two most important predictors based on the mean SHapley Additive exPlanations (SHAP) values (Fig 1). Chorioamnionitis and congenital anomalies were the least important predictors (S1 Fig, S2 Table). Most of the predicted risks derived from the nine machine learning approaches were below 20% (S3 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Beeswarm plot showing the SHapley Additive exPlanations (SHAP) value of the predictors (ranked in the y-axis by their mean absolute SHAP values) for the overall ‘test’ cohort and by weeks of gestation based on the logistic regression model.

Chorioamnionitis and congenital anomalies predictors were dropped from the logistic regression model after backward stepwise selection.

https://doi.org/10.1371/journal.pdig.0000955.g001

Internal validation (discrimination, calibration and utility).

Discrimination (Area under the receiver operating characteristics curve): There was little difference in the discrimination performance in the ‘test’ cohort in terms of area under the receiver operating characteristics curve (AUROC) between the seven approaches of extreme gradient boosting, feedforward neural network, random forest, long short-term memory, adaptive neuro-fuzzy inference system, AutoPrognosis 2.0, and logistic regression with AUROCs ranging from 0.746 to 0.757 (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Model performance of the machine learning approaches and previously published models in the ‘test’ cohort (n = 5,879).

https://doi.org/10.1371/journal.pdig.0000955.t002

Calibration (Calibration plot, calibration-in-the-large and calibration slope): Extreme gradient boosting, feedforward neural network, long short-term memory, and logistic regression had good calibration across the range of predicted mortality risk (Table 2, Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Calibration plot of the machine learning approaches in the ‘test’ cohort (N = 5,879).

Perfect calibration is seen when the predicted value (x-axis) matches the observed average of mortality (y-axis), i.e., when the machine learning approach follows the diagonal grey line.

https://doi.org/10.1371/journal.pdig.0000955.g002

Utility (Decision curve analysis): All the approaches, except for support vector machine, also had a similar superior net benefit compared with providing palliative/comfort care for all or no infants across a reasonable range of mortality threshold probabilities between 10% to 70% when deciding perinatal management in the decision curve analysis (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Decision curve analysis of the machine learning approaches in the ‘test’ cohort (N = 5,879).

The threshold probability denotes the mortality risk threshold above which healthcare professionals would provide palliative/comfort care. Net benefit denotes the ‘net’ mortality cases detected per 100 infants. Net benefit is calculated as a weighted combination of true and false positives based on the threshold probability.

https://doi.org/10.1371/journal.pdig.0000955.g003

The logistic regression model (S4 Table) was further developed in preference to the other machine learning approaches due to its familiarity among healthcare professionals and ease of interpretation, while achieving similar model performance in the ‘test’ cohort.

Comparison with other published models.

Although the discrimination performance of the Manktelow 2013 [17] and Santhakumaran 2018 [2] models (AUROC 0.758 and 0.754, respectively) were similar to our developed logistic regression mortality prediction model (AUROC 0.746) in the ‘test’ cohort, they had worse calibration performance (Table 2 and Fig 4). Our logistic regression model was also found to have superior net benefit across the range of mortality probability thresholds of 20% to 80% than the four previously published models (Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. A, Calibration plot and B, decision curve analysis comparing the logistic regression model with four previously published models in the ‘test’ cohort (N = 5,879).

https://doi.org/10.1371/journal.pdig.0000955.g004

Subgroup analysis.

Of the included infants, 1,500 (26%) and 55 (1%) in the ‘test’ cohort had missing data on maternal ethnicity and neonatal network, respectively. The performance of the logistic regression mortality prediction model in infants born to mothers who identified as Black was worse than in infants born to mothers of other ethnicities. The AUROC was 0.713 with a 95% confidence interval (CI) of 0.655 to 0.770 (S5 Table) with an overestimation of mortality risk (S2 Fig) in infants born to mothers who identified as Black. However, the logistic regression model demonstrated superior net benefit across a range of mortality threshold probabilities between 10% to 50% in all four maternal ethnicities (S3 Fig). There was also variation in the logistic regression model performance across the 13 neonatal networks. In networks with lower neonatal mortality risk, such as Networks 3 and 10, an overestimation of neonatal mortality risk was observed (S5 Table and S2 Fig), and the net benefit of using the model was only found across a narrower range of mortality threshold probabilities (S3 Fig).

Study population.

Our logistic regression model was externally validated on the 1,051 infants recruited to the international, multicentre PLUSS trial [19], after excluding seven infants born before 23 weeks’ gestation and one infant with missing data on prolonged rupture of membranes and chorioamnionitis. In this PLUSS cohort, 206 (20%) infants died before neonatal discharge. Infant characteristics are reported in Table 1.

Model performance.

The logistic regression model had similar discrimination in the ‘external validation’ cohort with an AUROC of 0.728 (95% CI 0.700 to 0.767) with an overestimation of mortality risk (calibration-in-the-large of -0.27 (95% CI -0.52 to -0.01) and calibration slope of 0.94 (95% CI 0.75 to 1.13)) (Fig 5A). The logistic regression model demonstrated superior net benefit across a range of mortality threshold probabilities between 10% to 40% (Fig 5B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. A, Calibration plot and B, decision curve analysis of the developed logistic regression model in the ‘external validation’ cohort (N = 1,051).

https://doi.org/10.1371/journal.pdig.0000955.g005

Quality of the dataset.

Data inaccuracies and missing data could not be controlled for as the data were entered at the point of care. However, predictors used in the dataset were derived from multiple data items within the dataset to ensure the validity of the variables extracted and reduce the missing data rate. At present, the dataset used to develop the tool does not include infants who died in the labour suite or obstetric theatre before admission to neonatal units. This may result in underestimating the true mortality risk among the highest-risk group of infants, typically those <24 weeks of gestation. We used birth weight z scores, which are not known prior to birth, rather than estimated fetal weight z scores, which may be quite different to the actual birth weight, potentially leading to inaccurate mortality estimates.

Generalisability of the tool.

The dataset used to develop our tool encompasses all neonatal admissions of extremely preterm infants in England and Wales from 2012 and 2014, respectively. Hence, it reflected current perinatal practice with a diverse infant cohort representative of the national population. However, the variation in the model performance of our developed tool in different maternal ethnicities and neonatal networks must be interpreted with caution due to the smaller sample size and number of deaths within each subgroup, and the high missing data of 26% in maternal ethnicities. The variation in neonatal outcomes across different maternal ethnicities [24] and neonatal networks [25] may also partly explain some of the differences in model performance found. However, maternal ethnicity and region of care were not included as predictors, as our tool was intended to be used widely across high-income countries. Caution is needed when using the tool in the subgroups with poor performance. Predictions provided by the tool should be considered within the clinical context of the pregnancy and infant. Furthermore, the performance of our tool in different ethnicities and regions of care needs to be continuously monitored to avoid exacerbating ingrained inequalities in the current healthcare system.

Computational expense.

Temporal internal validation was used as it was computationally expensive to analyse the entire dataset using a bootstrapping approach. Our models did not account for birth year as a predictor, as we anticipated that these would be accounted for by the changes in perinatal predictors across the birth years. Furthermore, our tool demonstrated good model performance in the temporal internal validation despite the changes in perinatal characteristics and neonatal outcomes with time.

Perinatal predictors.

Mortality is predicted for the whole neonatal admission, but is based on early perinatal characteristics. The accuracy is therefore unlikely to significantly improve beyond the current routinely collected baseline data without the inclusion of important postnatal events during the NICU stay, and this likely explains why the different models show similar results. Furthermore, the majority of the predicted risks derived by the nine machine learning approaches were below 20% (S3 Table), despite good calibration for some of the approaches (Fig 2). This may be a reflection of the imbalanced dataset, with only 20% infant mortality in the ‘test’ cohort. The current models may also be too conservative due to a lack of variation within the perinatal predictors, and the distribution of predicted risk may improve with further postnatal clinical data.

Further development of the tool will explore the addition of dynamic postnatal clinical data and events, alongside multi-omics biomarkers. This will also allow prediction of other important morbidities, such as severe bronchopulmonary dysplasia or intraventricular haemorrhage, and neurodevelopmental outcomes, which are of prime importance to parents and are often discussed during the perinatal counselling process [13,21,26].

Clinical implementation and explainability.

Despite our tool demonstrating promising findings, further external validation and impact studies on the usability and explanability of the tool, alongside its impact on real-world clinical decision-making, are needed before consideration for use in clinical practice to support shared clinical decision-making and to facilitate clinical implementation. The variable importance demonstrated by the tool for each individualised prediction provided a “technical” measure of explanability, rather than true explanability, as more work is needed to assess the tool from a causal reasoning or decision logic perspective. However, the visualisation of variable importance within the online tool, coupled with the co-design of the tool with focus groups of parents of extremely preterm infants, is a key first step taken in facilitating clinical implementation and explainability.

It is also unclear how our tool performs compared to clinical judgement alone in predicting mortality in extremely preterm infants. Previously published studies demonstrated the complexity and variation of mortality risk assessment in extremely preterm infants among perinatal healthcare professionals. Healthcare professionals were found to underestimate [27,28] or overestimate [29,30] the mortality risk, demonstrating a need for an objective measure in risk assessment and consideration of care plans for extremely preterm infants.

Model development and internal validation.

Routinely entered electronic patient record data from a retrospective population-based cohort of extremely preterm infants born between 23⁺⁰ and 27⁺⁶ weeks’ gestation and admitted to all 185 neonatal units in England and Wales between January 2010 and December 2020 were extracted from the National Neonatal Research Database (NNRD) [33]. This represented over 90% of neonatal units in England in 2010 with complete coverage in England from 2012 and Wales from 2014. Infants with birth weight z score < -4 or >4 based on the UK WHO preterm growth reference [34] or discharged to non-participating neonatal units were excluded as these were potentially erroneous or incomplete entries.

The study population was split into three cohorts depending on the birth year of the infants. Firstly, infants born from 2010 to 2015 (‘training’ cohort) were used to develop the mortality prediction models using different hyperparameters for each of the nine machine learning approaches. Secondly, infants born from 2016 to 2017 (‘validation’ cohort) were used to identify the optimal hyperparameters that produced the best discrimination performance for each of the machine learning approaches. Lastly, infants born from 2018 to 2020 (‘test’ cohort) were used to internally validate the performance of each of the machine learning approaches using the optimal hyperparameters identified previously [35]. The ‘test’ cohort was also used to compare the performance of our preferred final model implemented in our developed online mortality prediction tool with previously published models [2,15–17].

External validation.

The developed prediction tool was externally validated on infants born between 23⁺⁰ and 27⁺⁶ weeks’ gestation enrolled in the PLUSS trial [19] from 21 NICUs in four countries (Australia, New Zealand, Singapore and Canada) from January 2018 to March 2023. Infants were recruited in the first 48 hours of age if they were either invasively ventilated or receiving non-invasive ventilation with a decision to treat with surfactant.

Machine learning approaches.

The prediction models were developed in the ‘training and validation’ cohort using R version 4.3.2 in Rstudio [42] and Python version 3.11.7 in Jupyter Notebook [43] for nine machine learning approaches. The nine approaches were Logistic Regression, AutoPrognosis 2.0, Adaptive Neuro-Fuzzy Inference System, Extreme Gradient Boosting, Feedforward Neural Network, K-Nearest Neighbour, Long Short-Term Memory, Random Forest and Radial Kernel Support Vector Machine (Table 2). For the latter seven machine learning approaches, the dataset was z-score normalised based on the training data values to ensure stability and speed up the convergence of the algorithms. A complete case analysis was used for all approaches, excluding infants with missing data for the predictor variables. Further details of the modelling approaches are described in S1 File. For each of the nine machine learning approaches, the importance of each predictor to the prediction in the ‘test’ cohort was determined using the mean SHAP values. These were derived from Kernel SHAP [44] using the kernelshap package in R [45] and a sample of 100 infants to calculate the marginal expectation. As a subgroup analysis, the importance of the predictors to the prediction of the best-performing algorithm was determined for each gestational week in the ‘test’ cohort using mean SHAP values.

Model performance.

Performance of the nine machine learning approaches was assessed by temporal internal validation in the ‘test’ cohort across the three domains of (i) discrimination (AUROC) [46], (ii) calibration (calibration plot, calibration-in-the-large and calibration slope) [47], and (iii) utility measures (decision curve analysis) [48]. Decision curve analysis is a decision-analytic measure to summarise the ‘net’ benefit of the model in detecting neonatal death across a range of mortality risk thresholds above which healthcare professionals would provide palliative/comfort care (threshold probability).

The performance of the previously developed mortality prediction models (Tyson 2008 [18], Manktelow 2013 [17], Santhakumaran 2018 [2] and NICHD 2020 [16]) was also assessed in the ‘test’ cohort using published regression coefficients for the first three models and the freely available online NICHD 2020 [16] tool. Only infants within the ‘test’ cohort who fulfilled the inclusion criteria for each of the previously published models were used to assess their performance. Subgroup analysis was also performed to assess the performance of the best-performing algorithm across different maternal ethnicities and regions where perinatal care was provided (neonatal networks). The performance of the best-performing algorithm was then externally validated on the PLUSS trial [19] cohort of infants.

Online prediction tool.

An online prediction tool was developed based on the best-performing algorithm using the shiny package in R [49] and deployed on shinyapps.io. The tool displays the predicted mortality risk for each infant alongside the predicted mortality risk if the modifiable predictors of exposure to antenatal corticosteroids and being born in a centre with a co-located NICU were optimised. Explainability of the individual predictions made was provided using the Local Interpretable Model-Agnostic Explanation (LIME) package in R [50]. The ‘training and validation’ cohort was used to train the LIME explainer. Once the LIME explainer is created, 5,000 permutations using forward selection in a ridge regression model and Gower’s distance were used to explain how the individual predictions were made. The tool was co-designed with parents of ex-preterm infants through two focus group sessions organised with the Spoons charity.