Introduction

Artificial intelligence (AI) and deep learning (DL) are transforming the way we use medical images, offering new opportunities for more reliable, objective, and scalable tools to aid in clinical decision-making. This type of approach shows promise in evaluating neurological conditions such as Alzheimer’s disease [1]. Another neurological syndrome, affecting older people, is idiopathic normal pressure hydrocephalus (INPH), which radiologically is characterized by enlarged cerebral ventricles [2]. This condition affects both men and women and becomes more common with age, with an estimated prevalence of 7% among the people over 80 years [3,4]. Patients present with a characteristic gait/balance disorder, often accompanied by cognitive decline and incontinence [5]. Particularly the gait can be improved with neurosurgical insertion of a cerebrospinal fluid (CSF) shunt [6]. INPH is thus a treatable disease, but clinical signs and symptoms overlap with several differential diagnoses, such as Parkinson’s and Alzheimer’s disease, that do not respond to shunting, highlighting a key challenge in INPH care: identifying patients that will improve after shunt surgery [7].

Brain imaging is an essential component of the diagnostic workup for INPH. Ventricular size and morphology measurements have been utilized not only for diagnosis but also to predict surgical outcome [5]. However, the scientific evidence supporting their predictive value remains conflicting [8]. Many approaches exist to improve the selection of INPH patients to surgery and invasive test are today common, in addition to radiology. In a meta-analysis [9], the authors recommend tap-test or infusion testing as first line of invasive test to predict shunt outcome, with an area under the receiving operating curve (AUROC) of 0.71 and 0.73 respectively. Regarding the radiological measures, another meta-analysis found that the callosal angle is the most reliable radiological predictor of shunt response (diagnostic odds ratio = 1.88), though its predictive value is still much lower than invasive tests (tap-test: DOR = 3.86 and infusion testing: DOR = 5.7) and the authors suggest that combining multiple radiological markers using machine learning (ML) may significantly enhance predictive accuracy [10]. This motivates investigation of whether advanced methods that make full use of the information available in the MR images can improve prediction of shunt responsiveness.

Two recently published studies support the application of AI to brain MRI in INPH. Using a 3D convolutional neural network (CNN) ensemble approach, it was demonstrated that AI could identify individuals in the general population who exhibit the characteristic gait pattern associated with INPH [11]. Another group applied 3D CNNs to dual-sequence brain MRI with somewhat promising results in predicting which patients are likely to benefit from treatment [12]. However, further advancements in performance are needed before such models can be clinically useful. Additionally, a recognized challenge in applying AI in radiology is the limited reproducibility of seemingly successful single-center studies [13], therefore confirmatory studies from other centers are needed.

In this study, we adopted a comprehensive approach to determine the effectiveness of AI on structural MRI to predict shunt outcome. We made use of the ensemble search method [11] applied to three standard brain MRI sequences, as well as an alternative DL methodology with intermediate fusion, which enables the model to learn joint patterns in the different sequences. We also tested several ML models, using radiomic features extracted from the images. Our aim was to explore whether suggested AI-based approaches could provide clinically useful predictions of shunt responsiveness in INPH patients.

Method

In summary, we aimed to predict shunt outcome based on gait speed, in patients with INPH using preoperative brain MRI. We analyzed the images with three different methodologies (ensemble search with late fusion, intermediate fusion, and ML on radiomic data) and evaluated them using 10-fold cross validation for balanced accuracy (accounted for class imbalance), sensitivity, specificity, and AUROC.

Based on the idea that different networks might capture different aspects of the data [14] we performed a DL model ensemble search, based on 19 separately trained 3D CNN architectures. While the ensemble strategy applies late fusion at the decision level [11], previous work [12,15] suggests that intermediate fusion of different MRI sequences may have advantages for this type of problem. To explore this, we implemented an intermediate fusion method based on three (one per sequence) ResNet18 networks, connected by information-sharing modules. Considering the limited dataset and complexity of whole brain imaging, for comparison, we also implemented a traditional ML pipeline on radiomic features. Several commonly used ML classifiers were evaluated, after feature selection.

MR Images

Three brain MRI sequences T1w, T2w and FLAIR, were collected from the preoperative scans. As the study is retrospective, the images differed in size and resolution. Images with too low resolution or number of slices were excluded. Table 1 presents the averages along with the least favorable value for size and resolution of included images.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Summary of MRI characteristics.

https://doi.org/10.1371/journal.pone.0350335.t001

Preprocessing

Fig 1 illustrates the preprocessing pipeline. All images underwent co-registration and re-slicing to the ICBM-152 template brain and bias field correction using SPM12 [17]. Images were skull stripped to retain only the intracranial volume using SynthStrip [18]. To mitigate scanner-related intensity differences, voxel intensities greater than 3 standard deviations above the mean were clipped, and all voxel intensities were scaled to range between −1 and 1. Background voxels were set to −1. After preprocessing, all images had uniform resolution of 1 mm isotropic. When fed to the DL networks, the size of each input image was 160x192x160 pixels.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Preprocessing pipeline for both the Deep Learning (DL) and Machine Learning (ML) workflows.

The process is here illustrated for one FLAIR image but the same steps were applied to all MRI sequences. After co-registration and re-slicing, all images were standardized to the same size. Skull stripping was performed to keep only the intracranial volume, followed by intensity scaling to the range [−1, 1], with background voxels set to −1. The images were additionally cropped to exclude redundant background. For ML models, Laplacian of Gaussian filters and wavelet transforms were applied to generate multiple image representations, from which radiomic features were extracted.

https://doi.org/10.1371/journal.pone.0350335.g001

For extracting radiomic features, preprocessed images were used with the brain treated as a single region of interest. Features were extracted separately for each sequence from original, wavelet-transformed, and Laplacian of Gaussian (LoG) filtered images, capturing intensity and texture characteristics across spatial resolutions.

Neural networks

Ensemble search.

We trained 19 well-known 3D CNN architectures, including ResNet [19], DenseNet [20], SENet [21], and SFCN [22] independently on each MRI sequence, to predict shunt response. See S1 Table for the full list of networks, including pretraining. Among all 57 CNNs, those obtaining specificity and sensitivity above 60% on the validation set were included in the late fusion ensemble where their predictions were combined at decision level. A backward search algorithm [11], iteratively removed models based on ensemble performance (balanced accuracy, F1-score) and the diversity of their classification outputs (rho, kappa, and Q-statistic), identifying the most effective subset. This strategy enabled integration of modality-specific features without explicit feature sharing during training.

Multimodal model.

We also trained a multimodal model which used an intermediate fusion strategy [15]. In brief, it consisted of three parallel 3D ResNet-18 streams, one per MRI sequence, interconnected by four Multimodal Transfer Modules [23] (MMTMs). These MMTMs can be selectively activated or deactivated; when all are deactivated, the model effectively behaves as three independent networks with a joint classification layer at the output. If an MMTM is active, information from all modalities is fused and fed back into each stream, allowing the model to learn from all sequences simultaneously. The model was tried with 6 different configurations: no MMTM active, each one of the four MMTMs active at a time, and all MMTMs active. Both the ensemble search and the multimodal model are illustrated in Fig 2.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. The two multisequence approaches.

(A) For the ensemble search the process begins with multiple 3D convolutional neural networks, each independently trained on a single MRI sequence. Networks are removed one by one by a backwards search algorithm based on their contribution to overall performance and output diversity. The procedure stops when further removal no longer improves ensemble performance. Final predictions are aggregated at the decision level from the remaining networks.(B) The multimodal model is composed of three parallel ResNet-18 networks, each receiving one MRI modality (T1w, T2w, FLAIR) as input. The networks are interconnected by four Multimodal Transfer Modules (MMTMs), which enable information exchange between modalities. In the illustrated example, only the second MMTM is activated; at this point, features from all three streams are fused and redistributed to each network. The final classification is produced jointly from the outputs of all three streams. In total, six training configurations were evaluated: one for each MMTM activated individually, one with no MMTMs activated (i.e., independent streams), and one with all MMTMs activated.

https://doi.org/10.1371/journal.pone.0350335.g002

Machine learning models.

The extracted radiomic features included first-order statistics, shape descriptors, and texture metrics derived from gray-level co-occurrence, run-length, size zone, tone difference, and dependence matrices [24]. All features were computed in 3D, using the full brain mask. We evaluated 25 commonly used ML classifiers, including Random Forest, Extreme Gradient Boosting, support vector machines, linear discriminant analysis and neural networks. See S2 Table for the full list.

Results

Performance on the full dataset

We selected the best model from each approach and modality based on the highest validation AUROC, results for the test sets are presented in Fig 3. The backwards search algorithm, resulted in an ensemble consisting of 4 CNNs; 1 using T1w, 2 using T2w and 1 using FLAIR (see Table 2). For the multimodal model with intermediate fusion, the best result was achieved with one MMTM activated on the second position. Detailed results are presented in Table 2. Among all models, none reached an AUROC above 70%, which can be considered poor performance [25]. The best-performing model in terms of both balanced accuracy and AUROC was the Linear Discriminant Analysis classifier based on radiomic features from T1w with 63.74% and 63.75% respectively. As a comparison, we also tested the ensemble consisting of ResNet50 for T2w and FLAIR, resembling the approach of a previous study [12], which had a marginally better AUROC, but performed worse in other metrics.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Test set performance for selected models on the full dataset (shunt responder = 113, non-responder = 36).

https://doi.org/10.1371/journal.pone.0350335.t002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Test set performance on whole dataset.

Mean balanced accuracy and AUROC on the test set over 10 folds of the full dataset (n = 149). The models shown are the best performers from each method and MRI sequence. The yellow ellipse corresponds to the results obtained by the ensemble consisting of ResNet50 for T2w and FLAIR, which was created specifically for comparison to a previous study [12]. The lengths of the axes correspond to the standard error of the mean for the balanced accuracy and AUROC respectively.

https://doi.org/10.1371/journal.pone.0350335.g003

Among the radiomic features, those derived from 3 mm LoG filtered images and gray level run length matrix (GLRLM) metrics were most frequently incorporated into the ML models for T2w and FLAIR. Specifically, the LoG-based Short Run Low Gray Level Emphasis was selected in all 10 folds for FLAIR, while Long Run Low Gray Level Emphasis was consistently selected in all 10 folds for T2w images. The LoG-based gray level co-occurrence matrix (GLCM) Cluster Prominence feature was also present across all folds for T2w. For T1w, the most frequently selected features were wavelet-derived, with GLCM Difference Entropy being selected in 9 out of 10 folds. The 20 selected features varied across folds, in total 92 radiomic features were selected in at least on fold for T1w, 94 for T2w and 102 for FLAIR.

Most improved responders for balanced groups

If brain morphology is related to symptom severity, or shunt outcome, making the group separation clearer would potentially facilitate the models’ ability to identify the important features. To explore whether clearer group separation would improve model performance, we conducted a post-hoc test where we repeated the experiments on a reduced subset of the data. This subset included all 36 non-responders and the 36 shunt responders with the highest improvement in gait speed (Δ > 0.42 m/s). Fig 4 presents mean balanced accuracy and AUROC scores across folds for the best models from each modality.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Test set performance on the reduced dataset.

The figure shows mean balanced accuracy and AUROC on the test set over 10 folds, using the most extreme shunt responders and all non-responders (n = 72). The models shown are the best performing from each method and MRI sequence. The yellow ellipse corresponds to the results obtained by the ensemble consisting of ResNet50 for T2w and FLAIR, which was created specifically for comparison to a previous study [12]. The lengths of the axes correspond to the standard error of the mean for the balanced accuracy and AUROC respectively.

https://doi.org/10.1371/journal.pone.0350335.g004

Table 3 summarizes the detailed results for the subset. Overall performance remained modest in all models. The best-performing model in terms of balanced accuracy was ResNet34 on T2w, achieving 62.50%, and in terms of AUROC, the ensemble model based on five CNNs, at 69.17%. Thus, despite better separation of the groups in the training data, the models did not achieve clinically reliable predictive performance.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Test set performance on reduced dataset (shunt responder = 36, non-responder = 36).

https://doi.org/10.1371/journal.pone.0350335.t003

In the reduced dataset, wavelet-derived features dominated among the selected radiomic features. For T1w, the GLCM Difference Entropy was again the most frequently selected feature, appearing in 9 of 10 folds. For FLAIR images, wavelet-based first-order Skewness was selected in 8 of 10 folds, while for T2w, GLRLM High Gray Level Run Emphasis was selected in 7 of 10 folds. The number of total features used in any fold was similar to when the whole dataset was used with 92 for T1w, 106 for T2w and 103 for FLAIR.

Discussion

In this study, we investigated a range of DL and radiomics-based ML models to predict shunt response, in terms of gait speed improvement, in patients with INPH using preoperative MRI. Despite exploring multiple architectures and modalities, no model achieved clinically useful predictive performance. This suggests that the predictive information in the structural MRI is limited, making it difficult to capture the complexity of the condition. Our findings thus indicate that supplementary data or substantially larger datasets are needed to create reliable prediction models that can improve current clinical practice for INPH.

The proposed ensemble search model, which had previously shown good results on a similar task [11], did not manage to predict shunt outcome and neither did the radiomics-based models. The best result on the whole dataset, in terms of AUROC and balanced accuracy, was obtained by a linear discriminant analysis classifier on radiomic features from T1w images, with 64%. Among all models, the highest AUROC was obtained by the ensemble model applied to the reduced dataset, achieving 69%. That the reduced dataset did not yield better results implies that a critical factor may be the size of the non-responder group, rather than group separation or class imbalance. This is further supported by the observation that different radiomic features were selected in the reduced dataset. This may indicate that the data heterogeneity did not decrease in the reduced group of improvers, rather that the smaller fold sizes may have limited the model’s ability to generalize. This may be promising for future research since class imbalance is natural for INPH cohorts, given that selecting the appropriate patients for surgery is part of the clinical routine. Increasing the group size on the other hand can be managed by multi-center studies. Addressing such factors is needed, since the present results are not good enough to substantially improve the current established clinical practice for shunt investigation [7,9,10].

The ensemble search model has previously, in a population-based study, shown good results in identifying INPH-typical gait impairment compared to controls, based on brain MRI [11], but on this much more challenging task it did not perform as well. Additionally, Leary et al. reported encouraging results (AUROC = 88%) using 3D CNNs on dual-sequence MRI for predicting shunt response [12]. However, our attempt to replicate this approach using a wider set of models and a larger modality combination (including T1w) did not yield comparable results. Nor did using only the same sequences and network [12]; the ResNet50 ensemble obtained slightly better results in AUROC, but performed comparably in other metrics. Our results could be seen as surprising, given the successful outcome in these previous studies, but there are some possible explanations. First, it is important to distinguish diagnostic and predictive tests. Classifying our study participants compared to controls is likely an easier task, since radiology is an essential part of the diagnostic workup in INPH [5,26], whereas the role of structural MRI is more debated in prediction [10]. Second, in our cohort, in contrast to the Leary cohort, the patients have undergone a more extensive clinical protocol including tap test and infusion testing before being selected for shunt surgery. This means we are investigating what more information there are in the images, that can contribute to outcome prediction, beyond the information from these tests. This is probably a more complicated task than investigating which information the images can contribute with, with no additional invasive tests.

Other differences than cohort composition between our study and the one by Leary et al., are outcome definitions and model training protocols. While the previous study used the categorical scoring of the INPH scale [27] for assessing the gait impairment, the present study used maximum gait speed improvement as the outcome measure. Data collection began before the INPH scale was published, consequently it is not applicable for this cohort, preventing a direct comparison with the same outcome metric. Gait and balance are the only symptoms in the INPH triad that are proven to improve by shunting in a randomized, placebo-controlled study, with improvement in gait speed showing the strongest effect [6]. This is therefore a well suited outcome measure for testing these types of alternative approaches to determine shunt outcome. Furthermore, given the extended inclusion period of our study, use of a prospectively and objectively assessed quantitative metric is a strength.

Regardless of the specific reason behind the discrepancy in results, this highlights the challenges of reproducibility and generalizability in AI-based medical imaging research.

Our results could also be explained simply by the fact that although we used a relatively large cohort in the context of INPH research, the sample size is limited from a DL perspective. This may have constrained the models’ ability to generalize, especially when trying to learn high-dimensional features from volumetric MRI data. On the other hand, current radiological measures also have limited predictive value [10], which suggests that structural MRI alone may be insufficient for reliable prediction of shunt outcome in INPH. Shunt response is complex and can be influenced by many factors [7] that are not visible in preoperative structural MRI. For example, comorbidities, CSF dynamics [28], or subtle cognitive and gait features may contribute significantly to patient outcomes, yet potentially remain undetectable on conventional imaging. To determine the reason, bigger datasets are needed in future studies.

From a methodological perspective, this study has several strengths. We used consistent preprocessing and systematically compared a broad array of models, including 19 DL architectures and 25 ML methods, across multiple MRI sequences. We also explored both late and intermediate fusion strategies for multimodal modeling and investigated model performance on a reduced, more clearly defined patient subset. All the models performed quite poorly, supporting our negative conclusion.

There are some limitations worth noting. The analysis was restricted to structural MRI and did not incorporate clinical data, CSF biomarkers, or other potentially informative features [7,26]. Additionally, although follow-up assessments were standardized, shunt response was defined using a single gait speed metric, with an absolute cut-off for improvement (0.16 m/s). For reference, we note that if the improvement was expressed in percentage or preoperative gait speed (relative improvement), this corresponded to the same group division with a cutoff of 12%. Other dimensions of improvement, such as cognitive or urinary symptoms, were not included in the outcome definition, with the exception of excluding those patients who substantially improved in balance but not in gait speed. Typically, improvements in cognition and urinary symptoms are less common than improvement in gait following shunt surgery [29–31], and gait and balance are the only symptoms proven to improve randomized, placebo-controlled study [6], which supports our focus on this symptom. Furthermore, while external validation is generally desirable to assess generalizability, the limited performance observed in the main cohort indicates that no strong predictive pattern was identified. In this context, external validation would primarily have confirmed the modest performance rather than altering the overall interpretation. Lastly, the number of shunt responders was substantially larger than the number of non-responders, resulting in an imbalanced dataset. While this reflects the general situation for existing INPH cohorts rather than a limitation unique to our study, such imbalance can impact model training and evaluation. To mitigate this, we oversampled the non-responder group during training and used class-weighted loss functions.

Future research, if focusing solely on imaging, will require substantially larger datasets. To give predictive models a fair opportunity to learn meaningful patterns, data must capture the full spectrum of imaging presentations in INPH. The extracted radiomic features on the whole dataset were mainly describing texture patterns, characteristics that are difficult to quantify manually, which motivates the use of DL also in future models. However, considering the incompletely understood pathophysiology of INPH, imaging alone is unlikely to yield clinically useful predictive models for shunt outcome and a multimodal approach will probably be necessary. Integrating clinical and physiological data is likely essential. In addition to age, and duration and severity of symptoms, relevant variables could also include CSF outflow resistance, quantitative measures of improvement following the tap test or biomarkers from CSF [32]. Regarding the imaging, incorporating segmented volumes or quantitative measures of the shapes of the brain and the CSF spaces could also be of value. Using these additional features as input to a network, together with brain MRI, would more closely mirror the clinical work-up for INPH patients today. Although none of these measures alone can reliably predict shunt outcome [9,10], a combined model might identify patterns of how these features relate, potentially creating a stronger predictive model. Such multimodal approaches are therefore likely better positioned to capture the complexity of INPH and may offer more robust and clinically useful predictions. Bigger datasets would still be needed to capture the heterogeneity of the patient group, in particular the non-responders. Thus achieving this will likely require multi-center collaboration with harmonized clinical protocols and outcome measures to ensure consistency and generalizability.

Conclusion

In conclusion, our findings suggest that image-based AI models are currently not reliable enough for predicting gait outcome after shunt surgery for INPH patients in a clinical setting. Given that the current radiological measures also have limited predictive value, these findings suggest that AI on structural MRI provides limited additional predictive value for shunt outcome in INPH patients pre-selected using tap test and infusion testing. To capture the complexity of the condition and enable clinically meaningful predictions, our findings indicate the need for research implementing multimodal input and large multi-center datasets.

Supporting information

Acknowledgments

Computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) and the Swedish National Infrastructure for Computing (SNIC) at Chalmers Centre for Computational Science and Engineering (C3SE).