https://orcid.org/0000-0002-4538-4445
Figures
Abstract
In resource-limited settings, timely treatment of acute stroke is dependent upon accurate diagnosis that draws on non-contrast computed tomography (NCCT) scans of the head. Artificial Intelligence (AI) based devices may be able to assist non-specialist physicians in NCCT interpretation, thereby enabling faster interventions for acute stroke patients in these settings. We evaluated the impact of an AI device by comparing the time to intervention (TTI) from NCCT imaging to significant intervention before (baseline) and after the deployment of AI, in patients diagnosed with stroke (ischemic or hemorrhagic) through a retrospective interrupted time series analysis at a rural hospital managed by non-specialists in India. Significant intervention was defined as thrombolysis or antiplatelet therapy in ischemic strokes, and mannitol for hemorrhagic strokes or mass effect. We also evaluated the diagnostic accuracy of the software using the teleradiologists’ reports as ground truth. Impact analysis in a total of 174 stroke patients (72 in baseline and 102 after deployment) demonstrated a significant reduction of median TTI from 80 minutes (IQR: 56·8–139·5) during baseline to 58·50 (IQR: 30·3–118.2) minutes after AI deployment (Wilcoxon rank sum test—location shift: -21·0, 95% CI: -38·0, -7·0). Diagnostic accuracy evaluation in a total of 864 NCCT scans demonstrated the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) in detecting intracranial hemorrhage to be 0·89 (95% CI: 0·83–0·93), 0·99 (0·98–1·00), 0·96 (0·91–0·98) and 0·97 (0·96–0·98) respectively, and for infarct these were 0·84 (0·79–0·89), 0·81 (0·77–0·84), 0·58 (0·52–0·63), and 0·94 (0·92–0·96), respectively. AI-based NCCT interpretation supported faster abnormality detection with high accuracy, resulting in persons with acute stroke receiving significantly earlier treatment. Our results suggest that AI-based NCCT interpretation can potentially improve uptake of lifesaving interventions for acute stroke in resource-limited settings.
Citation: Chiramal JA, Johnson J, Webster J, Nag DR, Robert D, Ghosh T, et al. (2024) Artificial Intelligence-based automated CT brain interpretation to accelerate treatment for acute stroke in rural India: An interrupted time series study. PLOS Glob Public Health 4(7): e0003351. https://doi.org/10.1371/journal.pgph.0003351
Editor: Andreas K. Demetriades, Royal Infirmary of Edinburgh, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
Received: March 11, 2024; Accepted: June 19, 2024; Published: July 24, 2024
Copyright: © 2024 Chiramal et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data that support the findings of this study are publicly available from figshare with the following identifiers: - Chiramal, Justy Antony (2024). PreAI_Stroke. figshare. Dataset. https://doi.org/10.6084/m9.figshare.25093697.v1 - Chiramal, Justy Antony (2024). Radioreports_coded_Dxaccuracy. figshare. Dataset. https://doi.org/10.6084/m9.figshare.25093709.v1 - Chiramal, Justy Antony (2024). AI_scores_Dx accuracy. figshare. Dataset. https://doi.org/10.6084/m9.figshare.25093733.v1 - Chiramal, Justy Antony (2024). PostAI_Stroke. figshare. Dataset. https://doi.org/10.6084/m9.figshare.25093703.v1.
Funding: This study was funded by Qure.ai. JAC, DR, TG, SG, PK and SP are employees at Qure.ai, and DRN's salary was funded by Qure.ai. Funder was involved in the study design, data collection and analysis, decision to publish, as well as preparation of the manuscript.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: This study was funded by Qure.ai. JAC currently serves as Clinical Director for Global Health at Qure.ai, but formerly was a student and Research Assistant at the Department of Epidemiology at University of Washington during the study period. DRN is a full-time employees at the Baptist Christian Hospital, but DRN’s salary alone for the study period was funded by Qure.ai. DR, TG, SG, PK and SP are employees at Qure.ai. On behalf of these authors, I confirm that these competing interests will not alter adherence to PLOS Global Public Health policies on sharing data and materials. All other authors declare no competing interests.
1. Introduction
Stroke is the second leading cause of death and the third leading cause of disability globally [1]. There is a loss of >32,000 neurons per second after stroke, thus time to intervention (TTI) sharply affects the morbidity and mortality in stroke patients [2]. The American Stroke Association guidelines recommend completing a clinical evaluation by neurologist/stroke specialist within 15 minutes, analyzing brain imaging within 45 minutes, and initiating intervention within 60 minutes of patients’ arrival into the Emergency Room (ER) [3]. However, selecting the appropriate intervention requires distinguishing between hemorrhagic and ischemic stroke, since the interventions differ for each. For ischemic stroke, interventions include thrombolysis (if onset ≤4.5 hours), and/or mechanical thrombectomy and antiplatelet agents like Aspirin/ Clopidogrel, after ruling out bleed. For hemorrhagic strokes, interventions like Mannitol or neurosurgical procedures target to reduce intracranial pressure. It is therefore vital that physicians can readily access brain imaging findings, including non-contrast computed tomography (NCCT) scans, which may be the only brain imaging available in many of the Low-and middle-income (LMIC) settings.
Thrombolysis and thrombectomy are not optimally utilized across the world, particularly worse in LMIC settings, that witness increasing incidence and prevalence of stroke, alongside dearth of radiology and neurology expertise [4–7]. Though multiple strategies have been implemented to overcome these barriers, including teleradiology and tele-stroke platforms, recent evidence continues to show incomplete access to and utilization of these platforms and interventions [8, 9].
Artificial Intelligence (AI) based computer-aided diagnostic (CAD) assistance software devices may reduce the TTI in many illnesses like intracranial hemorrhage (ICH), traumatic brain injury, and tuberculosis [10–13]. Our objectives were to evaluate the impact of one such AI that automatically interprets NCCT brain scans in acute stroke. We also evaluated the diagnostic accuracy of the AI in detecting multiple findings in NCCT brain using radiologist report as reference standard (RS).
2. Material and methods
2.1 Ethics statement
We obtained Institutional Ethics Committee approval from the Emmanuel Health Association, New Delhi (Protocol No:255 [Version 3]) and from the University of Washington, Seattle, United States of America (ID: STUDY00017314).
2.2 Study setting
This study was conducted at the Baptist Christian Hospital (BCH), Tezpur, Assam, India, which is a 130 bedded charitable hospital under the Emmanuel Health Association. Anecdotal reports and hospital-based studies indicate a huge burden of young stroke and hemorrhagic type of stroke in Assam. A recent study showed that hemorrhagic strokes accounted for more than 50% of the cases in Assam, compared to only about 20% of the strokes in the rest of India [14]. One of the potential factors for higher incidence of hemorrhagic strokes among the Assamese indigenous population and the tea garden workers could be a high prevalence of hypertension reportedly between 33% to 60.8%, which is the largest single risk factor of stroke [15–18].
The ten-bedded ER at the BCH was managed by the same number of healthcare staff throughout the study period (November 2020-November 2022), with every 8-hour shift covered by one junior medical officer and one consultant physician along with three shift nurses and two support staff. The hospital has a CT scanner; official radiology reports were obtained through a teleradiology service. A regulatory approved AI device (qER from Qure.ai) capable of detecting critical abnormalities on NCCT brain scans was deployed at BCH for the purpose of assisting the non-specialist physicians in interpreting those scans. Though there was a radiologist employed towards the end of the project, he was not involved in emergency or routine reporting of the CT scans as his primary responsibilities were managing other imaging like ultrasonography. Since there were no neurologists at the hospital, the physicians availed of tele-stroke assistance from the Christian Medical College, Ludhiana as a part of physician-based stroke unit model [19].
2.3 AI Device description and deployment in study setting
qER (manufacturer: Qure.ai) is an AI-based CAD device that can detect critical findings in NCCT brain scans such as ICH, infarct, mass effect, and midline shift. qER is a hardware agnostic multi-label classifier and has been trained on approximately 300,000 NCCT scans from across the world. In a retrospective validation study, Chilamkurthy etal. observed that qER achieved an AUC of 0·92 (95% CI 0·91–0·93) and of 0·94 (0·92–0·97) for ICH in two different datasets against consensus from a panel of radiologists’ reports as RS. AUCs were 0·93 (0·91–0·94) for midline shift, and 0·86 (0·85–0·87) for mass effect [20]. In a retrospective observational study with data from Swedish Stroke Register, Hillal etal. observed that qER had a sensitivity of 97% in ICH detection against radiologists’ report as RS (n = 1649) [21].
qER version 2 was deployed at BCH in February 2021, on the computer system connected to the CT machine, by the information technologist at the hospital with guidance from engineers of the AI device manufacturer. During the initial few months (Phase 2) the radiology technician manually uploaded the scans to the cloud. In June 2022, this step was automated (auto-push), instantly uploading to the cloud once the imaging was complete. Once the scan was received, qER processed it and generated a radiology report. A proprietary smartphone application developed by the manufacturer notified and alerted the physicians on their phones within 4–5 minutes of the imaging. This application was HIPAA compliant and served as a Cloud-PACS (Picture Archiving and Communications System) for viewing scans, AI-generated radiology reports, and notifications of annotated abnormalities on the key slices; illustrated in Fig 1. The teleradiologists had no access to the AI report at any point.
A. Top Left: the notification alert on a locked screen; B. Top Right: the notification alert on a screen that is pulled down; C. Bottom Left: the report with findings; D. Bottom Right: key slice with annotations of abnormalities, for e.g. Infarct in the given picture. (Name displayed as patient name is of an imaginary person meant only for demonstrating the user interface).
2.4 Study design and the phases of the study
Our interrupted time series analysis used retrospectively collected clinical data from existing hospital records, radiology reports and AI device outputs to compare the TTI in patients diagnosed with stroke before and after the deployment of the AI. We also evaluated diagnostic accuracy of the AI. We collected data for nine months, which was divided into three phases that spanned three months each.
2.4.1 Pre-AI phase (Phase 1): November 2020 to January 2021 (3 months).
We chose the three months just before the deployment of the software as Phase 1 or the baseline period. There was no in-house radiologist or AI at the hospital during this phase. The standard of care was that the physicians would interpret the NCCT images and confirm the findings with a teleradiologist, whenever it was available. For critically ill patients, if the physicians were not confident of their interpretation from scans, they would contact the on-call teleradiologist to get the report early. For acute ischemic stroke patients who were eligible for thrombolysis, they consulted the tele-stroke team for next steps.
2.4.2 Post-AI phase: Phase 2- January 2022 to March 2022, and phase 3- September 2022 to November 2022.
In post-AI phase (three months each in phases 2 and 3), the clinicians had additional input from the AI report and the notification system, in addition to the standard of care. Auto-push feature was available during phase 3 only. We chose the three-month periods when there was more stable uploading of scans, which was 11 months after baseline for phase 2, and five months after phase 2 for phase 3. This was because of the shift in priorities during pandemic, pandemic related travel restrictions resulting in time lag in tackling technical issues, including the delayed enabling of the automatic upload of CTs on the cloud. Though there was a radiologist posted in BCH during latter half of phase 3, physicians were still dependent on the teleradiologist for the routine and emergency CT reporting. The clinical workflow in both pre-AI and post-AI phases is depicted in Fig 2.
Dashed line- represents pathways that were not necessarily part of the clinical decision-making process. Scans were manually uploaded to the cloud in phase 2 and automatically uploaded in phase 3. Description of turnaround times: TTI: from step 3 to step 6 and TTD: from step 3 to step 7.
2.5 Study population and objectives
All consecutive adults aged ≥18 years who underwent NCCT brain imaging for any indication at BCH during the study period were considered potentially eligible for analysis.
2.5.1 Impact evaluation.
The primary objective was an impact evaluation aimed at comparing the time from NCCT imaging to significant intervention (TTI) in patients diagnosed with either hemorrhagic or ischemic stroke before and after deployment of AI. Patients who were diagnosed with acute stroke and had a significant intervention in phases 1 (pre-AI) and 3 (post-AI) were included. We did not include patients from phase 2, because there was only manual uploading of CT scans in that phase, which was dependent on the only radiographer on duty, thus delaying uploading of scans sometimes by hours delaying the AI reports and notifications. Significant intervention was defined as either thrombolysis or antiplatelet agents like Aspirin/Clopidogrel in ischemic stroke or Mannitol for hemorrhagic strokes or for mass effect associated with any kind of stroke. We excluded patients who had diagnoses other than stroke, those with no documented evidence of significant intervention, those with missing date/time for NCCT imaging and/or intervention, those with missing AI reports, and those for whom records indicated significant intervention was before the imaging. As an exploratory analysis, we did a descriptive comparison of time from NCCT imaging to ER disposition (TTD), and scan processing times by AI. TTI and TTD are depicted in Fig 2.
2.5.2 Diagnostic accuracy evaluation.
All consecutive patients who had undergone NCCT brain scans for any indication during phases 2 and 3, with available teleradiologist and AI report were included in this evaluation. The primary objective was to estimate sensitivity and specificity of the AI device in detecting four radiological findings in NCCT brain–ICH, infarct, mass effect, and midline shift using teleradiology report as RS. As secondary objectives, we estimated positive predictive value (PPV), negative predictive value (NPV), and kappa statistics for agreement between AI and teleradiologist. Definitions of the outcome measures and radiological findings used for both the impact and diagnostic accuracy evaluations are described in Table 1.
2.6 Data collection
The data were collected by a staff nurse (DRN) at the hospital, on Microsoft Excel spreadsheets, and included demographic details, clinical and radiology reports and time stamps, without patient identifiers. The data collection for this study was initiated on January 5th, 2023, from the electronic health records, PACS archive, and the physical records of the patients. Only authors from BCH, Tezpur- JJ, JW and DRN, had access to the database with the patient identifiers. The rest of the authors only had access to the excel sheets with the numeric hospital number, which could not be used to identify patients after data collection since the access to the electronic health record system or PACS or physical records is restricted to the hospital staff alone.
2.7 Data analysis
2.7.1 Impact evaluation.
JAC and DR have formal medical training and independently identified cases of acute stroke based on clinical features including time of onset, comorbidities, radiology report, final diagnosis, and interventions by manually reviewing the retrospectively collected data from phases 1 and 3. Discrepancies were resolved by consensus meetings and in consultation with the physician from BCH (JJ). Descriptive statistics and comparisons of the distribution of patients in 30-minute time intervals are reported for TTI and TTD. We used the Wilcoxon Rank Sum test to compare TTI.
TTI was investigated additionally using multivariable regression models by fitting TTI as the dependent variable and adding age, gender, type of significant intervention and phase of the study (pre- or post-AI). We used quantile regression models by fitting 50th (median) quantile of TTI [22]. Additionally, 25th and 75th quantile regression models were fitted as part of exploratory analyses, and standard errors were estimated using bootstrapping [23].
2.7.2 Diagnostic accuracy evaluation.
We used the teleradiologist report as RS for four target findings (ICH, infarct, mass effect and midline shift). JAC and DR reviewed the radiology reports manually and ascertained the presence or absence of target findings for each NCCT brain scan in phases 2 and 3. In cases of discrepancies, a radiologist (TG) with 5 years of experience was consulted for review of reports and original scans, and his decision was considered the ground truth. Point estimates of sensitivity, specificity, PPV and NPV and 95% exact confidence intervals (CI) for each target finding are reported. Agreement between teleradiologist and AI were estimated by percentage agreement, Cohen’s Kappa and Prevalence-adjusted and Bias-adjusted Kappa (PABAK) along with corresponding 95% CI [24].
Statistical analyses for both impact and diagnostic accuracy were independently performed by JAC and DR, and results were finalized after multiple consensus meetings between them. The study is reported following the STARD guidelines, the checklist is available in the supplementary material (S1 Appendix) [25, 26]. All analyses were done using R version 4.2.1 [27].
Though this was human subject research the need for informed consent was waived since the study was done using retrospectively collected data that was anonymized. All the patient identifiers including contact details were removed in the data spreadsheets, except for the hospital numbers that were required to track the radiology and AI reports.
3. Results
Over a nine-month period, 1313 adults underwent NCCT imaging during the pre-AI (n = 323) and post-AI study phases (330 and 660 patients in phases 2 and 3 respectively). There were 174 acute stroke patients for impact evaluation and 851 patients with scans for diagnostic accuracy evaluation. The baseline characteristics of the study population for both impact and diagnostic accuracy evaluation are detailed in Table 2.
3.1 Impact evaluation
There were a total number of 284 patients in this cohort. After excluding 82 patients with no documented evidence of significant intervention (n = 15 in pre-qER and n = 67 in post-qER) and 28 patients for whom records indicated treatment initiation before CT acquisition or those with no record of treatment initiation date and time (n = 12 in pre-qER and n = 16 in post qER), we had 72 stroke patients in the pre-AI phase and 102 in the post-AI phase (38.7% excluded in total). Fig 3 illustrates the data flow of study participants.
The number of patients diagnosed with ischemic stroke in pre-AI and post-AI phase 3 were 39 (54·2%) and 60 (58·8%) respectively, and the rest of the patients experienced hemorrhagic stroke. 37 (51·4%) patients received either thrombolysis or an antiplatelet therapy as significant intervention in the pre-AI phase compared to 60 (58·8%) in post-AI phase. Thrombolysis was administered to 1 (1·4%) and 5 (4·9%) patients in the pre-AI and the post-AI phases respectively. Overall, 74% had hypertension. 19% of the strokes were young stroke (<50 years age) with one patient in ≤30 years age group.
Descriptive statistics of distribution of patients in each of the 30-minute intervals for the pre-AI and post-AI phases are detailed in Table 3 for the timestamps. Median TTI was found to be 80 minutes (IQR: 56·8–139·5) in pre-AI phase and 58·5 minutes (IQR: 30·3–118·2) and the difference in location shift of TTI was significantly different in post-AI phase based on the Wilcoxon rank sum test (location shift: -21·0, 95% CI: -38·0, -7·0, p: 0·007). Fig 4 shows the distribution of TTI across both phases. The proportion of patients who received significant intervention in less than 30 minutes had increased from 4·2% in pre-AI phase to 25·5% in the post-AI phase (p = 0·0004). Most of the patients in the pre-AI phase received their significant intervention within 30–90 minutes, whereas in the post-AI phase most of them received it within the first 60 minutes after imaging.
The peak and the distribution of TTI post-AI is earlier than the peak and distribution of TTI pre-AI.
In the regression analysis the median TTI was found to be significantly reduced in the post-AI phases (regression coefficient: -21·79, standard error [SE]: 8·57, p = 0·011). Patients who received Mannitol received it faster than patients who received any blood thinners (regression coefficient [beta]: -36·08, SE: 9.90, p = 0·0003). The 25th percentile TTI was also significantly different as per the regression analysis (beta: -26·43, SE: 7·81, p = 0·0008), but the 75th percentile TTI, although reduced in the post-AI phase 3, was not statistically significant (beta: -14·20, SE: 24·06, p = 0·555). More details of the regression analysis are available in the supplementary material (S2 Appendix).
The median TTD showed an increase in post-AI (median: 156, IQR: 102–214) compared to pre-AI phase (median: 135.5 IQR: 104·5–214.5). The median time from imaging to AI report notification was found to be 16 minutes (IQR: 13–19). Within this, the median time for NCCT imaging to acquisition by AI cloud was 12 minutes (IQR: 11–13), and the median time from receiving the NCCT scan by AI cloud to the generation of the AI report was 4 minutes (IQR: 2·25–5).
3.2 Diagnostic accuracy evaluation
There were 990 consecutive NCCT brain scans done for any indication, available for analysis from Phase 2 (n = 330) and Phase 3 (n = 660). We excluded two scans with missing AI reports and 124 scans with missing radiology reports (12.7% excluded in total). 124 tele-radiology reports were lost when the hard drive got corrupted during a power surge. Thus, we had 864 eligible NCCT brain scans from 851 distinct patients for this analysis. Thirteen patients each had two NCCT scans taken during multiple visits to BCH. After the independent manual review of the radiology reports of 864 scans by JAC and DR, there were 26 scans that had discrepant RS coding for any of the target findings (ICH, infarct, mass effect and midline shift), and 17 of them were resolved in consensus meetings. TG reviewed the remaining 9 scans along with radiology reports and determined RS for the presence/absence of target findings.
42% of this cohort had hypertension and the most common indication for NCCT imaging was trauma (n = 213, 25%). There were 159 scans with ICH and 206 with infarct. The sensitivity of all findings was ≥0·84 and the specificity were ≥0·93 for all the findings except for infarct. NPV for all findings were ≥ 0·94 and ICH had the highest PPV of 0·96 (95% CI: 0·91–0·98). Cohen’s kappa (k) was 0.90 for ICH suggesting almost perfect agreement between the teleradiologist and AI. There was substantial agreement (0·61< k< 0·8) for mass effect, midline shift, and infarct. All measures of diagnostic accuracy and Kappa statistics with 95% CI for each target finding are reported in Table 4.
4. Discussion
Our results support the use of AI for NCCT brain interpretation to treat acute stroke patients more quickly. Our main finding is the significant reduction in median TTI by 21·5 minutes (approximately 30%) for acute stroke patients when using AI. Diagnostic accuracy evaluation demonstrated high performance of AI in terms of high specificity and NPV for all findings. There was a higher burden of hemorrhagic stroke (43%) and hypertension (74%) in this stroke cohort corroborating with existing evidence peculiar to the geographical area of the study site, which is different from other regions of India [14–16].
Our findings of 30% reduction in TTI were significantly different from what Seyam et al. observed when using AI for ICH in a setting where radiologists and neurologists were present. They found a 10% reduction in turnaround time from a baseline of 70 minutes to communicate scans that were positive for ICH [10]. Though there was a radiologist at BCH in latter half of phase 3, since he was not involved with routine and emergency reporting of CT scans, this reduction in time can likely be attributed to AI. We also observed a significant increase in the proportion of patients who got intervention in the first 30 minutes after the imaging in the post-AI phase when compared to the pre-AI phase, 26% vs 4%. We did not observe a reduction in TTD in the post-AI phase which may be dependent on many factors beyond the scope of this study such as the lack of availability of in-patient/ ICU bed or delay in referral. But this serves as a ground reality to show that there was no escalation in ER services/ infrastructure between phases, although they were many months apart.
Our diagnostic accuracy evaluation showed that the sensitivity and specificity of the AI device to detect ICH were 0·89 (95% CI: 0·83–0·93) and 0·99 (95% CI: 0·98–1·00) respectively. Schmitt et al. reported a sensitivity and specificity of 0·91 and 0·89 respectively for a similar AI device in ICH detection [28]. Similarly, Seyam et al. reported sensitivity, specificity, and negative predictive values to be 0·87, 0·94 and 0·98 to detect ICH respectively for another device [10]. The specificity and the NPV of qER to detect ICH were ≥0·97. This is a very crucial finding for deciding on thrombolysis in patients who present within 4·5 hours of onset of acute ischemic stroke, since hyperacute infarct may not be visible on NCCT until many hours later. Diffusion Weighted Imaging sequence of MRI brain is the earliest modality to pick up acute infarcts, which is inaccessible and unaffordable for patients in resource limited settings. Thus, NCCT brain that is negative for ICH in a stroke suspect, could help improve uptake of lifesaving measures like thrombolysis for ischemic strokes in these settings, which is in accordance with the guidelines [3]. There was almost perfect to substantial agreement between AI and radiologist for all of the target findings.
Our results are consistent with the recommendation to use AI in hospitals where there are no in-house radiologists. T.J. Yun and colleagues concluded from a multi-reader diagnostic accuracy project that there was a significant improvement in AUC among physicians with AI assistance, the most noticeable improvement was among non-radiology physicians from 0.92 to 0.95 [29]. qER empowered the physicians as an added boost to their confidence in making early intervention decisions for neurocritical patients. The notification system on their phones alerted them to look at the scans promptly.
The median time from NCCT imaging to generation of AI report was found to be 16 minutes which was unusually high based on experiences from other deployments of the same device. The main limiting factor in this setting was the low internet connectivity, which prolonged the time from imaging to NCCT acquisition by the AI cloud at a median time of 12 minutes. Once the images were acquired, the report was generated within a median time of 4 minutes. A reliable and stable internet connection could enable faster AI report generation.
Our study had many strengths, including analysis from a real-world deployment of a commercially available product in a context where AI can improve time to care. We note below a few limitations. The pre-AI and post-AI phase 3 for impact evaluation were separated by 19 months owing to the COVID-19 pandemic, thus the chance of confounding cannot be ruled out. The effect of the pandemic was also noticeable in the number of patients who underwent CT: phase 3 patients had almost double the number of scans than phases 1 and 2, when hospitals were overwhelmed with COVID patients. Another limitation was that RS for diagnostic accuracy evaluation was based on a single teleradiologist’s report whereas ideal RS should have been collected from a panel of expert radiologists. In the impact evaluation cohort, about 38.7% of potentially eligible patients were excluded mainly due to no documented evidence of administration of a significant intervention. Similarly, about 12.7% of potentially eligible NCCT scans were excluded from the diagnostic accuracy evaluation mainly due to missing radiology reports. These reports were missing from the EMR database, because as per the physicians at BCH the online tele-reporting at night used to come through a different software but was not captured in the EMR. They were saved in a hard drive, which got corrupted during a power surge and thus lost access to this set of reports. Therefore, we cannot exclude the possibility of selection bias affecting our results. Although our study only covered one hospital, we could observe meaningful outcomes from this pilot which is crucial since there have not been any publications that have evaluated the usefulness of this kind of technology in LMICs, where stroke care has been dismal [8, 9].
5. Conclusion
Our work demonstrated that using AI that has high accuracy for NCCT interpretation had positive impact on empowering the non-specialist physicians to reduce the time to intervention for patients with acute stroke in resource limited settings, making more patients eligible for time sensitive interventions. We recommend larger randomized controlled trials and qualitative research in LMICs to study the impact of similar devices in the clinical workflow on patient outcomes.
Acknowledgments
The authors thank Asolie Chase MD. (Medical Director at the Baptist Christian Hospital, Tezpur) who approved the deployment of the AI device and helped in the smooth functioning of the study. We also thank Qure.ai employees Swetha Tanamala for the study design and the development of the AI device, Ayan Kumar for facilitating data collection and Ranjana Devi for the feedback on data analysis and write up.
References
- 1. Feigin VL, Stark BA, Johnson CO, Roth GA, Bisignano C, Abady GG, et al. Global, regional, and national burden of stroke and its risk factors, 1990–2019: A systematic analysis for the Global Burden of Disease Study 2019. Lancet Neurol. 2021;20(10):1–26. pmid:34487721
- 2. Saver JL. Time is brain—quantified. Stroke. 2006 Jan;37(1):263–6. pmid:16339467
- 3. Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Bambakidis NC, Becker K, et al. Guidelines for the Early Management of Patients With Acute Ischemic Stroke: 2019 Update to the 2018 Guidelines for the Early Management of Acute Ischemic Stroke: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke. Stroke [Internet]. 2019;50(12): e344–418. Available from: https://www.ahajournals.org/doi/abs/10.1161/STR.0000000000000211
- 4. Damasceno A, Gomes J, Azevedo A, Carrilho C, Lobo V, Lopes H, et al. An epidemiological study of stroke hospitalizations in Maputo, Mozambique: a high burden of disease in a resource-poor country. Stroke. 2010 Nov;41(11):2463–9. pmid:20930157
- 5. Pandian JD, Kalkonde Y, Sebastian IA, Felix C, Urimubenshi G, Bosch J. Stroke systems of care in low-income and middle-income countries: challenges and opportunities. Lancet (London, England). 2020 Oct;396(10260):1443–51. pmid:33129395
- 6. Hricak H, Abdel-Wahab M, Atun R, Lette MM, Paez D, Brink JA, et al. Medical imaging and nuclear medicine: a Lancet Oncology Commission. Lancet Oncol. 2021 Apr;22(4): e136–72. pmid:33676609
- 7. Ogbole GI, Owolabi MO, Ogun O, Ogunseyinde OA, Ogunniyi A. TIME OF PRESENTATION OF STROKE PATIENTS FOR CT IMAGING IN A NIGERIAN TERTIARY HOSPITAL. Ann Ibadan Postgrad Med. 2015 Jun;13(1):23–8.
- 8. Sarfo FS, Akpa OM, Ovbiagele B, Akpalu A, Wahab K, Obiako R, et al. Patient-level and system-level determinants of stroke fatality across 16 large hospitals in Ghana and Nigeria: a prospective cohort study. Lancet Glob Heal. 2023 Apr;11(4): e575–85. pmid:36805867
- 9. Asif KS, Otite FO, Desai SM, Herial N, Inoa V, Al-Mufti F, et al. Mechanical Thrombectomy Global Access For Stroke (MT-GLASS): A Mission Thrombectomy (MT-2020 Plus) Study. Circulation. 2023 Apr;147(16):1208–20. pmid:36883458
- 10. Seyam M, Weikert T, Sauter A, Brehm A, Psychogios M-N, Blackham KA. Utilization of Artificial Intelligence-based Intracranial Hemorrhage Detection on Emergent Noncontrast CT Images in Clinical Workflow. Radiol Artif Intell. 2022 Mar;4(2): e210168. pmid:35391777
- 11. Hibi A, Jaberipour M, Cusimano MD, Bilbily A, Krishnan RG, Aviv RI, et al. Automated identification and quantification of traumatic brain injury from CT scans: Are we there yet? Medicine (Baltimore). 2022 Nov;101(47): e31848. pmid:36451512
- 12. MacPherson P, Webb EL, Kamchedzera W, Joekes E, Mjoli G, Lalloo DG, et al. Computer-aided X-ray screening for tuberculosis and HIV testing among adults with cough in Malawi (the PROSPECT study): A randomised trial and cost-effectiveness analysis. PLoS Med. 2021 Sep;18(9): e1003752. pmid:34499665
- 13. Zhan Y, Wang Y, Zhang W, Ying B, Wang C. Diagnostic Accuracy of the Artificial Intelligence Methods in Medical Imaging for Pulmonary Tuberculosis: A Systematic Review and Meta-Analysis. J Clin Med. 2022 Dec;12(1). pmid:36615102
- 14. Mahanta BN, Mahanta TG, Gogoi P. Clinico-epidemiological profile of stroke patients admitted in a tertiary care Hospital of Assam. Clin Epidemiol Glob Health. 2018 Sep 1;6(3):122–9.
- 15. Hazarika NC, Biswas D, Narain K, Kalita HC, Mahanta J. Hypertension and its risk factors in tea garden workers of Assam. Natl Med J India. 2002 Mar-Apr;15(2):63–8. pmid:12044117.
- 16. Hazarika NC, Narain K, Biswas D, Kalita HC, Mahanta J. Hypertension in the native rural population of Assam. Natl Med J India. 2004;17(6):300–4. pmid:15736549
- 17. Biswas D, Narain K, Mahanta J. Hypertension in the native rural population of Assam. 2003 Nov; Available from: https://www.researchgate.net/publication/7996484
- 18.
Anchala R, Kannuri NK, Pant H, Khan H, Franco OH, Di Angelantonio E, et al. Hypertension in India: A systematic review and meta-analysis of prevalence, awareness, and control of hypertension. Vol. 32, Journal of Hypertension. Lippincott Williams and Wilkins; 2014. p. 1170–7.
- 19. John L, William A, Dawar D, Khatter H, Singh P, Andrias A, et al. Implementation of a Physician-Based Stroke Unit in a Remote Hospital of North-East India-Tezpur Model. J Neurosci Rural Pract. 2021 Apr;12(2):356–61. pmid:33927525
- 20. Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet (London, England). 2018 Dec;392(10162):2388–96. pmid:30318264
- 21. Hillal A, Sultani G, Ramgren B, Norrving B, Wassélius J, Ullberg T. Accuracy of automated intracerebral hemorrhage volume measurement on non-contrast computed tomography: a Swedish Stroke Register cohort study. Neuroradiology. 2023 Mar;65(3):479–88. pmid:36323862
- 22. Koenker R, Hallock KF. Quantile Regression. J Econ Perspect [Internet]. 2001 Dec;15(4):143–56. Available from: https://www.aeaweb.org/articles?id=10.1257/jep.15.4.143
- 23. Hao L, Naiman D. Quantile Regression [Internet]. Thousand Oaks, California; 2007. Available from: https://methods.sagepub.com/book/quantile-regression
- 24. Chen G, Faris P, Hemmelgarn B, Walker RL, Quan H. Measuring agreement of administrative data with chart data using prevalence unadjusted and adjusted kappa. BMC Med Res Methodol [Internet]. 2009;9(1):5. Available from: pmid:19159474
- 25. Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK, Chan A-W, et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nat Med [Internet]. 2020;26(9):1364–74. Available from: pmid:32908283
- 26. Cohen JF, Korevaar DA, Altman DG, Bruns DE, Gatsonis CA, Hooft L, et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open [Internet]. 2016;6(11). Available from: https://bmjopen.bmj.com/content/6/11/e012799 pmid:28137831
- 27.
R Core Team. R: A Language and Environment for Statistical Computing [Internet]. Vienna, Austria; 2022. Available from: https://www.r-project.org/
- 28. Schmitt N, Mokli Y, Weyland CS, Gerry S, Herweh C, Ringleb PA, et al. Automated detection and segmentation of intracranial hemorrhage suspect hyperdensities in non-contrast-enhanced CT scans of acute stroke patients. Eur Radiol. 2022 Apr;32(4):2246–54. pmid:34773465
- 29. Yun TJ, Choi JW, Han M, Jung WS, Choi SH, Yoo R-E, et al. Deep learning based automatic detection algorithm for acute intracranial haemorrhage: a pivotal randomized clinical trial. npj Digit Med [Internet]. 2023;6(1):61. Available from: pmid:37029272