Collection, storage, and integration

The data collection process initiates with the patient providing their information upon arrival at the examination area. Following registration and identification of the examination type, the patient proceeds to the imaging diagnostic device for the procedure. However, not all imaging devices automatically deliver images in the DICOM format. The use of this standard varies depending on the device type, manufacturer, and the healthcare setting’s specific needs. Certain older devices or those used in specialties like endoscopy or ophthalmology often produce images in proprietary formats, necessitating their conversion or export of the images and metadata to DICOM. Fig 2 displays the typical operational cycle in imaging diagnostic settings, highlighting the handling of various examination formats, both DICOM and proprietary. After storage, these examinations are accessible for further evaluation and annotation by specialist doctors.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Data collection cycle by a specific department or institution.

https://doi.org/10.1371/journal.pone.0307022.g002

Diagnostic imaging protocols vary across different modalities and clinical areas, yet share key common elements. These elements include pre-examination preparation instructions, patient positioning guidelines for optimal image acquisition, and technical parameters such as radiation exposure and scanning duration. In MRI and CT scans, protocols are tailored with specific sequences for various body parts or clinical conditions. When contrast agents are used, guidelines cover their application, necessary precautions, and patient monitoring. Maintaining a detailed record of patient information, clinical conditions, and procedural specifics is necessary. Quality criteria ensure image suitability, while post-examination care guidelines are also provided. These generic elements are adaptable to each modality and clinical area’s unique requirements. Standardizing these protocols enhances the quality and reliability of diagnostic imaging results.

The proposed methodology begins after examinations are complete. As illustrated in Fig 2, examinations first pass through a gateway before reaching the PACS. This gateway ensures the secure transfer of DICOM images and other formats between institutions. In the PACS, illustrated in Fig 1, the anonymizer processes the examinations to protect patient confidentiality by removing identifiable information. This de-identification is done using the Clinical Trials Processor (CTP) DICOM Pixel Anonymizer (DICOM Pixel Anonymizer: https://github.com/susom/mirc-ctp) [57]. The DICOM Pixel Anonymizer uses a script to decide which parts of the image to erase, based on information not contained in the DICOM file pixels. This step involves analyzing incoming DICOM objects to inspect if they are images, and if so, checking their compression status. When images meet the specified criteria, the anonymizer executes the script to remove selected areas. These areas are precisely defined by rectangular coordinates. If an object does not meet the criteria, it remains unmodified and passes through the system.

Following de-identification, the examinations proceed to the standardizer, as illustrated in Fig 1. This tool corrects specific fields in the DICOM images, aiding AI models in recognizing the type of examination and the body areas examined. After standardization, the data is transferred to AWS S3/MinIO, which constitutes the set of storage systems with an abstraction layer. This abstraction layer, termed ‘abstrador’, decouples the object storage system from the rest of the project. The abstraction layer’s role is to provide a consistent interface that encapsulates the complexities of the underlying storage provider. This allows other project components to interact with the abstraction layer without needing to deal with the specific nuances of each storage provider. This design makes the project vendor-agnostic, avoiding reliance on any specific provider and enabling a seamless transition between different storage services (cloud or on-premise infrastructures). Following the standardizer and ensuring that all examinations are in DICOM format, the Fast Healthcare Interoperability Resources (FHIR) [58] protocol is employed. Developed by HL7, the FHIR standard streamlines the exchange of electronic health information, enhancing the integration of healthcare systems and standardizing data sharing.

With the integrated system in place, all data is forwarded to the VNA. The VNA is connected to AWS S3/MinIO, an encrypted database that maintains correspondences between the original examination dataset and the anonymized examination dataset. This process is utilized to link the original examinations with the examinations that will undergo computational manipulations, being necessary for data integration, migration, and transformation, ensuring consistency and correspondence between the examinations. Thus, during computational manipulation, the patient’s identity associated with the examination under analysis is kept confidential. However, after the entire process, the specialist doctor needs to compare the original examination with the analyses generated by the AI models. The technique used for creating the encrypted identification pointer is developed in the Research Electronic Data Capture (REDCap) Web Platform for data collection and management [59, 60]. In REDCap, access numbers identifying the anonymized examinations are registered, establishing the connection with the examinations stored in AWS S3/MinIO. Only the specialist doctor associated with the patient has permission to access the identification number in REDCap.

The VNA also connects to the NoSQL/MongoDB database, used for flexible and scalable data storage, employing JSON or BSON documents in collections. NoSQL/MongoDB is chosen for its ability to handle semi-structured data, horizontal scalability, and support for complex operations, proving especially useful in environments with rapid development requirements, efficient storage of large data volumes, and dynamic data. At the output of the VNA, there are the dispatcher and the DICOMWeb. The dispatcher notifies the Integration and Storage System to send a new examination, providing a delayed notice to ensure that all information from the previous examination has arrived in the Prediction System.

The dispatcher connects directly to the router, which confirms the arrival of all examination information. The router analyzes the information contained in the forwarded examinations, identifying the type of examination and verifying whether the AI model for that examination exists in the AI model database. Between the dispatcher and the router, the information is in FHIR standard, however, when manipulating data in the AI models, DICOM formats are required. Thus, when it is necessary to manipulate the examinations in the AI models, DICOMWeb is requested. DICOMWeb is the set of web services that uses the principles of Representational State Transfer (RESTful) architecture (These principles guide the design of Web distributed systems, prioritizing a unique representation of resources and stateless communication, promoting flexibility and interoperability in Web APIs). for communication and information exchange in DICOM-format images, adapted for web-based environments and promoting interoperability among different systems and providers.

Deployment of a model to assist in diagnoses

If the router identifies that the model for the examination does not exist, it is sent to the manual analysis database. The information that the model is nonexistent is automatically forwarded to the model management system. If the AI model exists, the examination is directed toward an analysis of compatibility (adequate exam) parameters with the model. If the examination has all the necessary parameters, the AI model is applied to the examination. If the results of the manipulation by the AI model are the desired results, the examination is sent to the predicted exams database, otherwise, the examination is sent to the failure database.

In the case where the examination lacks all the necessary parameters, it is routed to the model management system. The model management system provides access to exam types that do not have defined models, to exam types that were not adequate with pre-existing models due to a lack of parameters, and to all registered models. In the manual database, examinations can be manually analyzed or wait for a model to analyze them. In the AI model database, all existing models for diagnostic analysis are stored, and these models can be in any language, as the platform is language-agnostic.

In the Prediction System, all logs of exams with unsatisfactory results are forwarded to the failure database for manual analysis. Exams with satisfactory results are sent to the predicted exams database, making them available for consumption by the image viewer of the VNA. The image viewing system is based on the Open Health Imaging Foundation (OHIF) [36, 61]. In this tool, it is possible to compare the original exam in the VNA with the predicted exam, presenting heatmaps, and segmentations, among other features. Only the physician responsible for the patient has access to the original exams and the exams predicted by AI models.

Analysis, testing, and validation

The analysis, testing, and validation of the proposed system are fundamental processes to ensure successful deployment. Efficiency, security, and compliance with clinical and regulatory requirements are necessary to ensure safety and accuracy in examinations [62]. These stages include requirements analysis to understand the specific functionalities needed, detailed test planning identifying critical cases and acceptance criteria, and unit testing to verify the proper functioning of each component [63]. Integration tests ensure efficient communication between different modules of the proposed system, while performance tests assess behavior under different conditions, including maximum load [64].

Security is specifically tested to identify and address potential vulnerabilities in the system. Compliance tests are conducted to verify adherence to standards and regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) (U.S. legislation from 1996 that protects health information, establishes privacy and security standards, and regulates the electronic transfer of data. Applicable to healthcare organizations and professionals, it imposes strict protocols and penalties for non-compliance). Disaster recovery capability is assessed to ensure operational continuity in adverse situations [63]. Additionally, clinical validation is performed through clinical tests involving healthcare professionals to ensure the system meets clinical needs and provides accurate and reliable results [65]. User training is provided to ensure a comprehensive understanding of the system and its functionalities [64].

Continuous evaluation is deployed to monitor the system’s performance and make updates as necessary [62]. All these tests are thoroughly analyzed and validated to ensure the system is robust, secure, efficient, and complies with clinical and regulatory requirements. Collaboration among Information Technology (IT) teams, healthcare professionals, and system providers is necessary to achieve maximum efficiency in system deployment [63]. Although the initial deployment cost is high, uniform and defined criteria are necessary for cost calculation, leading to savings over time after deployment [24]. In 1992, Choplin et al [66] emphasized that the goals of such a system are to improve operational efficiency while simultaneously maintaining or enhancing diagnostic capacity [66].

Identifying ventricular walls in myocardial scintigraphy

In this study, the proposal is to use AI techniques, specifically the nnU-Net convolutional neural network, to enhance the identification of left ventricular walls in myocardial perfusion scintigraphy (MPS) images, aiming to improve the diagnosis and treatment of coronary artery disease. The methodology included data collection in a clinical setting, followed by data preparation and analysis using the 3D Slicer platform for manual segmentation and subsequently the application of AI models for automated segmentation. A total of 83 clinical routine exams were collected, each containing 50 slices, totaling 4,150 images, at the Albert Einstein Hospital, São Paulo, Brazil, during February and March 2023. For the MPS procedure, the standard protocol with acquisition in the supine position at rest was employed. The inclusion criterion was the signing of the Informed Consent Form (ICF), including all patients who underwent the examination during the collection period and signed the ICF. The exclusion criterion was the non-signing of the ICF.

All exams had reports, but only copies of the DICOM images were used for this study, with the originals stored in the PACS. Demographic data were collected and recorded in REDCap by the principal investigator, with no other researchers having access to sensitive patient data. After collection, the exams were anonymized to ensure patient privacy during analysis. Only the principal investigator has access to the I_N associated with the exam, stored in the REDCap form. A total of 83 patients were analyzed, aged 25 to 88 years. Of the 83 exams, 43 were used for validation, and three were randomly selected for submission to specialists. Each exam contains 50 slices, totaling 150 images per specialist. The remaining exams were divided into 31 for training and nine for testing.

Three nuclear medicine specialists, with 18 years of experience, with 20 years of experience, and with 12 years of experience, independently and blindly evaluated the delineation of the left ventricle walls in each mask. One mask was manually created and used for training the network, and the other was generated by the proposed AI model, both focusing exclusively on the left ventricle walls. This analysis showed a high degree of agreement among the specialists most of the time, with evaluations indicating between 80% and 100% concordance in the delineation of the masks in both groups, as shown in Table 1. Fig 3, extracted from Nogueira et al. [73], visually presents the results, displaying Test 1, Test 2, and Test 3 in the rows, and the original images, the masks generated by AI, and the masks manually created by the specialists in the columns, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. General medical evaluation of the tests.

https://doi.org/10.1371/journal.pone.0307022.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Images chosen at random and forwarded to the M_N in charge of evaluation.

https://doi.org/10.1371/journal.pone.0307022.g003

Table 2 presents the individual assessments of the tests and their agreement indices. In the qualitative analysis conducted by the medical specialists M_N, at least 80% agreement was observed regarding the positioning of the masks generated by the proposed model. In Test 1, which recorded the greatest divergence among the evaluators, the lowest score was assigned to the manual mask. The discrepancy was primarily due to the proximity of the heart to the artifact zone, resulting in 60% agreement in the positioning of the manual mask.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Individual medical assessment by test.

https://doi.org/10.1371/journal.pone.0307022.t002

This study demonstrated the efficiency of AI techniques in identifying the left ventricular walls in MPS images, necessary for the diagnosis and treatment of coronary artery disease (CAD). The AI model achieved a Dice coefficient of 87% and an average Intersection over Union (IoU) of 0.8, aligning with the manual segmentations performed by specialists. The objectives were met, highlighting the model’s accuracy and feasibility in medical practice, significantly advancing medical imaging by improving diagnostic precision and optimizing processes, establishing a solid foundation for the continued development of AI in healthcare.

Unsupervised model for brain structure segmentation

This study presents an unsupervised method for segmenting brain computed tomography scans. The proposed methodology involves image feature extraction and the application of similarity and continuity constraints to generate segmentation maps of the anatomical head structures. Designed for real-world datasets, this approach utilizes a spatial continuity scoring function adapted to the desired number of structures. The primary objective is to assist medical specialists in diagnosis by identifying regions with specific anomalies.

The unsupervised methodology produces segmentation masks, in contrast to the results generated by the CTSeg tool. To test the AI model, the Computed Tomography Quality 500 (CQ500) database, containing 500 brain CT scans performed by various hospitals in India, was used. This dataset was included in the ISMIR PACS to test data from another platform and the AI model for volumetry. The images are high quality and properly anonymized, along with clinical reports prepared by three radiologists with 8, 12, and 20 years of experience in interpreting brain CT scans.

Six exams were selected from the set of 500 exams. A total of twelve simulations were performed for each label or class: gray matter c₁, white matter c₂, cerebrospinal fluid c₃, skull bone c₄, soft tissue c₅, and background c₆. The gold standard segmentations used to calculate D_c are from the CTSeg tool, and the calculation of D_c is performed using 3DSlicer. Fig 4, extracted from Santos et al. [74], shows the overlap of the CT195 exam segmentation performed by the CTSeg tool and the segmentation obtained by the proposed method. By relating the six labels used by CTSeg to the labels obtained by the proposed method, it was possible to identify overlap in all classes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Segmentation overlap of the CT195 exam: (a) c₁, (b) c₂, (c) c₃, (d) c₄, (e) c₅ e (f) c₆.

https://doi.org/10.1371/journal.pone.0307022.g004

For the segmentation of classes c₁ and c₂, shown in Fig 4(a) and 4(b) respectively, the result in the dark shade represents the entire set, while the segmentation by the CTSeg tool in the light shade is the correct subset. In contrast, for the segmentation of classes c₃, c₄, and c₅, shown in Fig 4(c)–4(e) respectively, there is an intersection where the segmentation by the CTSeg tool in the light shade is the set and the segmentation by the proposed method in the dark shade is the correct subset. Finally, for the segmentation of class c₆, shown in Fig 4(f), the CTSeg tool incorrectly considered classes c₄, c₅, and c₆, resulting in inaccurate segmentation, while the proposed method performed the segmentation correctly.

The developed unsupervised segmentation method achieved an accuracy exceeding 65%, with notable performance in white matter segmentation and results similar or superior to CTSeg in cranial volumetry. Experts recommended a range of 3≤ labels ≤6 for the best visual results. The methodology demonstrated efficiency in training the neural network, reducing resources, training time, and costs, and facilitating the early detection of anomalies in brain CT scans.

Pulmonary embolism classification

This study presents a classification model using AI for the detection of pulmonary embolism (PE) in computed tomographic pulmonary angiography (CTPA). The proposed model, developed from ISMIR data, utilizes a two-dimensional approach that integrates temporal series to classify each slice of the examination and make predictions at both slice and examination levels. The training process consists of two stages: first, using the InceptionResNet V2 convolutional neural network, and then, the long short-term memory (LSTM) recurrent neural network model.

Randomly, 800 exams were collected from ISMIR data, consisting of 400 positive and 400 negative for PE, with a maximum limit of 300 slices per exam. The Python pydicom library was used to load the exams, totaling 181,747 slices, with 16,492 positive and 165,255 negative, as shown in Table 3. Automatic segmentation was performed for each exam to identify the lung region. In this region, windowing was applied with a window of 1,400 and a level of -600. Completing the segmentation, background removal operations, edge region extraction, and morphological operations of erosion, dilation, OTSU thresholding, and closing were performed using the Python OpenCV library to generate the mask applied to the image. The original resolution of each slice was maintained at 512 × 512 pixels. To highlight the region of interest for PE classification, windowing with a level = 100 and width = 700 was applied to each slice. Each image was converted from DICOM to PNG 8-bit.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Dataset information.

https://doi.org/10.1371/journal.pone.0307022.t003

Subsequently, data augmentation was performed, generating new images to improve the generalization of the InceptionResNet V2 CNN model by applying rotation, translation, brightness alteration, among others. Ten variations were created for each positive slice, totaling 164,920 images. The number of variations in data augmentation was determined by dividing the original number of negative slices by the number of positive slices. InceptionResNet V2 was trained to learn the features of positive or negative slices for PE. Only images with the visible lung region were considered valid, resulting in the exclusion of 50 positive slices. A total of 330,125 slices were considered for training the 2D model, with 164,870 positive and 165,255 negative. The feature encodings of each exam were extracted from the penultimate layer, the dense layer. Finally, the RNN LSTM model was trained to generate predictions at the image and exam levels. Both techniques, CNN InceptionResNet V2 and RNN LSTM, were coded using the Keras/Tensorflow libraries. Fig 5, extracted from Silva et al. [75], shows the application of windowing and the proposed automatic segmentation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Application of automatic windowing and segmentation: (a) original image, (b) windowing for PE, (c) windowing for lungs, and (d) automatic segmentation.

https://doi.org/10.1371/journal.pone.0307022.g005

Therefore, the model adopting the CRNN classification approach was implemented, and the 2D and 2D+LSTM models were trained using cross-validation with k = 5 for 25 and 50 epochs, respectively. The NVIDIA TeslaT4 16GB was used to train the model. The hyperparameters of the InceptionResNet V2 model were configured as follows: i) sigmoid activation function, ii) binary cross-entropy loss (BCE), iii) Adam optimizer, iv) learning rate η_adam = 0.0005, and v) batch size T_L = 16. The hyperparameters of the LSTM model were defined as follows: i) sigmoid activation function, ii) binary cross-entropy loss (BCE), iii) Nadam optimizer, iv) learning rate η_nadam = 0.005, and v) batch size T_L = 16.

The study developed a deep neural network to classify PE in angiography images, achieving 93% accuracy at the slice level and 77% at the exam level. Validation on hospital data showed 86% accuracy for positive PE cases and 69% for negative cases. The model excelled in excluding PE, with 73% precision and 82% recall, demonstrating its clinical applicability. The demographic diversity of the validation set reinforced the model’s generalizability, indicating its potential to improve diagnosis and patient outcomes.

Performance and security metrics protocol for ISMIR

Performance metrics are utilized at various stages of the ISMIR workflow, including: i) system availability, which measures the time the system is operational, ii) response time, which assesses the speed of response to requests, iii) storage and management capacity for large volumes of medical images, iv) data transfer rate, which measures the speed of data transfer, v) efficiency in image retrieval, vi) security and compliance with regulations such as HIPAA and GDPR, and vii) user satisfaction. Other metrics include the number of exams annotated by radiologists with and without AI and the maximum number of exams processed by AI per day.

Performance analyses were conducted for system integration using three types of exams: tuberculosis, melanoma, and neurological, with 1,690 exams of each type. Among the tuberculosis exams, 426 were analyzed by AI models, for melanoma, 141, and for neurological exams, 143. The remaining exams were returned for manual processing or data correction, as illustrated in the flow of Fig 1. Common errors detected include corrupted FHIR arriving at the dispatcher, lack of AI for FHIR requiring manual processing, presence of AI and FHIR but with model errors, AI inference failures, and model persistence failures of AI inferences. These issues were addressed by dividing the system flow into two parts: i) the main flow and ii) the alternative or exception flow.

The main flow is followed when all data is correct, the alternative flow occurs when there is no AI model for the exam, and the exception flow is used for unexpected subsystem failures. Errors and failures are restored in real-time without system downtime. Performance analyses used the same exams: tuberculosis, melanoma, and neurological, with 1,690 exams each. The average data transfer times are 20s for tuberculosis, 59s for melanoma, and 2m05s for neurological. Tuberculosis exams are single black-and-white images, melanoma exams are single color images, and neurological exams are three-dimensional black-and-white images.

Regarding data storage in ISMIR, the expected load is 16,000 exams per day. In the worst-case scenario, considering volumetric neurological DICOM exams of approximately 1GB each, this equates to 16TB per day for SUS. This was the test conducted for data storage statistics. For expected data input growth, ISMIR was stress-tested by inserting approximately 40,000 exams in a single day without failures. Security measures for confidential data transmission include: encryption in transit (SSL/TLS), encryption at rest in VNA, MongoDB, and S3/MinIO, multifactor authentication, role-based access control (RBAC), audit logs, continuous monitoring, secure access tokens, rate limiting, network segmentation, firewalls, updates and patches, secure configurations, sandboxing, and strict anonymization process control policies.

To avoid performance errors in diagnosis, extensive validation tests were conducted using a comprehensive dataset to ensure the system’s proper functioning. Cross-validation was performed by comparing diagnoses produced by radiologists with the system’s automatic segmentations. Training, retraining, and updating of models are carried out periodically with new data and techniques to improve segmentation accuracy. Quality monitoring occurs in real time to identify error patterns and notify administrators of significant deviations. In critical cases or when uncertainty arises, manual segmentation review is established. Double reading, where multiple radiologists review automatic segmentations, is performed to avoid biases.

Specific tests are conducted, including system availability and reliability tests to verify the Gateway and VNA uptime. Response time and latency tests are applied to the anonymizer, standardizer, VNA response, MongoDB and S3/MinIO, dispatcher, and DicomWeb. Network performance and data transfer tests evaluate transfer rates and FHIR compliance. Processing and modeling tests measure model management efficiency, AI model execution, accuracy, and adequacy. Display and result tests verify image display and predictive results. Failure and recovery tests include recovery mechanisms and failure notifications. Integration tests ensure compliance with international medical integration protocols.

Regarding the impact of the system on healthcare cost reduction, several tuberculosis exams were analyzed by AI and radiologists, including conditions such as atelectasis, opacity, edema, pleural effusion, mediastinal widening, cardiomegaly, pneumothorax, consolidation, and lung lesions. Kazemzadeh et al. [76] demonstrated that AI identification of tuberculosis for NAAT confirmation reduced specialist time by 40% to 80% per patient. Viney et al. [77] reported that a two-view chest X-ray at the Centers for Disease Control and Prevention, Atlanta, costs approximately $34. The Brazilian labeled chest X-ray dataset (BRAX) [69] contains 24,959 chest X-ray studies, totaling 40,967 images verified and anonymized by trained radiologists. Without AI, costs would be $848,606. With AI segmenting the exams and subsequent radiologist analysis, there is an average reduction of 60%, totaling $339,442, saving approximately $509,163. These values are modest, considered below the market average.

The need to develop AI models is based on the priority queue of SUS. This queue prioritizes people with physical disabilities, seniors aged 65 or older, pregnant women, lactating women, and individuals with young children, as mandated by law. In the PACS, as the number of exams increases, bottlenecks such as the need for reports, exam analysis, and annotations arise. Therefore, models are created according to SUS priorities and these bottlenecks. The protocol ensures that AI models initially present positive results for diseases and are then refined for segmentations and other tasks. Once the AI model is built, tested, and validated, it is put into production. Periodically, AI models are retrained with new patient data for better generalization. Currently, this retraining process occurs manually since most models are supervised and require targets.