Introduction

The rate of medical imaging data availability for research is ever increasing as we continue to integrate technology into the healthcare system [1–4]. This influx of imaging data, especially magnetic resonance imaging (MRI) in the field of neuroimaging [5–9], allows us to perform large-scale data analyses important for drawing generalizable and reproducible conclusions about human health [10–13] or more robustly training deep learning algorithms [14]. However, researchers often do not run analyses on these unprocessed image data directly; rather, image processing pipelines are used to extract derived quantitative metrics for subsequent scientific investigations [15]. The scope of derived data can range from first-order features such as total brain volume [16] to complex 3D representations of white matter brain pathways [17]. Unfortunately, not all these available derived data are suitable for use in scientific research, as the algorithmic outputs may not exactly match the features these pipelines are attempting to capture for any given input scan [18]. Implementing robust quality control (QC) protocols can help ensure the quality and consistency of data used for research, facilitating more accurate and reproducible scientific breakthroughs [19] and better machine learning models [20].

There are several considerations for ensuring effective QC of large-scale datasets. First, as neuroimaging datatypes, such as diffusion-weighted imaging (DWI), have multiple preprocessing steps before arriving at the desired downstream outputs, there are several steps that must be checked for quality. As a result, it quickly becomes infeasible for a single researcher to perform all QC tasks as more data are introduced; thus, QC can become more achievable when it is partitioned to members of the team that process the data. However, there are additional considerations for a team-driven effort, such as the manner in which QC for a pipeline is performed. There are a multitude of freely downloadable image viewers available for researchers, and while most provide the same general functionality, each is usually specialized for a specific data type. For example, FSLeyes [21] is specialized for viewing timeseries data and outputs from FSL [22] processing, whereas MI-brain [23] is built for visualizing tractography, but both are capable of viewing 4D imaging data in similar ways with differences in presentation to the user and the graphical user interface (GUI). Moreover, such types of image navigation with viewers for QC are completely unconstrained, further increasing the variability in the QC process, and each individual’s own personal method of image navigation is difficult to communicate to others and reproduce. Any such discrepancies in any aspect of the QC process can greatly increase the amount of time spent aggregating results across a team or can result in data being assessed with different standards. Thus, the remaining time available to perform analyses on the data can be greatly impacted and can even result in data of different quality standards being introduced in the dataset, which can impact results and conclusions from scientific studies [19,24–26]. One innovative method for multi-user QC is an approach like the MRIQC Web-API [27], which aims to crowdsource quality ratings of data submitted by users of the tool. However, crowdsourcing can be problematic, as it relies on a diverse set of raters who may have varying levels of expertise and differing opinions, leading to inconsistent results for contested data. Additionally, for in-house pipelines that are not widely used by researchers, such crowdsourcing methods are not a viable option due to the limited pool of contributors.

Further challenges related to the aspect of team QC are how the QC status of data is being reported and the standards by which researchers assess poor or good quality. While a more complex QC method that assesses multiple dimensions of quality or indicates presence of specific artifacts can provide more insight into the data [28–30], such an assessment would greatly increase the amount of time spent performing QC. For example, a metric of quality such as “accuracy” as defined by Kindling et al., Goasdoue et al., and Taleb et al. [29,31,28], could be used in a deeper dive for what makes certain qualities of data more susceptible to algorithm failure. However, the foremost goal of pipeline QC when handling large-scale MRI data should not be complex judgement of data quality, especially for image processing researchers whose area of expertise does not pertain to MRI scanner systems and software that collect data. No image processing algorithm is perfection incarnate, especially not deep learning-based processing pipelines [18]. Thus, in QC of this scale, the most important aspect is to assess if the algorithm performed as expected or if it failed to produce a believable output.

To that end, another critical concern is the amount of time spent actually performing QC on the data. Though QC is essential for any research study, it is beneficial to optimize the amount of time spent on the process. As mentioned above, there is a panoply of visualization tools available for looking at data. However, opening individual files in these image viewers is far too time-consuming. Time minimization of QC is a well-researched area in imaging informatics. MRIQC and other works have focused on fully automated or semi-automated outlier detection using image-derived metrics [9,32,33]. Semi-automated methods such as MRIQC [32] and ExploreQC [9] provide interactive reports of the image-derived metrics for datasets of minimally processed or unprocessed MRI, with the functionality to select individual outlier scans to perform additional manual QC through a visual report. Fully automated QC methods, such as the method described in Alfaro-Almagro et al. for UK Biobank [33] and the automated classifier released alongside MRIQC [32], use image-derived metrics to exclude images without any visualization. There have also been efforts to automate QC of outputs from processing pipelines such as segmentation algorithms [34,35]. While such methods and tools can considerably cut down the time spent on QC, they are techniques for batch QC of data reliant on quantitative summary metrics rather than QC of individual images and outputs. These batch QC methods cannot consistently capture when an algorithm breaks or fails because they rely on summary metrics of the data that could be unrepresentative of the true data quality: one cannot correctly represent quality of a high-dimensional signal such as a 3D MRI using a single, derived quantitative value. Additionally, an algorithm can work as expected, but still flag data as an outlier due to the nature of the original data, even when the quality is good. Although these “false negatives”, or good quality data flagged as problematic, can be caught with semi-automated approaches to QC, “false positives”, or poor quality data that are not flagged, can still slip through the cracks.

Thus, proper QC requires visualization of every image or data output to ensure that the algorithm performed as expected while remaining efficient with regard to the amount of time spent. In addition to the research on reducing the time spent on manual QC, there exist many tools that are built to provide ease of image inspection so that every output can be visualized. Benhajali et. al proposed a method for quick QC of image registration that involved visualizing each result but was limited only to image registration pipelines [36]. The FSL tool slicesdir [22] allows for bulk QC by creating an HTML with orthogonal slices of several NIFTI files for visual inspection but cannot overlay processing such as segmentations. Moreover, slicesdir does not provide functionality for nicely aggregated QC results. snaprate, a QC tool built to work with XNAT databases, provides more flexibility for QC of pipelines, but does not allow for separation or aggregation of QC results across pipelines [37]. The comprehensive visual reports provided by tools such as MRIQC [32] and fMRIPrep [38] provide a great level of depth in visualizing image data and explaining the steps of their respective workflows; however, such extensive reports are time-inefficient when QC must be performed on a larger-scale and are limited to specific processing and modalities. Some pipelines, such as PreQual [39], already create QC documents designed to assess the quality of specific outputs. However, most times these pre-existing QC documents, like fMRIPrep and MRIQC, are not in a form that is optimized for high throughput quality checks.

To address the above concerns (Fig 1), we propose that an effective QC process for team-driven, large-scale neuroimaging datasets should meet the following design criteria: 1.) a consistent way to visualize the outputs of each image processing pipeline that is run on the data, 2.) a method for quick visualization and QC assessment in order to minimize the amount of time spent on the QC process without compromising the integrity of the QC process, and 3.) a manner by which QC results from different researchers can be easily and seamlessly aggregated across datasets and pipelines to provide a comprehensive QC of the entire database. Addressing these criteria in a QC pipeline can help ensure brevity and completeness in QC while providing a coherency of global QC across all datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Issues with automatic and team-based QC.

When maintaining large neuroimaging datasets with multiple processing pipelines, shallow quality control processes that rely on derived metrics can fail to catch instances of algorithmic failures. However, deep QC processes quickly become unscalable and inefficient as the amount of data available increases due to the required time for mass visualization of outputs. For example, opening 50,000 T1w images separately in an image viewer for deep QC can take over 60 hours if it takes five seconds to load images in and out of the viewer. Team driven efforts to alleviate such large time costs come with additional challenges due to inconsistencies in reporting and methods of performing QC.

https://doi.org/10.1371/journal.pone.0327388.g001

We propose a QC system for a database consisting of national-scale diffusion-weighted imaging (DWI) and T1-weighted (T1w) MRI that adheres to the aforementioned design criteria. Our database consists of 20 large-scale datasets with 16 distinct preprocessing and postprocessing pipelines. We visualize all outputs in the portable network graphics (PNG) format. For viewing the PNGs and assessing quality, we build an application using a Flask (https://github.com/pallets/flask) server that allows researchers to quickly look through PNGs and easily document when data processing has failed (Fig 2). The documentation method also allows us to easily aggregate QC assessments across pipelines and datasets in a CSV format that can easily be shared with collaborators or other team members. Further, we compare our pipeline with an existing method for QC of T1w MRI, mriqc-learn, and assess the variability of raters for several different processing pipelines using four different datasets.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Proposed pipeline for efficient manual QC.

We propose an efficient and scalable method for standardized deep QC of neuroimaging pipeline outputs that works within a team setting. For each pipeline output, we provide a visualization as a PNG file. The PNGs are then loaded into a QC app that permits quick scanning of outputs for abnormalities and failures. Any team member can start an instance of the app, and results are standardized in a structured CSV file for each pipeline.

https://doi.org/10.1371/journal.pone.0327388.g002

Methods

Upon receiving MRI data, we convert from the Digital Imaging and Communications in Medicine (DICOM) [40] to Neuroimaging Informatics Technology Initiative (NIFTI) (https://nifti.nimh.nih.gov/nifti-1) format when data are provided as DICOM files; no conversion is required if data are only provided in the NIFTI format. For an initial rapid filtering of data, we initially ignore any DWI that do not have corresponding BVAL and BVEC files that indicate the direction and strength of the diffusion-weighting, asking data providers for missing files if possible. We ignore DWI with fewer than 6 volumes unless it is a reverse-phase encoding scan for susceptibility distortion correction that accompanies a more highly sampled DWI. We also visualize raw T1w and DWI in batches using nibabel [41] to load images and matplotlib [42] to display and output them as PNGs to ensure that scans are the correct modality. We organize raw and derived data according to the Brain Imaging Data Structure (BIDS) (v.1.9.0) format as mentioned in our previous work [43].

Data processing pipelines

For processing raw MRI scans, we run 16 different pipelines in total across T1w and DWI modalities (Table 1). The pipelines have a variety of purposes, including preprocessing, signal modeling, image registration, segmentation, tractography, connectomics, and surface reconstruction. There exists a dependency chain for the pipelines, with many also requiring both T1w and DWI modalities (Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Pipeline list. A list of all the processing pipelines that are run on our datasets.

https://doi.org/10.1371/journal.pone.0327388.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Processing pipeline dependency chain.

The 16 pipelines we run on our data form a complex dependency chain, and thus, it is essential to perform quality control on each step so that data are not misinterpreted during downstream analyses. Created with flourish.studio.

https://doi.org/10.1371/journal.pone.0327388.g003

Performing quality control for a dataset and pipeline

Once we have generated the PNG files for a pipeline that has been run on a dataset, we visually inspect each file using a QC app running on a Python backend with Flask (Fig 4). Each researcher can start their own instance of the QC app and select the dataset and pipeline that they wish to perform QC on. PNG files are then pulled from the QC database and loaded in the QC app for the user to view. Whenever a failed pipeline is detected, the user can document that the output is a failure, and the results will be pushed in real-time to a database of the QC results for that pipeline (Fig 5). The source code for the app can be found here, in a publicly available repository: (https://github.com/MASILab/ADSP_AutoQA.git). We also make a BIDS-agnostic version available to download as well for researchers who cannot adhere to the BIDS organizational standard, but still wish to perform QC using our app: (https://github.com/MASILab/GeneralQATool).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. QC app interface.

The proposed QC app enforces consistent visualization of pipelines outputs through uniformly generated QC PNG documents for each respective pipeline. The homogenous format of the documents and high success rate for pipelines allow QC users to very quickly cycle through PNGs to catch any abnormalities. Moving through PNG documents can be done either manually with the arrow keys or automatically through the montage feature of the app (not shown). A counter is maintained (red box) so the user can know how many documents are left to view. Most outputs are expected to be good, and thus all QC results are initialized as “Yes” (passes QC) in order to minimize the manual effort in reporting. If the output is not satisfactory, the user can click the corresponding button to indicate their decision (yellow box) and articulate the reason for their verdict with some accompanying text (green box). Any changes are automatically pushed to a CSV document that maintains structured information about the results of the QC for a pipeline and dataset. All QA CSV results documents are formatted the same way and can thus be easily merged across pipelines and datasets so that QC decisions can be shared among team members and collaborators in an easily parseable format.

https://doi.org/10.1371/journal.pone.0327388.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Diagram for QC workflow.

Our method allows for an efficient quality control pipeline: 1.) QC files are generated by querying the data storage from control nodes 2.) and then stored on an archive separate from the data. 3.) To perform QC, files are pulled from the QC archive 4.) and then QC results are pushed back in real time for other researchers to use.

https://doi.org/10.1371/journal.pone.0327388.g005

Results

When running MRIQC on all data, only one scan was unable to run through MRIQC without early termination. As there were no image quality metrics generated, this single scan was removed from the analysis (). Ratings given to the T1w images were mostly positive for both the MRIQC classifier and the rater using the QC app, with only a small number of rejections from both methods. The agreement between the two methods was also very high, as seen in the resulting confusion matrix (Table 2). For the purpose of brevity, we use “false negative” to indicate a scan that received a passable rating by the rater using the QC app but was rejected from the classifier, and “false positive” for a scan that was rejected by the rater but not rejected by the classifier. Only two scans were rejected from the rater whereas five were rejected by the classifier, with four of those being false negatives. When visualizing all rejected scans, the false negatives all had qualities typical of aging brains, such as large ventricles, whereas the one false positive and “true negative” (when both methods reject the scan) both had moderate to high motion artifacts (Fig 6). The recorded time for the rater using the QC app was 10 minutes and 36 seconds.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Confusion matrix for MRIQC classifier experiment. A confusion matrix indicates that agreement between the QC app and the MRIQC classifier for T1w images was high with a few exceptions. We note that one scan failed to run through MRIQC and thus could not be included in this analysis.

https://doi.org/10.1371/journal.pone.0327388.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Qualitative results for MRIQC classifier comparison experiment.

Moderate to severe motion artifacts can be seen in the scan rejected for both methods (“True Negative”) and the scan deemed acceptable by MRIQC but rejected by the reviewer using the QC app (“False Positive”). Scans that were rejected by MRIQC but passed by the reviewer (“False Negative”) had qualities typical of older brains, such as larger ventricles, despite being of acceptable quality.

https://doi.org/10.1371/journal.pone.0327388.g006

The inter-rater variability among the four was either substantial or almost perfect agreement for most pipelines. According to Landis and Koch, agreement among raters is “fair” when the Fleiss’ Kappa value is between 0.21 and 0.4, “moderate” for 0.41 to 0.6, “substantial” for 0.61 to 0.8, and “almost perfect” when greater than 0.8 [78]. The only pipeline not at least at a “substantial” level of agreement was PreQual, which had a Fleiss’ Kappa of 0.313, whereas agreement for the Lung CT Harmonization and TPOSTC right, SLFI left, and AF right tracts fell into the substantial category and SLANT-TICV, CST left, and CC4 fell into the almost perfect category (Fig 7, Fig 8, Supplementary Figure Section C: Fig SC in S1 File). We do note that the amount of information present for each pipeline specific QC PNG is different, which may affect the variability of raters. Timing information of all raters for each pipeline can be found in Supplementary Information Section B: Table ST.B1. Additionally, examples of pipeline outputs can be found in Supplementary Information Section C: Figs SC.F2 and SC.F3 in S1 File.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Inter-rater variability results for PreQual, SLANT-TICV, and the CC4 tract.

Inter-rater variability of the four raters for QC of both SLANT-TICV (top) and TractSeg tract CC4 (bottom) using the QC app showed very high agreement, with Fleiss’ Kappa values greater than 0.8. The agreement was fair for PreQual QC (middle) with a Fleiss’ Kappa in the range 0.2–0.4. Blue indicates that the rater selected “yes” for the output while yellow and red indicate selection of “maybe” and “no” respectively.

https://doi.org/10.1371/journal.pone.0327388.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Fleiss’ Kappa scores for QC inter-rater variability experiments.

Fleiss’ Kappa agreement scores for QC pipelines across four raters suggest high inter-rater variability. Different levels of agreement as presented in Landis and Koch20 are shaded in with labels to the right of the plot. SLANT-TICV, TractSeg CC4, and TractSeg CSTleft are QCed with “almost perfect” agreement, whereas TractSeg AF right, TractSeg SLFIleft, TractSeg TPOSTCright, and Lung CT Harmonization have “substantial” agreement. PreQual is the only pipeline QCed with “fair” agreement.

https://doi.org/10.1371/journal.pone.0327388.g008

Discussion

For a comprehensive QC of pipeline outputs, visual inspection of every output is paramount to ensure that no instances of failed algorithms are used in scientific analyses. As dataset size and the number of processes scale up, multiple researchers are needed to perform QC in a timely manner. We have demonstrated a proposed method of facilitating visual inspection that permits consistent and efficient QC across a team setting. The use of a unified QC app that enforces reporting of results in a specific manner promotes stability of the team QC. Additionally, the QC PNGs are generated in the same way for each individual pipeline. The expectation of what a proper pipeline output should be, coupled with the consistency in format of the output PNGs, increases efficiency by permitting rapid cycling through the QC PNG documents with the QC app. The standardized visualization of outputs can also help reduce the amount of training required for new team members who have little to no experience with QC of the pipelines. Further, our automated documentation of QC results to the QC archive and ease in aggregating QC across pipelines reduces the manual effort for compiling the efforts made across the entire team. Combining results in an easily parseable CSV format also helps to expedite research and analysis for team members who use the data and for external collaborators. In these ways, we have successfully met the design criteria for a deep QC pipeline. We have also demonstrated in our QC experiment for the Lung CT Harmonization pipeline that this QC tool can be applied to fields outside of the neuroimaging domain.

In previous work, researchers have proposed methods for visual QC of outputs. The Mindcontrol QC application developed by Keshevan et al. links visualization of outputs with quantitative metrics and manual annotation tools [79]. However, Mindcontrol is not suited for visualizing large volumes of data as it loads in entire 3D medical images in order to allow users to annotate and modify errors in the processing pipelines. The MRIQC and fMRIPrep QC tools create QC visualizations without loading in entire 3D volumes, but their visualization methods prevent quick QC as every single axial slice is rendered in the QC image [32,38]. Alternative solutions that permit rapid visualization, such as FSL’s slicesdir, do not allow flexibility in how visualizations can be structured [22]. Our QC tool is unique in that it provides a method for rapid visualization of outputs in a pipeline agnostic manner.

For our visualizations of each pipeline presented in Table 1, we build off established methods from other researchers. With regard to deep learning segmentation, Benhajali et al. posit that triple slice, tri-planar visualizations of label maps overlayed on a structural scan is best for rapid QC [36]. As segmentation of white matter regions is the ultimate task for our atlas-based registration pipelines, we have found that overlaying the registered label maps in a similar manner is effective for efficient QC of registration. In comparing their visualization method to others, Raamana et al. suggest that contour maps and comprehensive views of surface-based parcellations result in more reliable QC between raters [80]. Following their example, we ensure visualizations of the BISCUIT and FreeSurfer pipeline outputs contain these as well. For bundle-specific tractography QC, He et al. provide a semi-automated method with manual editing of erroneous outputs [81]. However, manual editing-based methods like Mindcontrol or that in He et al. are not appropriate for efficient QC of a large volume of data. Still, He et al find that overlaying white matter bundles on an FA map is appropriate for identifying erroneous outputs. Thus, we combine this concept with the tri-planar viewing method to allow for both efficient and appropriate QC of TractSeg outputs. For pipelines with larger QC documents included as part of the outputs, such as PreQual [39], Nextflow bundle analysis, and BISCUIT toolbox, we maintain as much information in the original documents as possible that permits us to perform efficient, yet effective, QC. As structural connectomes are two-dimensional datatypes whose size is limited to the number of brain regions of interest, most visualizations of connectome pipeline outputs consist of the entire connectome [56,82]. Thus, our visualization for connectomics pipelines follows this practice. Note that we have not tested any hypotheses comparing different methods of visualizations across pipelines. We also wish to highlight that, while other alternative pipelines exist, such as FSL’s dtifit instead of MRtrix3’s dwi2tensor for tensor fitting or TRACULA for bundle-specific tractography instead of and TractSeg, we have chosen the specific tools in Table 1 for our purposes. Using any alternatives may result in different outputs, even when the tools are intended to perform very similar tasks. Nevertheless, the QC app can be applicable to any pipeline as long as the output can be visually inspected.

Including data of poor quality can impact results, as shown in Ducharme et al. with regard to cortical thickness trajectories changing from non-linear to linear depending on data quality [19]. Such changes in results are likely be due to additional unexplained variances in images being introduced into the dataset, making it more difficult to capture or identify underlying trends in the data as it would require a larger sample size. The inclusion of low-quality data can pose significant challenges for machine learning and deep learning models, as these models are highly sensitive and may learn incorrect data distributions [83]. For FreeSurfer outputs, Monereo-Sánchez et al. found that the largest decrease in unexplained variance occurred when manual QC was performed as opposed to automated QC methods [84]. We hope that the tool presented here will help promote scientific discoveries and better-trained deep learning models by removing unwanted variability due to poor data quality.

Although we advocate for rapid visualization of every image and pipeline output when faced with massive datasets in order to perform a comprehensive QC process, we are not disparaging alternative tools or methods that are slower due to being more comprehensive, such as the individual visual reports generated by MRIQC. Rather, we caution researchers against using these tools in a semi-automated manner where not every output or image is visually inspected and suggest a high degree of caution for fully automated methods. Scenarios of false negatives can be easily remedied for semi-automated methods where visualization occurs only for rejected scans, but in cases of false positives poor quality data can end up in analyses. For example, in our experiment comparing the automated MRIQC classifier to the QC app, one scan out of 1560 was classified as good quality by the classifier but was poor quality when visualized (Fig 6). This low number is in part due to the high quality of the WRAP dataset, where not many scans were qualitatively problematic. Analyses of larger scale, such as the one in Bethlehem et al. [85] that used over 120,000 T1w scans, or datasets of worse overall quality could result in many more scans of bad quality being included and affecting results. Performance of automated or semi-automated classifiers could be improved by training with increased sample sizes and multiple different population distributions (such as different age ranges, patient populations, etc.).

Inter-rater variability was low, with high Fleiss’ Kappa scores for most pipelines (Fig 7, Supplementary Fig in S1 File), even for outputs such as the TPOSTC right tract from TractSeg that had a high number of problematic outputs included in the data sample used for the experiment (Supplementary Table in S1 File). These differences in number of included problematic samples, amount of information in each pipeline’s QC PNG, or the difference in total sample size for the case with PreQual output QC could have influenced the differences in agreement scores across pipelines. Despite our experiments showing mostly low inter-rater variability, there is not perfect agreement. Our results suggest that for team-based QC of pipelines on large-scale datasets, QC of each process should be consistently performed by the same team members, rather than each member conducting QC of all pipelines on a subset of the data. Additionally, we note that the experiments on inter-rater variability were not intended to compare performance across raters. Further, although the four raters for the inter-rater variability experiments all have different levels of experience when working with the datatypes used in the experiments, all come from the same research group. Thus, the variability may change for an alternative set of raters.

One of our main goals in using the PNG format to view outputs was to promote brevity and efficiency in the QC process. We acknowledge that other image formats, such as scalable vector graphics (SVG), exist and could be used as well. However, as efficiency in visualization was our main goal, the PNG format was sufficient for this approach. Furthermore, despite the increased efficiency, we do recognize that visualization with an image viewer that was built to render the datatype being examined would be a more comprehensive level of QC, as a user would have more control in inspecting any part of high-dimensional data rather than a fixed viewpoint. As the QC design criteria were to assess outputs for failure of the algorithm, not a grading of quality level, we believe the decision to draw the line for the efficiency-thoroughness tradeoff should lie closer to the efficiency end. Thus, it was important to briefly visualize the data in a way that would maximize the amount of information in order to prevent caliber of the QC from being compromised. We understand the choice is subjective for what the best ways are to represent the data, and our decisions were based on preliminary QC efforts that we have built upon for the past two years. As we continue to maintain our MRI database, we will look to integrate new and existing tools into our current processing pipeline, such as FastSurfer [86], that will also require QC procedures. We note that the QC tool presented here is agnostic to processing pipelines, whether for newly incorporated tools or those discussed in this manuscript. Specifically, our QC tool provides flexibility to users regarding the content of the images. Researchers can insert visualizations they feel are most appropriate for their QC process, as long as each QC file is in the PNG format. Thus, our tool facilitates the implementation of a standardized QC procedure, which can be customized by each research group to optimize an efficient QC process for large volumes of data. For the rare cases when there is still ambiguity in whether or not the algorithm failed, we do advocate visualization in a proper image viewer to better understand if the output is a successful result. We are not trying to discern why the algorithm may have failed in our pipeline, but if users wish to understand the algorithmic shortcomings, such visualization might help elucidate the issues.

We also note that our QC process is mostly focused on qualitative visualization of outputs rather than quantitative visualization. This is because quantitative visualization on an individual/session level would greatly reduce the efficiency of performing QC, as visualization of images is quicker to investigate than a series of numbers and is often more effective. For instance, a tractogram is a collection of lines in 3D space, where each line (or streamline) is composed of several coordinates. To quantitatively read the coordinates for millions of streamlines is an infeasible task. Representing the tractogram as a condensed quantitative metric would make the quantitative QC process easier, but then one would be relying on a derived measure instead of visualizing the data itself. As we continue performing QC on our data, we will consider including batch-level quantitative methods of QC on derived metrics, such non-parametric statistical methods like interquartile range (IQR) and median absolute deviation (MAD) for outlier detection [87], in addition to our current pipeline in order to supplement the process. Such additions will be essential for pipelines like the white matter brain age that output only a single value.

Finally, in our approach, we are focused on identification of failed data processing. We do not attempt to correct data beyond any preprocessing that is common in the field for the imaging modality. We also are not trying to perform any iterative algorithmic fine-tuning to produce slightly higher quality outputs (for example, retraining a segmentation algorithm on more data and viewing the outputs for each new model). We suggest researchers who are interested in such algorithmic optimization to use a different method for QC of data, as our method is not intended for grading quality of outputs. We also do not attempt to assess how the quality of outputs from one pipeline influences the performance of a subsequent downstream analysis. Although we opted to use the “yes/maybe/no” categorization for QC ratings, there exist alternative categorizations that could be used instead, such as the one used for the MRIQC Web-API [27] or a Likert scale, which measures a range of ratings and is a frequently used option for grading MRI scan quality [88,89]. Such a rating scale can be beneficial because it allows for image quality to be used as a covariate for downstream analyses [88–90]. However, a recent study from Hoeijmakers et al. found that the rating scales in QC result in decreased reliability and inter-rater agreement when compared to a pairwise comparison (PC) quality assessment, suggesting that the greater availability of rating options reduces consistency [91]. Tang et al. similarly find that PC-like methods result in more reliable scores, even among board certified radiologists, and also observe a significantly faster QC time when compared to rating with a Likert scale [92]. Both Hoejimakers et al. and Tang et al. posit that rating scales increase the amount of subjectivity in QC, even despite having a defined rubric for scale ratings as in Tang et al. As our goal is to decrease subjectivity while increasing efficiency and consistency, we feel having concrete “yes/no” options to indicate processing pipeline failure is an optimal choice to satisfy the design criteria. Further, the “maybe” option allows for more flexibility in cases where a decision is not clear, with the text box provided in the QC app allowing raters to further elaborate on their decisions if necessary. Finally, we note our method is not defined as PC as in Hoejimakers et al. and Tang et al. Such a QC method results in performing QC ratings for outputs as opposed to ratings when assessing each output individually, which can greatly increase the amount of time performing QC for large datasets as we do. However, the QC app allows raters to easily and quickly return to previously rated PNGs should they need comparison data similar to a PC-based method. In this way, the QC app has the benefits of PC-based methods without incurring the additional overhead in multiple comparisons.

Supporting information

Acknowledgments

We use generative AI to create code segments based on task descriptions, as well as debug, edit, and autocomplete code. Additionally, generative AI technologies have been employed to assist in structuring sentences and performing grammatical checks. It is imperative to highlight that the conceptualization, ideation, and all prompts provided to the AI originate entirely from the authors’ creative and intellectual efforts. We take accountability for the review of all content generated by AI in this work.

Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf.

HABS-HD PIs: Sid E O’Bryant (main: Sid.OBryant@unthsc.edu), Kristine Yaffe, Arthur Toga, Robert Rissman, & Leigh Johnson; and the HABS-HD Investigators: Meredith Braskie, Kevin King, James R Hall, Melissa Petersen, Raymond Palmer, Robert Barber, Yonggang Shi, Fan Zhang, Rajesh Nandy, Roderick McColl, David Mason, Bradley Christian, Nicole Phillips, Stephanie Large, Joe Lee, Badri Vardarajan, Monica Rivera Mindt, Amrita Cheema, Lisa Barnes, Mark Mapstone, Annie Cohen, Amy Kind, Ozioma Okonkwo, Raul Vintimilla, Zhengyang Zhou, Michael Donohue, Rema Raman, Matthew Borzage, Michelle Mielke, Beau Ances, Ganesh Babulal, Jorge Llibre-Guerra, Carl Hill and Rocky Vig. A list of ADNI PIs and contact information is available at: https://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf.