1.1 COVID-19 diagnosis

In the fight against the COVID-19 pandemic, enormous effort was dedicated to employ Deep Learning models for more accurate medical image analysis for both X-ray and CT images. With low radiation and low cost, chest X-ray has been widely leveraged to provide initial assessment of COVID-19 infections. Therefore, many models, specifically deep 2D CNNs, have been developed to classify and detect COVID-19 infections from chest X-ray images [9–12].

Chest CT, on the other hand, is a key part of the diagnostic process for suspected COVID-19 infections [13]. Previous research efforts on COVID-19 diagnosis from chest CT images have approached the task as either classification [7,14–16] or segmentation [17–19]. The classification task is employed to either distinguish between normal, pneumonia, and COVID-19 cases or to determine the severity of COVID-19 infection. Segmentation, on the other hand, is focused on infection quantification within the lungs. Ardakani et al. [14] conducted a comparison between ten common 2D deep CNN architectures and achieved the best performance using ResNet-101 and Xception models. Similarly, Gozes et al. [15] used an ImageNet pretrained ResNet-50 model to classify positive COVID-19 cases based on 2D image slices from input CT scans. Furthermore, Ning et al. [7] developed a dual Deep CNN architecture with a simplified VGG-16 model to first classify positive for COVID-19 CT slices, which are then fed into a secondary VGG-16 model to determine the severity of the case. More recently, Mondal et al. [16] proposed a Vision Transformer (ViT) classification model to screen COVID-19 from chest CT and X-ray images. Slice-based approaches ignore important 3D spatial relationships and are not inherently designed for explainability, while ViT models require a large amount of training data that is difficult to acquire in the medical domain. In contrast, 3D convolutional models allow for 3D spatial features to be considered during classification. In such manner, Xu et al. [20] trained a 3D DenseNet model to classify the CT volumes into four classes: healthy, bacterial pneumonia, COVID-19, or other viral pneumonia.

1.2 Explainability

Explainability is crucial for diagnostic methods to be considered for clinical adoption [21]. As a result, some works have focused on explainability using gradient-based approaches from classification CNNs [15] or from ViT models [16]. On the other hand, segmentation models are explainability-focused, where the model is trained to segment and quantify infection in the lungs. However, manual delineation of infection regions in CT images is a tedious task, labour-consuming, expensive, and challenging due to the highly varied textures, sizes, and shapes. Wu et al. [18] collected a dataset of 3,855 CT image slices from 200 patients and annotated fine-grained pixel-level labels of lung opacifications. They used this dataset to jointly train classification and segmentation models to classify the image and segment lung infection. Zhou et al. [22] approached explainability from a different perspective. They proposed an ensemble of Deep Learning-based segmentation models to exclude tissue irrelevant to the pulmonary parenchyma from the CT volume. Then, following a human-in-the-loop training scheme, they extract interpretable subvisual CT lesions. Formulating the task as an infection detection, one can slightly reduce the annotation workload. For example, Ji et al. [23] proposed a single-stage model for automatic lung tumor detection in CT images, but the need for expert-level detection labels is still present.

1.3 Weak supervision

In 2D natural scene images, Zhou et al. [24] explored weak-supervision for instance segmentation with image-level labels. They exploit class peak responses in CNN models to enable instance mask extraction. In a similar manner, Ou et al. [25] leverage local peaks in image-level attention along with relationships between them to infer pseudo labels. For COVID-19 diagnosis, tackling annotation scarcity for infection regions, Han et al. [8] proposed a weakly-supervised attention-based multiple instance learning (MIL) approach for COVID-19 screening from chest CT volumes. This involves generating 3D instances using a deep instance generator, which are then passed to an attention-based MIL pooling to combine instances into a single representation. Converting the 3D CT scan into a bag of instances allows for the attention-based mechanism to provide insight into the contribution of each instance to the bag class label, and thus providing an interpretable output.

2.1 Proposal generator model

The proposal generator is composed of two components: (a) A cuboid nominator, which is a 3D CNN model that carries out the cuboid nomination. (b) A cuboid classifier, which is a small 3D CNN model tasked with cuboid-level classification.

2.2 Volume-level classification

A Multi-Layer Perception (MLP) network is used to take the combined cuboid representations from each volume V and provide an overall classification for the infection severity. Inspired by work done recently in vision Transformer models [26], we utilise 1D positional embedding for each cuboid to ensure that the MLP takes the position of the cuboid into consideration to learn inter-cuboid correspondence. The sum of each cuboid’s positional embedding vector with its representation is concatenated with the volume-level representation and used to classify V. Furthermore, the learned embedding from the proposal generator is also passed through the MLP to provide the model with global features from V.

2.3 Weak supervision

To train the cuboid nominator model, we utilize the outputs of the cuboid-level and volume-level classifiers. As such, we use the following Binary cross-entropy function to train the Cuboid Nominator:

(1)

where and x_n is the probability of the n^th cuboid being useful as predicted by the cuboid nominator, while y_n is determined using the following weakly supervised criteria:

(2)

where and are the confidence increase and decrease coefficients at epoch i, respectively. This is built on the intuition that cuboids leading to a correct classification are likely to contain relevant visual information, which is a region of infection in the cases of positive infection cases. Conversely, cuboids that lead to the wrong overall classification are less likely to contain relevant information. Intuitively, the criteria should also decrease the probability of cuboids that are not selected when the volume is classified correctly. However, due to the large number of cuboids that fall under this category, this leads to a deflation of the total cuboid probability scores. The two middle cases in (2) allow for a transition between the two ends of the confidence spectrum. In such manner, the confidence coefficients are given by the following formulae:

(3)

(4)

where and τ is an initial temperature value that controls the impact of the coefficients as the training progresses.

2.4 Infection clusters

Lung infections often cluster in particular areas of the lungs and include various types such as Viral Pneumonia (including COVID-19 and Influenza), Bacterial Pneumonia, Fungal Pneumonia, and Tuberculosis. In Viral Pneumonia cases, bilateral ground-glass opacities commonly appear in a clustered distribution, typically located subpleurally and predominantly at the base of the lungs. To deal with this, we propose a further step for cuboids to impact each other’s probability of being picked by the cuboid nominator, as depicted in Fig 2. For this, prior to applying the weak supervision policy in (2), we adjust y_n as follows:

(5)

where K is the number of cuboids in volume V and is given by:

(6)

where ω is a confidence threshold to limit the cuboids’ impact after a certain limit, while is the normalized distance between cuboids n and j as given by:

(7)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. An illustration of the infection clusters impact from two cuboids c_f and c_l on cuboid c_n along the z axis (Left).

The relationship between δ and ε for a given probability x = 0.8 along with the impact of c_f and c_l (Right).

https://doi.org/10.1371/journal.pone.0334431.g002

To ensure that the cuboid does not have an impact on itself, we set . Fig 2 illustrates an example of the infection clusters comparing the impact of two cuboids at varying distances from a given cuboid c_n. For an infection like Miliary Pneumonia, which presents in small pulmonary nodules scattered across the lungs, we can set ω to a very small value to limit the cuboid’s impact and maintain a local impact.

3.1 Datasets

We use two datasets, HUST [7] and MosMed [27], collected from China and Russia, respectively. This is to evaluate our proposed method on diverse data and show its ability to perform well compared to existing methods. We match the infection severity labels between the two datasets. The HUST dataset followed the guidance of the Chinese National Health Commission [28] in labelling the COVID-19 infection severities, which employs a combination of clinical features and CT observations. Similarly, MosMed uses a combination of clinical features along with visual features from the patient’s CT scan to determine the severity as summarized in [27]. In both datasets, each patient is diagnosed as one of five classes: control, mild, medium, severe, and critical. With slight variations between the class-determining criteria in terms of clinical features, the CT visual representation is more consistent, where a mild case is determined by little to no signs of pneumonia in the CT scan. A medium case is determined by some signs of pneumonia ( coverage as defined in MosMed), while a severe case would have signs of more than 50% pulmonary parenchymal involvement. Finally, a critical case is determined when there is respiratory failure, shock, and more than 75% coverage.

We use chest CT images from a total of 2,631 patients, 1,345 are part of the HUST dataset collected from two hospitals in China. The remainder 1,110 patients are part of the MosMed dataset collected from municipal hospitals in Moscow, Russia. The average age of patients in the HUST dataset is 56.6 years, with an average 51.1 years in the control class [7]. There is a subtle difference between the average age for different severities, with older age groups being more susceptible to develop more severe infections. As for the MosMed dataset, the median age is 47 years, with ages ranging from 18 to 97 years. The HUST dataset contains 249 control, 222 mild, 434 moderate, 131 severe, and 27 critical cases, while the remaining suspected but unconfirmed cases. The MosMed dataset contains 254 control, 684 mild, 125 moderate, 45 severe, and 2 critical cases. Due to the class imbalance challenge between the control, mild, medium, severe and critical samples, Ning et al. [7] combined mild and medium cases as Type I class and severe and critically ill cases as Type II class.

We follow four strategies in splitting the datasets: (a) In a similar approach to Ning et al. [7], for the HUST dataset, we combine the mild and medium cases to form a single class (Type I) and we combine the severe and critical cases to form another class (Type II), while the control class remains independent. In this split, we use the same cross-site approach to split the training and testing data, with the training set being from one hospital and the test set from another hospital. (b) We follow the same approach in merging the classes from the MosMed dataset into 3 classes, while splitting the MosMed dataset randomly into 80% and 20% splits for training and testing, respectively. (c) We combine the HUST and MosMed datasets by aligning their classes and we also use the 3-class approach. We conduct a 5-fold cross-validation split for this with 80% for training and 20% for testing. (d) The MosMed and HUST datasets are combined while keeping the 5 severity classes, where we also conduct 5-fold cross-validation with 80% training and 20% testing splits from both MosMed and HUST datasets. In all settings, we omit the suspected class from the HUST dataset, because they do not have a definitive laboratory confirmation of COVID-19 at the time of enrolment. Both splits are carried out at patient-level, with a single scan per patient, without any follow-up scans. Furthermore, patients from both datasets are enrolled from different hospitals, and from different countries in the case of HUST and MosMed datasets, with minimal likelihood of double patient enrolment between datasets.

A subset of 50 CT scans belonging to mild cases from the MosMed dataset [27] has been annotated by experts of the Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Healthcare Department. The annotators marked ground-glass opacifications and regions of consolidation in each of the 50 scans. This subset is used to evaluate the explainability of the model by measuring infection coverage.

3.2 Image preprocessing

The CT images from the two datasets are resampled to a unified isotropic resolution with 1mm × 1mm × 1mm voxel size. All images are also resized to 64 × 128 × 128 voxels using first-order spline interpolation, while the Hounsfield units of every voxel are clipped to values between –1250 and 200 and then normalized. To enable better generalization and to prevent overfitting, we apply on-the-fly volume augmentation with random scaling between 0.9 and 1.1. We finally apply a random lateral flip with a probability of 0.5.

3.3 Experiments

We implement an end-to-end training approach for the three components of the pipeline, the cuboid nominator, the cuboid-level classifier and the volume-level classifier. While the cuboid nominator is trained to reduce the Binary Cross Entropy loss in (1), the cuboid-level and volume-level classifiers are both trained using the Cross Entropy (CE) criterion, where soft labels for the cuboids are deduced from the volume’s label. Regardless of the number of classes for the volume, we observe that training the cuboid-level classifier using binary labels (infected or not) achieves better overall performance. Furthermore, weights are applied to the CE criterion at the volume-level to deal with the class imbalance problem. This is applied to the 3-class and 5-class tasks. Stratified sampling is also used for each batch of volumes to better deal with the class imbalance challenges.

We empirically chose the 3D-DenseNet-121 model [29] as the cuboid nominator with a penultimate fully-connected layer of size 1024, used as the volume-level representation. We choose a cuboid size of 8 × 16 × 16 as it is small enough to provide a good localisation precision but also large enough to contain significant features of infection to be classified. A shallow 3D CNN model is used for the cuboid-level classification, with two 3D convolutional layers of () and () kernel sizes along with a stride of (1,2,2) for each layer. Following each convolutional layer, we use a max pooling layer with a window size of (). Then, three fully-connected layers (2048, 1024 and 128 neurons) are used before the model ends with an output size of 2 to determine whether the cuboid is infected or not. The output of the penultimate layer with 128 neurons is considered cuboid-level representation. We then use an MLP model for the volume-level classification, where the 128 cuboid representation is added to the positional embeddings for the selected cuboids. The result is then concatenated with the 1024 volume representation from the cuboid nominator and passed through two fully connected layers of 512 and 128, ending with an output size equivalent to the number of classes in the task. The cuboid-level and volume-level classifiers’ design ensures minimal added computational complexity over the vanilla 3D-DenseNet-121 model, with less than 0.1% of added floating point operations per second at inference time.

All models in the pipeline are trained using the ADAM optimizer [30] with 0.9 and 0.999 values for and respectively. All models are trained with a starting learning rate of 0.001 and a reduce-on-plateau learning rate scheduler for each model to reduce the learning rate by a factor of 0.1 after 25 epochs without reduction in the respective model’s training loss, with a lower bound of . Each experiment was run on a single NVIDIA V100 GPU.

5.1 Explainability

Explainability is an integral part of diagnosis methods, which could enable them to be considered for adoption in clinical practice [21]. The explainability of our approach stems from the weakly-supervised cuboid nominator, which is learning to identify “useful” cuboids for the classification task. The resulting cuboids can provide an idea of where the infection is located within the volume as demonstrated in Fig 4. To quantify explainability, we measure the cuboids’ coverage rate C for the infected area using the subset of labelled volumes provided with the MosMed test data. We define C as the percentage of infection covered by the proposed cuboids to the overall infection in the volume. It is important to note that the aim is to achieve a coverage level that provides an explainable result, but does not necessarily compete with the performance of a supervised infection segmentation model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Explainability visualization as a result of highlighting the contents of the selected cuboids (red) compared to manually annotated infection mask (green).

(a) shows two validation samples with the highest infection coverage score while (b) shows the two validation samples with the lowest infection coverage score.

https://doi.org/10.1371/journal.pone.0334431.g004

Fig 5 illustrates the increase in the infection coverage for training and validation subsets, demonstrating that the cuboid nominator is learning to pick cuboids containing infection. Our approach can achieve an infection coverage of around 65% on unseen data (Fig 5) when setting k number of cuboids to be proposed by the cuboid nominator to 64 out of 512 possible cuboids. This infection coverage rate along with the visualizations in Fig 4 highlight the significance of our approach towards clinical adoption. Our method can provide clinicians with an infection severity classification along with areas to focus on in the CT scan, thus accelerating the diagnosis process and enabling wider adoption when compared with black-box methods. All annotated CT volumes used for validation are from the subset of annotated CT scans from the MosMed dataset. All annotated samples belong to the mild infection class, which has small lesion coverage making it more challenging.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The impact of varying the k number of nominated cuboids on: (a) the training and validation infection coverage rate during the training process.

(b) The validation F₁ measure, with the size of the dots indicating how early in the training process, the best F₁ measure was achieved.

https://doi.org/10.1371/journal.pone.0334431.g005

In Fig 5, we analyse the impact of varying the k number of selected cuboids on the F₁ measure and the infection coverage. Intuitively, we observe an increase in infection coverage as we increase k, until we reach a saturation level at k = 64, where we also achieve a maximum F₁ score. We also observe an unstable infection coverage score and F₁ measure with k = 80, caused by the large number of cuboids shifting within the volume, especially at the beginning of training. As for determining the cuboid dimensions; a smaller cuboid size can provide higher localisation resolution, but a very small size would make it challenging for the cuboid-level classifier to determine whether the cuboid contains infection or not. We experimentally tested varying the cuboid dimensions, through which we noticed that the cuboid proximity factor allows us to use small cuboid dimensions to achieve higher infection coverage while maintaining classification performance.

5.2 Ablation study

5.2.1 Combined datasets.

In Table 2, we test our method using a five-fold cross-validation split of our combination of the HUST [7] and MosMed [27] datasets. The prediction for the combination of the HUST and MosMed datasets shows a compromise between the two individual dataset predictions. The increase in the number of samples leads to increased performance for all methods when combining the two datasets, when it comes to the three-class setting. We also test on the five-class severity setting, which is significantly more imbalanced than the three-class setting. The impact of this is clear, with significantly lower F₁ measures reported from all methods compared to the three-class setting. This is likely due to the increased task complexity in learning the low and high-level features from two datasets with aligned infection severity classes. In Fig 7, we observe the highest confusion in severe cases that are misclassified as either medium or critical cases, with a tendency to go for the less severe ones. Even though zero control and mild cases were classified as critical, there were some critical cases misclassified as control, which is likely caused by the class imbalance, where the critical class has a minimal number of samples in both datasets. Furthermore, even though the five-class classification task proved to be more challenging and resulted in lower F₁ score, our cuboid nominator is still able to achieve similar infection coverage to that of the three class classification task (Fig 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2.

Classification results on the following three settings: (1) Three combined infection severity classes from the the HUST dataset [7]: control, Type I (Mild and medium cases), and Type II (Severe and critical cases). (2) The same three classes, but from the combined HUST and MosMed datasets. (3) All five classes, control, mild, medium, severe and critical, from the same split of the combined HUST and MosMed datasets.

https://doi.org/10.1371/journal.pone.0334431.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. (a) The impact of changing the epoch at which the proximity factor for infection clusters is activated on the validation infection coverage and F₁ measure.

(b) The ablation results when removing the influence of the increase and decrease coefficients from (2) on the validation infection coverage as well as the impact of removing the volume-level features. (c) Comparison between the validation infection coverage pattern during training of the three and five class settings.

https://doi.org/10.1371/journal.pone.0334431.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The confusion matrix with mean and standard deviations for the five (a) and three (b) classes in the HUST, MosMed dataset combination when testing our proposed approach using 5-fold cross validation.

https://doi.org/10.1371/journal.pone.0334431.g007