Breast cancer detection context

Test screenings are used to detect early signs of breast cancer before the appearance of any symptoms. The main method used for breast cancer screenings is mammograms, low-dosage x-rays around the breast area [7]. These scans reveal a high contrast background in black with dense areas in whites, which may correspond to calcifications or masses (e.g. lumps or cysts). If suspicious areas are detected, mammograms are followed by breast ultrasounds and MRIs (Magnetic Resonance Imaging) [8]. If any of the screenings raise suspicion or reveal a potential presence of breast cancer, then a breast biopsy, the removal of a small section of breast tissue to be analysed by a pathologist, can be conducted to confirm the screening tests’ results [9]. Due to the invasive nature of biopsies, it is ideal for doctors or clinicians to use medical imagery tools to detect early signs of breast cancer that can be treated early. Despite being the primary imagery method used, mammograms rely on the conventional diagnoses of expert radiologists [10]. These diagnoses rest on the correct interpretation of the mammograms, which may be subject to errors and interobserver variability due to the difficulty in correctly interpreting them [5, 11].

Supervised machine learning techniques are replacing these expert systems, allowing for hidden patterns in mammography data, which can not be perceived by radiologists, to now be recognised by these new algorithms [12]. Machine learning-based approaches were selected over statistical approaches, especially when dealing with large, complex and high-dimensional data like mammography datasets [13]. However, these machine learning models such as k-Nearest Neighbours (kNN) [14], decision trees, Support-Vector Machines (SVM) [7] and Artificial Neural Networks (ANN) [13] can not accurately operate on raw data as they required relevant features ranging from visual information (colours, edges, corners, shapes and textures [15]) to extracted information (cell size, clump thickness, bare nuclei, etc. [13]) to first be engineered from the images. Deep learning models, which corresponds to neural networks with hundreds of hidden layers, are based on the concept that these features can be learned by models on their own, directly from the data [16]. However, these deep models have not been broadly implemented until recent years as they require high powered Graphical Processing Units (GPU) to be trained effectively [17, 18].

Deep learning applications with transfer learning

Due to recent advances in deep learning techniques, this method of image analysis has shown promise in being transferred to the medical image domain, and more specifically for use in disease classification, detection and segmentation.

Given the size of deep learning networks, training an entire model from scratch is an intimidating task which requires huge amounts of data, time and compute resources to achieve accurate results [19]. To alleviate this issue, transfer learning is often utilised. This involves using networks which are pre-trained on large yet different datasets, and using these pre-trained weights as a starting point for further training on the desired dataset for the specific task, effectively fine-tuning the model [20]. The aim of this is to improve a network designed for a specific domain by transferring that which has already been learnt from another similar domain [21]. An analogy used [22] helps to explain this in real life; If one was to compare the ability of an experienced guitarist and someone with no musical experience to learn the piano, the guitarist would learn the piano in a more efficient way through using prior knowledge. This is much like how a network uses its knowledge from previous training as a starting point for learning similar tasks.

The need for transfer learning comes when there is insufficient training data, whether this is due to expense, scarcity or inability to generate labelled data. This is a common problem in the medical imaging domain and more specifically for mammograms as there are very few public databases which exists that contain an extensive number of cases with labelled data. One of the largest public databases, the Digital Database for Screening Mammography (DDSM) [23] has as few as 2600 cases. Second to this, released in April 2021, is The Chinese Mammography Database (CMMD) [24] which contains 3728 images for a total of 1775 patients. It should be noted that both DDSM and CMMD datasets are available freely for research usage via The Cancer Imaging Archive [25]. Papers focusing on the DDSM dataset tend to use complex pipelines consisting of extensive pre-processing procedures. A prominent example is Mohiyuddin et al. who achieved a record accuracy of 96.50% [26] with the use of state-of-the-art algorithms such as YOLOv5 [27]. However, in their work, they design their experiments such that macro calcification are omitted completely and only masses are considered, whereas this study considers both masses and calcifications both together and separately. Another paper with a complex pipeline consisting of several pre-processing, segmentation, false-positive reduction and classification steps, achieved an accuracy of 90.44% and an AUC of 90% [28].

Aboutalib et al. [29] explored the use of transfer learning through an incremental approach for a 3-class classification task on mammograms for negative, benign and malignant cases. The study used an AlexNet [30] architecture. This model made use of transfer learning by using ImageNet for pre-training and then further learning on the DDSM database (9648 images) before finally fine-tuning the model on the Full-Field Digital Mammography (FFDM) dataset (5212 images), a private dataset [29]. To explore the effects of transfer learning, and given that the DDSM database was larger than the FFDM database, Aboutalib et al. compared a model with further pre-training on the DDSM database before fine-tuning on the FFDM dataset against a model with only fine-tuning on the FFDM dataset (note, both models were pre-trained on the ImageNet database and testing was done on the FFDM dataset). It was observed that the effects of further pre-training on the DDSM dataset helped to improve the model’s Area Under the Curve (AUC) performance by an average of 3.2% across the 6 classification pairs. That being said the final performance only managed to generate an AUC of 76% and the results showed the model had a clear weakness in diagnosing malignant cases as opposed to benign. This points towards the fact that there are imaging features unique to the recalled benign cases which the network is able to successfully identify, features that are much harder to pick up for the malignant cases where the network is less successful.

Comparing full image against regions of interest as model input

The lack of accuracy generated by some models and deep learning techniques can often be traced back to the size of the image and the corresponding Regions of Interest (ROI) where breast masses and calcifications occur. The ROI can often vary in size and be a very small proportion of the entire image. As a result, this makes it more difficult for diagnosis [31]. A common setup in research has involved using pre-annotated ROIs for classification purposes. This approach was taken by Chun-ming et al. [32], in undertaking a 5-class classification task: benign calcification, malignant calcification, benign mass, malignant mass and a normal breast. The paper explores the creation of a Deep Cooperation Neural Network consisting of two parallel CNNs that feed into a fully connected network. The research made use of data from the DDSM archive, where pre-annotated ROIs were used for images with calcifications and masses, and cropped images of the same size were used for normal breast images, all with image sizes of 299×299 pixels. For a negative-class classification task, this network was able to achieve an accuracy of 91% with an AUC of 98%, a drastic improvement from the full mammogram classification carried out by Aboutalib et al, emphasising the advantage of using cropped images. Similarly, Levy et al. [33] used a similar approach in determining the malignancy of masses using the DDSM database. This research involved a comparison of three CNNs, a baseline shallow CNN with three convolutional layers and three fully connected layers along with AlexNet and GoogLeNet [34]. As before, pre-annotated ROIs were used for the classification task, with padding equal to 50 pixels (small context) and padding equal to a given amount such that the ROI equals to twice the size of the mass bounding box (large context). The results showed that, by utilising the large context enhanced classification method, accuracy with the GoogLeNet CNN architecture reached 93%. From this research, however, it is apparent that the accuracy of using CNNs for breast cancer classification is improved by using more specific ROIs, for a closer view of tumours, in order to generate more accurate results. This is an important point as it is a critical idea that can aid classification accuracy, a primary objective of this research paper. The issue with this however is similar to the issue represented by using ANNs, SVMs or kNN algorithms for classification using pre-extracted features, in that, image pre-processing and ROI tagging is required for an accurate prediction. This is then again reliant on trained radiologists generating ROIs to be used as an input to a CNN for breast cancer detection. Thus, bringing back the initial issues of being reliant on human input.

The inherent difficulties and limitations associated with classification of breast tumours using whole images alone leads onto the next wave of research based on automated extraction of ROIs and/or feature extraction for cancer classification using whole images. The effectiveness of using ROIs is clear; given the size of most tumours in respect to the entire image, a large majority of the image is essentially background noise that can throw off a classifier which is why it is important for a classifier to be able to learn and recognise the ROIs for classification tasks. This is achieved in multiple ways, from ROI tagging to mass segmentation and classification. An example of this is carried out by Punitha et al. [35]. In this research, an optimised region growing technique is used for breast mass segmentation. The segmentation algorithm uses a thresholding technique for region growing with thresholds and optimised seed points. Once segmented, GLCM and GLRLM (Gray Level Co-occurrence Matrix and Gray Level Run-Length Matrix) feature texture extraction is carried out on the mass. The results are then fed into a simple feed-forward neural network for classification. Through the use of segmentation and feature extraction, the classifier was able to achieve a sensitivity and specificity of 98.1% and 97.8% respectively and the final classification task had an accuracy of 98%. This process has effectively brought forward the strengths of using prior machine learning classifiers with high accuracy used on extracted features whilst managing to automate the feature generation through segmentation and automatic feature extraction. In doing so, research such as this highlights the importance of being able to detect where a mass or calcification is, either via bounding boxes or segmentation, in order to enhance classification tasks without the need for prior human input.

However, the implementation of multi-stage detection systems can add greater degrees of error into the system. Although these may appear to have high accuracy, the drawback of such a system is that the entire classification becomes increasingly dependent on the accuracy of the earlier detection stages. If, for example, one uses a mass segmentation system followed by a classification of the segmentation for cancer detection, this firstly relies on the full successful segmentation of a tumour followed by the successful classification of that tumour. As such, it can be seen why there is value in a single stage whole image classification system that has no prior reliance on other levels of image analysis.

The literature covered in this context review analysed the various techniques already used in mammography classification, ranging from machine learning and deep learning techniques to pre-processing steps. Early breast cancer detection methods such as expert systems and supervised machine learning-based techniques, lacked the performance attainable from deep learning techniques. Recent datasets have been beneficial in medical imaging, such as CBIS-DDSM for mammography classification. However, most studies using these datasets have made use of complex pre-processing techniques to obtain improved performance. Unfortunately, this makes their models reliant on tailored pre-processing pipelines to extract this performance and can reduce model generalisability. Alternatively, this study concentrates on these deep learning techniques and the performance gained by combining them to maximise classification accuracy instead. Accordingly, the divide and conquer approach proposed in the Materials and Methods section focuses on achieving that, rather than focusing on a single model and relying on optimised data that is pre-processed.

Reproducibility

To maximise reproducibility, the code used to implement the deep learning techniques in this paper, as well as the instructions to download the relevant data and run the pipeline, have all been made open-source and can be accessed online at https://doi.org/10.5281/zenodo.6828154.

Resources

Computational resources, in the form of a GeForce GTX 1060 6Gb GPU, were used to train models. To make use of the GPU’s compute capabilities, Tensorflow [36] and Keras [37] were used. These provide deep learning frameworks with CUDA support (to enable GPU optimisation and parallelisation), CNN support and pre-trained models.

Ethics statement

Datasets utilised in this paper include CBIS-DDSM and mini-MIAS, both of which are publicly available with all personal data anonymised and removed by default. In addition to the mammography images, both datasets supply accompanying diagnostic information. This accompanying data does not include any identifiable labels. Additionally, this study was approved by the University of St Andrews School of Computer Science Ethics Committee on behalf of the University Teaching and Research Ethics Committee (UTREC). Both datasets were not created by the authors and no participants were involved in this study. Therefore, no further consent-taking or ethics approval was needed.

Data

The deep learning models implemented in this work require clinical data to learn the underlying patterns necessary to detect cases of breast cancer in mammograms and to evaluate its performance. Therefore, the CBIS-DDSM and mini-MIAS datasets, fully anonymised, open-source and public datasets, were used.

The Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) dataset, an open-sourced, anonymised public dataset [38], is available online from The Cancer Imaging Archive [25]. The dataset, which contains a total of 10,239 images in Digital Imaging and Communications in Medicine (DICOM) format gathered from 1,566 patients across 6,775 studies [39], is the main dataset used in this work. This dataset is an updated and standardised subset of the older DDSM dataset [40], containing only abnormal cases with benign and malignant tumours (no normal cases). The scans found in the dataset are a mix of the two most commonly used projections in routine mammogram X-ray scans: bilateral craniocaudal (CC) and mediolateral oblique (MLO) [5]. The dataset can be further be separated into two different types of structures that radiologists usually look for to detect early signs of breast cancer: calcifications (small flecks of calcium usually clustered together) and masses (e.g. cysts or lumps) [41].

The mini Mammography Image Analysis Society dataset (mini-MIAS) [6] is a smaller open-sourced, anonymised public dataset available online from Pilot European Image Processing Archive (PEIPA). It is used solely for transfer learning purposes. It contains 322 images in greyscale Portable Gray Map (PGM) format with associated ground truth data and images all reduced to a uniform size of 1024×1024 pixels [42]. It is divided into normal, benign and malignant cases [43], but only the benign and malignant cases were used in this work.

The methods outlined in this section are implemented using a divide and conquer approach, whererin a combination of techniques are iteratively tested and evaluated to build an understanding of which perform best. They include a mixture of data preparation and model training techniques, combined as shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Divide and conquer experiments summary table.

Table comparing the different divide and conquer models tested using various deep learning techniques covered in the Materials and Methods section. * Histogram equalisation + Gaussian filtering. α: All layers instantiated with pre-trained weights from a binarised mini-MIAS dataset. β: Base CNN layers instantiated with ImageNet weights; fully-connected layers instantiated with pre-trained weights from a binarised mini-MIAS dataset. γ: Base CNN layers instantiated with ImageNet weights; fully-connected layers instantiated with random weights. δ: Balanced class weights (0.907 for majority benign class, 1.113 for minority malignant class). ϵ: +50% class weight for minority class (1.0 for majority benign class, 1.5 for minority malignant class). ζ: Two sets of convolutional layers before base CNN. η: Convolutional layers before fully-connected layers.

https://doi.org/10.1371/journal.pone.0280841.g002

Data preparation techniques

Dataset balance.

The class distributions seen in Fig 3 reveal that the CBIS-DDSM dataset is imbalanced, which must be taken into account to avoid fitting a biased CNN model. Potential solutions for countering this imbalance are to either undersample the majority class or oversample the minority class. Undersampling the dataset can be considered inefficient as it will diminish the number of samples the model could learn from [44]. However, oversampling by creating new artificial images resembling the original data, referred to as data augmentation, is a viable solution as it was proven to decrease the risk of overfitting [45]. Alternatively, a cheaper option in terms of computing resources would be to add class weights, which will cause the loss to become a weighted average giving more importance to less frequent classes [46].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Dataset imbalance.

Bar chart illustrating the imbalance of classes in the CBIS-DDSM dataset.

https://doi.org/10.1371/journal.pone.0280841.g003

Image enhancement.

A number of image processing steps were taken as optional augmentations to aid network performance. These were:

• Optimised image cropping
• Gaussian filtering
• Histogram equalisation

The first step was to use optimised cropping dimensions. From inspecting the mammography images, it was clear that image sizes ranged from roughly 2000×4000 to 3000×5000 pixels. In this work, VGG19 and DenseNet, models that are originally designed for square images of size 224×224 pixels, were used, which was taken into account for the initial sizes. It was however decided that because sizes of 224×224 pixels would require significant downsizing, hence the model input was scaled up to 512x512 pixels, and a square image ratio was preserved.

There are two ways an image can be resized: resizing it such that the image takes up the entire dimensions, i.e. stretch or compress it to a desired dimension, or resizing the image with padding to maintain the image’s aspect ratio. Although the images were initially scaled to a square size without padding, it was determined that this would distort the images’ spatial features through expansion along the horizontal axis. Accordingly, resizing with padding was used to maintain aspect ratios. The issue, however, was that given the usual dimensions of the image, by resizing with padding, a large amount of the image would become blank space. In order to reduce this, the best option was to change the input size to reflect the natural aspect ratio of the mammogram images. In general this aspect ratio was around 1.5:1. Choosing input dimensions that conformed to this level resulted in much less black space in the images as compared to the square images, whilst preserving the natural aspect ratio. An example of the varying resizing of the same image can be seen in Fig 4 below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Image resizing and padding.

Demonstration of varying resizing and padding of the same image: Resized to fill image without padding—aspect ratio not preserved (left), resized to fill image with padding—aspect ratio preserved (middle), resized and cropped without padding—aspect ratio preserved (right).

https://doi.org/10.1371/journal.pone.0280841.g004

Additionally, it has been shown that the use of Gaussian filtering and histogram equalisation can enhance the details in mammograms, to optimise them for use with CNNs. As implemented by George et al. [47], a Gaussian filter with σ = 0.5 was applied, along with histogram equalisation as applied by Cheng et al [48]. A Gaussian filter is a 2D convolutional filter that is used to blur an image and remove detail. In general, depending on the value of σ, a large blur is not used since vital details will be lost. However, this filter is used to remove noise inherent to the image itself.

Histogram equalisation is an image processing technique used to improve the contrast in an image. It does this by spreading out the most frequent intensity values and stretching them over the intensity range. The combination of Gaussian filtering and histogram equalisation is used as another pre-processing step to enhance the quality of the image input. This step can be visualised in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Image enhancements.

Mammogram scans with (left) and without (right) histogram equalisation and Gaussian filtering.

https://doi.org/10.1371/journal.pone.0280841.g005

Model training techniques

At this stage of the pipeline, the training data is ready to be fed into the CNN model, where it will learn from the images that were processed using the steps above, before making predictions on unseen test dataset, which will be compared with the ground truth labels for evaluation.

CNN model.

Due to the very small nature of the CBIS-DDSM dataset (approximately 3000 samples), the knowledge of CNN models pre-trained on large general datasets such as ImageNet (which contains over 14 million images belonging to 22000 categories [30]) is used via transfer learning, a technique that is proven to work on breast cancer detection tasks [49, 50]. This leads a general CNN to converge towards one that is tailored for the task of breast cancer detection using mammography images, rather than creating one from scratch. Different CNN architectures can be used as the base of a custom CNN model tailored for breast cancer detection. This is achieved by using popular CNN architectures natively available within Keras such as VGG19 [51], ResNet50 [52], InceptionV3 [53], DenseNet121 [54] and MobileNetV2 [55] as the base of the CNN [56]. The fully connected layers of these models, originally designed for general classification of 1,000 different categories, were dropped from the model and replaced with a custom MLP (Multi-Layer Perceptron) [57]. The fully connected MLP contains hidden layers and an output layer with different activation functions based on the dataset used.

Depending on the size of the image used, additional convolutional and pooling layers were added before the base model to downsample the image to smaller sizes and learn lower-level features. This was followed by the pre-trained base model, a flatten layer to convert the output from 2D to 1D, and finally the MLP which will make the final prediction. Dropout layers were added in the MLP to avoid overfitting [58]. The full model can be visualised in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Model architecture.

Model architecture implementation, which includes input images, additional convolutional and max pooling layers, ImageNet pre-trained convolutional layers (e.g., on either VGG19 or MobileNetV2 architectures), fully connected layers (flatten, dropout) and the output layer. Original image, “ImageNet pre-trained model” by Aphex34 [59].

https://doi.org/10.1371/journal.pone.0280841.g006

Hyper-parameter optimisation.

Traditionally, a grid search approach would have been preferred to fine-tune the model’s hyperparameters. However, due to the project’s time constraints, the significant training runtime and the number of hyperparameters to fine-tune, a grid search would have been an unrealistic approach. Instead, a divide and conquer approach was selected, manually implementing and validating different combinations of deep learning techniques mentioned in this section. The combinations tested are summarised in Fig 2, and include using different pre-trained CNN models, variations in transfer learning, data augmentation, dropout values and input image sizes. The methods mentioned across this section and the pipeline proposed are summarised in Fig 7, covering the data preparation steps followed, the model training steps, and the model validation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Flowchart pipeline.

A detailed flowchart of the implemented breast cancer detection deep learning pipeline.

https://doi.org/10.1371/journal.pone.0280841.g007

Model evaluation metrics

In mammography classification, overall accuracy (see Eq 1, [49]) is often complemented with additional metrics to evaluate how well the classifier fitted the data. The following terminology is used to define the metrics that are broken down below:

• P: Positives (all positive predictions).
• N: Negatives (all negative predictions).
• TP: True Positives (positive case correctly predicted as positive).
• TN: True Negatives (negative case correctly predicted as negative).
• FP: False Positives (negative case incorrectly predicted as positive).
• FN: False Negatives (positive case incorrectly predicted as negative).

Detecting FPs and FNs is essential to avoid interpreting malignant tumours as benign and vice versa. Such an interpretation could harm the patient and potentially lead to their death. Therefore, a mixture of additional metrics were used to assess how well the model learns the mammograms data and generalises to unseen cases.

Precision, which corresponds to the number of correct positive predictions (see Eq 2, [44]), shows the model’s ability to avoid labelling negative instances as positive. Recall, which is the number of positive instances that are correctly predicted (see Eq 3, [44]), shows how well the model can find all positive instances. Together, they can be combined into a more concise metric, the F1 score (see Eq 4, [20]). To achieve a high F1 score, both precision and recall must be high (unlike a regular mean) because as the precision goes down, the recall goes up, and vice versa, making the F1 score a reliable metric for evaluating a classifier since a balance between precision and recall must be found [20]. (1) (2) (3) (4)

Pipeline generalisation

This workflow can be generalised to other breast cancer detection mammography classification tasks as it can be applied to any dataset similar in nature, and could also be used as a starting point for comparable medical imaging tasks such as MRI classification. To evaluate the generalisability of the model across a larger range of demographics and medical imagery techniques, future work would be needed to test the model’s performance on other mammography datasets acquired from other geographical locations and medical devices. Additionally, all of the code implemented for the steps in this Materials And Methods section has been made publicly available and is hosted on GitHub for reproducibility [63].

The different techniques covered in this section are all tested iteratively in a divide and conquer fashion. The results of each experiment from Fig 2 are presented and discussed in the following sections.

Base CNN architectures

Five different CNN architectures, pre-trained on ImageNet (VGG19, ResNet50, InceptionV3, DenseNet121 and MobileNetV2), are tested out as the base architecture of the model. For this test, the CBIS-DDSM dataset is used with whole images resized to 512×512 pixels, a batch size of 2 and a learning rate of 1 × 10⁻⁴.

The results clearly reveal that MobileNetV2 outperforms the other CNN architectures, achieving a higher accuracy and F1 score. The baseline model’s VGG19 architecture is outperformed by more efficient models such as DenseNet121 (model T) and MobileNetV2 (model C), while still performing better than both the models that use ResNet50 (model V) and InceptionV3 (model U) as base architectures. These results contradict Falconi’s results on the CBIS-DDSM dataset, which find that ResNet50 outperforms MobileNetV2 [50]. However, MobileNetV2 still outperforms InceptionV3. These results may differ due to the different pre-processing techniques being used as Falconi et al. uses cropped images around ROIs, whereas whole images are used in this experiment. It is also worth noting that using MobileNetV2 as base architecture already surpasses the baseline by 4.66%, and beats models that use traditional machine learning methods like SVMs with GLCM features (63.03% accuracy) on the CBIS-DDSM dataset [64].

However, observing the training and testing runtimes reveals that VGG19 takes the longest time to train (3h50m), whereas the more efficient MobileNetV2 architecture takes less time (2h46m). Additionally, inference runtime is 2.3 times faster with MobileNetV2 compared to VGG19, which ultimately is more useful, in practice, for clinics as mammogram diagnosis results can be returned faster.

Class imbalance

Distinct variations of class weights are used on the CBIS-DDSM dataset in attempt to rectify the adverse effects that can be introduced by imbalanced datasets. This method avoids the process of data augmentation, which considerably slows down the training time. Three unique class weights values were tested using the imbalanced CBIS-DDSM dataset with whole images resized to 512×512 pixels, a batch size of 2 and a learning rate of 1 × 10⁻⁴:

• No class weights (dataset remains imbalanced).
• Balanced class weights:
  • 0.907 for majority class (benign).
  • 1.113 for minority class (malignant).
• Pre-determined +50% class weight for minority class:
  • 1.0 for benign samples.
  • 1.5 for malignant samples.

The results using using balanced weights clearly show a 0.62% increase to the accuracy (model H) relative to the one that uses no class weights (model C), thus helping against the imbalanced dataset issue without the need for techniques like data augmentation. However, the pre-determined weight increase for the minority class decreases the relative accuracy by 1.41% (model I) to that same model that uses no class weights, revealing the complexity and importance of finding the right parameters for balancing datasets since a 50% weight increase for malignant samples introduced even greater imbalance.

Input image sizes

Degrees of transfer learning

Instead of using a CNN pre-trained on ImageNet, model weights trained on a binarised version of a small dataset, mini-MIAS, are transferred to the CBIS-DDSM dataset. Four different experiments using identical CNN architectures with variations in transfer learning are tested to assess the effect of transfer learning from the binarised mini-MIAS dataset and ImageNet on the CBIS-DDSM dataset:

• Transfer learning of all layer weights (MobileNetV2 and fully connected MLP layers instantiated with binary mini-MIAS weights).
• Transfer learning of fully connected layer weights (MLP layers instantiated with binary mini-MIAS weights, MobileNetV2 layers instantiated with ImageNet weights).
• Transfer learning of ImageNet weights only (fully connected MLP layers instantiated with random weights, MobileNetV2 layers instantiated with ImageNet weights).
• No transfer learning (MobileNetV2 and MLP fully connected layers instantiated with random weights).

The results clearly indicate that any given form of transfer learning provides an improvement over random weight initialisation (model G). On the other hand, too much transfer learning, by using all the weights from the model trained on the binary mini-MIAS dataset, does not generalise well to the CBIS-DDSM dataset (model A). The best performance of this experiment came from initialising MobileNetV2 with ImageNet weights and the MLP fully connected layers with weights trained on the binarised mini-MIAS dataset (model B), achieving an accuracy increase of 5.28% relative to the baseline, which corresponds to an F1 score of 67.23%. This model was closely outperformed by again using ImageNet weights for MobileNetV2 and random weights for the MLP layers which reached a 5.6% relative accuracy increase (model H). The ImageNet weight transfer method provided a performance gain whilst proving the effective and adaptive nature of CNNs when leveraging knowledge learned from large general datasets on a more specific task like mammography classification. It is worth noting that training is marginally faster when using weights from binary mini-MIAS as the model converges more quickly.

Mammogram types

To assess how the model would adapt to learning samples from a single type of mammogram, the CBIS-DDSM dataset was separated into only mass samples and only calcification samples. These results show that the model using VGG19 as a base architecture achieves moderate improvement when mass and calcification samples are separated, reaching a 2.55% accuracy increase (model L) and a 4.87% accuracy increase (model M) relative to the baseline respectively on the test set, while achieving an relative accuracy increase of of 2.79% when using the full CBIS-DDSM dataset (model K). Indeed, all instances are classified as “benign” on the full CBIS-DDSM dataset (either true negatives or false negatives), indicating that the model fails to consistently deal with multiple views and cannot differentiate benign and malignant cases from each other. This outcome is in line with Hepsag’s et al. results, which achieve higher accuracies when classifying either masses or calcifications on another dataset [43], and confirms Elter’s et al. claim that masses are more difficult to detect than calcifications [5].

However, the opposite effect is witnessed when MobileNetV2 is used as a base model, reaching an accuracy increase relative to the baseline of 4.66% (model C) when the full dataset is used and only 1.32% (model E) and 1.43% (model D) for calcifications and masses respectively, contradicting the previous results. Because CNNs automatically learn features, it can be hard to know exactly what goes on underneath the hood of these models, especially when using architectures like VGG19 and MobileNetV2 due to their complex structures. Visualising heatmaps of the feature maps for each convolution layer could help understand why these models react differently when using all images or only specific types of mammograms, but is out of the scope of this work.

Image pre-processing steps

Following the successful implementation of various models, the next step was to move beyond the model fine-tuning and towards the data itself to improve model performance. This was undertaken through image processing techniques that involved histogram equalisation and Gaussian filtering. Histogram equalisation, in particular, is used to help with contrast adjustment. This can be helpful in highlighting important features by increasing contrasts to further highlight masses and/or calcifications. Through applying these image processing techniques, the data was then used to train and test using the VGG19 architecture with image sizes of 2048×2048 pixels and two additional sets of two convolutional and pooling layers. This generated the greatest improvement in accuracy with an accuracy increase of 1.8% (model P) in comparison to the same model without any image pre-processing steps (model Q). This further emphasises the effectiveness of image pre-processing steps such as Gaussian blur and histogram equalisation for mammography classification tasks.

Dropout rate

Although early stopping was implemented during training to avoid overfitting, it has been shown that adding dropout layers to the fully connected portion of the model can help eliminate overfitting [58]. Applying dropout layers with a drop rate p = 0.2 to the VGG19 architecture with image sizes of 2048×2048 pixels and two additional sets of two convolutional and pooling layers provides an increased accuracy of 1.10% (model O) compared to the same settings without any dropout (model P). Previously, the classification results showed a lot of bias within the baseline model with a considerable offset between precision and recall due to a heavy bias towards predicting the benign class. However, through adding dropout layers, the inherent biases in the model are reduced, resulting in a reduction in this disparity to only 1%, and more importantly, a greater ability in classifying malignant samples. Ultimately, for tasks like cancer detection, it is more important that a model is sensitive to the malignant class to avoid incorrectly diagnosing patients that have a malignant tumour as having a benign tumour.

Limitations

A known limitation concerning all breast cancer detection systems lies with the data itself, as the most widely used datasets of mammograms (e.g. DDSM) contain data that mainly originates from white females located in North America, which naturally introduces bias to the model learning this data [16]. Different body types linked to the geographic location of the patients used to create these databases can have a direct impact on the mammograms themselves and not generalise to females from other cultures. For example, a recent study with 53,000 North American females showed how diets that include dairy milk consumption might increase the risk of breast cancer by a maximum of 80% based on the consumption [65]. This means that if these deep learning algorithms were implemented in clinics outside western countries, they might not generalise well to other body morphologies (e.g. due to different diets based on the geolocation’s culture). This limitation could be resolved by collecting more varied data from multiple locations around the world, not just a single region, which would also help the deep learning algorithms used as more data is always more beneficial [66].

Another limitation in terms of the detection system’s usability is the confidence of the predictions. Indeed, when given new test samples, the model predicts a class label, e.g. benign or malignant. However, these do not indicate the prediction’s confidence, as it can be anywhere between the decision boundary’s limit (not confident) and far from the decision boundary (confident). Therefore, from a clinical point of view, it is hard to make a decision based on the predictions made by a system similar to this one. Ideally, a probability-based confidence metric would be coupled with the predictions to motivate the next step after the diagnosis. For example, if the confidence of a malignant tumour is high (e.g. 99%), then breast-conserving surgery or chemotherapy can be recommended, whereas if the confidence is low (e.g. low 50%s), then further screening tests can be recommended instead.

Future work

The main area of work that requires improvements is the mammogram imagery pre-processing as it is often an area where significant performance gains can be found [12] by using techniques such as Global Contrast Normalisation (GCN), local contrast normalisation, and Otsu’s threshold segmentation. Artefacts such as tags on the x-rays and black backgrounds should all be removed using computer vision techniques to avoid the CNN learning irrelevant features. A couple of ways to do so could involve cropping the image directly around the breast area or introducing bounding boxes for ROIs. Those improvements, accompanied with providing the degree of certainty with regards to the binary classification, would increase the clarity of the predictions for clinical practitioners and allow for improved human cross-validation and/or investigation of the pipeline’s output.

Another area where improvements can be made is the fine-tuning to extract better performance on the datasets and avoid overfitting. With the data pre-processing mentioned above, images would be smaller (e.g. no redundant dark background), which would in turn allow for quicker fitting and inference runtimes revealed, allowing hyperparameter fine-tuning algorithms like grid search or Bayesian optimisation to explore more combinations of configurations in order to unlock better solutions.

Finally, future research could look into applying the results found in this paper on other mammography datasets, such as the recently released Chinese Mammography Database (CMMD) dataset [24] and other datasets that introduce greater heterogeneity across its samples. This would improve generalisability with regards to imaging and cross-continental variation in body morphologies. Expanding the classification task from a binary to multi-class one by using different datasets would also help make the pipeline more versatile as it could not only be able to classify benign and malignant cases, but also normal cases with no masses or calcifications.