4.1.1 Wang algorithm assessment.

The winning algorithm submitted to the Camelyon 16 challenge was by Wang et al. [18]. It should be noted that additional details of the algorithm are included in Bejnordi et al. [16] and it’s supplementary details.

System specification and software—Red. No information was provided about the hardware or software platform in either Wang et al. [18] or Bejnordi et al. [16]. When reproducing, the hardware and software platform, outlined in the common architecture section of S1 Text, was assumed to be an acceptable choice.

Dataset splitting—Red. No information was provided about how or if the slides were split into training and validation sets. For the reproduction, an 80:20 split between training and validation set was applied as it matched the split in the paper by Liu et al. [21] and there was no other information to go on. Splitting was applied at the slide level as outlined in the common architecture section of S1 Text.

Stain normalisation—Yellow. Stain normalisation is not mentioned in the paper of Wang et al. [18]. However, in the supplementary details provided by Bejnordi et al. [16], two methods are described: method I, without stain normalisation, and method II, that applies Whole-Slide Image Color Standardizer (WSICS) stain normalisation [27]. WSICS is a template based normalisation technique and the supplementary details did not state what template was used or how it was selected. There is ambiguity in if stain normalisation was applied or not, if it was the technique is clear but there are key parameters missing, therefore it was rated as yellow. As it was unclear which method was reported in Wang et al. [18], stain normalisation was omitted from the reimplementation, this should match the method I reported by Bejnord et al. [27].

Patch extraction—Yellow. It is not clear what technique was used for patch extraction, for example if the patches were extracted from random locations, or on an non-intersecting grid. The paper and the supplementary information state different patch sizes: Wang et al. [18] states they were 256x256 and Bejnordi et al. [16] states they were 224 × 224. For the patch sampling strategy, no details are given about if they were balanced across slides or if they were extracted with replacement or not. Due to these pieces of missing information patch extraction was rated as yellow. The reimplementation used a regular grid, patches of size 256 × 256 were used and then cropped down to 224 × 224 as random cropping was stated as one of the augmentations applied to each patch. The assumption was that the cropping from 256 × 256 to 224 × 224 explains the difference in the reported methods.

Patch labelling—Red. The description in Wang et al. [18] is unclear as to what area on the patch had to be covered by an annotation polygon for it to be labelled as that class of annotation. Due to the lack of information, when reproducing the patch labelling strategy outlined in the common architecture section of S1 Text, was assumed to be an acceptable choice.

Patch augmentation—Yellow. No mention of patch augmentations is given in Wang et al. [18], however in Bejnordi et al. [16] random rotation is stated, but no parameters were given, therefore this was rated as yellow. In the reimplementation a random rotation was selected from 0, 90, 180, or 270 degrees.

Patch classification—Green. The details of the software platform, the number of GPUs used and details any parallelism used were missing. Not specifying these details was not a hindrance to reproducing the algorithm, though they may have small variations in the final values reported. As the algorithm described by Wang et al. [18] could still be reproduced this was rated as green.

Hard negative mining—Red. It was stated that hard negative mining was carried out, however details of the hard-negative mining were lacking. Therefore, in the reproduction it was assumed that all the false-positive patches were added into the training set and the model was retrained using the weights from the previous training. The previous weights were used in order to take advantage of transfer learning from the existing model and maximise performance. Given the number of false positives that were found, adding them into the training set was deemed not to significantly unbalance the dataset and this seemed like the most straight-forward approach.

Heatmap generation—Red. Details of how to generate heatmaps from the probabilities of the patches is not given and so this was rated as red. It is not clear if overlapping patches have been used, if so by how much the patches overlap and how different probabilities in overlapping areas are resolved. Since no more complex details were given, in the reproduction the simplest method that is a regular grid of non overlapping patches, was used.

Whole slide classification—Yellow. A random forest algorithm was used to classify the whole slide images based on features extracted from the heatmaps. However, no details of the hyperparameters of the random forest were given or software packages were reported, therefore it was rated as yellow. In the reproduction, the random forest classifier from scikit-learn was trained using it’s default hyperparameters. The defaults were assumed since taking hyperparameters from other studies did not seem sensible as it was assumed any study will have tuned hyperparameters for their own purposes. It is noted in the documentation for the scikit-learn random forest classifier that the defaults can lead to over fitting. When implementing an original model the approach in this situation would be to carry out a hyperparameter search to determine the best settings. When reimplementing from published literature these parameters should be specified so this should not be necessary. With no other information to go from the default parameters have been assumed to be acceptable, if these parameters are critical they should be specified in the published paper.

Reported Metrics—Yellow. The paper by Wang et al. [18] reports patch level results but it is not clear if this is on a balanced dataset or all patches from a dataset. It’s also not clear if this accuracy is reported for the test set of the validation set. For slide classification and lesion detection Wang et al. [18] and Bejnordi et al. [16] report slightly different values, we have assumed that the first is reported on valid and second on test. The reported metrics have been classed as yellow due to the lack of clarity on the patch metrics.

4.1.2 Lee algorithm assessment.

The leading algorithm on the Camelyon 17 leaderboard [20] in January 2021 is described in [28], the same team has published an alternate version with better results as [19] notably for reproducibility this paper contains more details of the algorithm. This paper was selected to reproduce. The team also provided additional supplementary information as part of the challenge [24].

System specification and software—Red. No information was provided about the hardware or software platform in any of the publications [19, 24, 28]. When reproducing the hardware and software platform, outlined in the common architecture section of S1 Text, was assumed to be an acceptable choice.

Dataset splitting—Yellow. The information given is the number of patches for each of the train/validation and test sets. There is no information on whether split is done on a patch, slide or patient level and how it is distributed between Camelyon 16 & 17 slides. For reproduction all the slides from Camelyon 16 and and only those from Camelyon 17 with annotations were used for training. Camelyon 16 was split at slide level and Camelyon 17 at patient level to train and validation sets.

Stain normalisation—Red. The only information given was that GAN is used for stain normalisation. There was insufficient information to make reasonable assumptions about the technique they have used and so in the reimplementation stain normalisation has not been done.

Tissue segmentation—Red. No information has been provided on tissue segmentation. It was assumed tissue segmentation was necessary to help make the implementation more efficient by discarding the parts of the slide that don’t contain any useful information. In the reimplementation the same segmentation method outlined in Liu et al. [21] was applied.

Patch extraction—Yellow. It is not clear what technique was used for patch extraction. It was stated that patches were extracted at random without intersection, with no details of how this was implemented. The level at which the patches were extracted was not given and no details were given on how to sample the patches from slides or classes. The patch size and the number of patches for train, validation and test sets were given. The reimplementation sampled patches on a regular grid of size 256 × 256 to ensure no overlapping patches and then randomly sampled from these patches. Finally the sampled patches were cropped down to 240 × 240. As no information was given on what level patches were extracted from, in the reimplementation we extracted patches at level zero as patches extracted at level zero are the highest resolution, this also matched what was done by Wang et al. [18] and Liu et al. [21]. In the reimplementation it was assumed that there was a 50:50 split to balance the classes.

Patch labelling—Red. There is no description in Lee et al. [19] as to what area on the patch had to be covered by an annotation polygon for it to be labelled with that annotations class. Due to the lack of information, when reproducing the patch labelling strategy, outlined in the common architecture section of S1 Text, was assumed to be an acceptable choice.

Patch classification—Yellow. Not all details needed to carry out patch classification were provided, in particular, the number of epochs to run the training, the batch size and, the loss function were missing. The values chosen for these parameters in the reimplementation were the simplest version of these methods available in the Wang algorithm or the Liu algorithm. The details of the software platform, the number of GPUs used and any parallelism used were also missing, however not specifying these details was not a hindrance to reproducing the algorithm, though they may have small variations in the final values reported.

Hard negative mining—Yellow. The paper by Lee et al. [19] does not specify how many patches were added to their initial training dataset by hard negative mining. The initial training dataset for this reimplementation had 45,000 normal patches. When hard negative mining was carried out in the reimplementation 450,000 false positive patches were found. Adding all these patches to the initial training dataset would change both the scale of the training being carried out as well as the balance between normal and tumour patches. It was assumed that if a change this large was found by the authors of the paper [19] then it would have been reported in the paper, perhaps either many fewer false positive patches were found or only a subset of the patches found were added to maintain the scale and balance of the training dataset. It was decided that adding all 450,000 patches found in the reimplementation was not a sensible approach. Therefore the 45,000 of the patches with the highest probabilities found were added to the initial training dataset. To retrain the network with the new training dataset after hard negative mining either the network could be initialised from ImageNet weights or from the weights of the first trained model. Since it was not stated in Lee et al. [19] which one was carried out, the assumption was made to use the weights from the first trained model to take advantage of transfer learning.

Heatmap generation—Red. It was stated in Lee et al. [19] that a patch is represented as one pixel in the heatmap, but as there was not enough information on how to extract the patches, it is not clear how the heatmaps are generated. In the reproduction, the simplest method, that is a regular grid of non overlapping patches, was used.

Whole slide classification—Yellow. It was stated by Lee et al. [19] that DBScan was used to find clusters of tumor but the parameters used were not stated. A list of 7 features to be extracted from each of the 3 largest tumor clusters found by DBScan was given, this results in 21 features in total. However, it was stated the 24 features were used, and so some information is missing. In the reimplementation the 21 features are used and for DBScan the default parameters are used.

For classification it is stated in Lee et al. [19] that they have used XGBoost but no parameters for this are given. There is further confusion because it is not clear what slides have been used for training and validation and whether the sets used for training and validation at patch level classification is consistent with slide level classification. In the reimplementation, training and validation sets for Camelyon 17 were created at patient level with 62% of the patients in the training set which corresponds to 310 slides in the training. This split remained consistent throughout patch classification, slide classification and patient classification.

Since there are multiple slide for each patient, it is advisable that all the slides for a patient be in the same set due to their similarity. In the paper by Lee et al. [19] there are different splits for patch classification, slide classification and patient level classification which does not seem to maintain the consistency of slides in training and validation sets between patch, slide and patient level classification stages. In the reimplementation it was decided to ensure consistency so that mixing of the train and validation sets could not happen accidentally when moving between different classification stages.

Patient level detection—Yellow. The method to go from slide level classification to patient level classification is clearly stated in Lee et al. [19], however as discussed previously the procedure of splitting data is not clear. Therefore we have used the same sets at this stage of classification as we did in patch and slide level classification.

Reported Metrics—Yellow. The metrics reported are clearly stated in Lee et al. [19], however as the splits are not clear as stated in 4.1.2, it is not clear what datasets the metrics are reported on.

4.1.3 Liu algorithm assessment.

The third paper selected was the one which gave the best results found at the time of the search, in early 2020. This paper by Liu et al. [21] was this final paper selected to reproduce as it had the best reported results on Camelyon 16.

System specification and software—Green. The number and type of GPUs used was given by Liu et al. [21], and the software platform used was TensorFlow. In the reimplementation we have used the same number of GPUs but of a different type. It is assumed that GPU architecture is unlikely to have a significant effect on the outcome of the experiment. The reimplementation used PyTorch, there are some differences between PyTorch and TensorFlow but it was expected that these libraries were interchangeable from a methodological point of view.

Dataset splitting—Yellow. The method used for allocating slides to training or validation set is not given in Liu et al. [21]. In the reimplementation, the same ratio between the two sets was maintained and slides were sorted randomly into these two sets. A random split, when no other information was provided, is a reasonable way to proceed.

Tissue segmentation—Yellow. The level of magnification at which the tissue segmentation was applied was not stated. In the reimplementation, tissue segmentation was carried out at level 5 which was same level in Wang et al. [18]. All other information for tissue segmentation was given and the same was applied in the reimplementation.

Patch extraction—Yellow. The size of patches and the level they are extracted are clearly stated in Liu et al. [21], what is not clear is the number of patches used, how they are sampled from the slides and balanced across the classes. The paper presents a patch sampling method designed to avoid bias towards slides with more patches. However, based on the number of patches stated in the appendix and their patch sizes, that would imply they are using all the available patches, meaning that no sampling would be required. In the reimplementation, the number of patches was set as 10,000,000 in the training set with a validation set of 1,250,000. This is in a similar order of magnitude but it’s hard to know the exact numbers from the original system. A similar weighted sampling method was applied when selecting these patches, rather then using all the available patches. Given the uncertainty this may be a source of difference in the results between the original project and the reproduction.

Patch classification—Yellow. The description of the patch classifier in Liu et al. [21] was fairly comprehensive, however, the loss function used and the criteria used to determine when training was complete were missing. In our reimplementation cross-entropy loss was used, as this was used with the original GoogLeNet architecture [29]. The model was trained for 15 epochs with early stopping if the validation accuracy didn’t improve for five epochs. The model weights used were those from the epoch with the best accuracy, which was found to be the second epoch.

5.1.1 Mismatched data distributions.

The same algorithm trained on different input data can lead to very different output predictions. In particular, differences in the proportion of samples from each class and the data diversity within each class can drastically alter results. The balance and diversity of data is affected by several stages in the pipeline. Such issues affect patch classification, slide classification, and lesion detection.

How the dataset is split into validation and test sets can lead to differences in data distributions between the original experiment and the reimplementation. This can also be the case for how the patches were extracted and balanced between classes. Patch augmentation can also help balance the dataset through increasing the number of samples from a minority class by applying different augmentations such as rotation and flip. Therefore, missing information in any of these can cause the performance of the trained classifier to vary. For example, some samples may be harder to classify than others, if a training set has less of these, it is unlikely to generalise well to a testing set that contains many such samples.

Hard negative mining improves the performance by changing the distribution of the training set, showing the algorithm more of the samples that lead to confusion teaches the classifier to better distinguish between these. Therefore, missing information about the hard negative mining process can affect how the model does or does not generalise to the test set. For example, different ways of adding or replacing samples will lead to differences in the distributions of the training set between the original and reimplementation.

If slides come from different labs, differences in the staining protocols between the labs can lead to differences in the colour distributions of the slides. Stain normalisation compensates for this by transforming each image into the same colour range. Alternatively, patch augmentation can compensate using color augmentations to increase the diversity of the training set. Differences in stain normalisation or colour augmentation due to missing information means that the classifier could be trained on a dataset that is not as representative of the validation and testing sets.

5.1.2 Uncertainty in segmentation and labelling.

If areas of the slides are labelled differently in the reimplementation and the original papers this can lead to changes in the classification performance. This can be affected by differences due to missing information in both the patch labelling and tissue segmentation methods.

The amount of tumorous pixels in the patch required for the patch to be labelled as tumor will make it easier or harder to train the patch classifier. For example, if only a few pixels in the patch need to be tumorous to label that patch as tumor, it could be expected that there is more noise and uncertainty in the patch classification step compared to when larger amounts of the pixels in the patch are tumor pixels.

Poor tissue segmentation can cause two problems. Firstly it can exclude tissue that may contain tumor. Secondly including extra background which can introduce noise to the training process. Both of these can affect the patch level classifier results to some extent but their effect on slide and lesion level classification will be larger. Internal holes in the tissue are part of the tissue structure and are of diagnostic importance. Different segmentation methods may include or exclude these regions which can therefore result in different classification accuracies.

5.1.3 Model convergence differences.

Missing information can lead to differences in how the model converges for both the patch and slide classification models. The patch and slide classifiers could be overfitted to the data, alternatively they may be underfitted or converge to a local minima.

Here we concentrate on outlining the effects of the patch classification parameters that are missing from the papers, for example if network architecture details were missing that would have a massive effect, but this is not discussed here as these have been well specified. Knowing when to stop the training of the network is a key parameter to replicate performance as it will directly affect over or under fitting of the network. By not specifying other parameters for example, loss functions, optimisers, batch size these are more likely to lead to the network converging to different local minima to the original papers.

Whole slide classification relies on the features extracted from the heatmap to classify each slide. Much like in patch classification, lack of detail about the machine learning technique employed and it’s hyper-parameters can lead to the over or under fitting of the model or the model converging to a different local minima than the one reported.

5.1.4 Dealing with patch classification imperfections.

Slide classification and lesion detection could be inferred directly from a perfect patch classification. However, imperfect patch classification needs more complex processing to compensate. Heatmap generation, feature extraction and lesion detection can be carried out to overcome an imperfect patch classification. When any of these steps is not well described differences introduced when reimplementing can reduce the effectiveness. This is now discussed in more detail in the following paragraphs.

The heatmaps are the basic starting blocks for slide, patient and lesion level results and so how they are generated is key to the performance of these. How the heatmaps are generated depends on how the patches are extracted when inference is carried out. Note one set of patches could be used for training the model and then the model applied to a different set to generate the heatmaps. The other key information required is how differences in probabilities between overlapping patches is resolved. For example, averaging over multiple patches at the same location can reduce noise in the heatmaps, extracting patches at different sizes or strides will give different a resolution to the heatmap.

In addition, how the information in the heatmaps is extracted on the performance on the algorithms used for slide and patient classification. Extracting features that are not useful for downstream tasks can add noise the data, conversely, not extracting important features can mean information is lost. Both make the slide classification task more difficult.

Note that weakly-supervised approaches to histopathology slide classification based on Multiple Instance Learning (MIL), such as those evaluated by Sudharshan et al. [30], do not necessarily require heatmaps to be generated. Instead they treat each slide, or part of the slide, as an “bag” of features and train classifiers based on the slide level label. The extraction of these features from the WSI and aggregation of these features depends on the specific method. We are interested in future work to extending our approach and checklist to cover these.

Lesion level detection relies on knowing how to select lesions from the heatmap accurately, identifying true lesions and discarding noise. This is done in two steps: blob detection and blob scoring. There are a number of ways of doing each of these steps and without details of the algorithms used, it can be unclear how to replicate them. Different blob detection algorithms can lead to too many or too few blobs and differences in the scoring methods can make it harder to distinguish between true lesions and false.

5.1.5 Data leakage, maintaining separation between subsets.

As discussed in Bussola et. al [31], data leakage is a significant issue in WSI analysis. For example, if the dataset is not partitioned into train and test sets at a patient level then the information can leak from train to test without researchers realising. If the train and test sets contain any of the same data, then the evaluation will be unrepresentative of the models ability to generalise to the target domain. If the train and validation sets contain any of the same data, then the model is likely to overfit. Similar effects can occur when the data is not identical but similar enough to cause issues. For example, two slides that are from the same patient may be very similar and cause overfitting or poor evaluation if used in different splits of the dataset.

To prevent data leakage and maintain separation between subsets, where there are multiple slides per patient, the slides must be split into training, validation, and testing sets at the patient level at the start of the process. In our study, none of the papers included sufficient information to know if the original results were biased by these effects. In our reimplementations, we were careful not to share data between the subsets and it’s unlikely our results were effected by data leakage.

5.1.6 Definition of dataset for results comparison.

In order to compare the results, you need to know that you are comparing the same measurements calculated in the same way on the same data. For example, if accuracy is reported, it’s critical to know on which dataset the accuracy was measured, for example validation or test, and how that dataset was created—was it created by sampling all patches within the slide or is it a balanced set created for training. When this information is missing you can end up comparing the same metric on different datasets giving different results.

5.2.1 Wang.

A comparison of the results reported in [18] and the reimplementation is shown in Table 2. A marked difference can be seen in the patch classification results. In the reimplementation, patch classification accuracy was 15% lower than the value reported in the paper. First of all we do not know if we are comparing the metrics on the same dataset. Secondly dataset splitting, stain normalisation, patch augmentation and hard negative mining all affect the data distribution which can affect the patch classification for the reasons described in Section 5.1.1, and as shown in Table 1 these areas are poorly specified by Wang et al. [18].

In addition, a major difference was seen during slide level classification is shown in Table 2, where the model failed to generalise from the validation set to the test set. The results on the test set were more or less random. As shown in Table 1 there is missing information in patch extraction, heatmap generation and whole slide classification, for the reasons described in Section 5.1.4 these can impact the slide classification results, it may also be due to the differences in data distribution impacting the patch classification. Another possibility is that the random forest classifier used for slide level classification is sensitive to specific hyperparameters and, as these were not stated in the paper, important differences could have arisen.

The lesion level results shown in Table 2 are approximately half as good as those reported in the paper. In lesion level results, the reimplementation sees many more false positives in the predictions. This is an outcome of the lower patch classification accuracy (described above). The lesion identification process is very sensitive to isolated false positive patch predictions, counting each one of these as a separate lesion. As a result, the small difference in patch classification accuracy, increases the average false positive rate for lesions and thus decreases the area under the ROC curve by nearly a half.

5.2.2 Lee.

A comparison of the results reported by Lee et al. [19] and the reimplementation is shown in Table 3. It is difficult to figure out on which sets of data the results are reported as shown by the ratings in Table 1. Therefore, we do not know if we are comparing the same results on same datasets in the original paper and the reimplementation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of original and reimplementation results of Lee paper.

https://doi.org/10.1371/journal.pdig.0000145.t003

Patch classification is affected by the data distribution as described in Section 5.1.1, the data distribution is affected by the dataset splitting, stain normalisation and hard negative mining and as shown in Table 1 these areas are poorly specified by Lee et al. [19]. Alternatively the missing information on patch labelling and tissue segmentation as shown in Table 1 could affect the patch classification results as described in Section 5.1.2. Finally as shown in Table 1 patch classification parameters are not stated in Lee et al. [19] and differences will affect the results as described in Section 5.1.3. All these reason may contribute to the patch classification precision in the reimplementation being much lower compared to the original.

As shown in Table 3, we were not able to reimplement the slide level results reported by Lee et al. [19]. In addition to the patch level classification problem discussed in the previous paragraph, the heatmap generation, feature extraction and whole slide classification were also lacking in information as shown in Table 1. These affect the slide level results for the reasons described in Section 5.1.4. The missing information therefore impacts on the patient level classification results which should build on top of both the patch and slide level, this is seen when comparing the performance of the patient level classification results of the reimplementation to the original. In addition as shown in Table 1 because the reported metrics are poorly specified, in particular the dataset for the patient level classification, this can affect the patient level results as described in 5.1.5.

5.2.3 Liu.

A comparison of the results reported in Liu et al. [21] and the reimplementation is shown in Table 4. As shown in Table 4 the slide classification results reported in Liu et al. [21] are close for the validation set but not for the testing set. This can be due to different distribution of data between validation and testing sets in Liu et al. [21] and the reimplementation. The patch classification parameters are not fully specified in Liu et al. [21] and therefore this can impact the results for the reasons described in Section 5.1.3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Comparison of original and reimplementation results of Liu paper.

https://doi.org/10.1371/journal.pdig.0000145.t004

The lesion level classification results in the reimplementation are poor because lots of noise is detected as lesions which gives lots of false positives. This can be the result of our patch classification not being as good as that of Liu et al. [21]. This is due to the reason described in the previous paragraph.