Introduction

Melanoma diagnoses have increased in recent decades [1] and melanoma is the fifth most common cancer in the United States [2]. In spite of its relatively high frequency, melanoma is often difficult to be histopathologically differentiated from nevi, a high diagnostic discordance rate having been reported even among experienced histopathologists [3]. If a melanoma is initially misclassified as nevus and therefore diagnosed at a later stage, the patient’s chances of survival could be significantly reduced and therapy will probably have to be more intense. On the other hand, if harmless benign lesions are diagnosed as melanoma, the patient will suffer an unnecessary psychological and physical burden. In individual cases, overdiagnosis can even lead to unnecessary, expensive and stressful therapies, which can also be associated with high costs in the healthcare system and unnecessary toxicity for affected patients [4]. More precise diagnostic options could contribute to overcoming these problems.

Due to rapid technological advances of the last few years, AI-based assistance systems may become powerful tools for pathological cancer diagnostics. Deep Learning (DL) with Convolutional Neural Networks (CNN) has shown promise in studies aimed at distinguishing melanomas and nevi on digitized hematoxylin and eosin (H&E)-stained whole slide images (WSI) [5, 6]. In some cases, the DL approach could even outperform humans [7]. However, accuracy of these classifiers especially on external data still shows room for improvement.

In addition to standard H&E-stained slides, immunohistochemical (IHC)-stained tissue sections are often available for many cancer entities and represent a source of complementary prognostic and/or predictive information in addition to H&E-stained tissue. However, the analysis of IHC-stained slides by DL models is a relatively new area of research. Recent studies, however, have employed DL for successful classification of non-skin cancer entities, i.e., to determine HER2 status in breast cancer [8] and immune cell multistains as prognostic and predictive biomarkers in colorectal cancer [9] on IHC-stained slides. Moreover, as shown in previous work, the fusion of different data modalities often improves generalizability and performance of DL models [9–11].

IHC-stains routinely used by pathologists to better differentiate other, usually benign lesions from melanomas are MelanA (MART-1) [12], HMB-45 [13], Ki-67 [14], tyrosinase [15], S100 [16] and PRAME [17]. The expression of melanocyte antigen MelanA (also called melanoma antigen recognized by T-cells 1 (MART1)) is a lineage-specific melanocytic marker which is commonly used by histopathologists for routine diagnosis of melanocytic neoplasms, since it highlights the cytomorphology and the distribution of melanocytes.

IHC expression of MelanA can be automatically analyzed using state-of-the-art artificial intelligence (AI) methods. In this study, we investigate the use of DL-based image analysis models on MelanA-stained tissue for melanoma classification in comparison and in addition to the standard H&E-based diagnosis.

Materials and methods

The presented study investigates melanoma suspicious lesions based on dermatoscopic investigation, which were verified histopathologically as melanoma or nevus. We use DL models to classify whether a lesion is a melanoma or a nevus based on MelanA or H&E stained tumor tissue or a combination of both stains.

Ethics approval was obtained from the ethics committees of the technical university in Dresden. Patients provided informed written consent. This work was performed in accordance with the Declaration of Helsinki [18].

Datasets

The inclusion criteria to participate in our study was to be 18 years old with melanoma-suspicious skin lesions that were biopsied after dermoscopic examination. Suspicious lesions that were pre-biopsied or located near the eye, under the fingernails or toenails were excluded. The ground truth labels were histopathological confirmed by at least one reference dermatopathologist investigating at least the H&E-stained reference slide. MelanA (MART-1) [19, 20] immunohistochemical (IHC) and Hematoxylin and Eosin (H&E) stained tissue slides from the university hospital in Dresden were used for training, validation and hold-out testing. Slides from the university hospital in Erlangen and from the National Cancer Institute of Naples were used for out of distribution (OOD) testing. Table 1 describes the population of all three cohorts. The Dresden, Erlangen and Naples cohorts were collected prospectively, with data collection starting in April 2021 and ending in April 2023, participants provided informed written consent. Data was physically transferred in batches. Data received before 2023 from the university hospital in Dresden was used as a training set, data received later was used as a holdout test dataset. The labels of the datasets were pathologically verified. All 3 cohorts differ in the stains of MelanA slides. Antibodies from different manufacturers with different dilutions were used at each site (Table 2). Representative WSI thumbnails from all 3 cohorts and for both stains are shown in the supplements in S1 Fig.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Description of the population in our datasets.

For continuous features we report median, range, and number of NAs, for categorical features we report the total number of observations per group. Here the training population as well as all three test populations are described. Melanoma in situ describes the early stage of a malignant melanoma that has not yet broken through the basement membrane. However, features at the cellular level do not differ between melanoma in situ and malignant melanoma.

https://doi.org/10.1371/journal.pone.0297146.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Antibodies and parameters of staining methods used by the different clinics.

https://doi.org/10.1371/journal.pone.0297146.t002

Models

To classify pigmented lesions between melanomas and nevi, the ResNet architecture introduced by He et al. [22] was selected as a model for all data modalities. The hyperparameters of the different models were tuned individually using the Bayesian optimization framework Optuna [23] and five-fold cross-validation, all the models where load from the timm library [24] To avoid overfitting with respect to slides containing a huge amount of tiles, we used weighted sampling to train with a predefined amount of tiles per slide in all epochs. The hyperparameters we tuned were the size of the ResNet, the learning rate, the number of training epochs, the type of pooling, the number of tiles used per training epoch and whether or not the initialized ResNet was pretrained on ImageNet. The parameterization of all models is shown in the supplements in S1 Table.

The slide prediction procedure for the different image modalities is as follows: The models were trained at the tile level (with resolutions of 0.25, 0.5, 1.0 and 2.0μm/px), using the slide label for each tile of the slide. All tiles of the slide were predicted and the slide score was calculated by averaging all tile scores (Fig 1). To train models capable of handling domain shifts, the color jitter augmentation package of PyTorch [25] was used as part of the training process, as mentioned by Tellez et al. [26]. In contrast to H&E stained slides, features of protein expression can be distributed over a larger area in the cytoplasm. For this purpose, different magnifications were used to analyze these larger features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic diagram of the different models.

The red box shows the pipeline for MelanA-stained WSIs and the purple box the pipeline for H&E-stained WSIs. We tessellated MelanA-stained WSIs corresponding to different magnifications and trained individual models on each tile size. The class probabilities for each tile were predicted and aggregated into a slide score by averaging all tile scores. For the H&E-based model we proceeded in the same way.

https://doi.org/10.1371/journal.pone.0297146.g001

Results

W The unimodal H&E classifier is based on previous works [5, 10, 28–30] and was adapted to the corresponding MelanA resolutions. The MelanA-models were combined and fused with the H&E model into one multi-modal classifier. All described models were tested within internal distribution (InD) on the Dresden holdout set, and OOD on the Erlangen and Naples cohorts (see Table 1). AUROCs and bootstrapped CIs for all models are shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. AUROC values as well as 95% bootstrapped CIs for the three test cohorts and all evolved models.

https://doi.org/10.1371/journal.pone.0297146.t003

All results differ significantly from random guessing since no CI contains 0.5, the critical value. Thus, we are able to classify melanoma on all evolved MelanA-based models as well as with the benchmark H&E-based model as well. Beside this, it should be highlighted that almost all models on all cohorts perform with a AUROC significantly better than 0.7 which makes findings probably relevant for clinical practice. However, note that CIs overlap in several cases, indicating that different models perform similarly and thus, probably contain a high amount of shared Information.

In addition, we investigated another hierarchical approach motivated by clinical practice, using only MelanA-stained slides for cases where the H&E-based model is uncertain.

The ROC diagrams of the MelanA-based, the H&E-based, and the combined models for all three cohorts are shown in Fig 2. Another representation of this plot, to better compare models within one cohort is shown in the supplements in S4 Fig. Additional ROC plots of the individual MelanA models, which consider only one magnification, and results of the hierarchical approach are shown in the supplementary material in S2 and S3 Figs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. ROC plots by data modality with corresponding AUROC values.

The different subplots show results for the individual evolved models: A: MelanA-based performance B: H&E-based performance taking all magnifications into account C: combined model using H&E as well as MelanA by aggregating the individual scores. The different colors of the ROC curves show from which data source site the results come: Red: internal results (Dresden), Blue: external results (Erlangen), Purple: external results (Naples).

https://doi.org/10.1371/journal.pone.0297146.g002

Discussion

In this work, we were able to predict melanoma/nevi classification across multiple datasets on MelanA slides with a similar accuracy as on benchmark H&E slides using DL-based image analysis. Furthermore, the results may suggest that the multistain approach has the potential to improve prediction accuracy and robustness, since at least on both external cohorts the combined model reached the highest AUROCs.

To integrate the presented work into clinical practice, a method for AI-pathologist interaction needs to be developed. For this purpose, we are developing an Explainable artificial intelligence system in collaboration with dermatologists [31], which produces easily interpretable explanations based on dermatoscopic images and aims to be integrated as an AI tool into digital pathology and clinical practice. Such a system can be expanded to include other data modalities such as immunohistochemistry or routine histology.

In clinical practice, pathologists often use H&E-stained tissue sections for melanoma diagnosis and resort to IHC-stained tissue in uncertain cases [32]. While DL-assisted detection of melanoma on H&E sections has been well studied [7], few studies have been performed using additional routine IHC-stained slides. Digital image analysis by automated quantification of the proliferation marker Ki-67 was used to distinguish melanoma from nevi as a diagnostic and prognostic aid [33]. Recently, an improved DL annotation method for H&E/SOX10 dual stains was developed to better identify tumor cells in cutaneous melanoma [34]. In the study presented here, MelanA-stained tissue was selected as an additional diagnostic tool since it highlights the cytomorphology and the distribution of melanocytes, thereby allowing a more accurate evaluation of the architecture of any melanocytic tumor, along with the size and the shape of single cells. Other IHC stains such as HMB45, p16, and PRAME were excluded because they are useful only in selected cases. However, SOX10 was not chosen because it is a nuclear marker and gives no idea about the actual size of melanocytes and about the morphologic features of their dendritic processes. Finally, Ki67, although largely used in routine, is of little help in the recognition of in situ and early invasive melanoma [13–16].

In the current pathological routine, IHC markers including MelanA are used heterogeneously in different hospitals and laboratories. At the university hospital in Dresden, generally all dermatologically melanoma-suspicious skin lesions are stained with MelanA, providing an unbiased training dataset for our study. In contrast, the OOD datasets likely contain more challenging lesions since MelanA-stained tissue was only prepared at the university hospital in Erlangen in case the H&E-stained slides provided uncertain pathological results. The Naples dataset contained 40% in situ melanomas, all of which are small in size and generate few tiles, making classification in general potentially difficult.

The Dresden test set apparently does not benefit from the inclusion of additional data (S4 Fig), since the H&E-stained tissue slides are already sufficient to yield maximum accuracy. This may be due to the rather unambiguous dataset and thus, the very high performance and a broad data set with many subclasses. In contrast, the OOD datasets benefit from incorporating the additional MelanA-stained slides, making the classifier externally more robust. A combined classifier thus provides an advantage here, a finding we have already made in predicting BRAF status using H&E, clinical and methylation data in melanoma [10] suggesting that a multi stain based classifier can lead to better generalizability. Although the information contained in the H&E- and the MelanA-stained slides is probably partially redundant, one can still see a benefit of combining both stains on OOD data.

A detailed analysis of the various misclassifications revealed that it was not individual histopathological features that caused the model to underperform, but mainly technical artifacts such as overlapping section fragments, low staining intensity combined with strong pigmentation and very small lesions that resulted in only a small number of tiles. This also explains why the MelanA+H&E classifier did not perform better in the Dresden cohort. The simple H&E classifier is already sufficiently good; a combination in the Dresden hold out dataset does not lead to any improvement in individual cases, as there are individual unseen technical coloring artifacts here. This could be avoided by increasing the amount of data in the training set or by excluding the staining artifacts. In clinical pathology, MelanA staining is used in parallel as well as in addition to H&E staining to perform melanoma and nevi classifications. The combined model design was conceived as a parallel evaluation, whereby the model has the greatest possible information at its disposal.

Due to the cytoplasmic distribution of the MelanA protein, tiles from a higher magnification can potentially be too small to extract all relevant features. Pathologists frequently investigate the MelanA stains at lower magnifications to evaluate the silhouette and overall architecture of the lesion, which also contains valuable information. Our data could not show that there is an identifiable best magnification. However, each magnification seems to contain partly different information as the combination of all 4 magnifications brings a slight overall improvement, which can be attributed to ensembling.

Contrary to clinical practice, the hierarchical approach did not lead to any improvement on external datasets. This shows that an unbiased dataset is preferable for training a DL model, since the network can make better decisions with larger datasets. Interestingly, the uncertain Dresden specimens are lesions with large diameters of 8 mm to 17 mm, where a melanoma has developed in the center of a nevus, with melanoma features smoothly merge into nevus features which probably confuses the model, as all tiles are weighted equally in our model. In contrast, the uncertain lesion in Naples is very small with a diameter of <1.0 mm.

Conclusions

With DL analysis of MelanA-stained tissue, we were able to classify melanomas and nevi in two distinct OOD cohorts with similar accuracy as with H&E-stained tissue. The numerically, but not statistically significantly, better classification results achieved by combining H&E and MelanA classifiers suggests that the combination of these image modalities may lead to improved generalizability and performance. However, these results need to be confirmed in larger studies containing more lesions.

Supporting information