2.1. Study design and participants

We included patients aged 18 years or older who underwent skin biopsies at Yonsei University Wonju Severance Christian Hospital between January 1, 2001, and December 31, 2023, with histopathological findings suggestive of or consistent with GR or LMDF. Subsequently, clinical and histopathological findings were reviewed to distinguish between GR and LMDF, with the presence or absence of caseous necrosis in histopathological examinations serving as the key criterion for differentiation. Patients were excluded if clinical photographs were unavailable or of inadequate quality or if the biopsy site was not located on the face. Additionally, using histopathological specimens, the performance and results of Mycobacterium PCR tests, as well as the performance and results of interferon-gamma release assay (IGRA) on blood samples, were evaluated for each patient. Data for this study were accessed on July 1, 2024, for research purposes. Although clinical photographs were included, all images were anonymized by obscuring the eyes and mouth to prevent participant identification. The authors did not have access to any personally identifiable information of the participants during or after data collection. The study protocol was reviewed and approved by the Institutional Review Board of Yonsei University Wonju College of Medicine (approval no. CR324042). The patients in this manuscript have given written informed consent to the publication of their case details.

2.2. Data set

The GR-LMDF dataset consists of a total of 257 image samples, including front-view and both side-view photographs of patients with GR and LMDF (S2 Fig). To ensure anonymization, the eyes and mouth were obscured in all clinical photographs, and only areas containing skin lesions were cropped, resulting in 797 patch images.

2.3. Neural network model

In this study, five convolutional neural network (CNN) architectures [12–16] and two ViT models [17] were utilized to classify images of inflammatory skin lesions into two categories, GR and LMDF.

S1 Table provides a brief overview of the deep learning network models used for comparison.

All CNN and ViT models used in this study were initialized with pretrained weights from the ImageNet dataset [18] and subsequently fine-tuned on our GR-LMDF dataset using a transfer learning approach. This method leveraged the general feature representations learned from large-scale natural images while adapting the models to classify inflammatory skin lesions.

A weighted cross-entropy loss function was chosen for this study to address class imbalance [19]. All models were trained for 100 epochs. For CNN architectures, the mini-batch size was set to 24 with a learning rate of 0.001, while for ViT architectures, the mini-batch size was set to 8 with a learning rate of 0.00001. An adaptive moment estimation (Adam [20]) optimizer was used for all models. The models were trained with a fixed patience value of 10, utilizing the ReduceLROnPlateau scheduler (Pytorch [21]) to adjust the learning rate dynamically. The data augmentation techniques, including random rotations within a range of −30 to +30 degrees, horizontal and vertical flipping, and random adjustments to brightness and contrast, were applied to all images in the training set to enhance performance.

2.4. Multiple instance learning (MIL)

Training on individual patches achieved an average accuracy of 89%. However, the process of cropping and labelling each patch is not only time-consuming and costly but also differs distinctly from how clinicians typically perform diagnoses. To address this, a transformer-based correlated MIL (TransMIL) approach was adopted to classify GR and LMDF using entire images [22]. This method generates patches from the whole image and learns interactions between patches to predict the label of the entire “bag” [23]. TransMIL utilizes the ViT model to divide images into multiple patches and processes each patch as an independent sequence element. Positional encoding is added to each patch to preserve positional information, which is then fed into the Transformer Encoder. The Transformer Encoder, consisting of multi-head attention mechanisms and feedforward networks, learns the interactions between patches and assigns weights to highlight the most relevant ones. The core of TransMIL lies in its attention-based MIL approach [24], where the relationships between patches are learned, and each patch’s contribution to the overall “bag” is weighted. This approach allows the model to emphasize critical patches and ultimately predict the label of the entire image. Additionally, it effectively differentiated GR from LMDF by accurately identifying lesion areas, enhancing classification accuracy for the entire image.

2.5. Evaluation

All architectures were compared under identical conditions, with no additional preprocessing or postprocessing applied, except for resizing the input images to the dimensions required by each architecture. For example, all images were provided as RGB color images and resized to match the input dimensions required by each architecture: 224 × 224 for ResNet-50 and 299 × 299 for Inception-V3. The dataset was split into 80% for training and 10% each for validation and testing. The model that achieved the best performance during the validation phase was used to evaluate the test set.

The performance of each model was evaluated using metrics such as accuracy, precision, sensitivity, the F1 score, and Cohen’s kappa. In this analysis, GR was designated as the negative class, and LMDF as the positive class. Accuracy refers to the proportion of correctly classified cases relative to the total number of cases in the dataset. Precision measures the proportion of positive predictions that were true positives, while sensitivity (or recall) represents the proportion of actual positive cases correctly identified. The F1 score, combining precision and sensitivity, is calculated as their harmonic mean. Cohen’s kappa score measures the agreement between evaluators while adjusting for chance agreement [25], providing a more reliable assessment than simple percent agreement. It is particularly useful in scenarios with imbalanced data or multiple categories. Additionally, a confusion matrix was constructed from the results of the testing images to evaluate the performance of the MIL.

2.6. Clinician-MIL augmented diagnosis

We compared the accuracy of clinicians with different levels of expertise and that of the MIL model in the differential diagnosis of GR and LMDF. Additionally, we assessed whether clinicians’ accuracy and diagnostic time improved when they referred to the MIL-predicted diagnoses on the same test set. The evaluation included three board-certified dermatologists, five dermatology residents, and two general practitioners. In the first assessment (clinical diagnosis), participants were presented with a clinical test set consisting of front-view and side-view photographs (a total of three images per patient) from 16 GR patients and 14 LMDF patients presented in random order. The clinicians were asked to select the most likely diagnosis between GR and LMDF based solely on the clinical photographs.

The second assessment (clinician-MIL diagnosis) was conducted 5 months after the first assessment. Clinicians were provided with the TransMIL model’s diagnostic predictions, including its accuracy and confidence levels, for the same clinical test set. Based on this information, clinicians were asked to make their diagnoses. Subsequently, the accuracy and diagnostic time of each clinician from the first and second assessments were compared.

2.7. Statistical analysis

The baseline characteristics of the study population are presented as means with standard deviations or frequencies with percentages, according to the type of variable. Differences were evaluated using an independent two-tailed t-test or a χ² test, as appropriate. Fisher’s exact test was used in cases with small sample sizes and frequencies. The results of the first and second evaluations were compared using a paired t-test. All statistical analyses were performed using R statistical software (version 3.6.3; R Foundation for Statistical Computing, Vienna, Austria) at a significance level of 5%.

3.1. Study population

A total of 107 patients who underwent skin biopsies were diagnosed with GR or LMDF between January 1, 2001, and December 31, 2023. After applying the exclusion criteria, 96 patients were included in the analysis, comprising 62 GR patients and 34 LMDF patients (Fig 1). The clinical, histopathologic, and laboratory characteristics of the patients are summarized in Table 1. Eyelid involvement was significantly more common in patients with LMDF (70.6%) than in those with GR (29.0%) (p < 0.001). Likewise, caseating necrosis was observed in 76.5% of LMDF cases and in none of the GR cases (p < 0.001). While LMDF patients demonstrated higher positivity rates for Mycobacterium PCR and IGRA tests, these differences were not statistically significant. A total of 171 clinical photographs were obtained from GR patients and 34 from LMDF patients, which were cropped into 797 patch images for use in neural network model training.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Clinical, Histopathologic, and Laboratory Characteristics of Patients with Granulomatous Rosacea or Lupus Miliaris Disseminatus Faciei.

https://doi.org/10.1371/journal.pone.0326763.t001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of the study.

A total of 62 patients with GR and 34 patients with LMDF were included. The 797 patch images and 257 clinical photographs were utilized for the neural network models and TransMIL, respectively. GR, granulomatous rosacea; LMDF, lupus miliaris disseminatus faciei; TransMIL, transformer based correlated multiple instance learning.

https://doi.org/10.1371/journal.pone.0326763.g001

3.2. Comparison of the neural network model

First, we evaluated the ability of each neural network model to differentiate between GR and LMDF lesions. The performance metrics of each model, including accuracy, precision, recall, F1 score, and kappa score, are presented in Table 2. When trained on cropped lesion images, the transformer-based model, ViT_base_patch16_224, demonstrated the highest accuracy at 93.0%. Furthermore, ViT_base_patch16_224 showed the highest performance in both the F1 score, which reflects the balance between precision and recall, and the kappa score, which indicates the reliability of agreement. The transformer-based model, ViT_base_patch16_224, highlights its superior ability to balance diagnostic precision and consistency compared to other models. Meanwhile, when using the whole image without marking lesion areas and applying a TransMIL model with uniformly sized patches, the accuracy was 70% (Fig 2).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Accuracy, Precision, Recall, F1 Score, and Kappa Score for the AI Models used in this Study.

https://doi.org/10.1371/journal.pone.0326763.t002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Performance of multiple instance learning in the differentiation between granulomatous rosacea and lupus miliaris disseminatus faciei.

Confusion matrix shows that the accuracy was 0.700, outperforming the clinician’s diagnostic accuracy of 0.647.

https://doi.org/10.1371/journal.pone.0326763.g002

3.3. Clinician-AI-augmented diagnosis

The evaluation conducted on a clinical test set comprising 30 patients showed that the mean accuracy for clinical diagnosis was 64.7% ± 6.3%, whereas clinician-MIL diagnosis achieved a statistically significant improvement to 70.3% ± 4.7% (p = 0.0136). When analyzed by expertise level, the mean accuracy for clinical diagnosis was 68.9% ± 3.8% for board-certified dermatologists, 64.7% ± 3.8% for dermatology residents, and 58.3% ± 6.3% for general practitioners. With a clinician-MIL diagnosis, the accuracy improved across all levels of expertise: 72.2% ± 1.9% for board-certified dermatologists, 69.3% ± 6.4% for dermatology residents, and 68.3% ± 4.7% for general practitioners. Notably, the most substantial improvement was observed in general practitioners, who had the lowest level of expertise (Fig 3). The mean diagnostic time decreased from 10.7 minutes in the first assessment to 6.4 minutes in the second assessment, indicating a reduction in diagnostic time with AI assistance (p = 0.1591).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Clinical diagnosis and clinician-AI augmented diagnosis.

Clinical diagnosis refers to the first assessments made by three board-certified dermatologists (red), five dermatology residents (green), and two general practitioners (blue) on an independent clinical testing dataset. Their second assessment, incorporating assistance from multiple instance learning (MIL), is labelled as the clinician-MIL diagnosis. The dotted lines illustrate the diagnostic accuracy of each clinician during the first and second assessments, while the solid lines represent the average diagnostic accuracy across clinicians with different levels of expertise.

https://doi.org/10.1371/journal.pone.0326763.g003