Ethics statement

This study has been approved by the Institut National de la Santé et de la Recherche Médicale (INSERM) ethics committee (Institutional Review Board IRB00003888, approval reference 21-761) in March 2021. This study was conducted in accordance with the principles of the Declaration of Helsinki and all patients or their legal representatives (for minors) provided written informed consent.

2.1. Related works

Virtual staining has been proposed to enable efficient transformations between different types of stains for WSIs. In recent years, several improvements have been made in generating multiple stains using adversarial generative networks. However, substantial issues remain regarding scalability, accuracy, trustworthiness, and accessibility to clinicians [5,10]. This section explores the existing body of literature with an emphasis on the H&E to IHC transformation process, aiming to highlight these current limitations and provide a comprehensive understanding of the landscape in which our research is located.

2.2. Data

In this study, we introduce a rigorously curated dataset that is crucial for our research. This dataset was acquired at Robert Debré Hospital in Paris within a study on pediatric Crohn’s disease.

Population: The study focuses on pediatric and adult patients diagnosed with Crohn’s disease according to the ESPGHAN criteria (European Society for Pediatric Gastroenterology Hepatology and Nutrition). These patients were followed at Robert Debré Hospital for at least one year and had an initial biopsy at the time of diagnosis. The study includes all patients diagnosed at the center from 1988 to 2019. Patients for whom whole slides were too old were removed, resulting in one or multiple slides available for 59 patients. This population includes predominantly male individuals (69%) and has a mean age of 11.11 years (standard deviation 3.64).

Dataset description: The dataset comprises a total of 480 whole-slide images (WSIs), evenly distributed across eight paired combinations of H&E and IHC stains. Each combination includes 30 matched pairs of an H&E slide and its IHC-stained counterpart, featuring the following markers: Anticytokeratin AE1/AE3 (AE1AE3), CD117 (c-Kit), CD15 (Lewis X or SSEA-1), CD163 (macrophage marker), CD3 (T cell co-receptor), CD8 (T cell co-receptor), cluster D2-40 (D240), and Giemsa stain. These 480 slides were derived from 59 patients, with 30 matched pairs (H&E and IHC from the same tissue region) available. All slides were scanned at a uniform magnification of 40× using the same scanner, with a resolution of 0.22 per pixel to ensure consistency.

For the experiments, the dataset was randomly divided into two subsets: 20% for testing and 80% for training. This split was performed at the slide level and considered the amount of tissue per slide, ensuring that each slide contained at least 10% tissue coverage (see S1 Text for exact slide IDs used for testing). Both the paired and unpaired training settings utilized the same total number of WSIs. In the paired setting, the data loader ensured a strict correspondence between an H&E slide and its IHC counterpart from the same tissue region during training. Conversely, in the unpaired setting, no such correspondence was maintained, and the slides were presented independently. Since the dataset itself remained unchanged, any observed performance differences arose solely from how the data was structured and presented to the model. The unpaired setting introduced a form of data augmentation by allowing the model greater flexibility, potentially reducing biases inherent to tissue-specific staining patterns and contributing to the improved performance observed in this setting.

Additionally, for S8 Fig, we used a separate in-house dataset comprising H&E-stained slides only. This dataset, which cannot be made publicly available due to privacy concerns, consists of a cohort of pediatric patients diagnosed with Crohn’s disease and followed at Robert Debré Hospital in Paris, France. The biological material included paraffin-embedded tissue blocks obtained from diagnostic gastrointestinal biopsies collected as part of routine clinical care. A total of 2022 slides were available for analysis. Each slide (5 μm thick) was stained with H&E to obtain high-quality sections suitable for scanning. All slides were digitized using a Hamamatsu slide scanner at the pathology department of AP-HP and were semi-anonymized prior to analysis.

2.3. Multi virtual staining model architecture and training methodology

Our research focuses mainly on adapting two notable neural network architectures, ComboGAN [25] and CycleGAN [17], for a specific application: transforming H&E-stained histological slides into various other types of stain. The core architectural framework of our approach includes key components such as an encoder E_i, a generator G_i, and a discriminator D_i, where i denotes the index representing each unique stain type within the set {1,..., S}. A crucial contribution of our method is the utilization of shared (unique) H&E-specific encoder, generator, and discriminator in all S stains, denoted as , , and , respectively. This strategic choice aims to improve the efficacy and specificity of the transformation process.

Synthesis training: The training methodology we employ is based on a dual cycle process aimed at transforming and reconstructing stains in histopathology slides. This approach focuses on seamlessly converting between H&E stained tiles and various other stain types, while ensuring preservation of core structural features throughout transformations.

In the first cycle of this process "H&E cycle", we start with an H&E-stained tile (refer to S5A Fig). This tile is first processed by an encoder specific to H&E stains, denoted as , transforming it into a latent representation . This latent representation is then fed into a generator G_i, corresponding to the type i of target stain, producing an image that mirrors the characteristics of the desired stain. To complete the cycle, this generated image is passed through another encoder E_i, obtaining a new latent representation , which is subsequently used by the H&E generator to recreate an H&E-stained tile . Mathematically, this reconversion process, which ensures the cycle from an H&E stain to a target stain i and back to H&E, can be encapsulated as follows:

(1)

(2)

In the second cycle "stain i cycle", we address the reverse process: starting with a tile originally stained with a specific type i, our objective is to convert it into an H&E-stained representation and subsequently revert it to its original stain (refer to S5B Fig). Initially, tile X_i undergoes encoding via E_i to produce a latent representation Z_i. This is then converted into an H&E-stained tile by the H&E generator , effectively translating X_i into the H&E domain. The resulting H&E image is re-encoded by into a new latent representation , which serves as input for the generator G_i, reconstructing the original stained image . This enables the conversion of any specific stain type to an H&E representation and back, formulated mathematically as:

(3)

(4)

To effectively train our architecture, we introduce a comprehensive global synthesis loss, which we denote as . This loss facilitates the translation from H&E stained images to a specific target stain, represented by i. The process involves comparing the reconstructed image and the target image X_i for the specified stain i, as well as comparing the reconstructed H&E image and the original H&E image . These comparisons are used to estimate the cycle loss for each stain i, denoted as , which quantifies the fidelity of translation between the H&E stain and stain i in a bidirectional manner.

In addition to the cycle loss, our model also incorporates an adversarial loss, . This component utilizes discriminators in inference, specifically for the H&E stain and D_i for the target stain i, to assess the authenticity of the generated images and . The discriminators aim to distinguish between real and synthesized images, thus encouraging the generation of images that are indistinguishable from genuine stained samples. Furthermore, the model incorporates a regularization loss, , which helps the model converge. The overall synthesis loss function for each specific stain i, out of a total of S stains, is given by the equation:

(5)

Here, , and are weighting coefficients. This approach ensures a versatile and effective training regime that can accommodate a range of stains, denoted by S, improving the model’s ability to generalize across a wide range of stains.

Discriminator training: In our approach, we employ two types of discriminators: one for the H&E stains, denoted as , and individual discriminators for each IHC stain, represented as D_i for the i^th stain. Each discriminator is trained using its respective loss function to optimize its performance.

For the H&E discriminator, , the loss function is defined as follows:

(6)

This equation represents a combination of the losses and from real and synthetic H&E stained images respectively (refer to Eq (15)), scaled by a weighting factor , to measure the discriminator’s performance in distinguishing between genuine and artificially generated H&E images.

Similarly, for each stain discriminator D_i, the loss function is customized to assess its ability to discern real images from synthetic stained images of that particular type, defined as:

(7)

Here, and correspond to the losses from real and synthetic images of the i^th stain, respectively (refer to Eq (14)). The sum of these losses, weighted by , constitutes the total loss for that discriminator, ensuring that it effectively learns to differentiate between actual and generated samples of its specific stain. This structure is applied in all S stains, allowing discriminators to specialize in their respective stains for improved performance in identifying authentic images versus generated images.

2.3.1. Integrating annotation-free knowledge through loss function optimization.

In the development of computational models for synthesizing IHC slides, it has become evident that traditional distances such as , , and mean square error (MSE) pose significant challenges. A key issue with these metrics lies in their indiscriminate treatment of different regions on the slides, as they fail to differentiate between tissue sections and areas activated by IHC staining. This becomes particularly problematic due to the inherent staining imbalance in IHC slides, where the vast majority of the slide is IHC-negative, underscoring that IHC staining is specific to only a small fraction of the total slide content.

To address these shortcomings, our approach harnesses the cycle consistency loss and adversarial loss [17,25]. Building on these foundations, we propose a novel method that integrates IHC-activated areas into the model’s training process for synthesizing S stains from H&E stained slides. This strategy facilitates the extraction of annotation-free knowledge, seamlessly incorporating the distinctive characteristics of each stain into the model. Consequently, our model can generate features with enhanced fidelity, thus improving the overall quality of synthesis. By guiding the synthesis process with stain-specific insights, our method not only enhances performance but also the transparency and trustworthiness of the synthesized images.

To automate the extraction of a precise mask for the IHC-activated region, denoted M_i, from the i^th IHC-stained image, X_i, we begin by transforming X_i from its original RGB color space to the HSV color space. This conversion is crucial because it significantly enhances the ability to isolate the target regions based on their color and brightness attributes. After converting the image to the HSV color space, we apply a threshold to isolate a distinct mask, M_i, which is shown in S2 Fig. This mask M_i is used to calculate the dynamic weighting factors, and , which are integrated into the computation of the loss function. These factors are defined as follows:

(8)

In this equation, N_i represents the total number of pixels in the background, which corresponds to the IHC-activated regions, and represents the total number of pixels in the background, or the non-activated IHC regions, of the i^th IHC-stained image. This approach enables nuanced differentiation between the regions of interest and their background, facilitating more accurate analyses.

Integrating knowledge during the synthesis training phase (cycle loss): Integration of knowledge during the synthesis training phase involves a detailed process that utilizes both H&E stained tiles and IHC images from specific stains. This process is important in creating accurate high-quality synthetic images that mirror the unique characteristics of each stain.

In scenarios where direct pairs of H&E-stained images and IHC images are not available, an unpaired setting approach is used. This uses a specifically designed cycle loss, , to facilitate synthesis in the absence of paired images. The cycle loss formula for the unpaired setting is as follows:

(9)

This approach meticulously focuses on the differentiation between IHC-activated regions (M_i) and non-activated regions (), ensuring the integrity of the synthesis process despite the absence of direct image correspondences (refer to S5 Fig).

In contrast, in paired settings where there are direct correspondences between H&E and IHC images exist, the cycle loss is formulated to ensure both the overall fidelity of the image reconstructions and the accurate replication of specific IHC-activated regions while taking advantage of the direct correspondence (see S4 Fig). The comprehensive equation for the paired setting is the following:

(10)

Both settings employ dynamic weighting factors, and , determined by the ratio of IHC-activated regions to non-activated regions within each image. This careful consideration ensures that the model’s training prioritizes not only the overall accuracy of stain transformation but also the faithful replication of regions crucial for IHC analysis. Furthermore, the methodology incorporates masks M_i and in the calculation of the cycle loss, enhancing the model’s capacity to embed the distinctive characteristics of each stain directly into its architecture. This meticulous approach facilitates the integration of annotation-free knowledge, resulting in the production of high-quality, reliable synthetic images that accurately reproduce the complexities of IHC staining.

Integrating knowledge during the synthesis training phase (adversarial loss): In the context of the unpaired setting, the adversarial loss, denoted , is calculated by evaluating the authenticity of the generated images and through discriminators D_i and , respectively. This process is enhanced by using the stain mask M_i, which helps focus on the areas activated by the IHC, as shown in S5 Fig. The formula for computing adversarial loss in unpaired settings is as follows:

(11)

For paired setting, the approach mirrors that of the unpaired setting but with an added emphasis on the direct correspondence between the stain and image i, as shown in S4 Fig. The computation of adversarial loss for paired settings incorporates this direct correspondence and is expressed by the following equation:

(12)

Direct supervision loss (paired setting): In the paired setting of our framework, the synthesis loss is meticulously designed to include a variety of components, notably the supervised loss (), alongside the cycle consistency loss () as mentioned in Eq (10) and the adversarial loss () as detailed in Eq (12). Crucially, supervised loss establishes a direct connection between the H&E cycle and the specific stain cycle i. It does this by evaluating the fidelity of the generated stains and H&E images against their actual counterparts (X_i and , respectively). Furthermore, it incorporates a common mask (M_i), derived from X_i, to concentrate the loss computation on relevant areas of the image. This ensures that the generated images maintain both structural and stylistic integrity in relation to the original samples. The formula for the supervised loss is articulated as follows:

(13)

Integrating knowledge during the discriminator training phase: To effectively train the discriminator D_i, we employ authentic X_i samples to generate corresponding stain masks M_i. These masks enable the discriminator to discern nuances within IHC-activated regions through the loss function. Concurrently, the discriminator is trained to recognize synthetic images as inauthentic using the loss function. The formulation of both loss functions is as follows:

(14)

In a similar vein, for the discriminator, we define and to, respectively, discern the authenticity of H&E stained images and identify synthetic counterparts. These loss functions are described as follows:

(15)

2.3.2. Enhancing training using regularization.

One H&E representation to rule all the staining modalities: The main objective of our approach is to develop a universal H&E encoder and generator capable of handling all staining modalities used in histology. Traditional methods often face challenges due to the varying requirements for different stains during the training phase. Some stains are inherently more complex to replicate, leading to uneven learning progress and possible neglect of less dominant stains. To address this issue, we have implemented a novel regularization strategy. This approach ensures an even distribution of learning focus across all stains by the components of our H&E encoder, thereby minimizing bias towards any specific stain.

The regularization process involves a systematic selection in which the stains are randomly chosen from a complete set identified by numbers ranging from 1 to S, ensuring complete coverage. The updates are then applied to the encoder E_i and the generator G_i for each selected stain i, alongside the updates for the shared H&E components ( and ), ensuring a fair representation of each stain throughout the training cycles. After cycling through all S stains in a random sequence, we calculate a mean synthetic loss across them using the equation:

(16)

This calculated loss, , specifically refines the H&E and , marking the completion of a training iteration. This methodology ensures equal attention to each stain and significantly enhances the versatility of the model in various modalities. As a result, we achieve a more stable and scalable training process, thereby enhancing the model’s overall effectiveness in dealing with a wide range of staining modalities.

Stain synthesis regularization: We propose a comprehensive methodology designed to encapsulate the entire spectrum of considerations for regularization in virtual staining. By integrating critical knowledge through IHC-activated regions captured by the stain mask M_i. This process involves the calculation of a regularized loss, denoted as , formulated as a weighted sum of three principal loss functions: identity (), latent (), and forward () losses.

For every stain index i within the set , the regularized loss is precisely defined as:

(17)

Here, the coefficients and represent the respective weights of each loss component within the cumulative regularized loss framework.

Identity loss () quantifies the deviation between the original and generated images, using the same domain encoder and generator within an autoencoder setup. This ensures that the encoder captures enough features to directly reproduce the input image. This concept is further extended to include M_i as follows:

(18)

The latent loss () aims to mitigate the disparities within the latent space, specifically by capturing the variance between the latent representation and its reconstructed version. This facilitates the alignment of embeddings from both H&E and stain i encoders, incorporating IHC-activated regions as follows:

(19)

Finally, the forward loss assesses the divergence between the degraded versions (represented using the lower case) of the original images () and their corresponding outputs (), specified as:

(20)

By incorporating three distinct loss functions (, , and ) and leveraging knowledge from IHC-activated regions, this regularization approach opens up a wide range of opportunities for fine-tuning the handling of task-specific challenges. This method is particularly designed for unpaired settings, where it contributes significantly to the nuanced management of such tasks. In paired settings, these approaches do not offer substantial benefits due to the direct correspondences between the H&E and the S stains, which are already extensively addressed by supervised loss functions. However, the theoretical potential of integrating these regularization strategies suggests a broader applicability beyond their initial intended contexts.

Note that the degradation process involved downscaling the H&E- and IHC-stained images using an average pooling operation. Specifically, we applied PyTorch’s AvgPool2d(3, stride=2) function, which performs a two-dimensional average pooling with a kernel size of 3 and a stride of 2. This operation reduces the spatial dimensions of the images by computing the average value over each region and moving the window by two pixels at a time, thereby generating a degraded version of the images.

2.4. Trust in virtual stains through self-inspection–anomaly detection

As described in Sect 2.3, our architecture incorporates encoder, decoder, and discriminator components inspired by CycleGAN [17] and ComboGAN [25]. Central to our methodology is the use of a PatchGAN discriminator [17,36,37], which is explicitly designed to differentiate between synthetic and authentic images. This discriminator features dual heads: one that focuses on luminance (emitting a confidence map C_lum) and the other on RGB space (emitting a confidence map C_rgb). Each map rates the authenticity of the image on a clamped scale from -1 (’anomaly’) to 1 (’authentic’), and both maps are resized to match the size of the input image.

To simplify the analysis for pathologists and reduce cognitive load, we combine these two maps into a single map by calculating the pixel-wise minimum of C_lum and C_rgb, resulting in C_all. This combined confidence map C_all is then normalized to a range of 0 to 1, where 0 indicates an anomaly and 1 indicates authenticity. This map can be used to calculate various metrics, such as the standard deviation shown in S8 Fig. Additionally, we apply a Jet-color map using OpenCV version 4.9.0 [38] to transform C_all into an 8-bit unsigned integer RGB confidence map, as illustrated in S9 Fig.

This approach provides comprehensive confidence maps that can identify a wide range of anomalies, as demonstrated in S8 and S9 Figs. Given that the same discriminator architecture is utilized in all stains, this methodology is applicable to all of them.

2.5. Tile-stitching for clean virtual staining WSI generation using a Hamming window-based approach.

To address the inevitable stitching artifacts encountered during the reconstruction of synthetic WSIs, we applied a tailored image processing approach. Central to our methodology was the use of a two-dimensional (2D) Hamming window [39,40], designed to smooth the transitions between adjacent image patches and mitigate edge effects.

The Hamming window, traditionally used in signal processing [39,40] to taper the signal edges, was adapted to two dimensions to suit the image patches. Each patch, representing a portion of the larger image, was processed through this window to ensure a gradual transition at the borders. With an overlap > 0, this was achieved by computing the outer product of a one-dimensional Hamming window with itself, thus creating a symmetric 2D window w(x,y) for a patch of size is defined as:

(21)

where x,y range from 0 to M–1. This results in a 2D Hamming window, which reduces the pixel values towards the edges of each patch. Then, this window was applied across the three color channels of the image. Each image patch (across all RGB channels) was element-wise multiplied by this matrix, reducing the intensity at the peripheries and thereby softening the boundaries between stitched patches. This operation is described by the following equation:

(22)

Where P(x,y) is the original pixel value at coordinates (x,y) within the patch for a given color channel, and w(x,y) is the value from the 2D Hamming window at these coordinates.

After applying the Hamming window, the weighted patches were summed to form the complete WSI. In regions where patches overlapped, pixel values from multiple patches were combined. To ensure uniformity, the accumulated weights of the patches were recorded and used to normalize the pixel values in these overlapping areas. This normalization process was crucial for maintaining consistent intensity across the WSI, preventing visual discontinuities that could hinder the quality of the synthetically stained WSIs. This methodology can be applied to any tile-based virtual staining approach to reconstruct a clean WSI output.

The final processed image was saved in pyramidal TIFF format, suitable for high-quality WSI. The processing pipeline was implemented using Python, using libraries such as NumPy [41] v1.26.3 for numerical operations and PyVIPS [42] v2.2.2 for image handling, ensuring efficient memory usage and scalability.

2.6. Cloud-based platform

We use open-source Cytomine Community Edition Legacy 3.1.0 software to transform our virtual staining model into a web application. It operates based on a containerized architecture using Docker, which facilitates the creation and deployment of Cytomine applications through various modules (applications, web UI, databases, nginx proxy, jobs management, etc.). The core component for the implementation of deep learning-based applications is a software Docker container.

Our Python-based application performing virtual staining is itself Dockerized and uploaded to software_router, where it is transformed into a Singularity image. Our python-based application includes the code for virtual staining, as well as the code to import the input WSI and upload the output virtual stains in the database. For these two last tasks, we use the Cytomine Python API for communication between the software container and the image database container.

The inputs to this Python-based application are specified in a JSON descriptor, also uploaded to the software container, which is then transformed into a user-friendly web interface for users to select the input H&E WSI.

When the user launches the algorithm initiating the virtual staining, its execution is managed by a SLURM-based job scheduling system, which launches the Singularity image with the corresponding inputs. Upon completion, the generated stains could be visualized directly within the Web UI.

Cytomine is optimized for the display of multiple instances of aligned WSIs, allowing for simultaneous visualization of stains. This functionality significantly improves the ability to compare and analyze different staining results within a unified interface, providing a powerful tool for digital pathology and related research fields.

2.7. Experimental configurations

To ensure reproducibility, it is important to note that all the experiments conducted in this study utilized the same architecture for the encoder, decoder, and discriminator. The number of parameters was aligned with those specified in [25] and implemented using the PyTorch library (version 2.2.0 with CUDA v12.1 and cuDNN v8.902) [43]. All training sessions were performed using 2048 x 2048 tiles resized to 512 x 512 tiles (no overlap) from the Crohn’s dataset, as discussed in Sect 2.2, in either paired or unpaired settings. The models employed an Adam optimizer with parameters and , and a batch size of 6. We used only random flip and random rotation (data augmentation strategies). Each training epoch contains 728 iterations. Each training was conducted on a single NVIDIA A100 80GB GPU. In this study, the cycle-consistency loss coefficient () was set to 10, consistent with the original CycleGAN implementation, which demonstrated stable convergence using this value across multiple image-to-image translation tasks, including artistic style transfer [17]. Given its widespread validation in the literature, this hyperparameter was adopted without an extensive hyperparameter search. Nevertheless, optimizing through a systematic strategy (e.g., grid search or Bayesian optimization) may further improve model performance in this specific application and represents a potential direction for future work.

2.8. Study of unified vs. individual H&E encoders in multi virtual staining

In our study, our objective was to develop an enhanced technique for generating multiple virtual stains simultaneously. Existing methods, as discussed in Sect 2.1.1, are limited to producing at most three stains concurrently. Inspired to improve upon these limitations, we utilized style transfer techniques from frameworks such as ComboGAN [25], which can handle up to 14 different art styles. We adapt this approach to histopathological applications, incorporating a novel architecture with a dedicated H&E encoder, generator, and discriminator for the concurrent training of H&E to multiple S stains, as illustrated in Sect 2.3 - S1, S4 and S5 Figs.

Our results, detailed in S1 Table, demonstrate the advantages of using a unified H&E encoder for multi virtual staining. Synthetic stains from this encoder consistently surpassed those of separate encoders for each stain. We evaluated performance using the mean square error (MSE), comparing synthetic stains with authentic counterparts with a paired test set of H&E samples. This comparison revealed that our method significantly outperforms the CycleGAN approach.

The unified and CycleGAN methods were tested under identical conditions, including the same dataset, training duration, and architecture for encoders, generators, and discriminators. Our method not only offers significant gains in computational efficiency by employing a single encoder, decoder, and discriminator throughout the staining process, but also requires fewer trainable parameters than the CycleGAN approach (see S3 Fig). This streamlined architecture boosts computational efficiency and supports scalability, accommodating a broader range of output stains, and accelerating the training process.

In conclusion, the unified H&E encoder method excels in producing more accurate synthetic stains and achieving greater computational efficiency, making it a scalable and effective solution for large-scale histopathological studies.

This approach also enables the development of robust H&E representations for downstream tasks. We evaluated our pretrained H&E encoder on a HER2-positive vs. HER2-negative classification task from H&E patches. Training from our pretrained H&E encoder weights demonstrated faster convergence (20 versus 110 epochs) and superior performance (AUC 0.877 versus 0.840) compared to the model trained from scratch with random initialization. These results highlight the transferability of the learned H&E features, suggesting that the unified encoder captures generalizable tissue morphology and microenvironmental characteristics that are valuable for various applications beyond virtual staining. More details on the classification method are provided in S1 Text. A per-stain evaluation (see S1 Text.) further demonstrated that reconstruction fidelity depends on subcellular localization, with membrane-associated and pan-cellular stains (e.g. D2-40 and GIEMSA) exhibiting high PSNR and SSIM.

2.9. Annotation-free knowledge guided training and overall H&E regularization

To improve the trustworthiness and robustness of virtual staining techniques in histopathology, we developed a novel methodology that incorporates additional constraints into the training model by leveraging information from stained slides. Contrary to style transfer applications in art [23–25], where domains differ significantly, simplifying the discriminator’s task and placing greater emphasis on the generator for accurate image synthesis, functional staining challenges are rooted in common morphological features in all stains, with variations primarily in activation reactions targeting specific proteins. This introduces two main challenges: (i) the model may underestimate regions with activated stains, which are statistically less frequent than non-activated tissue areas, leading to errors in stain generation as the discriminator struggles to differentiate between true activated regions and false negatives generated by the model; (ii) as the number of output stains increases, the encoder may disproportionately prioritize certain stains, potentially skewing the learning process and impeding overall performance.

To address these challenges, our approach incorporates loss functions that automatically recognize stain-specific properties (see S2 Fig), and adaptively modulate the loss functions to emphasize underrepresented activated regions (refer to Sect 2.3, S4 and S5 Figs), thereby minimizing errors and reducing hallucinations. Stain-activated regions are first identified and then used to spatially weight the loss functions, improving the focus on the relevant tissue areas.

Furthermore, we introduce an H&E regularization to maintain balanced attention across various stains, recalibrating the model to equally consider all stains by backpropagating the mean error of generated stains through the H&E components exclusively.

This strategy not only stabilizes the training process but also scales effectively, as demonstrated by the improved results in S2 Table. The combination of with enhances the performance of the model in paired and unpaired settings, ensuring consistent focus across different stains and significantly increasing overall efficacy.

2.10. Context-importance in multi virtual staining quality and scalability

Due to the enormous size of WSIs, which typically measure around 10,000 x 10,000 pixels, current GPUs cannot process an entire slide at once during training. As a result, virtual staining techniques often utilize a sliding window tiling approach. This method involves dividing the slide into smaller, more manageable patches that are compatible with deep learning models and GPU capacities. The common dimensions for these patches are 128 x 128 pixels and 256 x 256 pixels [22,44]. Using this approach requires careful consideration of the optimal magnification level (x10, x20, and x40) for analysis. The choice of magnification impacts the training process in paired and unpaired learning scenarios. Using smaller patches increases the total number of patches per WSI, raising questions about inference time – specifically, the duration required to reconstruct a virtually stained slide. Addressing these technical issues is crucial not only for optimizing performance but also for understanding the practical implications related to inference time, a critical factor for pathologists.

In our empirical experiments, using a modular approach that does not require loading all model parts simultaneously, we were able to process 512 x 512 tiles while simultaneously outputting 8 stains plus H&E during training on a standard 16GB GPU. This setup provides at least four times more spatial resolution than those reported in [22,44], thus offering more flexibility and the ability to incorporate more context within each patch. To determine the optimal magnification level for virtual staining, we trained our model using various magnifications, each resized to a uniform dimension of 512 x 512 pixels for consistent image processing. The specific magnifications tested were x10 (original tile size of 2048 x 2048 pixels ≈ 450.56 x 450.56), x20 (original tile size of 1024 x 1024 pixels ≈ 225.28 x 225.28), and x40 (original tile size of 512 x 512 pixels ≈ 112.64 x 112.64). These experiments were carried out in paired and unpaired learning settings to evaluate the impact of magnification on model performance. As shown in S3 Table, in the paired setting, all magnifications yielded comparable results due to the direct correspondence between the H&E-stained WSI and other stained WSIs. Our experiments in unpaired settings revealed a valuable insight: lower magnifications, which provide broader contextual views, enhance the performances. This suggests that embracing more extensive contextual information is crucial for effective learning – where direct stain correspondences are lacking –, guiding future improvements in virtual staining techniques.

In the paired analysis, we initially utilized a 10x magnification, corresponding to a resolution of 512 x 512 pixels (approximately 0.88 μm per pixel). We further processed the original images by resizing them to 1024 x 1024 pixels (0.44 μm per pixel) to better evaluate the impact of pixel density on high-context paired training. To maximize the full capabilities of high-end GPUs, such as the NVIDIA A100 80GB, we also experimented with a maximum image size of 1400 x 1400 pixels (0.32 μm per pixel) during the training phase.

S5 Table illustrates the scalability of our modular approach, which can successfully process images with eight stains plus H&E up to the resolution of 1400 x 1400. The comparative results in S3 and S5 Tables emphasize that improving contextual information within images proves substantially more advantageous than increasing the density of pixels.

2.11. Regularization techniques impact on unpaired multi virtual staining quality

Most style transfer methods emphasize the importance of regularization in improving and stabilizing the training process. For example, regularization of identity mapping loss is critical in preserving the color of input paintings in artistic applications, as seen in the CycleGAN [17] framework. Similarly, forward loss regularization plays a crucial role in virtual staining by preserving morphological characteristics when translating from H&E staining to other types, as highlighted in the UMDST [44] model. Despite the variety of available regularization techniques, comprehensive ablation studies are lacking to evaluate their effectiveness in virtual staining.

In our research, presented in S4 Table, we conducted an extensive ablation study to evaluate the individual and combined effects of various regularization techniques on the quality of virtual stain synthesis. This study examines different combinations of synthesis loss functions and regularization methods to identify the most effective configurations. The metrics used in the study include MSE, PSNR, and SSIM, which gauge the error, quality, and visual similarity of the synthesized images, respectively.

Our results, as presented in S4 Table, offer a detailed analysis of how various combinations of loss functions—specifically identity loss , latent loss , and forward loss (refer to Sect 2.3.2)—impact key performance metrics such as MSE, PSNR, and SSIM. Each row in the table represents a different combination of these loss functions, illustrating their respective effects on the evaluation metrics. This detailed evaluation provides essential information on the efficacy of each approach.

First, applying the forward loss alone has demonstrated superior results compared to baselines that include or exclude the combination of and . The is particularly effective as it assists the model in preserving the morphological features that are highlighted by the H&E staining, crucial for accurate virtual staining.

Secondly, the best performance is achieved when is combined with . This combination not only preserves the morphological integrity of the stains but also maintains the original features of the input images, thereby ensuring high fidelity in the virtual staining process. This suggests that integrating both forward and identity losses provides a robust method to enhance the quality and accuracy of the synthesized stains, making it particularly suitable for applications requiring high precision in unpaired virtual staining.

In contrast, the inclusion of latent loss in the combinations tested does not appear to contribute positively to the staining results. In fact, configurations incorporating consistently underperformed in all metrics compared to those without it. This observation suggests that latent loss may interfere with the preservation of crucial stain-specific characteristics, thus making it a less desirable option in the context of virtual staining, where accuracy and fidelity are paramount.

2.12. Mitigating stitching artifacts in WSI virtual staining

In our review of existing virtual staining approaches [6,11,12,19,21,22,44], both paired and unpaired, it is evident that most employ a sliding window tiling approach during model training, as discussed in Sects 2.1.1 and 2.10. This training method inevitably leads to challenges in reconstructing WSIs from the resultant patches. In particular, it can cause visible artifacts, such as sudden color changes at tile borders and errors near these boundaries, as demonstrated in S7B Fig (0% overlap in both settings, marked with red arrows). These artifacts not only undermine the trust of pathologists in the tools but can also increase cognitive load and error rates during slide examination. This issue is pervasive across all tile-based virtual staining methods, underscoring the necessity for a universal solution.

To mitigate these problems, we developed a post-processing technique tailored for WSIs in the context of virtual staining. Our observations indicate that the models are context-sensitive, performing with high accuracy at the tile’s center and less so near the edges. Using this insight, our approach involves stitching tiles with an intentional overlap that prioritizes the central regions of the tiles using a Hamming [39,40] window (refer to Sect 2.5), effectively enhancing performance without additional training. This method, depicted in S7 Fig, significantly improves all evaluated metrics in both paired and unpaired settings and results in higher perceived image quality relative to the ground-truth. To further refine our approach, it is important to note that while our post-processing technique does introduce a slight increase in processing time, it offers a significant benefit in terms of performance-to-time ratio. An optimal overlap of 60%, as illustrated in the figures, provides the best balance between performance enhancement and execution time (>1min per 8 stains).This post-hoc processing strategy not only effectively addresses stitch artifacts, but also enhances the overall utility of virtual staining technologies in clinical settings. By maintaining a streamlined workflow, it facilitates the routine use of these technologies in a fast-paced clinical environment, potentially broadening their adoption and building trust among pathologists. This adjustment ensures the high quality of the virtual WSI stain generated (refer to S6 and S13 Figs) while offering spatial context, keeping processing time manageable, thus aligning with the needs and dynamics of modern anatomopathological practice.

2.13. Trust in virtual stains through self-inspection–anomaly detection

A significant barrier to integrating generative models, particularly in healthcare, is the absence of a confidence score with the generated output. This limitation raises critical questions: How can we detect problems in the input H&E data? What is the model’s confidence in its virtual stains? How can we assess the quality of synthetic stains and identify errors that might influence a pathologist’s decision to rely on virtual stains or request traditional chemical stains for confirmation?

Such concerns are paramount in high-stakes decision-making processes. There is a pressing need for interpretable methods to ensure that these powerful generative approaches are not sidelined because of a lack of trustworthiness. In this study, we leverage knowledge-guided training not only to improve control over the learning process-which has demonstrated improvements in performance (refer to Sect 2.9)—but also to provide pathologists with an interpretable narrative. This approach considers stain masks, shifting the model’s focus to medically relevant features, thus providing a clearer explanation than a fully black-box model.

In addition, we use the learned knowledge of the discriminator in the training phase, an element that is often discarded after training, which models the authenticity of images. This unique application allows the inspection of data quality and its deviation from the learned distribution.

To demonstrate the utility of this approach, we processed H&E tiles and examined global degradation that could arise from incorrect stain concentration or scanner configurations, as illustrated in S9 Fig. The discriminator effectively detects the domain shift in these degraded images, aligning with an anomaly detection framework. It flags deviations from the learned distribution in red, as shown in S9 Fig. These results support the hypothesis that H&E tiles presenting new defects can be detected using the model discriminator.

An extensive evaluation was conducted on 2,022 authentic WSIs. This H&E only dataset, which cannot be publicly released due to privacy constraints, consists of slides from pediatric Crohn’s disease patients followed at Robert Debré Hospital (Paris, France). Slides (5 μm thick) were stained with H&E, digitized using a Hamamatsu slide scanner, and semi-anonymized prior to analysis. The dataset comprises 47,984 tiles (512 × 512 H&E tiles).

Given that anomalies can be local or global, we employed the standard deviation of the discriminator’s confidence map as an indicator, as depicted in S8 Fig. This analysis not only confirms that the discriminator can detect outliers (e.g., artifact tiles, mostly background tiles) but also facilitates the empirical determination of a confidence interval (e.g., 3.11% ≤ acceptable ≤ 14.86%). This method introduces an effective filter to avoid feeding the multi virtual staining approach with unsound H&E images (garbage in garbage out), thus, reduces the potential error rate in synthetic stains, enhancing reliability and trust in generated outputs.

In addition, we employed the discriminator’s confidence maps on the outputted virtual stains to generate pixel-wise confidence scores. These scores empower pathologists by highlighting regions where the virtual staining deviates from the expected representation of a stain. This feature acts as a secondary filter in the output stage of our pipeline, visually represented through heatmaps as depicted in S10 Fig. The Figure contrasts the discriminator’s responses to identical tissue sections—one from an authentically stained WSI and the other from a virtually stained WSI with a staining error. In particular, the discriminator indicates red staining discrepancies, aligning with the actual differences computed between the authentic and virtual images. This methodology demonstrates the approach’s ability to provide pathologists with a reliable confidence score, offering additional context to determine the significance of the region in question for specific use cases. It also assesses whether it is necessary to perform a chemical stain to confirm findings, thus eliminating any uncertainties. By clearly indicating areas of uncertainty in the output, this tool builds greater trust in virtual staining technologies, reassuring users of its reliability, and enhancing overall confidence in the outputs provided.

This study highlights the quality check capability of integrating discriminator confidence maps into the workflow of digital and virtual staining in pathology (refer to S1 Fig). Our approach’s ability to identify discrepancies and artifacts at both input and output stages ensures that only high-quality, reliable data are utilized and generated, addressing the critical issue of "garbage in, garbage out" in medical imaging. In the future, the adoption of such advanced tools promises to refine the precision of digital pathology, potentially leading to more personalized and timely therapeutic interventions.

2.14. Cloud-based digital pathology for enhanced efficiency and usability–proof-of-concept

Configuring the software and hardware required for complex generative models can be time consuming and resource intensive, demanding specific technical skills that may not be readily available in the busy environments typical of pathology laboratories. An anywhere accessible system, preferably through a browser, could significantly enhance time efficiency and work comfort. In this study, our goal is to provide a holistic approach to managing multi virtual staining. Therefore, we have chosen to use Cytomine [35], an open-source platform, as a proof of concept to deploy our multi virtual staining technique (discussed in Sect 2.1.3). This choice allows us to bridge the gap between cutting-edge DL models and their day-to-day application, offering replicable guidelines for utilizing open-source, cloud-based platforms.

To achieve this, we deployed the platform and containerized our multi virtual staining implementation before integrating it into the cloud-based platform. This approach enables pathologists to easily execute complex algorithms directly through their web browser, as illustrated in S11 Fig. This streamlined integration simplifies the use of advanced DL models in routine pathological analysis, enhancing the accessibility and practicality of digital histopathological tools.

2.15. Paired H&E-multi-stains dataset in the context of pediatric Crohn’s disease at diagnosis

As detailed in Sect 2.1.2, one primary challenge in multi-stain data analysis is the scarcity of publicly available datasets. For example, the dataset of de Haan et al. (2021) was not shared [6]. Furthermore, the availability of high-quality paired data is limited; typically, datasets such as AHNIR [45] are compiled from adjacent slides, resulting in imperfectly matched samples. Specifically, the AHNIR kidney dataset contains only a limited set of slides, with five slides for each stain type: H&E, PAS, PASM, and MAS.

This issue is also prevalent in other studies as well; for example, MVFStain [22] used only a fraction of the AHNIR dataset for lung lesions, employing one WSI for training and another for testing. Similar methodologies are applied to datasets related to lung lobes and breast tissues, using two WSIs for training and one for testing to maintain methodological consistency.

These challenges not only hinder the public availability of such data, but also limit the diversity and quality of the datasets. With samples derived from adjacent slides, significant pairing challenges arise, complicating the objective evaluation of computational methods. This necessitates the use of elastic registration (e.g. VALIS [46]), which is error-prone due to variations in tissue characteristics.

To overcome these limitations, we propose the introduction of a new dataset that provides paired H&E to eight different stains, focusing on pediatric Crohn’s disease. This dataset aims to catalyze further research in computational pathology by including 30 H&E WSIs and 30 stained WSIs in eight stains (AE1AE3, CD117, CD15, CD163, CD3, CD8, D240, and GIEMSA), culminating in a total of 480 WSIs. Each sample consists of perfectly matched data from identical tissue sections, as depicted in S12 Fig. This comprehensive collection of 480 WSIs aims to drive advances in computational pathology. By providing high-quality, diverse data, we anticipate setting a new benchmark for methodologies not only in virtual staining but also in segmentation, detection, and other computational histopathology applications.