Challenges and limitations in existing methods

Before the advent of deep learning algorithms, traditional machine-learning techniques were used for mitochondria segmentation. For instance, Jorstad et al. [7] proposed an active surface-based method that refined boundary surfaces by leveraging the thick, dark (electron-dense) membranes of mitochondria. The inner membrane served as a reference for accurately delineating the outer membrane surface. Their approach demonstrated higher accuracy in fitting 3D mesh surfaces to mitochondrial membranes than manual annotation. Similarly, Lucchi et al. [8] proposed an approximate subgradient descent algorithm for optimizing margin-sensitive hinge loss in structured support vector machine (SSVM) frameworks for automated image segmentation. Their approach improves convergence, allows random constraint sampling, and achieves state-of-the-art results on electron microscopy datasets.

Likewise, Rigamonti et al. [9] introduced an enhanced kernelBoost classifier that incorporates previous segmentation results and original images iteratively. This recursive approach enables the classifier to focus on challenging regions and effectively leverage the decision tree paradigm, mitigating the risk of overfitting. Cetina et al. [10] conducted a performance comparison of feature descriptors for mitochondria and synapse segmentation, employing Gaussian and Random Forest Classifiers. These methods rely on manually designed features to detect and segment mitochondria. Additionally, Peng et al. [11] introduced an image segmentation method using contextual features called local patch patterns (LPPs) to handle appearance ambiguity in mitochondria datasets. Their approach involves iterative learning of a hierarchical structured contextual forest, incorporating spatial and temporal contexts through a median fusion strategy. They also propose a 2D variant with multi-view fusion for accurate segmentation. However, these methods have limitations regarding generalization and handling large electron microscopy datasets.

Current approaches to mitochondrial segmentation

Traditional machine learning methods struggle with larger datasets because they rely on manual feature engineering. In contrast, deep learning, mainly using convolutional neural networks (CNNs) [12, 13], has emerged as a powerful approach, inspired by the research of Hubel and Wiesel on the primary visual cortex of cats [14]. CNNs extract features from images using convolution and pooling operations, enabling accurate classification of complex images by applying these operations repeatedly [15].

For semantic segmentation, fully convolutional neural networks (FCNNs) like U-Net [16] are widely used in biomedical image analysis. U-Net architectures were successfully used in various segmentation tasks, even with only a limited amount of hand-segmented images being available [16]. This shift to deep learning has addressed the limitations of traditional methods and improved generalization performance and segmentation accuracy, making it suitable for handling large amounts of electron microscopy data.

Based on existing research, numerous deep-learning techniques have been developed specifically for the automated segmentation of mitochondria. Oztel et al. [17] introduced a fully convolutional network (FCN) with four convolutional layers, three pooling layers, and one fully connected layer for automated mitochondria segmentation. They further improved the segmentation results by applying post-processing techniques, including 2D spurious detection filtering, boundary refinement, and 3D filtering. Similarly, the SyConn framework was introduced by Dorkenwald et al. [18], employing recursive deep convolutional neural networks and random forest classifiers to locate mitochondria. The primary objective of their framework is to infer a comprehensive synaptic connectivity matrix by automatically identifying different cellular components based on manual neurite skeleton reconstructions. However, it is worth noting that the accuracy of mitochondria segmentation within the framework was not deemed satisfactory, as the main focus was on synapse segmentation.

Furthermore, Liu et al. [19] introduced a technique that employs Mask region-based (R-CNN) [20] for segmenting FIB-SEM images. Their significant contribution focuses on the post-processing phase of the obtained segmentation masks using the deep network. The post-processing steps involve eliminating small regions, smoothing large regions through morphological opening, removing shorter mitochondria using a 3D multi-layer fusion algorithm, and enhancing consistency across adjacent layers. Similarly, Casser et al. [21] introduced a modified version of the 2D U-Net [16] designed explicitly for real-time segmentation. Their approach includes reducing the number of convolutional filters, replacing transpose convolutions with lightweight bilinear upsampling layers, and using padding layers instead of center-cropping to maintain dense output at full resolution. Additionally, in the post-processing stage, they applied the median Z-filtering method.

The approaches mentioned above, including Oztel et al. [17], Dorkenwald et al. [18], Liu et al. [19], and Casser et al. [21] did not incorporate 3D spatial information in their network training process, relying exclusively on 2D convolutions.

However, some publications specifically concentrate on developing 3D automated segmentation pipelines for mitochondria. For instance, Xiao et al. [2] proposed a method utilizing a modified version of the 3D U-Net [22] model with residual blocks. They addressed the vanishing gradient problem by incorporating two auxiliary outputs in the decoder section. Their ensemble prediction approach involved considering 16 possible 3D variations, achieved through flips and axis rotations, for each 3D subvolume. To address previous studies’ limitations in utilizing labeled and unlabeled data for mitochondria segmentation, Yuan et al. [23] proposed a multi-task network named EM-Net. Their approach includes an auxiliary centerline detection task to capture the shape information of mitochondria, even with limited labeled data, resulting in improved accuracy and robustness in segmentation. However, it is worth noting that these studies did not effectively leverage large volumes of unlabeled data for model training.

Yuan et al. [24] proposed HIVE-Net, an interconnected multi-branch architecture that decomposed 3D convolutions into 2D convolutions. This approach achieved state-of-the-art performance while maintaining a compact model size and low computational requirements. Mekuč et al. [5] proposed the HighRes3DZMNet pipeline, a modified version of the HighRes3DNet [25] architecture, for automated segmentation of mitochondria and endolysosome regions. As part of their approach, they addressed the issue of varying brightness within the image by modifying the input layer of the algorithm, incorporating a zero-mean convolution layer.

In a recent study, Guo et al. [26] developed a two-stage cascaded architecture that combines top-down and bottom-up approaches for mitochondria segmentation. The first stage focuses on precise contour delineation through object localization, while the second stage considers 3D connectivity for accurate segmentation. Segmentation cues generated from multi-slice fusion in the detection stage are used to supervise the 3D CNN segmentation in the second stage, incorporating pixel properties and 3D connectivity information. Pan et al. [27] present an adaptive template transformer (AT-Former) combined with a hierarchical attention learning mechanism for mitochondria segmentation. The templates are used to accurately learn specific semantic information about the data, such as the images’ background, foreground, and contour. The hierarchical attention mechanism allows the algorithm to work at different scale levels of the image, granting the algorithm better performance results at different scales.

Similarly, Luo et al. [28] proposed a coherent fragment vision transformer (FragViT) that utilizes affinity learning to manipulate features on 3D fragments while exploring mutual relationships to model fragment-wise context. FragViT includes a fragment encoder and a hierarchical fragment aggregation module. The fragment encoder transforms the tokens into fragments with homogeneous semantics using affinity heads, and multi-layer self-attention is used to learn inter-fragment relations with long-range dependencies. The hierarchical fragment aggregation module aggregates fragment-wise predictions back to the final voxel-wise prediction in a progressive manner. This approach allows the model to maintain locality prior without sacrificing global reception.

Advancements in 2D and 3D deep learning algorithms for segmentation techniques are promising in enhancing our understanding of mitochondria and their functions. The integration of such automated segmentation methods with FIB-SEM imaging data holds the potential to unlock valuable insights into mitochondrial dynamics, structure, and interactions. This has significant implications for fields like neuroscience and clinical studies, offering a better understanding of conditions such as bipolar disorder and diabetes.

Addressing gaps with ensemble approaches

In this article, we propose a simple and effective post-processing technique that combines the predicted outputs from Mask R-CNN [20] and 3D U-Net [22] networks using an ensemble learning-based entropy-weighted average linear fusion method. This approach allows us to generate more accurate segmentation results by taking advantage of both object detection and pixel-wise segmentation networks.

Our motivation for adopting these methods is based on their demonstrated advantages in other biomedical research, where they have successfully segmented a high percentage of foreground regions of interest compared to other deep models. To ensure that Mask R-CNN and 3D U-Net were the most suitable choices for our ensemble method, we evaluated their performance alongside four other advanced networks, Ddrnet-23 [29], Segformer-B2 [30], YoloV8 [31], and YoloV9 [32], on four publicly available mitochondrial datasets: (1) Lucchi, (2) Lucchi++, (3) Kasthuri++, and (4) UroCell.

Our comparative analysis showed that Mask R-CNN and 3D U-Net consistently outperformed the other models on these datasets, thereby justifying their selection for our ensemble approach. Another key focus of our study was to develop a pipeline capable of accurately segmenting the different structures of mitochondria. We evaluated our proposed method on the aforementioned mitochondrial datasets with varying mitochondrial structures to achieve this. We then compared our results to other promising studies that have conducted similar work on these datasets.

In summary, this work makes two main contributions:

• Method: We have designed an automated pipeline for segmenting mitochondria in isotropic FIB-SEM images. Our method combines two computer vision techniques: object detection with segmentation using Mask R-CNN [20] and semantic image segmentation using the 3D U-Net [22] architecture. To account for the diverse shapes of mitochondria, we have introduced a post-processing stage where we merge the predicted outputs of both models using an ensemble learning weight fusion strategy. While Mask R-CNN and 3D U-Net are well-established models, their unique combination tailored for this specific task represents a significant contribution. The innovative aspect of our approach is the entropy-weighted fusion strategy, which dynamically adjusts weights based on model performance metrics, enhancing segmentation accuracy and robustness. Our approach significantly advances the current state-of-the-art by providing a more accurate and reliable method for mitochondrial segmentation, as demonstrated by our superior performance metrics compared to existing methods. This enables us to reconstruct complete mitochondrial shapes. Finally, we compare the performance of our proposed fusion strategy with previously published state-of-the-art results. Overall, our approach aims to improve the accuracy and completeness of mitochondrial segmentation by leveraging the strengths of both Mask R-CNN [20] and 3D U-Net [22] through our fusion post-processing strategy.
• Varying mitochondria structures: To assess the effectiveness of our proposed method in segmenting varying mitochondrial structures, we conducted experiments on four publicly available datasets: Lucchi, Lucchi++, Kasthuri++, and UroCell. The selection of these datasets was deliberate, as they encompass a wide range of mitochondrial structures and imaging conditions. Moreover, their extensive use in existing research provides a solid benchmark to evaluate our model’s performance. This comprehensive approach allowed us to evaluate our method’s robustness and generalization capabilities across diverse scenarios.

The structure of our article is designed to provide a comprehensive understanding of our research. The Materials and Methods section describes the datasets used and outlines our unique approach. Next, we present the experimental results and discuss the findings comprehensively, providing a thorough analysis of our method’s performance. Finally, the conclusion summarizes the key points and offers insights into future works, ensuring the reader is fully informed about our research and its implications.

Image volume datasets

Our experiment utilized four publicly available mitochondrial datasets, each containing a diverse range of mitochondrial structures, including oval and tube-like shapes. We had the input and corresponding labeled images for each dataset, which domain experts annotated. We selected these datasets for their high quality and the extensive research conducted using them, which provides a benchmark for comparing our work.

• Lucchi (dataset-1): The dataset used in this study was obtained from the EPFL homepage (https://www.epfl.ch/labs/cvlab/data/data-em/) [33]. The original image volume represents a section of the CA1 hippocampus region of a mouse brain, measuring 5 x 5 x 5 μm with a voxel resolution of 5 x 5 x 5 nm. The FIB-SEM microscope was used to acquire a total of 2048 x 1536 x 1065 voxels. On their homepage, the dataset consists of two subvolumes for training and testing, containing 165 slices of images with dimensions of 1024 x 768 pixels each. These images are accompanied by corresponding labeled data, which have been expertly annotated. Fig 1a illustrates the Lucchi dataset, where the image is overlapped with its corresponding labeled image.
• Lucchi++ (dataset-2): This dataset was obtained from the Connectomics homepage (https://sites.google.com/view/connectomics). Dataset-2, also known as Lucchi++ [21], is an updated version of Dataset-1 (Lucchi) [33]. The decision to create Lucchi++ was made by [21] and a neuroscientist due to observed boundary inconsistencies in the ground-truth annotations provided in Dataset-1. To address this issue, two experienced senior neuroscientists re-annotated the data. Fig 1b visually demonstrates the difference in annotation between Lucchi++ [21] and Lucchi (Fig 1a) [33] for the same slice, highlighted by a yellow circle.
• Kasthuri++ (dataset-3): The Kasthuri++ dataset (dataset-3) was also downloaded from the Connectomics homepage (https://sites.google.com/view/connectomics). This dataset is a re-annotation of the dataset provided by [34]. The volumes in [34] correspond to the S1 primary somatosensory cortex of the 3-cylinder mouse cortex volume, acquired using a serial section electron microscopy (ssEM) device. Like the Lucchi++ dataset, [21] enlisted the help of the same two experienced senior neuroscientists to re-annotate these two volumes. The training volume has dimensions of 1463 x 1613 x 85 voxels, and the testing volume has dimensions of 1334 x 1553 x 75 voxels, with a voxel size of 3 x 3 x 3 nm. Fig 1c illustrates the Kasthuri++ dataset, where the image is overlapped with its corresponding labeled image.
• UroCell (dataset-4): The UroCell dataset was downloaded from the webpage provided by [5] (https://github.com/MancaZerovnikMekuc/UroCell). This dataset consists of volumes obtained from the urinary bladder of male mice using a FIB-SEM dual-beam electron microscope device. The authors have provided eight sub-volumes, out of which five sub-volumes include both the input images and their respective annotated data. The remaining three sub-volumes (fib1-2-2-3, fib1-2-3-0, and fib1-2-3-2) contain raw images without annotations. Each sub-volume has dimensions of 256 x 256 x 256 voxels, with a voxel size of 16 x 16 x 15 nm. Fig 1d in the paper illustrates the UroCell dataset, showing the image overlapped with its corresponding labeled image.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Four publicly available datasets, (a) Lucchi, (b) Lucchi++, (c) Kasthuri++, and (d) UroCell were used in this study.

Each dataset was overlaid with its corresponding (ground-truth) label data, annotated by domain experts. For dataset-1(a) and dataset-2(b), different experts annotated the same slice, and the differences in annotations were highlighted using a yellow circle.

https://doi.org/10.1371/journal.pone.0313000.g001

Mask R-CNN & 3D U-Net

The Mask Region-based CNN (R-CNN) [20] is a state-of-the-art convolutional neural network (CNN) model used for image segmentation tasks. It is a variant of the Faster R-CNN [35] architecture, a popular computer-vision object detection algorithm. The Mask R-CNN [20] detects objects in an image and generates precise pixel-to-pixel segmentation masks for each instance. While the majority of layers in Mask R-CNN [20] is the same as in Faster R-CNN [35], it incorporates three additional layers: MaskHeads, MaskPredictor, and MaskRoiPool as shown in Fig 2 (P1) d. These extra layers facilitate the segmentation of detected objects. In Faster R-CNN [35], the user provides the input image and labeled bounding boxes and expects to receive two outputs: the bounding boxes for each object and their corresponding class labels with confidence scores. However, in Mask R-CNN [20], the user must provide the input image, labeled bounding boxes, and corresponding mask images. As a result, the user receives three outputs: the bounding boxes for each object, class labels with confidence scores for each detected box, and the segmentation masks. Mask R-CNN architectures, as introduced by [20], have been widely adopted in biomedical research studies [19, 26, 36], and similarly, we leverage these architectures in our work.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Proposed pipeline for the training and testing using the four datasets.

(P1) represents the working pipeline for Mask R-CNN, while (P2) illustrates the image segmentation pipeline for the 3D U-Net architecture. The pipeline outlines the sequential steps involved in preprocessing, training, and saving a trained model for each dataset.

https://doi.org/10.1371/journal.pone.0313000.g002

The 3D U-Net [22] architecture is an extension of the 2D U-Net [16] architecture, specifically designed to handle three-dimensional volumetric inputs (h x w x d) for pixel-wise image segmentation tasks as shown in Fig 2(P2) c. Similar to the 2D U-Net [16], the 3D U-Net [22] consists of both a 3D contractive (encoder) and a 3D expanding (decoder) structure. The 3D contractive structure takes the 3D volume as input and performs dimensionality reduction using 3D max-pooling layers. This structure aims to create a bottleneck in the center part of the network, where the spatial information is condensed through a combination of 3D convolution and pooling operations. On the other hand, the 3D expanding structure performs upsampling operations to reconstruct the segmented image. It achieves this by combining 3D convolutions and 3D upsampling layers to restore the original resolution of the input volume. Similar to the 2D U-Net [16], the 3D U-Net [22] architecture has been widely used in various biomedical research studies [2, 26, 37] for tasks such as image segmentation and as well as for boundary prediction, leveraging its ability to capture spatial information in three dimensions.

Proposed method

The proposed method workflow consists of three main subsections: pre-processing, training and testing, and post-processing. These subsections are integral parts of the overall working pipeline of the method, as illustrated in Fig 2, where two deep neural networks are depicted: (P1) represents the pipeline for the Mask R-CNN, and (P2) represents the pipeline for the 3D U-Net. The pre-processing stage involves preparing the input data, while the training and testing process focuses on training the models and evaluating their performance of testing data. Finally, the post-processing stage is applied to refine the segmentation results obtained from the earlier workflow. The clear visual representation in Fig 2 helps to illustrate the different stages and the involvement of the two deep neural networks in the proposed method workflow.

Pre-processing.

In the pre-processing phase, the 3D volume input data undergoes multiple steps before feeding into the training model (Fig 2), as illustrated in Fig 3. First, contrast enhancement employs the contrast-limited adaptive histogram equalization (CLAHE) method. Our implementation uses a clip limit of 2 and an 8 x 8 tile grid size, balancing contrast enhancement while preserving image details (Fig 3b). Next, Gaussian blur with a small 3 x 3 kernel is applied to reduce noise, maintaining image integrity (Fig 3c). Lastly, a geometric transformation aligns volume slices for accurate segmentation (Fig 3d). These steps prepare the pre-processed input volumes for training and testing with the Mask R-CNN and 3D U-Net neural network models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Pre-processing steps.

Dataset-4 (UroCell) undergoes several pre-processing stages before being fed into six different selected deep neural networks.

https://doi.org/10.1371/journal.pone.0313000.g003

Training and testing process.

The training and testing process involves initializing, training, and optimizing deep learning models on pre-processed data. This iterative process improves model performance through parameter adjustments via backpropagation. Once trained, the models are assessed on new, unseen data to gauge segmentation accuracy and generalizability.

Our approach includes training separate models for the four datasets using Mask R-CNN and 3D U-Net networks. This individualized training per dataset aims to capture distinct mitochondrial data characteristics, enhancing segmentation performance specific to each dataset.

In our Mask R-CNN pipeline (Fig 2P1), we constructed custom dataset file readers tailored to each dataset. These readers facilitated image and mask reading, object counting, and bounding box generation on the original image without resizing them for training, thereby preserving the original image dimensions. Our model utilized the “maskrcnn_resnet50_fpn” architecture, integrating a Feature Pyramid Network (FPN) for multi-scale feature extraction. We employed the Stochastic Gradient Descent (SGD) optimizer to optimize model parameters with a learning rate of 0.005, momentum of 0.9, and weight decay of 0.005. Additionally, we utilized a stepLR scheduler, reducing the learning rate by 0.1 every three epochs. The choice of 100 epochs for training was based on a balance between achieving effective learning and computational efficiency.

The model utilized data augmentation during training to prevent overfitting, and it was exclusively trained on the provided dataset. A separate validation set, comprising 20% of the data, enabled ongoing performance evaluation on unseen data ([5, 21, 33, 34]). After completing 100 epochs, models were saved for subsequent test image predictions. Implementing this inference process for each dataset, we obtained predicted binary masks for all data (refer to Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The image shows the predicted boundary boxes (in red) generated along with their respective segmented images (in random color) after passing each volume slice through the Mask R-CNN trained model (M1).

https://doi.org/10.1371/journal.pone.0313000.g004

After the Mask R-CNN pipeline, we applied the pre-processed data to the 3D U-Net pipeline (Fig 2P2) using Pytorch-3dunet (https://github.com/wolny/pytorch-3dunet) based on [38]. Instead of building our own 3D U-Net pipeline from scratch, we used the pytorch-3dunet library due to its proven performance in various research papers. In their library, [38] implemented three semantic segmentation algorithms, including 3D U-Net [22], ResidualUNet3D [39], and ResidualUNetSE3D [40]. We attempted to use both 3D U-Net and ResidualUNet3D from this library for our experiment. However, we found that ResidualUNet3D did not yield satisfactory results. As a result, we proceeded to run each dataset through the 3D U-Net network only.

The train-and-test configuration YAML file defined the model, loss function, optimizer, and learning rate scheduler. We utilized BCEDiceLoss as the loss function, combining binary cross-entropy and dice losses. The Adam optimizer with a learning rate of 0.0002 and weight decay of 0.00001 was chosen. Employing the ReduceLROnPlateau scheduler, we monitored metric improvements and reduced the learning rate by 0.2 if no improvement was observed within ten epochs, optimizing resources. Training for 1000 epochs used a patch shape of 64 x 64 x 64 for each data point, accompanied by data augmentation techniques (flipping, rotation, elastic deformation) to prevent overfitting. The 3D U-Net architecture processes volumetric inputs, reduces dimensionality through convolutions and pooling, and reconstructs segmented images using convolutions and upsampling layers.

After completing the training phase, we saved each dataset’s trained 3D U-Net model (M2). We employ the saved models (M2) by passing the test image volume through the inference pipeline to make predictions on unseen test image volumes. This process involves inputting the test volume into the trained model and generating a predicted binary mask. Repeating this inference for each dataset results in predicted binary masks, as illustrated in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The image shows the predicted mask generated for all four datasets after passing each volume through a 3D U-Net trained model (M2).

https://doi.org/10.1371/journal.pone.0313000.g005

Post-processing.

In our post-processing phase, we aim to refine the segmentation results obtained from modes P1 and P2. Instead of traditional techniques like thresholding or morphology operations, we adopt a fusion strategy based on ensemble learning [41], illustrated in Fig 6. This weighted average fusion method aims to utilize the best features of each model to create a more detailed and accurate segmentation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Post-processing stage.

The image illustrates the final segmentation result obtained by combining the predicted outputs from each trained model (M1) and (M2), Mask R-CNN, and 3D U-Net models, using the ensemble learning weight average fusion technique. This fusion process enhances the accuracy of the segmentation and produces the final prediction output.

https://doi.org/10.1371/journal.pone.0313000.g006

Ensemble learning usually assigns weights to models based on data characteristics. We use an entropy weight measure method, a scientific approach that considers data specifics to determine weights. This method assesses variables or indices based on entropy values, reflecting their disorder or information content. Higher entropy variables receive smaller weights, while lower entropy variables get larger weights. This approach objectively assigns weights, considering discrimination ability and offering a deeper understanding of data integration dynamics. This fusion technique linearly combines the model outputs based on their entropy-determined weights, providing a strategic advantage over simple average voting methods.

In average voting, each model in the ensemble contributes equally to the final prediction, regardless of its performance or reliability. This can be problematic when some models are more accurate or relevant to specific data segments than others, as average voting does not differentiate between the quality of input models. It treats each model’s prediction with the same level of importance, potentially diluting the influence of more accurate models with those of lesser accuracy.

Conversely, the linear model approach adjusts the influence of each model’s prediction according to its weight, determined by entropy measures.

Employing entropy-weighted linear fusion directly addresses the limitations of individual models. Specifically, it compensates for areas where a model may produce false negatives, effectively filling in gaps in the segmentation. This tailored combination not only leverages the unique strengths of each model but also mitigates its weaknesses, leading to a more accurate and complete segmentation, as shown in Fig 7. This strategy significantly boosts the segmentation’s accuracy and completeness, offering a deeper and more holistic understanding of the analyzed objects. Importantly, this method dynamically adjusts weights to enhance overall segmentation accuracy, reassuring us about the adaptability of our approach.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The image represents the results of passing the first slice of the Kasthuri++ testing data through trained M1 and M2 models to obtain their respective predicted outputs.

In (a), the predicted outputs from both models exhibit false negatives, indicated by the yellow circles within the image. (b) shows the one found false negative object. To address this limitation, we propose an ensemble learning weight averaging fusion method, shown in (c), which combines the outputs based on the weights obtained using entropy. The resulting new segmentation result, displayed in (d), demonstrates the effectiveness of this technique in overcoming the limitations of individual models. Finally, the segmentation result in (d) is visually compared with the expert-annotated label image (e).

https://doi.org/10.1371/journal.pone.0313000.g007

The proposed ensemble learning-based entropy-weighted average fusion can be defined using the below expressions:

In our case, we have two models, let us assume; (1) Where n is the number of individual models (n = 2 in this study, as we have utilized two models in this experiment),

And evaluation metrics for each model; (2) Where l is the number of individual model evaluation metrics, such as the score of Jaacard_Foreground, mean_IoU, precision, recall, dice, etc.

Utilizing (2), we can have Em_{mask_rcnn} and Em_3d−unet

First, we need to calculate the entropy for each model, which can be calculated by using the obtained evaluation metrics of each model; (3) Where p represents the probability of each metric for the model.

By utilizing Eq (3), we can calculate the entropy for each model as follows;

Now, after calculating the entropy, we can calculate the weight for each model by the below expression; (4)

Now, utilizing (4), we can calculate Wt₁ and Wt₂ for Mask-R-CNN and 3D U-Net;

In the next step, we have to normalize the weight;

Now, we can pass these normalized weights for each respective model to perform ensemble learning weighted averaging fusion; (5)

The expression (5) signifies the ensemble learning weighted average fusion using entropy weights, denoted by P as the ensemble prediction and (×) representing multiplication. This fusion method linearly combines the predictions of two models based on their respective weights.

Evaluation metrics

In our evaluation phase, we rigorously computed several metric scores to assess the performance of our segmentation models thoroughly. After processing the test data through individually trained models, M1 and M2, and subsequently employing our proposed ensemble learning method, we meticulously calculated a range of metrics. These metrics were carefully chosen to gauge the segmentation results comprehensively and provide a nuanced understanding of the models’ effectiveness in identifying regions of interest.

• Area Under the Receiver Operating Characteristic (AUROC): The AUROC value quantifies the overall ability of a model to distinguish between the two classes (usually positive and negative) across different classification thresholds. A higher AUROC score indicates better model performance regarding classification accuracy and the ability to discriminate between the classes. The AUROC curve represents the trade-off between a true positive rate (sensitivity) and a false positive rate (1-specificity) at various threshold values.
• Pixel Accuracy: It is the ratio of correctly classified pixels to the total number of pixels in the segmentation mask. Pixel Accuracy is a simple and intuitive evaluation metric used in image segmentation tasks to assess the accuracy of pixel-level predictions made by a segmentation model. It measures the percentage of correctly classified pixels in the predicted segmentation mask compared to the corresponding pixels in the ground truth mask. This metric provides a straightforward way to gauge the overall accuracy of a segmentation model’s predictions. Pixel Accuracy values range from 0 to 1, where 1 indicates perfect pixel-level accuracy (all pixels are correctly classified), and 0 indicates no pixel-level accuracy (none of the pixels are correctly classified). (6)
• Error Rate: It is a metric that quantifies the percentage of incorrectly classified samples in a dataset and complements accuracy. To calculate it, we need to know the number of misclassified samples (i.e., the number of samples classified incorrectly), which refers to the count of samples classified incorrectly by a model or system. These are instances where the predicted class or label does not match the true class or label and the total number of samples in the dataset (this represents the entire set of samples or instances you are evaluating. It includes correctly and incorrectly classified samples). (7)
  In summary, the error rate measures how well a model or system performs in classifying samples correctly. A lower error rate indicates better performance, while a higher error rate suggests that more samples are misclassified.

Besides the above metrics, we also computed the Jaccard index for background (JI_B) and foreground (JI_F) and their mean (m_IoU). Similarly, we computed each model’s dice score coefficient (DSC) on the four diverse datasets.

Experimental settings

In our experiments, the training and testing of both the P1 and P2 models were conducted on a desktop environment with comprehensive system specifications: Memory: 64 GB, Processor: Intel Core i7–6700K CPU (4 GHz), Graphics: NVIDIA GeForce GTX TITAN, and OS Type: 64-bit. The Ubuntu 20.04.4 LTS (Focal Fossa) operating system was used to conduct the experiments. We implemented the training and testing codes in the PyTorch framework, leveraging its capabilities for deep learning tasks.

We utilized four publicly available mitochondria datasets during the experiments, as shown in Fig 1. These datasets consist of diverse mitochondria structures, posing a challenge for the model and the proposed fusion method to segment the regions of interest accurately. The variations in mitochondrial structure across the datasets added complexity to the segmentation task. However, our model’s ability to adapt and generalize well to different shapes and textures reassures its robustness.

The Mask R-CNN model parameters, such as learning rate, number of epochs, loss function, etc., were consistent across all datasets during the training and testing phases. Similarly, the 3D U-Net model followed the same parameter settings for each dataset. All four datasets underwent the pre-processing stage in both the training and testing stages, as depicted in Fig 3. This ensured that the input data for both models was prepared and standardized, allowing for fair and consistent comparisons across the datasets.

Model selection for ensemble method

Based on our comprehensive evaluation, we selected Mask R-CNN and 3D U-Net for our ensemble method due to their superior performance across various metrics on the four mitochondrial datasets: Lucchi, Lucchi++, Kasthuri++, and UroCell. As depicted in Tables 2 and 3, Mask R-CNN and 3D U-Net consistently achieved higher JI_B and JI_F, as well as higher m_IoU and DSC values, compared to other models such as Ddrnet-23, Segformer-B2, YoloV8, and YoloV9. Additionally, their AUROC and pixel accuracy metrics were notably better. Fig 8 further illustrates that the segmentations produced by Mask R-CNN and 3D U-Net closely align with the ground truth, demonstrating fewer misclassifications and artifacts. Similarly, Fig 9 shows the obtained error rate for each model, and it further validates that 3D U-Net and Mask R-CNN models have achieved a low error rate compared to the other four models.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Qualitative comparison of segmentation results across different deep learning models for Lucchi++ slices.

The ground truth image of slice 32 is shown alongside predicted segmentations from all six models: Mask R-CNN, 3D U-Net, YoloV8, Ddrnet-23, Segformer-B2, and YoloV9.

https://doi.org/10.1371/journal.pone.0313000.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Computed mean error rate for each algorithm and dataset.

https://doi.org/10.1371/journal.pone.0313000.g009

The architectural strengths of Mask R-CNN in object detection and 3D U-Net in volumetric segmentation make them particularly effective for detailed and accurate segmentation tasks. In contrast, Ddrnet-23 and Segformer-B2, while competitive, displayed slightly lower performance metrics. Ddrnet-23, which focuses on high-resolution segmentation, struggled with the fine-grained details in mitochondrial structures. Segformer-B2, which combines Transformer and CNN features, showed potential but was less adept at capturing intricate mitochondrial details, leading to lower precision and recall scores. The Yolo models, YoloV8 and YoloV9, primarily designed for real-time object detection, performed well in identifying objects but were less effective in pixel-wise segmentation tasks, as evidenced by their lower mIoU and DSC scores. This is likely due to their optimization for speed and detection rather than the fine-tuned segmentation required for biomedical images. The parameters, such as loss function, optimizer, and learning rate, were consistent while the results were validated.

Thus, the empirical evidence supports our choice of Mask R-CNN and 3D U-Net for the ensemble, leveraging their strengths to achieve more accurate segmentation results. The output variations between models can be attributed to their inherent architectural differences and design perspectives. Mask R-CNN and 3D U-Net are more suited for the detailed and accurate segmentation needed for these datasets.

Model complexity

The computational complexity of our ensemble approach is primarily characterized by the models’ parameters and GFLOPs (Giga Floating Point Operations per Second), reflecting the scalability and efficiency of the method, as provided in Table 1. The models in our ensemble, such as the 3D U-Net and Mask R-CNN, contain 19 million and 43 million parameters, respectively, highlighting their substantial capacity for detailed feature extraction. The GFLOPs metric further quantifies this computational demand, with values ranging from 17.944 for DDRNet-23 to 344.9 for YoloV8. These values indicate the varying computational costs associated with each model. Specifically, YoloV8, with the highest GFLOPs, provides powerful object detection and segmentation capabilities but requires significant computational resources. In contrast, DDRNet-23, with lower GFLOPs, offers a more computationally efficient option while maintaining robust performance. Our choice of models balances these factors, leveraging the strengths of both high-capacity and efficient models. The consistent use of a batch size of 2 across all models ensures a uniform approach to training. At the same time, the learning rates and optimization strategies (e.g., ADAM and SGD) are tailored to each model’s architecture to optimize convergence and performance. Overall, the ensemble approach strategically integrates diverse models, providing enhanced segmentation accuracy while maintaining feasible computational demands. This balance supports the method’s application to large datasets, as evidenced by the detailed computational costs manageable within current hardware capabilities.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of model configurations and computational complexities of each model.

The GFLOPs metric indicates the computational cost associated with each model, providing insight into their scalability and efficiency.

https://doi.org/10.1371/journal.pone.0313000.t001

Segmentation results

To validate the applicability and effectiveness of our proposed approaches, we conducted experiments on four diverse datasets: Lucchi, Lucchi++, Kasthuri++, and UroCell. Tables 2 and 3 present, respectively, the performance metrics and the computed AUROC, pixel accuracy, and error rates for all four distinct datasets and all three proposed algorithms. Fig 9 presents the mean error rate for each algorithm and dataset. By testing our methods on these datasets, which encompass various mitochondrial structures and imaging conditions, we aimed to assess our approaches’ robustness and generalization capability across various scenarios.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance metrics evaluation of each algorithm on four diverse datasets.

https://doi.org/10.1371/journal.pone.0313000.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Computed AUROC and pixel accuracy for all four distinct datasets using all seven proposed algorithms.

https://doi.org/10.1371/journal.pone.0313000.t003

Our fusion method consistently outperforms the Ddrnet-23, Segformer-B2, YoloV8, and YoloV9, Mask R-CNN, and 3D U-Net model (Fig 10) and other state-of-the-art methods regarding foreground segmentation accuracy as shown in Tables 4–7. This indicates that the fusion of predictions from multiple models enhances the overall performance and leads to more accurate and robust segmentation results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. The figure displays the final segmentation results achieved by applying the ensemble learning weight average fusion technique across four datasets, presented alongside their respective annotated label images.

The black circle indicates the slice number of each image. The fused output highlights the advantages of the weight fusion technique, enhancing segmentation accuracy by combining the strengths of Mask R-CNN and 3D U-Net predictions.

https://doi.org/10.1371/journal.pone.0313000.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance comparison of segmentation methods on the Lucchi dataset.

https://doi.org/10.1371/journal.pone.0313000.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Performance comparison of segmentation methods on the Lucchi++ dataset.

https://doi.org/10.1371/journal.pone.0313000.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Performance comparison of segmentation methods on the Kasthuri++ dataset.

https://doi.org/10.1371/journal.pone.0313000.t006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Performance comparison of segmentation methods on the UroCell dataset.

https://doi.org/10.1371/journal.pone.0313000.t007

Lucchi dataset.

On the testing data, our proposed fusion method achieved an accuracy of 0.9337, 0.9954, 0.9623, 0.9633, 0.9944, 0.9761, and 0.9928 scores in terms of JI_F, JI_B, m_IoU, DSC, accuracy, precision, and recall, respectively, as shown in Table 2. Likewise, in the case of the 3D U-Net model (which we used from [38]), which is a pixel-wise segmentation method designed to track the 3D connectivity of volumes in 3D data, the performance of the 3D U-Net model on this dataset was slightly lower than the fusion method, achieving scores of 0.912, 0.9941, 0.9502, 0.9512, 0.9944, 0.9673, and 0.9503 for JI_F, JI_B, m_IoU, DSC, accuracy, precision, and recall, respectively, as shown in Table 2.

Despite this, the 3D U-Net model still demonstrated exemplary performance in accurately segmenting the mitochondria regions of interest, exhibiting its effectiveness in capturing the 3D connectivity of the structures, which can be seen in Fig 12a. The performance of the Mask R-CNN method on this dataset was relatively average compared to the earlier two methods. The scores obtained for JI_F, JI_B, m_IoU, DSC, accuracy, precision, and recall were 0.9112, 0.9915, 0.9512, 0.9533, 0.9921, 0.9416, and 0.9416, respectively, as shown in Table 2. One possible reason for this average performance could be the limitation of the two-dimensional object localization technique used in Mask R-CNN when dealing with 3D-connected regions. The method may struggle to accurately track and localize these connected components, leading to localization problems. This can be observed in Fig 11a, where some 3D connected objects are not precisely localized or localized in the first slice but did not localize in the second or fourth slice.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. The image illustrates the predicted output of Mask R-CNN (green channel) overlapped with the ground-truth image (red channel) for various datasets.

The yellow-colored regions represent true positive regions where Mask R-CNN accurately segmented the interested regions of interest (ROI). However, limitations of the model can also be observed. For example, in the Lucchi dataset, a particular object was not detected in slice numbers 52 and 53 (indicated by the orange arrow) but detected in slices 54 and 55, highlighting challenges in object localization for connected objects. Similar limitations can be seen in the Lucchi++ dataset. In the case of the Kasturi++ dataset, while most objects were correctly detected, there are instances of false negative ROIs (indicated by the green circle). The blue circle highlights regions where the model predicted correctly, but the segmentation did not fully cover the entire ROI.

https://doi.org/10.1371/journal.pone.0313000.g011

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. The image displays the comparison between the predicted output of the 3D U-Net model (shown in the green channel) and the ground-truth image (shown in the red channel) for various datasets.

The yellow-colored regions indicate the true positive regions where the 3D U-Net model accurately segmented most ROIs. Overall, the performance of the 3D U-Net model is relatively good compared to the predictions from Mask R-CNN. However, there are still instances, such as in slice number 52, where the 3D U-Net model did not fully predict certain objects. An example of such an object is highlighted with a blue circle, indicating that it was not completely captured by the trained model (M2). The green circle shows the false negative ROIs, which should have been predicted, but the model failed to predict it.

https://doi.org/10.1371/journal.pone.0313000.g012

Likewise, in Table 3, we have computed the AUROC, pixel accuracy, and error rate for each algorithm. It is evident from this table that the fusion algorithm has achieved impressive scores, with 0.9937 and 0.9941 for AUROC and pixel accuracy, respectively. These scores are notably higher than those obtained by the 3D U-Net and Mask R-CNN algorithms. Furthermore, in terms of the mean error rate, the fusion method stands out as the best performer, with a mean error rate of 0.0059 (Fig 9). In comparison, the 3D U-Net has a mean error rate of 0.0085, and Mask R-CNN has a mean error rate of 0.0093. Table 4 provides a comprehensive evaluation of our proposed fusion method, offering a detailed comparison with the results obtained by other authors on the same dataset. The table shows that our proposed fusion method has obtained competitive scores across all the listed metrics compared to other methods.

UroCell dataset.

In the case of the UroCell dataset, which is a newly available dataset, our proposed ensemble learning-based weight average fusion method, as well as the Mask R-CNN and 3D U-Net models, performed exceptionally well compared to the methods proposed by [5]. The proposed fusion method achieved high scores in all seven metrics, with scores of 0.9432, 0.9956, 0.9644, 0.9749, 0.9958, 0.9689, and 0.9895 for JI_F, JI_B, m_IoU, DSC, accuracy, precision, and recall, respectively, as shown in Table 2. The 3D U-Net model also performed well, with scores of 0.9329, 0.9949, 0.9601, 0.9622, 0.9952, 0.9344, and 0.9622 for the same metrics. The Mask R-CNN model achieved scores of 0.9231, 0.9951, 0.9623, 0.9654, 0.9954, 0.9579, and 0.9579 for JI_F, JI_B, m_IoU, DSC, accuracy, precision, and recall, respectively, on the UroCell dataset. These results demonstrate the effectiveness of our proposed fusion method and the individual models in accurately segmenting the UroCell dataset.

Figs 11d and 12d demonstrate the performance of the Mask R-CNN and 3D U-Net models on the UroCell testing volume slices from 13 to 16. Both models performed well on this dataset, with the Mask R-CNN model missing some objects (false negatives) in slices 14 and 15 but successfully predicting the same region in slice 16. This limitation is attributed to the object localization technique’s difficulty segmenting 3D-connected regions. On the other hand, the 3D U-Net model accurately predicted all objects, including the previously missed false negative object, by the Mask R-CNN model.

Similarly, we computed AUROC, pixel accuracy, and mean error rate for each algorithm on this dataset, and the scores are presented in Table 3. Our proposed fusion method achieved impressive scores of 0.9985, 0.9938, and 0.0062 for AUROC, pixel accuracy, and mean error rate, respectively. In comparison, 3D U-Net obtained scores of 0.9825, 0.9922, and 0.0078 for the same metrics, while Mask R-CNN scored 0.9717, 0.9888, and 0.008. These results highlight the superior performance of the fusion method. Additionally, Fig 9 displays the mean error rate for each algorithm across all datasets.

Fig 13a and 13b showcase the predicted outputs of the proposed fusion method in 2D and z-stack form, respectively. The predicted slices are highlighted in blue color. Additionally, Fig 13c visualizes the same z-stack predicted slices in 3D using the visualization tool proposed by Gupta et al. [48]. These figures assess whether our proposed fusion method accurately captures the relevant objects in their 3D shapes. By examining Fig 13c, it becomes evident that our proposed method successfully segments all the regions of interest. The 3D connectivity between the slices is blue, indicating accurate segmentation. Furthermore, we overlay the predicted output (shown in blue) with the expert-annotated ground truth (shown in green) to assess the consistency between the proposed fusion segmentation result and the labeled image. Remarkably, the predicted output aligns remarkably well with the labeled image Fig 13d, further confirming the effectiveness and accuracy of our proposed fusion method.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13.

The image shows the visual representation of the segmented UroCell testing dataset by applying the fusion method in (a) 2D, in (b) z-stack, in (c) 3D reconstruction, and in (d) shows the overlapped of predicted volume (shown in blue) with expert annotated ground truth volume of the same testing dataset (shown in green), to see the 3D binding between these two.

https://doi.org/10.1371/journal.pone.0313000.g013

Limitations

While our proposed entropy-weighted ensemble approach combining Mask R-CNN and 3D U-Net has demonstrated significant improvements in mitochondrial segmentation, as shown in Fig 10, several limitations must be acknowledged, which we found while experimenting:

1. Handling Diverse Mitochondrial Structures:
   • Structural Variability: Mitochondria exhibit a wide range of shapes and sizes, varying significantly across different cell types and imaging conditions, as shown in Fig 1. While our method performs well on the datasets used, it may face challenges in generalizing to other datasets with highly irregular or uncommon mitochondrial structures. Future work could explore incorporating additional models specialized in capturing such variability or using more advanced data augmentation techniques to improve generalization.
2. Imaging Conditions:
   • Noise and Artifacts: FIB-SEM images often contain noise and artifacts that can impact segmentation accuracy. Although our preprocessing steps aim to mitigate these issues by applying the Gaussian blur method (as shown in Fig 3c), certain types of noise and imaging artifacts may still pose challenges. Advanced noise reduction algorithms and artifact correction techniques could be integrated into the preprocessing pipeline to enhance robustness.
   • Contrast Variability: Variations in image contrast across different datasets can affect model performance, and therefore, in our pipeline, we have applied CLAHE (as shown in Fig 3b image contrast enhancement technique to enhance the contrast of the image, however, same clip-limit might not be suitable for all datasets; therefore, proposed method may require fine-tuning of parameters or additional contrast normalization steps to handle varying imaging conditions effectively.
3. Post-Processing Strategy:
   • Entropy-Weighted Fusion: While the entropy-weighted fusion strategy enhances segmentation accuracy by leveraging model confidence (as shown in Tables 4–7), it may not always perfectly capture the nuances of every mitochondrial structure. In some cases, misclassifications from one model might still influence the final output. Exploring alternative fusion strategies or incorporating additional validation steps could help address this limitation.