Fig 1.
Pipeline for segmenting nuclei with NuSeT.
(A) Deep-learning model structure of NuSeT. The inputs of the model are gray scale images with different sizes. The outputs are binary masks with the same size as inputs, with predicted foreground regions as Ones and background regions as Zeroes. The model combines U-Net (gray and orange) and Region Proposal Network (purple), which performs nuclei segmentation and detection separately. The results are then merged and processed by watershed (dark blue) to generate final predictions. (B) Outlook of NuSeT Graphic User Interface(GUI), and example training and predicting pipelines using NuSeT GUI.
Fig 2.
Improved normalization performance by foreground normalization and synthetic training.
(A) The visual effects of normalizing sparse/dense samples using whole-image normalization showing images having inconsistent nuclear signals after normalization. (B) Foreground normalization during training and testing. During training, only pixels belonging to cell nuclei are used to normalize the image. During testing, a coarse segmentation prediction is generated by the model, and pixels belonging to the predicted nuclei are used to perform foreground normalization. The model then makes final predictions based on the normalized input images. (C) Distribution of pixel intensities over an entire training dataset after different normalizations, showing foreground normalization has wider dynamic range. (D) The visual effects of normalizing sparse/dense samples using foreground normalization showing images have a higher dynamic range and more consistent nuclear signals. (E, F) Line charts showing that the object-level performance (E) and the pixel-level performance (F) of the foreground normalization model depend on the pixel-level performance of the whole-image normalization model. Error bars represent three individual experiments. (G) Examples of synthetic images with labels used during training. Our algorithm can generate synthetic nuclei-shaped blobs with different sizes, as well as different types of artifacts to increase the robustness of the model. Overlapping nuclei were introduced to enhance NuSeT performance in touching nuclei separation. (H) Representative examples comparing the performances of different segmentation approaches. Training without synthetic images mis-identified artifacts (stripes) as foreground. The addition of synthetic data improved artifact detection. Switching to foreground normalization led the best performance including robust identification of imaging artifact, detected of more nuclei, and better separation of touching nuclei compared to Mask R-CNN and U-Net.
Fig 3.
NuSeT efficiently addresses common segmentation challenges.
(A) Implementing RPN-aided watershed algorithm improves touching cell separation. Bounding boxes and segmentation masks are computed by RPN and U-Net. Then the estimated centroid of each cell is computed from the coordinates of the bounding box. The watershed line is then estimated based on the binary mask and centroids. (B) Sample results showing that RPN successfully detects most of the cells, and watershed lines further separate touching cells. (C) Representative examples showing NuSeT detected more nuclei and better separated touching nuclei compared to Mask R-CNN and U-Net. (D,E) Examples nuclear masks generated using NuSeT for an image with high nuclei density (D). Comparison with the corresponding masks generated by Mask R-CNN and U-Net show subtle as well as prominent irregularities in boundary delineation that are circumvented by NuSeT (E).
Fig 4.
NuSeT effectively segments single nuclei in disorganizing dense mammary acini.
(A) Representative 3D MCF10AT acinus segmentation using NuSeT. (B) Nuclei tracking. For ease of visualization, only a few of the segmented nuclei are shown at different time points. (C) 3D tracks of the nuclei shown in (B) over time, from 0 h (dark) to 4.5 h (light). (D) Number of nuclei detected in disorganizing acini at different time points using different segmentation methods. Data were collected from 8 representative acini and were normalized by the total number of nuclei at the last time point. Data from the first 5 hours are shown. (E) Cumulative distribution function plots of area of nuclei segmented using different methods. (F) Box plots of nuclear area distribution. The median area for each method is indicated on the top. The area box plot for Otsu’s method (median area: 2816.6 ± 2845 μm2) is shown in S6 Fig. (G) Representative examples comparing nuclei segmentation in dense mammary acini using different methods. Scale bars are 20 μm.
Fig 5.
NuSeT can be applied to segment histopathology samples and detect mitotic events.
(A, B) Representative example of liver fat globule segmentation using NuSeT. Notice that NuSeT performs well on both macroglobule (A) and microglobule (B) segmentation. (C-E): Representative example of segmentation and detection of mitosis in breast cancer samples from ICPR 2012. (C) Weight map used for training the mitosis model. An ‘attention’ strategy has been used to focus more on the mitosis events and the environment surrounding them. The shaded region denotes the label for the mitotic event, and colors denote the weights applied during training process. (D) Representative example of mitosis detection and segmentation results with breast cancer sample. Scores on the top-left corner of the bounding boxes denote the possibility of a mitosis event evaluated from the model. Zoom-in of some detected mitotic events are shown in (E). (F) Representative example of mitosis detection and segmentation results with fluorescent nuclei in a time lapse movie of MCF10A epithelial cells stably expressing histone H2B-eGFP. NuSeT can detect mitotic progression from prophase to telophase (mitotic events were identified by NuSeT and then manually classified into different phases).