Fig 1.
The Nikon BioStation IM benchtop live cell imaging system.
(a) External features include a incubation unit, joystick for controlling the position of the camera during sample selection, and a monitor. (b) Culture dish sitting inside the BioStation IM incubator unit.
Fig 2.
Phase contrast images for the six different classes of hESC obtained from the Nikon BioStation IM.
Table 1.
Summary of the related work for detecting hESC.
Table 2.
Summary of the related work for classification of hESC and contributions of this paper.
Fig 3.
Example images of Cell clusters and Apoptically Blebbing cells.
The distinguishing features between Cell clusters and Apoptically Blebbing cells are the small cells in the Cell clusters packed close to each other.
Table 3.
Summary of the related work for generating synthetic hESC images and contributions of this paper.
Fig 4.
Archirecture of e-DCGAN for generating synthetic images of Cell clusters.
A noise vector Z is given as input to the generators of the four intrinsic cells, the corresponding synthetic images are passed through their corresponding discriminators to extract a feature vector. The resulting feature vector is given as input to a DCGAN trained to generate synthetic images of Cell clusters.
Fig 5.
Workflow of DeephESC 2.0 is split into three modules namely: Detection of hESC from video, Generation of synthetic hESC images and hierarchical classification of the hESC images into six different classes.
Fig 6.
Images of the cell body and the substrate and their corresponding intensity distribution.
Fig 7.
Detected cell bodies of a single frame using the approach proposed by [24].
The detected cell bodies are then cropped and passed through the hierarchical classifier to be classified into one of the aforementioned six classes.
Fig 8.
Workflow of the hierarchical classifier.
The input is either a real or synthetic image belonging to one of the six classes. The outputs of the CNN and Triplet CNNs are fused at the decision level using the product rule.
Table 4.
Architecture of the Convolution Neural Network in the hierarchical classifier.
Table 5.
Data augmentation performed to train the CNN.
Fig 9.
Architecture of Triplet CNN A and Triplet CNN B in Fig 8.
The parameters within the parenthesis indicate the kernel dimension, stride and padding. By skipping intermediate layers and concatenating the feature maps of branched layers, DeephESC 2.0 is able to extract much more robust features, further improving the classification.
Table 6.
Best hyper-parameters for training the networks in DeephESC 2.0.
Fig 10.
The generator is trained to take as input a random noise vector and generate an image that resembles the training data. The task of the N discriminators are to predict if the input image to the discriminator is either a real or a synthetic image. In our architecture of GMAN the softmax outputs of the N discriminators are combined together by computing their geometric mean.
Table 7.
Architecture of the generator and the three discriminators used in our Generative Multi Adversarial Network.
Table 8.
Comparison of the average classification accuracy of the networks used in DeephESC and DeephESC 2.0.
Table 9.
Confusion matrix for the classification of the 724 real hESC images using the CNN architecture of DeephESC 2.0.
Table 10.
Confusion matrix for the classification of the 724 real hESC images using the CNN-Triplet architecture of DeephESC 2.0.
Table 11.
Confusion matrix for the classification of the 724 real hESC images using the Fused CNN-Triplet architecture of DeephESC 2.0.
Fig 11.
Visualization of features extracted by the CNN in DeephESC 2.0 for (a) Apoptically Blebbing cell and (b) Unattached cell.
Fig 12.
Visualization of features learned by DeephESC 2.0.
(a) Image of a Cell cluster. (b) Image after masking the surrounding small cells using a window. Red bounding boxes are drawn across the masked area only for visualization purposes. (c) Probability heat map for the class Apoptically Blebbing cell.
Fig 13.
Visualization of features learned by the generators in DeephESC 2.0.
(a) Unattached cell and (b) Attached cell.
Fig 14.
The 600 synthetic images used for validating the quality in Table 12.
(a) Cell clusters, (b) Debris, (c) Unattached cells, (d) Attached cells, (e) Dynamically Blebbing cells, (f) Apoptically Blebbing cells.
Table 12.
Comparison of our GMAN architecture used in DeephESC 2.0 with e-DCGAN [5], DCGAN [35] and c-DCGAN [41] using the SSIM and PSNR metrics.
SSIM has no units and PSNR is measured in decibels (dB).
Table 13.
Accuracy and number of images of each fold for the 5-fold cross validation using the 724 real hESC images.
The number in the brackets indicates the number of images per class for Cell clusters, Debris, Unattached cells, Attached cells, Dynamically blebbing cells, and Apoptically blebbing cells respectively.
Table 14.
Comparison of using different data compositions of synthetic images for training the classifier and then testing it on the 724 real images.
Fig 15.
Classification accuracy Vs training time trade-off.
Fig 16.
Examples of images that were unintentionally labeled wrong by the biologist, but correctly classified by our classifier.
(a) Unattached cell mislabeled as Cell cluster, (b) Attached cell mislabeled as Dynamically Blebbing cell. (c) Apoptically Blebbing cell mislabeled as Cell cluster.