Fig 1.
Schematic overview of the proposed workflow.
Paired data comprising images from two different acquisition paradigms were collecting, z-stack projection and z-sweep images. (a) Z-projection images are generated by first collecting a stack of fluorescent images at discrete steps along the z-direction and projecting them down to a high quality 2D-image. (b) Z-sweeps are acquired with a single or consecutive exposures while the camera continuously moves through the axial dimension, yielding a single 2D image with intensity integrated over the z-axis. (c) Paired data of z-sweeps and z-projections are used to train a CNN, which is then capable of predicting the high-quality z-projection images from z-sweep images, to enable high-throughput acquisition of high-quality images.
Fig 2.
Diagrams of ANN architectures used in this study.
(a) U-Net, a standard encoder/decoder architecture, which our image-to-image generator models are based on. Each upsampling uses bilinear interpolation to double image size and each downsampling uses max pooling to half image size. (b) The standard U-Net convolutional block, which contains a 3x3 convolution (F3×3), batch normalization (BatchNorm) and standard ReLu. (c) PatchGAN, which comprises the discriminator of our CGAN-based models. The two channels in the first PatchGAN-layer indicate the input z-sweep image stacked onto the corresponding z-projection, either predicted or measured. Each arrow indicates a convolutional block containing a convolution with a 4x4 kernel and stride of 2 (Conv 4x4), batch normalization (BatchNorm) and leaky ReLU activation function with alpha = 0.2 (LReLU). (d) One-shot aggregation (OSA) block, adapted from [27], which replaces the standard U-net block in our improved U-Net variants. Here, F3×3 and F1×1 indicate 3x3 and 1x1 convolutional layers, FAdAvg adaptive average pooling over spatial dimensions, ⊗ indicate element-wise multiplication and ⊕ element-wise addition. FAdAvg followed by the 1 × 1 convolution and the hard Sigmoid activation make up the efficient Squeeze-and-Excitation (eSE)-module of the OSA-block. Note that F1×1 projects the concatenated tensor to the same dimensions as input tensor and that the element-wise multiplication is performed along the channel-axis and broadcasted over the spatial dimension.
Fig 3.
Schematic illustration of the conditional generative adversarial network (CGAN)-approach used to improve prediction of z-projection of images.
⊕ indicate stacking of the images. (a) During training, the generator G predicts a z-projection image based on the input sweep image S. S is stacked onto
to create a two-channel image, which the discriminator D attempts to classify as fake Df. (b) To provide negative examples, real z-projection images P are similarly stacked with z-sweeps and the discriminator then attempts to classify them as real Dr. The discriminator is optimized to accurately classify z-projection images as real or fake, while the generator network is optimized to make the discriminator prediction more difficult. The scale bar in the leftmost image applies to all images.
Fig 4.
Graphical summary of establishment and image acquisition of 3D tumour spheroid datasets.
(a) In a single spheroid assay, cells were seeded into a ultra-low attachment round bottom 96-well plate and allowed to form a single spheroid. (b) In a single plane multi spheroid assay format, cells were seeded into a polymerized bed of ®, where they grow into multiple spheroids in what is effectively a single optical plane. (c) In an embedded multispheroid assay, cells are resuspended in Matrigel® before placing in a flat bottom 96-well plate. Examples of brightfield, z-projection and z-sweep fluorescence images acquired 10 days post seeding are shown.
Table 1.
Z-projection reconstruction performance on test set images as quantified by Frechet Inception Distance (FID), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Mean Squared Error (MSE) and Multiscale SSIM (MS-SSIM).
Performance is reported for the compared models as well as the input z-sweep to provide a baseline. Arrows indicate direction of improvement, meaning that an upward arrow indicates that a larger score is better and the opposite for a downward arrow.
Fig 5.
Illustrative examples of z-sweeps, z-projections and model predicted reconstructions for the compared models.
The scale bar in the bottom left image is 50 μm and applies to all images. The brightness of the images are relative for visual purposes.
Fig 6.
Qualitative reconstruction performance of OSA-CGAN.
On single-spheroid images (a) and multi-spheroid ones (b). The brightness of the images are relative for visual purposes.
Fig 7.
Comparison of mean fluorescence intensity measurements of tumor spheroids shows fidelity between ground true and predicted z-projection images.
Representative example of (a) true z-projection and (b) OSA-CGAN predicted z-projection image, with mean intensity values overlaid for each tumor spheroid. (c) Logscale scatterplot of mean intensity values of predicted vs. ground truth (blue) and predicted vs. input z-sweep images (orange).
Fig 8.
Time courses of nuclear-restricted mKate2 fluorescence intensities of embedded tumor spheroids.
Quantified based on z-sweeps (a), OSA-CGAN predictions (b), and z-projections (c). Note the different scale of y-axis in A.