Introduction

Organoids are multicellular three-dimensional (3D) structures derived from primary or stem cells that are embedded into a biological matrix to create an extracellular environment that provides structural support and key growth factors. Organoids closely recapitulate cellular heterogeneity, structural morphology, and organ-specific functions of a variety of tissues, which improves modeling of in situ biological phenomena compared to traditional monolayer cell cultures [1]. Live-cell imaging can reveal important features of organoid dynamics, such as growth, apoptosis/necrosis, movement, and drug responses, which can reflect physiological and pathological events such as organ development, function, infarction, cancer, and infection [2–7].

While organoids have been successfully used to investigate important phenomena that might be obscured in simpler models, their use in data-intensive applications, such as high-throughput screening, has been more difficult. A major challenge for organoid experiments is drug-response measurement and analysis, which must be performed for a large number of microscopy images. Image analysis is particularly difficult for organoid experiments due to the movement of organoids across focal planes and variability in organoid size and shape between different tissue types, within the same tissue type, and within the same single culture sample [8,9]. Several recently developed organoid platforms, while powerful, rely on per-experiment or per-image tuning of brightfield image analysis parameters [10–12], or require manual labeling of each image [7,13,14], which limits experiment reproducibility and scale. Organoids can be fluorescently labeled to aid in image segmentation and tracking, such as through genetic modification for fluorescent protein expression [15–17], cell fixation and staining [10,16], or the use of membrane-permeable dyes. However, these approaches may alter intrinsic cellular dynamics from original samples [18,19], limit measurements to endpoint assays, or cause cumulative toxicity through longer growth times and limited diffusion through the hydrogel matrix [20]. There is a critical need for an automated image analysis tool that can robustly and reproducibly measure live-cell organoid responses in high-throughput experiments without the use of potentially toxic or confounding fluorescence techniques.

A number of software tools have been developed to automate the process of brightfield/phase-contrast organoid image analysis. These platforms use conventional image processing methods, such as adaptive thresholding and mathematical morphology [21], or convolutional neural networks [22–24] to identify organoids in sequences of microscopy images. Despite their advantages, many existing platforms require cellular nuclei to be transgenically labeled [23], which increases experiment time and complexity and may modify cellular dynamics. Other existing platforms require manual tuning of parameters for each image [21], focus on single-timepoint analysis [24], only provide population-averaged (bulk) measurements without single-organoid resolution, or are limited to bounding-box detection [22], which fails to capture potentially useful morphological information at the individual organoid level. Changes in organoid shape, such as spiking or blebbing, can reveal important responses to external stimuli and might be missed with bounding-box measurements [25]. Many of these existing platforms were also developed for the analysis of organoids derived from a single type of tissue and for images obtained with one specific optical configuration, precluding their use across different experimental settings.

To address these challenges, we have developed a software platform, OrganoID, that can identify and track individual organoids in a population derived from a wide range of tissue types, pixel by pixel, in both brightfield and phase-contrast microscopy images and in time-lapse experiments. OrganoID consists of (i) a convolutional neural network, which detects organoids in microscopy images, (ii) an identification module, which resolves contours to label individual organoids, and (iii) a tracking module, which follows identified organoids in time-lapse imaging experiments. Most importantly, OrganoID was able to accurately segment and track a wide range of organoid types, including those derived from pancreatic ductal adenocarcinoma, adenoid cystic carcinoma, and lung and colon epithelia, without cell labeling or parameter tuning. The OrganoID software overcomes a major hurdle to organoid image analysis and supports wider integration of the organoid model into high-throughput applications.

Design and implementation

A convolutional neural network for pixel-by-pixel organoid detection

We developed a deep learning-based image analysis pipeline, OrganoID, that recognizes and tracks individual organoids, pixel by pixel, for bulk and single-organoid analysis in brightfield and phase-contrast microscopy images (Fig 1A). The platform employs a convolutional neural network to transform microscopy images into organoid probability images, where brightness values represent the network belief that an organoid is present at a given pixel. The network structure was derived from the widely successful u-net approach to image segmentation [26] (Fig 1B). The u-net approach first passes each image through a contracting series of multidimensional convolutions and maximum filters to extract a set of deep feature maps that describe the image at various levels of detail and contexts, such as edges, shapes, and textures. The feature maps are then passed through an expanding series of transposed convolutions, which learn to localize the features and assemble a final output.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Architecture and evaluation of the OrganoID platform and neural network.

(a) The OrganoID software automates robust analysis of organoid microscopy images. Contours are detected pixel by pixel and then separated into distinct organoids for bulk or single-organoid analysis. Identified organoids can also be tracked across time-lapse image sequences to follow responses over time. (b) Microscopy images are processed by a convolutional neural network to produce images that represent the probability that an organoid is present at each pixel. The network follows the u-net architecture, which applies a series of two-dimension convolutions, maximum filters, and image concatenations to extract and localize image features. Feature channel depths were minimized to limit overfitting and computational power required to use the tool. Scale bar 100 μm. (c) The intersection-over-union (IOU) metric, defined as the ratio of the number true positive pixels to the union of all positive pixels, was used to measure the quality of the neural network detections. To compute the IOU, pixels above 0.5 in the network prediction image were marked as positives. Examples of IOU values for several degrees of overlap are shown. (d) A representative set of images of different organoid types from the test dataset are shown with the corresponding OrganoID detections. Detections are overlayed on top of ground truth measurements and colored according to the schema in (c). The mean and standard deviation of IOU for images of each organoid type in the test dataset are also shown. Scale bar 100 μm.

https://doi.org/10.1371/journal.pcbi.1010584.g001

The OrganoID neural network follows the u-net structure and was optimized to require far fewer feature channels than the original implementation, as a reduction in trainable model parameters limits overfitting and minimizes the amount of memory and computational power required to use the network in the final distribution [27]. The original u-net implementation uses 64 filters in the first layer, which results in quite a large number of trainable parameters across the full network (over 30 million) [26]. We sequentially reduced the number of filters in the first layer by powers of two to reach a minimal value that preserved performance on the validation dataset (validation cross-entropy loss was minimal at 0.088 for 8 filters in the first layer, compared to 0.098 for 4 filters and 0.1 for 16 filters). The final OrganoID u-net structure uses only 8 filters in the first layer, which results in a network structure with less than 500,000 trainable parameters (a 98% reduction compared to the original implementation). All convolutional neurons were set to compute outputs with the exponential linear unit activation function, which has been previously shown to produce higher accuracy than the rectified linear unit (ReLU, used in the original u-net implementation) and avoid vanishing gradient problems during network training [28]. Neurons in the final convolutional layer were set with a sigmoid activation function to produce a normalized output that corresponds to the probability that an organoid is present at each pixel in the original image. All images are automatically contrasted and resized to 512x512 pixels before training and inference. Python was used for the entire OrganoID platform and Keras (a neural network interface for the TensorFlow software library) was used for network expression, training, and operation.

An original dataset of 66 brightfield and phase-contrast microscopy images of organoids were manually segmented to produce black-and-white ground truth images for network training (52 image pairs) and validation (14 image pairs) (S1A Fig). Each image featured 5 to 50 organoids derived from human pancreatic ductal adenocarcinoma (PDAC) samples from two different patients. Organoids were either grown on a standard tissue culture plate or on a microfluidic organoid platform [29]. To teach the network that segmentations should be independent of imaging orientation, field of view, lens distortion, and other potential sources of overfitting, the training dataset was processed with the Augmentor Python package [30] with random rotation, zoom, elastic distortion, and shear transformations to produce an augmented set of 2,000 images for training (S1B Fig). The neural network was trained on this dataset with the Adam stochastic optimization algorithm [31] at a learning rate of 0.001. Unweighted binary cross-entropy between predicted and ground-truth segmentations was used for the loss function. Layer weights were initialized with the He method [32]. An early stopping rule was used to halt training once performance on the validation dataset reached a minimum (i.e. once 10 epochs pass with no improvement in validation loss). The batch size was set to train on 8 augmented images for every round of backpropagation. Dropout regularization was introduced after all convolutions to randomly set 12.5% of neuron weights to zero after each batch. After each epoch, a copy of the model was saved for additional evaluation of the training process.

Identification of individual organoids with diverse morphology and size

The convolutional neural network detects organoids in an image on a pixel-by-pixel basis, which can be used to measure bulk responses. For single-organoid analysis, pixels must be grouped together to identify individual organoids. This task is straightforward for isolated organoids, where all high-belief pixels in a cluster correspond to one organoid, but is more difficult for organoids that are in physical contact. To address this challenge, we developed an organoid separation pipeline (Fig 2A) that uses the raw network prediction image to group pixels into single-organoid clusters. Conventionally, neural network image segmentation methods set an absolute threshold on predicted pixels to produce a binary detection mask. This approach is effective but discards useful information about the strength of predictions. The segmentations that were used to train the neural network were produced with a 2-pixel boundary between organoids in contact that follows the contour of separation. As a result, the network predictions were marginally less confident about pixels near organoid boundaries (Fig 2A). We took advantage of this phenomenon to identify organoid contours with a Canny edge detector [33], an image transformation which (i) computes the partial derivative of pixel intensity to identify high-contrast areas, (ii) blurs the image to smooth noisy regions, and (iii) applies a hysteresis-based threshold to identify locally strong edges. Edges are removed from the thresholded prediction image to mark the centers of each organoid. These centers are then used as initializer basins in a watershed transformation [34], an algorithm inspired by the filling of geological drainage basins that is used to segment contacting objects in an image. The image is further refined to remove organoids that may be partially out of the field-of-view or below a particular size threshold. The pipeline outputs a labeled image, where the pixels that represent an individual organoid are all set to a unique organoid ID number, which can be used for single-organoid analysis (Fig 2A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2.

(a) OrganoID can identify individual organoid contours, including those in physical contact. An example image (left) is shown to demonstrate the steps of single-organoid identification. The neural network predictions (second-from-left) are observably less confident for pixels at organoid boundaries (enlarged view, indicated with white arrows), which enables edge detection with a Canny filter (second-from-right). Edges are used to identify organoid centers, which serve as basin initializers for a watershed transformation on the prediction image to produce a final single-organoid labeled image (right). (b) The identification pipeline was used to count the number of organoids in images from the test dataset. These counts were compared to the number of organoids in the corresponding manually segmented images. The concordance correlation coefficient (CCC) was computed to quantify measurement agreement (left). Bland-Altman analysis (right) demonstrates low measurement bias and limits of agreement. Black dots are test images. Green line in the left plot is y = x. (c) The area of each organoid in all test images was also measured manually and with OrganoID. Measurements were compared with CCC computation (left) and Bland-Altman analysis (right). Black dots are identified organoids. (d) Identified organoids in time-lapse microscopy images are matched across frames to generate single-organoid tracks and follow responses over time. Shown are images of three timepoints from an organoid culture experiment. (e) Automatically measured growth curves for a selected set of organoids from the experiment in (d).

https://doi.org/10.1371/journal.pcbi.1010584.g002

The scipy [35] and scikit-image [36] packages were used to identify individual organoids from the network detection images. Detection images were thresholded at 0.5 and passed through a morphological opening operation to remove weak and noisy predictions. Partial derivatives were computed with a Sobel filter, passed through a Gaussian filter (σ = 2), and converted to an edge mask with a hysteresis threshold filter (T_hi = 0.05, T_lo = 0.005), which was then was used to mark organoid centers. The watershed method was used to identify filled organoid contours, with the organoid centers as label initializers and the network prediction image as an inverted heightmap. Labeled organoids were morphologically filled and discarded if the total area was below 200 pixels. Images were morphologically processed with scikit-image to record properties of identified organoids.

Results

Network training halted after 37 epochs, when segmentation error (binary cross-entropy) on the validation dataset converged to a minimal value (S1C Fig and S1 Video). After hyperparameter tuning, the final model performance was assessed on a novel PDAC organoid testing dataset, previously unseen by the network. Performance was quantified with the intersection-over-union (IOU) metric, which is defined as the overlap between the predicted and actual organoid pixels divided by their union (Fig 1C). An IOU greater than 0.5 is generally considered to reflect a good prediction [38,39] and we chose this value to be the minimal benchmark for satisfactory model performance. All PDAC images passed the benchmark, with a mean IOU of 0.74 (SD = 0.081) (S2 Fig).

Because the network was trained and assessed solely with images of PDAC organoids, we were also curious to evaluate its capacity to generalize to organoids derived from non-PDAC tissues. Microscopy images of organoids derived from lung epithelia, colon epithelia, and salivary adenoid cystic carcinoma (ACC) were manually segmented (6 images for each tissue type). The OrganoID network passed the benchmark for all of these non-PDAC images, with a mean IOU of 0.79 (SD = 0.096) (S2 Fig). Additionally, 19 images of organoids derived from mouse small intestine were segmented and evaluated with the model and passed the benchmark with a mean IOU of 0.70 (SD = 0.15). These results support the potential of the OrganoID neural network to generalize to organoids from various tissues of origin without parameter tuning (Fig 1D). However, 3 images of mouse small intestinal organoids did not pass the benchmark, which suggests a limit to this generalizability. To demonstrate how performance can be improved on novel datasets, a copy of the neural network was additionally trained on a subset of mouse small intestinal organoid images. This retrained network showed considerable improvement on a test dataset of mouse intestinal organoids with a mean IOU of 0.82 (SD = 0.09) while maintaining performance on the original test dataset with a mean IOU of 0.76 (SD = 0.10) (S2 Fig).

The network was also evaluated for appropriate exclusion of non-organoid technical artifacts. Air bubbles in culture media or gel matrix were rarely detected by OrganoID with a false positive rate of 4.2% (S3A Fig). We observed that OrganoID segmentations also avoided cellular debris or dust embedded into the gel, ignored chamber borders, and performed robustly across microscope resolutions, organoid concentrations, and organoid shapes (S3B–S3G Fig). OrganoID proved to be computationally efficient; each image was segmented in ~300 milliseconds on a laptop CPU (Intel i7-9750H, 2.6GHz) with less than 200 megabytes of RAM usage.

For quantitative validation of single-organoid analysis, OrganoID was used to count and measure the area of organoids in images from the PDAC and non-PDAC human testing datasets (a total of 28 images). These data were then compared to the number and area of organoids in the corresponding manually segmented images. Organoid counts agreed with a concordance correlation coefficient (CCC) of 0.95 [95% CI 0.89–0.98]. OrganoID, on average, detected 1.1 fewer organoids per image than manual segmentation. The limits of agreement between OrganoID and manual counts were between -10.35 and 8.14 organoids (Fig 2B). Organoid area comparison demonstrated a CCC of 0.97 [95% CI 0.96–0.97]. OrganoID area measurements were biased to be 1.06x10³ μm² larger per organoid with limits of agreement between -3.42x10³ μm² and 5.55x10³ μm² (Fig 2C). Several additional morphological metrics were calculated, including circularity (the ratio of the organoid perimeter to the perimeter of a perfect circle), eccentricity (a measure of elliptical deviation from a circle), and solidity (the ratio of the organoid area to the area of its convex hull). These three metrics showed moderate concordance (0.52, 0.41, and 0.59 respectively) between automated and manual measurements (S4 Fig). Performance on brightfield microscopy images was higher than performance on phase-contrast microscopy images. Overall, the measurements produced by OrganoID were in considerable agreement with those obtained by hand in our test dataset, which supports the use of OrganoID for automated single-organoid analysis.

For validation of the organoid tracking module, microscopy images taken every 2 hours from a 92-hour organoid culture experiment were passed through the entire OrganoID pipeline to produce growth curves that followed single-organoid changes over time (Fig 2D–2E). The tracking step was also performed by hand to evaluate automated performance. OrganoID tracking maintained over 89% accuracy (defined as the fraction of identified organoids that were correctly matched at each time step) throughout the duration of the experiment (S5 Fig and S2 Video). Tracking accuracy was not affected when frames were discarded to simulate a 12-hour imaging interval, which demonstrates low susceptibility to imaging frequency.

OrganoID normalizes single-organoid death responses over time

PDAC organoids were treated with serial dilutions of gemcitabine (3nM-1000nM), an FDA-approved chemotherapeutic agent commonly used to treat pancreatic cancer. Propidium iodide (PI), a fluorescent reporter of cellular necrosis, was also added to the culture media to monitor death responses. Cultures were then imaged every 4 hours over 72 hours and the produced brightfield images were processed with the OrganoID platform to identify organoids and analyze timelapse features (Fig 3A). At gemcitabine concentrations above 3 nM, the total organoid area increased for the first several hours, which reflected initial organoid growth, but then decreased to a value and at a rate inversely proportional to gemcitabine concentration (Fig 3B). Identified organoid counts for gemcitabine concentrations above 10 nM also sharply decreased over time. The total fluorescence intensity of the PI signal increased over time to a value and at a rate proportional to gemcitabine concentration, however the response to 100 nM gemcitabine appeared to induce a stronger death signal than the response to 1000 nM gemcitabine (Fig 3C).

Measurements of death stains such as PI are typically normalized to a viability measurement that accounts for differences in the number and size of organoids between replicates that can compound over the duration of an experiment. We hypothesized that OrganoID could be used to measure morphological features indicative of organoid viability over time, which could then be used to normalize and decrease the variability of measurements across experimental replicates. Furthermore, organoid tracking with OrganoID would enable per-organoid measurement normalization, which we observed to significantly decrease the coefficient of variation of organoid area and fluorescence change across organoids imaged at the same time point and exposed to the same concentration of gemcitabine (S6 Fig). Normalization of PI intensity with tracked single-organoid area indeed increased the separation of responses between each treatment group over time, decreased standard error across replicates, and corrected the response discrepancy between the 100 nM and 1000 nM conditions (Fig 3C). We also compared dose-response measurements to an MTS proliferation assay ((3-(4,5-dimethylthiazol-2-yl)-5-(3-carboxymethoxyphenyl)-2-(4-sulfophenyl)-2H-tetrazolium), a gold-standard indicator of cell viability, at the experiment endpoint. The MTS assay determined the half-maximal effective concentration (EC₅₀) to be 26.13 nM of gemcitabine. However, total fluorescence and organoid area underestimated this value to be 16.95 nM and 12.67 nM, respectively. In contrast, the dose response of area-normalized fluorescence determined the EC₅₀ to be 24.45 nM, considerably closer to the MTS standard (Fig 3D).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. OrganoID facilitates morphological analysis of a chemotherapeutic dose-response experiment.

(a) PDAC organoids were treated with a serial dilution of gemcitabine (3 nM to 1,000 nM) and imaged over 72 hours. Propidium iodide (PI) was used to fluorescently label dead organoids. Shown are representative brightfield images from three time points for control and 30 nM gemcitabine conditions. OrganoID was used to identify organoid contours, which are displayed on top of the PI fluorescence channel. (b) OrganoID measurements of fold change (FC) organoid area (top) and number of organoids (bottom) over time for each concentration of gemcitabine. Measurements were normalized to the initial timepoint. Error bars represent standard error of the mean (n = 6). (c) Total PI fluorescence intensity above control for each concentration of gemcitabine (top). The fluorescence intensity per area of each OrganoID-identified and tracked organoid were then normalized to the timepoint of first detection, which improved separation of responses to difference concentrations of gemcitabine and reduced standard error across replicates (bottom). (d) An MTS assay was used as a gold standard measurement of viability to compute the half-maximal effective concentration (EC₅₀) at the end of the experiment. Organoid area, total fluorescence, and area-normalized fluorescence were also measured at endpoint to produce dose-response curves and estimate EC₅₀ without the need for live-cell staining. (e) Descriptive shape measurements, including circularity (the inverse ratio of organoid perimeter to the perimeter of a perfect circle with equivalent area), solidity (the ratio of organoid area to its convex hull), and eccentricity (elliptical deviation from a perfect circle), were computed for each organoid. The effect size (Cohen’s d statistic) between each gemcitabine concentration is shown. Effect size is computed from dosages on the x-axis to dosages on the y-axis.

https://doi.org/10.1371/journal.pcbi.1010584.g003

Availability and future directions

Organoids have revolutionized biomedical research through improved model representation of native tissues and organ systems. However, the field has yet to fully enter the high-throughput experimental space. A central bottleneck is the challenge of automated response measurement and analysis in large numbers of microscopy images. Organoids exhibit striking diversity in morphology and size and can move through their 3D environment into and out of the focal plane; current image processing tools have not quite been able to capture these aspects in a robust manner. We developed OrganoID to bridge this gap and automate the process of accurate pixel-by-pixel organoid identification and tracking over space and time.

Experimental replicates in organoid studies can differ in the number and distribution of sizes of organoids. This difference between identical conditions requires per-sample normalization of response measurements to baseline growth of organoid colonies. There are several commercially available live-cell assays that can facilitate normalization in 2D culture. However, these same assays have proven difficult for organoid use due to the production of toxic photobleaching byproducts, limited diffusion through the gel matrix, and nonspecific staining of the gel matrix that results in a considerable background signal. Another available option is to genetically modify each organoid sample to express fluorescent proteins; however, this method increases experimental time and complexity and may alter cellular dynamics from the original tissue sample. The OrganoID platform can be leveraged for accurate normalization of standard organoid assays without live-cell fluorescence methods. OrganoID is also uniquely useful for efficient quantification of single-organoid morphological features, such as circularity, solidity, and eccentricity, that can reflect important dynamic responses. In our gemcitabine dose-response experiment, we observed that PDAC organoid circularity and solidity decreased with an increase in gemcitabine dosage. This disturbance of organoid architecture is likely due to the interference of gemcitabine with RNA and DNA synthesis [40], which may in turn affect cell turnover and production of signaling and structural proteins.

In this work, we have also contributed a manually segmented organoid image dataset for use in other computational platforms. OrganoID has demonstrated compatibility with organoids of various sizes, shapes, and sample concentrations as well as various optical configurations. Most excitingly, the OrganoID model was trained and validated on images of PDAC organoids but still demonstrated excellent generalization to images of other types of organoids, including those derived from colon tissue, lung tissue, and adenoid cystic carcinoma.

OrganoID was trained with and tested on a diverse yet relatively small set of images. Despite the suggested generalizability of OrganoID to various samples and optical configurations, performance may still differ with other types of organoids with significantly different morphology. Performance on a dataset of mouse small intestinal organoids passed our benchmark on average, however there were several challenging images that were not processed adequately. We demonstrated further training on a handful of images of mouse small intestinal organoids to improve performance on such challenging datasets. As well, OrganoID can only detect and assign a single organoid to each pixel in an image. While the platform can appropriately identify contours of organoids in physical contact, it cannot distinguish organoids that overlap across the focal plane. These limitations could be overcome in future work with a network model that produces multiple outputs per pixel as well as additional validation, an expanded training dataset, and the use of multiple focal planes for image analysis.

We have released the OrganoID platform open-source and freely licensed on GitHub (https://github.com/jono-m/OrganoID). The repository includes all source code as well as usage instructions and scripts used for the examples presented in this paper. The network training module is also included on the repository to allow further model training to improve performance for any untested applications. The training and testing dataset is available at through the Open Science Framework (https://osf.io/xmes4/). Our image analysis platform serves as an important tool for the use of organoids as physiologically relevant models in high-throughput research. The platform can accurately capture detailed morphological measurements of individual organoids in live-cell microscopy experiments without the use of genetic modifications or potentially cytotoxic dyes. These metrics can reveal important organoid responses that might be otherwise obscured with the use of bounding-box tools. The ability of the platform to generalize to a range of organoid types without parameter tuning also reflects the potential for the platform to standardize morphological assay readouts and improve measurement reproducibility. OrganoID achieves comprehensive and expedient image analysis of organoid experiments to enable the broader use of organoids as tissue models for high-throughput investigations into biological systems.

Supporting information