Structure of a modular pipeline for tuning in situ transcriptomics image processing with crowdsourced annotations

This section provides a high-level overview of INSTA (Fig 1). Greater detail will be provided in Sections III and IV, and examples with two different in situ transcriptomics chemistries are provided in Sections V and VI.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. INSTA (IN situ Sequencing and Transcriptomics Annotation) is a pipeline for tuning and validating spot detection methods using crowdsourced annotations.

The steps for optimizing the spot detection parameters specific to a given chemistry are as follows. Row 1: A sample image from that chemistry is filtered and annotated by an expert. From this small amount of expert input, the tool learns what a spot in this chemistry should look like and passes these parameters to a spot detector. Row 2: The images that will receive consensus annotations are individually pre-processed. The spot detector uses the learned parameters to do rough, first-pass spot-calling. A function then detects the crowded regions and recursively zooms in to them until the sub-images are sufficiently uncrowded that a human worker should easily be able to click on all the spots without getting fatigued. Then, all the sub-images and parent images are sent to Quanti.us, which is a system for crowdsourced annotation through Amazon’s Mechanical Turk platform. Crowdsourced workers annotate each crop. Row 3: Quality control is performed based on characteristics of the spots and clusters to produce consensus annotations, which are then reassembled to produce an original image that has been annotated with high precision and recall. Row 4: These annotations can be used to tune and train other spot detectors. They can also be used to quantify the performance of the spot detector that this run of the pipeline attempted to optimize.

https://doi.org/10.1371/journal.pcbi.1009274.g001

The input to the pipeline consists of images from a particular in situ transcriptomics chemistry and a spot detection algorithm to be optimized for that chemistry. An expert designates one image as a representative image for the in situ chemistry used, and annotates it. The remaining images are assigned to the test dataset.

From this small amount of expert input, the tool learns approximately what a spot in this chemistry should look like. That is, a script extracts parameters which characterize the brightness (intensity) and size (sigma of a 2D Gaussian approximation of a spot) profiles of the spots of that chemistry. These parameters are passed to a spot detector that uses scikit-image’s implementation of the Laplacian of Gaussian algorithm.[18]

The pipeline then processes each test image separately. For each test image:

1. ○. The spot detector uses the spot parameters it learned from parameter extraction to do rough, first-pass spot-calling.
2. ○. A script detects the crowded regions and recursively crops the images until the sub-images are sufficiently uncrowded that a human worker should easily be able to click on all the spots without getting frustrated or tired.
3. ○. All the pieces of the image–the crops and the parent images–are sent to Quanti.us, which is a platform for crowdsourced image annotation through Amazon’s Mechanical Turk platform. Custom crowdsourced data ingestion classes can be written to allow the pipeline to work with any crowdsourced annotation system.
4. ○. Each image is annotated by a user-defined number of workers (typically 20 to 30). In the annotation analysis stage, all the crowdsourced annotations are clustered and QC is performed based on characteristics of the spots and clusters to produce consensus annotations, which are then reassembled to produce an original image that has been annotated with high precision and recall.
5. ○. These annotations can be used to tune and train other spot detectors. They can also be used to validate or quantify the performance of the spot detector that this run of the pipeline attempted to optimize, or of another detector.
6. ○. If the optimized spot detector’s performance is satisfactory, the detector may be useful for other images. If the detector’s performance is unsatisfactory, the parameters can be modified and the detector can be reevaluated against the worker annotations.

Two key aspects of the pipeline should be highlighted: First, individual segments of the pipeline may be used separately for assorted purposes. For example, to simply get annotations for images without optimizing any spot detectors, the latter portion of the pipeline can be used to crop images and QC the crowdsourced annotations, with cropping based on spots detected by some detector that has been deemed sufficiently good for the purpose of preliminary detection. Second, workers tend to perform poorly on full-size raw images because there are too many spots and the spots are too close together. The pipeline includes a recursive cropping functionality that automatically breaks up each image into sections that the workers can handle effectively.

Section III will discuss the limits of what workers can accurately annotate with regard to brightness, density, number of spots, etc. Section IV will further discuss ways to prepare images to optimize worker performance.

Worker performance is limited by spot crowding, visibility, and quantity

Given the large number of in situ transcriptomics technologies and even larger number of tissues to analyze, we sought to understand the fundamental properties of the images required for high quality crowd-sourced annotations. In particular, we aimed to characterize worker performance as a function of signal-to-noise ratio (SNR) and the distances between spots. We hypothesized that these two properties of in situ transcriptomics images would be fundamental determinants of annotation quality.

Many factors in sample preparation and imaging affect the SNR of spots in in situ transcriptomics images. For example, some tissues exhibit higher autofluorescence owing to the composition of the cells and extracellular matrix.[32,33] Light scattering due to tissue composition further degrades SNR, with shorter wavelengths generally more degraded than longer wavelengths.[34] Understanding how the protocol chemistry and tissue properties affect performance can help guide the sample preparation optimization process. For example, optimizing sample preparation via tissue clearing [35] and fixative selection [36] can improve SNR and contrast.

The ability to resolve adjacent spots depends on both their physical distances and the magnification and microscope quality used to image them. The physical distances between mRNA molecules in the cell vary both by their abundance and localization. Dense transcripts can be imaged with higher magnification, which can improve resolution at the cost of imaging speed. Alternatively, expansion microscopy can be used to improve the spatial resolution of transcripts.[37,38] Anticipating that dense transcripts could present a challenge for workers to annotate, we developed quality control methods to identify and declump annotations from dense spots, and created the automatic image subdivision system.

We crowdsource annotations using Quanti.us (30) and perform clustering and quality control on the annotations to arrive at consensus coordinates for spot locations. Each image sent to Quanti.us is annotated by 25 workers (S2 Text). The resulting annotations are then clustered via Affinity Propagation to find the initial set of annotation clusters—this algorithm does not require a priori knowledge of the number of clusters.[39,40] Given that some of the annotations do not correspond to spot locations and some of the annotations cover adjacent spots (S4 Fig), we next perform QC to identify false positives and unmix adjacent clusters.

To determine if a cluster is a false positive, we threshold the clusters by the number of annotations in the cluster (Fig 2A). In annotations of synthetic images we observed that clusters are distributed bimodally (S3 Text) by number of annotations. Clusters with few annotations tend to be incorrect (that is, the cluster centroid is not within a given threshold pixel radius of the closest actual spot location). So for the first QC step, clusters are sorted by number of annotations and one-dimensional k-means with k = 2 is applied to find the threshold number of annotations. All clusters with fewer annotations than this threshold are removed. This thresholding method tends to be aggressive; we would rather miss spots than “detect” incorrect spots, since in reality it is inevitable that some spots will be missed anyway when signals are too faint or overlap. In an experiment with nine synthetic images (with 50, 100, and 150 spots and mean SNR of 5, 10, and 15), this QC step yielded 100%, 100%, and 100% precision (40%, 13%, and 7% increase compared with the precision obtained without thresholding clusters by number of annotations) and 51%, 78%, and 99% recall (19%, 12%, and 0% decrease) for images with mean SNR = 5, 10, and 15 respectively. For each SNR, the reported precision and recall value is the average across the three images with different numbers of spots (S5A Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. QC, including cluster size thresholding and declumping, improves precision, sometimes at the expense of recall, for images with lower SNR values.

(A) Clusters with fewer workers tend to be incorrect. Sort clusters by number of unique workers annotating them. The fraction of workers who contribute once can predict whether a cluster is clumpy (it corresponds to multiple image spots that are close together). (B) Sort clusters by fraction of unique workers contributing. Isolate and declump the clusters where many workers contribute more than once, as the inset demonstrates using real data. (Inset–orange circle: original centroid, green and purple circles: new centroids found by declumping, green and purple dots: worker annotations assigned to new centroids by declumping, stars: actual spot locations.) Clumps are declumped using 2D k-means. For more examples of declumped clusters, see S2 Fig. (C) Thresholding clusters by the number of annotations in the cluster and by the fraction of unique workers who contribute multiple times to the cluster improved recall by 17%, while decreasing precision by 11% on average, in an experiment with nine images with 50, 100, and 150 spots; mean SNR = 5, 10, and 15; and average nearest neighbor distance (NND) ≈ 11.5, 15, and 20.5.

https://doi.org/10.1371/journal.pcbi.1009274.g002

To detect whether a cluster corresponds to multiple spots that are very close together, we threshold the clusters by the fraction of unique workers who contribute multiple times to the cluster. We observed that when spots are very close together, the clusters associated with multiple spots may clump together into one annotation cluster, but some workers do detect that the spots are supposed to be separate and those workers contribute more than one click within the region that the clustering algorithm detects as one cluster. (See pink clusters in S6 Fig.) The threshold fraction is found between the main mode of the distribution and the tail by identifying the point of steepest increase in histogram values. All clusters with a greater fraction of multiple-clicking workers than this threshold fraction are removed. Therefore, the fraction of workers who contribute only once can predict whether a cluster is actually clumpy, even if sometimes the actual spots are so close that most of the workers looking at them interpret them as one spot and it is not possible to identify that cluster as clumpy (Fig 2B). We declump each clumpy cluster using two-dimensional k-means (Fig 2B). In the same experiment with synthetic images, this QC step yielded 64%, 88%, and 94% precision (4%, 1%, and 0% increase over results without QC) and 67%, 87%, and 96% recall (3%, 3%, and 3% decrease) for images with mean SNR = 5, 10, and 15 respectively (S5B Fig).

Performing declumping after false positive detection boosts recall. In the same experiment with synthetic images, removing small clusters and separating clumpy clusters yielded 100%, 100%, and 100% precision (40%, 13%, and 7% increase in precision) and 51%, 78%, and 98% recall (19%, 12%, and 2% decrease in recall) for images with mean SNR = 5, 10, and 15 respectively (Fig 2C).

We observed some limits on the utility of QC. When spot visibility was poor, nearest neighbor distance was very small, or an image had too many spots, the quality of the raw annotations was very low, so QC was unable to improve precision and recall to acceptable levels.

Using the NND-varying and SNR-varying features of our SpotImage tool, we tested the limits of spot visibility and crowdedness that workers can accurately annotate. In an experiment with synthetic images of mean SNR = 3, 5, 7, 9, and 11, and spot size = 0.5, 1.0, and 1.75 pixels (S7A Fig), small spot sizes required larger mean SNR to achieve recall greater than 50%. For example, when the spot size (sigma of a gaussian fit to the intensity) was half a pixel (about 5 microns), recall was zero until SNR > 9 (S7B Fig). At lower SNR values, even spots with large nearest neighbor distances tended to be missed, and as spot SNR increased the mean NND of undetected spots decreased (S7C Fig). While the size of cellular structures in pixels will be a function of the imaging system and chemistry used, note that the cells in the smFISH image used for Fig 3C are 250 to 300 pixels wide, and 130 to 180 pixels tall. A typical nucleus from an RCA image (e.g. idr0071-feldman-crisprko/experimentE/10X_A11_Tile-0.aligned from Feldman et al. 2018 [41]) is approximately 8–12 pixels in diameter.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Images are filtered and subdivided so that they are easier for workers to annotate.

(A) The raw image (which contains 268 spots) is pre-processed with a gaussian high pass filter, a Laplace filter, and a maximum intensity projection over z. (B) Crowded spots detected and bound. Rough, first-pass spot-calling enables the detection of crowded spots and subsequent automatic subdivision into smaller images. (C) True positive = consensus in concordance with expert annotation, false positive = consensus not in concordance with expert annotation, and false negative = no consensus found for an expert annotation. The distance between a correct consensus annotation and the nearest expert annotation is no more than the user-defined correctness threshold. The distance between a detected expert annotation and the nearest consensus annotation is also no more than the user-defined correctness threshold. (D) Applying cropping resulted in precision and recall of 97% and 87%, improvements of 50% and 64%, respectively, over the un-cropped image.

https://doi.org/10.1371/journal.pcbi.1009274.g003

We observed that the radius of the symbol that Quanti.us uses to mark spots identified by workers limits the minimum nearest neighbor distance between spots which we can reasonably expect workers to discern to 4% of the image’s width. When spots are closer than this distance, the mark on one spot obscures neighboring spots. We tested inverting the images (dark spots on a light background) before submitting them to Quanti.us to see if this would improve worker performance. A linear regression between recall with inversion and recall without inversion resulted in a slope of 1.004, with a Pearson’s correlation coefficient of r = 0.985 (S7D Fig), making it clear that inversion is not helpful.

We also sought to understand the total number of spots that workers can annotate in one image. In an experiment with SNR = 3, 5, 7 and number of spots per image between 50 and 225, the number of clicks per worker per image increased as the number of spots in the image increased, until it leveled off at around 120 on average, suggesting that 120 was the upper bound on the total number of spots workers were willing to click for the payment offered in these experiments ($0.05 per image) (S8A Fig). For example, in an assay detecting 31 genes in fresh-frozen breast cancer tissue, Ke et al. report an average of 25 spots per cell [42], which means that a worker can reliably annotate about 4.8 cells per image. If the average number of spots per cell for one gene is instead 16 (Fig 2B in [43,44]), the same worker would be able to annotate 7.9 cells per image. As the number of spots increased beyond 50, the fraction of spots that workers were willing to click decreased. On average, workers annotated almost all spots for images with 50 spots but only about 60% of all spots for images with 200 spots (S8B Fig). However, even though each worker annotated a smaller fraction of the spots as the number of spots in the image increased, most spots were still getting annotations from at least half the workers (S8C Fig). In other words, some workers detected spots which other workers missed.

These effects of spot visibility, crowding, and quantity on worker performance are intuitive, but these results provide quantitative metrics to understand worker performance as a function of image quality, and objective guidelines for image pre-processing that allow us to optimize the performance for a given image dataset. We found that for best performance the spot signal-to-noise ratio must be at least 10 and the mean nearest neighbor center-to-center spot distance must be between 12 and 15 pixels. The performance characterization in this section provides targets for the SNR and density of spots that can aid in guiding the optimization of in situ technologies to new samples. In the next section, we present the methods we developed to prepare images which workers are more likely to annotate reliably.

Image pre-processing improves workers’ ability to annotate images reliably

Given the guidelines we identified for the spot density, total number of spots and number of spots that can be accurately annotated in a given image, raw in situ transcriptomics images would need to be pre-processed in order to meet the guidelines.

In the first preparation step, the raw images are pre-processed with filters to enhance contrast between spots and the rest of the image (Fig 3A). This process is described in greater detail in S4 Text. In the second step, the filtered images are automatically subdivided to produce child images with sufficiently small spot densities and large nearest neighbor distances to be effectively annotated by workers (Fig 3B). S4 Text provides more details about preprocessing. In an experiment with a real single molecule fluorescence in situ hybridization (smFISH) image with 268 spots of typical contrast and density, annotated by 25 workers, applying cropping resulted in precision and recall of 97% and 87%, improvements of 50% and 64% respectively, over the un-cropped image (Fig 3C).

There are limits on how much an image can be cropped to increase recall. Some spots are so close together that the spots themselves appear to clump in the parent image, so no amount of zooming in would enable them to be separated. The user should also keep in mind that in the Quanti.us website cropped images are upscaled by interpolation to a size which workers can annotate, so the pixels in the upscaled image which the workers are annotating are not a perfect representation of the original data. The user should also consider the relationship between the number of crops and the overall cost of the experiment. Annotations from Quanti.us cost only five cents per worker per image (in 2020), but many workers may be needed to ensure good coverage and proper clustering and de-clumping of clusters.

Helper images

Because images from experiments using RCA (Rolling Circle Amplification, an in situ transcriptomics chemistry which is described in S5 Text) have a lot of variation in spot sizes, we sought to investigate whether we could improve the quality of the worker annotations by providing better guidance at the crowdsourcing interface, and whether including the parent image with the crops removed in the stack (e.g. second image from left, S9B Fig) affects worker accuracy. The parent image appears to the workers on the Quanti.us interface at the same size as the crops, so the spots in the parent image appear much smaller to the workers than the spots in the crops. Additionally, we designed four different helper images–two variants, with or without circles drawn around the correct spots–which illustrate what the workers should click on (S11 Fig).

For RCA image ISS_rnd1_ch1_z0, which contains 287 spots, the inclusion of helper images on average increased precision by 14% (95% with helper images and 81% without) and decreased recall by 16% (59% with helper images and 76% without) (S12 Fig). When the spots in the helper image were circled, precision was 0.4% higher and recall was 3.4% higher. Workers expressed little preference between the two variants of helper images. On average, precision and recall with images of the first variant were only 0.5% greater and 4% less than precision and recall of the second variant, respectively. However, including the parent image in the stack, which workers would also annotate, decreased both precision and recall by 6% and 4% respectively.

These results suggest that the workers were less likely to click spots they felt unsure about when helper images were provided. These results also demonstrate that for some images it is disadvantageous to show the workers the parent image, as it confuses them because of the drastic difference in scale between the parent image and the crops.

These image preparation steps complete the pipeline described in Section II (Fig 1). The performance of the pipeline will now be demonstrated with a vignette, using the rolling circle amplification (RCA) chemistry.

The results of validating spot calling algorithms using crowdsourced ground truth are comparable to the results using expert annotations

We tested the usage of crowdsourced annotations resulting from the pipeline as ground truth to validate spot calling algorithms. Using available images from RCA experiments, we sought to evaluate how well crowdsourced annotations agree with expert annotations to assess the generalizability of the tuned spot-calling parameters.

To produce the crowdsourced annotations, the inputs to the annotation generation pipeline were: One sample image with the RCA chemistry, expert spot location annotations for that image, and three test images without annotations. All images were downloaded from an in situ sequencing (ISS) experiment in the starfish database.[45] The spots which an expert had annotated in the sample image were analyzed to extract spot detection parameters required by starfish’s BlobDetector method, which implements the Laplacian of Gaussian spot detection approach. These parameters were used for first-pass spot detection on the test images, and the resulting spot coordinates were used to automatically subdivide the test images. All crops were then sent to Quanti.us to be annotated by 25 workers each. Further details are explained in S6 Text.

We sought to evaluate how well the resulting crowdsourced annotations agreed with expert annotations. Precision and recall for the consensus annotations, based on an expert’s evaluations of the test images, were 95% and 70%, 92% and 89%, and 81% and 76% for images ISS_rnd0_ch1_z0, ISS_rnd0_ch3_z0, and ISS_rnd1_ch1_z0, which contain 1236, 416, and 287 spots respectively (Fig 4). The Jaccard similarity indices (intersection over union) between the consensus annotations and the expert annotations were 71%, 85%, and 68%, respectively. We then sought to understand how well the consensus annotations and the expert annotations agreed with each other by evaluating the level of concurrence between different experts and between the same expert annotating the same image twice six months apart (at times t₀ and t₁ = t₀ + 6 mos) (S7 Text). The Jaccard indices (intersection over union) for Expert #1 at t₀ and Expert #1 at t₁, Expert #1 at t₀ and Expert #2 at t₁, and Expert #1 at t₁ and Expert #2 at t₁ were 73%, 78%, and 82% respectively (S13 Fig). Thus, the agreement we saw between the consensus and expert annotations for the RCA images are in the same range as the intra- and inter-expert Jaccard indices. We also observed that consensus annotations are more likely to include false positives for specks of debris that experts would ignore (S14A Fig). These specks are smaller and fainter than the spots an expert would annotate. However, this common error mode in the consensus annotations does not necessarily hamper the performance of a spot detector trained on consensus annotations. For image ISS_rnd0_ch1_z0, the blobs found using scikit-learn’s blob_log() algorithm with parameters based on worker consensus annotations actually had higher precision and recall (84.8% and 73.1%) than the blobs found using scikit-learn’s blob_log() algorithm with parameters based on expert annotations (75.1% and 72.8%) (S14B Fig). While recalls differ by only 0.4%, precision is significantly lower (12.1% difference) for spots found based on expert consensus because the expert identified spots with a larger range of sizes and intensities than the workers. Consequently, parameter tuning that took these edge cases into consideration caused blob_log() to capture more false positives and more spots overall (2019 vs. 1388).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The in situ transcriptomics annotation pipeline was tested using RCA (Rolling Circle Amplification) images from an in situ sequencing (ISS) experiment in the starfish database.

Worker consensus annotations for RCA test image ISS_rnd0_ch3_z0, which contains 416 spots, achieved 92% precision and 89% recall based on expert consensus annotations. The Jaccard similarity index (intersection over union) between the consensus annotations and the expert annotations was 0.85.

https://doi.org/10.1371/journal.pcbi.1009274.g004

We also sought to assess the generalizability of the tuned spot-calling parameters and the potential utility of using crowd-sourced annotations to generate ground truth for a large number of images. To do this, we ran starfish’s BlobDetector method using the spot parameters which had been extracted in this vignette on thirteen other images from starfish’s RCA dataset which had not been annotated by experts. By visual inspection, these images had a greater variation in spot size, brightness, and quantity than the other images. As ground truth, we used consensus annotations for these images. The precision of the BlobDetector method was 82 ± 10%, and the recall was 68 ± 17% (mean ± standard deviation, S16 Fig). These results suggest that when a set of spot parameters tuned to a particular channel and field of view for a chemistry are used for other channels and fields of view for the same chemistry, the spots detected are likely to be correct but fewer spots may be detected.

The finding that the Jaccard indices for the consensus and expert annotations for the RCA images were similar to the intra- and inter-expert Jaccard indices strongly suggests that consensus annotations can be used in place of expert annotations. We also found that spot parameters found for a given chemistry using consensus annotations as ground truth can be used to automatically find spots with precision (82 ± 10%, mean ± standard deviation) comparable to the agreement between two experts annotating the same image (82%). Together, these findings are especially useful for the processing of large datasets that would be infeasible for an expert to annotate by hand.

Crowdsourced ground truth is useful for tuning and validating spot calling parameters

This section tests the usage of crowdsourced annotations resulting from the pipeline as ground truth to tune and validate their parameters. Ground truth is essential for testing how well a spot-calling algorithm generalizes to other in situ transcriptomics chemistries and tuning parameters (S8 Text). We tested whether consensus and expert annotations function similarly well as ground truth to tune parameters for spot-calling algorithms, using the RCA dataset from starfish.[31] We also bootstrapped our consensus and expert annotations for the RCA dataset to explore the minimum number of ground truth annotations needed to effectively tune a spot-calling algorithm.

Usage of consensus annotations and expert annotations as ground truth to tune the LMP (starfish’s LocalMaxPeakfinder) stringency parameter, which controls the intensity cutoff to detect a peak, resulted in similar precision vs. stringency trends, as well as recall vs. stringency trends (Fig 5). The optimal stringency parameter found using consensus annotations from crowdsourced workers alone as ground truth resulted in a lower precision and slightly higher recall (89.4% and 95.4% respectively, compared with 94.3% and 94.8% from using expert annotations alone as ground truth– 5.3% and 0.63% difference respectively) for RCA image ISS_rnd0_ch1_z0, which contains 1236 spots. These results reflect the fact that some of the spots annotated by the worker consensus were not annotated by the more conservative expert.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The optimal “stringency” parameter for starfish’s LocalMaxPeakfinder (LMP) with Rolling Circle Amplification (RCA) images from the starfish database resulted in lower precision and slightly higher recall (89.4% and 95.4% respectively) for RCA image ISS_rnd0_ch1_z0, which contains 1236 spots, when consensus annotations from crowdsourced workers alone were used as ground truth for parameter tuning, compared with 94.3% and 94.8% precision and recall which were achieved when expert annotations alone were used as ground truth for parameter tuning.

The optimal stringency parameter found using consensus annotations from crowdsourced worker annotations pooled together with the expert’s annotations resulted in precision and recall almost equivalent to the results using consensus annotations from crowdsourced workers alone (89.5% and 95.3%– 0.1% percent difference for both).

https://doi.org/10.1371/journal.pcbi.1009274.g005

The optimal stringency parameter found using consensus annotations from crowdsourced worker annotations pooled together with the expert’s annotations resulted in precision and recall almost equivalent to the results using consensus annotations from crowdsourced workers alone (89.5% and 95.3%– 0.1% percent difference for both). This result likely reflects the fact that the number of expert annotations is typically much smaller than that of worker annotations, so their effect on the pool is minimal. Nevertheless, combining crowdsourced annotations with expert annotations provides a way to extend the amount of ground truth available.

S9 Text shows that the training behavior of the expert and crowdsourced annotations is very similar. That is, performance converges to roughly the same level and at roughly the same rate. When blob detection was executed on RCA image ISS_rnd0_ch1_z0, which contains 1236 spots, fifteen ground truth annotations (spots) were enough to get 99.1% and 98.1% of the maximum precision performance when the annotations were produced by experts and worker consensus, respectively. With the same number of annotations, 97.6% and 96.6% of the maximum recall performance was achieved with annotations produced by experts and worker consensus, respectively. Together with the result that consensus annotations and expert annotations resulted in similar precision vs. stringency and recall vs. stringency trends when used as ground truth to tune the LMP stringency parameter (Fig 5), this suggests that consensus annotations are a viable substitute for expert-generated ground truth for both parameter tuning and algorithm validation.