Fig 1.
INSTA (IN situ Sequencing and Transcriptomics Annotation) is a pipeline for tuning and validating spot detection methods using crowdsourced annotations.
The steps for optimizing the spot detection parameters specific to a given chemistry are as follows. Row 1: A sample image from that chemistry is filtered and annotated by an expert. From this small amount of expert input, the tool learns what a spot in this chemistry should look like and passes these parameters to a spot detector. Row 2: The images that will receive consensus annotations are individually pre-processed. The spot detector uses the learned parameters to do rough, first-pass spot-calling. A function then detects the crowded regions and recursively zooms in to them until the sub-images are sufficiently uncrowded that a human worker should easily be able to click on all the spots without getting fatigued. Then, all the sub-images and parent images are sent to Quanti.us, which is a system for crowdsourced annotation through Amazon’s Mechanical Turk platform. Crowdsourced workers annotate each crop. Row 3: Quality control is performed based on characteristics of the spots and clusters to produce consensus annotations, which are then reassembled to produce an original image that has been annotated with high precision and recall. Row 4: These annotations can be used to tune and train other spot detectors. They can also be used to quantify the performance of the spot detector that this run of the pipeline attempted to optimize.
Fig 2.
QC, including cluster size thresholding and declumping, improves precision, sometimes at the expense of recall, for images with lower SNR values.
(A) Clusters with fewer workers tend to be incorrect. Sort clusters by number of unique workers annotating them. The fraction of workers who contribute once can predict whether a cluster is clumpy (it corresponds to multiple image spots that are close together). (B) Sort clusters by fraction of unique workers contributing. Isolate and declump the clusters where many workers contribute more than once, as the inset demonstrates using real data. (Inset–orange circle: original centroid, green and purple circles: new centroids found by declumping, green and purple dots: worker annotations assigned to new centroids by declumping, stars: actual spot locations.) Clumps are declumped using 2D k-means. For more examples of declumped clusters, see S2 Fig. (C) Thresholding clusters by the number of annotations in the cluster and by the fraction of unique workers who contribute multiple times to the cluster improved recall by 17%, while decreasing precision by 11% on average, in an experiment with nine images with 50, 100, and 150 spots; mean SNR = 5, 10, and 15; and average nearest neighbor distance (NND) ≈ 11.5, 15, and 20.5.
Fig 3.
Images are filtered and subdivided so that they are easier for workers to annotate.
(A) The raw image (which contains 268 spots) is pre-processed with a gaussian high pass filter, a Laplace filter, and a maximum intensity projection over z. (B) Crowded spots detected and bound. Rough, first-pass spot-calling enables the detection of crowded spots and subsequent automatic subdivision into smaller images. (C) True positive = consensus in concordance with expert annotation, false positive = consensus not in concordance with expert annotation, and false negative = no consensus found for an expert annotation. The distance between a correct consensus annotation and the nearest expert annotation is no more than the user-defined correctness threshold. The distance between a detected expert annotation and the nearest consensus annotation is also no more than the user-defined correctness threshold. (D) Applying cropping resulted in precision and recall of 97% and 87%, improvements of 50% and 64%, respectively, over the un-cropped image.
Fig 4.
The in situ transcriptomics annotation pipeline was tested using RCA (Rolling Circle Amplification) images from an in situ sequencing (ISS) experiment in the starfish database.
Worker consensus annotations for RCA test image ISS_rnd0_ch3_z0, which contains 416 spots, achieved 92% precision and 89% recall based on expert consensus annotations. The Jaccard similarity index (intersection over union) between the consensus annotations and the expert annotations was 0.85.
Fig 5.
The optimal “stringency” parameter for starfish’s LocalMaxPeakfinder (LMP) with Rolling Circle Amplification (RCA) images from the starfish database resulted in lower precision and slightly higher recall (89.4% and 95.4% respectively) for RCA image ISS_rnd0_ch1_z0, which contains 1236 spots, when consensus annotations from crowdsourced workers alone were used as ground truth for parameter tuning, compared with 94.3% and 94.8% precision and recall which were achieved when expert annotations alone were used as ground truth for parameter tuning.
The optimal stringency parameter found using consensus annotations from crowdsourced worker annotations pooled together with the expert’s annotations resulted in precision and recall almost equivalent to the results using consensus annotations from crowdsourced workers alone (89.5% and 95.3%– 0.1% percent difference for both).