Figure 1.
Example image and manual focus identification.
(A) Spread nuclear DNA from budding yeast meiosis stained with DAPI (DNA, left) and antibodies against Zip3-GFP (middle). The merged image is shown in false colour on the right. Scale bar = 2 µm. (B) Experimenter-labelled maxima (foci) of Zip3-GFP within the DNA region (determined by the experimenter). (C) Quantification of foci from the same 21 images by three different experimenters (P1 to P3). ‘Expert’ refers to scientists with previous knowledge of Zip3 or Msh4 quantification. Magenta bars represent the arithmetic mean, the black bar in the box-and-whisker plot shows the median value and all individual data points are shown as dots. Whiskers extend 1.5× of the interquartile range or to the minimum/maximum value, when these fall within 1.5× of the interquartile range.
Table 1.
Comparison of manual focus selection by two different experimenters across 21 imagesa.
Figure 2.
Comparison of focus assignments between experimenters.
An example image shows matches (within 8 pixels) of two experimenters in green using the position of experimenter P1. Unmatched foci that were selected only by experimenter P1 are shown in yellow and unmatched foci selected only by experimenter P2 are indicated in blue. The Jaccard score for this comparison was 0.81. Only spots within the DAPI stained region were extracted for analysis. Arrows indicate examples where experimenters have clicked a short distance from a maximum.
Figure 3.
Sources of inconsistency in focus selection between experimenters.
Matches are shown in green and unmatched points in magenta. (A) Dual assignment of a single maximum by experimenter 2 (‘double clicking’); (B) mislabelling of non-true maxima; (C) interpretation of diffraction-limited foci as a single focus or dual foci (‘doublet’); and (D) arbitrary selection of different background levels to determine inclusion of foci in the analysis. Arrows indicate the discordant foci in magenta.
Figure 4.
Interpretation of low intensity foci causes variation in focus quantification between experimenters.
Plotted are the pixel intensity of the foci selected by experimenter P1 and P2 (left panel); P1 and P3 (middle panel); and P2 and P3 (right panel) for an example image from the dataset. Foci that were selected by both experimenters (within 8 pixels of each other) are shown as crosses (‘Match’); foci selected by experimenter P1 only are shown as a dash on the X-axis; and foci selected by experimenter P2 are shown as a dash on the Y-axis. A best fit line for the matched pairs is shown in blue.
Figure 5.
Agreement of focus selection between any two experimenters decreases with reduced focus intensity.
Foci were arranged by pixel intensity and divided into quartiles. The match statistics (Jaccard scores) were computed for each quartile (Q1–Q4) and the entire set (‘All’). Images are arranged in descending order of overall Jaccard score. Q1 consisted of the 25% of foci with the lowest intensity, Q2 the 26th–50th percentile (‘low intensity’); Q3 the 51st–75th percentile (‘medium intensity’), and Q4 the 25% of foci with the highest intensity (‘highest intensity’).
Figure 6.
The average focus intensity is higher with increasing agreement between experimenters (cluster size or concordance).
Greedy clustering was performed by first comparing foci selected by two experimenters using an 8 pixel radius. When the focus was selected by both experimenters, a cluster of size 2 was generated. Foci that were only picked by a single experimenter were used to seed a cluster of size 1. The clusters were compared to the third experimenter and any matches used to increase the cluster sizes such that matches between all three experimenters gave rise to cluster size 3. Clustering was performed using experimenter order 123, 231 and 312 and all results combined for plotting (A). Because experimenters clicked on slightly different parts of the focus, the focus intensity was calculated as the average pixel value selected by the experimenters. The normalized values for the cluster were calculated by dividing the average pixel value by the median value of all of the selected foci from the same image. The horizontal, dashed line indicates a normalized value of 1. The analysis of individual images are shown in (B).
Table 2.
The FindFoci algorithm closely matches human assignmentsa.
Figure 7.
Parameters obtained from a single image show wide variation in performance when applied across the entire dataset.
The heatmap shows the F1 scores for the optimal parameters from the training image (‘Image used for training’) applied to all 21 assigned images from each experimenter (‘Image with manual focus selection’ (‘Test’)). P1, P2, and P3 refer to the experimenter ID. Arrows labelled A to C are discussed in the main text. The colour key and histogram of all F1 scores are shown above the heatmap.
Figure 8.
Training on multiple images improves consistency in focus detection.
The FindFoci Optimiser was trained using 1 to 21 images. When the number of combinations of images used for training was small (n = 1 or n = 20 has 21 possible combinations; n = 2 or n = 19 has 210 possible combination) all possible combinations were used. For larger numbers of possible combinations, a random subset of 100 combinations were used to select images for training of the algorithm. The parameters obtained from training were tested against the manually marked images from each experimenter and the F1-score calculated. P1 to P3 refers to experimenter ID, with the first mention indicating the images used for training and the second referring to the manually-assigned images against which the parameter combination was assessed. I.e. ‘P1 versus P2' means that P1's marked images were used for training and the parameters were used to predict focus selection by P2.
Figure 9.
Flow chart of the automated workflow provided by the FindFoci software.
Interactive tools are shown in green; automated scriptable tools are shown in yellow. After images have been collected, manual assignment of foci can be improved by the FindFoci Helper, which aligns clicked points to their true maximum. The FindFoci Optimiser can then be trained on the resulting labelled image to identify the best parameters for the algorithm. The FindFoci GUI provides a real-time update of the results while the user changes the parameters. Training on multiple images can be achieved using the Multi-Image Optimiser plugin in order to improve consistency. The number of initial images used for training is discussed in the main text. The parameters can be applied to a large set of images using the FindFoci Batch plugin.
Table 3.
Comparison of F1-scores from human assignments or automated focus assignment to other experimentersa.
Table 4.
Focus alignment prior to training of the algorithm improves the F1-scorea.
Table 5.
FindFoci outperforms CellProfilera.