Fig 1.
The anatomy of the cochlea [35].
Sensory hair cell stereocilia sit along a tonotopic axis that follows the length of the spiral-shaped cochlea with a striking pattern of increasing stereocilia lengths from the base to the apex of the cochlea. The morphological and spatial features of cochlear hair cell stereocilia follow extremely predictable tonotopic patterns between individuals and species: Stereocilia lengths increase as a function of position along the tonotopic axis of the cochlea, which in turn is reflective of the frequency of sound they are tuned to detect. Thus, the relationship between the morphological and spatial features of these cells and their function are relatively well-defined compared to many other biological systems. Due to their highly patterned organization, cochlear tissues thus present a particularly striking opportunity for automated computer vision tasks.
Table 1.
Comparison of architectures based on performance metrics for Z-focus tracker.
The results highlight that ZFT-Net outperforms all other architectures across all evaluation metrics. Notably, ZFT-Net achieved a switch count of 0, meaning its predictions consistently transitioned from class 0 (pre-cellular zone) to class 1 (cellular clarity phase), and finally to class 2 (noise saturation zone), without fluctuations or misclassifications. This indicates the model’s robustness in maintaining a stable and accurate prediction sequence. Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 2.
The confusion matrix for ZFT-Net is presented alongside a corresponding bar plot, illustrating the overlapping predictions for the CCZ class, represented by the digit 1.
Errors are color-coded and oriented to reflect the different cases in the confusion matrix. GT0, GT1, and GT2 correspond to the Pre-Cellular Zone (PCZ), Cellular Clarity Zone (CCZ), and Noise Saturation Zone (NSZ), respectively. At the bottom of the figure, we provide an example of the first subject’s prediction, visualized through a barplot. It shows a smooth transition in prediction from 0 (PCZ), to 1 (CCZ), to 2 (NSZ) without any fluctuation errors, except for two minor boundary errors that are likely due to annotator subjectivity. For ZFT-Net, Conv represents Convolution layers, BN stands for Batch Normalization, ReLU is the Rectified Linear Unit activation function, Pool refers to Pooling layers, and FC denotes Fully Connected layers.
Table 2.
Performance comparison of different architectures using various error metrics reveals that DenseNet121 has optimal performance across all the metrics.
Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 3.
Panels A, B, and C show instances where inter-row stereocilia bundles are in close proximity, yet the algorithm successfully separates them, demonstrating robust performance.
Panel D presents a scenario where intra-row bundles are tightly clustered; however, the algorithm efficiently distinguishes each bundle, highlighting its ability to resolve complex spatial relationships within the samples. In Panel E, despite the raw image being notably dark, the algorithm remains effective in detecting and segmenting individual bundles.
Fig 4.
The algorithm successfully detects stereocilia bundles in 2D across successive frames, assigning unique IDs to each detected object within the 2D plane.
The variation in IDs across frames reflects the independent detection process in each 2D frame. Subsequently, a multi-object assignment algorithm intervenes to reconcile these IDs, effectively re-assigning them to maintain consistency across frames. This step is crucial for reconstructing accurate 3D objects, ensuring that each bundle retains a consistent ID throughout all frames.
Fig 5.
Detailed performance indicators are presented.
The metrics are evaluated across two categories: typical cases that do not present extreme challenges and complex cases affected by factors such as contrast variations, noise, overlapping bundles, and close proximity between them. Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 6.
Comparison of raw crops (first row), GT masks for the first human annotator (second row), GT masks for the second human annotator (third row), and the Predicted masks (fourth row) for 10 different crops from the same stack, the IoU score underneath the crops indicate the score between the 3D GT masks and the 3D predicted masks.
We examined the margin of error between two annotators for a single 3D stack over 47 instances. The average overlap was 0.70 between two annotators, 0.74 between one annotator and the prediction, and 0.76 between the second annotator and the prediction.
Fig 7.
Comparison of the IoU distribution (left) and density plot (right) between model predictions and annotators. Source data are provided in Supplementary Information S1 Data.xlsx.
[A] Box plot comparing the IoU alignment of model predictions with Human 1 and Human 2, and the agreement between Human 1 and Human 2. Statistical tests indicate that the model predictions are significantly better aligned with both Human 1 (p = 0.0002) and Human 2 (p = 0.0000) compared to the agreement between the two annotators. Notably, there is no significant difference between the model’s alignment with Human 1 and Human 2 (p = 0.0879). [B] A density plot (KDE) estimates the distribution of IoU values by smoothing the data with small curves (kernels) at each point. The y-axis shows the density, representing the relative likelihood of observing the IoU values at different points on the x-axis. The density curves for Model vs. Human 1 and Model vs. Human 2 show considerable overlap, indicating similar IoU distributions between the model and both annotators. In contrast, Human 1 versus Human 2 shows less overlap, suggesting greater variability between the annotators’ manual segmentation.
Fig 8.
An illustration of the plugin’s automated method for measuring the distance from the tip to the bottom of stereocilia bundles.
Users can adjust the positions of the upper and lower points. Upon making these adjustments, the plugin’s listeners automatically detect the changes, redraw the connecting line, and recalculate the distance accordingly. Left: shows the 3D visualization using Napari. Right: demonstrates how the distance is computed from x, y, and z, considering the physical resolution in microns.
Fig 9.
Data set crops that are associated to the computations used in Table 3, See Fig S2 A-D, where all four complete datasets are visualized for reference.
Table 3.
Measurements from VASCilia and Observers with Descriptive Statistics
Table 4.
Pearson correlation coefficients, p-values, and paired t-test results.
Fig 10.
This figure consists of two panels.
Panel (A) on the left shows bundle heights and tonotopic variations of phalloidin fluorescence intensity in P5 mouse cochlear hair cells across all cochlear regions. The tonotopic gradient observed here may not be representative of other developmental ages, which were not examined in this study. Panel (B) presents the anti-EPS8 analysis, comparing anti-EPS8 fluorescence intensity between wild type and knockout stereocilia bundles from the mid-cochlear region at P30. Source data are provided Supplementary Information S1 Data.xlsx.
Fig 11.
The plot demonstrates the utility and ease with which users can compare data between bar plots and actual cellular images within the plugin.
Top: Segmented bundles; class legend (IHC WT = yellow, OHC WT = red). Green numbers mark each bundle’s ID. Middle: Per-bundle fluorescence (phalloidin) for three depth-aggregation modes: Mode 1:All layers (use each bundle’s full depth); Mode 2: Common depth only (truncate to depth across bundles); Mode 3: Pad to max depth (pad shallower bundles to
depth by repeating the weakest-layer value, then sum). Bottom: Galleries from four example cells (shared same depth). Across cells, IHC bundles show consistently higher fluorescence than OHC bundles, reflecting greater F-actin signal. Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 12.
This figure presents the texture analysis displayed at the top as panel (A), and the anti-CDH23 fluorescence quantification analysis included at the bottom as panel (B).
Source data are provided in Supplementary Information S1 Data.xlsx. (Panel A) [Top row]: t-SNE of texture features [I to IV] and PCA projection show clear genotype separation in feature space, with crops from the two genotypes providing visual representation. (Panel A) [I to V]: GLCM energy by genotype (Cdh23−/− bundles have lower energy than Cdh23+/−, reflecting less uniform, more complex textures; differences are statistically significant); GLCM correlation by genotype (Cdh23−/− bundles exhibit lower correlation than Cdh23+/−, indicating weaker spatial regularity; differences are statistically significant); GLCM contrast by genotype (Cdh23−/− bundles show higher contrast than Cdh23+/−, consistent with greater local intensity variation; differences are statistically significant); GLCM homogeneity by genotype (Cdh23−/− bundles show reduced homogeneity relative to Cdh23+/−, consistent with more uneven local texture; differences are statistically significant); and Random forest feature importance highlights key GLCM/LBP drivers. (Panel B): Anti-CDH23 immunofluorescence quantification analysis for P7 IHC and OHC stereocilia bundles from a mid-cochlear location. Each data point represents a value obtained from an individual bundle, normalized to the maximum value within the dataset. Exemplar images on the right show stereocilia bundles stained with phalloidin (white) and anti-CDH23 antibody (magenta), with the automatically generated stereocilia bundle mask overlaid.
Fig 13.
left: covariance matrix for model prediction, others: Grad-CAM Visualization for Base, Middle, and Apex; Resized and Overlaid Response from the Last Convolutional Layer Highlighting Focus on Bundles During Decision Making.
Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 14.
Automated Computation of Stereocilia Bundle Orientation Using a Height-Only Method.
Top Left: illustrates the bundle orientations superimposed on the raw data, Top right: displays the 3D segmentation masks with bundle orientation highlighted. Bottom Right and Left are cropped regions for a closer look. Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 15.
The limitations of the Height-Only orientation computation.
While the Height-Only method excels with cells exhibiting a clear V-shape Fig 14, it struggles with some apical region hair cells that lack this distinct structure. To address these challenges, we developed the Height and Distance approach, which effectively handles a wider variety of cell shapes. The visual comparison includes the frames from which the crops are taken (first column), Height-Only results superimposed on the 3D segmented labels (second column) and the original images (third column), alongside the height and distance results superimposed on the 3D segmented labels (fourth column) and the original images (fifth column). We observe that the Height and Distance method overcomes the limitations of the Height-Only computation. Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 16.
First row: Successful cases for all methods—KMeans, GMM, and Deep Learning—in accurately clustering the four rows into their respective categories are clearly demonstrated.
This scenario represents an ideal case where each row is well-separated, linearly aligned, and free from outliers that simplify the task of accurate clustering. Second and third rows: Failure cases for KMeans and GMM in accurately clustering the four rows into their respective categories are evident: IHC1 in yellow, OHC1 in cyan, OHC2 in green, and OHC3 in magenta. These traditional methods often struggle to precisely segregate the rows due to their inherent limitations in handling complex data distributions, outliers, and overlapping clusters. In contrast, Deep Learning significantly outperforms both KMeans and GMM, providing accurate and reliable clustering for all cell types. Errors are represented by red bounding boxes. For the sample in the second row, there are Five errors in KMeans, three errors in GMM, and no errors with deep learning. For the sample in the third row, there are Sixteen errors in KMeans, twenty eight errors in GMM, and no errors with deep learning
Fig 17.
Comprehensive Examination of Error Rates, Heatmaps, and Cumulative Errors for Cell Type Identification in the Apex Region.
Subplots (A, B, E, and F) illustrate error rate and cumulative errors, with blue representing KMeans, brown for GMM, and green denoting Deep Learning. Subplots (C and D) utilize the ’inferno’ colormap to depict error rates, transitioning from black (low errors) to yellow (high errors), providing a visual gradient of error severity. This color-coded representation aids in distinguishing the methodologies applied across different datasets and highlights the specific error dynamics associated with each method. Source data are provided in Supplementary Information S1 Data.xlsx.
Fig 18.
VASCilia enables the ear research community to process their cochlea samples through an end-to-end workflow, all within a user-friendly interface.
Fig 19.
Determining the precise orientation of the frame is challenging, as not all stereocilia bundle rows share the same horizontal alignment.
Achieving an exact degree out of 360 is practically impossible, even manually. To address this, we decided to use 72-degree intervals with 5-degree increments. We have adapted the augmentation process, shown in Fig 20, to increase the number of image samples.
Fig 20.
To expand our dataset and represent all possible angles, we augmented each frame by rotating it and cropping the largest area without empty pixels from padded regions.
This process results in frames with varying perspectives and scales. In the example shown, the original slide is highlighted in orange, while all others are augmented versions.
Table 5.
Summary of 3D stereocilia bundle instances across different data sets.
Fig 21.
Augmentation of maximum projection images from 3D confocal stacks for training a classification model that discriminates between IHC, OHC1, OHC2, and OHC3.