Dataset preparation

We used an extensive library of standardized, laboratory-based photographs of small, bottom-dwelling (‘cryptobenthic’) fish species across several ocean basins. Each photo was obtained from specimens that were collected in the field [32,33,44,45], immediately placed on ice and then transported to the laboratory. There, each fish was placed in a small photo-tank [46] and photographed laterally against either a black or white background, facing to the left with the fins elevated whenever possible. From this digital archive, we arbitrarily pre-selected 113 target species by applying two complementary criteria. First, we required at least 30 distinct images per species to guarantee sufficient sample size for future steps. Second, we selected species to capture as much of the morphological and phylogenetic diversity inherent in our full imagery collection. Across the selected images, only adult individuals were selected; juvenile or larval stage were not included due to variation in coloration and body shape. All candidate images were organized into a standard folder hierarchy named for the corresponding family, genus, and species. These images were then subjected to our quality-control pipeline (see below), resulting in a laboratory image dataset that comprised 7,626 high-resolution images, with an average of over 67 photos per species.

To guarantee the model’s exposure to photos obtained in underwater conditions and validate its performance, we increased our dataset with web-sourced images from three trusted repositories. These web-sourced images represent real-world, uncontrolled conditions, including wide variation in lighting, pose, background complexity, and image quality. For iNaturalist (non-commercial API access), we mined every available photo for our 113 selected species using a custom script. For FishBase and the STRI database [47], we manually scraped each species page to download all displayed images. All files were then again passed through our unified quality-control pipeline. This process yielded approximately 18,800 web-based photos, approximately 71% of our total collection.

All collected images were then subjected to a quality-control pipeline to ensure consistency and to prevent the same or similar images from appearing in both training and evaluation sets. First, we computed perceptual hashes to detect and remove exact and near-duplicate frames. Next, we used automated blur (via Laplacian variance computed with OpenCV [48]) and size checks to flag outlier images that were below our sharpness or minimum-resolution threshold for manual review and, if appropriate, removal. Every image’s label was cross-checked against its visible diagnostic traits (fish shape, body proportions, color patterns), discarding any label-morphology mismatch, suboptimal framing, or ambiguous anatomy. Finally, we manually applied a square crop to all web-sourced images, preserving fish natural morphology while eliminating distracting background clutter. After this multi-stage data QCQA process, our combined dataset contained approximately 26,500 images that passed checks and were standardized for further augmentation and model training.

The final CryptoVision dataset comprised images across 20 taxonomic families, 62 genera, and 113 species. To support robust training and unbiased evaluation, we partitioned these images into training (70%), validation (15%), and test (15%) subsets. These splits were stratified at the species level—each species retains the same relative representation in every subset as in the full dataset—ensuring that even the rarest taxa appear in all three sets and preventing “unseen” species during validation or testing.

Model architecture

To develop CryptoVision, we built upon a standard convolutional model with specialized modules for robust, taxonomically-informed fish classification. We began with an ImageNet pretrained model working as our feature extractor and inserting an augmentation pipeline to improve generalization and mitigate the small number of images per class. Then, we attached three parallel “heads” to predict family, genus and species simultaneously (Fig 1). To respect the hierarchical relationships among taxonomic levels, we introduced both novel loss functions and fusion blocks that allow higher‐level predictions to inform and correct lower‐level decisions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. End-to-End CryptoVision Workflow.

Overview of the workflow including image acquisition from laboratory and web sources, QA/QC filtering, and model training with four fusion strategies—standard (STD), feature concatenation (CONCAT), attention (ATT), and gated (GATED). Models are evaluated using precision, recall, accuracy, and our custom Taxonomic Alignment Score (TAS), and interpreted via saliency maps.

https://doi.org/10.1371/journal.pone.0349646.g001

To maximize generalization and reduce overfitting on our relatively small lab‐standard archive and more variable web imagery, we applied several image transformations during the training stage. Specifically, following established data augmentation practices in computer vision tasks for image limited datasets [49], we implemented an augmentation block (implemented via TensorFlow-Keras) including the following settings: random horizontal flips and rotations (±10%), random zoom (height/width factors 5–10%), random contrast (±20%) and brightness (±20%), random translation (±10% in both axes), a final random crop back to 352x352 pixels, and Gaussian noise (σ = 0.2). Because these operations occur in GPU memory only during training, we avoided inflated storage requirements while still presenting the model with thousands of unique “views” of each specimen.

We selected ResNet50v2 (ImageNet-pretrained model; [8] as our feature extractor due to its high initial precision, recall and accuracy and seamless integration with gradient-based interpretability tools, and a moderate parameter size (~25M). From this pretrained model we obtained a matrix of convolutional feature maps, which we then recalibrated via a Squeeze-and-Excitation (SE) block [50], characterized by the following steps sequence: 1) Squeeze: global average pooling collapses each feature channel to its mean activation. 2) Excitation: two fully connected layers—with ReLU activation after the first layer and sigmoid activation after the second—learn per-channel weights. 3) Re-scale: these weights multiply the original feature maps, amplifying more important channels. The SE-recalibrated feature maps were then reduced to a fixed-length vector by global max-pooling. We fed this vector through a shared dense layer (2048 neurons, ReLU activation and 0.3 dropout) to produce our shared embedding. This task-agnostic embedding serves as the input to all downstream taxonomic heads (family, genus, and species).

To leverage the inherent taxonomic hierarchy in nature and extract the full potential of a single deep learning model, we designed our classifier with three parallel “heads” immediately after the shared embeddings. This so-called multi-output approach created three fully connected blocks (dense layers with SoftMax activation), which map the previously shared vector into a probability distribution over its own label set. By reusing the same embeddings across all three prediction tasks, we not only reduce the model size, but also force the model to learn representations that are simultaneously useful at coarse (family) and fine (species) granularity. This multi-task arrangement lets information flow between ranks (e.g., family cues strengthen genus predictions), removes the need for separate models at each level, and ultimately yields independent confidence scores for each taxonomic level (family, genus and species).

To enhance alignment across family, genus, and species outputs, we extended our multi-output model with tree “fusion” strategies, each connected with the same shared embedding but differing in how they inject higher-level context into downstream heads. We compared these against a standard (STD) baseline that shared no information between predictions. Specifically, we applied the following designs to compare their influence on model outputs:

• STD (Standard Multi-Output): In the simplest configuration, each head considers only the shared embedding and operates independently.
• CONCAT (Logit Concatenation): The embedding vector concatenates with the family head’s raw class scores (its “logits”). The same procedure is applied to the genus prediction, where the genus head receives the shared embedding concatenated with the family logits; similarly, the species head input combines the shared embedding with the genus logits.
• GATED (Learned Gate Fusion): Instead of directly using the SoftMax output from the family prediction as input, we added a sigmoid “gate” for each embedding dimension that dynamically balances the original features against a transformed version of the family logits. The gate is also created between genus and species layers.
• ATT (Taxonomy-Conditioned Attention): For this design, we added an “attention mask” (via a small two-layer perceptron with sigmoid activations) for which values between 0 and 1 indicate how relevant each feature dimension should be for the genus predictions. By multiplying this mask with the embedding layer, we highlight channels that are most diagnostic for the family output, a process comparable to pinpointing the features that matter most given the higher-level context. The same genus attention-mask was created and coupled to species head input.

All four designs were implemented and compared in terms of their accuracy, precision, recall, and the Taxonomic Alignment Score (TAS). By progressively training and testing all these architectures, we were able to measure how much each fusion style improves hierarchical consistency and overall classification performance, ultimately revealing the optimal design for our hierarchical classification tasks.

Given that our dataset had some imbalances (between 90–270 images per species), standard cross-entropy may bias the model towards over-represented taxa and potentially ignore the strict relationships that are inherent to biological classification (i.e., misassign a species to the wrong genus or family). To address this, we developed a custom loss function—Taxonomy-Focal Cross-Loss (TFCL)—which combines class reweighting through focal loss with soft penalties that encourage hierarchical consistency across taxonomic ranks. TFCL builds upon Focal Cross-Entropy, which augments the standard cross-entropy with a modulating factor to down-weight easy examples and focus learning on harder instances [51]:

(1)

Here, is the model’s estimated probability for the true class, is the balance between positive and negative examples and is the focusing parameter, adjustable to learn on hard misclassified cases. We extended this by adding a soft consistency penalty that encourages each lower-level head to “agree” with its parent heads. Specifically, we implemented it where:

• , , are the SoftMax scores over families , genera , and species
• their true labels,
• the parent family of genus j, and the parent genus of species k.

Using these penalties, the loss for a single sample is computed as:

(2)

(3)

(4)

Following this, our final loss is computed as:

(5)

By combining focal reweighting with hierarchy-aware penalties, TFCL drives the model to excel on rare taxa and to maintain biologically valid predictions across all three taxonomic levels.

Experimental design and evaluation

We conducted two complementary analyses to isolate the contributions of our standardized lab image dataset and the hierarchy-aware fusion designs to overall model output quality. All runs shared the same general training setup, which used ResNet50v2 connected with the SE Block for feature enhancement, followed by a global-max-pooling layer to combine the three parallel SoftMax heads (family, genus and species). All training methods also used our TFCL and shared the same hyper-parameters. We report common classification metrics (accuracy, precision, recall) at each taxonomic level, plus a newly derived Taxonomic Alignment Score (TAS), which directly measures taxonomic consistency among the hierarchical levels. Specifically, while traditional metrics (precision, recall and accuracy) capture each head’s performance in an isolated fashion, TAS quantifies whether the three predictions form a biologically valid family-genus-species chain. Concretely, a test example () with predicted family (), genus () and species () is valid only if:

(6)

We define TAS as the fraction of samples satisfying both conditions, as detailed by:

(7)

Where the parent comparison is the indicator (1 if true, 0 otherwise). A perfect TAS = 1 means every genus falls within its predicted family and every species within its predicted genus. In turn, lower values indicate cross-level inconsistencies that violate biological relationships.

To specifically test the effect of our proprietary lab-based image archive, we trained two classification models using two size-matched datasets of either web only images (100% web-source images) or a 50:50 mixed composition (half of the images from the web, while the other half was from our laboratory-based images). We limited this experiment to species that had at least 100 web-sourced and 50 lab-based images. Thus, for the web-only run, all species had 100 images, while in the mixed set, we randomly replaced 50 web-based images with 50 lab-based images, while retaining the same number of images per species, total dataset sizes, and a balanced number of images for each run.

We next evaluated the impact of hierarchy-aware fusion, by comparing our four model variants under identical training conditions. All experiments used the full CryptoVision dataset, split into 70% training, 15% validation, and 15% test. Aside from the fusion strategy, all hyperparameters were identical (ResNet50v2 with SE block, input resolution (352x352 pixels), dropout = 0.3, batch size = 32), and all random seeds were set to be the same. Each model had two training stages, consisting of 1) 15 training epochs with all pretrained layers frozen, updating only the shared embedding and taxonomic heads, and 2) a fine-tuning stage in which the top 75 pretrained layers were unfrozen and trained with an additional ten epochs, allowing high-level features to adapt while preserving low-level stability. By keeping all other parameters constant, this protocol ensures that any differences in classification metrics and the TAS arise solely from the choice of STD, CONCAT, GATED, or ATT fusion.

To interpret the morphological features driving the model’s predictions and compare their alignment with expert-defined traits, we generated saliency maps—visual explanations that indicate which regions of an input image most influence the model’s output. These maps are computed by taking the gradient of a given model output with respect to each input pixel [52,53]. Thus, this method measures how each pixel affects the prediction’s confidence, with pixels with higher gradient magnitudes representing those that the model relies on most heavily to make its decision. To enhance interpretability via suppression of noise from irrelevant gradients, we applied guided backpropagation [54,55], which restricts the saliency visualization to positive contributory activations in the network. Saliency computation was implemented using the tf-keras-vis library [56], with a linearization step to ensure compatibility with our model and explicitly targeting the species output head.

Finally, we assessed whether model attention aligns with expert-defined diagnostic traits. To do so, we focused our saliency analysis on a subset of correctly classified test images from the dwarfgoby genus Eviota, the most abundant and taxonomically diverse group in our dataset. Specifically, Eviota now contains more than 130 species, which are unified by similar anatomical and morphological features but often differ in subtle aspects of their coloration or patterning [35,57]. Using diagnostic descriptions from [36]), which represents the most comprehensive and detailed dichotomous key for the genus, we compared the high-saliency regions identified by the model with externally visible, image-based diagnostic traits used in species identification, such as coloration patterns, pigment markings, body regions, and anatomically salient body features (e.g., head, eye, and caudal regions). The resulting saliency maps were normalized and overlaid on the original images to facilitate visual inspection and comparison.

High-quality image effect

We compared models trained on two datasets—Web-Only and Mixed (web plus high-quality lab images)—to evaluate how image quality influences overall model performance. The Mixed model consistently outperformed the Web Only model across all metrics (Fig 2). The most pronounced improvement was observed in both Recall and Accuracy (which increased by approximately 25%), indicating that the inclusion of high-quality images improves the model’s sensitivity to minority or difficult classes, reducing false negatives. Additionally, the TAS increased by 14%, reflecting better consistency across hierarchical predictions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Model performance comparison across data sources.

Radar plot comparing models trained on Web Only (purple) vs. Mixed (green) datasets. Vertices reflect the different metrics, all of which are constrained between 0 and 1. TAS = Taxonomic Alignment Score; F1 = Harmonic mean between precision and recall.

https://doi.org/10.1371/journal.pone.0349646.g002

Furthermore, using qualitative saliency visualizations, we examined how the model’s attention is spatially distributed across the fish body when trained on the two different datasets (Fig 3). The Web-Only saliency map overlay exhibited a perceptibly noisier and more dispersed signal, activating both relevant anatomical regions and background artifacts. In contrast, the Mixed model showed more spatially constrained and localized attention on the fish-body patterns (Fig 3a).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Training dataset influences model attention patterns and saliency distributions.

(A) Saliency map overlays for a cryptobenthic fish image of the dwarfgoby Eviota afelei. Photos (left to right) show the original image, the saliency map overlay from a model trained on only web-based images, and the overlay from the model trained with the lab-based images. Colors represent the saliency score. Original photograph taken by the authors as part of the laboratory-standard image dataset used in this study. (B) Distribution of saliency values from the maps displayed in the image.

https://doi.org/10.1371/journal.pone.0349646.g003

These visual trends were quantitatively supported by the saliency‐value distributions computed across the images sample (Fig 3b). The Mixed model exhibited a more concentrated profile, with a sharply peaked distribution (mean ≈ 0.099; standard deviation ≈ 0.100), indicating highly focused attention on a small subset of pixels. By contrast, the Web-Only model’s distribution was noticeably broader (mean ≈ 0.129; standard deviation ≈ 0.108), indicating a more diffuse and less discriminative allocation of saliency.

Hierarchical architecture comparison

To assess the value of our hierarchical model architecture, we first focused on the overall metrics between the standard and all three fusion schemes (based on concatenated family–genus–species predictions). While all models achieved very similar overall accuracy and recall, the baseline STD scored the highest precision, and both GATED and ATT substantially outperformed the other schemes in TAS (Table 1). Delta scores of the different metrics compared to the baseline (STD) highlighted a possible trade-off between improving TAS and decreasing the overall precision (Fig 4a). While marginally so, GATED and ATT also exceeded STD in accuracy, recall, and TAS, three out of four available metrics.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Mean metrics of different model architectures when run on the test set.

https://doi.org/10.1371/journal.pone.0349646.t001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Relative model performance and taxonomic consistency.

(A) Model performance relative to the standard (STD) baseline (Δ) across the core metrics—Accuracy, Precision, Recall—and the Taxonomic Assignment Score (TAS). (B) Fraction of genus-level misclassifications that still map to the correct family. (C) Fraction of species-level misclassifications that still map to the correct genus.

https://doi.org/10.1371/journal.pone.0349646.g004

We then examined the hierarchical awareness for links between family-genus and genus-species by asking how frequently the model assigns misclassifications at the lowest taxonomic level to the correct higher taxonomic rank (i.e., how often is an incorrectly classified species assigned to the right genus)? GATED improved this metric substantially, exhibiting an increase of up to 10% compared with other methods, demonstrating that GATED is far more likely to suggest a closely related species even when it misses the exact label. In turn, STD showed by far the lowest score, highlighting the utility of any fusion strategy for hierarchical classification (Fig 4b and 4c).

To determine whether the observed differences in overall accuracy between fusion strategies reflect systematic performance improvements, we conducted paired McNemar’s tests on the test set. No statistically significant differences were detected between the GATED architecture and any alternative fusion scheme (STD, CONCAT, ATT; all p ≥ 0.12), despite small numerical differences in accuracy (ΔAcc ranging from −3.7% to +0.5%). In all comparisons, disagreement counts were approximately symmetric, indicating similar error patterns rather than consistent gains in exact classification performance.

Finally, we assessed model calibration via reliability diagrams (Fig 5) alongside the expected calibration error (ECE). A well-calibrated model’s predicted confidence should align with its observed accuracy, and ECE quantifies the average gap between them. Although all four models exhibit generally low calibration errors, GATED delivered the strongest performance (ECE = 0.0108 (family), 0.0084 (genus), and 0.0107 (species), averaging ≈ 0.01). In contrast, ATT showed the highest errors (0.0124, 0.0133, and 0.0151 at family, genus, and species, respectively (average ≈ 0.0135)), suggesting a tendency toward overconfidence despite its strong TAS performance.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Calibration of model predictions across taxonomic tasks.

Reliability diagrams for the three taxonomic prediction tasks, showing the empirical fraction of positives (y-axis) versus the mean predicted probability (x-axis). Each colored curve corresponds to one model variant—GATED (dark blue), ATT (cyan), CONCAT (green) and STD (orange)—while the dashed diagonal line marks perfect calibration.

https://doi.org/10.1371/journal.pone.0349646.g005

Taken together, these analyses show that the GATED fusion strategy delivers the best overall balance, providing substantially higher cross-level consistency (TAS) and genus-to-species alignment, at only a minor precision cost, and with the best calibration, thus making it the preferred design for taxonomy-aware deep classification.

To investigate model performance beyond aggregate accuracy metrics, we conducted a per-species error analysis based on recall-derived error rates (1 − recall). Species were grouped into three categories according to their classification error: low (0–10%), moderate (10–25%), and high (>25%) error classes (S1–S3 Figs). The majority of species (58 of 113) fell into the low-error category, exhibiting consistently high recall with error rates below 10%. An additional 36 species showed moderate error rates, while 19 species exhibited substantially reduced performance, with error rates exceeding 25%.

Species in the high-error category showed persistent difficulty in being distinguished based on visual features alone, often coinciding with strong visual similarity among closely related taxa. To assess whether taxonomic complexity was associated with classification performance, we evaluated the relationship between species-level error and genus richness (number of species per genus; S4 Fig.). A weak positive trend was observed, with error rates tending to increase with genus richness; however, this relationship was modest (Pearson’s r = 0.228). Together, these results indicate that, while more diverse genera are more challenging to classify, per-species classification performance is not explained by genus richness alone.

Saliency maps & trait overlaps

Comparing and contrasting saliency maps with morphological features outlined in taxonomic keys revealed broad overlap, with model attention frequently highlighting even subtle diagnostic features. Below, we display three examples of Eviota species that exemplify this alignment.

A clear example of trait alignment is observed in the saliency map of the whitelined dwarfgoby Eviota albolineata (Fig 6a), where model attention is concentrated around the head, eye, and upper pectoral-fin base. These focal areas coincide precisely with the species’ main diagnostic characters as described by [36], which state: “Two unbroken stripes behind eye, upper across nape, lower across operculum” and “oblique wide stripe of melanophores across center of pectoral-fin base”. The model’s focused attention on these regions indicates that it has learned to prioritize the same visual traits that taxonomists use to distinguish this species.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Representative saliency-map outputs for selected species.

Saliency map for three fish specimens: (A) Eviota albolineata, (B) Eviota infulata, and (C) Eviota teresae. In each row, panels show (left) the original input image, (middle) the normalized saliency map (pixel importance scores from 0.0–1.0), and (right) the saliency heatmap overlaid on the original image. Original photographs taken by the authors as part of the laboratory-standard image dataset used in this study.

https://doi.org/10.1371/journal.pone.0349646.g006

In Eviota infulata (Fig 6b), the saliency map highlights the upper anterior body, just above the pectoral-fin base. This region corresponds precisely to the highly characteristic W-shaped black mark that is primarily used to identify the species. As described by [36], diagnostic characters include: “Irregular or W-shaped black mark on upper anterior body above and just posterior to base of pectoral fin”, “no distinct black spot at caudal-fin base”, and “7 postanal ventral-midline dark spots from subcutaneous body bars”. While the model’s saliency map shows minimal activation on the caudal fin—consistent with the absence of a defining mark—moderate saliency in other body areas suggests that broader morphological context is also considered. This balance between localized and distributed attention reinforces the model’s interpretability and its nuanced approach to species classification.

Finally, in Eviota teresae (Fig 6c), the saliency map is strongly focused on the abdomen and upper part of the eye, which align well with the species’ diagnostic traits. As described by [36], these include: “reddish blotches on abdomen taller than wide”, “dorsal part of eye reddish with small spots”, and “No prominent dark spots on body along base of dorsal fins”. The reduced saliency along the dorsal midline mirrors the absence of defining features in this body region, reinforcing the model’s sensitivity not only to prominent traits but also to their absence. This example highlights how even fine-scale pigment patterns are integrated into the GATED model’s classification strategy, illustrating its capacity to combine expert-level trait recognition with broader pattern synthesis.