4.1 Feature extraction

In the present study, the log-Mel spectrogram was chosen as the input to the proposed CNN model. Such a feature has been found to be effective in a wide variety of speech and generalized sound recognition tasks [26,27] including computational bioacoustics applications [13,28]. The log-Mel spectrogram was extracted using the following parameters: sample rate of 22050 Hz, 128 Mel-bands, window size equal to 1024 samples with a 50% overlap between subsequent windows [29]. The extraction process was performed on each audio sample, which was first split into 1-second duration with a sliding window of 50% overlap. Unlike handcrafted features, such a standardized feature extraction process minimizes the need for domain expertise for characterizing the available goat vocalizations. To further increase the diversity of the data set, various data augmentation techniques were applied as described next.

4.2 Data augmentation

CNNs, characterized by their substantial model capacity and large number of parameters, heavily rely on access to extensive training data. This dependency is crucial for effectively learning a non-linear function from input to output, facilitating generalization and achieving high classification accuracy on previously unseen data. Thus, we suggest employing audio data augmentation as a solution to address the challenge of limited data availability. Additionally, we investigate the way various augmentations impact the performance of the proposed CNN architecture. The methods used in this study include pitch shifting and time stretching, which have been successfully applied in the audio pattern recognition literature [30]. For each original audio sample, pitch shifting was applied with parameters ranging from -2 to 2 in semitones, which were expressed as [-2, 1, 0, 1, 2]. Additionally, time stretching was carried out at rates of 0.8, 1.0, and 1.2. In particular, we generated three time-stretched samples for every one-second audio sample, and for each time-stretched sample, we applied five different pitch shift values. As a result, a total of 15 log mel spectrograms were generated for each one-second audio sample. By increasing the number of samples, we obtained a larger and more diverse dataset used to train the proposed model. The augmentation parameters were selected ensuring that the identity of the goat vocalization class would not be altered up to the extent that a human listener can assess. It should be mentioned that model testing is carried out only on original data without considering their augmented versions [31].

4.3 The proposed convolutional architecture

CNNs have effectively addressed a wide variety of audio pattern recognition tasks [27,32], including the computational bioacoustics [13]. Thus, this work designs a CNN accommodating classification of goat vocalizations. Commencing with the conventional multilayer perceptron model, a CNN undergoes straightforward yet significant adjustments that empower it to efficiently handle audio signals [27]. Typically, a CNN is structured with several layers stacked together, hence creating a deep topology. The convolutional layers arrange the hidden units in a way that exposes local structures on the 2-dimensional plane, which are then utilized. This is achieved by linking each hidden unit to a small segment, known as a receptive field, of the input spectrogram. The units’ weights act as filters, also known as convolutional kernels, which are employed across the entire input plane to extract a feature map. The key assumption here is that locally extracted features are valuable in different regions of the input plane, leading to the application of the same weights throughout the spectrogram. This assumption is crucial as it not only reduces the number of trainable parameters but also ensures that the network is unaffected by translational shifts in the input data [33]. Last but not least, due to the potentially vast resulting dimensionality, max-pooling layers are introduced to retain only the maximum value within a specific region.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The proposed model architecture composed of stacked convolutional layers for classifying goat vocalizations.

https://doi.org/10.1371/journal.pone.0318543.g002

In addressing the present task, we were not able to directly employ existing CNN architectures as the extracted log-mel spectrogram images might not necessarily fit the required input shape [34]. Importantly, resizing such inputs would cause degradation of the time-frequency content and even add undesired information. To avoid such loss of information and maintain the original feature shapes, a novel CNN architecture is proposed based on extensive empirical experimentations and inspired by the VGGish model which has been shown to be effective in audio classification tasks [35].

The proposed CNN structure is illustrated in Fig 2 and is composed as follows: there are 4 blocks of convolutional and max pooling layers. The first three blocks each contain 2 convolutional layers with 64, 128, and 256 filters, respectively, followed by a max-pooling one. The final block contains 2 convolutional layers with 512 filters each, followed by a max-pooling layer. These blocks of convolutional layers increase the depth of the neural network and allow it to learn complex and abstract features. We utilized Rectified Linear Units, i.e. the activation function is f ( x ) = max ( 0 , x ) ; they have been particularly popular since they provide a) quicker gradient propagation compared to traditional units (such as logistic sigmoid, hyperbolic tangent, etc.), b) biological plausibility, and c) an activation form distinguished by high sparsity [36]. After the last max-pooling layer, the output is flattened into a 1D vector and passed through two fully connected layers each one encompassing 4096 units, followed by a dropout layer to remove irrelevant relationships and prevent overfitting. Finally, the output layer consists of 8 units with softmax activation, which is suitable for multi-class classification.

4.4 Model learning

The proposed CNN model was trained on the augmented dataset comprising 4161 recordings of goat vocalizations. The primary evaluation metric utilized was the classification accuracy computed on the original test set. For model compilation, Adam optimizer was chosen with a learning rate of 0.0001. A batch size of 12 was selected, and the model was trained for 14 epochs adopting early-stopping. Adam was chosen due to its efficiency in optimizing complex models [37]. The low learning rate value was chosen to ensure stable model convergence, while the batch size was determined based on available memory. To address class imbalance, we adopted a stratified 5-fold cross-validation scheme. As regards to the overfitting phenomenon, we monitored training and validation accuracy as well as the respective losses, where we confirmed a steady mutual increase indicating improved performance after each iteration.

5.1 The contrasted approaches

To the best of our knowledge, this is the first time that automatic classification of goat vocalizations is addressed, hence there are no existing approaches to compare to. As such, the proposed CNN was compared to two diverse and popular deep architectures [38], namely MobileNet [34] and EfficientNet [39], which are explained next. The employed structures are depicted in Fig 4.

Learning MobileNet and EfficientNet.

Both MobileNet and EfficientNet models were trained using the available augmented dataset, while the classification accuracy over the original test set served as the primary evaluation metric. During the training phase, we employed a batch size equal to 64 and 100 epochs. Following early experimentations, the learning rate was optimized to 0.0001.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Class-wise recordings’ distribution of the VOCAPRA dataset.

https://doi.org/10.1371/journal.pone.0318543.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Illustration of the structures characterizing the contrasted approaches, namely (a) EfficientNetB0 and (b) MobileNetV2.

https://doi.org/10.1371/journal.pone.0318543.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Confusion matrix assessing the performance of the proposed and contrasted approaches (presentation format CustomCNN/EfficientNetB0/MobileNet).

https://doi.org/10.1371/journal.pone.0318543.g005

5.2 Results

Fig 5 presents the confusion matrix assessing the performance of the proposed and contrasted approaches. The matrix includes average classification rates over the 5 folds with respect to each class present in the test set.

There, we observe that the proposed approach surpasses the contrasted ones in classifying all considered classes of goat vocalizations. The average rates as regards to the considered models are 95.8%, 75%, 68.3% for the proposed, EfficientNet, and MobileNet based approaches respectively. Importantly, all classes are recognized with a rate higher or equal to 91%, demonstrating promising performance. The class recognized with the highest rate is the isolation one which presents a quite characteristic pattern as we see in Fig 1. It is followed by heat, visitors, injury-death, and reunion.

In addition, the rates reveal that classes separation and parturition consistently demonstrated relatively lower rates compared to the rest of the classes. This observation could be attributed to the smaller number of available audio samples in these classes, as well as to the acoustic similarity of those vocalizations with other classes to the extent that the human listener can judge. For instance, presence of unknown visitors was often misclassified as class feed distribution, possibly indicating that the goats experienced similar emotions or stimuli during these situations.

Furthermore, examination of the confusion matrix reveals that neither MobileNet nor EfficientNet models attained performance levels comparable with the proposed CNN model. It is worth noting that both approaches presented a tendency towards overfitting (see the implementation page).

During the conducted experiments, it became apparent that transfer learning, as applied through MobileNet and EfficientNet architectures, did not yield outcomes matching the predictive accuracy demonstrated by the proposed CNN model. Despite their pretrained weights, such models failed to leverage the distinctive structures present in the available goat vocalization dataset when expressed via log Mel spectrograms.

These findings underscore the subtle interaction between transfer learning and the intricacies of specific datasets and feature representations. It is evident that the MobileNet and EfficientNet models, while benefiting from prior knowledge encapsulated in pretrained weights, encountered limitations in adaptability when confronted with the distinct characteristics of goat vocalizations.

It is interesting to note that, similarly to the proposed CNN, the class which was poorly recognised is mother-kid separation and the one associated with the highest rates is isolation. This may be attributed to the distinctness of the acoustic patterns exhibited by such vocalizations in the time-frequency plane. Importantly, the implementation of the presented experimental pipeline is publicly available ensuring full reproducibility of the achieved results at https://colab.research.google.com/drive/1Xu-U3HY_FHgI11Ni35I69DdlP7jFFOLD#scrollTo=AMb1H9dgFsNp.

We argue that the rates achieved by the proposed model are more than satisfactory and highlight the efficacy of the implemented architecture. The proposed model may be efficiently employed to serve precision livestock farming applications [23]. However, applying the existing model to unseen locations/farms/scenarios may result in decreased performance depending on the similarity of the overall soundscape to the original training and testing data. As such, model finetuning and retraining using additional data representing the unknown setting should be carried out.

6.1 Concept relevance propagation

To gain insights into the predictive mechanisms of the proposed model regarding the classification of spectrograms, we employed the Concept Relevance Propagation (CRP) technique [24]. Built upon the principles of Layer-wise Relevance Propagation [42], CRP introduces the concept of conditions, extending the capability of back-propagation. Interestingly, CRP allows for the identification of specific channels and layers that significantly influence the final prediction.

In the subsequent sections, we delve into the application of CRP to analyze the predictions generated by the proposed model, thus enhancing interpretability and gaining meaningful insights into the model’s decision-making process.

The initial exploration of CPR was centered on uncovering pivotal features within the available goat vocalizations. Similarly to the way features like fur or dog eyes can be readily identified in a dataset of dog images, CRP allows us to discern the learned features of the model, providing a quantitative score that indicates the influence of each feature on the final prediction. However, when applied to spectrogram images, the identified features are not as immediately interpretable as those found in image data. Indeed, when analyzing spectrograms representing audio data, the identified features are more abstract, representing specific regions of the spectrogram, which may be characterized by a certain shape or more generally low, mid, or high-frequency bands in the spectrum.

6.2 Sample-wise comparative explainability analysis

We applied CRP in a uniform way to elucidate the predictions of the two best-performing models, i.e. the proposed CNN and EfficientNetB0. Initially, we employed CRP to gain a broad, high-level understanding of the features learned by each model. More specifically, we examined the significance of different features such as regions, shapes, patterns, and areas of varying frequency within the spectrogram. For each layer, we systematically analyzed all channels to determine their focus on distinct features within the spectrogram. Subsequently, we identified the top six channels in each layer that exert the most significant influence on the final prediction, and visualized their impact through suitable heatmaps. Figs 6 and 7 illustrate the relevance maps generated when the proposed model and EfficientNetB0 process a given input.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. CRP for a single spectrogram when processed by the proposed CNN. (a) Heatmaps displaying the top six most relevant channels for the first fully connected layer of the proposed CNN. (b) Heatmaps illustrating the top six most relevant channels for the eighth convolutional layer in the proposed CNN.

https://doi.org/10.1371/journal.pone.0318543.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. CRP for a single spectrogram when processed by EfficientNetB0. (a) Heatmaps showing the top six most relevant channels for the second fully connected layer of the EfficientNetB0 model. (b) Heatmaps representing the top six most relevant channels for the eighth convolutional layer in the EfficientNetB0 model.

https://doi.org/10.1371/journal.pone.0318543.g007

This preliminary investigation facilitated the understanding of the way the proposed CNN and EfficientNet models carry our predictions. As we see in Fig 6, the predictions of the proposed CNN are made by dividing the spectrogram in specific regions, such as low, mid, or high-frequency bands and considering them with diverse weights. On the contrary, the division of the spectrogram into high, mid, and low regions is not applicable to the prediction strategy of the EfficientNet model; instead, the CRP analysis suggests that the EfficientNet model exploits almost the entirety of the spectrogram using a grid approach while excluding most of the content associate with the low-frequency regions. This concise, yet comprehensive exploration has guided the subsequent phase of our investigation aiming at a class-level interpretation analysis.

6.3 Class-wise relevance heatmaps

The specific interpretation phase aims at offering in-depth and comprehensible insights into the way the considered models predict each goat vocalization class. Thus, we implemented the following procedure:

• selection of n goat vocalizations to represent each each class as the most central ones following a k-means clustering scheme using the Euclidean distance.
• application of CRP analysis on each log-Mel spectrogram extracted from the selected goat vocalizations. This involved computing the relevance heatmaps of the top g most relevant channels impacting the final prediction with respect to each layer.
• computation of the class-wise average heatmap by aggregating the corresponding heatmaps.

It should be noted that in the case of the proposed CNN model, the average heatmap for each class was computed using a total of 1600 relevance heatmaps. The formula used to determine the total number of relevance heatmaps used to compute the average heatmap for a single class is the following:

(1)

where  | D |  denotes the cardinality of the class dictionary (see Sect 2), g represents the number of relevance heatmaps per layer, and n the number of spectrograms per class. In our specific case,  | D | = 8, g = 20 and n = 100.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. The class-wise heatmaps with respect to the proposed model (top row), EfficientNetB0 (middle row), and MobileNet (bottom row).

https://doi.org/10.1371/journal.pone.0318543.g008

6.4 Class-wise comparative explainability analysis

The resulting class-wise average heatmaps characterizing the operation of the proposed (top row), EfficientNetB0 (middle row) and MobileNet (bottom row) models are depicted in Fig 8. Importantly, there is a notable disparity in the focus areas across the various spectrogram regions, i.e.

• Custom CNN (top row): Here, the heatmaps indicate that the custom CNN learnt to place emphasis upon a specific region with respect to each class of goat vocalizations. Interestingly, the model attends to distinct frequency bands across the spectrograms. The activations are more localized, suggesting that the custom CNN has learned to focus on precise features within the spectrograms for each category. For instance: a) in “Heat" and “Feed Distribution" classes, the model focuses on the mid and lower frequency ranges. b) in “Injury or death" classes, it highlights certain diagonal patterns in the mid-range, capturing distinct features. c) “mother kid reunion" and “Unknown Visitors" show attention to broader regions, with concentration in specific time-frequency regions. Such a focused and specialized attention contributes the almost excellent performance achieved by the proposed CNN model.
• EfficientNetB0 (middle row): The respective heatmaps display broader attention regions compared to the proposed CNN. Instead of focusing on narrow bands or specific spectrogram parts, the model spreads its focus across a larger portion of the spectrogram. For example: a) in the “Heat" and “Feed Distribution" classes, the attention is more diffused, indicating that the model might struggle to pinpoint critical features as effectively as the custom CNN, b) “injury or death" and “Social Isolation" still show some distinct patterns, but the activations are less precise the ones observed by the maps characterizing the operation of the proposed model. Such a broad focus could explain why EfficientNet might not perform as well, since it captures less specific features, which are hard to detect in novel spectrograms.
• MobileNetV2 (bottom row): the heatmaps of MobileNet also show relatively diffused attention, though slightly more structured than EfficientNet. MobileNet focuses on larger frequency ranges in each audio class. For example: a) in the “Heat" and “Feed Distribution" classes, MobileNet seems to activate larger areas in the lower and mid frequencies, but the patterns are not as crisp as in the proposed CNN. b) in “Injury or death", the model struggles to focus on the distinct diagonal pattern that the proposed CNN captures well, suggesting less precise feature extraction. Similar to EfficientNet, MobileNet’s broader focus likely leads to poorer performance in classifying these specific vocalizations compared to the custom CNN.

6.5 Comments on model interpretability

One overarching consideration stems from the fact that CNNs are primarily designed to learn from images, rather than directly “listening" to the sounds represented by spectrograms. As evidenced by the case of the EfficientNetB0 model, while relevance maps may visually highlight certain regions, they may not necessarily correspond to the expected constituent frequencies crucial for sound classification. In several instances, these highlighted regions might have contributed to distinguishing between classes, but ultimately may not represent meaningful features from the animal scientist’s point of view. Such discrepancy could be indicative of inadequate data or lack of generalization ability. Despite being trained on a vast image dataset, the need to fine-tune precomputed weights for spectrogram classification underscores the challenges in transferring knowledge from general image datasets to spectrogram-specific tasks. Spectrograms, being inherently similar across different classes, may necessitate a specialized model for accurate classification. The clarity of the average maps produced by the proposed CNN model suggests a more robust learning of relevant features, which likely contributes to its superior generalization performance.

Another important consideration is the potential utility of model prediction inspection in identifying corrupted data. For instance, by subtly embedding class labels into spectrogram images, one could assess whether the model has indeed learned to classify based on the visual cues provided by the labels. This approach enables the rapid identification of any biases or anomalies in the dataset that may affect model performance. However, it remains challenging to determine whether such analyses can effectively discern the optimal training data, as the criteria for optimal data may vary depending on the specific task and dataset characteristics. Nonetheless, the ability to detect corrupted data through model prediction inspection underscores its value in ensuring the integrity and reliability of the dataset used for training and evaluation.