ResNet-50.

ResNet is a principal deep learning architecture, presenting variations with 50, 101, and 152 layers [42], and its learning mechanism revolves around residual representation, where the prior layer’s output operates as the input to the next layer without alterations.

The architecture includes two key types of blocks: Convolutional and Identity. These blocks, merged, construct the foundation for creating ResNet. Residual entities, categorized as pooling, convolution, and batch normalization layers, form the building blocks, as illustrated in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Graphical representation of ResNet architecture.

https://doi.org/10.1371/journal.pone.0322171.g004

Mathematically, each unit can be defined using Eqs 1, 2, and 3. Here, x_n is the input, x_n + 1 is the output of the n-th unit, and f is the residual function of the n-th unit.

(1)

(2)

(3)

Here, h(x_n) describes the identity mapping of the activation function. Notably, ResNet uses back-propagation to address the vanishing gradient issue since its residual representation contains short-cut networks. As seen in Figs 5 and 6, these short-cut networks use skip connections, which are similar to basic convolutional layers, to add input values to the output after specific weight layers.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. ResNet convolutional block architecture.

https://doi.org/10.1371/journal.pone.0322171.g005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. ResNet identity block: A foundational unit in ResNet.

https://doi.org/10.1371/journal.pone.0322171.g006

ResNet-50, utilizing 50 parameterized layers with recurrent connections as residual blocks, uses transfer learning. The architecture culminates in a personalized softmax layer for detecting mosquitoes on human skin, enhanced with an L2 regularizer. Noteworthy elements of ResNet-50 include:

• Fully connected layers are previous to the softmax output layer,
• A 77 average pooling layer with stride 7,
• 16 residual building blocks,
• A 77 convolution layer, and
• A 33 max pooling layer with stride 2.

DenseNet-121.

DenseNet, an architecture in Convolutional Neural Networks (CNNs), is created to streamline connectivity patterns among layers by allowing direct connections. To overcome the restriction on the maximum information flow, this enables direct connections between each layer and the other layers. In comparison with traditional CNN, there are fewer feature maps in DenseNet than in regular CNN. DenseNet uses feature reuse to maximize the potential of its layers, and it also fixes the problems caused by the gradients. Using a classic feed-forward architecture, the DenseNet associates each layer’s output with the layer before it through a composite operation that combines activation functions, batch normalization, and layer pooling.

The foundational equation governing DenseNet is is given in Eq 4:

(4)

In the ResNet architecture, this equation is extended by containing the skip connection, reformulating as in Eq 5.

(5)

In comparison to ResNet, which expands the equation with skip connections, DenseNet opts for a concatenation approach without the summation of outgoing and incoming feature maps. Therefore, the equation reshapes again into Eq 6.

(6)

DenseNet employs Dense Blocks as fundamental building blocks within the model structure. These Dense Blocks consist of layers directed as transition layers, which handle down-sampling by including batch normalization. Despite changes in the number of filters, the feature map dimensions stay constant throughout the model. However, the dimensions of each channel experience an expansion as a result of the concatenation of feature maps from every layer.

Every time H_n is employed to produce k feature maps in the generalized n-th layer as defined in Eq 7.

(7)

The expansion rate of hyperparameter k is regulated by the level of information added to the network at each layer. DenseNet demonstrates a distinctive architecture where input volumes and the output of two operations become intricately linked. This connection concerns not only the additive infusion of information into the shared knowledge of the network but also the concatenation of k feature maps, thereby constructing an outstanding interweaving of information flow within the network.

VGG-16.

The VGG-16 convolutional neural network model is a highly prosperous architecture in computer vision. Well-known for its simplicity and versatility, VGG-16 has become a widely embraced model due to its precise design principles, giving valuable insights for the exploration of deep neural networks.

To maintain spatial resolution post-convolution, the model utilizes 33 convolutional filters with both a stride and padding set to 1. Feature extraction concerns the integration of certain convolutional layers with a subsequent 22 max-pooling layer, directing with a stride of 2 over a pixel window of 22. Individual passage through a pooling layer results in a halving of the input’s spatial resolution.

There are three fully connected levels at the end of the architecture. Both of the initial layers have 4,096 channels each, while the number of channels in the third layer varies depending on which particular dataset is being used. The last layer is a softmax layer responsible for classification predictions. ReLU assists in the overall efficacy of the model by serving as the activation function for every layer in the VGG-16 convolutional layer architecture. The deep learning model VGG-16 is constructed of 13 convolution layers, three fully connected layers, and five max-pooling layers, as demonstrated in Fig 7.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Graphical representation of VGG-16 architecture.

https://doi.org/10.1371/journal.pone.0322171.g007

InceptionV3.

InceptionV3 desires to scale up networks by dealing with the computation as efficiently as possible. It utilizes appropriately factorized convolutions, aggressive regularization, and an advancement in Inception architecture. Inception architecture remains different from conventional approaches to expand network accuracy. The model is carefully created based on four key principles: representational bottlenecks in the network should be ignored; high-dimensional representations are easy to process; low-dimensional embeddings can be utilized for spatial aggregation with no loss in a representative capacity, and the network’s width and depth can be extended concurrently to reach the best performance for a provided amount of computation. Inception architecture stays different from traditional methods of improving network accuracy by following these guidelines.

Table 2 and Fig 8 represent the general structure of the InceptionV3 network architecture.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Inception modules: (a) two 33 convolutions are employed in place of each 55 convolution, (b) factorization of the nn convolutions, and (c) modules with expanded filter bank outputs.

https://doi.org/10.1371/journal.pone.0322171.g008

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. General structure of InceptionV3 network architecture.

https://doi.org/10.1371/journal.pone.0322171.t002

Proposed novel ensemble learning strategy.

The four pre-trained base learner models are fine-tuned using the training data, and the resultant models are saved (as .h5 files) as shown in Fig 9.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Fine tuning and saving of four base models.

https://doi.org/10.1371/journal.pone.0322171.g009

Here, our goal is to create ensembles using those four base models. In order to generate more variety, instead of creating just one ensemble containing all four base models, we opt to create four ensemble models, each containing three different base models (see the subsequent section for more details).

Let us consider one particular ensemble model (out of four ensembles). Suppose this ensemble contains three base learner models (namely, Base Learner 1, Base Learner 2, and Base Learner 3). Those saved models (.h5 files) from those three models are loaded into memory in order to generate the ensemble model. The input data of shape (299, 299, 3) is defined for a single input layer, which is then transmitted through each loaded model to produce its final outputs. A single composite output is produced by averaging these distinct outputs. The ensemble model’s final classification output is directly derived from the averaged output. By combining the predictions of the three models, this technique makes use of their strengths and may outperform individual models. The ensemble model as a whole (containing the three base models) is trained using the same training data. The weights of the three base models are further updated during the ensemble training process. The block diagram is shown in Fig 10.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Block diagram of proposed ensemble model.

https://doi.org/10.1371/journal.pone.0322171.g010

Our approach is unique because it builds an ensemble model based on transfer learning specifically designed to handle parameter efficiency and computational complexity. By using pre-trained models with significant amounts of frozen layers, our method dramatically minimizes these complexities compared to existing deep learning models, which frequently exhibit high parameter counts, resulting in significant processing demands during ensemble combinations. We optimize and fine-tune pre-trained models to meet our particular mosquito species classification job by utilizing transfer learning approaches. The development of this customized ensemble model framework based on transfer learning is the main innovation of our work.

This methodology significantly enhances the efficiency and efficacy of ensemble learning in complex classification tasks such as mosquito species identification while also optimizing computational resources. To our knowledge, the use of such an ensemble model idea based on transfer learning for the classification of mosquito species has not been explored before, making it a unique and promising contribution to the field of machine learning in disease vector management and biodiversity conservation.

Our strategy is distinct from other ensemble strategies involving the pre-trained deep learning models such as majority voting [45], average voting [46], weighted voting [47], or combined ensemble outputs being fed into a dense (fully-connected) neural network [48,49]. In all those ensemble models, the base learners (pre-trained models) are frozen and no longer updated once they have been trained with the training data. In our case, another round of end-to-end training of the whole ensemble network was carried out, and the participating base learner models are still updated accordingly.

Building ensemble models.

As discussed above, the basic idea of our ensemble learning is an average ensemble model. Our learning procedure involves applying various base models to the same set of data. We have utilized transfer learning as our base model, and for using the different transfer learning models, the structure of the models was different. We have described the base learners’ architecture above. This technique is only possible on deep learning ensemble models where traditional ensemble models cannot perform this learning strategy [43].

In the pursuit of constructing an ensemble model for our research, we sought to explore the various combinations of four pre-trained models. In order to improve generalization and predictive performance, deep learning ensemble methods combine several deep learning models. To improve the performance in deep learning tasks, pre-trained CNN models like VGG-16, InceptionV3, DenseNet-121, and ResNet-50 are assembled by combining their respective strengths. Every pre-trained model excels at capturing various facets of intricate data patterns through their specialized features and hierarchical representations. We developed a more thorough and reliable understanding of the underlying features in the dataset by combining these models. The variety of architectures improves generalization, reduces overfitting, and allows for flexibility with different kinds of input. Through the use of pre-trained models’ transfer learning capabilities, this ensemble approach facilitates efficient knowledge transfer from the massive datasets used in the models’ initial training. We selected three base models at a time to form ensembles with diverse architectures. The calculation of the number of unique combinations was performed using Eq 8.

(8)

Here, n represents the total number of base models, and r denotes the number of models to be selected for each ensemble. For our case, n = 4 (the number of available base models) and r = 3 (the desired ensemble size). Substituting these values into the formula, we obtained:

Thus, selecting three models out of the four aforementioned architectures allows for four unique combinations. These combinations represent distinct ensemble configurations with diverse model architectures, contributing to our research’s comprehensive exploration of the model ensemble space.

Here, the ensemble model’s final prediction is obtained by averaging the base learners’ results. Because the DL architectures have low bias and high variance, generalization performance can be improved by simply averaging the ensemble models, which reduces variance among the models. Let D represent the pre-trained model set, consisting of {“InceptionV3”, “ResNet-50”, “DenseNet-121”, “VGG-16”}. The images of six different species of mosquitoes from the dataset (X_i and Y_i), where X_i is the total number of images and Y_i is the correspondent labels of images Y = {“Aedes albopictus landing”, “Aedes albopictus smashed”, “Aedes aegypti landing”, “Aedes aegypti smashed”, “Culex quinquefasciatus landing”, “Culex quinquefasciatus smashed”}.

The size of each image is (299299). Mini batches are created by dividing the training set by n, which lowers the empirical loss and improves the accuracy of the DL models. The base learners’ outputs directly or the classes’ anticipated probabilities using the softmax function are used to average the base learners as shown in Eq 9.

(9)

Here, P_ji is the probability outcome of the i-th unit on the j-th base learner, O_ji is the output of the i-th unit of the j-th base learner, and k is the number of classes. In our case, k is 6. As indicated in [35], unweighted averaging makes sense when the base learners’ performance is comparable. In our research, the base learners’ performance is indeed comparable. The changes in the training samples did not lead to significant changes in accuracy in our base learners, which was also a reason to apply unweighted ensemble techniques in our research. This technique is suitable for low model variance. However, because it is influenced by the performance of the overconfident yet weak learners, naive unweighted averaging may lead to sub-optimal performance when the ensemble consists of heterogeneous base learners [50].

The overview of the proposed ensemble model is shown in Fig 11.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Overview of proposed ensemble model.

(The mosquito images are reproduced with permission by the original images’ owners [16]).

https://doi.org/10.1371/journal.pone.0322171.g011

Algorithm 2 Building mosquito species classification ensemble models.

1: Resize images in dataset (to )

2: Split dataset into training data and validation data

3: Preprocess training data (using Algorithm 1)

4:

5: Create and fine-tune pre-trained base learner models

6: D = {“InceptionV3”, “ResNet-50”, “DenseNet-121”, “VGG-16” }

7: for i = 1 to do

8:   initialize(D[i])

9:   fine-tune base learner model D[i] on preprocessed training data

10:   save model D[i] (as .h5 file)

11: end for

12:

13: Prepare ensemble models through combinations of base learners

14: Calculate number of unique combinations using C(n,r) formula

15: number of base learner models (4)

16: desired ensemble size (3)

17:

18:

19:

20:

21:

22:

23: Create, train, and evaluate ensemble models

24: for i = 1 to do

25:   Load saved models

26:   for j = 1 to r do

27:    load base model EM[i][j] (from original .h5 file)

28:   end for

29:   Create ensemble model

30:   initialize input layer:

31:   for j = 1 to r do

32:    output of base learner EM[i][j] on

33:   end for

34:   Perform unweighted averaging

35:   average()

36:   Prepare ensemble model

37:   Model(, )

38:   Configure ensemble model

39:   Configure (optimizer: Adam or Nadam, learning

   rate: 0.001, loss function: categorical cross-entropy)

40:   Train ensemble model

41:   Train on preprocessed training data

42:   Evaluate ensemble model

43:   Evaluate on validation data

44:   Print validation accuracy of

45: end for

46:

47: Select the best ensemble model

48: Identify the ensemble model with the highest validation accuracy

Four ensemble models are created using Algorithm 2 described above. They are:

1. Ensemble Model 1 (EM1): InceptionV3 + ResNet-50 + DenseNet-121
2. Ensemble Model 2 (EM2): InceptionV3 + ResNet-50 + VGG-16
3. Ensemble Model 3 (EM3): ResNet-50 + DenseNet-121 + VGG-16
4. Ensemble Model 4 (EM4): InceptionV3 + DenseNet-121 + VGG-16

Hyperparameters.

The hyperparameters of the proposed method represent the pre-trained models’ estimated computational behavior. In our research, the hyperparameter values were empirically determined. The model was able to achieve an improvement in accuracy at a learning rate of 0.001 when compared to 0.0001, which conversely converged slowly and exhibited poor performance. The optimizer had major implications for performance enhancement. Although Adam and Nadam optimizers exhibited competitive results among themselves, Adam always outperformed Nadam for all metrics. This can be accounted for by Adam’s adaptive learning rate mechanism that smoothly mediates between momentum and adaptive gradient estimation, resulting in faster convergence and better generalization.

The proposed architecture uses Softmax activation functions for the output layer and Rectified linear unit (ReLU) for the hidden layer activation, as stated in Eqs 10 and 11.

(10)

(11)

Optimizers: We used two optimizers, namely, (i) Adam [51] and (ii) Nadam [52] optimizers, in building our ensemble models.

1. Adam [51] is a method for first-order gradient-based optimization of stochastic objective functions. The algorithm is based on adaptive estimates of lower-order moments. It is straightforward to implement, is computationally efficient, and has little memory requirements. It is one of the most commonly used optimizers in deep learning [53].
2. Nadam [52] utilizes an advanced momentum algorithm named Nesterov’s accelerated gradient (NAG) instead of the regular momentum used in Adam. The authors presented preliminary evidence suggesting that making this substitution improved the speed of convergence and the quality of the learned models.

Epochs: When training a neural network, an epoch is a single run through the whole dataset. Mathematically, one epoch in our dataset with N samples (where N is the number of our training samples) entails using each of the N samples once to update the model’s parameters. Usually, this entails two types of propagation: backward propagation (where gradients are computed and the loss is calculated to update the model’s parameters) and forward propagation (where input data is passed through the network to generate predictions). The model has seen the entire dataset once after finishing one epoch. Several epochs are typically carried out during training to enhance the model’s performance by iteratively updating its parameters using the complete dataset.

Loss function: In our multi-class classification vector mosquito and species classification, we make use of the categorical cross-entropy loss function. For each sample i, it calculates the difference between the predicted class probabilities and the true class distribution y_i. Eq 12 gives the formula for categorical cross-entropy loss.

(12)

where:

• n denotes the number of samples,
• C denotes the number of classes,
• y_ij is the indicator function that equals 1 in the event that sample i truly belongs to class j and 0 otherwise,
• represents the expected probability that sample i belongs to class j.

The experimental hyperparameters details are shown in Table 3.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Hyperparameters of ensemble models.

https://doi.org/10.1371/journal.pone.0322171.t003

Trainable and non-trainable parameters.

Trainable and non-trainable parameters make up the parameter spaces for our pre-trained models (base models of the ensemble). The number of parameters (a.k.a. weights) that the backpropagation algorithm can modify is indicated by the term “trainable parameters.” The non-trainable parameters, on the other hand, are static values that are not changed during training. The numbers of parameters in our base pre-trained CNN models are displayed in Table 4.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Total parameters, trainable parameters, and non-trainable parameters of the pre-trained CNN models.

https://doi.org/10.1371/journal.pone.0322171.t004

Evaluation metrics.

In order to thoroughly evaluate the effectiveness of our model, we utilized various statistical and graphical techniques.

Firstly, we plot a confusion matrix (explained in Fig 12) for each classification exercise. We evaluated our pre-trained models and the ensemble models using the evaluation metrics of accuracy, precision, recall, and F1-score, which are all derivable from the confusion matrix. Moreover, we assessed our ensemble model using additional criteria such as Cohen’s Kappa () indicator, Matthews correlation coefficient (MCC), receiver operating characteristic (ROC) curve and its area under the curve (AUC), and precision-recall curve.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 12. An example of confusion matrix with 6 classes.

If the current class i is 2, blue represents TP, green TN, yellow FP, and orange FN.

https://doi.org/10.1371/journal.pone.0322171.g012

This multifaceted approach to evaluation allowed us to gain a more comprehensive understanding of our model’s strengths and weaknesses. By analyzing both statistical and graphical data, we identified areas in which our model excelled and areas in which further improvement was necessary. Furthermore, our use of additional evaluation criteria, such as Kappa, MCC, and AUC, allowed us to assess our model’s performance in a more nuanced and sophisticated manner. This, in turn, allowed us to draw more meaningful conclusions about our model’s effectiveness in real-world scenarios.

The precision of positive predictions is measured. It is the proportion of accurate positive predictions to all of the model’s positive predictions. Recall gauges the capacity to record every positive instance; it is also referred to as sensitivity or true positive rate. It is the proportion of all actual positive occurrences to all true positive predictions. The harmonic mean of recall and precision is known as the F1-score; the other name for it is the F-score or F-measure. The model’s overall correctness is measured by accuracy. It is the proportion of cases that were accurately predicted for all instances. They are calculated as follows:

When considering class i (, where C = 6 is the number of classes, we regard i as the positive class and the remaining classes as a negative class.

Here,

• TP = no. of true positives (no. of instances belonging to class i correctly predicted as class i),
• FP = no. of false positives (no. of instances belonging to other classes incorrectly predicted as class i),
• TN = no. of true negatives (no. of instances belonging to other classes correctly predicted as other classes), and
• FN = no. of false negatives (no. of instances belonging to class i incorrectly predicted as other classes).

The concepts of TP, FP, TN, and FN are illustrated in Fig 12.

For a multi-class classification task with C number of classes (C = 6 in our case), C sets of metrics for precision, recall, F1-score, and accuracy are generated, and the average metric values are taken.

For classification tasks, Cohen’s Kappa is a statistical indicator of inter-rater agreement. It takes into consideration the chance that an agreement could occur. The value of Kappa is a number between -1 and 1, where -1 denotes perfect disagreement, 0 represents an agreement that is equal to chance, and 1 indicates perfect agreement.

To handle multiple classes, Cohen’s Kappa for multi-class classification can be modified from the binary case. When dealing with multiple classes, the Kappa statistic is usually calculated by taking into account the degree of agreement and disagreement that exceeds chance in each class. The multi-class Cohen’s Kappa formula is as follows:

where:

and

where:

• is the number of instances where the predicted class and the true class are both i.
• is the sum of row i in the confusion matrix.
• is the sum of column i in the confusion matrix.
• is the total number of instances.

To handle multiple classes, MCC for multi-class classification can be modified from the binary case. Usually, in a multi-class scenario, the confusion matrix for each class is taken into account when computing the MCC. The multi-class MCC formula is as follows:

where:

• C is the number of classes.
• TP_i: True positives for class i.
• TN_i: True negatives for class i.
• FP_i: False positives for class i.
• FN_i: False negatives for class i.

InceptionV3.

The pre-trained model InceptionV3 consists of 48 layers, including convolutional layers, pooling layers, fully connected layers, and auxiliary layers. It has achieved 90.30% training accuracy and 84% validation accuracy, with the Nadam optimizer having 21,817,127 Total parameters, 1,949,793 trainable parameters, and 19,867,424 non-trainable parameters. The difference between training and validation accuracy is 6.3%, which indicates the model might be overfitted.

Fig 13a has described the InceptionV3 five-epoch training process to represent a full cycle through the dataset. In the first epoch, the model achieved a validation accuracy of 56.00% and 1.0716 validation loss with a training accuracy of 34.37% and a training loss of 2.1857. Then, both accuracies improved significantly in the second epoch, with the validation accuracy reaching 70.00% and the training accuracy at 66.74%. The loss reached 0.8787 for training and 0.7000 for validation loss. While the training accuracy increased to 78.07% and loss decreased to 0.5971 in the third epoch, the validation accuracy remained at 70.00%, though the validation loss has been slightly increased to 0.7875. Then, after the fourth epoch, the validation accuracy improved to 77.33%, and the training accuracy rose to 85.04%, with the loss reducing further to 0.4291 and 0.5574 for both training and validation losses. In the first two epochs, the model exhibited underfitting, suggesting that it hasn’t yet learned the complexities of the data. On the other hand, overfitting began to emerge in the third epoch because the training accuracy surpassed the validation accuracy. Finally, the model achieved its best performance in the fifth epoch with a training accuracy of 90.30%, a validation accuracy of 84.00%, and the lowest training loss of 0.3156, where the validation loss is 0.4673. The model was saved after each epoch that saw an improvement in validation accuracy. The training process was computationally intensive, with each epoch taking between 405 to 449 seconds to complete.

ResNet-50.

ResNet-50 has a total of 23,602,055 parameters, 1,443 trainable parameters and 23,587,712 non-trainable parameters with a depth of 50 layers. The training accuracy obtained by ResNet-50 is 78.37%, and the validation accuracy is 85.33%. In this case, an underfitting issue was presented.

Fig 13b shows RenseNet-50 performance in every epoch. In epoch 1, the model leaped from an undefined initial validation accuracy of 70.67% with 1.0227 validation loss. This significant upturn is coupled with a modest training accuracy of 34.22% and a loss of 1.6998. This epoch indicates underfitting as the model has a low training accuracy compared to a higher validation accuracy. Epoch 2 climbed to 74.00% in validation accuracy with training accuracy, which has raised to 59.04%, and losses have been reduced to 1.0560 validation loss and 0.8004 training loss. The model hit a stride by reaching a validation accuracy of 79.33% and training accuracy of 69.26%, where losses are 0.8316 for training and 0.7152 for validation loss. The model’s consistency and improvement are evident in Epoch 4. It achieved 80.00% in validation accuracy where the loss is 0.6164 and a training accuracy of 73.11% with a loss of 0.7326. Lastly, epoch 5 culminated this upward trajectory, marking the peak with an impressive 85.33% validation accuracy and a solid 78.37% training accuracy, where the loss is 0.6261 for training and 0.5476 for validation.

DenseNet-121.

DenseNet-121 has performed very well by getting 91.48% training accuracy and 88% validation accuracy with the help of the Nadam optimizer. The validation accuracy is reasonably close to the training accuracy, and it indicates effective generalization. DenseNet-121 has comprised 3,985,920 non-trainable parameters, 3,058,759 trainable parameters, and 7,044,679 total parameters, having a total of 121 layers where each layer is connected to every other layer in a dense block to contribute to its depth and efficiency in feature learning.

Throughout five intriguing epochs, DenseNet-121 (Fig 13c) validation accuracy has soared to 68% with a promising debut complemented by a 46.89% training accuracy where the losses are 1.3716 for training and 0.7898 for validation. In the last and final epoch, the model reached its zenith with a stellar 88% validation accuracy and 91.48% training accuracy, where the training loss is 0.2637 and the validation loss is 0.3938. The model’s validation accuracy is 88%.

VGG-16.

VGG-16 presented the lowest performance among all the pre-training models, with the Nadam optimizer containing 14,718,279 total parameters, where 3,591 are trainable and 14,714,688 are non-trainable. The model has a total of 16 layers, including 13 convolutional layers and 3 fully connected layers. But, when we customized the model, we used one fully connected layer in our research. The accuracy of both training and validation accuracy values is much lower than other pre-trained models. It has yielded 45.85% training accuracy and 56.67% validation accuracy. The model has appeared to be underfitted as validation accuracy is relatively lower than training accuracy, which indicates that the model is not capturing the underlying patterns in the data effectively.

We have presented the five-epoch journey of VGG-16 in Fig 13d. Starting with Epoch 1, the model showed a validation accuracy of about 22.67% and a training accuracy of 17.48%, where training loss is 2.9269 and validation loss is 2.2223. There’s a consistent improvement as the epochs progress. The validation accuracy rises to 34.67%, with the training accuracy at 25.48% in epoch two, having a 2.2080 training loss and 1.8132 validation loss. By moving to epoch 3, the model reached 42.67% in validation accuracy and 35.33% in training accuracy. Here, 1.8170 is the training loss, and 1.5151 is the validation loss. Having a training loss of 1.5944 and validation loss of 1.3085 in the fourth epoch, the validation accuracy goes up to 48.00% and the training accuracy to 41.93%. In final epoch 5, the model hits a validation accuracy of 56.67% and a training accuracy of 45.85% with a validation loss of 1.1472 and a training loss of 1.4109. Though the model does improve its accuracy over time, the training accuracy remains relatively low compared to the validation accuracy, suggesting that the model’s capacity may be too limited to grasp the nuances of the training dataset fully. So, despite the steady increase in accuracy throughout the five epochs of training the VGG-16 model, persistent underfitting was evident.

InceptionV3.

Firstly, InceptionV3 achieved a precision of 87.00%, meaning that 87% of the time, it is correct in predicting a class. With a recall of 84.00%, the model is able to capture 84% of the relevant instances. The F1-score, which harmonizes precision and recall, also stands at 84.00%, suggesting a balanced performance.

ResNet-50.

Another pre-trained model, ResNet-50, performs slightly lower in terms of precision compared to InceptionV3. The model has achieved 86% precision. However, the recall value of the ResNet-50 model is higher, almost 1%, than that of InceptionV3. This model’s performance is more balanced than that of InceptionV3. The harmonic mean of this model is 85%, which is 1% higher than the previous one.

DenseNet-121.

DenseNet-121 achieved the same value for both precision and recall, about 88%, which is higher than the previous models. Also, this model is able to obtain a perfectly balanced F1-score, which is 88%. That typically means that the pre-trained model DenseNet-121 performed very well, especially in terms of both identifying true positives and avoiding false positives.

VGG-16.

At last, we applied VGG-16 as a pre-train CNN model. But in this case, we got much lower values. The VGG-16 performs noticeably worse in terms of precision, recall, and F1-score. VGG-16 got only 58% precision and recall value, and the F1-score is 57%, whereas InceptionV3, ResNet-50, and DenseNet-121 achieved higher scores. VGG-16 obtained 18-20% lower precision and 16-20% lower recall than the other three pre-trained models.

So, our experiments showed that DenseNet-121 is the best pre-trained CNN model among those that have been applied; it offers the highest F1-score, recall, and precision. Conversely, VGG-16 performs the worst, having the lowest F1-score, recall, and precision. These findings highlight that to classify the mosquito species, we need extensive experiments to achieve better performance.

Ensemble Model 1 (EM1).

The first model, EM1, is the combination of InceptionV3, ResNet-50, and DenseNet-121, showing about 99% and 93% accuracy on training and validation, respectively, using the Nadam optimizer. EM1’s training and validation accuracy increased significantly using the Adam optimizer. We achieved 95.33% validation accuracy when Adam is used as an optimizer in this model. However, EM1 still outperformed the best-performing single pre-trained model DenseNet-121.

Ensemble Model 2 (EM2).

The second one, EM2, which is a combination of (VGG-16 + ResNet-50 + InceptionV3), gives the same training accuracy as the first one but reduces 3% overfitting. The validation accuracy is 96% by the Nadam optimizer. Using the Adam optimizer, the model achieved the same validation accuracy, whereas training accuracy slightly increased.

Ensemble Model 3 (EM3).

The third ensemble model, EM3, performs worse than EM1 and EM2. This model is a combination of the ResNet-50, VGG-16, and DenseNet-121 weak learners. However, the model’s accuracy is 99% and 92% in training and validation accuracy for both optimizers.

Ensemble Model 4 (EM4).

Finally, the fourth ensemble model, EM4, which is the combination of InceptionV3, VGG-16, and DenseNet-121, performs best when the optimizer is used as Adam. The model’s validation accuracy is 91% for Nadam and 94% for Adam. The analysis shows that the best-performing pre-trained model (DensNet-121) combinations give lower accuracy in the ensemble model most of the time.

Among the four ensemble models, EM2 performs best and does not suffer from any overfitting issues. Notably, DenseNet-121 was not involved in the combination of EM2. The worse ensemble model performances were observed in the combinations that involved the DenseNet-121 model. Using the Adam as an optimizer performed best in our research. However, all of the ensemble models outperformed all the pre-trained CNN models. So, the experiments showed that the combination of weak learners increases the performance in our research.

Ensemble Model 1 (EM1).

EM1 achieved 96% precision, 95% recall, and 95% F1-score with Adam optimizer. The Nadam optimizer, a variant of the Adam optimizer, produced 93% precision, recall, and F1-score, which is 3% lower than the Adam optimizer performance.

Ensemble Model 2 (EM2).

EM2 demonstrated the robustness of the Adam optimizer in optimizing the model’s parameters and striking a harmonious balance between precision and recall, with an astounding 96% precision, recall, and F1-score. In terms of precision, recall, and F1-score, the model performed better than EM1, but more significantly, it maintained a balanced performance.

Ensemble Model 3 (EM3).

EM3, using the Nadam optimizer, consistently produced results with 93% precision, recall, and F1-score. Still, the model’s performance was inferior to that of EM1 and EM2.

Ensemble Model 4 (EM4).

EM4 has marginally better performance than EM3 using the Adam optimizer as well. EM4 performed worse than EM1 and EM2 with 94% precision, 94% recall, and 94% F1-score.

Out of the four models, EM2 with the Adam optimizer is the most efficient and well-balanced model, with the best precision, recall, and F1-score. Some of the classification results produced by EM2 are demonstrated as examples in Fig 19.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 17. Precision-recall curves of four ensemble models using Adam and Nadam optimizers.

https://doi.org/10.1371/journal.pone.0322171.g017

Download:

• PNG
  larger image
• TIFF
  original image

Fig 18. ROC curves of four ensemble models using Adam and Nadam optimizers.

https://doi.org/10.1371/journal.pone.0322171.g018

EM1 exhibited competitive results with EM2, using the Adam optimizer. EM3 and EM4 perform marginally worse than the other two, with EM3 being the least optimal, especially when it comes to recall and F1-score.

Kappa and MCC results.

To evaluate the model extensively, We have also calculated the Kappa () and MCC values of our model, which are shown in Table 9. The findings provided insight into the models’ consistency and dependability in making predictions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 9. Kappa and MCC values of four ensemble models.

https://doi.org/10.1371/journal.pone.0322171.t009

Regardless of the optimizer used, EM2 performed exceptionally well, displaying high Kappa values that indicate a significant degree of agreement between its predicted classifications and the actual results (Nadam: =0.9519; Adam: =0.9519). These findings imply that when it came to correctly classifying observations, EM2 consistently performed better than the others. Even with slightly lower Kappa values (Adam: =0.9118; Nadam: =0.9038), EM1 showed a good degree of agreement.

Higher Kappa values are indicative of more robust and consistent predictive performance, offering practitioners and researchers insightful information about how well ensemble models perform under various optimization schemes.

The MCC results provide more information about the models’ performance. In addition to consistently displaying high Kappa values, EM2 also showed remarkable MCC values, highlighting its resilience in producing precise predictions (MCC = 0.9523 for Adam and MCC = 0.9523 for Nadam).

EM1 also displayed the second-highest performance in this criterion. The subtle variations in MCC values between the ensemble models and optimizers offer a thorough grasp of their predictive capacities. Models with higher MCC values are better able to predict outcomes accurately in a variety of classes. When paired with Kappa values, these MCC results help provide a more complete evaluation of the ensemble models.

Graphical evaluation.

The performance of the positive class is typically the minority or less frequent class and is the focus of precision-recall curves. More useful metrics for evaluating how well a model recognizes instances of the positive class are recall (a.k.a. true positive rate or sensitivity) and precision (a.k.a. positive predictive value). These metrics make it possible to track how the decision threshold is adjusted and how the trade-off between recall and precision changes.

Fig 17 provided an overview of the precision vs. recall performance evaluation of our ensemble models, whereas Figs 17a–17d showed the Adam optimizer curves and Figs 17e–17h the Nadam optimizer curves. The illustrated graphs also showed that EM2 with Adam

optimizer performed better than the other models. For every class, an additional examination of the precision-recall performance was conducted.

Impressively, the Aedes albopictus landing class achieved a 96% recall value and 100% precision. The Aedes aegypti landing class, on the other hand, showed a higher true positive rate than the positive predicted value, with a precision value of 92% and a recall value of 96%. With a sensitivity rate of 91%, EM2 positively predicted 95% of the Aedes aegypti smashed class. The precision and recall values of the Aedes albopictus smashed class were 93% and 100%, respectively. As seen in Fig 17b, the Culex quinquefasciatus landing class showed a balance between precision and recall values. For this class, EM2 achieved a 97% true positive rate and positive prediction rate. Finally, the Culex quinquefasciatus smashed class and attained an astounding 100% precision and 95% recall.

In summary, our findings suggested that EM2 equipped with the Adam optimizer is a dependable and efficient instrument for precisely forecasting mosquito species. EM2, with Nadam optimizer, performed the same, but for the Aedes albopictus smashed class, the precision value increased by 2%, reaching 93% from 91%.

Receiver operating characteristic (ROC) curves are generated to visually represent the relationship between sensitivity and specificity for different cut-off points in experimental combinations. From ROC curves, we also calculated the area under the ROC curve (AUC) values for all the classes. The maximum possible AUC value is 1.0. The higher the AUC value, the better the classifier.

The ROC curve analysis for the Adam and Nadam optimizer of the four ensemble models with six different mosquito classes is demonstrated in Fig 18. In Figs 18a–18d, the

performance of the Adam optimizer is shown, while in Figs 18e–18h, that of the Nadam optimizer is demonstrated.

A detailed examination of the AUC performance metric in the complex field of mosquito classification models reveals interesting trends among various ensembles and optimizers.

Aedes albopictus landing, where consistently high scores of 0.98 were achieved by EM1, EM2, and EM3 with the Adam optimizer. However, EM4, using the Nadam optimizer, experienced a considerable drop, recording a decreased landing score of 0.88.

Shifting our focus to the Aedes aegypti landing, the pinnacle was achieved at an outstanding 0.99 with EM1 using the Nadam optimizer. Yet, the journey for EM3 was challenging, as it struggled to gain only a 0.93 success rate, irrespective of whether it was the Adam or Nadam optimizer.

Aedes aegypti smashed, on the other hand, exhibited an impressive 0.95 score with EM2 in both Nadam and Adam optimizers. In contrast, EM1 lagged at 0.91, particularly in the Nadam optimizer.

The saga continues with Aedes albopictus smashed, showcasing remarkable performance at 0.99 with EM2 and EM4 in Adam, as well as EM1 and EM2 in Nadam optimizers. However, a drop in performance was marked with EM3 and EM4 in the Nadam optimizer, alongside EM1 in Adam, showing comparatively lower scores at 0.97.

Culex quinquefasciatus landing emerged as a strong performer, especially with EM2, earning a commendable 0.98 in both the Nadam and Adam optimizers. Yet, EM4 with the Adam optimizer experienced a fall, dropping to 0.94.

Finally, the class Culex quinquefasciatus smashed reached its peak at 0.98 with EM1, EM2, and EM4 in Adam, as well as EM2 and EM3 with the Nadam optimizer. However, EM1 with the Nadam optimizer demonstrated a relatively lower value of 0.96.

After a thorough examination of the mosquito classification models, EM2 has proven to perform exceptionally well in a variety of scenarios and classes. This ensemble model consistently attained outstanding peaks of 0.99 in Aedes aegypti landing, 0.95 in Aedes aegypti smashed, and 0.99 in Aedes albopictus smashed.

Strengths.

The two main strengths of our proposed method are as follows.

Reducing complexity: Although ensemble-based models are recognized for their increased accuracy, real-time applicability is questioned due to their computational cost. Even for delicate activities like vector mosquitoes, we prioritize gaining better classification performance, and we are open to investigating efficient learning strategies to lower computational costs. To achieve our desired results, the ensemble model that we have suggested makes use of multiple customized Deep Convolutional Neural Networks (DCNNs) that are adapted to different learning constraints. With 138 million parameters, the original pre-trained VGG-16 model is known for its complicated architectural design; however, this complexity has been significantly reduced by our fine-tuning procedure. This decrease in the number of parameters is a common occurrence seen in all of the pre-trained models used as base learners in our study.

For a comprehensive analysis of the parameters for each base model, we have presented Table 4. A typical deep ensemble model usually entails using many CNN designs and averaging the predictions that result from them. Our work seeks to set itself apart, nevertheless, by proving the effectiveness of ensembles based on transfer learning while also cutting down on computing complexity. We reduced the requirement for extensive training from the start by leveraging the knowledge gained from pre-trained models by utilizing the power of transfer learning.

We have found that our EM2 model (InceptionV3 + ResNet-50 + VGG-16) performed the best. The large number of trainable parameters for DenseNet-121 (over 3 million, ref. Table 4) is not in this best-performing combination. So, our model not only improves performance but also significantly lowers its computing overhead after customizing the pre-trained models.

Prediction quality and robustness: The proposed method did not suffer from any “accuracy fallacy” or “accuracy paradox.” The term accuracy fallacy/paradox refers to a situation in which a model achieves a high accuracy but is unable to predict correctly. This phenomenon occurs when a model performs well overall but finds it difficult to classify instances correctly, especially in datasets that are imbalanced or when misclassification costs differ between classes. The accuracy paradox draws attention to the drawbacks of using accuracy as the only performance metric. This is because accuracy may not accurately represent the efficacy of the model, particularly in practical applications where precise predictions are crucial.

To acquire a more thorough grasp of the model’s overall prediction quality and robustness, it is necessary to take into account alternative metrics such as AUC, F1-score, precision, recall, MCC, and Kappa. The proposed model has also been evaluated on those measurement criteria, and the analysis of the outcomes shows that the model can overcome the problem of accuracy fallacy. As stated before, the dataset is imbalanced, but our model does not exhibit much bias against minority classes. The confusion matrices, precision-recall curves, and ROC curves showed excellent class-wise performances of the proposed method graphically.

Limitations.

On the other hand, our proposed method has its limitations.

Dependence on transfer learning: Because our model depends on pre-trained networks, its initial knowledge may derive from datasets that are not entirely relevant to our particular topic. By utilizing past information, transfer learning dramatically improves performance, yet it also restricts the model’s capacity to learn features that are unique to a given domain. Predictive performance may suffer when our dataset differs significantly from the ones used to pre-train the models, requiring further fine-tuning or even starting over.

Data imbalance challenges: Although our model has demonstrated encouraging outcomes in managing data imbalance, it is not completely impervious to the difficulties presented by severely skewed datasets. The suggested method’s performance may be further improved if the data distribution is more evenly distributed.

Interpretability issues: Interpretability/explainability is still a major challenge even though our model performs exceptionally well in terms of accuracy and other performance measures. Ensemble models typically operate as “black boxes,” making it challenging to comprehend the decision-making process, particularly for those based on deep learning architectures. This lack of transparency may be an issue when implementing models in industries where interpretability is critical, such as healthcare, where knowing the reasoning behind predictions is just as vital as the predictions themselves.