1.1 Related work

There is a long history of combining separate pieces of information to improve the learning process and resulting models. In [6], Abu-Mostafa used hints in the form of prior knowledge about the unknown function to improve the model being trained, whereas we use additional information about a specific instance of input to improve its classification. There is a growing interest in including knowledge bases or metadata in the learning process for hybrid models combining neural networks with symbolic knowledge (e.g. [7, 8]).

There are also works focused on enhancing image classification with context metadata in various applications (e.g., [9–13]). In comparison, our approach makes use of already existing large pre-trained models eliminating the need for processing and incorporating the metadata into a complicated pipeline.

There are many approaches to combining multiple modalities [14–16]. Integration can happen at the input level (early fusion), at the decision level (late fusion) or intermediately [16]. Hybrid fusion [17] combines all the approaches. Like Axelsen et al. [1] and Chen et al. [2] derive a late Bayesian fusion scheme for integrating multiple visual classifiers with conditionally independent modalities, similar to our assumption. As noted, we are concerned with joining independent evidence in the form of hints of any type. Axelsen et al. [1] also propose a permutation test method to discover dependence among modalities in a dataset.

The efforts of combining vision and language have been explored in several works. Relevant tasks include visual question answering [18, 19] and visual reasoning [20, 21]. Models jointly trained on text and images have been developed (e.g. [22, 23]). These approaches typically solve more complex problems leading to intermediate-level data fusion.

2.1 Multimodal fusion by Bayesian inference

Theorem 1. Given N observations x₁, …, x_N and logits (i.e., outputs of the last linear layer, before the softmax function) such that for all relevant i, j: , and assume for all classes c_i that P(x₁, …, x_N, c_i) > 0. Then where π and κ(x₁, …, x_N) are vectors in with elements with C being the number of classes, and the logarithm is applied element-wise.

Remark. Remind that is defined as

Proof. Using Bayes’ rule and multiplying by , we obtain that: (1)

We notice that the fraction can be rewritten as such: (2) where . Now we substitute softmax for P(c_i|x_j) as well as the above result into Eq 1 to get:

Remark. If we assume P(x₁,…, x_N, c_i) = 0 for some possible realization, then ln κ_i(x₁, …, x_N) or ln π_i is undefined and we have P(c_i|x₁, …, x_N) = 0.

If we assume that P(x₁, …, x_N|c_i) = P(x₁|c_i) · … · P(x_N|c_i), and avoid using κ_i(x₁, …, x_N) to resolve dependencies in the derivation, we get the same result as in Eq 3. Here we relax the assumption to P(c_i) > 0.

Remark. We see that for ordinary logistic regression on the concatenated embeddings, a weight and a bias exist such that it is equal to the naive Bayes fusion (i.e. when ln κ(x₁, …, x_N) = 0). Assume we have for all N classifiers, as well as the block-matrices W = [W₁| … |W_N], and , and the bias . We then see that

We notice that in the case of N = 2, the ln κ_i elements are the conditional mutual information of the observations conditioned on each class i. We further notice that for equiprobable classes, the (N − 1) ln π term can be left out since softmax is invariant to translation by any scalar multiple of the one-vector. The result generalizes the derivation in [1, 2] to include a term correcting for the naive Bayes assumption of independent modalities given class. If we suppose that modalities (observations and hints) are independent given class, we can simplify the result in Theorem 1 (κ(x₁, …, x_N) = 1) and get (3) We discuss conditional independence further in Section 2.3: Conditional independence. The vector ln π can in practice be computed by counting before performing any inference in the combined model.

Supposing we have a classifier of each modality, we can use this formula to estimate posterior probabilities by combining the logits and prior probabilities. This new model combines the original predictions (logits) with new information coming from processing hints. If the original classifiers are good, this combination generates better predictions.

2.2 Calibration

Since the result was derived using the posterior class probabilities, approximating these conditional probabilities well is crucial to using the model fusion approach in practice. However, as noted in [24], modern neural network classifiers are not guaranteed to be well-calibrated. For this reason, we quantify the effect of calibration of the classifiers on the accuracy of the fusion model. Since we add the logit vectors in our fusion scheme, their magnitudes play a major role—if the discrepancy is too large, one model will dominate the other in the decision. In our experiments, we used temperature scaling [24] to make the magnitudes reflect the accuracy of the model. We calibrated the models on their respective validation sets using 25 bins. We can summarize the approach of our calibrated Bayesian fusion (see Fig 1, bottom-right corner) in the following steps:

1. Get or create unimodal classifiers for the primary observation and all hint modalities.
2. Calibrate all the models.
3. Insert each modality of the input into its respective model and collect the logit vectors.
4. Sum all the logits, subtract prior probabilities and apply the softmax function on this vector (following Eq (3)).

We compare the performance of our late fusion model to an intermediate fusion scheme based on support vector machines. A linear SVM classifier [25] is trained on concatenated embeddings coming from each unimodal classifier. As embeddings, we take outputs of the second-to-last layer of the classifiers or the immediate outputs of the large-scale pre-trained models. They can vary in dimensions for each modality. We do it to compare the performance of our fusion model to what can be achieved with the embedding vectors as inputs.

2.3 Conditional independence

An important assumption of Eq (3) is the conditional independence of all modalities, conditioned on the class (label). In practice, this assumption is often satisfied. For example, imagine we have images of different objects and textual descriptions of the same types of objects. Once we know that the object is a chair, the images and the textual descriptions are independent.

When classifying MNIST images into two classes: even and odd, the independence condition is unfulfilled. A text describing a certain digit would reveal more information than just parity. However, we could repair the dependence if we divide the classes further into individual digits and condition on these classes. This can be done in general: even if the mutual independence of x₁|c, …, x_N|c is not possible, one can often further partition into a set of subclasses denoted such that x₁|c′, …, x_N|c′ are mutually independent for all . One could then build a model classifying into all partitions and use Theorem 1 because the subclasses would be conditionally independent. To obtain the original class probability, one would sum over the probabilities of its subclasses: .

An example illustrating this partition could be a classification of images of places with an additional image hint (e.g. an image of the same place taken from a different angle). Imagine a class “park” and two pictures taken in the same park during winter. Since it snows in the winter, both images contain snow, whereas images taken during summer do not. Conditioning only on class is not enough in this case. However, if we condition on both class and season (“park during winter”), we get the independence and can use our scheme.

If we are able to identify all latent variables, then we can condition on these variables, get conditionally independent data and use Theorem 1.

3.1 Training details

We used the Adam optimizer and the cross-entropy loss The metrics we used to compare our models are the top-1 and top-5 accuracies (i.e., proportion of the data such that the target label is within the 1 (or 5) highest-scoring predicted class(es)). In the tables, we used the conventional notation of displaying the accuracies in percentages. We used the 95% Jeffreys intervals to quantify uncertainty for the accuracies. Since the text modalities have fewer observations than the image modalities, we based the interval radii for the fusion models on the text only. This approach is likely overestimating uncertainty.

For CMPlaces, we used 20500 observations for the image models and 2050 observations for text/fusion. Similarly, we used 49550 and 1708 for ImageNet, in the same order.

We used Scikit-Learn [39] to implement the SVM classifier. To tune the hyperparameters of the text classifier, we used the TPESampler (Tree-structured Parzen Estimator) procedure in Optuna [40]. Here we optimized for top-1 accuracy in the text-classification task alone. We experimented both with using weight decay and dropout for regularizing the classification head. Also, we tuned the learning rate and the amount of layers (and hidden units) at the top part of the network. Since we found the best performance using only dropout, a linear layer, and a tuned learning rate, we tuned these exclusively in the end. Throughout the search, we tuned the model on a training split (80%) for 5 epochs and validated on a validation split (20%). The splits were picked randomly at the beginning of the tuning and were fixed across all trials.

We tuned the regularization strength in the SVM-based fusion scheme manually on the training split and validated it on the validation split. We optimized for the best top-1 accuracy in the combined task. We presented a large spectrum of regularization values and tested the performance after training for 1 epoch. We chose 1 epoch since it gave a good performance but was also very time-demanding. Note here that the training split is comparable in size to the size of the entire training set, which proved to be very time-consuming to train on (see Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Accuracies vs time.

The top-1 accuracies of each model plotted with uncertainties against the time spent on preparing and evaluating them. The best models are in the upper left corner as indicated by the green arrows. Here we suppose that we are provided with trained constituent classifiers. Hence, for the Bayesian fusion models, the preparation time only includes the time to calibrate and estimate the log-prior. The preparation time of SVM comprises training and calibrating the SVM head.

https://doi.org/10.1371/journal.pone.0301360.g002

In order to calibrate the image and text classifiers, we used LBFGS with a fixed learning rate. We used the implementation from [41] which can be found on the associated GitHub repository. The calibrations of the image and text classifiers were done on the validation sets of the image and text modalities respectively. We find this choice justifiable since calibration does not affect the predictions of the models individually, but only affects their confidence in prediction. Furthermore, the calibration aligns the prediction confidence of the model with the prediction accuracy. And since we want the model to be as close as possible to predicting the true conditional probability p(c|x), it makes sense to calibrate it on the validation set assuming it is representative enough of the true underlying distribution.

The implementations and settings of the searches are all found in the associated GitHub repository of this paper [42]. The selected hyperparameters are found set in the code in the file main.py to make the experiments reproducible.

3.2 Results

Table 1 summarizes the results of the experiments described in the previous part. In both experiments, we see that fusing calibrated models produced a significant improvement over each of the two constituent classifiers, as well as the uncalibrated fusion model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Results of the experiments.

https://doi.org/10.1371/journal.pone.0301360.t001

We broke the calibrations of the unimodal classifiers by temperature scaling to observe their importance on the results of our fusion scheme. If badly-calibrated models are used, the logits may be of different magnitudes, and one of the classifiers may outweigh the other. Logits with comparable magnitudes are needed for a balanced voting process. Our findings agree with the observation Chen et al. made in [2] that calibration is crucial for this type of model fusion.

Calibrating the unimodal classifiers is important not only for classification accuracy but also for getting a well-calibrated fusion model. Our results indicate that fusing uncalibrated models yields an uncalibrated fusion model, and fusing well-calibrated unimodal classifiers results in a well-calibrated fusion model.

We can also observe that the calibrated fusion models have better or comparable performance to the linear SVM classifiers. It tells us that we cannot get substantially better models built on top of the embedding vectors.

We report the time needed to conduct the experiments in Fig 2. It shows that SVMs require considerably more time and resources to perform well than calibrated fusion models. This indicates that our fusion scheme can give a roughly equal success rate with only negligible resource requirements.