1.1 Problem definition

The ability to present the information to the user is an important task of every dialogue system. Dialog manager decides on what to say to the user but how to say the information is the sole responsibility of natural language generation (NLG) module of a dialogue system. In our current work, we focus on response generation in a multimodal setup. In Fig 1, it is evident that the responses are dependent on the visual features as well as the textual information. Therefore, our present work is based on a recently released multimodal dialogue [5] dataset. Our primary focus is to generate diverse and polite responses in a multimodal system by utilizing the information of text and image. The existing systems concentrate on generating the responses, while in this work our target is to make the responses interesting and interactive to increase the customer satisfaction leading to customer retention. The task is challenging as we need to consider the information present in both the images and text, and to make the responses diverse and polite simultaneously. Existing unimodal systems suffer from the problem of generating dull and generic responses like Yes, No, Okay, I don’t know etc. The researchers, therefore, focused on making the responses diverse [10, 11]. Our current work is greatly motivated from these prior works, but our focus is on building a multimodal dialogue system.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. An example from the MMD dataset.

https://doi.org/10.1371/journal.pone.0241271.g001

The user utterances play a significant role in building the dialogue context for generating coherent and appropriate responses, in accordance to the user demands. We focus on generating the textual responses only in a similar manner as [8, 12, 13]. Here, the task of multi-modal dialog generation is defined as follows: we consider both the modalities, i.e. text and images by effectively combining these two information for the generation of the textual response as opposed to the unimodal systems that typically use only textual information for generating the next response. The primary objective of this work is to generate the diversified and polite responses for increasing the human-machine interaction, and we do not focus on image retrieval or generation in our current work.

1.2 Motivation and contribution

Task-oriented dialogue systems are based primarily on textual (unimodal) information. Growing requirements in various fields, such as travel, entertainment, retail etc. require conversational agents to be able to communicate by incorporating information from the different modalities in order to build a robust system. Our research is based on the previously proposed multimodal dialog [5] (MMD) dataset, composed of the conversations related to e-commerce (fashion domain). The work is focused on generating textual responses based on both text and images present in the conversational history. An example from the MMD dataset is shown in Fig 1. It is evident from the figure that the responses are generic and not very interactive. It can be seen that the system’s responses are repetitive and mostly in one word like yes or no. Hence, in our current research, we aim at generating the diverse textual responses by capturing information from both text and images. Diversification in textual responses is an open and exciting research problem [10, 14]. This becomes more challenging in multimodal dialogue systems, as information from the images is essential for making the responses diverse while preserving the content as there is a strong correlation between the text and images. For every goal-oriented dialogue systems, the interaction between the user and the system should lead to a better user experience and satisfaction. Hence, the generation of interactive, informative and polite responses is imperative for every conversational agent. Our primary objective is the generation of responses in accordance to the contextual information while making it more diverse and courteous.

In this work, we employ a multimodal hierarchical encoder to encode the information from both text and image. Due to strong dependency between the textual contents and images, we apply parallel co-attention in order to capture the effective evidences from image and text, and to create an useful context for the decoder. We employ BLOCK [15] fusion technique which is based on the block-superdiagonal tensor decomposition for learning better multimodal representation. For increasing the diversity in responses, we apply stochastic beam-search [16] which uses Gumbel Top k-trick for generating responses. To the best of our knowledge, we are the first in presenting a novel approach for making textual responses more diverse in a goal-oriented multimodal dialogue system.

The key characteristics of our present work are as follows:

• We employ a parallel co-attention mechanism to derive the dependencies between the text and image by focusing on the important textual and image information.
• We incorporate BLOCK—Bilinear superdiagonal fusion module to obtain the improved multimodal representations.
• We integrate Stochastic beam search with Gumbel Top k-trick for incorporating diversity in responses while preserving the contextual information.
• We achieve the state-of-the-art performance in making diverse and informative textual responses on the MMD dataset.

2.1 Unimodal dialogue systems

Deep learning’s efficacy has shown notable improvement in the generation of dialogue. As shown in [17, 18], deep neural models are quite successful in modelling the conversations. To capture the context of previous queries by the user, the authors in [19] proposed a hierarchical framework capable of preserving the past information. Similarly, to preserve the dependencies among the utterances, a hierarchical encoder-decoder framework was investigated in [20, 21]. Sequence to sequence (seq2seq) neural models often generate incomplete and boring responses, such as I don’t know, Okay, Yes, No, etc. Hence, bringing the diversity in responses is an extremely challenging and interesting research problem for every conversational agent.

Generation of interesting responses has been the objective in many dialogue generation works, such as [10, 22–27]. In [10], maximum mutual information (MMI) as objective function was proposed for generating the diverse responses in neural models. Similarly, the authors in [23] used inverse token frequency as an objective function for generating the interesting responses. In [24, 25] and [27], conditional variational auto-encoders were used to generate coherent and diversified responses. In [22], adversarial learning was employed for generating informative and diverse responses. Deep reinforcement learning models [26] have also shown remarkable improvement in generating interesting responses. In [28], the inter-sibling ranking penalty was added to favour responses from diverse parents to be generated instead of using the standard beam search algorithm. The authors in [14] proposed diverse beam search algorithm that decodes a list of diverse outputs by optimizing for a diversity-augmented objective.

In [29], a dialogue generation model was proposed that directly captures the variability in possible responses to a given input, which reduces the ‘boring/monotonous output’ issue of deterministic dialogue models. The generative adversarial network was employed in [30] for generating informative and diversified text. In [31], SpaceFusion model was proposed to jointly optimize diversity and relevance of a sequence-to-sequence model with that of an autoencoder model by leveraging regularization terms. In [32], the authors proposed a reinforcement learning-based approach which considers a set of responses jointly and generates multiple diverse responses simultaneously. The authors in [11] propose a Frequency-Aware Cross-Entropy (FACE) loss function for generating diverse responses by incorporating a weighting mechanism conditioned on token frequency. In [33], the authors proposed an easy-to-extend learning framework named MEMD (Multi-Encoder to Multi-Decoder), in which an auxiliary encoder and an auxiliary decoder are introduced to provide essential training guidance for generating diverse responses. A multi-mapping mechanism was proposed in [34] to capture the one-to-many relationship, where multiple mapping modules are employed as latent mechanisms to model the semantic mappings from an input post to its diverse responses.

2.2 Multimodal dialogue systems

Recently, research in dialogue systems has shifted towards incorporating different modalities such as images, audio and video for capturing information to make the robust systems. The research reported in [7, 35–38] has been effective in narrowing the gap between vision and language. In [36], an Image Grounded Conversations (IGC) task was proposed, in which natural-sounding conversations were generated about a shared image. Similarly, the authors in [7] introduced the task of Visual Dialog, which required an AI agent to hold a meaningful dialogue with humans in natural, conversational language about visual content. In [39], a combination of generative adversarial networks (GANs) and reinforcement learning was employed to generate more human-like responses to the questions having visual information as well. The authors in [40] addressed the task of cross-modal semantic correlation for Visual dialog that utilized a dual visual attention mechanism for answering the questions. Similarly, for better visual and textual correlation, in [41], a textual-visual Reference-Aware Attention Network was employed for the generation of correct answers in accordance to the input image and dialog history. Our work varies from these as the Multimodal Dialog (MMD) conversations [5] deal with various images, and the growth in conversation depends on both image and text as opposed to a single image conversation.

Recently, video and textual modalities were investigated with the release of the DSTC7 dataset in [9] that made use of a multimodal transformer network to encode videos and incorporate information from the different modalities. Similarly in [6, 42, 43], DSTC7 dataset has been used for generation by incorporating audio and visual features. Earlier works on the MMD dataset reported in [12, 13, 44] used the hierarchical encoder-decoder model to generate responses by capturing information from text, images and the knowledge base. Recently, [8] proposed attribute and position-aware attention for generating textual responses. The authors in [45] used an hierarchical attention mechanism for generating responses on the MMD dataset.

Our work differs from the existing works (mainly, on the MMD dataset) in a sense that we aim here to generate diversified responses using stochastic beam search with Gumble k-tricks at the time of generation for a multimodal dialogue system. The task is challenging as we have to consider the images also for generating the coherent and informative responses. Hence, we focus on capturing vital contextual information by applying co-attention between text and image to achieve important details from both the modalities. We employ BLOCK fusion technique instead of the linear concatenation of modalities to obtain better multimodal representation. The end goal is to achieve responses that are not only coherent to the conversational history but also interesting, diverse and polite.

3.1 Formal problem definition

In this paper, we address the task of generating diverse and polite textual responses in accordance to the conversational history in a multimodal dialog setting. The dialogues consist of textual utterance along with the multiple images. Given a context history of p turns, we address the task of generating the next response that is coherent, diverse and polite, leading to better and more engaging human-machine conversation. More precisely, given an user utterance U_p = u_p,1, u_p,2, …, u_p,j, a set of images I_p = img_p,1, img_p,2, …, img_p,j′, with the dialog history H_p = (U₁, I₁), (U₂, I₂), …, (U_p−1, I_p−1), we focus on generating interesting, informative, polite and context-aware response Y_p = (y_p,1, y_p,2, …, y_p,k) instead of template like generic and monotonous responses, such as I don’t know, Yes, No, Similar to…, etc. This will enhance human-machine conversations by keeping the users engaged in the conversation. Here, p is the p^th turn of a given dialogue, while j is the number of words in a given textual utterance and j′ is the number of images in a given utterance. Note that in every turn, the number of images j′ ≤ 5, so in-case of only text, vectors of zeros are considered in place of image representation.

3.2 Multimodal hierarchical encoder decoder

We construct a generative model for response generation, as shown in Fig 2, which is an extension of recently introduced Hierarchical Encoder-Decoder (HRED) architecture [20, 21]. As opposed to a standard sequence-to-sequence models [46], the dialogue context among the utterances is captured by adding utterance-level RNN (Recurrent Neural Network) over the word-level RNN increasing the efficacy of the encoder to capture the hierarchy in dialogue. The multimodal HRED (MHRED) is built upon the HRED to include text and image information in a single framework. The key components of MHRED are utterance encoder, image encoder, context encoder and decoder.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Architectural diagram of our baseline multimodal hierarchical encoder-decoder.

https://doi.org/10.1371/journal.pone.0241271.g002

3.3 Proposed approach

To improve the performance of the MHRED model, we use an attention layer to mask out the insignificant information instead of merely concatenating the representations of the text and image encoder. It is essential to focus on both image and text modalities to achieve informative and coherent responses while generating exciting and diverse responses. For better representation across the modalities, we employ the recently proposed BLOCK fusion technique [15]. We provide the knowledge base information to the decoder for generating informative and context-aware responses. Finally, to generate diverse responses, we incorporate stochastic beam search at the time of generation. Beam search makes use of Gumbel max trick for top-K samples. The architectural diagram of the proposed framework is given in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Architectural diagram of our proposed multi-modal hierarchical encoder-decoder with parallel co-attention mechanism along with BLOCK fusion technique between the text and image.

Input to the decoder is the attended and fused context vector along with the decoder and KB inputs.

https://doi.org/10.1371/journal.pone.0241271.g003

3.4 Training and inference

The whole model is trained using teacher-forced cross entropy [51] at every decoding step to minimize the negative log likelihood on the model distribution. We define as the ground truth of the given input sequence. (27)

For diversity, we use Stochastic beam search for generating the response, which is more diversified and also capable of preserving the contextual information. Stochastic beam search was derived from the Gumbel Top-K trick to sample sequences without replacement. This requires instantiating all the sequences in the domain to get the largest perturbed probability. Then we transit to a top-down sampling of the perturbed log probabilities.

3.5 Baseline models

For our experiment, we implement the following baseline models.

Model 1 (MHRED): The first model is the baseline MHRED model as previously described in the methodology section. This is shown in Fig 2, that has utterance encoder, image encoder, context encoder and the decoder for generating the responses.

Model 2 (MHRED+KB): In this model, we add the KB information at the decoder to incorporate contextual queries and celebrity information for endorsing different brands and products for informative response generation that is in accordance with the conversational history.

Model 3 (MHRED+KB+A): In this model, we employ global attention [49] at the decoder for better decoding at each time step.

Model 4 (MHRED+KB+A+DAA(I,T)): In this model, for different attributes on the text, we implement dynamic attribute attention to obtain the enhanced representation computed by: (28) (29) Here, a self-attended text embedding is used as query U_p to calculate the attention distribution over the image feature representation as H_I = [H_I,1, H_I,2, H_I,3, …, H_I,j′]. (30) (31)

Model 5 (MHRED+KB+A+PCA(I,T)): In this model, we employ parallel co-attention mechanism as described in 3.3.1 to attend the image and text simultaneously. We connect image and text by calculating the similarity between the image and text features at all the pairs of image and text utterances, as previously described in the methodology section.

Model 6 (MHRED+KB+A+PCA(I,T)+MFB(I,T)): In this model, we concatenate pairwise text and image features U_f and I_f obtained after parallel co-attention mechanism as input to the MFB module for better interaction between the modalities as used in [8].

Model 7 (MHRED+KB+A+PCA(I,T)+BLOCK(I,T)): In this model, we concatenate the pairwise output of text and image features U_a and I_a obtained after parallel co-attention mechanism as input to the BLOCK module where the output of BLOCK serves as input to the context encoder.

5.1 Implementation details

All the implementations have been performed using PyTorch https://pytorch.org/ framework. We use 512-dimensional word embedding initialized randomly. We did not use any delexicalisation, and our model learns independent of the context encoder and knowledge base (KB). All the encoder and decoder have 1-layer GRU cell with 512 hidden dimensions. For image representation, we use 4096-dimensional FC6 layer from VGG-16 [48], pre-trained on the Imagenet. We use AMSGrad [52] as the model training optimizer to alleviate the problems of slow convergence. We use dropout [53] with the probability of 0.4 to avoid over-fitting. All the parameters are randomly initialized using Gaussian distribution with Xavier scheme [54]. For generating diversified responses, we use the different methods, such as beam search, diverse beam search and greedy sampling. Finally, in our proposed model, we employ stochastic beam search with sample size k: 5, 10 as implemented in fairseq https://ai.facebook.com/tools/fairseq/. It works based on the Gumbel Top-k trick that samples sequences without replacement from the sequence model. We experiment with different learning rates and lastly fix it as 0.0004.

To measure the politeness quotient in the generated response we design a politeness classifier as presented in Fig 5. The input is first converted into embeddings using the embedding layer. We use 300-dimensional Glove embedding for representing the utterances. The embedded utterance is fed to the convolutional layer with filter size 3. After convolution, we apply max pooling to obtain the hidden representations. For more abstract representation of the dialogue utterance, we apply a unidirectional LSTM network on the hidden representation followed by self-attention. The number of neurons on the LSTM layer is fixed to be 200. Finally, we apply the softmax layer for attaining the politeness accuracy of the given utterance. To avoid over-fitting of the neural network a useful regularisation technique known as dropout has been used in our model. At the time of forward propagation, the neurons are randomly tuned-off so that the convergence of weights is restricted to the identical positions. For optimisation and regularisation, we use Adam optimiser along with 15% dropout in our model. Categorical cross-entropy is employed to update the model parameters.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Architectural diagram of politeness classifier.

https://doi.org/10.1371/journal.pone.0241271.g005

6.1 Evaluation metrics

6.2 Results of automatic evaluation

We present the results of our experiments in Table 2. It is evident from the table that our proposed approach outperforms the baselines in generating diversified responses, and these improvements are statistically significant We perform statistical significance t-test [60] and it is conducted at 5% (0.05) significance level. The perplexity score of the proposed approach with k = 5 is the least, thereby proving the fact that the generated responses are better from the rest. This suggests that our proposed framework employing stochastic beam search with gumble top k-tricks(here k = 5) is able to generate better responses in contrast to the baseline models and the existing system. In the baseline approaches, the framework employing PCA and BLOCK fusion technique outperforms all the other baseline networks. This indicates that both PCA and BLOCK fusion employed together makes the network perform better as opposed to the frameworks in which they are employed individually. Similarly, there is a decrease in perplexity of about 15 points when we use BLOCK instead of MFB fusion technique that states the superiority of BLOCK fusion in comparison to the MFB fusion technique.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Results of different models on MMD dataset.

https://doi.org/10.1371/journal.pone.0241271.t002

Similarly, the distinct-1 and distinct-2 metrics of the final model with stochastic beam search decoding proves that the proposed framework have been successful in generating diverse responses for the MMD dataset. Our proposed framework outperforms the existing approaches such as MMI [10], MMI-antiLM [10] and MMI-Bidi [10] for both distinct-1 and distinct-2 metrics. From the baseline models, there is an improvement in the distinct-1 and distinct-2 measures for the proposed approach having stochastic beam search decoder for both k = 5 and 10. This implies the efficacy of the stochastic beam search that ensures diverseness in the responses. The distinct-1 and distinct-2 scores demonstrate a huge jump in comparison to the best performing baseline network MHRED+KB+A+PCA(I,T)+BLOCK(I,T), thereby illustrating the effectiveness of stochastic beam search with Gumble k-tricks. From the table it is visible that our baseline framework shows improved performance in case of BLEU-4 metric in comparison to the existing systems. We observe that there is a drop in BLEU-4 metric in our proposed model,MHRED+KB+A+PCA(I,T)+BLOCK(I,T) compared to the baseline. This favors our assumption that the generated response is diverse in nature, and hence is not completely similar to the ground-truth. The politeness score also increases in our proposed framework with an increment of 4% with respect to the final baseline model. This improvement indicates that the generated responses are more polite than the baselines. Therefore, our proposed approach ensures that in the multi-modal setup, both diversity and politeness are preserved.

6.3 Analysis of network parameters

We provide the analysis of different hyper-parameters such as dropout and learning rate for our proposed framework. We used Perplexity (PPL) as the primary metric to fine-tune the framework for determining the network parameters in case of dropout and learning rate. After fixing these parameters we do the complete evaluation in terms of both automatic and human metrics for all the baselines and proposed framework. We used politeness accuracy (PA) as the primary metric for determining the number of LSTM layers in the politeness classifier. The dropout probability was determined to be 0.4 by considering the range of 0.1—0.8. The statistical analysis of the results for the different values of dropout probability for the proposed framework is provided in Table 3 of the revised manuscript. In Fig 6(a) we depict the performance of the proposed framework in case of dropout. It is evident that the proposed framework performs best when the dropout probability is 0.4. Therefore, for all the experiments (all baselines) and metrics we fix the dropout probability as stated. For learning rate we determined the value to be 0.0004 similarly as [12, 13, 45]. Also, we cross-verified by taking a range of 0.001 to 0.0001 to determine the learning rate in case of our proposed framework. As shown in Table 3 and Fig 6(b), the proposed framework performs best when the learning rate is fixed at 0.0004 in a similar manner as the existing literature.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Analysis on proposed framework for learning rate and dropout.

https://doi.org/10.1371/journal.pone.0241271.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Analysis of different network parameters on the proposed framework.

https://doi.org/10.1371/journal.pone.0241271.g006

For the politeness classifier, we had fixed the LSTM layers at 200 by evaluating the performance of the classifier for the responses generated by our proposed framework. We checked the performance of the classifier by using different layers of LSTM as shown in Table 4 of the revised manuscript. From the Fig 6(c), it is evident that by increasing the number of LSTM layers the performance of the classifier improved with the maximum accuracy being 0.84 in case of 200 layers. On increasing the number of layers the performance declined and eventually became constant with accuracy lesser than the best performing classifier at 200 layers. This analysis helped in deciding the number of lSTM layers for the classifier.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Analysis of LSTM layers on politeness classifier for the proposed framework.

https://doi.org/10.1371/journal.pone.0241271.t004

6.4 Comparisons to the existing systems

In Table 2, we provide the results of our proposed framework in comparison to the existing state-of-the-art methods. For response generation in multi-modal systems, we compare our current work with the existing systems [8, 13]. For a fair comparison with the existing systems, the data used for experimentation should be similar in terms of its structure, genre as well as annotation in order to draw correct conclusions on the results obtained. Hence, we compare our proposed approach with these systems as the dataset is identical. Also, both the existing systems used for comparing with our current system focuses on the task of textual response generation in multi-modal dialogue systems. In [13], the authors employed a knowledge base for generating the textual responses while using both the textual and visual information. Similarly, in [8], the authors have exploited the position and attribute information in text and images for generating the textual responses. As it is evident from Table 2, our proposed system outperforms the existing systems significantly in terms of Perplexity and BLEU-4 metrics. By applying parallel co-attention and BLOCK fusion technique, our best performing baseline framework achieves an improvement of around 5% in BLEU score from both the existing approaches. This is mainly due to the fact that the parallel co-attention network simultaneously captures the essential information from the entire utterance possessing textual and visual knowledge. We also attain enhanced representation of the utterance by having improved interaction between the modalities with the help of BLOCK fusion technique. One of the purposes of the current research is to accomplish diverse response generation. Evaluation shows that by applying stochastic beam search, we observe an improvement of 2% in BLEU score (with k = 5) as opposed to [8] and [13]. The increase in BLEU score signifies that the responses are diverse and different from the ground-truth responses, thereby, helping in achieving the desired task. Also, the decrease in perplexity score testifies that the generated responses are grammatically correct and better in comparison to all the baselines and the existing frameworks. The other existing systems on multi-modal dialog systems focus on the tasks of response generation and image retrieval both, which does not come within the scope of our work. Moreover, the datasets used in these works are different and therefore correct conclusions cannot be drawn for fair comparisons.

It is to be noted that as this is the very first work that focuses on generating diverse responses in a multi-modal setup, hence we are unable to compare our work with any other systems with respect to the diversity. Since the primary goal of this research is to bring diverseness in responses, therefore we compare our proposed system with the state-of-the-art techniques [10] for generating diverse responses. In the existing methods the authors employed various objective functions, such as maximum mutual information (MMI), anti language models (anti-LM), MMI-bidi approach for making the responses diverse. We analyze the responses generated by these techniques on the MMD dataset to have a thorough comparison in terms of distinct-1 and distinct-2 metrics. From the table, it is visible that our proposed framework performs better than the existing approaches for the task of diverse response generation. In contrast to the existing system and the baseline approaches, our proposed framework is capable of generating responses that are diverse in nature, thereby increasing the interactive property of the responses.

6.5 Human evaluation results

In Table 5, we present the human evaluation results for the baseline as well as the proposed approaches. The fluency of the proposed model is better in comparison to both the baseline models. We observe a fluency score of 41.31 for the proposed approach with stochastic beam search decoder (k = 5) along with BLOCK fusion techniques and attention modules. Hence, the models are capable of generating fluent responses. Also, there is an improvement of 4.98 in the relevance score over the baseline model. In case of diversity, it is evident from the table that the stochastic beam search decoder with k = 5 has achieved a score of 74.32, thereby, proving the fact that the responses generated are interesting and diverse in comparison to the baselines. Finally, the politeness score of the proposed approach is also better than the baseline model. Hence, the human evaluation results conclude that the proposed method has been successful in generating fluent, relevant, diverse as well as polite responses in a multi-modal dialogue system. In Fig 7, we present the responses generated by the different models. As it is visible from the figure, the responses generated are quite diverse and completely fluent. Also, instead of dull and direct Yes and No responses, the model has learned to be more polite by appreciating and apologizing, whenever required, according to the conversational history.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Results of human evaluation on different models for the MMD dataset.

https://doi.org/10.1371/journal.pone.0241271.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Examples of responses generated by different models along with contextual information and input.

https://doi.org/10.1371/journal.pone.0241271.g007

6.6 Attention analysis

For the model to be capable of learning different attribute information, we incorporate the knowledge base (KB) information in our model in the similar way as was done in [13]. To obtain a more focused information from KB, we apply self-attention on the KB inputs. The attention visualization of the KB is given in Fig 8. From the figure, it is noticeable that the model can focus on the relevant attributes for better dialogue generation. In Example 1 from the figure, it is evident that eyeglasses of brand Vincent is given more focus as the brand is an important feature for the user. Similarly, in Example 2 the print of the shirt is a significant aspect for the selection of appropriate shirt for the user. From these examples, it is clear that the KB input facilitates in capturing the correct attribute of a product, thereby, enhancing the quality of the generated response by making it more coherent to the user needs. Different attributes, such as the brand, print, type, name of a product gets attended by the self-attention applied on the KB input which is in accordance to the dialogue context as observed by the attention visualization in Fig 8 assists the generation of the textual responses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Attention map for KB Input.

https://doi.org/10.1371/journal.pone.0241271.g008

For better interaction between the image and text, we employ parallel co-attention in our proposed framework as described in Section 3.3.1 of the Methodology section. For a complete understanding of the parallel co-attention and its effectiveness we provide an attention visualization of the parallel co-attention module in Fig 9. The attention on the text and the corresponding image is needed for better response generation. By employing parallel co-attention it can be observed from the attention visualization that the attention mechanism simultaneously focuses on the given text and the images, providing an enhanced representation to the decoder for the generation of better responses. From the figure, it is quite clear that the attributes and different images being discussed in text are also getting focused on the corresponding image. We thereby provide more information and better evidences to the decoder for generating the response. In the first example from the Figure, the 4^th image gets attended as it is being discussed in the textual utterance as well. The color and material of the image and the information in the textual part of the utterance, such as cotton simultaneously gets attended providing improved representation of the entire utterance having both textual and visual features. Similarly, in Example 2 from the set of 5 images the 2^nd image gets selected and the attributes such as leather, rubber material gets attended in the textual part of the utterance. From the figure showcasing the attention visualization, it can be concluded that the parallel co-attention mechanism assists in capturing the important and critical features of both the visual and textual part of the utterance simultaneously.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Attention map for parallel co-attention between the text and image.

https://doi.org/10.1371/journal.pone.0241271.g009

6.7 Error analysis

We closely analyze the outputs of the generated response so that we are aware of the errors made by the proposed dialogue generation framework. The common errors made by the model are:

• Additional Information: The model sometimes generates additional/extra information in case of attributes for a few products. For example, Gold: The material of the trousers is cotton., Predicted: The trousers have cotton polyester material with check patterns.
• Incorrect information: The model is incapable of presenting the correct celebrity information in most of the cases. For example, Gold: Celebrities cel_205, cel_2254 and cel_101 endorse this kind of shoes., Predicted: This kind of shoes is endorsed by celebrity cel_123.
• Erroneous image selection: The model is incompetent in selecting the images having contextual information of more than 5 turns, thereby generating incorrect responses in some cases. There are also the cases, where due to the discussion of multiple images in the conversational history, wrong images get selected, making the responses incorrect.