Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Multimodal representation learning for tourism recommendation with two-tower architecture

Abstract

Personalized recommendation plays an important role in many online service fields. In the field of tourism recommendation, tourist attractions contain rich context and content information. These implicit features include not only text, but also images and videos. In order to make better use of these features, researchers usually introduce richer feature information or more efficient feature representation methods, but the unrestricted introduction of a large amount of feature information will undoubtedly reduce the performance of the recommendation system. We propose a novel heterogeneous multimodal representation learning method for tourism recommendation. The proposed model is based on two-tower architecture, in which the item tower handles multimodal latent features: Bidirectional Long Short-Term Memory (Bi-LSTM) is used to extract the text features of items, and an External Attention Transformer (EANet) is used to extract image features of items, and connect these feature vectors with item IDs to enrich the feature representation of items. In order to increase the expressiveness of the model, we introduce a deep fully connected stack layer to fuse multimodal feature vectors and capture the hidden relationship between them. The model is tested on the three different datasets, our model is better than the baseline models in NDCG and precision.

Citation: Cui Y, Liang S, Zhang Y (2024) Multimodal representation learning for tourism recommendation with two-tower architecture. PLoS ONE 19(2): e0299370. https://doi.org/10.1371/journal.pone.0299370

Editor: Sathishkumar Veerappampalayam Easwaramoorthy, Sunway University, MALAYSIA

Received: November 21, 2023; Accepted: February 9, 2024; Published: February 23, 2024

Copyright: © 2024 Cui et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: We provide access to the data used in our experiments through the following links: Tourism dataset: https://github.com/haiyuyuyuy/-Tourism-dataset MovieLen dataset: https://www.kaggle.com/datasets/odedgolden/movielens-1m-dataset/data Grocery & Food dataset: https://nijianmo.github.io/amazon/index.html

Funding: The work was supported by the FDCT Funding Scheme for Postdoctoral Researchers of Higher Education Institutions, grant number 0003/2021/APD, under the supervision of Prof. Shengbin Liang.

Competing interests: The authors have declared that no competing interests exist.

1 Introduction

With the rapid development of the Internet and mobile devices, our daily activities connect deeply to online services, such as online shopping, online music and videos. Online services also lead to a surge in data volumes, with information overload making it harder to choose from multitude of services. Recommendation systems (RS) are a powerful information filter for guiding people to find the items of interest in a gigantic and rapidly expanding pool of candidates. Providing users with efficient and accurate prediction results is the goal of RSs. The core methods of RSs include collaborative filtering (CF) [1], content-based recommendation [2] and hybrid recommendation [3]. The principle of the RSs is based on the idea that things are clustered together and people are divided into groups, then recommend items that meet their preferences for users. However, two major challenges faced by existing RSs are the problems of data sparsity and cold start. Data sparsity means that most users interact with only a few items, similarly, many items interact with only a few users. Cold start refers to a new user or item with none interaction, it is difficult for RSs to identify similar candidates and recommend them. The challenging aspect of recommending travel experiences arises from the complexity of multimodal features. In contrast to traditional unimodal data, tourism recommendation tasks require the consideration of information from various perceptual channels, such as images, text, audio, etc., to comprehensively understand users’ needs and preferences. The fusion of this multimodal data introduces a series of technical challenges. In order to solve these problems, the core idea is to add auxiliary features of users and items to build the relationship between users and items.

Recently, the two-tower recommendation model [4–6] is famous for its high efficiency and is widely used in the industry. The notable feature of the two-tower recommendation model is that user and item are two independent sub-networks, one side is the user tower, another side is the item tower, and the parameters of these two towers are not shared. The user tower side covers the features of users, such as user ID, age, historical behavior sequence and so on; the item tower side includes the features of the items, such as item ID, item category, item ratings and so on. The basic workflow of the two-tower model is as follows:

  1. Step 1. the user features and item features are respectively input into the feature extraction network to obtain user embedding and item embedding.
  2. Step 2. calculate the distance between user embedding and item embedding, the items which the user choose are closer, the items which the user do not choose are farther.
  3. Step 3. use the loss function to update the parameters of the model.
  4. Step 4. get all the items embedding through the item tower and store them, then get the top-N candidates for the target user.

Due to the two-tower model suffers from the lack of information interaction between user tower and item tower, and uneven distribution of features in the two towers. Yu et al. [7] proposed a Dual Augmented Two-tower model (DAT), which integrated adaptive mimic mechanism and a category alignment loss. The DAT model effectively alleviates the problems of insufficient information interaction and low query efficiency. A large-scale corpus makes query inefficient. Yang et al. [5] proposed a method called Mixed Negative Sampling (MNS) which uses a mixture of batch and uniformly sampled negatives to address the selection bias of implicit user feedback, the model helps to improve the query speed of features in the recommendation process. Wang et al. [8] applied the two-tower model to the field of video recommendation and explored the possibility of video recommendation based on the two-tower model using different types of metadata. Shen et al. [9] proposed an Adversarial Two-tower Neural Network (ATNN) model for new arrivals CTR predictions. Xu et al. [10] establish some asymptotic results of the two-tower model in terms of its strong convergence to the optimal recommendation system, showing that it achieves a fast convergence rate depending on the intrinsic dimensions of inputs features.

By building an explicit representation on the user-item interactions, literature [11–13] use context information to enrich the representation of users or items, so as to alleviate the cold start problem. Literature [14–16] introduce graph neural network (GNN) into the recommendation model, which can iteratively propagate information from the interactive items, and update the user vector (similarly to item), and can enhance the user/item representation. In order to better incorporate information from heterogeneous interaction types, some solutions such as [17–21] have been proposed in recommendation systems. MBGCN [17] addresses the challenges of data sparsity and cold start issues in traditional recommendation models by constructing a unified graph structure representing various user-item behavioral interactions. It achieves this by learning behavioral intensity and capturing behavioral semantics. HG-GNN [18] is a heterogeneous global graph neural network model that leverages user-item interaction information to better infer user preferences from both current and historical sessions. SCHGN [19] is a self-supervised heat-aware heterogeneous graph network, enhancing food recommendations by considering relationships among ingredients and user preferences for calories. MAINT [20] employs multiple projection mechanisms to capture diverse user preferences and intentions. It utilizes behavior-enhanced LSTM and multi-faceted refined attention mechanisms to adaptively integrate user preferences and intentions, effectively addressing constraints in multi-behavior sequence recommendations. DMRL [21] is a novel recommendation approach that employs disentangled multimodal representation learning, capturing users’ attention to different modalities for each factor in user preference modeling, and outperforms existing methods in extensive evaluations on real-world datasets.

In the field of tourism recommendation, there are various data types for users to evaluate and describe tourism attractions, including traditional manners such as ratings, comments, as well as travel notes, images and short videos. Rational use of these data as features can not only alleviate the cold start problem, but also it is beneficial to improve the quality of recommendation results. Inspired by the above research methods, we propose a multimodal heterogeneous representation learning and apply it to the two-tower recommendation model. In summary, our contributions are:

A multimodal two-tower recommendation framework. We showcase how to apply heterogeneous data integrated into the framework to improve recommendation quality. In the item tower, according to different modal information describing items, we use Bi-LSTM [22] model to extract text features from text data, and EANet [23] model to extract image features from images and short videos, then combine them with item ID.

Heterogeneous representation learning. In the item tower, there is a large amount of heterogeneous information, and we adopt different strategies to extract features for multimodal information. We designed a deep fully connected stack layer to integrated the features of user tower and item tower, so that the feature latent vectors of each modality extracted are fully cross-fused to capture more deep relationship between user tower and item tower.

Offline and online experiments. We conduct extensive offline and online experiments on datasets of different industries and scales, such as tourism, catering and movies to demonstrate the effectiveness of our model. We select a variety of metrics for evaluation, and the experimental results show that our model has better performance. Due to the lack of interaction between the two towers in the early stage, online feature extraction at the user tower is very important. In addition to basic user information, this paper uses Hard Negative Mining (HNM) strategy for negative sampling to extract user features in the online stage, in order to obtain a better user profile.

The rest of this paper is structured as follows. Section 2 introduces the related work. Section 3 introduces the method and framework of our model. Section 4 conducts a variety of numerical experiments and ablation tests to demonstrate the advantage of the model. A brief summary is provided in Section 5.

2 Related work

2.1 Two-tower model

In the past decade, deep learning has achieved great success in recommendation systems, from using CNN, RNN, GNN to deep reinforcement learning [3]. The two-tower model has two independent sub-networks: user tower and item tower. The core idea is to cache the features of the two towers respectively, an early popular design of two-tower models is DSSM [24]. When online recommendation is performed, it only needs to perform similarity calculation in the memory, which has the characteristics of fast speed and high efficiency. ATNN [9] model predicts the CTR predictions of new arrival by introducing an adversarial network to a two-tower network. The classic DNN [25] model proposed by YouTube is the pioneering work of the two-tower model, and it also includes the recall and ranking. DSSM [24] is a two-tower model for computing semantic similarity, which is used to rank Web documents. Yi et al. [4]. proposed a two-tower model framework for building large scale content-aware retrieval models for industrial scale applications. MIM [26] is a two-tower deep learning model that implements free-form textual descriptive medication phrases inference into patient-friendly medication names. Qin et al. [27] introduced split learning into the two-tower recommendation models and proposed a model called STTFedRec, a privacy-preserving and efficient cross-device federated recommendation framework. MNS [5] used a mixture of batch and uniformly sampled negatives to tackle the selection bias of implicit user feedback. COMET [28] simultaneously utilizes high-order interaction information between historical interactions and embedding dimensions. It captures interaction signals from implicit feedback in the recommendation system and independently learns the representation of the target user and target item.

In summary, the item side of the two-tower model does not depend on the user side, and the massive item-side information uses offline training to improve the efficiency of the model. Existing methods mostly focus on how to improve the feature representation of item-side information, such as positive and negative sampling, to better describe the items. However, this requires a large amount of item information as support, and its effect is limited for small-scale datasets.

2.2 Heterogeneous information representation

Although collaborative filtering recommendation systems based on user ratings have better performance, these traditional methods are limited to pre-selected information sources or domain knowledge such as classification standards, descriptions, sentiment analysis and so on, but the recommendation results of such recommendation algorithms lack personalization and they are not universal for some specific application fields. Therefore, researchers develop different recommendation models for user-item interaction in different fields. The recent promising progress in representation learning [29, 30] clarify this problem. With well-established representation learning theories and graph theory on texts [31], images [32], audios [30], videos [33] and many other aspects, we can conduct joint representation learning on heterogeneous information sources in the shared space to obtain more intelligent recommendations. It has an important effect on the rough ranking stage of the recommendation process. Furthermore, by incorporating time sequence relationship [34], context information [35], sentiment analysis [36] and knowledge graph [37] into the ranking stage of recommendation results, we can further optimize the recommendation order. IRGPR [38] model is an innovative re-ranking approach for large-scale commercial recommender systems, utilizing a heterogeneous graph and graph neural network to explicitly model transitive item relationships and personalized user intents, thereby improving the accuracy of re-ranking results compared to existing strategies. RAISE [39] is an advanced re-ranking model that achieves user-specific predictions based on individual intentions, demonstrating superior performance with significant improvements in Precision, MAP, and NDCG on four public datasets. By building representation learning on top of pair-wise learning to rank techniques, we can achieve highly promoted top-N recommendation quality, which is closely related to the business value in the real-world recommendation systems.

In terms of multimodal heterogeneous information representation, the above research has achieved a lot of results. This paper focuses on the efficient representation of implicit features of heterogeneous information such as text, images, and ratings in the item tower, so as to describe a richer description of the items.

3 Methodology

In this section, we first give the mathematical definition of a recommendation system based on two-tower architecture, and then construct a feature selection strategy to select candidate items from a high-dimensional, huge corpus; next, we introduce the similarity calculation method between item tower and user tower, and finally, introduce how to train the model.

3.1 The architecture of two-tower recommendation model

The retrieval task in recommendation systems aims to quickly select hundreds to thousands of candidate items from the entire item corpus given a certain query. Given a typical recommendation system with user covariates and item covariates , the two-tower model can be written as (1) where f: and are two deep neural networks mapping xu and into the same p-dementional embedded space. The recommendation mechanism of the two-tower model is based on the dot product of f(xu) and . Finally, we use the sigmoid function to output the result, as illustrated in Fig 1.

thumbnail
Download:
Fig 1. The network architecture of our proposed two-tower DNN model.

https://doi.org/10.1371/journal.pone.0299370.g001

In the tourism recommendation system, the number of features of item tower is much larger than that of user tower. At the same time, the data structure of item tower is more diverse. Many tourism attractions contain a large number of user comments, posted images and short videos. The information basically expresses tourists’ subjective impression of the tourism attractions.

3.1.1 Item tower.

The data on the item tower side is multimodal, large in quantity, and extremely complex in structure. We use a variety of neural networks to deal with heterogeneous information, such as text, image and video and structured data.

3.1.1.1 Text feature. LSTM as a deep learning model capable of capturing and understanding sequential data, has demonstrated outstanding performance in recommendation systems. Due to its effectiveness in handling users’ time-series behavioral data, such as click history and purchase records, LSTM excels in capturing both long-term and short-term changes in user interests [40]. By learning temporal correlations and sequence patterns, LSTM can more accurately predict users’ future behavior, thereby enhancing the precision and personalization levels of recommendation systems [3]. Additionally, applying LSTM to user sentiment analysis allows the extraction of user preferences from their reviews, further enriching user features [41]. The traditional recurrent neural network model cannot capture long-distance semantic connection, even if it can transfer semantic information between words. In the process of training, the gradient decreases gradually until it disappears. As a result, the length of sequential data is limited. LSTM overcomes the problem of gradient disappearance by introducing input gate, output gate, forget gate and memory cell. The LSTM network structure is shown in Fig 2.

thumbnail
Download:
Fig 2. The memory cell structure of LSTM.

https://doi.org/10.1371/journal.pone.0299370.g002

A LSTM memory cell includes three gates: input gate it, forget gate ft and output gate ot. While ct is cell state representing long-term memory, memory ht representing short-term memory, and candidate state waiting to be stored in long-term memory. At time step t, the input gate and the tanh function are used to control the input of new text information at this time, and its value is: (2) where Wi is the weight matrix of the input gate, bi is the bias term of the input gate, xt is the input feature at the current time step and σ is the sigmoid activation function.

The forget gate is the key component of LSTM, which controls the retention and forgetting of memory in the cellular state, and thus avoid the problem of gradient disappearance caused by gradient over time. It is a function of the short-term memory ht-1 of the previous moment and the input xt of the current moment. And the forget gate is multiplied by the cell state ct-1 of the previous moment to indicate the memory retained from the previous moment. It combines the input gate it and the candidate state to update the cell state to ct, the formula is as follows: (3)

The output gate ot is used to control the final output of the entire LSTM, which is combined with the cell state ct at time t to generate the output result of the memory ht at the current time: (4)

Bi-LSTM can solve the problem that LSTM cannot calculate reverse sequence context information, and can combine forward sequence and reverse sequence to output as Eq 5.

(5)

Among them, and are the results of the forward and reverse hidden layers of Bi-LSTM respectively, Bi-LSTM is calculated and updated in two directions, both sequences are directly connected to the output layer, providing complete contextual state for each word.

In the recommendation model, we design a two-layer Bi-LSTM for processing comments, in the first layer of Bi-LSTM, we output ht at each time step (return_sequences = True), and connect it as the input of the second Bi-LSTM layer. For example, the text such as “景点的环境非常好” (The environment of the attraction is very good). Firstly, the Chinese comments are segmented, then these words are converted into word embedding. Secondly, these word embeddings are sequentially input into the Bi-LSTM model to obtain the output vector rt. Finally, we send rt to a fully connected layer with 32 neurons and convert it into a one-dimensional vector of length 32, which fits the text features required for the recommendation model. The process is shown in Fig 3.

thumbnail
Download:
Fig 3. The workflow of text feature processing.

https://doi.org/10.1371/journal.pone.0299370.g003

3.1.1.2 Image and short video feature. We use EANet to extract features from images and videos. Since self-attention ignores the potential correlation between different samples, EANet proposes a new external attention mechanism to replace self-attention, which is based on two external, small-scale, learnable, shared memory. While external attention implicitly considers the correlations between samples, which helps us to mine the potential preferences of users. EANet further incorporates multi-head attention into the external attention mechanism, and has achieved very good results in tasks such as computer vision. We design an external attention MLP framework for extracting image features describing tourism attractions, and integrates it into our two-tower recommendation model.

The external attention mechanism is shown as Fig 4, We first calculate the attention map by computing the affinities between the self-query vectors and an external learnable key memory, and then produce a refined feature map by multiplying this attention map by another external learnable value memory. They are independent of individual samples and shared across the entire dataset, which plays a strong regularization role and improves the generalization capability of the attention mechanism. The external memories are designed to learn the most discriminative features across the whole dataset, capturing the most informative parts, as well as excluding interfering information from other samples. The structure of external attention mechanism is simple, but it is very effective for various computer vision tasks. Because of its simplicity, the TTHM model proposed in this paper integrates it to obtain the features contained in the images’ description of tourism attractions.

thumbnail
Download:
Fig 4. External-attention.

https://doi.org/10.1371/journal.pone.0299370.g004

External attention utilizes Eq 6 to compute the attention map between the input pixels and an external memory unit , (6)

Among them, (a)i,j is the similarity between the ith pixel and the jth row of M, where M is a learnable parameter independent of the input, which acts as a memory of the whole training dataset. A is the attention map inferred from this learned dataset-level prior knowledge. Finally, we update the input features from M by the similarities in A. To enhance the network’s capability, we use two different memory units Mk and Mv, as the key and value.

The attention map is sensitive to the scale of the input features. External attention does not use the softmax function to normalize the attention map but utilizes double normalization [42] to normalize the columns and rows. The double normalization is described as Eq 7, (7)

In Transformer [43], self-attention is computed many times on different input channels, which is called multi-head attention. Multi-head attention can capture different relations between tokens, improving upon the capacity of single head attention, external attention uses a similar method, the structure of external attention is shown as Fig 5.

thumbnail
Download:
Fig 5. Multi-head external attention.

https://doi.org/10.1371/journal.pone.0299370.g005

Multi-head external attention descripted as Eq 8, (8) where hi is the ith head, H is the number of heads and WO is a linear transformation matrix making the dimensions of input and output consistent. and are the shared memory units for different heads. For image data, the basic flow of its processing is as follows:

  1. Step 1. The image data is converted into a 32*32*3 gray scale image through the input layer;
  2. Step 2. Then gray scale images go through a patch extract layer and a patch embedding layer to extract the key features of the tourism attractions;
  3. Step 3. The initially extracted features are passed through the EANet model described above to obtain detailed image features, which contain the potential correlation between all samples;
  4. Step 4. Output the final result through a fully connected layer with 32 neurons, which is a one-dimensional vector of length 32.

3.1.1.3 Structured data. In addition to the above features, other features on the item tower side are structured data, our model use embedding layer convert them into embedded vectors of low dimension with a fixed size. Suppose there are n structured features at the item tower side, Kj is the dimension of the embedding layer corresponding to the jth feature. For each feature fi of , map to a low-dimensional dense layer through the corresponding embedded vector , then all the embedded vectors at the item tower side E∈ℝL, where L is the dimension of E, that is, .

Finally, In the item tower side, we perform the full connection operation on the BI-LSTM and EANet output results, and the embedded vectors generated by the structured data to obtain in Eq 1.

3.1.2 User tower.

The main features of the user tower include user ID and user profile. These features are basically structured data, so use the embedding layer to convert them into fixed dimensional embedded vectors. Then we use the full connection operation to concatenate the embedded vectors to generate user covariates xu in Eq 1.

3.1.3 Deep fully connected stack layer.

We perform the corresponding concatenation operation on all the embedded vectors at both sides of the user tower and item tower, and obtain two fusion vectors zu and zi. They are calculated as in Eq 9. (9) where || represents the vector concatenation operation. In this way, the final fusion vector contains various of multimodal features.

After getting zu and zi, we feed them into the corresponding two towers, which are stacked by gradually narrowing fully connected layer. We use ReLU activation function for the fully connected layer. Finally, the output result of the fully connected layer will pass through L2 regularization layer to obtain the fusion vectors pu and pi with fully crossed features, its structure is shown in Fig 6.

thumbnail
Download:
Fig 6. The structure of fully connected layer.

https://doi.org/10.1371/journal.pone.0299370.g006

We use Eq 10 to describe the operation of the fully connected layer, (10) Where dj denote the jth fully connected operation, wj and bj are the weight and bias vector of the jth fully connected layer, zu,i represents zu and zi, pu,i represents the output vector of the L2 regularization layer.

Finally, our model calculates the user-item similarity using the dot product as shown in Eq 11, where zi and zu represent the embedding of the item and user respectively.

(11)

3.2 Metadata

In this section, we introduce three types of metadata for the tourism recommendation: category feature, text feature and image feature. Category feature is a subset of metadata that generally listed by online tourism server providers, which are classifications of different types of tourism attractions. We first represent each categorical feature with a one-hot code. The categorical features mainly include the following five types:

  • Ticket price is the primary concern of users, since the ticket prices of various tourism attractions are different, if each price is considered separately, the feature coding will be very lengthy. In order to simplify this process, we divide the ticket prices into four levels: free, cheap, medium and expensive.
  • The recommended tour duration is the amount of time users need to visit the attractions, and we divide it into ten levels at hourly intervals.
  • Opening hours indicate the opening hours of attractions, and we divide them into three levels: short, medium and long.
  • Ratings indicate the overall evaluation of the attraction by all tourists, divided into five grades from 1 to 5 points.
  • The popularity of attractions represents the number of tourists in a fixed period, we set the popularity to five levels according to the number of tourists.

After we get the one-hot representation of each categorical feature, we concatenate it into a binary vector, which is then converted into a dense vector through an embedding layer.

The text features are mainly commenting on attractions in the online community, and various comments on attractions from the user’s point of view include service level, experience, etc. Since the text lengths of these corpora are different, and most of them are Chinese, we firstly perform word segmentation with Chinese word segmentation tools; then encode each word, in order to avoid the long tail problem, we keep the 20,000 most frequent words. They are converted into a 64-dimensional dense vector by embedding; finally, we use a two-layer Bi-LSTM to extract a 32-dimensional vector from the embedding layer.

Image features are very important way of describing tourist attractions. Images can increase people’s cognition of tourism attractions and show the characteristics of attractions. We extract relevant attractions from the user-item interaction history, and use the EANet model to extract image feature to 32-dimensional vectors. The feature vector that distinguishes different types of attractions from the input images, by comparing from different images of the same type of attractions are very close in the feature space, and different types of attractions are far apart in the feature space. The type of attractions is determined through feature extraction and comparison.

3.3 Model training

We introduce the structure of the model and the fusion and representation of multimodal features. In the recall phase of the recommendation system, in most cases, there are usually close semantic associations between candidates, and it is necessary to match the text at a finer attribute granularity. Appropriately increasing the difficulty of strong negative examples in the training data will help improve the model effect. The general practice is to sample from a ranked candidate paragraph, and the higher the negative example, the more difficult it is for the model. However, due to the unavoidable missing labeling situation, direct sampling has a high probability of introducing false negatives. We use the HNM strategy for negative sampling. In the model training stage, in addition to the positive samples, the negative sampling method is used to incorporate the same data that is mistaken as a negative sample into the training set.

  1. Step 1. Using Eq 12, determine the negative sample, that is, the sample whose predicted value differs greatly from the label.
  2. Step 2. Based on the logits, determine the number of negative sampling samples, our model sets the number of negative sampling samples as S.
  3. Step 3. Calculate the total number of samples T in the training set according to Eq 13.
  4. Step 4. Model calculation, and output recommendation results.
(12)(13)

where D is the number of positive feedback query-item pairs, T is the total number of train pairs, and σ represents the sigmoid function.

4 Experimental results

We perform experiments on one public dataset and one industrial dataset to fully evaluate our model. Furthermore, we conduct extensive ablation test on each proposed component. We also provide a visualization comparison with five other models.

4.1 Datasets

To evaluate the effectiveness of our model, we conduct extensive experiments on three different datasets: Tourism, MovieLens and Grocery & Food. The Tourism is a dataset about tourism topics and its data is collected from mafengwo.cn. These datasets are tourism-related datasets, and their data scale and number of interactions are quite different, including both Chinese and English corpora, which can more fully reflect the performance of our model. Table 1 summarizes the statistics of all the datasets.

thumbnail
Download:
Table 1. Statistics of evaluation datasets.

https://doi.org/10.1371/journal.pone.0299370.t001

4.2 Evaluation metrics

For all experiments, we evaluate our model and baseline in terms of Precision@k, Recall@k, F1-value@k and NDCG@k, we report with k = 5 and 10.

  • Precision@k: The metric represents the accuracy of the recommendation system, that is the proportion of relevant results that were correctly predicted out of all the results returned. The metric is often used to measure the precision of the recommendation system. It is computed as:
(14)

True positive (TP) stands for true examples, false positive (FP) stands for false positive example, False negative (FN) stands for false counterexample. The value of Precision @k range is [0,1], the bigger the better.

  • Recall@k: The metric represents the proportion of the number of samples predicted to be positive in all positive samples. It is useful for evaluating the performancSe of recommendation systems and can be used to make improvements to the system. By optimizing the system to increase the number of relevant items recommended, the overall user experience can be improved. Additionally, the metric can be used to compare the performance of different recommendation systems and determine which one is more effective. Is can be expressed as:
(15)
  • F1@k: The metric is the harmonic mean of Precison@k and Recall@k. It used to evaluate how many relevant items have been retrieved among the top k results returned by the model. Essentially, it measures the proportion of relevant items that were retrieved in the top k results. It can be expressed as:
(16)
  • NDCG@k: The Normalized Discounted Cumulative Gain (NDCG) is one the most frequently used evaluation measure, which considers the position of correctly recommended items. NDCG is averaged across all testing users. It is computed as:
(17)

where reli refers to the true relevance score of the ith result. |REL| indicates that the results are sorted according to the true correlation from large to small, and the number of sets consisting of the first k results is taken.

4.3 Baseline algorithms

To demonstrate the effectiveness, we compare and analyze our proposed TTHM model with the following baselines. Among these models, there are recommendation systems based on Bayesian methods, collaborative filtering algorithm based on neural network method, heterogeneous information recommendation models based on two-tower architecture, and recommendation system based on graph neural networks. They are state-of-the-art methods among different recommendation methods.

4.3.1 BPR [44].

Bayesian Personalized Ranking (BPR) is an approach for personal recommendation, it generates a recommendation list based on implicit feedback. It assumes that different behavior reflects different order preference between user and item, and this can be used as prior knowledge to build more diverse training pairs.

4.3.2 NCF [45].

Neural Collaborative Filtering (NCF) is a classical deep neural network model for RS, NCF is a general framework that can model user-item interactions in different ways. NCF complements the mainstream shallow models for collaborative filtering, NCF proposes to leverage a multi-layer perceptron to learn the user–item interaction function.

4.3.3 HERec [46].

HERec is designed to solve the problem of heterogeneous information network, it includes a meta-path based random walk strategy to generate meaningful node sequences for network embedding. The learned node embeddings are first transformed by a set of fusion functions, and subsequently integrated into an extended matrix factorization (MF) model. The extended MF model together with fusion functions are jointly optimized for the rating prediction task.

4.3.4 MoHINRec [47].

MoHINRec is a motif-enhanced meta-path, which further captures the high-order relationship between nodes of the same type, and then inputs the embedding representation into the factorization machine for training.

4.3.5 ATTR [48].

ATTR is a neural sequence recommendation model designed for scenic spots, employing a Self-Attention mechanism to capture sequence representations and model multiple relationships within items. This individualized Tourism Recommendation system analyzes user interactions through self-attention, capturing both long- and short-term preferences. The model enhances item embedding by preserving the relationship structure between scenic items, enabling accurate analysis of user interests for effective item prediction.

4.3.6 TMFUN [49].

TMFUN is a novel and effective Attention-guided Multi-step Fusion Network designed for multimodal recommendation. The main objective of this model is to leverage the rational utilization of item multimodal information for enhanced recommendation performance. In contrast to previous approaches that directly integrate multimodal features with item ID embeddings, TMFUN focuses on preserving inherent semantic relations within multimodal features. The model constructs modality feature graphs and item feature graphs to capture latent item-item semantic structures.

4.3.7 SGFD [50].

SGFD is a model-agnostic approach, Semantic-guided Feature Distillation, devised to address challenges in multimodal recommendation. It employs a teacher-student framework to enhance feature extraction. The teacher model extracts rich modality features by considering semantic information and complementary details from multiple modalities. In this paper, we use SGFD-GRCN as the baseline.

4.4 Parameter settings

We optimize all models using the Adam optimizer, and the experimental hyperparameter settings in our model are shown in Table 2.

thumbnail
Download:
Table 2. Parameter settings.

https://doi.org/10.1371/journal.pone.0299370.t002

4.5 Result and discussion

Tables 3–5 report the overall performance compared with baselines. Each result is the average performance from 10 runs with random weight initializations.

thumbnail
Download:
Table 3. Experimental results on Tourism dataset.

https://doi.org/10.1371/journal.pone.0299370.t003

thumbnail
Download:
Table 4. Experimental results on MovieLens dataset.

https://doi.org/10.1371/journal.pone.0299370.t004

thumbnail
Download:
Table 5. Experimental results on Grocery & Food dataset.

https://doi.org/10.1371/journal.pone.0299370.t005

Table 3 show the experimental results on Tourism dataset. Tourism dataset is a Chinese corpus with relatively few interactions. The experiment results show that our model outperforms all baseline models in terms of most classification indicators such as precision, F1 and ranking indicators NDCG.

Table 4 show the experimental results on MovieLens dataset. The dataset is an English corpus with a greater number of interactions. The experimental results show that our proposed TTHM model outperforms the baseline models in both F1 and Recall. It indicates that TTHM model performs better on medium-scale dataset.

Table 5 show the experimental results on Grocery & Food dataset. Grocery & Food dataset is the biggest among the three datasets. Due to Grocery & Food dataset has a large number of category features compared to the previous two datasets, for example, each food has a large number of ingredient lists. TTHM model focuses on text features and image features, and only uses a simple embedding layer for classification features to convert them into low-dimensional embedding vectors. It makes the model not fully extract and integrate a large number of classification features. It shows that TTHM model has some limitations, that is, the representation of categorical feature embedding with many features needs to be improved.

4.6 Ablation studies

We conduct ablation studies on Tourism and MovieLens dataset to justify its effectiveness. As a heterogeneous two-tower recommendation model, TTHM introduces different categories and quantities of data, and its recommendation results are also different. In this ablation tests, firstly, we only added categories as feature to the model and used it as a benchmark. Secondly, we add heterogeneous data such as texts and images to the neural network of the item tower, then fused the features to get the recommendation results respectively. Finally, further modify the model structure and establish a fully connected stack layer between the user tower and the item tower to recommend heterogeneous data. At different stages, the accuracy of different structural models is significantly different. The experimental results in the three datasets are shown in Tables 6–8, Connect denote concatenate all types of metadata directly, DDS means to fuse different types of metadata using fully connected layer.

thumbnail
Download:
Table 6. Ablation tests results on MovieLens dataset.

https://doi.org/10.1371/journal.pone.0299370.t006

thumbnail
Download:
Table 7. Ablation tests results on Tourism dataset.

https://doi.org/10.1371/journal.pone.0299370.t007

thumbnail
Download:
Table 8. Ablation tests results on Grocery & Food dataset.

https://doi.org/10.1371/journal.pone.0299370.t008

The experimental results are shown in Figs 7–9. Compared with the basic features, multimodality features are gradually integrated with the introduction of various features, the accuracy of the model continues to improve. But the importance of each feature is different, as we can see, for the Tourism dataset, the text features are significantly better than the image features, but the opposite is true for the MovieLens dataset. Compared to simply concatenating individual features, the fully-connected stack layers can bring significant improvements to the model, suggesting that the introduction of fully-connected stack layers is beneficial.

thumbnail
Download:
Fig 7. Ablation tests on Tourism dataset.

https://doi.org/10.1371/journal.pone.0299370.g007

thumbnail
Download:
Fig 8. Ablation tests on MovieLens dataset.

https://doi.org/10.1371/journal.pone.0299370.g008

thumbnail
Download:
Fig 9. Ablation tests on Grocery & Food dataset.

https://doi.org/10.1371/journal.pone.0299370.g009

5 Conclusion

In this paper we have presented a novel heterogeneous two-tower recommendation model TTHM with two key components: feature fusion method for heterogeneous data and fully connected stack layer in two-tower framework. We discussed different heterogeneous data such as texts, images, videos into feature fusion. Compared to the baseline models, the overall performance of TTHM is good, especially in the terms of NDCG, which is better than the baseline models in the three datasets, indicating that our model is better in ranking. At the same time, there are certain advantages in Recall, Precision and F1, especially for small and medium-scale datasets. We will solve the problem of classification performance on large-scale dataset in the future.

For future work, we also plan to conduct experiment on online systems with A/B testing to evaluate the recommendation performance. We also plan to improve the speed of model training, so that the model has better training efficiency when it is compatible with more features.

References

  1. 1. Chen J., Zhao C. and Chen L.. (2020). Collaborative filtering recommendation algorithm based on user correlation and evolutionary clustering. Complex & Intelligent Systems, 6, 147–156.
  2. 2. Javed U., Shaukat K., Hameed I. A., Iqbal F., Alam T. M. & Luo S.. (2021). A review of content-based and context-based recommendation systems, International Journal of Emerging Technologies in Learning (iJET),16(3), 274–306.
  3. 3. Liang S., Jin J., Ren J., Du W. & Qu S.. (2023). An Improved Dual-Channel Deep Q-Network Model for Tourism Recommendation. Big Data, 11(4), 268–281. pmid:36930787
  4. 4. Yi X., Yang J., Hong L., Cheng D. Z., Heldt L., Kumthekar A. & Chi E.. (2019). Sampling-bias-corrected neural modeling for large corpus item recommendations. Proceedings of the 13th ACM Conference on Recommender Systems, pp. 269–277.
  5. 5. Yang J., Yi X., Cheng D. Z., Hong L., Li Y., Wang S. X., et al. (2020). Mixed negative sampling for learning two-tower neural networks in recommendations. Companion Proceedings of the Web Conference 2020, pp. 441–447.
  6. 6. Wang J., Zhu J. & He X.. (2021). Cross-batch negative sampling for training two-tower recommenders. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1632–1636.
  7. 7. Yu Y., Wang W., Feng Z. & Xue D.. (2021). A dual augmented two-tower model for online large-scale recommendation. DLP-KDD.
  8. 8. Wang J., Yessenalina A. & Roshan-Ghias A.. (2021). Exploring heterogeneous metadata for video recommendation with two-tower model. arXiv preprint arXiv:2109.11059.
  9. 9. S. Xin. E. N. H, Z. Li, P. Zou, C. Long, J. Zhang, J. Bu et al. (2021). ATNN: adversarial two-tower neural network for new item’s popularity prediction in E-commerce. 2021 IEEE 37th International Conference on Data Engineering, IEEE, pp. 2499–2510.
    • 10. XU S. & Wang J.. (2022). On strong convergence of the two-tower model for recommender system. ICLR 2022 Conference.
    • 11. Wu L., Quan C., Li C., Wang Q., Zheng B. & Luo X.. (2019). A context-aware user-item representation learning for item recommendation. ACM Transactions on Information Systems, 37(2), 1–29.
    • 12. Xin X., Chen B., He X., Wang D., Ding Y. & Jose J. M.. (2019). CFM: Convolutional Factorization Machines for Context-Aware Recommendation. IJCAI, pp. 3926–3932.
    • 13. Jeong J. H, Lee K. H. & Kim M. H.. (2021). DGC: Dynamic group behavior modeling that utilizes context information for group recommendation. Knowledge-Based System, 213.
    • 14. Fan W., Ma Y., Li Q., He Y., Zhao E., Tang J. & Yin D.. (2019). Graph neural networks for social recommendation. The world wide web conference, pp. 417–426.
    • 15. Wu S., Tang Y., Zhu Y., Wang L., Xie X., & Tan T.. (2019). Session-based recommendation with graph neural networks. Proceedings of the AAAI conference on artificial intelligence, pp. 346–353.
    • 16. Chang J., Gao C., Zheng Y., Hui Y., Niu Y., Song Y., Jin D. & Li Y.. (2021). Sequential recommendation with graph neural networks. Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 378–387.
    • 17. Jin B.., Gao C., He X., Jin D. & Li Y.. (2020). Multi-behavior recommendation with graph convolutional networks. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 378–387.
    • 18. Pang Y., Wu L., Shen Q., Zhang Y., Wei Z., Xu F., et al. (2022). Heterogeneous global graph neural networks for personalized session-based recommendation. Proceedings of the fifteenth ACM international conference on web search and data mining, pp. 775–783.
    • 19. Song Y., Yang X., & Xu C.. (2023). Self-supervised calorie-aware heterogeneous graph networks for food recommendation. ACM Transactions on Multimedia Computing, Communications and Applications, 19(1s),1–23.
    • 20. Liu H., Ding J., Zhu Y., Tang F., Yu J., Jiang R., et al. (2023). Modeling multi-aspect preferences and intents for multi-behavioral sequential recommendation. Knowledge-Based Systems, 280, 111013.
    • 21. Liu F., Chen H., Cheng Z., Liu A., Nie L. & Kankanhalli M.. (2022). Disentangled multimodal representation learning for recommendation. IEEE Transactions on Multimedia, 25, 7149–7159.
    • 22. Huang Z., Xu W., & Yu K.. (2015). Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991.
    • 23. Guo M. H., Liu Z. N., Mu T. J. & Hu S. M.. (2022). Beyond self-attention: External attention using two linear layers for visual tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5), 5436–5447.
    • 24. Huang P. S., He X., Gao J., Deng L., Acero A. & Heck L.. (2013). Learning deep structured semantic models for web search using clickthrough data. Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333–2338.
    • 25. Covington P., Adams J. & Sargin E.. (2016). Deep neural networks for youtube recommendations. Proceedings of the 10th ACM conference on recommender systems, pp. 191–198.
    • 26. Yuan S., Bhatia P., Celikkaya B., Liu H. & Choi K.. (2021). Towards User Friendly Medication Mapping Using Entity-Boosted Two-Tower Neural Network. In International Workshop on Deep Learning for Human Activity Recognition, pp. 126–138.
    • 27. Qin J., Liu B., Zhang X. & Qian J.. (2022). Split Two-Tower Model for Efficient and Privacy-Preserving Cross-device Federated Recommendation. arXiv preprint arXiv:2206.13715.
    • 28. Lin Z., Feng L., Guo X., Zhang Y., Yin R., Kwoh C. K., et al. (2023). Comet: Convolutional dimension interaction for collaborative filtering. ACM Transactions on Intelligent Systems and Technology, 14(4), 1–18.
    • 29. Zhang Y., Ai Q., Chen X. & Croft W. B.. (2017). Joint representation learning for top-n recommendation with heterogeneous information sources. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1449–1458.
    • 30. Huang Z., Xu X., Ni J., Zhu H. & Wang C.. (2019). Multimodal representation learning for recommendation in Internet of Things. IEEE Internet of Things Journal, 6(6),10675–10685.
    • 31. Liu H., Wang Y., Peng Q., Wu F., Gan L., Pan L., et al. (2020). Hybrid neural recommendation with joint deep representation learning of ratings and reviews. Neurocomputing, 374, 77–85.
    • 32. Kang W. C., Fang C., Wang Z. & McAuley J.. (2017). Visually-aware fashion recommendation and design with generative image models. 2017 IEEE international conference on data mining (ICDM), IEEE, pp. 207–216.
    • 33. Du X., Yin H., Chen L., Wang Y., Yang Y. & Zhou X.. (2018). Personalized video recommendation using rich contents from videos. IEEE Transactions on Knowledge and Data Engineering, 32(3), 492–505.
    • 34. Mizrachi S. & Pavel L.. (2019). Combining Context Features in Sequence-Aware Recommender Systems. RecSys (Late-Breaking Results), pp. 11–15.
    • 35. Hu B., Shi C., Zhao W. X. & Yu P. S.. (2018). Leveraging meta-path based context for top-n recommendation with a neural co-attention model. Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 1531–1540.
    • 36. Dang C. N., Moreno-García M. N. & Prieta F. D. L.. (2021). An approach to integrating sentiment analysis into recommender systems, Sensors, 21(16).
    • 37. Wang X., He X., Cao Y., Liu M. & Chua T. S.. (2019). Kgat: Knowledge graph attention network for recommendation. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 950–958.
    • 38. Liu W., Liu Q., Tang R., Chen J., He X. & Heng P. A.. (2020). Personalized Re-ranking with Item Relationships for E-commerce. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 925–934.
    • 39. Lin Z., Zang S., Wang R., Sun Z., Senthilnath J., Xu C., et al. (2022). Attention over self-attention: Intention-aware re-ranking with dynamic transformer encoders for recommendation. IEEE Transactions on Knowledge and Data Engineering, 35(8), 7282–7295.
    • 40. Zhu G., Wang Y., Cao J., Bu Z., Yang S., Liang W., et al. (2021). Neural attentive travel package recommendation via exploiting long-term and short-term behaviors. Knowledge-Based Systems, 211, 106511.
    • 41. An H. W. & Moon N.. (2022). Design of recommendation system for tourist spot using sentiment analysis based on CNN-LSTM. Journal of Ambient Intelligence and Humanized Computing, 13, 1–11.
    • 42. Guo M. H., Cai J. X., Liu Z. N., Mu T. J., Martin R. R. & Hu S. M.. (2021). Pct: Point cloud transformer. Computational Visual Media, 7,187–199.
    • 43. Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N. & Polosukhin . (2017). Attention is all you need. Advances in neural information processing systems.
    • 44. Rendle S., Freudenthaler C., Gantner Z. & Schmidt-Thieme L.. (2012). BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.
    • 45. He X., Liao L., Zhang H., Nie L., Hu X. & Chua T. S.. (2017). Neural collaborative filtering. Proceedings of the 26th international conference on world wide web, pp. 173–182.
    • 46. Shi C., Hu B., Zhao W. X. & Philip S. Y.. (2018). Heterogeneous information network embedding for recommendation. IEEE Transactions on Knowledge and Data Engineering, 31(2),357–370.
    • 47. Zhao H., Zhou Y., Song Y. & Lee D. L.. (2019). Motif enhanced recommendation over heterogeneous information network. Proceedings of the 28th ACM international conference on information and knowledge management, pp. 2189–2192.
    • 48. Liu G., Ma X., Zhu J., Zhang Y., Yang D., Wang J., et al. (2022). Individualized tourism recommendation based on self-attention. Plos one, 17(8), e0272319. pmid:36006968
    • 49. Zhou Y., Guo J., Sun H., Song B. & Yu F. R.. (2023). Attention-guided Multi-step Fusion: A Hierarchical Fusion Network for Multimodal Recommendation. arXiv preprint arXiv:2304.11979.
    • 50. Liu F., Chen H., Cheng Z., Nie L. & Kankanhalli M.. (2023). Semantic-Guided Feature Distillation for Multimodal Recommendation. Proceedings of the 31st ACM International Conference on Multimedia, pp. 6567–6575.