2.1 Consumer behavior analysis based on machine learning methods

Sılahtaroğlu et al. [8] gathered data on consumers’ past consumption behaviors alongside statistical insights, employing neural networks and decision trees to prognosticate the likelihood of consumer purchases. Webb et al. [9] devised a model delineating customers’ purchasing behaviors, addressing four pivotal facets: data preparation, data labeling, concept transfer, and intricate calculations to forecast consumer purchasing patterns based on these aspects. ZUO et al. [10] scrutinized supermarket data, demonstrating superior results with the support vector machine model through comparative analyses. Chang et al. [11] established a model akin to traditional statistical methods to predict purchase probabilities. Guadagni et al. [12] assembled a cohort of one hundred purchasers of generic items, utilizing logistic regression to gauge brand loyalty, thereby predicting sales figures. Cui et al. [13] employed support vector machines in conjunction with multinomial logistic regression for predictive analysis, offering a framework that contrasts support vector machines with contemporary modeling techniques, delineating their strengths and limitations. Potharst [14] harnessed neural network algorithms to construct a predictive model for repeat purchasing behaviors, leveraging the algorithm’s adeptness in accommodating nonlinear feature variables. Baesens et al. [15] employed Bayesian neural network algorithms to mitigate data overfitting issues in constructing predictive models.

The research above underscores the burgeoning interest in consumer analysis within e-commerce, particularly through machine learning methodologies. However, as the volume and complexity of data continue to burgeon, traditional machine-learning approaches are rendered somewhat obsolete. Consequently, the evolution of deep learning technologies has ushered in more intelligent methods capable of dynamic environmental interaction, offering promising solutions to these evolving challenges.

2.2 Consumer behavior analysis using deep learning methods

With the ongoing advancement of deep learning technology, leveraging this technology for time series analysis of user data has emerged as a crucial avenue for predictive consumer behavior analysis. Inspired by gated circuits, researchers have introduced enhancements such as input, output, and forgetting gates in models based on gating mechanisms like LSTM [16] and GRU [17] to mitigate such problems as vanishing and exploding gradients. Hidasi et al. [18] integrated the GRU model into recommendation systems, proposing the GRU4Rec model to capture dependencies among users’ behavioral histories, thereby enhancing recommendation efficacy. However, the sequential computation inherent in RNN and their variants constrain model parallelism, leading to challenges in handling longer sequence data, where issues like gradient vanishing and explosion persist, especially for sequences exceeding a length of 100. Addressing this, the Transformer model introduced by Vaswani et al. [19] leverages attention mechanisms to overhaul the computational structure, enabling parallel processing and effective handling of extended time-series data. This innovation marked a significant improvement over traditional RNN structures, particularly in tasks like language translation and text classification, catalyzing further advancements in sequence modeling. Zhou et al. [20] identified the need for distinct user interest representations for various candidate items, proposing the DIN model equipped with an Attention mechanism to extract sequence features comprehensively. An Activation Unit (AU) was also introduced to tailor user embedding vectors, generating the user’s Attention Score for each item. Kang et al. [21] leveraged the Transformer encoder to capture dependencies among users’ historical behaviors, thereby extracting interest features to predict the next item of interest. This approach yielded superior results in experimental evaluations, showcasing the efficacy of leveraging Transformer-based architectures in consumer behavior prediction tasks.

3.1 Transformer network

The Transformer model epitomizes a deep learning paradigm, harnessing the Self-Attention mechanism to tackle sequential data, thereby adeptly addressing long-range dependency challenges and facilitating efficient parallel data processing [22]. Preceding the advent of the Transformer, sequence-to-sequence (Seq2Seq) tasks like machine translation and text summarization heavily relied on RNNs and their derivatives, such as LSTM and GRUs. These models iteratively processed each sequence element to capture contextual information. However, RNNs encounter issues with vanishing or exploding gradients when handling lengthy sequences. Although LSTMs and GRUs somewhat alleviate this problem to some extent, the inherent sequential nature of RNNs impedes parallel processing, resulting in suboptimal training efficiency [23]. A schematic illustration of a typical Transformer network structure is presented in Fig 1:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Framework for the transformer.

https://doi.org/10.1371/journal.pone.0327548.g001

In Fig 1, the TRANSFORMER network comprises four primary components: the self-attention mechanism, the multi-attention mechanism, positional encoding, and the encoding-decoding structure. Notably, the self-attention mechanism empowers the model to consider all elements within the sequence while processing each sequence element, ensuring that each element’s representation encapsulates information about the entire sequence. At the heart of the self-attention mechanism lies the computation of attention scores for each element concerning all other elements, followed by a weighted summation based on these scores to derive the new representation of each element, as depicted in Equation (1):

(1)

Where are the Query, Key, and Value matrices, respectively, and is the dimension of the key vector. This formula determines the weight of each value by calculating the similarity between the query and all the keys. The Transformer model amplifies the self-attention capability by utilizing a multi-head attention mechanism. Specifically, it conducts multiple parallel executions of the self-attention process, termed “heads,” wherein each head learns information about distinct sequence segments. Subsequently, the outputs of all heads are concatenated and processed through a linear layer. Given that the Transformer model inherently lacks the sequential processing capability of RNN, position encoding becomes imperative to furnish positional information to the model. Position encoding entails a vector that is added to the embedding vectors of each sequence element, thereby imbuing each element’s representation with positional information. The computational methodology for position encoding is delineated in equations (2) and (3):

(2)

(3)

Where pos is the position and is the dimension. The purpose of this is to allow the model to obtain the meaning of a word based on its relative or absolute position. The comprehensive process described above is embedded within the overarching structure of encoding and decoding, wherein the encoder handles input sequence processing while the decoder undertakes output sequence generation. In tasks like machine translation, the input sequence undergoes conversion into a sequence of contextual representations by the encoder, following which the decoder generates the output sequence based on these representations. Upon finalizing the model’s design through these constituent elements, the Transformer employs a normalization step to stabilize the training process at the output of each sub-layer (comprising the self-attention layer and the feed-forward neural network layer) in each module. The normalization formula is as follows:

(4)

Where and are mean and standard deviation of and are the learnable parameters, and denotes the element-by-element multiplication. In addition, the feedforward network in each encoding and decoding layer can be computed by Eq. (5):

(5)

Among them. , and are learnable parameters. The feedforward network is applied independently, position by position, thereby enhancing the model’s nonlinear capabilities. Transformer incorporates residual connections between the inputs and outputs of each sublayer (self-attention and feedforward network), which are subsequently normalized using the following equation:

(6)

Residual connectivity aids in mitigating the issue of vanishing gradients in deep networks, thereby facilitating the training of deeper networks.

3.2 Reinforcement learning DQN

Reinforcement learning constitutes a pivotal branch of machine learning, focusing on how an intelligent agent learns an optimal behavioral strategy to maximize cumulative rewards through interactions with its environment. In this paradigm, the agent commences from an environmental state, executes actions, and receives rewards from the environment based on these actions, transitioning to a new state [24]. This iterative process continues, enabling the agent to gradually discern the optimal actions to maximize future rewards given the current state.

DQN, an acronym for Deep Q-Network, represents a reinforcement learning algorithm amalgamating deep learning and Q-learning. It primarily addresses the challenge of traditional reinforcement learning algorithms in coping with high-dimensional continuous state spaces. By leveraging a deep neural network as a function approximator, DQN effectively learns the action-value function, denoted as the Q function, by leveraging a deep neural network as a function approximator from high-dimensional perceptual inputs (e.g., pixels) [25]. In essence, DQN extends the Q-Learning framework by integrating deep learning techniques.

Within the Q-Learning framework, the fundamental update rule can be expressed as follows:

(7)

Where is in the state The action to be taken in the state , is the discount factor, which is used to regulate the importance of the reward at the end, and is the next state the maximum of all possible actions in the next state value. In DQN, we use a deep neural network to approximate the Q-value function, and the network parameters are denoted by θ. The goal of DQN is to minimize the difference between the predicted Q-value and the target value. This goal can be achieved by minimizing the following loss function:

(8)

Among them is the target value, calculated as follows.

(9)

On this basis, experience playback and target network updates are carried out. The purpose of experience playback is to increase the diversity of training data and reduce the temporal correlation between samples, which is realized by transferring the transfer obtained from the interaction of the intelligent body with the environment is stored in the playback buffer in the playback buffer, and when training, a batch of samples is randomly selected from for learning. The target network is updated by periodically copying the parameters θ of the main network to the target network parameters θ*, which can be realized by soft updating, i.e., gradually mixing the weights of the main network into the target network.

3.3 RL-Trans for the user behavior analysis

After introducing both the Transformer network and the reinforcement learning, we constructed the RL-Trans network by integrating elements from both. The overall structure of the model is depicted in Fig 2:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Framework for the RL-Trans.

https://doi.org/10.1371/journal.pone.0327548.g002

The network initially processes various user information, preparing the corresponding vectors. Subsequently, leveraging the Transformer network, we execute combined feature extraction across diverse information through multi-head attention. Next, we employ the corresponding DQN network to facilitate dynamic interaction with the environment for reinforcement learning. Following these operations, we proceed with user interest recognition. Before computing the final result via sigmoid activation, we incorporate a DNN neural network structure to transform the concatenated feature information nonlinearly, enhancing the model’s nonlinear learning capabilities. The complexity of the model architecture can be adjusted according to specific scenarios by controlling the number of layers in the upper Transformer layer and the DNN layer. The DQN component in RL-Trans dynamically adapts to changes in consumer behavior by continuously updating its Q-values based on real-time interactions. To optimize this adaptation, a tailored reward function was designed to balance short-term engagement and long-term satisfaction. Anε-greedy exploration strategy was also implemented to ensure a trade-off between exploiting known consumer preferences and exploring new recommendation opportunities. The experience replay mechanism further enhances learning stability by storing past experiences and allowing the model to learn from diverse behavioral patterns, ensuring effective adaptation to evolving consumer interests.

In the training process outlined, we adopt the cross-entropy loss function. This loss function trains the model by minimizing the disparity between the model output and the actual label, thereby enhancing the model’s predictive classification capability. The cross-entropy loss function is calculated as shown in Equation (10):

(10)

Where is the total number of categories, and is an indicator function when the sample belongs to the category is 1 when the sample belongs to the category, and 0 otherwise, and is the model prediction that the sample belongs to category of the sample. The enhanced prediction layer in RL-Trans was designed to improve both accuracy and efficiency by incorporating adaptive attention fusion and reinforcement-aware optimization compared to conventional architectures. Instead of relying solely on standard dense layers, this layer integrates multi-head attention outputs with dynamically learned Q-values from the DQN module. This fusion ensures that both sequential consumer preferences and learned action policies contribute to the final prediction.

4.1 Dataset and experiment setup

The dataset selected for this paper is MovieLens-1M (MovieLens-1M: https://zenodo.org/records/5916501, https://doi.org/10.5281/zenodo.5916501) [26], a comprehensive collection of movie rating-related data obtained from the MovieLens movie recommendation system by GroupLens. It encompasses movie ratings, movie attributes, and user-related statistical features. Specifically, this paper utilizes one of the MovieLens-1M datasets comprising three files: ratings.dat, movies.dat, and users.dat. This dataset is a standard benchmark for testing and evaluating various recommender system algorithms, including collaborative filtering, content-based recommendation, and hybrid recommender systems. The MovieLens-1M dataset was preprocessed to ensure data quality and compatibility with RL-Trans. First, data was cleaned by removing duplicate entries and filtering users with fewer than 10 interactions to retain meaningful behavior patterns. Interaction sequences were then constructed by sorting user-movie interactions chronologically and segmenting sessions based on interaction gaps for reinforcement learning.

Feature engineering included one-hot encoding of user demographics, multi-hot encoding of movie genres, and learned embeddings for movie representations. Temporal features such as interaction recency were also incorporated. Finally, normalization techniques were applied, including scaling ratings to a [0,1] range and standardizing embeddings for stable model training. While proprietary data cannot be shared, we propose releasing a synthetic dataset that replicates key statistical properties, ensuring reproducibility and enabling further research in reinforcement learning-based recommendation systems. While a sizable dataset is suitable for addressing complex recommender system challenges, it is not as daunting as some larger datasets. The consumer behavior analysis in RL-Trans was conducted using a publicly available dataset, which underwent several preprocessing steps to ensure diversity and representativeness. First, data was cleaned to remove duplicates, inconsistencies, and missing values. Next, user interactions were tokenized and structured into sequential representations to capture behavioral patterns. Session-based segmentation was applied to enhance representativeness, preserving temporal dependencies in consumer actions. Additionally, categorical features such as product categories and user demographics were encoded for effective learning. Finally, the dataset was split using stratified sampling to maintain distribution consistency across training, validation, and test sets, ensuring that RL-Trans learns from a well-balanced and diverse set of consumer behaviors.

The primary objectives entail predicting users’ ratings for unseen movies based on their historical ratings and generating personalized movie recommendations based on users’ characteristics and movie attributes. For evaluation metrics, we regard each instance of user preference as a positive sample and assess the hit accuracy of predicting preference for specific products. This can essentially be viewed as a multi-source classification and recognition task, for which precision, recall, and F1 scores are selected as metrics to evaluate model performance. Additionally, we employ the hit rate (HR@k) to evaluate recommendation algorithms, which is calculated as shown in Equation (11):

(11)

In addition to the more mainstream recommendation algorithms, our model also explores classic methods such as LSTM, GRU, and BI-LSTM, BI-GRU, alongside other established approaches. The analysis encompasses well-known algorithms including DSSM [27], SDM [28], DeepFM [29], and DIN [30]. Among them, the DSSM can be regarded as a basic benchmark model. DSSM (Deep Structured Semantic Model), used as a benchmark model, leverages deep neural networks to learn semantic representations of users and items, enabling recommendation through similarity-based vector matching. However, DSSM relies on fixed vector representations, making it less effective in capturing dynamic user interest changes and limiting its performance on long-sequence interactions. Additionally, it lacks a reinforcement learning mechanism, preventing adaptive optimization based on long-term user feedback. In contrast, RL-Trans, integrating Transformer and DQN, effectively captures long-range dependencies and dynamically optimizes recommendations, making it more suitable for complex consumer behavior modeling. Referring to the previous studies’ model establishement [31], the hyperparameters are set as follows Table 1:

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. The hyperparameters setting.

https://doi.org/10.1371/journal.pone.0327548.t001

4.2 The model comparison on the public dataset and ablation experiment

After establishing the respective models and defining the data indicators, we conducted a comprehensive analysis and comparison. Initially, we compared classical modeling methods by replacing the transformer module with LSTM, GRU, and other corresponding modules. The obtained results are presented in Fig 3:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. The comparison result for the classical method.

https://doi.org/10.1371/journal.pone.0327548.g003

In Fig 3, it is evident that the RL-Trans framework proposed outperforms traditional methods such as LSTM across all three categories of indicators: precision, recall, and F1-score. The improvement is notably significant, with all three indicators exceeding 0.5, representing a substantial enhancement over other methods.

Following the comparison of classical models, we evaluated the widely used push recognition recall model. The results are depicted in Fig 4:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. The comparison result for the currently used method.

https://doi.org/10.1371/journal.pone.0327548.g004

Through Fig 4, it is evident that the RL-Trans method proposed maintains a distinct advantage over the more recent recognition recommendation algorithms. Across precision and recall metrics, all five methods achieve values exceeding 0.5, yet the performance of the RL-Trans method stands out as superior. This observation underscores the efficacy of the proposed method.

Following comparing and optimizing of the recognition aspect, we proceeded to simulate the number of hotspot pushes. Subsequently, we analyzed the hit results under varying numbers of pushes, with the findings presented in Table 2 and Fig 5:

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. The comparison result for the HR@k with different k numbers.

https://doi.org/10.1371/journal.pone.0327548.t002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. The HR@k result among different k.

https://doi.org/10.1371/journal.pone.0327548.g005

As depicted in Fig 5, the hit rate of the method proposed consistently outperforms other push methods across different numbers of pushes. Notably, its overall results remain stable even when pushing 10 corresponding movies. This observation suggests that the proposed method delivers higher precision with lower standard requirements, thus effectively meeting the need for precise pushes.

4.3 The practical test for the RL-Trans and ablation experiment

Upon completing the model’s training of the model using the public dataset, we assessed its performance on a self-constructed dataset. The self-constructed dataset comprises shopping preferences and relevant shopping data of nearly 100 types of products from a total of 500 users on shopping websites in this region. Following meticulous data cleaning and feature extraction, we conducted further model analysis. The results obtained for precision, recall, and F1-score are depicted in Fig 6:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. The method comparison on the self-established dataset.

https://doi.org/10.1371/journal.pone.0327548.g006

In Fig 6, it’s evident that the proposed method demonstrates superior recognition performance across all three evaluation metrics. When comparing precision indices, the recognition results of our method and the DeepFM method are relatively close, with values of 0.91 and 0.69, respectively.

However, in terms of recall indices, our method outperforms DeepFM significantly, indicating better stability and generalization performance.

To further evaluate the model, we compared different push targets with and without reinforcement learning across various test subjects. The results are illustrated in Fig 7:

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. The ablation experiment result for the HR@k.

https://doi.org/10.1371/journal.pone.0327548.g007

In Fig 7, we present the push hit results of RL-Trans under three different push quantity settings: 5, 10, and 15. The results indicate a clear pattern as the number of pushes increases, showing how different modules contribute to model performance.

Firstly, the Transformer module is crucial in capturing long-range dependencies, effectively modeling users’ long-term interest shifts and ensuring more globally consistent recommendations. Compared to traditional sequential models like RNNs or LSTMs, Transformer leverages the multi-head self-attention mechanism to focus on different time-step dependencies, making the push content more aligned with users’ potential preferences. Even without reinforcement learning, the Transformer maintains relatively stable recommendation accuracy, leading to a more evenly distributed performance across different push settings. However, Transformer alone lacks adaptability in dynamic environments, as it does not adjust its recommendation strategy based on long-term user feedback. We incorporate the DQN reinforcement learning module to address this limitation, which optimizes push strategies through interactive learning. DQN enables the model to adjust recommendations dynamically under different push quantities and user groups, learning the optimal sequence of recommendations for long-term rewards. Experimental results demonstrate that, across different push tasks (5, 10, or 15 pushes), RL-Trans with DQN consistently outperforms the Transformer-only variant, confirming that reinforcement learning enhances recommendation effectiveness and improves adaptability and generalization.

In summary, the Transformer module constructs robust feature representations and global interest modeling, while DQN further strengthens the model’s decision-making ability. This synergy enables RL-Trans to maintain superior performance across various push settings, delivering more precise consumer behavior predictions and personalized recommendations.