Fuzzy number.

Definition 1. Triangular Fuzzy Number

A fuzzy number (fuzzy set) on a real domain can be defined as a triangular fuzzy number if its membership function is equal to [14], and it is shown in S1 Fig.

(1)

where l ≤ m ≤ u, l and u stand for the lower and upper values of the support of fuzzy number (fuzzy set), respectively; and m is the most likely value (middle value), that is, when x = m, x belongs to . The triangular fuzzy number can be denoted by (l, m, u). The support of is the set of elements {x ∈ R|l < x < u}. When l = m = u, it is a non-fuzzy number.

Definition 2. α-cuts of a fuzzy set

The α-cut of fuzzy set , represented by , is defined as , where 0 < α ≤ 1, θ_L is the lower value of and θ_R is the upper value of [15, 16].

Fuzzy function.

Definition 3. Fuzzy Function

Extended from a real-valued function f: Y = f(x, θ), the fuzzy function can be defined as [16] (2) where θ and are parameters of functions f and [16, 17], is the fuzzy output set.

The membership function can be expressed as (3) where θ = (θ₁, …, θ_n)^T, and

Property 1. Real Domain Interval Arithmetic

For two real domain intervals [a, b] and [c, d], the interval arithmetic can be defined as [16, 18]

[a, b] + [c, d] = [a + c, b + d];

[a, b] − [c, d] = [a − c, b − d];

[a, b] × [c, d] = [min(a × c, a × d, b × c, b × d), max(a × c, a × d, b × c, b × d)];

[a, b] ÷ [c, d] = [min(a ÷ c, a ÷ d, b ÷ c, b ÷ d), max(a ÷ c, a ÷ d, b ÷ c, b ÷ d)].

Definition 4. α-Cuts of a Fuzzy Function

α-Cuts of : For a continuous function f, the α-Cuts of , namely , can be expressed as [16] (4)

It is infeasible to calculate the membership function using formulas (2) and (3) because this requires maximization and minimization of the original function. This paper adopts α-cuts and interval arithmetic to solve this problem.

(5)

The membership function of a fuzzy function can be obtained by interval arithmetic because intervals are easy to calculate. However, when f is very complex, interval arithmetic will be NP-hard; therefore, this paper introduces a defuzzification approach to handle this problem.

Defuzzification of the fuzzy function.

The centroid approximate solution [19] is employed to defuzzify the fuzzy function in this paper.

The centroid of fuzzy function can be denoted by [19]: (6) However, directly calculating formula (6) is difficult because it involves integrals. The centroid can be approximated through many α-cuts of the fuzzy function in discrete form. is a vector of fuzzy numbers and an α-cut of can be denoted as , where θ_L and θ_R are lower and upper bounds, respectively, of the interval with respect to α. When x is nonnegative, is a monotonically decreasing function with respect to parameters . Thus, according to the interval arithmetic principle, definition 3 and formula (5), the α-cut of is given by (7)

The approximate centroid of fuzzy function can be obtained by [20] (8) where α = (α₁, …, α_N), α ∈ [0, 1]^N and .

Although all the α-cutsare bounded intervals, this paper takes only the special case where α_i = 1 into consideration. Let θ = [θ_L, θ_R]. Formula (8) can be expressed as [20] (9) Therefore, fuzzy function can be expressed as (10)

Basic fuzzy components and corresponding learning algorithms.

Inspired by the idea of above fuzzy theory and methods [16–20], we first provide a detailed introduction and describe the corresponding learning algorithms of several basic components that have been modified using fuzzy technology.

A. FRBM and its Learning Algorithm

Fuzzy restricted Boltzmann machine (FRBM) is a symmetric neural network with binary nodes that is based on an energy model. It contains a set of visual binary nodes v ∈ {0, 1}^D and another set of hidden binary nodes h ∈ {0, 1}^F. There are no connections between different nodes of the same hidden layer or between different nodes of the same visual layer. Fuzzy parameters are employed to govern the FRBM model. The structure of FRBM is described in S2 Fig.

The fuzzy energy function of FRBM can be expressed as [20] (11) In this formula, , and are the offsets, and is the connection weight between the ith visible node and the jth hidden node.

According to the free energy function , the fuzzy free energy function is expressed as (12) If is used directly to define the probability, it is difficult to calculate the fuzzy probability, fuzzy maximum likelihood and fuzzy objective function in the optimization. Therefore, needs to be defuzzifized to transform the optimization into regular maximum-likelihood optimization. This paper adopts a centroid method to defuzzify .

The centroid of is expressed as , and the probability can be defined as (13) This paper selects the negative log-likelihood as the objective function of FRBM: (14) In the formula, D denotes the training dataset.

The learning algorithm of FRBM finds the parameters that minimize the objective function (). The paper adopts the centroid approximation method to defuzzify : (15) The gradients of with respect to θ_L can be expressed as (16) In the formula, E_p(⋅) is the expectation over the target probability distribution P.

Correspondingly, (17) Because it is difficult to calculate the expectation , an approximation approach is employed. After defuzzifying the objective function, Gibbs sampling [21] is adopted to sample from these conditional distributions.

For FRBM, the fuzzy conditional probabilities can be expressed as and . The α-cuts of the fuzzy conditional probabilities are described as and .

In these formulas, P_L(h_j|x), P_R(h_j|x), P_L(x_i|h) and P_R(x_i|h) are the conditional probabilities with respect to the lower bounds and upper bounds of the parameters. Consequently, (18) and (19) In these formulas, and are the lower bound and upper bound of the connection weight, and are the lower bound and upper bound of the visible bias, and and are the lower bound and upper bound of the hidden bias [20].

According to (15), (16), (17) and the expectation estimation method of [20, 22], the gradients of with respect to the fuzzy parameters of FRBM can be obtained as follows: where P_c(x) is the centroid probability.

The CD1 algorithm [22] is employed to obtain the updating rules for parameters (θ_L and θ_R)to approximate the expectation [20, 22]. The above description is summarized as Algorithm 1.

Algorithm 1 The learning algorithm of FRBM

Input: x⁽⁰⁾ is an input sample of the training set;

  ε is the learning rate;

  W^L and W^R are the lower and upper bounds of connection weight matrices of the visible and hidden layers;

  b^L and b^R are the lower and upper bounds of bias vectors of the visible nodes;

  c^L and c^R are the lower and upper bounds of bias vectors of the hidden nodes.

Output: The latest parameters of FRBM: W^L, W^R, b^L, b^R, c^L, c^R.

 BEGIN

 1. For all hidden nodes j do

  obtain and by means of formula (18);

  sample from ;

  sample from ;

  end for

 2. For all visible nodes i do

  gain and by means of formula (19);

  sample from ;

  sample from ;

  end for

 3. For all hidden nodes j do

  obtain and by means of formula (18);

  sample from ;

  sample from ;

  end for

 4. The parameters can be updated according to the following rules:

  W^L = W^L + ε(x⁽⁰⁾ ⋅ P_L(h^L(0) = 1|x⁽⁰⁾) − x^L(1) ⋅ P_L(h^L(1) = 1|x^L(1)));

  b^L = b^L + ε(x⁽⁰⁾ − x^L(1))

  c^L = c^L + ε(P_L(h^L(0) = 1|x⁽⁰⁾ − P_L(h^L(1) = 1|x^L(1)));

  W^R = W^R + ε(x⁽⁰⁾ ⋅ P_R(h^R(0) = 1|x⁽⁰⁾) − x^R(1) ⋅ P_R(h^R(1) = 1|x^R(1)));

  b^R = b^R + ε(x⁽⁰⁾ − x^R(1));

  c^R = c^R + ε(P_R(h^R(0) = 1|x⁽⁰⁾ − P_R(h^R(1) = 1|x^R(1)));

 5. return W^L, W^R, b^L, b^R, c^L, c^R.

 END

B. FGBRBM and Its Learning Algorithm

The nodes of the visible and hidden layers in the fuzzy Gaussian-Bernoulli restricted Boltzmann machine (FGBRBM) model correspond to the Gaussian-Bernoulli nodes. In other words, the nodes of visible layer v ∈ R^D are Gaussian nodes and the nodes of the first hidden layer h ∈ {0, 1}^F are Bernoulli nodes (binary nodes) [23]. Their structures are described in S3 Fig. The fuzzy energy function of FGBRBM can be defined as (20) where σ_i denotes the standard deviation of v_i in the ith dimension of the Gaussian visual node.

Correspondingly, as for FGBRBM, the fuzzy conditional probability distribution for obtaining nodes of the hidden layer through nodes of the visual layer can be described as: (21) where g(⋅) denotes the sigmoid function.

The formula for constructing the nodes of the visual layer through nodes of the hidden layer can be expressed as (22) In this formula, N(μ, σ²) is a Gaussian probability density function, where μ is the mean value and σ² denotes the variance.

The α-cuts of fuzzy conditional probabilities can be expressed as , , where P_L(h_j|v) and P_R(h_j|v) are the conditional probabilities with respect to the lower and upper bounds of the parameters governing the FGBRBM, and v_iL and v_iR are the probabilities of the normal distribution. Consequently, (23) and (24) where and are the lower and upper bounds of the connection weight, respectively; and are the lower and upper bounds of visible node bias, respectively; and and are the lower and upper bounds of the hidden node bias, respectively.

The CD-1 algorithm can also be used for learning parameters of the fuzzy Gaussian-Bernoulli RBM model. In the process of training, the input data of the whole training set should first be normalized so that each dimension of the input data in the FGBRBM model obeys the normal distribution, i.e., the mean value is 0 and the variance is 1. When the formula is calculated, σ = 1. In the process of reconstructing the nodes of the visual layer, this paper does not adopt binary data instead of the probabilities of the normal distribution. The above description is summarized as Algorithm 2.

Algorithm 2 The learning algorithm of FGBRBM

Input: x⁽⁰⁾ is a sample of the training dataset;

  ε is the learning rate;

  W^L and W^R are the lower and upper bounds of connection weight matrices of the visible and hidden layers;

  b^L and b^R are the lower and upper bounds of bias vectors of the visible nodes;

  c^L and c^R are the lower and upper bounds of bias vectors of the hidden nodes.

Output: The latest parameters of FGBRBM: W^L, W^R, b^L, b^R, c^L, c^R.

 BEGIN

 1. For all hidden nodes j do

  obtain and by means of formula (23);

  sample from ;

  sample from ;

  end for

 2. For all visible nodes i do

  obtain v_iL and v_iR by means of formula (24);

  ;

  ;

  end for

 3. For all hidden nodes j do

  obtain and by means of formula (23);

  sample from ;

  sample from ;

  end for

 4. The parameters can be updated according to the following rules:

  W^L = W^L + ε(v⁽⁰⁾ ⋅ P_L(h^L(0) = 1|v⁽⁰⁾) − v^L(1) ⋅ P_L(h^L(1) = 1|v^L(1)));

  b^L = b^L + ε(v⁽⁰⁾ − v^L(1))

  c^L = c^L + ε(P_L(h^L(0) = 1|v⁽⁰⁾ − P_L(h^L(1) = 1|v^L(1)));

  W^R = W^R + ε(v⁽⁰⁾ ⋅ P_R(h^R(0) = 1|v⁽⁰⁾) − v^R(1) ⋅ P_R(h^R(1) = 1|v^R(1)));

  b^R = b^R + ε(v⁽⁰⁾ − v^R(1))

  c^R = c^R + ε(P_R(h^R(0) = 1|v⁽⁰⁾ − P_R(h^R(1) = 1|v^R(1)));

 5. return W^L, W^R, b^L, b^R, c^L, c^R.

 END

C. FLR and Its Learning Algorithm Used for Modeling the Click Through Rate

The logistic regression (LR) model [24] is a classic model in the field of advertising click-rate prediction. The LR model is described in S4 Fig. The activation function of the node in the output layer is a sigmoid function [25]. The output value of the node (value of the activation function) h_θ(x′) is the probability that a user will click on the advertisement; it can also be described as P(y = 1|x′; θ′). When a sample () in the training set is given, the probability value of advertising click rate prediction can be obtained in terms of logistic regression and can be described by formula (25) (25) Correspondingly, .

In this formula, θ′^T denotes the vector of parameters of the standardized logistic regression model, a refers to the features of the advertisement, u denotes the features of users, and c represents the features of the context environment. These three features compose a vector, which can be denoted as . This vector is fed into the logistic regression model for CTR prediction.

For a single sample (), the probability of correctly predicting the advertising click-through rate can be described as . This paper selects the likelihood function as the objective function: (26) In formula (26), the click label y_d denotes the expected output and is the real output value of the neuron node in the output layer (output of the activation function); can be described as , where and g(x) denotes the sigmoid function.

As for the fuzzy logistic regression model, the objective function can be described as (27) where , and .

According to formulas (5) and (7), the gradient of the objective function of FLR with respect to fuzzy parameter can be expressed as (28) (29) The above description is summarized as Algorithm 3.

Algorithm 3 The learning algorithm of FLR

Input: is a sample of training sets;

  ε is the learning rate;

   and , where W^L and W^R are the lower and upper bounds of connection weight matrices of FLR, and c^L and c^R are the lower and upper bounds of bias vectors of the output node of FLR;

Output: The latest parameters of FLR: , .

 BEGIN

 1. obtain by means of formula (28) based on the input data ;

 2. obtain by means of formula (29) based on the input data ;

 3. The parameters can be updated according to the following rules:

  ;

  ;

 4. return .

 END

Advertising CTR prediction model FDNN.

A. Construction of Advertising CTR Prediction Model FDNN

This section describes the proposed fuzzy deep neural network (FDNN) model for advertising CTR prediction and its learning algorithm in detail. The architecture of the method and its process of predicting CTR are illustrated in S5 Fig.

First, the fuzzy deep belief network (FDBN) constituted by FRBM and GBRBM is used to automatically learn the discriminative features of raw input data from an advertising dataset, and these features are able to capture internal explanatory information that is hidden in the raw data, amplify the important information for discrimination and weaken irrelevant information [26].

Next, FLR is adopted to model the CTR prediction and map the predicted CTR value to the range between 0 and 1.

The raw input data of advertising CTR prediction are the fields that include the corresponding advertisement features, users’ features, and context features, which are extracted from the advertisement click log. These fields contain not only numerical data but also multi-category data; therefore, the raw input data of advertising CTR prediction are composed of vectors with different data types.

Because FRBM only models input with binary variance (0, 1), this paper adopts fuzzy Gaussian-Bernoulli RBM (FGBRBM) to address the input data of advertising CTR prediction.

Then, FRBM can be used to construct FDBN through layer-by-layer stacking; the detailed steps can be found in [22].

In real applications, advertising click rate prediction refers to binary classification or regression; therefore, this paper uses fuzzy logistic regression to model user behaviors in clicking advertisements. After the input data vector x = (a, u, c) = (x₁, x₂, …, x_n) is extracted by FDBN, the vector , which is composed of the binary values, in the last hidden layer of FDBN is fed into fuzzy the logistic regression model for CTR prediction.

The advantage of the proposed method is that fuzzy set techniques are introduced, and the uncertainties in the relationships between nodes located in adjacent layers is taken into consideration. Nodes in adjacent layers of FDNN often interact in uncertain ways. Because the parameters that represent the relationships between nodes in adjacent layers are fuzzy numbers and the learning process of fuzzy parameters is extended to a relatively wider space, this advantage will be reflected in the fitness of the joint probability distribution. Combined with the merits of deep learning, the proposed method demonstrates superior performance in coping with data with noises.

B. Construction and Training of the Advertising CTR Prediction Model FDNN

The process of constructing FDNN is illustrated in S6 Fig, and the detailed steps of constructing the FDNN are as follows:

Step 1: The training of FRBM layer by layer

First, the vectors that are constituted by related data of textual advertising click logs are regarded as the input of FGBRBM, and the CD1 algorithm is used to train FGBRBM; in this way, the parameter θ₁ of FGBRBM is obtained. Based on the input vectors and the parameter θ₁, the hidden nodes h₁ of FGBRBM are obtained.

Second, the parameter θ₂ of the current FRBM can be obtained when h₁ is regarded as the input data for the CD1 algorithm. Based on h₁ and θ₂, the hidden nodes h₂ of the current FRBM can be obtained.

Third, the previous process is repeated until reaching the fixed layer.

Step 2: Constructing FDBN by means of FRBMs

According to the training procedure of FRBM in step 1, the weight values of FRBMs are connected from bottom to top to construct FDBN. In this FDBN, the connection between the top two layers is undirected and the other connection is directed from top to bottom [22].

Step 3: Constructing the FDNN that is used for advertising CTR prediction by means of FDBN

After an FLR model is added onto the top of the FDBN, the probability that the user clicks this advertisement is equal to the output value of the nodes of the input layer. At this point, the network becomes the initially trained fuzzy deep neural network (FDNN).

Step 4: Fine-tuning the parameters of FDNN

After FDNN is initially trained layer by layer, a fine-tuning process needs to be conducted. In contrast to the unsupervised learning approach in the initial training, the fine-tuning is implemented using supervised learning. Based on the weight values of the network that were obtained from the initial training, error backpropagation [27, 29] and stochastic gradient descent are used to update the weight values and bias of the neural network. After fine-tuning, the process of training FDNN is complete. The output of FDNN is in the form of a probability: .

The process of fine-tuning the parameters of FDNN contains two phases.

1) Forward calculation of the input signal:

When labeled data (x_d, y_d) of click logs are given, that is, the combined feature vector x = (a, u, c) = (x₁, x₂, …, x_n) is given, the jth node in the lth layer (l = 2, ‥, L_n) of the FDNN deep neural network can be defined as follows:

L_n denotes the total number of layers in the FDNN network;

s_l denotes the number of nodes in the lth layer of the FDNN network;

denotes the connection weight between the jth node in the lth layer and the ith node in the l − 1th layer; denotes the bias of the jth node in the lth layer; denotes the weighted-sum input of the jth node in the lth layer;

denotes the output value of the activation function of the jth node in the lth layer;

The selected activation function is sigmoid function.

In the forward calculation, the input signal is transmitted from the input layer to the output layer. The input value of the jth node in the second layer of FDNN (l = 2) can be described as . When the input of node j is transferred by the activation function, its output value can be described as . Therefore, the input value of the jth node in the lth layer of FDNN (l = 3, …, L_n) can be described as and its corresponding output value can be described as .

In particular, the output value of the jth node in the th layer of FDNN can be described as (30)

2) Backpropagation of the error signal:

This paper regards the cross-entropy as the objective function [25]. It can be described as formula (31). The local gradient will be refined using backpropagation. The error signal between the output value and labeled signal will be transmitted from the output layer to the input layer in the process.

(31)

In this formula, y_d is the label value of the sample, which can be represented as y_d ∈ [0, 1].

The gradient descent method will be adopted to learn the model. This paper will first define the residual error [28] of the jth node in the lth layerlayer as the partialand of objective function with respect to the weighted sum of node j. The residual error of the j th neuron node of the L_n th layer can be calculated by formula (32) (32) The residual error of the jth neuron node of the lth layer (l = 2, …, L_n − 1) can be calculated by formula (33) according to the chain rule [29].

(33)

where denotes the connection weight between the jth node of the lth layer and the ith node of l + 1th layer.

According to formulas (2) and (10), the gradient of cost function with respect to parameters and can be calculated by formulas (34), (35), (36) and (37).

(34)(35)(36)(37)

The above process is described in Algorithm 4.

Algorithm 4 The algorithm for fine-tuning the parameters of FDNN using BP

Input: (x_d, y_d) is the labeled training data of textual advertising click logs;

  ε is the learning rate;

   is the lower bound of the connection weight between the jth node in the lth layer and the ith node in the (l − 1)th layer of FDNN;

   is the lower bound of the bias of the jth node in the lth layer of FDNN;

   is the upper bound of the connection weight between the jth node in the lth layer and the ith node in the (l − 1)th layer of FDNN;

   is the upper bound of the bias of the jth node in the lth layer of FDNN;

Output: The latest parameters: , , ,

 BEGIN

 1. In the forward-propagation calculation, when the input data vector x_d is fed into the input layer, the output value of FDNN can be obtained using formula (30);

 2. In the calculation of error backpropagation,

  obtain by means of formula (34);

  obtain by means of formula (35);

  obtain by means of formula (36);

  obtain by means of formula (37);

 3. The parameters can be updated according to the following rules:

  ;

  ;

  ;

  ;

 4. return , , , .

 END

Description of dataset.

This paper uses the Criteo Display Advertising Challenge dataset, which is provided by the Criteo company on the famous machine learning website Kaggle for advertising CTR competition [30]. Every advertisement log record contains the set of features of the displayed advertisement, which is composed of 13 numerical features and 26 categorical features. The detailed formats of these features are given in S1 Table. The class Label is used to indicate whether this advertisement is clicked: if the advertisement is clicked, the value of Label is 1; otherwise, the value of Label is 0. Categorical features are desensitized using Harsh mapping; therefore, real meaning can not be obtained from them.

To adequately evaluate the performance of this model and prevent the interference of many factors such as data timeliness, we randomly disrupt the order of the raw samples in the experiment. Meanwhile, to avoid the over-fitting, the 5-fold cross validation method [31] is adopted in the training process of all experiments.

Experiment environment.

Six high-performance workstations are used for the experiments. The hardware and software configurations of the experimental environment are shown as S2 Table.

AUC.

This paper uses the area under the receiver operating characteristic curve (AUC) [33, 34] as the main performance evaluation metric for advertising CTR prediction. The confusion matrix is shown as S3 Table [35, 36]. Formulas TPR = TP/(TP + FN) and FPR = FP/(FP + TN) are used to calculate the corresponding TPR and FPR values of the confusion matrix to obtain a co-ordinate pair (dot pair). The dot pairs make up the receiver operating characteristic (ROC) curve [37]. The AUC is the total area under the ROC. The larger the AUC is, the more accurate the advertising CTR prediction is, and the better the effect achieved by the advertising placement is.

Log-loss.

This paper selects the logarithmic loss function ‘log-loss’ [38, 39] as another auxiliary performance evaluation metric for advertising CTR prediction.

(38)

where L is the number of samples, y_i is the true label (0 or 1), and is the predicted probability. The value of log-loss reflects the similarity between the CTR predicted by model and real CTR. The smaller the value of log-loss is, the more accurate the advertising CTR prediction is.

Configuration optimization and performance analysis of FDNN and DBNLR.

A. Feature Pre-processing

FDNN is similar with DBNLR in the architecture. Their difference lies in the components of model. Therefore, this paper selects the DBNLR as a baseline model.

In the experiment on the FDNN and DBNLR models, the numerical features of the raw dataset are directly adopted, and these categorical features are expanded to binary vectors using one-hot representation.

Although the method of one-hot representation is straightforward and effective, it makes the feature vectors sparse and high-dimensional. In the deep learning models, when the dimensionality of the features is large, the scale of the parameters of the model becomes large, which leads to a sharp increase in training time. Moreover, the hardware will be a bottleneck in the training process. Therefore, this paper adopts a feature hashing strategy (signed hashing trick) [40] to reduce the dimensionality of these sparse binary feature vectors. Finally, the joint vector that is composed of numerical features and categorical features and has been processed by the feature hashing strategy is regarded as the input of the two deep architecture models.

Input vectors can be obtained when the data of the Display Advertising Challenge dataset are preprocessed using the feature preprocessing methods described above. In this experiment, after preprocessing, the dimensionality of the input vector is 1746088. Therefore, the number of nodes in the visible layer (input layer) is 1746088.

B. The Parameter Settings for Training FDNN and DBNLR

These two models are developed based on the Theano framework [41] by means of Python. In all experiments, the maximum number of learning epochs in the training process is set to a sufficiently large value (800) and the Early-Stopping strategy is adopted to automatically cut epochs for obtaining the best validation score. Furthermore, this strategy can also avoid over-fitting.

The steps of the Early-Stopping strategy are as follows:

1. Split the dataset into a training set and a validation set;

2. When the training of every epoch has been finished, calculate the validation error (loss) in the validation dataset;

3. When the validation error of the validation dataset no longer decreases, record the number of epochs and the current validation error value. If the validation error of the validation dataset does not increase after continuation, the training process is terminated, the best validation score is obtained, and optimization is complete. Meanwhile, the optimal parameter estimation values are obtained.

In the initial training process of FDBN and DBNLR, the gradient descent method with minibatch is used for training on these samples and each minibatch includes 2000 training samples. For FGBRBM (or GBRBM), the learning rate is 0.01, while for FRBM (or RBM) with the binary variable, the learning rate is 0.1. The decay factor of the weight value is 0.0002.

In the fine-tuning process of FDNN and DBNLR, the gradient descent method with minibatch is used for training on these samples and each minibatch includes 4000 training samples. The decay factor of the weight value is 0 and the learning rate is set as 0.005.

Dropout represents the probability that a node is retained in the neural network [42, 43]. A reasonable sparse dropout strategy can strengthen the robustness of a neural network. Dropout is utilized to reduce over-fitting and the complexity of these two deep learning models. The paper sets the dropout rate to 0.5, 0.6, 0.7, 0.8, and 0.9 to implement the testing experiments. Under the condition of having the same configuration structure, the dropout rates are respectively 0.5 and 0.6 when FDNN and DBNLR use their own optimal settings.

C. Optimization of the number of hidden layers and hidden nodes

Currently, there is no method for quickly setting the number of hidden layers and the number of hidden nodes in each layer. Therefore, many experiments and much experience are utilized to search for the optimal structure. According to past experience, when the number of nodes of the visible layer is large, the nodes of the hidden layer need to perform “dimensionality reduction- dimensionality addition—dimensionality reduction”. However, when the number of nodes of the visible layer is small, the nodes of the hidden layer only need to perform “dimensionality addition—dimensionality reduction”. In general, the number of hidden nodes will be increased or decreased with multiple. By comparing the performances in many experiments with different configurations (different numbers of nodes of each hidden layer and numbers of hidden layers), the configuration with relatively optimal performance can be obtained.

In S4 Table, information on different configurations in many experiments is given. In S7 Fig, the corresponding AUC values of DBNLR and FDNN with different configurations are given. In S8 Fig, the corresponding log-loss values of DBNLR and FDNN with different configurations are given.

As presented in S7 and S8 Figs, in the experiments for training FDNN, in the beginning, increasing the number of hidden layers can improve the performance of the proposed method; however, when the number of hidden layers is more than 3, the performance is degraded. This is because FDNN does not adequately capture feature interactions when the number of hidden layers is small. However, when the number of hidden layers and the number of nodes in the hidden layers exceed a certain scale, cross-features disturb and degrade the accuracy of CTR prediction. The over-fitting phenomenon is observed and the algorithm spends more time learning the parameters.

After many experiments have been completed, a configuration with relatively optimal performance is obtained: Conf11, the number of hidden layers is 3; the number of nodes of the first hidden layer is 170; the number of nodes of the second hidden layer is 1700; and the number of nodes of the third hidden layer is 17. This paper chooses this configuration as the final structure of FDNN. Corresponding AUC and log-loss on the test dataset can be seen in S9 and S10 Figs.

In the experiments for training DBNLR, relatively optimal performance is obtained when the configuration structure is as follows: Conf7, the number of hidden layers is 3; the number of nodes of the first hidden layer is 150; the number of nodes of the second hidden layer is 1500; and the number of nodes of the third hidden layer is 15. This paper chooses this configuration as the final structure of DBNLR. Corresponding AUC and log-loss on the test dataset can be seen in S9 and S10 Figs.

We observe that the curve of the AUC value of the DBNLR model and that of the FDNN model are indented. However, when they have the same configuration, the overall level of performance of DBNLR is worse than that of FDNN; this is true for the log-loss index. It is because fuzzy set techniques are introduced in the proposed FDNN model. The components of FDNN model are fuzzy RBM, fuzzy GBRBM and fuzzy LR. With the data of many records in the Display Advertising Challenge dataset which contains outliers and missing values implying many uncertainties, the uncertainties can be taken into consideration and efficiently handled by fuzzy set techniques; therefore, the performance of the FDNN model for CTR prediction is better than that of DBNLR in the experiment.

Configuration optimization of other baseline models.

A. LR Solution

CTR prediction in a real advertising system involves a large number of samples, while logistic regression (LR) model can appropriately address a large number of features and can be trained rapidly, so the LR model is selected as a baseline model in this paper. This paper selects LIBLINEAR [44] to implement L2-regularized logistic regression, and the stochastic gradient method (SG) is adopted to optimize the parameters of LR. In the LR solution, the numerical features are used directly and the categorical features are first represented with one-hot encoding and then used to establish the feature space with MurmurHash 3. After many experiments, this paper adopts the following parameters: η = 0.2, λ = 0.

B. FM Solution

For advertising CTR prediction, Factorization Machines (FMs) combine the advantages of Support Vector Machines with factorization models, and can characterize the basic feature interactions of a large number of advertising data, therefore, this paper selects the FM model as a baseline model [45]. For advertising CTR prediction, Factorization Machines (FMs) is also adopted, which is based on feature engineering and matrix design. This paper adopts LIBFM to implement factorization machines [46, 47] and the stochastic gradient method (SG) is adopted to optimize the parameters of FM. The numerical features are used directly and the categorical features are first represented with one-hot encoding. After many experiments, this paper adopts the following parameters: λ = 40, k = 30.

C. GBDT+FM Solution

Gradient Boost Decision Tree (GBDT) is a nonlinear model that is based on the concept of collective learning boosting. GBDT has strong extracting feature ability and nonlinear fitting ability, and it can flexibly handle various types of data in the advertising dataset including continuous value and discrete value, but no over-fitting occurs. Therefore, this paper selects GBDT+FM solution as a baseline model. Based on the work of Y Juan et al., this paper implements GBDT+FM solutions for performance comparison and analysis. First, in Preprocessing-A, all numerical features are used directly and all categorical features are represented with one-hot encoding. Then, these features are fed into decision trees in GBDT to generate GBDT features. This paper selects Xgboost to implement the GBDT [48]. In Preprocessing-A, the number of decision trees in GBDT is 30 and the depth is 7. Thirty features are generated for each impression. The following process is Preprocessing-B, which is used for generating input features for FM. In Preprocessing-B, numerical features (I1-I13) with values greater than 2 are transformed by v ← ⌊log(v)²⌋; categorical features (C1-C26) that appear fewer than 10 times are transformed into a special value; and GBDT features are directly included. Therefore, each record contains 69 features: 13 numerical features, 26 categorical features and 30 GBDT features. Then, these three groups of features are hashed into 1 M dimensions using a hashing trick. Finally, the processed features are fed into the FM model for CTR prediction. The parameters of the FM model that achieve the relatively optimal results are “λ = 40, k = 100”.

Experimental results and analysis.

According to S9 and S10 Figs, the proposed FDNN method has the best performance and LR has the worst performance. In the process of training models, it can be found that logistic regression model can be trained rapidly, but its effect is in the general level. One reason is that when LR is used to predict CTR, it requires the experts with sophisticated experience to implement feature engineering through the human labor. The other reason is that the simple structure of LR restrains its representation ability. These two disadvantages cause that LR cannot learn complex feature interactions and is insensitive to outliers and missing values. Therefore, LR has the relatively bad performance in the experiment.

According to S9 and S10 Figs, FM is superior in performance to LR. The reason is that FM extends the idea of LR because FM adds to second order fitting on the basis of first order fitting and it can automatically learn cross features of any two features from different dimensions. And cross features are expressed with embedding vector. Compared with LR, FM model has fewer manual intervene but higher efficiency. Therefore, FM has better representation ability than LR in the experiment. However, FM also cannot learn much more complex feature interactions because of its relatively simple structure.

In this experiment, GBDT+FM solution outperform the FM model. The GBDT+FM solution obtains good prediction results, which are slightly inferior to those of the proposed method. The GBDT can learn useful high-order feature interactions because it has strong nonlinear fitting ability and the robustness for hyperparameter, so it can efficiently extracts discriminative features and cross-features of advertising dataset, and shows better performance than FM model in our CTR prediction experiment.

Compared with the GBDT+FM solution, the proposed method achieves more than 0.09% improvement in terms of AUC (0.01% in terms of log-loss) on the dataset. The reasons are as follows. The first is that the quality of data exerts a great influence on effect of model. Because the dataset contains many noisy data, there are bottlenecks in the effects of GBDT+FM solution. Compared with this model, the proposed FDNN model, as a result of being endowed with fuzzy technology, can capture more-complex high-order feature interactions of the advertising dataset, which contains many outliers and missing values. The second is that GBDT+FM solution, belonging to a kind of stacking combined model without backproportion, is not always better than the proposed model with backproportion in effect improvement for CTR prediction from the theoretical perspective. Therefore, it demonstrates relatively good performance in terms of both data representation capability and prediction accuracy. In a real production environment for an advertising system, a small improvement in advertising CTR is likely to result in a significant increase in financial income for the internet company and an improved user experience.