Minority clustering in accumulation sequence of unbalanced data based on natural nearest neighbor.

The density clustering of minority samples in the accumulation sequence of unbalanced data based on the natural nearest neighbor relationship mainly includes three steps [10]: firstly, the core point is determined according to the natural nearest neighbor relationship of all minority samples in the accumulation sequence of unbalanced data; the self-neighborhood radius of the core point is calculated, to determine the core point directly connected with its density, and cluster the core points according to the process of density clustering [11], so as to classify the two core points within their own neighborhood radius values into the same category; According to the nearest neighbor relationship between non core points and core points, non core points are classified into the existing classification.

The implementation process of clustering algorithm based on natural nearest neighbor for minority classes in accumulation sequence of unbalanced data is as follows.

Input: natural eigenvalue s_k of a few samples in the accumulation sequence of unbalanced data, natural nearest neighbor N_N(x_i) and natural nearest neighbor N_B(x_i) of sample point x_i.

Output: core point set C_s of minority samples, non core point set N_Cs, number of clusters G and sample set C_G of various clusters.

Step 1: determine C_s and N_Cs from s_k, N_N(x_i) and N_B(x_i) according to definition clustering algorithm.

Step 2: cluster x_i, x_i∈C_s.

Step 2.1: calculate the neighborhood radius ϕ_i of x_i and the core point set D_i of which the direct density can be reached.

Step 2.2: using ϕ_i and D_i and referring to the density clustering process, classify all density connected core points into the same cluster to obtain C_G and G.

Step 3: divide x_j into G classes, x_j∈N_Cs.

Step 3.1: calculate ϕ_j for x_j.

Step 3.2: get all core points that are natural neighbors to x_j and within the radius of each other’s neighborhood.

Step 3.3: determine the core point x_i closest to x_j, and add x_j to the cluster where x_i is located; If there is no such core point, mark x_j as a noise point.

In the above process, new samples are generated by core points and non core points belonging to the same cluster. Noise points do not participate in the synthesis of sample points, which can avoid the introduction of new noise points.

Oversampling based on natural nearest neighbor.

After completing the clustering operation for a few samples in the accumulation sequence of unbalanced data, it can determine the number of samples to be synthesized for each class according to the proportion of non core points in each class. The more the non core points in the class are, the more new samples to be synthesized are. The reason is that the non core points are the samples whose number of natural nearest neighbors is less than the natural eigenvalue [12]. The distribution of samples around them is relatively sparse. More new samples are synthesized inside them, which can avoid the excessive concentration of new samples in densely distributed areas and reduce the overlap of composite samples. The specific steps of oversampling based on natural nearest neighbors are as follows.

Input: accumulation sequence of unbalanced data S.

Output: minority class sample set S_g generated by oversampling.

Step 1: search the natural nearest neighbor of a few samples [13] to obtain s_k, N_N(x_i) and N_B(x_i).

Step 2: cluster a few samples to get G, C_K, C_s and N_Cs.

Step 3: number of newly synthesized samples N = number of samples of most classes—number of samples of few classes.

Step 4: calculate the proportion p_i of non core points in class C_i, and determine the number of samples to be synthesized for class C_i according to the proportion.

Step 5: randomly select non core point x and core point y in class C_i to synthesize new samples, i = 1,2,⋯,G. z = x+a×(y−x), a is a random number in the range of [0,1], add z to the set S_g and repeat this step until the required number of samples is obtained for each category.

S_g is the composite sample set of a few classes.

Through the above series of processes, new minority samples are synthesized and added to the original data accumulation sequence, so as to balance the data accumulation sequence.

Mining of accumulation sequence of unbalanced data based on probability matrix decomposition.

Probabilistic Matrix Factorization (PMF) is a common recommendation system algorithm that uses a user-item rating matrix to recommend items that users might be interested in. In PMF, the original score matrix is decomposed to map both users and items into a potentially low-dimensional space, and then the score matrix is recalculated until its error from the original score matrix is minimal. Probability matrix decomposition is to map some potential information of users and projects (accumulation sequence of unbalanced data after balanced processing, hereinafter referred to as data sequence) to low dimensional feature space [14] from the perspective of probability, and then use the linear combination of low dimensional feature vectors to explain the preference of specific users for data sequences. Given the scoring matrix R_m×n of user data series, MATLAB is used to generate a random number matrix H_f×m with mean value of 0, variance of and a random number matrix Q_f×n with variance of , where f is the decomposition dimension, H_f×m is defined as the user’s f-dimensional characteristic matrix, Q_f×n is the f-dimensional characteristic matrix of data series, and its column vectors H_i and Q_j represent the corresponding potential characteristic vectors respectively, generally . After learning the training set, H_f×m and Q_f×n are constantly updated, to make .

Assuming that the error between the real score R_ij and the predicted score conforms to the Gaussian distribution [15] with a mean value of 0 and a variance of , there is a probability distribution: , and there is through translation, then the conditional probability of the scoring matrix R is as shown in Eq (1): (1)

Where, I_ij is the indicator function, I_ij value of 1 indicates that user i has scored the data sequence j, and I_ij value of 0 indicates that user i has not scored the data sequence j. N is the actual error, δ_R is the forecast error.

Since H and Q are independent of each other and satisfy the Gaussian distribution with mean value of 0 and variance of and respectively, there are Eqs (2) and (3): (2) (3)

Where, I is the function value. The joint probability distribution of H and Q can be obtained from Eqs (1), (2) and (3): (4)

Taking logarithm of probability distribution of U and V can get: (5)

The maximum value of Eq (5) can be equivalently replaced by the minimum value of the error function with regularization parameters [16], as shown in Eq (6): (6)

Where, , , and , the objective function can be described by Eq (7): (7)

The relationship between the regularization parameter λ and , , is obtained.

In order to solve the objective function, the random gradient descent method is used. This algorithm finds the direction in which the parameters of the objective function decline fastest by deriving the parameters [17], and makes the variables move along this direction until they move to the minimum point.

By deriving from Eq (7), it can be found that at each iteration, the updated equations of H_i and Q_j become: (8) (9) (10)

Where α is the learning rate of random gradient descent.

In addition, in order to improve the recommendation efficiency of the algorithm, the batch processing module [18] is added. For 90000 training data in the experiment, it is divided into 9 batches and 10000 data are processed each time, which greatly reduces the amount of calculation of model training and the instability of model convergence caused by the calculation of each training data.

The data sequence mining algorithm based on probability matrix decomposition is as follows:

Input: training set train_vce, test set probe_vce.

Output: mining result pred_out, square root error REST.

1. Set the regularization parameters, the maximum number of iterations maxepoch, and decompose the dimension feat;
2. Generate two standard normal distribution matrices: data sequence (1682) feat and number of users (943) feat;
3. Iteration times epoch < maxepoch;
4. Batch processing is adopted, which is divided into 9 batches, 10000 scoring records are processed each time, and the batch processing times are batch < 9;
5. The loss function p is calculated, and the two matrices of (2) are continuously updated according to the negative gradient direction;
6. End.

Through the above process of accumulation sequence of unbalanced data preprocessing and probability matrix decomposition, accumulation sequence of unbalanced data mining can be realized. The unbalanced data accumulation sequence is partially coded as follows:

#include <iostream>#include <cstring>#include <map>using namespace std;const int N = 1e6+10;#define int long longint n, maxn, minn, top1, top2, ans;int a[N], Lb[N], Rb[N], Ls[N], Rs[N];int st1[N], st2[N];signed main(){

  cin >> n;

  for (int i = 1; i < = n; i++) cin >> a[i];

  top1 = 0, top2 = 0;

  for (int i = 1; i < = n; i++)

  {

 while (top1 && a[st1[top1]] > a[i]) top1—;

 while (top2 && a[st2[top2]] < a[i]) top2—;

 if (!top1) Ls[i] = 0;

 else Ls[i] = st1[top1];

 if (!top2) Lb[i] = 0;

 else Lb[i] = st2[top2];

 st1[++top1] = i;

 st2[++top2] = i;

  }

  memset(st1, 0, sizeof(st1));

  memset(st2, 0, sizeof(st2));

  top1 = 0, top2 = 0;

  for (int i = n; i > = 1; i—)

  {

 while (top1 && a[st1[top1]] > = a[i]) top1—;

 while (top2 && a[st2[top2]] < = a[i]) top2—;

 if (!top1) Rs[i] = n+1;

 else Rs[i] = st1[top1];

 if (!top2) Rb[i] = n+1;

 else Rb[i] = st2[top2];

 st1[++top1] = i;

 st2[++top2] = i;

  }

  for (int i = 1; i < = n; i++)

  {

 ans + = (i—Lb[i]) * (Rb[i]—i) * a[i];

 ans - = (i—Ls[i]) * (Rs[i]—i) * a[i];

  }

  cout << ans;

  return 0;

}

Optimization of probability matrix decomposition.

In the probability matrix decomposition algorithm, the random gradient descent method is used to learn the characteristic matrix of users and data sequences. However, each prediction error in the learning process is only scored for a single user data sequence, resulting in the random gradient descent method without considering the overall prediction error effect. No matter how to optimize the feature vectors of users and data sequences, all predictions cannot be accurate, which leads to the phenomenon that although the algorithm has good prediction ability in the training set, it has poor prediction ability in the test set. To solve this problem, an adaptive error Perr is introduced, which is calculated by Eq (11): (11)

Where: w is the sample weight, and the initial value conforms to the uniform distribution; numRate is the number of scores in the training set, which results in the minimum initial value of w. Therefore, in Eq (11), w is multiplied by numRate to amplify the influence of w. In the optimization process, when the prediction error err is large, the sample weight w will become larger, and the adaptive error Perr will be larger; Similarly, the smaller the values of err and w are, the smaller the value of Perr is. The variation of w makes Perr associated with the global error. The random gradient descent method finds the appropriate eigenvector by minimizing the test error err of the training set [14], which is modified to minimize the adaptive error Perr of the training set to find the appropriate eigenvector. Therefore, the core idea of this paper is to optimize the global error as well as the single sample error more effectively.

Here, it is worth considering how to define the judgment threshold of prediction error. When Perr is small to a certain extent, if it continues to learn, it will lead to over fitting of the algorithm. Similar to the classification threshold of ADA Boost, it needs to meet the global needs. It can be known that standard deviation is the most commonly used quantitative form to reflect the degree of data dispersion. If the mean errMean of the error is used to represent the mean of the absolute prediction error of all samples, then the standard deviation errStd represents the dispersion degree of the overall error of the samples, that is, the greater the value of errStd is, the lower the prediction accuracy is, in turn, the higher the prediction accuracy is. Suppose φ represents the number of scores in the training set, then errStd is calculated with Eq (12): (12)

In Ada Boost, the weight of the weak classifier represents the impact of the weak classifier on the final strong classification, which is derived by optimizing an index loss [19–21]. Similarly, in order to improve the optimization speed of the algorithm, this weight is also added. If the threshold value of prediction error is η, then: (13)

Where: a represents the weight of the learner, and the weight calculation method is the same as that of ADA boost weak classifier. In Ada Boost, the error rate represents the proportion of samples with classification errors in the total samples [22, 23]. However, in the mining problem, the error rate C is used to represent the global error in the current learner, and φ is set to represent the number of scores in the training set, then Eq (14) calculates C: (14)

Therefore, according to ε, the sample weight w can be adjusted automatically to prevent over fitting to a certain extent [24, 25]. The method of adjusting the sample weight w is shown in Eq (15): (15) (16)

In Eq (15): D_t represents the weight distribution of the sample at the t-th learning; D_t(i) represents the i-th sample weight at the t-th learning; Z_t represents the normalization factor of the t-th learning. In Eq (16), Z_t+1 represents the calculation method of normalization factor at the t+1-th learning. C_t represents the feature weights of the sample at the t-th learning.

From the implementation process of the optimized probability matrix decomposition algorithm, it can be seen that in the t-th learning process, the characteristic matrices H_t and Q_t are mainly learned according to Perr, and then Perr is updated according to the value of the characteristic matrix. ADA Boost only adjusts the sample weight within each random gradient descent learning. After learning convergence, ADA Boost algorithm also ends; However, the idea of ADA Boost integration is to combine multiple probability matrix decomposition algorithms with different parameters, but does not optimize a single probability matrix decomposition model itself. This is the essential difference between ADA Boost adaptive lifting idea and ADA Boost integration idea. At the same time, according to the needs of real data, the parameters ξ₁ and ξ₂ are introduced: in order to ensure that the algorithm does not converge at the beginning, ξ₁ is used to reduce the influence of w; ξ₂ is to restore w in the learning process and strengthen the intervention of w in the prediction error.

Eq (17) is used to initialize the sample weight: (17)

In Eq (17), μ represents the error correction coefficient.

Meanwhile, improved Eq (11) is Eq (18) to calculate Perr. The values of parameters ξ₁ and ξ₂ can be obtained from cross validation experiments.

(18)

Influence of parameter setting on mining performance of the proposed algorithm.

According to Eq (7), and , and , where is the variance of Gaussian observation noise = . The cross validation method and empirical parameters of can be obtained according to the experimental data. In the experiment, the influence of λ = 0 = 0.01, 0.01, 0.05, 0.1, 0.5 on RMSE is dynamically considered. During the experiment, the decomposition dimension is taken as 5 and the result is iterated for 50 times. The experimental results are shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Influence of parameter setting on mining performance.

https://doi.org/10.1371/journal.pone.0288140.g002

It can be seen from Fig 2 that when λ∈[0.05,0.1], the square root error of the model is obviously lower. In the following experiments, λ = 0.1 is taken as the optimal value.

Impact of decomposition dimension and iteration times on mining performance.

In the experiment, the influence of decomposition dimension and iteration times is dynamically considered. The iteration times are 10, 20, 30, 40 and 50 respectively, and the decomposition dimensions are 3, 6, 10 and 20. The experimental results are shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Impact of decomposition dimension and iteration times on mining performance.

https://doi.org/10.1371/journal.pone.0288140.g003

As can be seen from Fig 3, when the decomposition dimension is unchanged, RMSE decreases slightly with the increase of iteration times; When the number of iterations does not change, the decomposition dimension first decreases and then increases. When the decomposition dimension is, 20, the minimum RMSE is obtained. Therefore, the number of iterations is 50, and when the decomposition dimension is 5, the minimum RMSE is obtained.

Analysis of mining accuracy.

In order to further verify the mining effect of the proposed method, it is compared with the method in literature [6] and the method in literature [7] to test the mining accuracy, and the comparison results are shown in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Comparison of mining accuracy.

https://doi.org/10.1371/journal.pone.0288140.t004

According to the analysis of Table 4, the mining accuracy of the method in this paper is above 0.92, and that of the two literature methods is below 0.84. Compared with the two literature methods, the mining accuracy of the method in this paper is higher.