Link Prediction in Complex Networks: A Mutual Information Perspective

Topological properties of networks are widely applied to study the link-prediction problem recently. Common Neighbors, for example, is a natural yet efficient framework. Many variants of Common Neighbors have been thus proposed to further boost the discriminative resolution of candidate links. In this paper, we reexamine the role of network topology in predicting missing links from the perspective of information theory, and present a practical approach based on the mutual information of network structures. It not only can improve the prediction accuracy substantially, but also experiences reasonable computing complexity.


Introduction
Link prediction attempts to estimate the likelihood of the existence of links between nodes based on the available network information, such as the observed links and nodes' attributes [1,2]. On the one hand, the link-prediction problem is a long-standing practical scientific issue. It can find broad applications in both identifying missing and spurious links and predicting the candidate links that are expected to appear with the evolution of networks [1,3,4]. In biological networks (such as protein-protein interaction networks [5] and metabolic networks [6]), for example, the discovery of interactions is usually costly. Therefore, accurate prediction is more reasonable compared with blindly checking all latent interaction links [3,4]. In addition, the detection of inactive or anomalous connections in online social networks may improve the performance of link-based ranking algorithms [7]. Furthermore, in online social networks, very promising candidate links (non-connected node pairs) can be recommended to the relevant users as potential friendships [8,9]. It can help them to find new friends and thus enhance their loyalties to the web sites. In ref. [9], the authors even proposed the potential theory to facilitate the missing link prediction of directed networks. The hypothesis can find broad applications in friendship recommendation of largescale directed social networks, such as Twitter, Weibo and so on. On the other hand, theoretically, link prediction can provide a useful methodology for the modeling of networks [10]. The evolving mechanisms of networks have been widely studied. Many evolving models have been proposed to capture the evolving process of real-world networks [11][12][13][14]. However, it is very hard to quantify the degree to which the proposed evolving models govern real networks. Actually, each evolving model can be viewed as the corresponding predictor, we can thus apply evaluating metrics on prediction accuracy to measure the performance of different models.
Therefore, link prediction has attracted much attention from various scientific communities. Within computer society, for example, scientists have employed Markov chains [15,16] and machine learning techniques [17][18][19][20][21] to extract features of networks. These methods, however, depend on the attributes of nodes for particular networks such as social and textual features. Obviously, the attributes of nodes are generally hidden, and it is thus difficult for people to obtain them [2].
Over the last 15 years, network science has been developed as a novel framework for understanding structures of many real-world networked systems. Recently, a wealth of algorithms based on structural information have been proposed [2,4,[22][23][24][25][26][27][28]. Among various node-neighbor-based indices, Common Neighbors (CN) is undoubtedly the precursor with low computing complexity. It has also been revealed that CN achieves high prediction accuracy compared with other classical prediction indices [25]. CN, however, only emphasizes the number of common neighbors but ignores the difference in their contributions. In this case, several variants of CN to correct such a defect were put forwarded. Consider, for example, Adamic-Adar [24] and Resource Allocation [25], in which low-degree common neighbors are advocated by assigning more weight to them. In addition, based on the Bayesian theory, a Local Nave Bayes model [27] was presented to differentiate the roles of neighboring nodes. Furthermore, node centrality (including degree, closeness and betweenness) was applied to make neighbors more distinguishable. Besides such CN-based indices, the evolving patterns and organizing principles of networks can also provide useful insights for coping with the link-prediction problem. The well-known mechanism of preferential attachment [11], for instance, has been viewed as a prediction measure [25,29]. For networks exhibiting hierarchical structure, Hierarchical Random Graph can be employed to predict missing links accordingly [4]. Recently, communities have been reinvented as groups of links rather than nodes [30]. Motivated by the shift in perspective of communities, Cannistraci et al. developed the localcommunity-paradigm to enhance the performance of classical prediction techniques [28].
All the aforementioned methods aim to quantify the existence likelihood of candidate links. In information theory, the likelihood can be measured by the self-information. In this article, we thus try to give a more theoretical analysis of the link-prediction problem from the perspective of information theory. Then a general prediction approach based on mutual information is presented accordingly. Our framework outperforms other prediction methods greatly.

A Mutual Information Approach to Link Prediction
We here introduce the definitions of the self-information and of the mutual information, respectively.
Definition 1 Considering a random variable X associated with outcome x k with probability p(x k ), its self-information I(x k ) can be denoted as [31] I where the base of the logarithm is specified as 2, thus the unit of self-information is bit. This is applicable for the following if not otherwise specified. The self-information indicates the uncertainty of the outcome x k . Obviously, the higher the self-information is, the less likely the outcome x k occurs. Definition 2 Consider two random variables X and Y with a joint probability mass function p(x,y) and marginal probability mass functions p(x) and p(y). The mutual information I(X ; Y ) can be denoted as follows [32]: Hence, the mutual information I(x k ; y j )~I(X~x k ; Y~y j ) can be obtained as The mutual information is the reduction in uncertainty due to another variable. Thus, it is a measure of the dependence between two variables. It is equal to zero if and only if two variables are independent. Now consider the link-prediction problem. Our idea is to use the local structural information to facilitate the prediction. To do that, we denote the set of node x's neighboring nodes by C(x). For node pair (x,y), the set of their common neighbors is denoted as O xy~C (x)\C(y).
Given a disconnected node pair (x,y), if the set of their common neighbors O xy is available, the likelihood score of node pair (x,y) is defined as where I(L 1 xy DO xy ) is the conditional self-information of the existence of a link between node pair (x,y) when their common neighbors are known. According to the property of the selfinformation, the smaller I(L 1 xy DO xy ) is, the higher the likelihood of existence of links is. Thus, we define the score as the negation of I(L 1 xy DO xy ). According to the definition of mutual information, I(L 1 xy DO xy ) can thus be derived as where I(L 1 xy ) is the self-information of that node pair (x,y) is connected. I(L 1 xy ; O xy ) is the mutual information between the event that node pair (x,y) has one link between them and the event that the node pair's common neighbors are known. Note that I(L 1 xy ) is calculated by the prior probability of that node x and node y are connected. In our method, without knowing the common neighbors of node pair (x,y), we could use I(L 1 xy ) to estimate the existence of a link between node pair (x,y). I(L 1 xy ; O xy ) indicates the reduction in uncertainty of the connection between nodes x and y due to the information given by their common neighbors. Since the mutual information plays a significant role in our method, this framework is called MI for short.
If the elements of O xy are assumed to be independent of each other, then Here I(L 1 xy ; z) can be estimated by I(L 1 ; z), which is defined as the average mutual information over all node pairs connected to node z Now we try to calculate the above mutual information. According to its definition (3), I(L 1 mn ; z) can be denoted as where I(L 1 mn Dz) is the conditional self-information of that node pair (m,n) is connected when node z is one of their common neighbors, and I(L 1 mn ) denotes the self-information of that node pair (m,n) has one link. The right-hand side of eq. (8) is composed of the (conditional) self-information. Based on the definition of (conditional) self-information, it can be calculated based on the (conditional) probability. The conditional probability p(L 1 mn Dz) can be estimated by the clustering coefficient of node z, defined as where N Dz and N^z are the numbers of connected and of disconnected node pairs with node z being a common neighbor, respectively. Once p(L 1 mn Dz) is available, I(L 1 mn Dz) can be calculated.
In order to calculate the probability p(L 1 mn ), we assume that no degree-degree correlation is considered. When nodes' degrees are known, the probability that node pair (m,n) is disconnected is derived as where k m and k n are the degrees of nodes m and n, respectively. M is the total number of links in the training set. Obviously this formula is symmetric, namely Thus, and I(L 1 nm ) can be calculated accordingly. Collecting these results, we can obtain where I(L 1 xy ; z) and I(L 1 xy ) can be calculated by eqs. (13) and (12) respectively.
To facilitate the understanding of MI, we illustrate it with an example as shown in fig. 1. First, consider node u1, for example, which is the common neighbor of nodes u2, u3 and u4. Using eq.  5). That is to say, node pair (u2,u3) is more likely to be connected than node pair (u3,u4). The six prediction methods mentioned in section ''Previous Prediction Methods'', however, cannot distinguish these two node pairs. In this sense, MI has higher discriminative resolution than them. Second, MI can distinguish node pairs even if they all have no common neighbors. For instance, I(L 1 u3u5 )~log 10 3~1 :7370 and I(L 1 u3u8 )~log 5~2:3219. That is to say, node pair (u3,u5) is more likely to be connected than node pair (u3,u8). This is undoubtedly beyond the distinguishing ability of previous methods. Thirdly, the mutual information of node u6 can be calculated as I(L 1 ; u6)~I(L 1 ; u7)~0:1854. Thus I(L 1 u5u8 DO u5u8 )~0:5361. We note that I(L 1 u5u8 DO u5u8 )vI(L 1 u2u3 DO u2u3 ), namely, node pair (u5,u8) with two common neighbors has higher connection likelihood compared to node pair (u2,u3) with only one common neighbor. This is in agreement with our intuition very well. Lastly, different nodes may provide different mutual information to reduce the uncertainty of connections. The extent to which node u6 (I(L 1 ; u6)~0:1854) contributes to the reduction of link uncertainty, for example, is greater than that of node u1 (I(L 1 ; u1)~0:0703).

Experimental Results
In this section, we compare our MI approach with other six representative prediction indices which are introduced in section ''Previous Prediction Methods''. Tables 1 and 2 show the prediction accuracy measured by AUC and precision, respectively. The overall prediction performance of MI outperforms them greatly. Table 1 demonstrates that for AUC, MI model gives much higher prediction accuracy than all 6 other indices for real-world networks except network Grid. Especially for networks EPA and INT, AUC of six indices is all around 0.6. MI model can experience AUC of more than 0.9. Such great difference may arise from that previous methods can't distinguish those node pairs without common neighbors. Unfortunately, the lack of common neighbors between two nodes often appear in real-world networks. For example, more than 99% of node pairs in network INT have no common neighbors. But MI approach is able to discriminate them greatly. Another finding is that CAR-based indices (CAR and CRA) achieve the worst prediction performance for ten Table 2. Comparison of the prediction accuracy measured by precision (top-100) on ten real-world networks. September 2014 | Volume 9 | Issue 9 | e107056 ∈ networks. Actually, for node pairs with few common neighbors, the distinguishing ability of CAR-based indices degenerates remarkably due to their emphasis on the links among common neighbors. For example, all node pairs with less than two common neighbors share the same connection likelihood because they all have no links among common neighbors. Table 2 shows the comparisons of precision for ten real-world networks. We can see that MI is much better than CN, RA, LBN-CN, and LNB-RA for all networks. CAR-based indices, however, achieve higher precision than MI for some networks. The efficiency of CAR-based indices in predicting top-ranked candidate links is very high for networks with notable link communities. Consider, for example, network Wikivote with high average degree, in which CAR-based indices overwhelmingly win MI and other methods. Obviously, the extent to which CAR-based indices excel MI is positively related to link communities. The computing complexity of CAR-based indices, however, depends on the density of networks greatly.
It is thus necessary to compare the computing complexity of CAR-based indices and our MI model. Here the average degree is denoted as SkT. According to eq. (23), the time complexity of computing c(z) and O xy is O(SkT 4 ) and O(SkT 2 ), respectively. The total computing complexity of CAR is thus O(N 2: SkT 6 ). Similarly to CAR, the computing complexity of CRA is also O(N 2: SkT 6 ) because C(z) has the computing complexity of O(1) based on eq. (24). For MI, the computing complexity of I(L 1 mn ) and averaging all neighboring node pairs of node z is both O(SkT 2 ). Thus, I(L 1 ; z) has the computing complexity of O(SkT 4 ). The computing complexity of MI model can be derived as O(N 2: SkT 4 ) accordingly. Taking precision and the computing complexity of CAR-based indices together, we note that they outperform MI in some networks but with the computing complexity as SkT 2 times as that of MI. It is intolerable especially for networks with the high average degree.
We also conduct experiments on an ASUS RS500-E6-PS4 workstation with 16 GB RAM and a Inter (R) Xeon (R) E5606 @ 2.13 GHz quad-core processor. The detailed comparison of computational time on ten real-world networks is summarized in Table 3. The results indicate that the MI index overwhelms CARbased methods while remains similar time scale to other CN-based methods.
Altogether, MI has a good tradeoff among AUC, precision and the computing complexity.

Discussion
In this paper, we develop a novel framework to uncover missing edges in networks via the mutual information of network topology. Note that our approach differs crucially from previous prediction methods in that it is derived from the information theory. We compare our model with six typical prediction indices on ten networks from disparate fields. The simulation results show that MI model overwhelms them. Furthermore, we compare the computing complexity of MI model with that of CAR-based indices and find that our approach is less time-consuming.
Notice that the calculation of the mutual information depends on the assumption that the network is free of assortativity. However, we find that MI method performs very well not only in uncorrelated networks but also in networks with high assortativity coefficient such as PB, Yeast and EPA. Actually, the assortativity coefficient refers to the global network-level property [33] as showed in Table 4, which can't convey sufficient property about local structure. Considering that our method is mainly focusing on Table 3. Comparison of the computational efficiency of seven methods on ten real-world networks. the neighbors of two nodes, we utilize local assortativity [34,35] to explain such a phenomenon. For a network with N nodes and M links, its excess degree (which is equal to the node's degree minus one) distribution is denoted as q(k). Then, the local assortativity of node v is defined as [35] r v~j (jz1)( where j is node v's excess degree, k k denotes the average excess degree of node v's neighbors, m q is defined as the expectation of distribution q(k) and s q is the standard deviation of distribution q(k). Based on this definition, the sum of all nodes' local assortativity is equal to the network assortativity coefficient. Fig. 2 shows the cumulative distribution function of nodes' local assortativity. We find that i) both locally assortative and disassortative nodes exist regardless of the network-level assortative mixing pattern; ii) most nodes do not show the local assortative property, which is coincident with our assumption. Since our method is related to the local assortativity rather than the global one, it can achieve good prediction performance even in those globally correlated networks.

Data and Problem Description
In this article, in order to better capture the statistical perspective of our method, we choose ten example data sets from various areas with the size of its giant component being greater than 1000. They are listed as follows. i) Email [37]: A network of Alex Arenas's email. ii) PB [38]: A network of the US political blogs. iii) Yeast [39]: A protein-protein interaction network. iv) SciMet [40]: A network of articles from or citing Scientometrics. v) Kohonen [40]:  Table 4.
In this paper, only an undirected simple network G(V ,E) is studied, where V and E are the sets of nodes and of links, respectively. That is to say, the direction of links, self-connections and multiple links are ignored here. The framework of prediction indices can be described as follows [2]. Given a disconnected node pair (x,y), where x,y V, we should try to predict the likelihood of connectivity between them. For each non-existent link (x,y) U{E, where U represents the universal set, a score s xy will be given to measure its existence likelihood according to a specific predictor. The higher the score is, the more possible the node pair has a candidate link. To figure out the latent links, all disconnected ones are first sorted in the descending order. The top-ranked node pairs are believed most likely to have links.
To validate the prediction performance of the algorithms, the observable links of the network are divided into two separate sets, i.e., the training set E T and the probe set E P . Obviously, E T is the available topological information, and E P is for the test and thus cannot be used for prediction. Therefore, E T |E P~E and E T \E P~= 0. In our model, the training set E T and probe set E P are assumed to contain 90% and 10% of links, respectively (see the review article [2] and references therein). As in many previous papers, two widely used metrics are adopted to evaluate the performance of prediction algorithms [2]. They are AUC (area under the receiver operating characteristic curve) [46] and precision [47]. AUC is denoted as follows: where among n times of independent comparisons, n' and n'' represent the time that a randomly chosen missing link has a higher score and the time that they share the same score compared with a randomly chosen nonexistent link, respectively. Clearly, AUC should be around 0:5 if all scores follow an independent and identical distribution. Therefore, as a macroscopic accuracy measure, the extent to which AUC exceeds 0.5 indicates the performance of a specific method compared with pure chance. Another popular measure is precision, which focuses on topranked latent links. It is defined as L r =L, where among top-L candidate links, L r is the number of accurate predicted links in the probe set.

Previous Prediction Methods
We here introduce six typical methods based on common neighbors. They are Common Neighbors (CN), Resource Allocation (RA) [25], the Local Naïve Bayes (LNB) forms of CN [27] and RA [27], CAR [28] and CRA [28], respectively. N CN. This method is the natural framework in which the more nodes x and y share common neighbors, the more likely they are connected. The score can be quantified by the number of their common neighbors, namely In this formula, g is denoted as In addition, R z is defined as where N Dz and N^z are as same as those in eq.
where c(z) refers to the subset of neighbors of node z that are also common neighbors of nodes x and y. is denoted as the percent of nodes with the local assortativity value not larger than r. r is the assortativity coefficient of the network which is presented in