Quantifying Discrepancies in Opinion Spectra from Online and Offline Networks

Online social media such as Twitter are widely used for mining public opinions and sentiments on various issues and topics. The sheer volume of the data generated and the eager adoption by the online-savvy public are helping to raise the profile of online media as a convenient source of news and public opinions on social and political issues as well. Due to the uncontrollable biases in the population who heavily use the media, however, it is often difficult to measure how accurately the online sphere reflects the offline world at large, undermining the usefulness of online media. One way of identifying and overcoming the online–offline discrepancies is to apply a common analytical and modeling framework to comparable data sets from online and offline sources and cross-analyzing the patterns found therein. In this paper we study the political spectra constructed from Twitter and from legislators' voting records as an example to demonstrate the potential limits of online media as the source for accurate public opinion mining, and how to overcome the limits by using offline data simultaneously.


Introduction
The proliferation of online social media services such as Twitter is widely recognized as signifying a revolution in how we utilize information and understand the way our society communicates.It also presents an opportunity open and accessible to any interested party to utilize the massive data accrued from such services for understanding the world as well as ourselves [1,2].Among many attempts to harness the potential of online social media, a prominent one is to mine the opinions and sentiments of the public for a variety of purposes, either academic or commercial [3][4][5][6].The use of social media for such purposes poses a potentially critical problem, however: Since online social media users constitute an uncontrolled, thus likely unfair, sample of the population unlike in well-designed opinion polls, it is unclear to what extent the online sphere accurately represents the offline world.User profiles collected by online social media and made available to researchers frequently fall short of giving a sufficiently accurate demographic information about the users, and the traditional polling techniques using questionnaires have to be used to augment them [7].The ideal case would be that the demographics of online users were identical to that of the offline public at large, guaranteeing the agreement of the opinions mined from the online sphere and with traditional polls.A less ideal, but a manageable case would be when the online demographics are precisely known, allowing us to employ straightforward calibrations to find accurate results.But for a limited number of cases the demographic differences between online service users and non-users are not precisely known [8], and given the breadth of topics discussed online and the difficulty of accurately estimating the number of users it does not seem realistic that such a state can ever be reached.
To those looking to mine for information from social media as a proxy for conducting polling, the potential disparity between the online and the offline world due to the unknowable demographics raises serious doubts on the usefulness of online social media, particularly where the offline reality is essential such as politics.In such cases the lack of a systematic method by which to compare the online and the offline would make finding relevant, useful information from online data infeasible.When researchers study such problems, they could circumvent this issue by restricting the scope of their study to the online sphere so that by design the real-world public at large is of little concern.This may be appropriate when the issue being studied is pertinent only to the online world; for instance, when the issue affects the online sphere only, or when one declares that their only interest is in the opinions of the online sphere.While certain scientific discoveries can still be made this way since the data themselves are nonetheless novel and interesting, the inability to draw conclusions regarding many issues pertinent to the offline potentially cripples the potential of online social media.
In order to overcome such a problem, it is important to identify the discrepancies between the online and the offline quantitatively.One way of achieving that is to apply a common analytical and modeling framework to comparable data sets representing the online and the offline so that we can perform a cross-analysis of the patterns found therein.In this paper we demonstrate the process using as an example the landscape of political spectra (signifying the political leanings or positions) constructed from distinct online and offline data of the US and South Korea, two nations with modern representative democracies.Specifically we study the political spectra of the legislators in each country, constructed in two ways.First, we construct a spectrum from the Twitter followership network, which we take as representing the online political landscape.There are some direct evidence that connection on social networks are biased towards those with similar political viewpoints.In a well-known study, Adamic and Glance showed a clear partisan divide on the blogosphere [9].Regarding Twitter, Parmalee and Bichard showed that over 70% of those espousing strong conservative or liberal ideology completely avoided following political leaders who challenge their beliefs [10].In the case of Korea (one of the two countries studied in this work), Chang found that both politicians and ordinary Twitter users exhibit a systematic relationship between their political choices and their position in the Twitter network [11].Second, we construct a spectrum from the legislators' roll call (voting) records, which we take as representing the offline political landscape that more closely reflects the true public one, since the legislators' voting is likely to be influenced by the offline public.The cross-examination of the two spectra should then reveal the differences between the online and the offline.

Data
We utilized the following four data sets: (i) The Twitter followership network of the members of the 111th U.S. Senate; (ii) the roll call of the Senators on legislative bills; (iii) The Twitter followership network of the members of the 18th Korean National Assembly; and (iv) the roll call of the Assembly members between the years 2008 to 2010 (covering three quarters of the term).The Twitter data are current as of June 2011, and Twitter users who followed two or more legislators were included in the data.The data contain all legislators who owned an official Twitter account at the time of data collection and maintained their seat during the entire term considered.This resulted in 67 senators (out of 100), 698 roll calls, and 139 806 Twitter users for the US, and 194 assembly members (our of 299), 1 119 roll calls, and 124 341 Twitter users for Korea.Both countries have two-party systems, the Republican Party (GOP) and the Democratic Party (DEM) that account for 99% of the US Senate, and the Grand National Party (GNP) and the Korean Democratic Party (DP) that account for 86% of the Korean National Assembly.[12] The data were modeled as bipartite networks.See Figure 1 for the schematic illustrations of the data.

Online and offline political spectra
The data introduced above can be modeled as bipartite networks on which the political spectra are constructed using matrix decomposition methods [13,14].In this study we use Classical Multidimensional Scaling (CMDS) [15,16].The method requires a dissimilarity function that act as the distance between two legislators.We use the Kulczynski dissimilarity [17].In essence, CMDS regards the dissimilarity values as the Euclidean distances between legislators in 1-dimensional space and determines the coordinates of the legislators that most closely matches the given distances.We then interpret the coordinates as the political positions of the legislators.Once the spectra of the legislators are determined this way using the Twitter network, we can construct the spectra of the Twitter users by averaging the positions of the legislators that each user follows (see the Methods section for more details).We consider the offline, roll call-based spectra as revealing the true political positions of the legislators as they results from their actual actions of voting.Furthermore, we accept the roll call-based spectra as more accurately reflecting true political positions of the offline public and the society, as the legislators' actions are likely to be heavily influenced by public interests.
The political spectra of the legislators thus constructed are shown in Figures 2A and 2B for the U.S., and in Figures 2C and 2D for Korea.The spectra of the Twitter users (the average of the positions of the legislators that they follow) are shown as the solid lines in Figures 2A and 2C.
A close examination of these spectra can reveals the disparities between the online and the offline.First, we find a stark indication of the significance of the disparity between Twitter and the offline in the majority-minority reversal in both countries: The majority of Twitter users align themselves with the minority parties.The mean and the median positions, and the overall distribution of the spectra are  A and C) show a majority-minority reversal, more people aligning themselves with the minority parties.
shown in Figure 2A for U.S. and Figure 2C for Korea.We find that in the US, Republican Senators John McCain (the presidential candidate in 2008) and Jim DeMint (of the populist Tea Party movement) boasting a disproportionately large number of followers are primarily responsible for this behavior.In Korea, the DP (green) overwhelms the GNP (blue) in the Twitter sphere despite being outnumbered by a nearly two-to-one margin in the number of seats in the Assembly.
Second, the comparison between the roll call-based spectra of the legislators as the proxy for the offline public at large and the spectra of the Twitter users also reveals a similar disparity.Although we cannot directly compare the online and offline spectra on the same plot because of the differences in scale, the comparison of relative biases in the distribution is still valid.For the U.S., the spectrum of the Twitter users is heavily skewed towards GOP in sharp contrast to the DEMs who are favored in the spectrum of the offline public (Figures 2A and 2B).The trend is similar in the Korean spectra: the offline public favors GNP in contrast to the Twitter user who favor DP (Figures 2C and 2D).
Third, there also exists a discrepancy between the real political positions of the legislators and their positions constructed from the Twitter data.The party-line split in the positions of the legislators (indicated as solid shapes in the figures) is immediately noticeable for both nations, and accordingly the roll call-based spectra and the Twitter-based spectra of the legislators exhibit some degree of overall agreement (similarity) as measured by Kendall's τ -coefficient (1 means identity; 0 means independence), with 0.68 ± 0.02 for US and 0.48 ± 0.01 for Korea (see Methods section for more detail).When we consider each party separately, however, we find a weaker agreement -for the US parties, the τ is 0.463 ± 0.003 for GOP and 0.291 ± 0.022 for DEM.For the Korean parties that is 0.029 ± 0.005 for GNP and 0.062 ± 0.051 for DP.
This indicates that Korean Twitter population is more noticeably impervious to the political positions of the legislators, resulting in the very low level of intra-party correlations between the Twitter based spectrum and the roll call based spectrum of the legislators.In US the correlation is more sizable.
These disparities between the Twitter users and the offline public show the limits of Twitter as a source for fair and accurate opinion mining, especially pertaining to political or social issues.In fact, the majority-minority reversal shows us that Twitter may be functioning as a de facto "alternative media."

Quantifying partisanship of twitter users
The spread structures of the Twitter users' spectra in Figures 2A and 2C imply that Twitter users are biased (show partisanship) in choosing which legislators to follow.We can quantify the strength of a user's bias by measuring the probability that a user follows certain parties, and comparing it with the random probability that a user becomes connected to a party if the connections were made at random.If a user is unbiased they must be identical, and deviations between the two probabilities signify the strength of the bias.
A widely-used measure for the discrepancy between two probability distributions is the Kullback-Leibler (KL) divergence D. It is non-negative, i.e.D ≥ 0; 0 when two distributions being compared are identical, and positive otherwise.To account for random fluctuations we calculate the Z-score of KL divergences (see Methods section for detail) Z D as a function of the number of politicians (degree) k that a user follows, shown in Figure 3. Z D 1 implies partisanship, while Z D −1 implies the opposite.The radius of the disc around each plot point is proportional to the the log of the number of users with the given degree k.The figure shows the widespread nature of partisanship on Twitter [9], with the exception of high-degree Twitter users who are more balanced in following each party (see Methods section for more detail).

Discussion
In this paper we proposed a method to compare the political spectra of the online and offline public.We found significant discrepancies between the online and the offline, demanding caution when one tries to use online social media such as Twitter as source for fair and accurate opinion mining.A quantitative measurement of political biases found widespread partisanship among Twitter users.
While our work shows the potential limits of the correctness of online social media-based findings, it is still meaningful that were able to obtain the political spectra of Twitter users based on their behavior observed objectively.This enables us to overcome uncertainty in methods based on self-assessment of political positions [18][19][20] caused by the lack of objective, common measure of political positions agreed upon by people.In our method, however, a person's political position is determined from the objective behavior of all other people in the data, allowing us to construct global and objective political spectra.
(k) (k) Figure 3.The Z-score of the Kullback-Leibler (KL) divergence as a function of the number of legislators k that a Twitter user follows (A) in the US, and (B) in Korea.The radius of each circle is proportional to the log of the number of Twitter users with given k.Z D 1 implies a statistically significant level of partisanship (political bias).We see that partisanship is widespread in both countries, except for high-degree Twitter users for whom D → 0.

Classical multi-Dimensional scaling
CMDS is a method to embed a set of objects in a pre-defined N -dimensional space such that the distance between objects match given dissimilarities as much as possible.Specifically, CMDS aims to produce the coordinates of the objects that minimize a loss function called STRAIN [15].In this paper we set N = 1 to produce a linear political spectrum that agrees with the common way of thinking of political polarization (e.g.left wing versus right wing).
CMDS is performed by the eigendecomposition of the matrix made from the dissimilarities.The quality of the resulting coordinates in reproducing the given dissimilarities is defined as the fraction of the first eigenvalue over the sum of all positive eigenvalues.The values are 0.297 for Korea and 0.872 for US with the roll call data, and 0.113 for Korea and 0.093 for US with the Twitter data.The noticeably high value for the US roll call data may be due to the stable two-party system of the US.
While one could theoretically gain more information from higher-dimension CMDS, we could not observe meaningful patterns allowing straightforward political interpretation beyond the first.Our We used the Kulczynski dissimilarity [17].In Twitter data it is given between two legislators i and j as δ T (i, j) = 1 − 1 2 where F x denotes the set of the followers of legislator x.The Kulczynski dissimilarity is recommended as an alternative to the commonly used Jaccard measure when the size of F x exhibits a wide range as in our Twitter data (Figure 4) [21].
The dissimilarity between legislators i and j in the roll call data is, similarly, δ R (i, j) = 1 − 1 2 where V x denotes the set of nay (opposition to a bill's passing) votes of legislator x.We ignore the yea votes here.The motivation for this is that most bills that pass are bipartisan, potentially limiting the discriminating power of the yea votes.and Z D (k) 1 indicates that Twitter users follow a biased set of legislators, that the bias is larger than the typical random fluctuation.Similarly, Z D (k) −1 means that Twitter users follow the parties in close agreement with the reference probability P i .As k approaches the actual number of legislators p i,j converges to P i , Z D (k) becomes more negative (since D → 0).

Figure 1 .
Figure 1.(A)The schematic illustrations for the data employed in our study.The legislators are included in two distinct bipartite networks.On the left is the legislator-Twitter user network, and on the right the legislator-legislative bill network.Of the two types of edges -a 'nay' vote and a 'yea' votewe consider the 'nay', since they are believed to carry more information in determining the legislators' political spectra (see the Method section for more detail).The structure of each bipartite network can reveal differences in political positions of the legislators, which is the origin of the online-offline discrepancy.Here, for example, the upper two legislators occupy similar positions that are different from that of the the lower one since their follower sets are disjoint.The two groups' voting patterns may show less clear differences.In this study we use Multidimensional Scaling (MDS) and Kendall's ranking correlation coefficient to quantify the spectra and their discrepancies.(B) A sample of the US senator-Twitter followership network consisting of ten legislators and their Twitter followers.The red squares are Republican (GOP) senators, and the blue squares are the Democratic (DEM) senators.All other nodes are Twitter users.

Figure 2 .
Figure 2. The political spectra determined from Twitter and roll call data.(A) The Twitter-based spectrum of the legislators, and the spectrum of Twitter users in the US; (B) The roll call-based spectrum of the legislators in the US; (C) The Twitter-based spectrum of the legislators, and the spectrum of Twitter users in Korea; (D) The roll call-based spectrum of the legislators in South Korea.The legislators are indicated by the shapes (one shape corresponds to one legislator) and the spectra of the Twitter users are indicated by solid lines.The party-line splits of the legislators are evident in both the Twitter-and the roll call-based spectra for both countries.The general Twitter public (solid lines in FiguresA and C) show a majority-minority reversal, more people aligning themselves with the minority parties.

Figure 4 .
Figure 4. (A) The distribution of the number of Twitter followers for each legislator.(B) The distribution of the number of legislators that Twitter users follow.All distribtuions are heavy-tailed.