Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

A network analysis of the propagation of evidence regarding the effectiveness of fat-controlled diets in the secondary prevention of coronary heart disease (CHD): Selective citation in reviews

  • Rhodri Ivor Leng

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

    rleng@exseed.ed.ac.uk

    Affiliation Department of Science, Technology and Innovation Studies, School of Social and Political Science, University of Edinburgh, Edinburgh, United Kingdom

A network analysis of the propagation of evidence regarding the effectiveness of fat-controlled diets in the secondary prevention of coronary heart disease (CHD): Selective citation in reviews

  • Rhodri Ivor Leng
PLOS
x

Abstract

Objective

To examine how the first randomised controlled trials (RCTs) evaluating the efficacy of cholesterol-lowering diets in the secondary prevention of coronary heart disease were interpreted in reviews of the literature prior to the National Institutes of Health consensus conference in 1984.

Design

Claim-specific citation network analysis was used to study the network of citations between reviews and RCTs over a defined period (1969–1984). RCTs were identified and classified according to whether their conclusions supported or opposed the use of dietary fat modification/restriction in the secondary prevention of coronary heart disease. Each review published in this period that cited any of the RCTs was classified as supportive, neutral, or unsupportive to the use of dietary fat modification based on a quotation analysis of its evaluation of the findings of these RCTs. Citation bias and underutilisation were detected by applying a comparative density measure, in-degree centrality, and out-degree in a series of sub-graph analyses.

Results

In total, 66 unique publications were identified (four RCTs—one supportive, three unsupportive; 62 reviews—28 supportive, 17 neutral, 17 unsupportive). On average, supportive reviews underutilised the available RCTs to a greater degree than other reviews. Amongst the supportive group, citation bias was common—23 (82%) reviews cited only the one RCT that was supportive.

Conclusion

Most reviews that disseminated a supportive evaluation of the results of RCTs in the context of secondary prevention cited only data that supported this position.

Introduction

In 1961, the American Heart Association published guidelines recommending that people who have had one or more atherosclerotic heart attacks should reduce their intake of saturated fatty acids and modestly increase that of polyunsaturated fatty acids [1]. The rationale was an early version of the diet–heart hypothesis, that dietary intake of saturated fatty acids, by increasing total serum cholesterol, contributes to coronary heart disease (CHD) via the development of atherosclerosis, while polyunsaturated fatty acids decrease serum cholesterol and lower the risk of CHD [2]. In the 1950s, this theory was supported by: (i) epidemiological evidence of an association between fat intake, particularly saturated fatty acids intake, and raised serum cholesterol levels; (ii) an association between serum cholesterol and coronary incidence and mortality; (iii) short-term metabolic ward trials that suggested that saturated fatty acids increased serum cholesterol while polyunsaturated fatty acids appeared to modestly lower serum cholesterol [3].

However, some scientists warned of the dangers of mistaking correlation for causation, particularly in population studies [4] and in a multifactorial disease such as CHD [5, 6]. By the time that the guidelines were issued, no randomised controlled trial (RCT) had tested the link between the consumption of different types of fat and coronary risk in secondary prevention (i.e., after a clinical manifestation of CHD), but, in the 1960s, five RCTs examined this. The Los Angeles Veterans Administration Diet Study [7] combined secondary prevention and primary prevention (i.e., before the development of CHD), but was generally regarded as relevant only to primary prevention due to the way the authors stratified their data and presented their findings. Four other trials specifically examined the effect of a modified diet in the secondary prevention of CHD in patients who had experienced a recent myocardial infarction: the Research Committee Low-Fat Diet Study [8] examined total fat restriction, while the Rose Corn Oil Study [9], the Oslo Diet-Heart Study [10], and the Medical Research Council’s Soya-bean Oil Study [11] examined the impact of restricting saturated fatty acids and increasing polyunsaturated fatty acid intake. These were the only published secondary prevention RCTs until the partial publication of the Sydney Diet–Heart Study [12] in 1978, which only published results concerning all-cause mortality. The full results of that trial on cardiovascular disease (CVD) and CHD mortality were recently recovered and re-analysed by Ramsden et al. [13], who found a significant harmful effect of replacing saturated fatty acids with polyunsaturated fatty acids on all-cause mortality, CVD mortality, and CHD mortality.

The four RCTs published in 1965 [8, 9], 1966 [10], and 1968 [11] had similar methodologies, and were published in major journals, but produced discrepant findings (S1S4 Tables). The Oslo study, published in 1966, found significantly lower levels of fatal reinfarction, non-fatal reinfarction, and angina pectoris in the intervention group, but found no significant relationship between diet and sudden death or all-cause mortality [10]. The other three studies all returned equivocal or negative results regarding reinfarction, coronary mortality, cardiovascular events, and all-cause mortality. The author of the Oslo study regarded his results as supporting dietary fat modification in secondary prevention, while the authors of the other studies regarded their results as evidence against such treatments.

In December 1984, the National Institutes of Health (NIH) gathered a panel of experts to resolve whether the link between raised serum cholesterol and heart attack was causal, and under what circumstances dietary or drug therapy should be initiated [14]. After reviewing the available evidence, the panel concluded that an elevated serum cholesterol level is a cause of CHD and that lowering it is protective. To lower serum levels of cholesterol, the panel recommended that all men, women, and children over 2 years of age should eat a diet consisting of no more than 30% fat, should reduce their intake of saturated fatty acids to less than 10% of total calories, and should increase their intake of polyunsaturated fatty acids to 10% of total calories. Specifically, on secondary prevention, the panel concluded:

“Studies are available that indicate a beneficial effect of treating high cholesterol levels in individuals with preexisting clinical disease (secondary intervention) … It is encouraging that the progression of established lesions may be retarded by appropriate dietary and drug therapy” (p.2084)

The conference was widely regarded as bringing a (temporary) end to the controversy over both the relationship of cholesterol with CHD and the relationship between dietary fats and CHD [15].

Yet, by 1984, the four RCTs discussed previously remained the only available RCTs with data on the effect of dietary modification on coronary mortality and morbidity in secondary prevention. However, the results of these trials were conflicting–an evaluation supported both by the directors of each study and by recent meta-analyses [1619]. According to Harcombe et al. [16], the NIH advice was not supported by the available RCTs, neither in primary or secondary prevention. As can be seen in the risk-ratio analysis in the supplement (S4 Table), the evidence from these RCTs available prior to 1984 leads to no clear conclusion as to whether dietary modification is an appropriate treatment in secondary prevention. Recent meta-analyses also agree that, when the findings of these four trials are considered together, no significant difference exists on the “hard” clinical end-points–the incidence of myocardial infarction, fatal CHD, and all-cause mortality–between the intervention and control [1619].

The present study examines how these four trials were utilised in the literature and their impact on published scientific opinion regarding the case for dietary fat modification/restriction in the secondary prevention of CHD, prior to the NIH consensus conference. By using a modified version of claim-specific citation network analysis, a methodology developed by Greenberg [20], it maps and evaluates the network of review papers that cited these RCTs over a defined period (1969–1984); the date by which all relevant RCT data would have been available to reviewers to the start of the NIH conference. The rationale for examining how these studies were cited is the claim that the diet–heart controversy stems from selective citation [21, 22]. In 1992, Uffe Ravnskov, in one of the first published demonstrations of citation bias, examined 22 cholesterol-lowering trials and found that supportive trials were cited almost six times as often as others [21]. In three highly-cited reviews associated with dietary policy [2325], he went on to document citation bias and quotation bias, which describes the erroneous or selective interpretation of experimental findings [22]. Finally, in a response to an article in the BMJ in 2003, he claimed that belief in the diet–heart hypothesis was kept alive by selective citation [26].

The present study tested these claims by examining how dietary RCT results were cited in reviews during the period leading up the publication of dietary guidelines and closure of controversy, following Greenberg’s [20] finding that reviews tend to be the major propagators of information regarding the findings of primary studies. To do so, it examines two outcomes of selective citation: citation bias–the selective citing of data supporting a particular outcome [20]; and research underutilisation–when authors fail to reference all available data relevant to a specific claim. These are outcomes of sub-optimal citation practices that distort the evidence base by inflating the influence of some studies while under-representing others [20, 27]. To detect these factors, network analysis was used to document citation bias and underutilisation by mapping the citation links between RCTs and reviews that were themselves classified by quotation analysis.

Materials and methods

Study selection

To be included, a trial had to be a unifactorial RCT that: (i) was published before 1984; (ii) examined the effect of dietary fat restriction (<30% of total calories derived from dietary fat) OR dietary fat modification (replacing saturated fatty acids with polyunsaturated fatty acids) on coronary benefit/risk in the secondary prevention of CHD; (iii) published data on sample numbers AND serum cholesterol changes AND mortality and morbidity figures regarding incidence of myocardial infarction, coronary mortality (including myocardial infarction and sudden death), and all-cause mortality; (iv) included only patients with established CHD (myocardial infarction, angina pectoris); (v) a minimum intervention and follow-up period of 12 months.

RCTs were identified via a literature search on Web of Science via the ‘all database’ function with the following Boolean string (“dietary fat” OR “fatty acids” OR “saturated fat” OR “low fat diet” OR “cholesterol lowering diet” OR “modified fat diet” OR “restricted fat diet” OR “corn oil” OR “soya bean oil” OR “unsaturated fat” OR “linoleic acid”) AND (“coronary heart disease” OR “ischaemic heart disease” OR “myocardial infarction” OR “cardiovascular disease” OR “atherosclerotic heart disease”) AND (“trial” OR “controlled trial” OR “randomised controlled trial” OR “intervention trial” OR “clinical trial”), limited to publications between 1950 and 1984. This returned 244 publications across four databases in March 2017 (MEDLINE, Web of Science Core Collection, BIOSIS Citation Index, FSTA® - the Food Science Resource). An additional examination of trials identified by recent systematic reviews and meta-analyses was also conducted to ensure all relevant studies were identified [1619]. After screening publication titles and abstracts, 19 trials were identified as potentially relevant. After reading these, four RCTs were identified that met the inclusion criteria, all of which were published in the decade after the publication of the American Heart Association guidelines. The findings of these four RCTs were published in seven papers: the results of the Oslo Trial were published in four journals [10, 2830]. These were merged, reducing the primary source studies to four. The results of these trials are included in the analysis in the supplement (S1S4 Tables).

One trial, the Los Angeles Veterans Administration Diet Study [7], was omitted due to the inability to assess data on the effectiveness of dietary modification in secondary prevention due to the way the authors stratified their data and reported their findings. This was further justified by the prevailing opinion of the scientific community as expressed in reviews, which appeared to regard it relevant only to primary prevention, and its exclusion from meta-analyses covering secondary prevention specifically [18]. Another trial, the Sydney Diet–Heart Study [12], was omitted because only results on all-cause mortality were reported within the time frame and because it remained uncited in the literature by reviews until 1985 [30]. The Finnish Mental Hospital Study [31], a commonly-cited study in this period, was excluded on the basis that it was not an RCT; it was a “cross-over” trial in two mental hospitals and combined secondary and primary prevention. Further, it was not a standard cross-over trial: the patient groups were not the same in the diet arm and the control arm, no blinding was mentioned, and investigators re-coded deaths post-hoc. Bierenbaum et al.’s [32] St Vincent’s Hospital Study was excluded because of the lack of appropriate randomisation due to the assigning of a control group post-hoc that had “similar characteristics” to the intervention. Five trials were excluded on the basis of being primary prevention trials [3335] and/or multifactorial [3637], and six early secondary prevention trials were excluded due to a lack of appropriate randomisation and/or control [3843].

Review papers, published in academic journals in English between 1969 and 1984 and which cited one or more of the studies, were identified by searching Scopus and Web of Science using indexed citation links associated with the relevant RCTs. To be included, a paper had to be a review or an extended editorial (with at least one citation) published in an academic journal within which the coronary mortality and morbidity results of the identified RCTs were discussed. During this period, only narrative reviews covered this topic, and none described a systematic search strategy. Excluded from analysis were letters, notes, book chapters, other primary articles, any paper not published in an academic journal, and any review that cited the RCTs without discussing mortality and morbidity. The search in Scopus returned 110 citing articles, of which 30 met the inclusion criteria. The search on Web of Science returned 215 citing articles, from which an additional 32 reviews were recovered that met the inclusion criteria. In total, 62 citing reviews were included in the analysis [44105].

Classification of papers

The RCTs were classified using ‘quotation analysis’ [20, 22, 27] according to whether the findings were evidence for or against the use of dietary fat modification/restriction in the secondary prevention of CHD. Quotations from the papers were collected to enable papers to be classified by the authors’ interpretation of their findings and these are reproduced in the supplement (S5 Table). For example, the authors of the Rose Corn Oil Study, an unsupportive trial, concluded that “under the circumstances of this trial corn oil cannot be recommended in the treatment of ischaemic heart disease” (p. 1533). By contrast, the author of the Oslo Study, the only supportive trial, concluded that the observed “reduction of the serum cholesterol level associated with a reduced CHD relapse rate strongly suggests a cause and effect relationship” (p. 79).

Reviews were similarly classified by quotation analysis by the authors’ evaluation of the RCT evidence. Quotations were identified either in the passage in which RCTs were cited or were stated explicitly in a conclusion, and these are reproduced in the supplement (S5 Table). Reviews were then classified into one of three categories: supportive reviews contained statements implying that the available trial evidence supported dietary intervention; neutral reviews contained statements highlighting the conflicting nature of the trial results and contained no statement either supporting or opposing dietary intervention; unsupportive reviews contained statements implying that the available trial evidence did not support the use of dietary intervention.

Network data

Each paper was given an identifier code (Vn). All authors, title of paper, year of publication, paper type (RCT or review), and classification were recorded to identify each vertex in a ‘node list’ format (S6 Table). This attribute information was used to examine the relationship between classification variables and citations behaviours. The citations to (incoming edges) and from (outgoing edges) all papers were recorded. The bibliographies of each review were screened to ensure that all citations to RCTs were identified within this sample. Citation information on the relationship between papers was recorded in an ‘edge list’ format in an Excel spreadsheet (S7 Table). This describes a network as a set of vertices (papers) connected by ‘edges’ (citation links). From these data, a graph was constructed and analysed using R’s ‘statnet’ package [106, 107] and visualised in Gephi [108]. Citation information was converted into an adjacency matrix that records edges (citations) between vertices (papers): A graph [G] was constructed of the vertex set [V] and edge set [E]: The vertices were coloured by classification and sized by in-degree, labelled by number assigned in the bibliography of this paper, and visualised initially using Gephi’s ‘ForceAtlas 2’ algorithm [108, 109] and later manually realigned in sub-graphs.

Two outcome measures–citation bias and research underutilisation–were used. Following Greenberg [20], citation bias refers to the act of citing only evidence of a particular classification. For example, if paper A references only supportive studies and no contradictory studies then it displays citation bias. Citation bias only describes the most extreme cases of evidence selection, but any underutilisation may create distortions in the literature–i.e., if reviews fail to use all relevant evidence even if a clear preference for a particular outcome does not influence selection. To detect these, the unit of analysis is the citation pathway, and the occupied citation pathway (an edge linking two vertices) is compared to all possible citation pathways between reviews and RCTs. Three measures are used to measure citation bias and underutilisation–a novel modified density measure, in-degree centrality, and out-degree.

The network in this study is an example of a directed acyclic network [110] because papers can only cite previously published papers and can never cite future literature or cite themselves. To know the density, the proportion of potential edges present, a measure of network connectivity is needed. Generally, if P is a set of primary studies, S a set of reviews, and E the number of edges from S to P, then captures the number of times that a set of reviews cites a set of RCTs as a proportion of the maximum possible number of citations, allowing comparison between different classifications of review. Thus, this equation can be used to understand divergent utilisation of primary data.

In-degree reflects how often a study has been cited. The in-degree of a vertex is the number of edges from other vertices to that vertex, and, for a vertex i, is given by:

Out-degree reflects the number of references from a particular review. The out-degree of a vertex is the number of edges from that vertex to other vertices, given by:

To establish citation bias, the network was divided into three sub-graphs of the RCTs with reviews of each classification (supportive, neutral, unsupportive) and only edges from reviews to RCTs. The vertices that cite only RCTs of a particular classification were then counted–providing an exact account of citation bias and research utilisation.

Finally, a Pearson’s chi square test was performed on the distribution of citations to these four RCTs between the three classifications of review to establish whether the difference in citation behaviour is significant or simply the product of chance.

Results

Of the four RCTs, one was supportive of dietary intervention [10] in the context of secondary prevention, while three were not [8, 9, 11]. Between 1969 and 1984, these were cited in 62 reviews, 28 of which were supportive of dietary intervention [78105], 17 were unsupportive [6177], and 17 ‘neutral’ reviews came to no clear conclusion on this issue [4460]. Of these, 28 were published in the first 7 years (12 supportive; 12 neutral; four unsupportive), and 34 in the next 8 years (16 supportive; five neutral; 13 unsupportive).

Fig 1 shows the citation network. Of the four RCTs, the most cited was the Oslo study with 57 citations. This was the only one of the four RCTs that was supportive of dietary intervention, but the other three (unsupportive) RCTs together attracted a similar number of citations (32 to MRC Soya-bean, 15 to Rose Corn, 17 to Research Committee Low-fat). Here, citation bias, particularly by supportive reviews, is signified by the many studies that cite only the Oslo study. Clearly, the distribution of citations to these studies is uneven. A high proportion of the reviews (27 of 62; 43%) cited only one of the RCTs, 20 cited two of them, six cited three and only nine cited all four. Nevertheless, taking the three unsupportive trials together, supportive trials are cited 57 times and unsupportive trials 64 times. Overall the graph has a density D of 0.49 –indicating only 49% of the possible citations exist between reviews and these RCTs.

thumbnail
Fig 1. Citation network mapping the citations from reviews to RCTs testing dietary fat modification in the secondary prevention of CHD (n = 66; m = 121).

Citation network graph with four RCTs connected to 62 citing reviews by a total of 121 citations. Each vertex represents a paper and the integer label corresponds to a paper in this paper’s bibliography. The colour of a vertex corresponds to a study’s classification (blue is supportive, green is neutral, and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link and the direction of the edge is established by the direction of the arrow head. Visualised with Gephi’s ‘Force Atlas 2’ algorithm [109].

https://doi.org/10.1371/journal.pone.0197716.g001

To create an exact account of the utilisation of the primary RCT data by reviews, three sub-graphs were constructed including only reviews of a particular classification (supportive, neutral, unsupportive) and only edges from reviews to RCTs.

Neutral reviews

Of the neutral reviews (Fig 2), one cites only supportive data, which represent the only case of citation bias in this group. Overall the density of this sub-graph (D = 0.68) is higher than that of the network as a whole, indicating that reviews which cited more of the available primary literature were more likely to come to no clear conclusion than the average. While the Oslo Study received most citations (17), its influence is nearly rivalled by the MRC Soya-bean Study, which received 15; the Rose Corn Oil Study and the Research Committee Low-fat Study were each cited seven times. Six of the reviews cited all four RCTs and only one cited only one of them. Thus, compared to the whole network, neutral reviews were more likely to fully utilise the available RCT evidence and less likely to cite only one RCT.

thumbnail
Fig 2. Sub-graph mapping the citations from neutral reviews to RCTs testing cholesterol lowering diets in the secondary prevention of CHD (n = 21; m = 46).

This graph is composed of four RCTs connected to 17 citing reviews by a total of 46 citations. Each vertex represents a paper and the integer label corresponds to a paper in this paper’s bibliography. The colour of a vertex corresponds to a study’s classification (blue is supportive, green is neutral, and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link, the direction of the edge is established by the direction of the arrow head, and the colour of an edge indicates the classification of the cited article. This was visualised manually in Gephi [108] by lining up reviews in the centre, placing supportive the RCT below and the unsupportive RCTs on top.

https://doi.org/10.1371/journal.pone.0197716.g002

Unsupportive reviews

The sub-graph of unsupportive reviews (Fig 3) had a density of 0.56. The Oslo study and the MRC study were each cited 12 times by unsupportive reviews, the Rose study six times and the Low-fat study eight times. Five of the 17 unsupportive reviews cited only evidence that supported their position. Three reviews cited only the supportive trial, implying a difference of interpretation between the reviewers and the authors of that study. Therefore, in total eight reviews (47%) exhibit citation bias for trials of a particular outcome. Three reviews cited just one RCT and two cited all four; most (nine) cited two of the RCTs.

thumbnail
Fig 3. Sub-graph mapping the citations from unsupportive reviews to RCTs testing cholesterol lowering diets in the secondary prevention of CHD (n = 21; m = 38).

This graph is composed of four RCTs connected to 17 reviews by a total of 38 citations. Each vertex represents a paper and the integer label corresponds to a paper in the reference list. The colour of a vertex corresponds to a study’s classification (blue is supportive and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link, the direction of the edge is established by the direction of the arrow head, and the colour of an edge indicates the classification of the cited article.

https://doi.org/10.1371/journal.pone.0197716.g003

Supportive reviews

Compared with the neutral and unsupportive reviews, the 28 supportive reviews (Fig 4) used much less of the available primary data (D = 0.33 vs 0.56 in unsupportive reviews and 0.68 in neutral reviews). All 28 cited the single supportive RCT, the Oslo study, while the three other RCTs were cited only nine times. In all, 23 of the supportive reviews (82%) cited only the Oslo study, and only five of these reviews cited any of the other RCTs (two cited one other, two cited two others, and just one cited all four RCTs).

thumbnail
Fig 4. Sub-graph mapping the citations from supportive reviews to RCTs testing cholesterol lowering diets in the secondary prevention of CHD (n = 32; m = 37).

This graph is composed of four RCTs connected to 28 citing reviews by a total of 37 citations. Each vertex represents a paper and the integer label corresponds to a paper in the reference list. The colour of a vertex corresponds to a study’s classification (blue is supportive and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link, the direction of the edge is established by the direction of the arrow head, and the colour of an edge indicates the classification of the cited article.

https://doi.org/10.1371/journal.pone.0197716.g004

Table 1 gives the distribution of citations to RCTs from different classifications of review.

thumbnail
Table 1. Contingency table of the distribution of citations from three different classifications of review to four secondary prevention RCTs.

https://doi.org/10.1371/journal.pone.0197716.t001

A Pearson’s chi square test was performed to test whether this distribution is significantly related to classification or the product of chance. The validity of a chi square test depends on the conditions that the expected count is greater than 5 for at least 80% of the cells, and that each cell has a count of at least 1: these conditions were both satisfied. The test result establishes that the difference observed in the citation behaviours between reviews is significant (Χ 2(6, N = 121) = 18.19, p = 0.006).

Discussion

This study examined the propagation of experimental evidence derived from RCTs in reviews of the literature during a period of scientific controversy. This was not a minor academic dispute; this was an issue of considerable public health importance at the time. It documented a major divide between reviews over what the available empirical evidence was, how it should be evaluated, and what the implications were for dietary intervention in the secondary prevention of CHD. Overwhelmingly, reviews that disseminated a supportive evaluation of the results of RCTs in the context of secondary prevention cited only data that supported this position. They did not critique the discrepant data, they simply ignored it, or were ignorant of its existence. Other reviews were overall much more balanced in their citation and this difference in citation behaviour appears to explain divergence of opinion on this matter.

While the Oslo trial, a supportive RCT, was by far the most influential, its influence was not uniform amongst different classifications of review. This was the only RCT to arrive at statistically significant findings. A number of studies have reported evidence of a bias in the literature, in that studies with statistically significant results tend to be cited more often than studies that report non-significant or equivocal results [21, 111114]. However, such a bias for significant outcomes fails to explain the discrepancies between the citing behaviours of different groups.

The degree of utilisation of primary data by reviews appeared to be mediated by a review’s position, with supportive reviews likely to use less of the available RCT evidence than others. Again, this may be explained by a preference for statistically significant findings, but, as discussed, there are problems with this explanation. An alternative hypothesis may be that the shape of the literature itself determines the selection of RCT evidence–where reviewers find evidence from other papers. In this case, if a reviewer believes another paper to have identified all relevant evidence, they may replicate selective citation. This has been described by Greenberg’s information cascade hypothesis–where chains of citations from one paper to another lead researcher’s to only a segment of the available evidence, if those authors simply follow bibliographies [20]. This hypothesis may be supported by Simkin and Roychowdhury’s [115] finding that on average only 20% of the cited literature in bibliographies has been read by the authors. If scientists do tend to gain their information about primary studies by reading other papers rather than the originals, this is particularly worrying in this case due to the amount of bias in the selection of evidence by reviews and the degree of underutilisation. Collectively, the citing reviews studied in this paper have been cited 2,157 times (by August 2017) according to Web of Science (supportive reviews 651; neutral 489; unsupportive 1017); however, without further analysis, no statement can be made about what influence these reviews had on the literature.

Thirdly, citation bias was considerably more prevalent in supportive reviews. This seems to be an example of homophily—the tendency for a vertex to share an edge with a vertex with which it shares an attribute, and is consistent with Heider’s [116] theory that individuals choose relationships that reinforce existing preferences and beliefs. However, this fails to explain the unsupportive reviews, which we would expect to only cite evidence that supports their perspective. We did not observe this.

Fourthly, few reviews cited all relevant data–with underutilisation common. In the period concerned, conventions governing systematic search strategies were in their infancy. The reviews were mostly of the narrative form, a type that has been criticised for selection bias and the promotion of partial perspectives of the evidence. Indeed, one finding of this paper is that reviews in this field were poor at considering all the available empirical evidence. Bushman and Wells [117] offer experimental evidence of the role of availability bias (a heuristic that describes the tendency of humans to judge the prevalence/probability of an event by the ease in which they are able to recall certain information), which the authors argue influence the propensity for selection bias in narrative reviews compared with systematic reviews and meta-analysis, which impose a rigorous structure of literature searching and study evaluation on the researcher. Importantly, narrative reviews are not the only papers in which citation bias has been documented, Robinson and Goodman [118] documented significant citation bias amongst RCTs addressing the same question and may be explained by the lack of systematic search strategies amongst these papers.

Finally, did the selection of evidence itself explain the divergence of opinion amongst reviews? That is, were these authors only aware of the data they cited? If so, it would seem those underutilising the literature arrive at positions that are based on partial perspectives. Or did pre-existing opinions held by reviewers determine their selection of evidence? That is, were opinions formed first and evidence selected afterwards? If so, this would be a case of confirmation bias. Here, we see the complexities involved in attempting to determine which direction the arrow of causality points–is evidence formative in scientific opinion? Or do pre-existing opinions dictate the evidence we choose to examine?

The major findings offer an alternative to the prevailing notion of what a scientific controversy is, i.e. that scientific disagreement stems from different theoretical positions, where researchers looking at the same evidence can justifiably come to different conclusions [119]. Disagreement in this instance appears to stem from the selection of different sources of evidence, rather than the possession of different theoretical perspectives. Trinquart et al. [27] came to similar conclusions when examining the controversy about the suggested benefits of restricting dietary salt intake. They observed “two almost distinct and disparate lines of scholarship, one supporting and one contradicting the hypothesis” (p.57)–lines that relied on different sources of evidence and which differed in their citing behaviour.

Implications

Scientists, at present, are concerned at a wide range of factors that they understand to be distorting science. These include publication bias [120, 121], problems with replication [122], and citation biases [20, 21, 27, 111114]. Their combined effect has undermined faith in the idea that our sciences are self-correcting, and increasingly commentators fear that our system of knowledge production is broken. De Solla Price [123] and later Ravetz [124] raised concerns that as science grew it may experience ‘senility’–with the evidence base and citing literature growing too large for effective quality control, but it is a view that is gaining increasing attention [125].

This study focused on citation bias, a structural bias that leads to distortions of authority, whereby some positions are held up by the inflated authority of supporting primary studies and where scientists do not interact with, and in some cases may be unaware of, rival perspectives and unsupportive data [20]. This can lead to polarisation, where different scientists draw from different sources of evidence and arrive at opposing conclusions to similar questions [27]. Unlike these two studies, which examined recent citation practices, the present study focused on a historic analysis of divergent citation practices amongst papers drawing different conclusions as to what the available empirical evidence supports. Citation bias and research underutilisation were exactly documented, as were the different citation behaviours of different classifications of review.

This case study was chosen because it represents a unique example where the relevant RCTs were published in quick succession (1965–1968), and these remained the only RCTs addressing this particular question prior to a major consensus building conference in 1984. This provided a 17-year citation window, from the time all results of the RCTs were available to researchers to the beginning of this conference, where the evidence-base, as derived from RCTs, remained constant (i.e. no additional secondary prevention RCTs were published that would have impacted the evidence-base and, possibly, citation behaviours). Further, as these RCTs were the first to test this particular treatment, the relevant evidence-base and citation behaviour could not have been influenced by prior studies, at-least not evidence derived from RCTs, which may have distorted the results. This provided a unique opportunity to study how scientists utilise a small evidence-base over a long and undisturbed period. One surprising findings is that, even with a small evidence-base, scientists did not generally cite these studies in a representative manner nor did their citation behaviours improve as this period unfolded, and this challenges our understanding that it may be the size of the literature responsible for this phenomenon.

Therefore, the major findings of this study demonstrate that citation bias is not a new phenomenon–challenging narratives that suggest our science has become increasingly less reliable in recent years because science has grown too large–and that scientific disagreement may revolve around something more simple than divergent theoretical positions–scientists may just be looking at different sources of data.

The results of this study should not be taken as making claims about the current state of knowledge regarding the efficacy of dietary intervention in secondary prevention. Nor should it be used to fuel the current controversy over scientific advice concerning dietary fat consumption. Indeed, the selection of a historical cases study was, in part, chosen to avoid this paper being mistaken for attempting to weigh in on what particular treatment is suitable in secondary prevention. Since the 1980s, much work has been conducted in this area and the results of this study do not impact the scientific case. Rather, the aim was to provide a deeper understanding of the ways in which empirical data is used by scientists, how it may fuel scientific disagreement, and to introduce a method capable of detecting this particularly efficiently. Its focus was on citation practices and evidence dissemination, not the validity of current scientific theories.

Supporting information

S1 Table. Study characteristics of four RCTs examining dietary fat restriction/modification in the secondary prevention of CHD.

A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.

https://doi.org/10.1371/journal.pone.0197716.s001

(DOCX)

S2 Table. Intervention and diet characteristics of four RCTs examining dietary fat restriction/modification in the secondary prevention of CHD.

A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.

https://doi.org/10.1371/journal.pone.0197716.s002

(DOCX)

S3 Table. A comparison of the incidence of myocardial infarction, CHD mortality (myocardial infarction + sudden death), combined fatal + non-fatal CHD events, all-cause mortality, and changes in total serum cholesterol in four secondary prevention RCTs.

A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.

https://doi.org/10.1371/journal.pone.0197716.s003

(DOCX)

S4 Table. A risk-ratio analysis of the results of four secondary prevention RCTs.

A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.

https://doi.org/10.1371/journal.pone.0197716.s004

(DOCX)

S5 Table. A quotation analysis of four RCTs and 62 citing reviews.

https://doi.org/10.1371/journal.pone.0197716.s005

(DOCX)

S6 Table. Attribute data on all studies (vertices) included in this study.

https://doi.org/10.1371/journal.pone.0197716.s006

(DOCX)

S7 Table. Edge data on all citations between reviews and RCTs included in this study.

https://doi.org/10.1371/journal.pone.0197716.s007

(DOCX)

Acknowledgments

I would like to thank Professor Steve Sturdy, Dr Valeria Skafida, Dr Gil Viry, Professor Steve Yearley, Professor Gareth Leng, and Dr Sharon Hollingsworth for their time and feedback on this manuscript.

References

  1. 1. Page IH, Allen EV, Chamberlain FL, Keys A, Stamler J, Stare F. Dietary fat and its relation to heart attacks and strokes. Circulation. 1961; 23: 133–136.
  2. 2. Keys A, Anderson JT, Grande F. Prediction of serum-cholesterol responses of man to changes in fats ill the diet. Lancet. 1957; 273(7003): 959–966. pmid:13482259
  3. 3. Jolliffe N. Fats, cholesterol, and coronary heart disease: a review of recent progress. Circulation. 1959; 20: 109–127 pmid:13663199
  4. 4. Yerushalmy J, Hilleboe HE. Fat in the diet and mortality from heart disease; a methodologic note. N Y State J Med. 1957; 57: 2343–2354. pmid:13441073
  5. 5. Mann GV. The epidemiology of coronary heart disease. Am J Med. 1957; 23: 463–480. pmid:13458210
  6. 6. Yudkin J. Diet and coronary thrombosis. Hypothesis and fact. Lancet. 1957; 273: 155–62. pmid:13450357
  7. 7. Dayton S, Pearce ML, Hashomoto S, Dixon WJ, Tomiyasu U. A controlled clinical trial of a diet high in unsaturated fat in preventing complications of atherosclerosis. Circulation. 1969; 40: II-1–II-63.
  8. 8. Rose GA, Thomson WB, Williams RT. Corn oil in treatment of ischaemic heart disease. BMJ. 1965; 1: 1531–1533. pmid:14288105
  9. 9. Committee Research. Low-fat diet in myocardial infarction: a controlled trial. Lancet. 1965; 2: 501–504. pmid:4158171
  10. 10. Leren P. The effect of plasma-cholesterol-lowering diet in male survivors of myocardial infarction. A controlled clinical trial. Acta Med Scand Suppl. 1966; 466: 1–92. pmid:5228820
  11. 11. Medical Research Council. Controlled trial of soya-bean oil in myocardial infarction: Report of a research committee to the Medical Research Council. Lancet. 1968; 292: 693–700.
  12. 12. Woodhill JM, Palmer AJ, Leelarthaepin B, McGilchrist C, Blacket RB. Low fat, low cholesterol diet in secondary prevention of coronary heart disease. Adv Exp Med Biol. 1978; 109: 317–330. pmid:727035
  13. 13. Ramsden CE, Zamora D, Leelarthaepin B, Majchrzak-Hong SF, Faurot KR, Suchindran CM et al. Use of dietary linoleic acid for secondary prevention of coronary heart disease and death: evaluation of recovered data from the Sydney Diet Heart Study and updated meta-analysis. BMJ. 2013; 346: e8707 pmid:23386268
  14. 14. National Institutes of Health Consensus Development Program Consensus conference. Lowering blood cholesterol to prevent heart disease. JAMA. 1985; 253: 2080–2086.
  15. 15. Garrety K. Social worlds, actor-networks and controversy: the case of cholesterol, dietary fat and heart disease. Soc Stud Sci. 1997; 21: 727–773.
  16. 16. Harcombe Z, Baker JS, Cooper SM, Davies B, Sculthorpe N, DiNicolantonio JJ et al. Evidence from randomised controlled trials did not support the introduction of dietary fat guidelines in 1977 and 1983: a systematic review and meta-analysis. Open Heart. 2015; 2(1): e000196.
  17. 17. Hooper L, Martin N, Abdelhamid A, Davey Smith G. Reduction in saturated fat intake for cardiovascular disease. Cochrane Database Syst Rev. 2015;(6): CD011737. pmid:26068959
  18. 18. Schwingshackl L, Hoffmann G. Dietary fatty acids in the secondary prevention of coronary heart disease: a systematic review, meta-analysis and meta-regression. BMJ Open 2014; 4: e004487. pmid:24747790
  19. 19. Hamley S. The effect of replacing saturated fat with mostly n-6 polyunsaturated fat on coronary heart disease: a meta-analysis of randomised controlled trials. Nutr J. 2017; 16: 30. pmid:28526025
  20. 20. Greenberg SA. How citation distortions create unfounded authority: analysis of a citation network. BMJ. 2009; 339:b2680 pmid:19622839
  21. 21. Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ. 1992; 305: 15–19. pmid:1638188
  22. 22. Ravnskov U. Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol. 1995; 48: 713–719. pmid:7730926
  23. 23. Kannel WB, Doyle JT, Ostfeld AM, Jenkins CD, Kuller L, Podell RN et al. Optimal resources for primary prevention of atherosclerotic diseases. Atherosclerosis study group. Circulation. 1984; 70: 157A–205A.
  24. 24. National Research Council Diet and Health. Implications for Reducing Chronic Disease Risk. Washington, DC: National Academy Press, 1989.
  25. 25. LaRosa JC, Hunninghake D, Bush D, Criqui MH, Getz GS, Gotto AM et al. The cholesterol facts. A summary of the evidence relating dietary fats, serum cholesterol, and coronary heart disease. A joint statement by the American Heart Association and the National Heart, Lung, and Blood Institute. Circulation. 1990; 81(5): 1721–1733. pmid:2184951
  26. 26. Ravnskov U. The diet-heart idea is kept alive by selective citation. BMJ. 2003; 327: 1348.
  27. 27. Trinquart L, Johns DM, Galea S. Why do we think we know what we know? A metaknowledge analysis of the salt controversy. Int J Epidemiol. 2016; 45: 251–260. pmid:26888870
  28. 28. Leren P. The effect of plasma-cholesterol-lowering diet in male survivors of myocardial infarction. A controlled clinical trial. Bull N Y Acad Med. 1968; 44: 1012–1020. pmid:4875746
  29. 29. Leren P. Plasma cholesterol lowering diet. The effect in male survivors of myocardial infarction. Minn Med. 1969; 52: 1231–1235. pmid:5804990
  30. 30. Leren P. The Oslo diet-heart study. Eleven-year report. Circulation. 1970; 42:935–942. pmid:5477261
  31. 31. Turpeinen O, Karvonen MJ, Pekkarinen M, Miettinin M, Elosuo R, Paavilainen E. Dietary prevention of coronary heart disease: the Finnish mental hospital study, Int J Epidemiol. 1979; 8: 99–118. pmid:393644
  32. 32. Bierenbaum ML, Fleischman AI, Green DP, Raichelson RI, Hayton T, Watson PB et al. The 5-year experience of modified fat diets on younger men with coronary heart disease. Circulation. 1970; 42: 943–952. pmid:5477262
  33. 33. National Diet-Heart Study Research Group. The National Diet-Heart Study Final Report. Circulation. 1968; 37(3 Suppl.): I1–428.
  34. 34. Christakis G, Rinzler SH, Archer M, Winslow G, Jampel S, Stephenson J et al. The anti-coronary club. A dietary approach to the prevention of coronary heart disease—a seven-year report. Am J Public Health Nations Health 1966; 56: 299–314. pmid:5948223
  35. 35. Frantz ID, Dawson EA, Kuba K, Brewer ER, Gatewood LC, Bartsch GE. The Minnesota coronary survey: Effect of diet on cardiovascular events and deaths. American Heart Association Scientific Proceedings. Circulation. 1975; 41 (Suppl. 2): 2–4.
  36. 36. Hjermann T, Velve-Byre K, Holme I, Leren P. The effect of diet and smoking intervention on the incidence of coronary heart disease. Lancet. 1981; 2(8259): 1303–1310. pmid:6118715
  37. 37. Multiple Risk Factor Intervention Trial Research Group. Multiple risk factor intervention trial. Risk factor changes and mortality results. JAMA. 1982; 248: 1465–1477. pmid:7050440
  38. 38. Morrison LM. Diet in coronary atherosclerosis. J Am Med Assoc. 1960; 173:884–888. pmid:14424373
  39. 39. Lyon TP, Yankley A, Gofman JW, Strisower B. Lipoproteins and diet in coronary heart disease. Calif Med. 1956; 84: 325–328. pmid:13316532
  40. 40. Pilkington TR, Stafford JL, Hankin VS, Simmonds FM, Koerselman HB. Practical diets for lower serum lipids. Brit Med J. 1960; 1(5165): 23–25. pmid:20788847
  41. 41. Hood B, Sanne H, Örndahl G, AhlstÖm M, Welin G. Long-term prognosis in essential hypercholesterolemia. Actu Med Scand. 1965; 178: 161–173.
  42. 42. Hansen PF, Geill T, Lund E. Dietary Fats and Thrombosis. Lancet 1962; 2:1193–1194. pmid:13952618
  43. 43. Nelson AM. Treatment of atherosclerosis by diet. I. Results in patients followed from 36 to 72 months. Northwest Med. 1965; 55: 643–649.
  44. 44. Cornfield J, Mitchell S. Selected risk factors in coronary disease. Possible intervention effects. Arch Environ Health. 1969; 19: 382–394. pmid:5807741
  45. 45. Brown DF. Blood lipids and lipoproteins in atherogenesis. Am J Med. 1969; 46: 691–704. pmid:4892338
  46. 46. Walker AR. Can expectation of life in Western populations be increased by changes in diet and manner of life? II. S Afr Med J. 1969; 43: 768–775. pmid:4895907
  47. 47. Simborg DW. The status of risk factors and coronary heart disease. J Chron Dis 1970; 22: 515–552. pmid:5437449
  48. 48. Zelis R, Mason DT, Spann JF. The hyperlipoproteinemias. A simplified classification and approach to therapy. Calif Med. 1970; 112: 32–37. pmid:4924875
  49. 49. Logue RB, Robinson PH. Medical management of angina pectoris Circulation. 1972; 46(6): 1132–1145. pmid:4404551
  50. 50. Stormorken H. Epidemiological studies. Relation between dietary fat and arterial and venous thrombosis. Haemostasis. 1973; 2:1–12. pmid:4597983
  51. 51. Jacobson NL. The controversy over the relationship of animal fats to heart disease. BioScience. 1974; 24(3): 141–148.
  52. 52. Howard AN. Hypolipaemic drugs and coronary heart disease. J Clin Pathol Suppl. (R Coll Pathol.) 1975; 9: 106–114.
  53. 53. Shaper AG. Primary and secondary prevention trials in coronary heart disease. Postgrad Med J. 1976; 52: 464–469. pmid:967765
  54. 54. Kritchevsky D. Diet and atherosclerosis. Am J Pathol 1976; 84: 615–632. pmid:786036
  55. 55. Yeshurun D, Gotto AM. Drug treatment of hyperlipidemia. Am J Med. 1976; 60: 379–396. pmid:1258886
  56. 56. Lewis B. Hypothesis into theory—the development of aetiological concepts of ischaemic heart disease: a review. J R Soc Med. 1978; 71: 809–818. pmid:215767
  57. 57. Hulley SB, Sherwin R, Nestle M, Lee PR. Epidemiology as a guide to clinical decisions II. Diet and coronary heart disease. West J Med. 1981; 135: 25–33. pmid:7257376
  58. 58. Hulley SB, Lo B. Choice and use of blood lipid tests. An epidemiologic perspective. Arch Intern Med. 1983; 143: 667–673. pmid:6340621
  59. 59. Goldman GL, Pichard AD. The natural history of coronary artery disease: does medical therapy improve the prognosis? Prog Cardiovasc Dis. 1983; 25: 513–552. pmid:6133314
  60. 60. Stallones RA. Ischemic heart disease and lipids in blood and diet. Annu Rev Nutr. 1983; 3: 155–185. pmid:6357234
  61. 61. Dayton S, Pearce ML. Prevention of coronary heart disease and other complications of arteriosclerosis by modified diet. Am J Med. 1969; 46: 751–762. pmid:4892340
  62. 62. Lees RS, Wilson DE. The treatment of hyperlipidemia. N Engl J Med. 1971; 284: 186–195 pmid:4923684
  63. 63. Carnie JA. Diet and heart disease. Sci Prog. 1972; 60: 231–243. pmid:4581384
  64. 64. Ahrens EH. The management of hyperlipidemia: whether, rather than how. Ann Intern Med. 1976; 85: 87–93. pmid:779574
  65. 65. Borhani NO. Primary prevention of coronary heart disease: a critique. Am J Cardiol. 1977; 40: 251–259. pmid:195453
  66. 66. Mann GV. Diet-heart: end of an era. N Engl J Med. 1977; 297: 644–650. pmid:197407
  67. 67. Mann GV. Coronary heart disease—the doctor’s dilemma. Am Heart J. 1978; 96: 569–571. pmid:263389
  68. 68. Mann JI. Fats and atheroma: a retrial. Br Med J. 1979; 1(6165): 732–734. pmid:435754
  69. 69. McMichael J. Fats and atheroma: an inquest. Br Med J. 1979; 1(6157): 173–175. pmid:105776
  70. 70. McMichael J. Fats and arterial disease. Am Heart J. 1979; 98: 409–412. pmid:484428
  71. 71. Walker AR. Dietary fat intake and serum cholesterol levels in coronary heart disease. S Afr Med J. 1980; 58(1): 7–12 pmid:6996145
  72. 72. Inkeles S, Eisenberg D. Hyperlipidemia and coronary atherosclerosis: a review. Medicine. 1981; 60(2): 110–123. pmid:7012540
  73. 73. Oliver MF. Serum cholesterol—the knave of hearts and the joker. Lancet. 1981; 2(8255): 1090–1095. pmid:6118533
  74. 74. May GS, Eberlein K, Furberg CD, Passamani ER, DeMets DL. Secondary prevention after myocardial infarction: a review of long-term trials. Prog Cardiovasc Dis. 1982; 24: 331–352. pmid:6119737
  75. 75. Buckwald H, Fitch L, Moore RB. Overview of randomized clinical trials of lipid intervention for atherosclerotic cardiovascular disease. Control Clin Trials. 1982; 3: 271–283. pmid:6759035
  76. 76. Coates ME. Reviews of the progress of dairy science: dietary lipids and ischaemic heart disease. J Dairy Res. 1983; 50: 541–557. pmid:6358290
  77. 77. Oliver MF. Why measure cholesterol after myocardial infarction, and when? Br Med J. (Clin Res Ed.) 1984; 289: 1641–1642.
  78. 78. Malmros H. Dietary prevention of atherosclerosis. Lancet. 1969; 2(7618): 479–484. pmid:4185482
  79. 79. Rathmann DM, Stockton JR, Melnick D, Stare FJ. Dynamic utilization of recent nutritional findings: Diet and cardiovascular disease. CRC Crit Rev Food Technol. 1970; 1: 331–338.
  80. 80. Little JA. Fats in Adult Nutrition. Can J Public Health. 1971; 62: 27–35. pmid:4925957
  81. 81. Vergroesen AJ. Dietary fat and cardiovascular disease: possible modes of action of linoleic acid. Proc Nutr Soc. 1972; 31: 323–329. pmid:4579353
  82. 82. Christakis G, Rathmann D. Position Statement on Diet and Coronary Heart Disease. American Health Foundation. Prev Med. 1972; 1: 255–286. pmid:4560604
  83. 83. Stamler J. Epidemiology of coronary heart disease. Med Clin North Am. 1973; 57(1): 5–46. pmid:4569832
  84. 84. Anderson JT, Grande F, Keys A. Cholesterol-lowering diets. Experimental trials and literature review. J Am Diet Assoc. 1973; 62: 133–142. pmid:4739366
  85. 85. Mcbean LD, Spechmann EW. An interpretive review: diet in early life and the prevention of atherosclerosis. Pediatr Res. 1974; 8: 837–842. pmid:4606213
  86. 86. Renaud S. Dietary fats and arterial thrombosis. Thromb Res. 1974; 4(Suppl. 1):25–35.
  87. 87. Grande F. Diet and Atherosclerosis. S Afr Med J. 1974; 48: 1660–1668. pmid:4604321
  88. 88. Glueck CJ, Mattson F, Bierman E. Diet and coronary heart disease: another view. N Engl J Med. 1978; 298: 1471–1473. pmid:206833
  89. 89. Abrahamson JH, Hopp C. The control of cardiovascular risk factors in the elderly. Prev Med. 1976; 5: 32–47. pmid:772670
  90. 90. Kannel WB. Prevention of cardiovascular disease. Curr Probl Cardiol. 1976; 1:1–68.
  91. 91. Grundy SM. Treatment of hypercholesterolemia. Am J Clin Nutr. 1977; 30: 985–992. pmid:326023
  92. 92. Hegsted DM. Dietary goals-a progressive view. Am J Clin Nutr. 1978; 31: 1504–1509. pmid:28662
  93. 93. Eaton RP. High density lipoprotein—key to anti-atherogenesis. J Chronic Dis. 1978; 31: 131–135. pmid:209070
  94. 94. Steinberg D. Research related to underlying mechanisms in atherosclerosis. Circulation. 1979; 60: 1559–1565. pmid:387283
  95. 95. Glueck CJ, Connor WE. Diet and atherosclerosis: past, present and future. West J Med. 1979; 130: 117–122. pmid:371145
  96. 96. Hegsted DM. Optimal nutrition. Cancer. 1979; 43(5 Suppl.): 1996–2003. pmid:445385
  97. 97. Nora JJ. Identifying the child at risk for coronary disease as an adult: A strategy for prevention. J Pediatr. 1980; 97: 706–714. pmid:7000993
  98. 98. Hymowitz N. Behavioural approaches to preventing heart disease: risk factor modification. Int J Mental Health. 1980; 9: 27–69.
  99. 99. Hornstra G. Dietary prevention of coronary heart disease. Effect of dietary fats on arterial thrombosis. Postgrad Med J. 1980; 56: 563–570. pmid:7008047
  100. 100. Stamler J. The established relationship among diet, serum cholesterol and coronary heart disease Acta Med Scand. 1980; 207: 433–446. pmid:7424562
  101. 101. Steinberg D. Underlying mechanisms in atherosclerosis. J Pathol. 1981; 133: 75–87. pmid:7009813
  102. 102. Hornstra G. The role of (n-6) fatty acids and their metabolites in arterial thrombus formation. Prog Lipid Res. 1981; 20: 407–413. pmid:6804970
  103. 103. Grundy SM, Bilheimer D, Blackburn H, Brown WV, Kwiterovjch PO, Mattson F et al. Rationale of the Diet-Heart Statement of the American Heart Association. Report of Nutrition Committee. Circulation. 1982; 65(4): 839A–854A pmid:7060268
  104. 104. Council of Scientific Affairs. Dietary and pharmacologic therapy for the lipid risk factors. JAMA. 1983; 250: 1873–1879. pmid:6620484
  105. 105. Furberg CD, May GS. Effect of long-term prophylactic treatment on survival after myocardial infarction. Am J Med. 1984; 76: 76–83. pmid:6146258
  106. 106. R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2008 [cited May 2017]. Available from: http://www.R-project.org.
  107. 107. Handcock M, Hunter D, Butts C, Goodreau S, Krivitsky P, Bender-deMoll S et al. ‘statnet: Software Tools for the Statistical Analysis of Network Data’. The Statnet Project, 2003 [cited April 2017]. Available from: http://www.statnet.org. in R package version 2016.9, Stable URL: CRAN.R-project.org/package=statnet.
  108. 108. Bastian M, Heymann S, Jacomy M. Gephi: an open source software for exploring and manipulating networks. International AAAI Conference on Weblogs and Social Media, 2009 [cited April 2017]. Available from: https://gephi.org/users/publications/.
  109. 109. Jacomy M, Venturini T, Heymann S, Bastian M. ForceAtlas2, a Continuous Graph Layout Algorithm for Handy Network Visualization Designed for the Gephi Software. PLoS One. 2014; 9(6): e98679. pmid:24914678
  110. 110. Newman MEJ. Networks: An Introduction. Oxford, Oxford University Press, 2010.
  111. 111. Gøtzsche PC. Reference bias in reports of drug trials. Br Med J. (Clin Res ed.) 1987; 295: 654–656.
  112. 112. Nieminen P, Rucker G, Miettunen J, Carpenter J, Schumacher M. Statistically significant papers in psychiatry were cited more often than others. J Clin Epidemiol. 2007; 60: 939–946. pmid:17689810
  113. 113. Fanelli D. Positive results receive more citations, but only in some disciplines. Scientometrics. 2013; 94: 701–709.
  114. 114. Misemer BS, Platts-Mills TF, Jones CW. Citation bias favoring positive clinical trials of thrombolytics for acute ischemic stroke: a cross-sectional analysis. Trials. 2016; 17: 473. pmid:27677444
  115. 115. Simkin MV, Roychowdhury VP. Read before you cite! Complex Syst. 2003; 14: 269–274.
  116. 116. Heider F. The Psychology of Interpersonal Relations. Hoboken, NJ, USA: John Wiley & Sons, 1958.
  117. 117. Bushman BJ, Wells GL. Narrative impressions of literature: The availability bias and the corrective properties of meta-analytic approaches. Pers Soc Psychol Bull. 2001; 27: 1123–1130.
  118. 118. Robinson KA, Goodman SN. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med. 2011; 154: 50–55. pmid:21200038
  119. 119. Kuhn T. The Structure of Scientific Revolutions (2nd edn.). Chicago; University of Chicago Press, 1962 [1970].
  120. 120. Young NS, Ioannidis JPA, Al-Ubaydli O. Why current publication practices may distort science. PLoS Med 2008; 5(10): e201. pmid:18844432
  121. 121. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010; 14(8): iii, ix-xi, 1–193. pmid:20181324
  122. 122. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005; 2(8): e124. pmid:16060722
  123. 123. de Solla Price DJ. Little Science, Big Science. New York: Columbia. University Press, 1963.
  124. 124. Ravetz JR. Scientific Knowledge and Its Social Problems. Oxford: Clarendon Press, 1971.
  125. 125. Parolo DB, Pan K, Ghosh R, Huberman R, Kimmo BA, Kaski K et al. Attention decay in science. J Informetr. 2015; 9: 734–745.