Published pharmaceutical industry–sponsored trials are more likely than non-industry-sponsored trials to report results and conclusions that favor drug over placebo. Little is known about potential biases in drug–drug comparisons. This study examined associations between research funding source, study design characteristics aimed at reducing bias, and other factors that potentially influence results and conclusions in randomized controlled trials (RCTs) of statin–drug comparisons.
Methods and Findings
This is a cross-sectional study of 192 published RCTs comparing a statin drug to another statin drug or non-statin drug. Data on concealment of allocation, selection bias, blinding, sample size, disclosed funding source, financial ties of authors, results for primary outcomes, and author conclusions were extracted by two coders (weighted kappa 0.80 to 0.97). Univariate and multivariate logistic regression identified associations between independent variables and favorable results and conclusions. Of the RCTs, 50% (95/192) were funded by industry, and 37% (70/192) did not disclose any funding source. Looking at the totality of available evidence, we found that almost all studies (98%, 189/192) used only surrogate outcome measures. Moreover, study design weaknesses common to published statin–drug comparisons included inadequate blinding, lack of concealment of allocation, poor follow-up, and lack of intention-to-treat analyses. In multivariate analysis of the full sample, trials with adequate blinding were less likely to report results favoring the test drug, and sample size was associated with favorable conclusions when controlling for other factors. In multivariate analysis of industry-funded RCTs, funding from the test drug company was associated with results (odds ratio = 20.16 [95% confidence interval 4.37–92.98], p < 0.001) and conclusions (odds ratio = 34.55 [95% confidence interval 7.09–168.4], p < 0.001) that favor the test drug when controlling for other factors. Studies with adequate blinding were less likely to report statistically significant results favoring the test drug.
Citation: Bero L, Oostvogel F, Bacchetti P, Lee K (2007) Factors Associated with Findings of Published Trials of Drug–Drug Comparisons: Why Some Statins Appear More Efficacious than Others. PLoS Med 4(6): e184. https://doi.org/10.1371/journal.pmed.0040184
Academic Editor: Alessandro Liberati, Italian Cochrane Centre, Italy
Received: September 4, 2006; Accepted: April 2, 2007; Published: June 5, 2007
Copyright: © 2007 Bero et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This research was supported by the California Tobacco-Related Disease Research Program Grant 13RT-0108 (principal investigator, L. A. Bero). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: CI, confidence interval; OR, odds ratio; RCT, randomized controlled trial
Randomized controlled trials are generally considered to be the most reliable type of experimental study for evaluating the effectiveness of different treatments. Randomization involves the assignment of participants in the trial to different treatment groups by the play of chance. Properly done, this procedure means that the different groups are comparable at outset, reducing the chance that outside factors could be responsible for treatment effects seen in the trial. When done properly, randomization also ensures that the clinicians recruiting participants into the trial cannot know the treatment group to which a patient will end up being assigned. However, despite these advantages, a large number of factors can still result in bias creeping in. Bias comes about when the findings of research appear to differ in some systematic way from the true result. Other research studies have suggested that funding is a source of bias; studies sponsored by drug companies seem to more often favor the sponsor's drug than trials not sponsored by drug companies.
Why Was This Study Done?
The researchers wanted to more precisely understand the impact of different possible sources of bias in the findings of randomized controlled trials. In particular, they wanted to study the outcomes of “head-to-head” drug comparison studies for one particular class of drugs, the statins. Drugs in this class are commonly prescribed to reduce the levels of cholesterol in blood amongst people who are at risk of heart and other types of disease. This drug class is a good example for studying the role of bias in drug–drug comparison trials, because these trials are extensively used in decision making by health-policy makers.
What Did the Researchers Do and Find?
This research study was based on searching PubMed, a biomedical literature database, with the aim of finding all randomized controlled trials of statins carried out between January 1999 and May 2005 (reference lists also were searched). Only trials which compared one statin to another statin or one statin to another type of drug were included. The researchers extracted the following information from each article: the study's source of funding, aspects of study design, the overall results, and the authors' conclusions. The results were categorized to show whether the findings were favorable to the test drug (the newer statin), inconclusive, or not favorable to the test drug. Aspects of each study's design were also categorized in relation to various features, such as how well the randomization was done (in particular, the degree to which the processes used would have prevented physicians from knowing which treatment a patient was likely to receive on enrollment); whether all participants enrolled in the trial were eventually analyzed; and whether investigators or participants knew what treatment an individual was receiving.
One hundred and ninety-two trials were included in this study, and of these, 95 declared drug company funding; 23 declared government or other nonprofit funding while 74 did not declare funding or were not funded. Trials that were properly blinded (where participants and investigators did not know what treatment an individual received) were less likely to have conclusions favoring the test drug. However, large trials were more likely to favor the test drug than smaller trials. When looking specifically at the trials funded by drug companies, the researchers found various factors that predicted whether a result or conclusion favored the test drug. These included the impact of the journal publishing the results; the size of the trial; and whether funding came from the maker of the test drug. However, properly blinded trials were less likely to produce results favoring the test drug. Even once all other factors were accounted for, the funding source for the study was still linked with results and conclusions that favored the maker of the test drug.
What Do These Findings Mean?
This study shows that the type of sponsorship available for randomized controlled trials of statins was strongly linked to the results and conclusions of those studies, even when other factors were taken into account. However, it is not clear from this study why sponsorship has such a strong link to the overall findings. There are many possible reasons why this might be. Some people have suggested that drug companies may deliberately choose lower dosages for the comparison drug when they carry out “head-to-head” trials; this tactic is likely to result in the company's product doing better in the trial. Others have suggested that trials which produce unfavorable results are not published, or that unfavorable outcomes are suppressed. Whatever the reasons for these findings, the implications are important, and suggest that the evidence base relating to statins may be substantially biased.
Please access these Web sites via the online version of this summary at http://dx.doi.org/10.1371/journal.pmed.0040184.
- The James Lind Library has been created to help people understand fair tests of treatments in health care by illustrating how fair tests have developed over the centuries
- The International Committee of Medical Journal Editors has provided guidance regarding sponsorship, authorship, and accountability
- The CONSORT statement is a research tool that provides an evidence-based approach for reporting the results of randomized controlled trials
- Good Publication Practice guidelines provide standards for responsible publication of research sponsored by pharmaceutical companies
- Information from Wikipedia on Statins. Wikipedia is an internet encyclopedia anyone can edit
Bias is the combination of various study design, data analysis, and presentation factors that make the results differ systematically from the truth . Various factors can lead to bias in randomized controlled trials (RCTs) of drug efficacy, including framing of the research question, design and conduct of the study, and analysis of the data [2,3]. Whether the results are reported in full, or whether there is selective reporting of outcomes can also contribute to biased results and conclusions [4–7].
Most studies that have attempted to identify factors that may be associated with bias have examined individual design features, including randomization [8,9], concealment of allocation , double blinding [9,10], sample size [11–14], choice of drug comparator [15–18], and choice of statistical analysis [19,20]. In this study, we examine the relative contributions of different factors to favorable results and conclusions.
One factor that has been associated with possible bias is funding source for a study. Trials supported by pharmaceutical companies are more likely than those with non-industry sponsors to report results and conclusions that are favorable towards the sponsor's product compared to placebo [14,15,21–28]. However, few studies examining the association of funding source and outcome adjust for potential confounders, such as study design characteristics or type of intervention. For example, study characteristics such as randomization sequence generation, concealment of allocation, blinding, sample size, or choice of drug comparator might also contribute to statistically significant results [28,29]. Although the association of pharmaceutical industry sponsorship with results and conclusions that favor the sponsor's product over placebo is clear [22,23], the potential influence of funding source when the funder manufactures one of two competing drugs undergoing comparison has not been well described. Although Heres et al. found that results of head-to-head comparisons of second-generation antipsychotics have contradictory conclusions depending on which company sponsored the study, they did not analyze the potential effects of other study design characteristics .
Statins are an interesting class of drug for investigating the influence of funding source on outcomes of head-to-head drug comparisons because a number of statins are manufactured by competing companies. Statins are widely prescribed as effective first-line agents for lowering cholesterol and other lipids. At the time of this study, seven statin drugs were marketed in the United States by competing companies, although one of these drugs has since been withdrawn. Alternative classes of drugs were also available to treat the same condition. As strong evidence suggests that statins are more effective than placebo in reducing lipids , drug–drug comparison trials involving statins are the most relevant for making policy decisions about choosing a statin. Choice of a statin should depend on data from statin–statin comparisons of low-density lipoprotein reduction at comparable doses, ability to achieve cholesterol-reduction goals, and effects on a variety of other outcomes such as death, coronary events, or stroke . For example, formulary committees use head-to-head drug comparisons to decide which of the statins will be placed on their formulary. Therefore, it is important to explore possible biases in statin–statin comparisons.
This cross-sectional study examines associations between research funding source, study design characteristics aimed at reducing bias, and other factors for which results and conclusions have been published in RCTs of statin–drug comparisons. We hypothesized that the results and conclusions of trials are more likely to favor the statin made by the sponsor of the study and that other design features, such as concealment of allocation, blinding, and sample size are also associated with statistically significant results that favor the statin produced by the study's sponsor.
We electronically searched PubMed to identify reports of RCTs published between January 1999 and May 2005. The following MeSH terms or Substance Names of the seven available statins were used: “simvastatin” OR “cerivastatin” OR “pravastatin” OR “atorvastatin” OR “fluvastatin” OR “rosuvastatin” OR “lovastatin”. The search was limited to “randomized controlled trials” and “humans”. We restricted our search to these years because journals strengthened their policies requiring disclosure of funding sources and financial ties of authors during this period . We also searched the reference lists of all potentially relevant articles identified through the PubMed search. Our search included articles published in any language.
Inclusion and Exclusion Criteria
We reviewed abstracts of all citations and retrieved articles based on the following inclusion criteria: (1) RCT; (2) statin drug compared to a different statin drug or another, non-statin, drug; (3) efficacy measured in humans; and (4) original research, defined as studies that appeared to present original data and did not specifically state that they were reviews. Studies with the primary objective of assessing the effect of a combination of a statin and another drug were included if there was a comparison of the statin alone with the other drug. If a placebo arm was also included in the trial, we included only the data from the statin–drug comparison.
The following exclusion criteria were used to screen all abstracts: (1) pharmacokinetic or pharmacodynamic studies, since they do not involve testing of clinical efficacy outcomes; (2) studies including only rationale and design elements, editorials, letters to the editor, commentaries, abstracts, unpublished reports, reviews; (3) studies comparing different doses of one type of statin; (4) studies comparing statins to placebo only; (5) studies comparing statins to a non-drug intervention (e.g., diet, exercise); (6) studies in which the statin was present in all the comparison groups; (7) absence of statistical comparison or lack of sufficient data; and (8) in vitro analyses. Any discrepancies about inclusion were discussed by the authors of the present paper until consensus was achieved. No identical publications were identified. However, as we were interested in the published reports from trials, we did include multiple publications from the same study if the publications reported different outcomes.
One investigator (F. Oostvogel), who was not blinded to author names and affiliations, funding sources, and financial disclosure, extracted all data from each article. A second coder (L. Bero), who was blinded to funding source and financial tie information, independently extracted data on concealment of allocation, selection bias, blinding, sample size, results for primary outcomes, and author conclusions. Inter-coder reliability was very good (weighted kappa 0.80 to 0.97). In cases of disagreement, the two coders discussed the papers and reached agreement. We extracted data on the following publication characteristics, which have been shown to be independently related to favorable results or conclusions of drug studies [8–10,13,22,27,34,35].
Each article was classified as peer reviewed, non-peer reviewed, or unknown, based on information found on the website of the journal where the article was published. A publication was considered peer reviewed if the website mentioned that the journal had a peer-review process or if it was stated that the manuscripts were evaluated by at least one external expert in the field; otherwise, a publication was considered non-peer reviewed. Peer-review status was classified as unknown if we could find no information on the journal.
Impact factor was obtained from the Institute for Scientific Information, 2004 data .
The institutional affiliation of the corresponding author was obtained from the article and classified into (1) academic/university, (2) government, (3) private nonprofit, (4) industry, (5) hospital, (6) other, or (7) unable to determine.
Country of origin.
The country of origin of the corresponding author was recorded and categorized into low income, lower-middle income, upper-middle income, and high income economies based on the World Bank Group classifications .
Study Design Characteristics
The study design for each article was classified as parallel or cross-over trial. Specific drugs being compared were recorded.
The comparisons for the primary outcome were classified as (1) statin versus statin or (2) statin versus other drug. In statin-versus-statin comparisons, the “test” drug was defined as the newest statin (most recent FDA approval date) and the older statin as the “comparator” drug. In statin-versus-other-drug comparisons, the “test” drug was defined as the statin and the other drug as the “comparator” drug.
Type of primary outcome measure.
The primary outcome measured was classified as (1) surrogate if the end point was a marker for a clinical event (e.g., lipid levels, artery diameter, endothelial function) or (2) clinical if a real clinical event (e.g., stroke, myocardial infarction, death) was measured.
For each published paper, the result reported for each primary outcome was categorized as (1) favorable if the result was statistically significant (p < 0.05 or confidence interval [CI] excluding no difference) and in the direction of the test drug being more efficacious or less harmful (in the case of side effects); (2) inconclusive if the result did not reach statistical significance; or (3) unfavorable if the result was statistically significant in the direction of the comparator drug being more efficacious or less harmful. If a study explicitly stated that it was designed as a non-inferiority study and the two comparisons drugs were equivalent, the result was coded as favorable.
The entire set of results for all primary outcomes in each paper was then classified as favorable if at least one primary outcome was favorable and none were unfavorable; otherwise, the entire set was classified as unfavorable.
The conclusions reported in the published papers were categorized as (1) favorable if the test drug was preferred to comparator; (2) about equal if the test drug was about equal to comparator; or (3) not favorable if the comparator drug was preferred to the test drug. If a study explicitly stated that it was designed as a non-inferiority study and the two drugs being compared were equivalent, the conclusion was coded as favorable. If an article did not clearly state that one of the two drugs was better or if the two drugs had different advantages, the conclusion was coded as “test drug about equal to comparator”. For analysis, conclusions were categorized as favorable or not favorable (combining about equal and not favorable).
The funding source of each published study was categorized as (1) industry, (2) private nonprofit, (3) government, (4) other, (5) multiple sources, (6) no funding, and (7) none disclosed. For analysis, the funding-source categories were collapsed into (1) industry, (2) none disclosed/no funding, and (3) government/private nonprofit.
Data about the financial ties of each author were extracted and coded for (1) whether or not there were any financial ties disclosed with the sponsor of the study and (2) whether or not there were any financial ties disclosed with any other company (yes, no, or none disclosed).
Study Design Characteristics Aimed at Reducing Bias
Studies that met the inclusion criteria were rated for study design features according to the components reported by Chalmers et al. . Chalmers used three different categories: method of treatment assignment (randomization and concealment of allocation), control for whether all participants enrolled in the trial have been included in the analysis (intention-to-treat analysis and loss to follow-up), and blinding of participants and investigators. For each category, the score can range from 0 to 3, where higher scores indicate better methodological quality. For analysis, we dichotomized each category into “adequate” (score of 2 or 3) or “inadequate” (score of 0 or 1).
We report the frequency of the different characteristics of each article. For characteristics where there was sufficient variability, we analyzed the characteristics by the direction of results and conclusions to determine whether certain characteristics were associated with favorable results or conclusions. Proportions of manuscripts with favorable results or conclusions were first analyzed using univariate logistic regression and estimating odds ratios (ORs) to identify associations between independent variables and favorable results and conclusions. Although impact factor and sample size were continuous variables, they were modeled categorically because their effects were clearly nonlinear.
To control for multiple variables simultaneously, we carried out multivariate logistic regression analysis and calculated ORs. These models included funding source and all factors that had p < 0.05 in univariate models for either favorable results or conclusions. For our primary analysis, we conducted the regression analyses on our full sample (n = 192). For our a priori analysis of drug industry–sponsored studies, we conducted the regression analyses on the subsample of studies that were industry funded (n = 95) in order to examine the association between funding from the test drug company and results or conclusions that favor the test drug.
Our target sample size was to have 40 trials that had results or conclusions favoring the test drug. We chose this sample size so there would be at least ten trials with favorable results or conclusions per predictor in a multivariate analysis with up to four simultaneous predictors. We achieved this target sample size for both the full sample and subsample of industry-funded trials. Data were analyzed with SAS software (version 9.1, SAS Institute, Cary, North Carolina, United States).
Characteristics of Included Studies
Our final sample consisted of 192 published RCTs (see Figure 1). The characteristics of the full sample are shown in Table 1, and the full list of references to included studies is presented in Table S1. Almost all (98%, 189/192) studies reported only surrogate outcome measures. There was also little variability in the peer-review status, study design, or country of origin or institutional affiliation of corresponding authors. Therefore, these variables were not included in our regression analyses. Impact factor and sample size were divided into quartiles with the upper three quartiles compared with the lowest quartile. Forty-nine percent of articles had conclusions that favored the test drug, 15% had a conclusion that favored the comparator drug, and 36% concluded that the two drugs were about equal. Of the 192 included trials, 95 (49%) disclosed funding from industry sponsors.
Characteristics of Included Articles by Funding Source (n = 192)
Among the 95 articles declaring industry funding, the role of the sponsor was disclosed in 20 (21%) of these. One trial stated that the sponsor was not involved in the study design and analyses, and 19 trials stated that the sponsor was involved by providing the study drug, data analyses, or writing and preparation of the manuscript.
Analysis of Full Sample
Table 2 shows the results of univariate logistic regression analyses. Studies with adequate blinding were substantially less likely to report results favoring the test drug than studies that did not include adequate blinding. Trials with larger sample sizes were more likely to report conclusions that favored the test drug, while trials with no disclosed funding sources were less likely to have conclusions favoring the test drug compared to trials with industry funding. In multivariate analyses, trials with adequate blinding remained significantly less likely to report statistically significant results favoring the test drug, and sample size remained associated with favorable conclusions when controlling for other factors (Table 3). Pooling non-industry-funded studies and those with no funding disclosure produced ORs of 1.49 (95% CI 0.75–3.0, p = 0.26) for results and 0.73 (95% CI 0.36–1.46, p = 0.37) for conclusions versus industry-funded studies. Adding interaction terms for industry funding versus all others with sample size quartile did not produce a statistically significant improvement in the fit to the data for results (p = 0.21 by likelihood ratio test) and also did not show a consistent pattern (OR of all others 2.1, 1.15, 3.3, 0.42 in first to fourth sample size quartiles, respectively). For conclusions, the interaction terms also did not reach statistical significance overall (p = 0.11), but the pattern was in a consistent direction (OR of all others 2.2, 1.02, 0.52, 0.18 in first to fourth sample size quartiles, respectively).
Association between Characteristics of Articles (n = 192) and Statistically Significant Results or Conclusions that Favor the Test Drug: Univariate Logistic Regression
Association between Characteristics of Manuscripts (n = 192) and Results or Conclusions that Favor Test Drug: Multivariate Analysis
We also conducted a multivariate analysis for the subset of articles that were statin–statin comparisons (n = 112), and the results were essentially the same as for the comparisons between statin and any non-statin or statin drug. Trials with adequate blinding remained significantly less likely to report statistically significant results favoring the test drug (OR = 0.28 [95% CI 0.11–0.73], p = 0.0095), and sample size remained associated with favorable conclusions (OR = 8.49 [95% CI 1.93–37.36], p = 0.0047) when controlling for other factors.
Analysis of Industry-Sponsored Trials
In univariate logistic regression analyses of the industry-sponsored trials, higher impact factor, larger sample size, and funding from the test drug company were associated with favorable results, while trials with adequate blinding were less likely to report statistically significant results favoring the test drug (Table 4). Larger sample size and funding from the test drug company were also associated with favorable conclusions (Table 4). In multivariate logistic regression analysis, funding from the test drug company remained associated with statistically significant results favoring the test drug (OR = 20.16 [95% CI 4.37–92.98], p < 0.001) or conclusions favoring the test drug (OR = 34.55 [95% CI 7.09–168.4], p < 0.001) (Table 5) when controlling for other factors. Studies with adequate blinding remained less likely to report statistically significant results favoring the test drug (Table 5). Adding interaction terms for funding by test drug company with sample size quartile did not produce a statistically significant improvement in the fit to the data for results (p = 0.38), although the ORs associated with test drug company funding did show an increasing pattern (3.8, 8.1, infinite, 59.5 in the first to fourth sample size quartiles, respectively). For conclusions, the interaction p-value was p = 0.066, with ORs associated with test drug company funding of infinite, 2.1, infinite, and 143.2 in the first to fourth sample size quartiles, respectively. The infinite estimated ORs result from no favorable outcomes for some combinations of funding and sample size quartile, making these results difficult to interpret.
Association between Characteristics of Industry-Funded Articles (n = 95) and Statistically Significant Results and Conclusions that Favor the Test Drug: Univariate Logistic Regression
Association between Characteristics of Industry-Funded Manuscripts (n = 95) and Results or Conclusions that Favor Test Drug: Multivariate Analysis
We also conducted the multivariate logistic regression analysis for the subset of industry-funded studies that were statin–statin comparisons (n = 63), and the results were essentially the same as for the comparisons between statin and any non-statin or statin drug. Funding from the test drug company remained associated with statistically significant results favoring the test drug (OR = 16.06 [95% CI 2.22–116.3], p = 0.043) or conclusions favoring the test drug (OR = 77.09 [95% CI 7.92–749.9], p < 0.001) when controlling for other factors.
We examined the association between study design characteristics and the results and conclusions of RCTs of head-to-head comparisons of statins with other drugs. We hypothesized that the results and conclusions of published trials would be more likely to favor the statin made by the sponsor of the study and that other design features, such as concealment of allocation, blinding, and sample size, would also be associated with results or conclusions that favor the statin. We found that the main factor associated with the results and conclusions of industry-sponsored research to compare statin drugs with statin or non-statin drugs is research sponsorship. Our study adds new information to the body of literature showing that pharmaceutical industry–sponsored studies comparing drug and placebo are more likely to favor the drug [14,21–23,25,27,28,39]. Our finding suggests that favorable results and outcomes are associated with the specific sponsor of a study, even when all the studies are industry funded. This finding may help explain why well-designed head-to-head comparisons of statins and other drugs sometimes have contradictory results.
There are several possible explanations for our finding of the strong association between funding source and outcomes that are favorable to the drug company sponsor. First, it is possible that pharmaceutical companies selectively fund trials on drugs that are likely to produce a statistically significant result. This can be accomplished by selecting nonequivalent doses of drugs for testing [15–17]. A recent review of 42 RCTs comparing the low-density lipoprotein–lowering ability of two or mores statins found that almost all of the trials compared nonequivalent doses of statins . Second, as we examined only published studies, publication bias, or the phenomenon of statistically significant results being published more frequently than statistically nonsignificant results, may explain the association of funding and outcome . Selective reporting of outcomes can also contribute to biased results and conclusions [4–6]. In addition, industry sponsorship may be associated with multiple reporting of studies with favorable findings, emphasizing the imbalance towards statistically significant results in the published literature [18,41,42]. Finally, more than one third of the studies in our sample had no disclosed sponsorship. It is possible that industry funders or industry-supported authors could fail to disclose the sponsorship of a published study if the findings do not support the sponsor's product. This, however, would have to be very prevalent to explain by itself the results in Tables 4 and 5 concerning sponsorship. For example, nearly all the 29 studies with favorable conclusions and undisclosed funding would have to be actually funded by the comparator drug company, and nearly all the 45 studies without favorable conclusions would have to be funded by the test drug company in order to erase the difference shown in Table 4.
We identified a number of weaknesses common to published statin versus statin or non-statin drug comparisons that bring into question their clinical relevance. The most important weakness was the lack of patient-related clinical-outcome measures. Looking at the totality of available evidence, we found that almost all studies (98%, 189/192) used only surrogate outcome measures. Inadequate blinding, lack of concealment of allocation, poor follow-up, and lack of intention-to-treat analyses were common among these studies. These weaknesses suggest that these types of studies should be used with great caution by those making regulatory and purchasing decisions.
We found that adequate double blinding was an influential design feature in our sample. Adequately blinded studies were less likely to report results favoring the sponsor's product. Although RCTs of poorer quality are more likely to reach biased conclusions [10,43,44], the unreliability of quality scores has led to the recommendation that trial quality be assessed for individual features that are aimed to reduce bias, such as concealment of allocation or blinding . Recent research further suggests that specific study design characteristics are not reliably associated with treatment effect sizes across different studies and medical areas . Thus, the study design characteristics associated with statistically significant results might vary with the type of research being examined. We used three items from the Chalmers et al.  methodological quality assessment scale to assess study design features that might be associated with results and conclusions. These items focus on important aspects of trial design: (1) concealment of allocation, (2) control for whether all patients enrolled in the trial were included in the analysis (drop outs and intention-to-treat analyses), and (3) blinding of participants and investigators. Previous studies showed that these specific characteristics are associated with bias in clinical trials [10,47,48]. For example, Schulz and colleagues found that estimates of treatment effects were exaggerated by 41% for inadequately concealed trials and by 17% for trials with inadequate double blinding .
Finally, journal characteristics may influence the results and conclusions of articles as the quality of the reporting may vary with the journal. For example, articles published in peer-reviewed journals have superior quality compared to articles published in non-peer-reviewed journals [15,27]. In our sample, we had no variability in peer-review status, but we did observe a small possible association between journal impact factors and results and conclusions that favored the test drug.
Our study has several limitations, including our ability to identify funding sources and financial ties. We categorized studies as industry funded or not based on each article's disclosure of a trial's funding source(s). Krimsky showed, however, that there is a lack of disclosure of industry research support and personal financial ties across a wide variety of journals [49,50]. Thus, we may be underestimating the number of industry-sponsored studies and personal financial ties of investigators.
The market for statins is competitive. The National Cholesterol Education Program update of the Adult Treatment Panel III guidelines expanded both the scope and intensity of low-density lipoprotein–lowering therapy for prevention of cardiovascular disease . To achieve the goals in the guideline, millions of Americans would need to be placed on cholesterol-lowering medication in higher doses and for a longer period, thereby increasing the number of prescriptions for statin drugs [51,52]. Eight of the nine members of the National Cholesterol Education Program panel had financial ties with pharmaceutical companies that manufactured statin drugs [51,52]. Our findings suggest that available data on choosing between statins based on head-to-head comparisons may also be influenced by financial conflicts of interest. Our findings may be generalizable to other classes of drugs with competitive markets.
There is increasing concern that the funding source influences outcomes and conclusions of medical research . At the same time, industry support of biomedical research has increased dramatically during the past few decades [53,54]. The growing proportion of industry-funded studies could shift the balance of published trials more towards studies that favor new drugs . This trend and our finding that, for one class of drugs, the results and conclusions of trials tend to favor the drug that is made by the sponsor raises important considerations for selecting drugs within a class. Sponsorship bias, even when controlling for other confounding study characteristics, may be the main explanation for contradictory findings of drug–drug comparison trials. This bias in drug–drug comparison trials should be considered when making health-policy decisions regarding drug choice, such as drug formulary decisions. Reviewers of published reports that disclose funding by the makers of the product being tested should be more critical of the methods than if the reports are not industry sponsored 
Table S1. Included Studies Table
(62 KB PDF)
Table S2. Statin Data: Adjudicated Full and Subsample
(105 KB XLS)
Table S3. Statin-Only Comparison Data
(48 KB XLS)
We thank Christina Zhukovsky for translating the Russian articles, entering data, and obtaining the peer-review status of articles. We thank Joaquin Barnoya for translating the Spanish articles. We thank Erika Campbell for administrative assistance. This work was conducted as an Extra Junior Research Project at the Department of Clinical Pharmacy, University of California, San Francisco.
LB designed the study, analyzed data, and drafted the manuscript. FO collected data. KL advised on study design. LB, FO, and KL coded data. FO, PB, and KL advised on analysis. PB conducted analysis. FO, PB, and KL revised the paper.
- 1. Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2: e124. JP Ioannidis2005Why most published research findings are false.PLoS Med2e124
- 2. Bero LA, Rennie D (1996) Influences on the quality of published drug studies. Int J Technol Assess Health Care 12: 209–237.LA BeroD. Rennie1996Influences on the quality of published drug studies.Int J Technol Assess Health Care12209237
- 3. Smith R (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Med 2: e138. R. Smith2005Medical journals are an extension of the marketing arm of pharmaceutical companies.PLoS Med2e138
- 4. Chan AW, Altman DG (2005) Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. BMJ 330: 753.AW ChanDG Altman2005Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors.BMJ330753
- 5. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG (2004) Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291: 2457–2465.AW ChanA. HrobjartssonMT HaahrPC GotzscheDG Altman2004Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles.JAMA29124572465
- 6. Chan AW, Krleza-Jeric K, Schmid I, Altman DG (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 171: 735–740.AW ChanK. Krleza-JericI. SchmidDG Altman2004Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.CMAJ171735740
- 7. Dickersin K (1990) The existence of publication bias and risk factors for its occurrence. JAMA 263: 1385–1389.K. Dickersin1990The existence of publication bias and risk factors for its occurrence.JAMA26313851389
- 8. Chalmers TC, Block JB, Lee S (1972) Controlled studies in clinical cancer research. N Engl J Med 287: 75–78.TC ChalmersJB BlockS. Lee1972Controlled studies in clinical cancer research.N Engl J Med2877578
- 9. Colditz GA, Miller JN, Mosteller F (1989) How study design affects outcomes in comparisons of therapy. I: Medical. Stat Med 8: 441–454.GA ColditzJN MillerF. Mosteller1989How study design affects outcomes in comparisons of therapy. I: Medical.Stat Med8441454
- 10. Schulz KF, Chalmers I, Hayes RJ, Altman DG (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273: 408–412.KF SchulzI. ChalmersRJ HayesDG Altman1995Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.JAMA273408412
- 11. Ioannidis JP, Trikalinos TA, Ntzani EE, Contopoulos-Ioannidis DG (2003) Genetic associations in large versus small studies: An empirical assessment. Lancet 361: 567–571.JP IoannidisTA TrikalinosEE NtzaniDG Contopoulos-Ioannidis2003Genetic associations in large versus small studies: An empirical assessment.Lancet361567571
- 12. Ioannidis JP, Cappelleri JC, Lau J (1998) Issues in comparisons between meta-analyses and large trials. JAMA 279: 1089–1093.JP IoannidisJC CappelleriJ. Lau1998Issues in comparisons between meta-analyses and large trials.JAMA27910891093
- 13. Misakian AL, Bero LA (1998) Publication bias and research on passive smoking: Comparison of published and unpublished studies. JAMA 280: 250–253.AL MisakianLA Bero1998Publication bias and research on passive smoking: Comparison of published and unpublished studies.JAMA280250253
- 14. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL (2003) Association of funding and conclusions in randomized drug trials: A reflection of treatment effect or adverse events? JAMA 290: 921–928.B. Als-NielsenW. ChenC. GluudLL Kjaergard2003Association of funding and conclusions in randomized drug trials: A reflection of treatment effect or adverse events?JAMA290921928
- 15. Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, et al. (1994) A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch Intern Med 154: 157–163.PA RochonJH GurwitzRW SimmsPR FortinDT Felson1994A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis.Arch Intern Med154157163
- 16. Safer DJ (2002) Design and reporting modifications in industry-sponsored comparative psychopharmacology trials. J Nerv Ment Dis 190: 583–592.DJ Safer2002Design and reporting modifications in industry-sponsored comparative psychopharmacology trials.J Nerv Ment Dis190583592
- 17. Djulbegovic B, Lacevic M, Cantor A, Fields KK, Bennett CL, et al. (2000) The uncertainty principle and industry-sponsored research. Lancet 356: 635–638.B. DjulbegovicM. LacevicA. CantorKK FieldsCL Bennett2000The uncertainty principle and industry-sponsored research.Lancet356635638
- 18. Johansen HK, Gotzsche PC (1999) Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis. JAMA 282: 1752–1759.HK JohansenPC Gotzsche1999Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis.JAMA28217521759
- 19. Brook O, Van Hout H, Nieuwenhuysea H, De Haan M (2003) Effects of coaching by community pharmacists on psychological symptoms of antidepressant users; a randomised controlled trial. Eur Neuropsychopharmacol 13: 347–354.O. BrookH. Van HoutH. NieuwenhuyseaM. De Haan2003Effects of coaching by community pharmacists on psychological symptoms of antidepressant users; a randomised controlled trial.Eur Neuropsychopharmacol13347354
- 20. Gomez Cerezo J, Lubomirov Hristov R, Carcas Sansuan AJ, Vazquez Rodriguez JJ (2003) Outcome trials of COX-2 selective inhibitors: Global safety evaluation does not promise benefits. Eur J Clin Pharmacol 59: 169–175.J. Gomez CerezoR. Lubomirov HristovAJ Carcas SansuanJJ Vazquez Rodriguez2003Outcome trials of COX-2 selective inhibitors: Global safety evaluation does not promise benefits.Eur J Clin Pharmacol59169175
- 21. Yaphe J, Edman R, Knishkowy B, Herman J (2001) The association between funding by commercial interests and study outcome in randomized controlled drug trials. Fam Pract 18: 565–568.J. YapheR. EdmanB. KnishkowyJ. Herman2001The association between funding by commercial interests and study outcome in randomized controlled drug trials.Fam Pract18565568
- 22. Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ 326: 1167–1170.J. LexchinLA BeroB. DjulbegovicO. Clark2003Pharmaceutical industry sponsorship and research outcome and quality: Systematic review.BMJ32611671170
- 23. Bekelman JE, Li Y, Gross CP (2003) Scope and impact of financial conflicts of interest in biomedical research: A systematic review. JAMA 289: 454–465.JE BekelmanY. LiCP Gross2003Scope and impact of financial conflicts of interest in biomedical research: A systematic review.JAMA289454465
- 24. Montgomery JH, Byerly M, Carmody T, Li B, Miller DR, et al. (2004) An analysis of the effect of funding source in randomized clinical trials of second generation antipsychotics for the treatment of schizophrenia. Control Clin Trials 25: 598–612.JH MontgomeryM. ByerlyT. CarmodyB. LiDR Miller2004An analysis of the effect of funding source in randomized clinical trials of second generation antipsychotics for the treatment of schizophrenia.Control Clin Trials25598612
- 25. Bhandari M, Busse JW, Jackowski D, Montori VM, Schuneman H, et al. (2004) Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ 170: 477–480.M. BhandariJW BusseD. JackowskiVM MontoriH. Schuneman2004Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials.CMAJ170477480
- 26. Procyshyn RM, Chau A, Fortin P, Jenkins W (2004) Prevalence and outcomes of pharmaceutical industry-sponsored clinical trials involving clozapine, risperidone, or olanzapine. Can J Psychiatry 49: 601–606.RM ProcyshynA. ChauP. FortinW. Jenkins2004Prevalence and outcomes of pharmaceutical industry-sponsored clinical trials involving clozapine, risperidone, or olanzapine.Can J Psychiatry49601606
- 27. Cho MK, Bero LA (1996) The quality of drug studies published in symposium proceedings. Ann Intern Med 124: 485–489.MK ChoLA Bero1996The quality of drug studies published in symposium proceedings.Ann Intern Med124485489
- 28. Kjaergard LL, Als-Nielsen B (2002) Association between competing interests and authors' conclusions: Epidemiological study of randomised clinical trials published in the BMJ. BMJ 325: 249.LL KjaergardB. Als-Nielsen2002Association between competing interests and authors' conclusions: Epidemiological study of randomised clinical trials published in the BMJ.BMJ325249
- 29. Barnes DE, Bero LA (1998) Why review articles on the health effects of passive smoking reach different conclusions. JAMA 279: 1566–1570.DE BarnesLA Bero1998Why review articles on the health effects of passive smoking reach different conclusions.JAMA27915661570
- 30. Heres S, Davis J, Maino K, Jetzinger E, Kissling W, et al. (2006) Why olazepine beats risperidone, risperidone beats quetiapine, and quetiapine beats olazepine: An exploratory analysis of head-to-head comparison studies of second generation antipsychotics. Am J Psychiatry 163: 185–194.S. HeresJ. DavisK. MainoE. JetzingerW. Kissling2006Why olazepine beats risperidone, risperidone beats quetiapine, and quetiapine beats olazepine: An exploratory analysis of head-to-head comparison studies of second generation antipsychotics.Am J Psychiatry163185194
- 31. Law M, Wald N, Rudnicka A (2003) Quantifying effect of statins on low density lipoprotein cholesterol, ischaemic heart disease, and stroke: Systematic review and meta-analysis. BMJ 326: 1423–1429.M. LawN. WaldA. Rudnicka2003Quantifying effect of statins on low density lipoprotein cholesterol, ischaemic heart disease, and stroke: Systematic review and meta-analysis.BMJ32614231429
- 32. Kelley C, Helfand M, Good C, Ganz M (2006) Drug class review: Hydroxymethylglutaryl-coenzyme A reductase inhibitors (statins). VHA Pharmacy Benefits Management Strategic Healthcare Group and Medical Advisory Panel. C. KelleyM. HelfandC. GoodM. Ganz2006Drug class review: Hydroxymethylglutaryl-coenzyme A reductase inhibitors (statins).VHA Pharmacy Benefits Management Strategic Healthcare Group and Medical Advisory PanelAvailable at: http://www.pbm.va.gov/reviews/hmgstatins04-09-03.pdf. Accessed 18 August 2006. Available at: http://www.pbm.va.gov/reviews/hmgstatins04-09-03.pdf. Accessed 18 August 2006.
- 33. International Committee of Medical Journal Editors (1997) Uniform requirements for manuscripts submitted to biomedical journals. Ann Intern Med 126: 36–47.International Committee of Medical Journal Editors1997Uniform requirements for manuscripts submitted to biomedical journals.Ann Intern Med1263647
- 34. Ioannidis JP, Cappelleri JC, Sacks HS, Lau J (1997) The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Control Clin Trials 18: 431–444.JP IoannidisJC CappelleriHS SacksJ. Lau1997The relationship between study design, results, and reporting of randomized clinical trials of HIV infection.Control Clin Trials18431444
- 35. Ioannidis JP, Polycarpou A, Ntais C, Pavlidis N (2003) Randomised trials comparing chemotherapy regimens for advanced non-small cell lung cancer: Biases and evolution over time. Eur J Cancer 39: 2278–2287.JP IoannidisA. PolycarpouC. NtaisN. Pavlidis2003Randomised trials comparing chemotherapy regimens for advanced non-small cell lung cancer: Biases and evolution over time.Eur J Cancer3922782287
- 36. Institute for Scientific Information (2005) Science citation index: Journal citation reports. Philadelphia (Pennsylvania): Institute for Scientific Information. Institute for Scientific Information2005Science citation index: Journal citation reportsPhiladelphia (Pennsylvania)Institute for Scientific InformationAvailable at: http://portal.isiknowledge.com/portal.cgi?DestApp=JCR&Func=Frame. Accessed 4 May 2007. Available at: http://portal.isiknowledge.com/portal.cgi?DestApp=JCR&Func=Frame. Accessed 4 May 2007.
- 37. The World Bank Group (2005) Country group data and statistics by income. Geneva: The World Bank. The World Bank Group2005Country group data and statistics by incomeGenevaThe World BankAvailable at: http://web.worldbank.org/WBSITE/EXTERNAL/DATASTATISTICS/0,,contentMDK:20421402~pagePK:64133150~piPK:64133175~theSitePK:239419,00.html#Low_income. Accessed 4 May 2007. Available at: http://web.worldbank.org/WBSITE/EXTERNAL/DATASTATISTICS/0,,contentMDK:20421402~pagePK:64133150~piPK:64133175~theSitePK:239419,00.html#Low_income. Accessed 4 May 2007.
- 38. Chalmers I, Adams M, Dickersin K, Hetherington J, Tarnow-Mordi W, et al. (1990) A cohort study of summary reports of controlled trials. JAMA 263: 1401–1405.I. ChalmersM. AdamsK. DickersinJ. HetheringtonW. Tarnow-Mordi1990A cohort study of summary reports of controlled trials.JAMA26314011405
- 39. Friedman L, Richter E (2004) Relationship between conflicts of interest and research results. J Gen Intern Med 19: 51–56.L. FriedmanE. Richter2004Relationship between conflicts of interest and research results.J Gen Intern Med195156
- 40. Dickersin K, Min Y-I, Meinert C (1992) Factors influencing publication of research results. JAMA 267: 374–378.K. DickersinY-I MinC. Meinert1992Factors influencing publication of research results.JAMA267374378
- 41. Gotzsche PC (1989) Multiple publication of reports of drug trials. Eur J Clin Pharmacol 36: 429–432.PC Gotzsche1989Multiple publication of reports of drug trials.Eur J Clin Pharmacol36429432
- 42. Tramer M, Reynolds D, Moore R, McQuay H (1997) Impact of covert duplication publication on meta-analysis: A case study. BMJ 315: 635–640.M. TramerD. ReynoldsR. MooreH. McQuay1997Impact of covert duplication publication on meta-analysis: A case study.BMJ315635640
- 43. Davidson R (1986) Source of funding and outcome of clinical trials. J Gen Intern Med 3: 155–158.R. Davidson1986Source of funding and outcome of clinical trials.J Gen Intern Med3155158
- 44. Moher D, Cook D, Jadad A, Tugwell P, Moher M, et al. (1999) Assessing the quality of reports of randomised trials: Implications for the conduct of meta-analysis. Health Technol Assess 3: 1–98.D. MoherD. CookA. JadadP. TugwellM. Moher1999Assessing the quality of reports of randomised trials: Implications for the conduct of meta-analysis.Health Technol Assess3198
- 45. Juni P, Witschi A, Bloch R, Egger M (1999) The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282: 1054–1060.P. JuniA. WitschiR. BlochM. Egger1999The hazards of scoring the quality of clinical trials for meta-analysis.JAMA28210541060
- 46. Balk E, Bonis P, Moskowitz H, Schmid CH, Ioannidis J, et al. (2002) Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 287: 2973–2982.E. BalkP. BonisH. MoskowitzCH SchmidJ. Ioannidis2002Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials.JAMA28729732982
- 47. Schulz K, Chalmers I, Altman D (2002) The landscape and lexicon of blinding in randomized trials. Ann Intern Med 136: 254–259.K. SchulzI. ChalmersD. Altman2002The landscape and lexicon of blinding in randomized trials.Ann Intern Med136254259
- 48. Schulz K, Chalmers I, Altman D, Grimes D, Dore C (1995) The methodologic quality of randomization as assessed from reports of trials in specialist and general medical journals. Online J Curr Clin Trials 187: 81.K. SchulzI. ChalmersD. AltmanD. GrimesC. Dore1995The methodologic quality of randomization as assessed from reports of trials in specialist and general medical journals.Online J Curr Clin Trials18781
- 49. Krimsky S, Rothenberg LS (2001) Conflict of interest policies in science and medical journals: Editorial practices and author disclosures. Sci Eng Ethics 7: 205–218.S. KrimskyLS Rothenberg2001Conflict of interest policies in science and medical journals: Editorial practices and author disclosures.Sci Eng Ethics7205218
- 50. Krimsky S, Rothernberg L (1998) Financial interest and its disclosure in scientific publications. JAMA 280: 225–226.S. KrimskyL. Rothernberg1998Financial interest and its disclosure in scientific publications.JAMA280225226
- 51. Grundy S, Cleeman J, Merz C, Brewer H, Clark L, et al. (2004) Implications of recent clinical trials for the National Cholesterol Education Program Adult Treatment Panel III Guidelines. Arterioscler Thromb Vasc Biol 24: e149–e161.S. GrundyJ. CleemanC. MerzH. BrewerL. Clark2004Implications of recent clinical trials for the National Cholesterol Education Program Adult Treatment Panel III Guidelines.Arterioscler Thromb Vasc Biol24e149e161
- 52. Quilliam B, Perez H, Andros V, Jones P (2004) Quantifying the effect of applying the NCEP ATP III criteria in a managed care population treated with statin therapy. J Manag Care Pharm 10: 244–250.B. QuilliamH. PerezV. AndrosP. Jones2004Quantifying the effect of applying the NCEP ATP III criteria in a managed care population treated with statin therapy.J Manag Care Pharm10244250
- 53. Buchkowsky S, Jewesson P (2004) Industry sponsorship and authorship of clinical trials over 20 years. Ann Pharmacother 38: 579–585.S. BuchkowskyP. Jewesson2004Industry sponsorship and authorship of clinical trials over 20 years.Ann Pharmacother38579585
- 54. Dorman P, Counsell C, Sandercock P (1999) Reports of randomized controlled trials in acute stroke, 1955–1995. What proportions were commercially sponsored? Stroke 30: 1995–1998.P. DormanC. CounsellP. Sandercock1999Reports of randomized controlled trials in acute stroke, 1955–1995. What proportions were commercially sponsored?Stroke3019951998
- 55. Patsopoulos N, Ioannidis J, Analatos A (2006) Origin and funding of the most frequently cited papers in medicine: Database analysis. BMJ 332: 1061–1064.N. PatsopoulosJ. IoannidisA. Analatos2006Origin and funding of the most frequently cited papers in medicine: Database analysis.BMJ33210611064
- 56. Chaudhry S, Schroter S, Smith R, Morris J (2002) Does declaration of competing interests affect readers' perceptions? A randomised trial. BMJ 325: 1391–1392.S. ChaudhryS. SchroterR. SmithJ. Morris2002Does declaration of competing interests affect readers' perceptions? A randomised trial.BMJ32513911392