The $1.1 billion investment in comparative effectiveness research will reshape the evidence-base supporting decisions about treatment effectiveness, safety, and cost. Defining the current prevalence and characteristics of comparative effectiveness (CE) research will enable future assessments of the impact of this program.
We conducted an observational study of clinical trials addressing priority research topics defined by the Institute of Medicine and conducted in the US between 2007 and 2010. Trials were identified in ClinicalTrials.gov. Main outcome measures were the prevalence of comparative effectiveness research, nature of comparators selected, funding sources, and impact of these factors on results.
231 (22.3%; 95% CI 19.8%–24.9%) studies were CE studies and 804 (77.7%; 95% CI, 75.1%–80.2%) were non-CE studies, with 379 (36.6%; 95% CI, 33.7%–39.6%) employing a placebo control and 425 (41.1%; 95% CI, 38.1%–44.1%) no control. The most common treatments examined in CE studies were drug interventions (37.2%), behavioral interventions (28.6%), and procedures (15.6%). Study findings were favorable for the experimental treatment in 34.8% of CE studies and greater than twice as many (78.6%) non-CE studies (P<0.001). CE studies were more likely to receive government funding (P = 0.003) and less likely to receive industry funding (P = 0.01), with 71.8% of CE studies primarily funded by a noncommercial source. The types of interventions studied differed based on funding source, with 95.4% of industry trials studying a drug or device. In addition, industry-funded CE studies were associated with the fewest pediatric subjects (P<0.001), the largest anticipated sample size (P<0.001), and the shortest study duration (P<0.001).
Citation: Bourgeois FT, Murthy S, Mandl KD (2012) Comparative Effectiveness Research: An Empirical Study of Trials Registered in ClinicalTrials.gov. PLoS ONE 7(1): e28820. https://doi.org/10.1371/journal.pone.0028820
Editor: Laxmaiah Manchikanti, University of Louisville, United States of America
Received: October 24, 2011; Accepted: November 15, 2011; Published: January 9, 2012
Copyright: © 2012 Bourgeois et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Dr. Bourgeois was supported by a training grant (5T32HD040128) from the National Institute of Child Health and Human Development, and Dr. Mandl by a grant from the National Library of Medicine (5G08LM009778), National Institutes of Health. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Comparative effectiveness (CE) research is the “generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat and monitor health conditions in ‘real world’ settings”. Recognizing that the evidence-base for the practice of medicine is often built on studies lacking active comparators and therefore falls short in supporting either high quality care or healthcare reform, there is now substantial focus on and investment in CE research. In the United Kingdom, for instance, the National Institute for Health and Clinical Excellence compiles and disseminates CE and cost-effectiveness data to support diagnostic and therapeutic decisions.,  Similar agencies in Canada and Australia—the Common Drug Review and the Pharmaceutical Benefits Advisory Committee, respectively—provide information on the effectiveness and cost of pharmaceuticals, specifically, compared to relevant alternatives., , 
In the United States, CE research was recently appropriated $1.1 billion through the American Recovery and Reinvestment Act of 2009.,  This funding reflects the growing awareness that improved data is needed on the relative benefits of therapies to enable patients and clinicians to make informed decisions and to reduce gross geographic variations in healthcare allocations seen across the United States., 
In order to envision how the evolution of CE research will shape the evidence-base for future healthcare delivery, we sought to leverage a novel data source of clinical trials—the web-based registry ClinicalTrials.gov—and measure the prevalence of CE research and characterize current CE research activity. We focus our empirical study on research conducted in the United States where the concerted effort to expand CE research has not yet had a substantial impact on studies performed. Specifically, we examine research areas highlighted in the 2009 Institute of Medicine (IOM) list of 100 priority topics deemed to be most pertinent to improving the health of the population, commissioned by the United States Congress to inform the initial investment in CE research. Since we focus on research activity in the United States, we limit our study to trials registered in ClinicalTrials.gov, which is the primary registry employed by investigators in the United States and which has previously been used to define and study large trial cohorts.,  To begin to anticipate the impact of the investment in CE research, we determine the prevalence of CE research to date, the types of interventions studied, and the role of funding sources sponsoring CE research.
Selection of Clinical Studies
We examined the 15 research areas among the top 25 topics on the IOM list of priority areas that addressed specific diseases or conditions as opposed to strategies for delivering care or diagnostic and treatment approaches for broad groups of conditions (Table S1). We identified studies pertaining to these research areas in ClinicalTrials.gov, selecting trials that were registered between January 1, 2007 and April 26, 2010 (date of data download from ClinicalTrials.gov) and that were conducted in the United States (Figure 1).
Selection of trials in ClinicalTrials.gov that address 15 research topics identified by the Institute of Medicine as being top priority for comparative effectiveness research.
ClinicalTrials.gov is a web-based registry of clinical studies that provides a publicly available source of information on clinical studies conducted in the United States and internationally. In 2005, the International Committee of Medical Journal Editors instituted a policy requiring prospective registration of all trials—regardless of intervention type—as a prerequisite for publication, resulting in a dramatic increase in the registration of trials and sustained wide-spread use since then.,  In addition, under the FDA Amendments Act of 2007, the FDA requires the registration in ClincialTrials.gov of all clinical investigations (except phase I trials) of a drug, biologic, or device that is subject to FDA regulation, regardless of trial design. Users can query the registry and identify specific types of trials using a search function that includes keyword searches. We employed keywords identified from published systematic reviews on the diseases or conditions of interest (Table S1). Studies selected using this search strategy were individually reviewed and those included that directly addressed the research topics of interest.
Definitions and Data Extraction
CE studies were defined as those comparing the experimental intervention to another active therapy as opposed to a placebo control or no comparator. Both the experimental treatment and the comparator were classified as an intervention involving a drug, device, procedure, behavioral change, or other treatment (e.g. dietary supplement). Active comparator studies were defined as studies that compared two treatment alternatives, including “optimal usual care” when these reflected appropriate current practice and standards.,  In determining the type of comparator employed, we did not rely on the investigator-assigned study labels in ClinicalTrials.gov but rather examined the detailed description of the study in the record.
The data elements obtained from the ClincialTrials.gov entry and recorded for each study were registration date, study start and completion dates, experimental treatment under study, comparator type, trial phase for drug and device studies, funding source, outcome measures, anticipated enrollment number, subject age groups, and elements of the study design.
Study outcome measures are specifically listed in the study record as primary and secondary outcomes and we determined whether these included measures of safety, including any side effects, adverse events, or other potential harms or risks related to the intervention, or cost assessments, including formal cost-analyses and general measures of resource utilization. For 17 studies that did not include specific outcome measures in the study record, we reviewed the study descriptions to identify the inclusion of safety and cost assessments.
Funding source was classified as government, industry, or nonprofit based on the funding sponsors listed in the record. We categorized “NIH”, “U.S. federal”, and “other government” as government funding; “industry” as industry funding; and “network”, “individual”, and “other” (which includes universities, hospitals, foundations, and other nonprofit organizations) as nonprofit funding. We applied this classification to primary and secondary funding sources. Subject ages are categorized in the registry as “Child” (up to 17 years), “Adult” (18 to 65 years), “Senior” (66 years and older), and combinations of these groups. We re-coded these data into a three-level variable of children only, children and adults/seniors, and adults/seniors.
Classification of experimental intervention, comparator type, and safety and cost assessments were performed independently by two of the authors (F.B. and S.M.) and disagreements resolved by consensus.
Assessment of Study Outcomes
Publications associated with studies were identified using a previously described method. Briefly, for studies that did not include results or a reference to a publication within the CinicalTrials.gov record, four electronic databases were searched. These included PubMed, the Cochrane Library, EMBASE, and the Cumulative Index to Nursing and Allied Health Literature (CINAHL). All searches were finalized by August 31, 2010. Each publication was reviewed and the results for the primary outcome classified as favorable (i.e. statistically significant based on P values or confidence intervals) or not favorable (i.e. not statistically significant) for the experimental treatment. For studies without a comparator or statistical analysis, the classification was based on the interpretation of the results provided in the study conclusions. Publications that did not describe results pertaining to the efficacy or safety of the intervention were classified as “neither”. Two of the authors (F.B. and S.M.) independently performed the outcome classification and resolved disagreements by consensus. Inter-coder agreement for assigning study outcomes was good with a kappa of 0.78 (95% CI, 0.65–0.91).
We calculated the proportion of studies that were CE studies and compared study characteristics for CE and non-CE studies. Sub-analyses were performed on CE studies based on funding source. Trials examining a pharmaceutical intervention were also specifically examined and CE and non-CE studies compared. Chi-square and Kruskal-Wallis tests were used to compare categorical and median values, respectively. We used the Cochran-Mantel-Haenszel test to control for funding source when examining study outcomes. All data were analyzed with SAS software (version 9.2, SAS Institute Inc., Cary, North Carolina).
Of the 3167 studies retrieved from ClinicalTrials.gov, 1035 were included in the final study sample after reviewing the study description (Figure 1). Among these, 231 (22.3%; 95% confidence interval [CI], 19.8%–24.9%) were CE studies and 804 (77.7%; 95% CI, 75.1%–80.2%) were non-CE studies, with 379 (36.6%; 95% CI, 33.7%–39.6%) employing a placebo control and 425 (41.1%; 95% CI, 38.1%–44.1%) no control.
Study characteristics are presented in Table 1. In half the studies examined (49.9%), the experimental treatment consisted of a pharmacological therapy and in 18.3% a behavioral intervention. The distribution of experimental treatments differed for the different study types, with drug treatments more likely to be studied with a placebo or no intervention (P<0.001). Studies with active comparators were more likely to be in advanced phases (Phase 3 or 4; P<0.001), to employ larger sample sizes (P<0.001), and to be longer in duration (P = 0.02). Fewer studies with active comparators included a primary safety outcome (8.2% vs. 14.0% and 23.8% for placebo-controlled and no comparator studies; P<0.001) and only 3.5% included a cost assessment.
Impact of Funding Source on Characteristics of CE Studies
The distribution of primary funding sources was similar among CE and non-CE studies (Table 2). Overall, 71.8% (n = 166) of CE studies were funded by non-commercial sources, including government and nonprofit organizations. CE studies were more likely to include government funding (32.5% compared with 22.9% of non-CE studies, P = 0.003) and less likely to include industry funding (37.2% compared with 46.4% of non-CE studies, P = 0.01).
We further examined study interventions and other characteristics for CE studies based on funding source (Table 3). Among CE studies funded primarily by industry, 95.4% involved the study of a drug or device and most compared the intervention to another drug or device (90.8%). Primarily industry-funded CE studies involved the largest anticipated sample size (median of 324 subjects vs. 175 and 100 subjects for government and nonprofit funding, respectively; P<0.001), were the least likely to enroll pediatric subjects (7.6% vs. 37.5% and 17.3% for government and nonprofit funding, respectively; P<0.001), and were the shortest in duration (median length 1.8 years vs. 3.0 and 2.4 years for government and nonprofit funding, respectively; P<0.001).
Studies with any type of government funding were less likely to study a drug or device (P<0.001) and more likely to include children (P<0.001) and be longer in duration (P<0.001). Government funding was not associated with an increase in the study of safety or cost outcomes.
Pharmaceutical and Device Studies and CE Research
Among the subset of 516 studies examining a pharmaceutical intervention, 86 (16.7%; 95% CI, 13.4%–19.9%) were CE studies and 430 (83.3%; 95% CI, 80.1%–86.6%) were non-CE studies.
Government sources provided primary funding for 11.6% (n = 10) and industry for 52.3% (n = 45) of CE drug studies (Table 4). CE studies were less likely to include a safety outcome compared with non-CE studies (P<0.001), involved larger anticipated sample sizes (median of 238 subjects vs. 80 subjects; P<0.001), and were less likely to be double-blinded (66.7% vs. 92.7%; P<0.001). Device studies included 32 (26.4%; 95% CI, 18.6%–34.3%) CE studies and 89 (73.6%; 95% CI, 65.7%–81.4%) non-CE studies. Government was the primary funding source for 6.2% (n = 2) and industry 53.1% (n = 17) of the CE device studies.
CE Study Outcomes
Results were identified for 115 (11.1%) studies. A total of 8/23 (34.8%) reports described positive findings for studies with active controls compared with 66/84 (78.9%) among non-CE studies (12/32 [73.9%] placebo-controlled trials and 6/32 [84.2%] trials without controls) (P<0.001). Among trials primarily funded by industry, 33/41 (80.5%) reported positive findings compared with 41/66 (61.7%) among all others (P = 0.04). After controlling for primary funding source, CE studies remained less likely to report positive findings (P<0.007 for Cochran-Mantel-Haenszel test). Among CE studies involving a drug therapy, findings were positive for 30.0% (n = 3) of CE studies compared with 81.6% (n = 40) of non-CE studies (P<0.001, Cochran-Mantel-Haenszel test controlling for primary funding source). None of the CE studies examining devices and 71.4% (n = 5) of non-CE studies involving devices reported findings favorable for the device (P = 0.04).
We provide a benchmark for the current state of CE research, demonstrating that for conditions deemed as highest priority by the IOM, less than a quarter of studies examined comparative effectiveness. The majority of CE studies were funded by government and nonprofit sources and outcomes were less likely to be positive for the experimental intervention among CE trials compared with non-CE trials. Funding sources had a substantial impact on the characteristics of CE studies, with industry-funded trials focusing primarily on drugs and devices and those funded by noncommercial sources addressing more diverse types of interventions. Industry-funded trials also differed in trial design with larger sample sizes, fewer studies involving pediatric patients, and shorter study periods.
Only a small proportion of CE studies address safety and cost outcomes, highlighting an opportunity for government-sponsored CE research to play a significant role. , , ,  Regardless of funding source, CE studies are less likely to examine safety outcomes, particularly among drug studies, demonstrating an emphasis on measuring treatment efficacy over measuring treatment risks and adverse events. Cost assessments are currently rare for both CE and non-CE studies across all funding sources.
Hochman et al previously found low prevalence of CE studies and a higher rate of positive outcomes among non-CE studies of pharmaceuticals compared with CE studies.  Using a comprehensive and growing data source of recent and ongoing research activity, our results corroborate those findings. We further demonstrate that the low prevalence of CE research and differences in outcomes exist across CE studies with a broad array of interventions and that characteristics of CE studies vary substantially based on the funding source sponsoring the study.
With an ever-expanding list of diagnostic and therapeutic options, CE studies fill an important gap in informing clinicians whether an intervention is superior to existing and familiar alternatives. Our findings suggest characteristics of CE research that may produce specific shifts in the evidence-base towards more critical and comprehensive assessments of the intervention under study. From our findings, we extrapolate that the projected increase in the number of CE studies—particularly studies of drugs and devices funded by noncommercial sources—will increase the proportion of studies that fail to support adoption of the experimental treatment. We base this prediction on two findings.
The first is that CE studies are less likely than non-CE studies to report results that promote the use of the experimental intervention, reinforcing that this study design may produce more conservative results in terms of the superiority of a therapy compared to other treatments. Trials with inactive comparators have previously been shown to have a greater likelihood of achieving favorable findings., ,  Drug and device studies that employ non-active comparators and yield favorable outcomes may encourage the adoption and use of the experimental intervention even though information is lacking on how the drug or device compares to current standards of care.
Secondly, while noncommercial sources funded 71% of CE studies overall, industry funded the majority of CE drug and device studies, which biases toward results supporting the use of a product., , ,  Industry trials investigating drugs and devices are typically designed and conducted by the company marketing the product and there is substantial and well-documented evidence that these studies are more likely to report findings supporting the efficacy and safety of the product than noncommercially funded studies., , , , ,  In our study sample, industry-funded studies were more likely to report an outcome favoring the use of the intervention than noncommercially-funded studies, and only 17% of drug studies and 26% of device studies used an active comparator. Research on drugs and devices would benefit from greater participation of non-stakeholders—such as government sponsors—as well as greater oversight in study design in order to ensure rigorous and valid assessments of the effectiveness of these treatments.
There are several factors critical to ensuring the success of the new CE research initiative and the ability of CE research to improve clinical decision-making. Methodologically, CE studies must be large enough and have a sufficient patient follow up period to demonstrate not only equivalence, but superiority of one treatment compared to another. Randomized controlled trials, which are typically designed to ascertain the efficacy of an intervention in select patient populations and tightly controlled settings, may not reflect real-world outcomes or be generalizable to routine clinical practice, which is one of the defining principles of CE research. By contrast, pragmatic clinical trials and observational studies may provide results that are directly pertinent to clinicians and patients choosing between available therapies. In addition, comparative efficacy data must be timely and available prior to the widespread adoption of new products or interventions, as the lack of comparative evidence has resulted in the extensive use of a number of treatments later found to be less efficacious or safe than existing alternatives., 
A limitation of our study is that outcomes data are not available for all studies since we chose to examine recent and ongoing studies, in order to ensure that our findings are most pertinent to the current state of CE research. However, it is unlikely that systematic bias produced our finding that CE studies are more likely to yield favorable outcomes. This finding is supported in prior literature and the sample of published results is of sufficient magnitude to demonstrate important and statistically significant differences in reported outcomes. ,  We were not able to verify the accuracy of data provided by investigators, but information such as experimental treatment, comparator type, and funding source, are likely properly and reliably reported. Finally, there are some missing data in ClincialTrials.gov, including anticipated sample size for 1% of trials and study duration for 17% of trials.
In conclusion, less than a quarter of studies use an active comparator to measure the CE of the treatment under investigation. Based on outcomes reported in CE and non-CE studies, CE studies in general appear to provide more rigorous assessments of the interventions under study. Boosting noncommercial funding of CE studies may be particularly critical to drug and device studies in order to ensure unbiased data on how the intervention compares to other available treatments. Further study is necessary to understand the impact of CE research on healthcare reform and cost, as there is a risk when new treatments face a higher barrier to acceptance that some innovation may be slowed and development costs increased. On the other hand, we can expect that CE research will provide physicians and patients with substantially stronger evidence about which therapies are effective.
Study Topics and Keywords for Study Selection. Clinical trials examined in the study pertained to these 15 research areas. The corresponding keywords were used to identify the trials in ClinicalTrials.gov using the embedded search function.
We would like to thank Professor Enrico Coiera, director of the Centre for Health Informatics, University of New South Wales, for his time in providing an insightful critical review of the manuscript.
Conceived and designed the experiments: FTB SM KDM. Performed the experiments: FTB SM. Analyzed the data: FTB SM. Contributed reagents/materials/analysis tools: FTB SM KDM. Wrote the paper: FTB SM KDM.
- 1. Federal Coordinating Council for Comparative Effectiveness Research (2009) Report to the President and the Congress. Washington, DC: Department of Health and Human Services.
- 2. Sox HC, Helfand M, Grimshaw J, Dickersin K, Tovey D, et al. (2010) Comparative effectiveness research: challenges for medical journals. PLoS Med 7: e1000269.
- 3. Rawlins MD (2004) NICE work–providing guidance to the British National Health Service. N Engl J Med 351: 1383–1385.
- 4. Steinbrook R (2008) Saying no isn't NICE - the travails of Britain's National Institute for Health and Clinical Excellence. N Engl J Med 359: 1977–1981.
- 5. Clement FM, Harris A, Li JJ, Yong K, Lee KM, et al. (2009) Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada. JAMA 302: 1437–1443.
- 6. Henry DA, Hill SR, Harris A (2005) Drug prices and value for money: the Australian Pharmaceutical Benefits Scheme. JAMA 294: 2630–2632.
- 7. Tierney M, Manns B (2008) Optimizing the use of prescription drugs in Canada through the Common Drug Review. CMAJ 178: 432–435.
- 8. (2009) U.S. House Appropriations Committee American Recovery and Reinvestment Act of 2009, HR, 111th Cong, 1st Sess.
- 9. Baicker K, Chandra A, Skinner JS, Wennberg JE (2004) Who you are and where you live: how race and geography affect the treatment of medicare beneficiaries. Health Aff (Millwood) Suppl Web Exclusives. pp. VAR33–44.
- 10. Mittler JN, Landon BE, Fisher ES, Cleary PD, Zaslavsky AM (2010) Market variations in intensity of Medicare service use and beneficiary experiences with care. Health Serv Res 45: 647–669.
- 11. Institute of Medicine (2009) Initial National Priorities for Comparative Effectiveness Research. Washington, DC: National Academies Press.
- 12. Bourgeois FT, Murthy S, Mandl KD (2010) Outcome reporting among drug trials registered in ClinicalTrials.gov. Ann Intern Med 153: 158–166.
- 13. Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM (2009) Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med 6: e1000144.
- 14. U.S. National Institutes of Health ClinicalTrials.gov.
- 15. DeAngelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. (2004) Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Med J Aust 181: 293–294.
- 16. Zarin DA, Tse T (2008) Medicine. Moving toward transparency of clinical trials. Science 319: 1340–1342.
- 17. U.S. Food and Drug Administration Amendments Act of 2007 (2007) Public Law No. pp. 105–185.
- 18. Hochman M, McCormick D (2010) Characteristics of published comparative effectiveness studies of medications. JAMA 303: 951–958.
- 19. American College of Physicians (2008) Information on cost-effectiveness: an essential product of a national comparative effectiveness program. Ann Intern Med 148: 956–961.
- 20. Alexander GC, Stafford RS (2009) Does comparative effectiveness have a comparative edge? JAMA 301: 2488–2490.
- 21. Weinstein MC, Skinner JA (2010) Comparative effectiveness and health care spending–implications for reform. N Engl J Med 362: 460–465.
- 22. Djulbegovic B, Lacevic M, Cantor A, Fields KK, Bennett CL, et al. (2000) The uncertainty principle and industry-sponsored research. Lancet 356: 635–638.
- 23. Stafford RS, Wagner TH, Lavori PW (2009) New, but not improved? Incorporating comparative-effectiveness information into FDA labeling. N Engl J Med 361: 1230–1233.
- 24. Bhandari M, Busse JW, Jackowski D, Montori VM, Schunemann H, et al. (2004) Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ 170: 477–480.
- 25. Ridker PM, Torres J (2006) Reported outcomes in major cardiovascular clinical trials funded by for-profit and not-for-profit organizations: 2000-2005. JAMA 295: 2270–2274.
- 26. Berger E (2008) Ghostwriters, data manipulation and dollar diplomacy: how drug companies pull the strings in clinical research. Ann Emerg Med 52: 137–139.
- 27. Smith R (2005) Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Med 2: e138.
- 28. Bero L, Oostvogel F, Bacchetti P, Lee K (2007) Factors associated with findings of published trials of drug-drug comparisons: why some statins appear more efficacious than others. PLoS Med 4: e184.
- 29. Vedula SS, Bero L, Scherer RW, Dickersin K (2009) Outcome reporting in industry-sponsored trials of gabapentin for off-label use. N Engl J Med 361: 1963–1971.
- 30. Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326: 1167–1170.
- 31. O'Connor AB (2010) Building comparative efficacy and tolerability into the FDA approval process. JAMA 303: 979–980.
- 32. Sorenson C, Naci H, Cylus J, Mossialos E (2011) Evidence of comparative efficacy should have a formal role in European drug approvals. BMJ 343: d4849.
- 33. Luce BR, Kramer JM, Goodman SN, Connor JT, Tunis S, et al. (2009) Rethinking randomized clinical trials for comparative effectiveness research: the need for transformational change. Ann Intern Med 151: 206–209.