The population with multiple chronic conditions is growing. Prior studies indicate that patients with comorbidities are frequently excluded from trials but do not address whether information is available in trials to draw conclusions about treatment effects for these patients.
Methods and Findings
We conducted a literature survey of trials from 11 Cochrane Reviews for four chronic diseases (diabetes, heart failure, chronic obstructive pulmonary disease, and stroke). The Cochrane Reviews systematically identified and summarized trials on the effectiveness of diuretics, metformin, anticoagulants, longacting beta-agonists alone or in combination with inhaled corticosteroids, lipid lowering agents, exercise and diet. Eligible studies were reports of trials included in the Cochrane reviews and additional papers that described the methods of these trials. We assessed the exclusion and inclusion of people with comorbidities, the reporting of comorbidities, and whether comorbidities were considered as potential modifiers of treatment effects. Overall, the replicability of both the inclusion criteria (mean [standard deviation (SD)]: 6.0 (2.1), range (min-max): 1–9.5) and exclusion criteria(mean(SD): 5.3 (2.1), range: 1–9.5) was only moderate. Trials excluded patients with many common comorbidities. The proportion of exclusions for comorbidities ranged from 0–42 percent for heart failure, 0–55 percent for COPD, 0–44 percent for diabetes, and 0–39 percent for stroke. Seventy of the 161 trials (43.5%) described the prevalence of any comorbidity among participants with the index disease. The reporting of comorbidities in trials was very limited, in terms of reporting an operational definition and method of ascertainment for the presence of comorbidity and treatments for the comorbidity. It was even less common that the trials assessed whether comorbidities were potential modifiers of treatment effects.
Citation: Boyd CM, Vollenweider D, Puhan MA (2012) Informing Evidence-Based Decision-Making for Patients with Comorbidity: Availability of Necessary Information in Clinical Trials for Chronic Diseases. PLoS ONE 7(8): e41601. doi:10.1371/journal.pone.0041601
Editor: Isabelle Boutron, University Paris Descartes, France
Received: April 5, 2012; Accepted: June 22, 2012; Published: August 3, 2012
Copyright: © Boyd et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported in part by the Johns Hopkins Bayview Center for Innovative Medicine, The Robert Wood Johnson Foundation Physician Faculty Scholars Program, and the Paul Beeson Career Development Award Program (NIA K23 AG032910, AFAR, The John A. Hartford Foundation, The Atlantic Philanthropies, The Starr Foundation, and an anonymous donor,) and AHRQ R21 HS018597-01 (Boyd). URLS include: http://rwjfpfsp.stanford.edu/about/index.html and http://www.afar.org/research/funding/beeson. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that (1) CMB and DPV had support from Robert Wood Johnson Foundation Physician Faculty Scholars Program for the submitted work and CB had support from the Paul Beeson Career Development Award Program (NIA K23 AG032910, AFAR, The John A. Hartford Foundation, The Atlantic Philanthropies, The Starr Foundation and an anonymous donor), and AHRQ R21 HS018597-01; (2) CMB, DV, and MAP have no relationships with companies that might have an interest in the submitted work in the previous 3 years; (3) their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and (4) CMB, MAP, and DPV have no non-financial interests that may be relevant to the submitted work. This does not alter the authors' adherence to all the PLoS ONE policies on sharing data and materials.
Recent reports have posed concerns about the challenges of caring for a growing population with multiple chronic conditions –. This population faces self-management challenges and adverse events and often experiences high health care utilization and poor quality of care , . One concern is an inadequate evidence base. The roots of this problem extend throughout the translational path, from the generation of the evidence to the synthesis of the evidence upon which clinical practice guidelines depend . Comorbid patients are frequently excluded from the relevant clinical trials –. And even when such patients are not explicitly excluded, it is often difficult to know whether the overall effect estimate can be extrapolated to individuals with comorbidities, or whether the effect of a treatment is different in those with or without comorbidities. Also the applicability of “average” trial results to specific individuals, or particular groups, such as complex patients with more competing risks than average, remains challenging –. These uncertainties about treatment effects in comorbid patients translate into systematic reviews and guideline development, making it difficult to establish treatment recommendations for these patients .
We designed this project to address these methodological issues. Many trials focus on treatments for a single condition, and may exclude people with comorbidities, and the typical primary hypothesis does not include effect modification. With rare exceptions, systematic reviews and guidelines also emphasize the single-disease perspective as they inherit the limitations of the trials and studies they review. For systematic reviews, the scientific community has not identified the best methods of determining and describing applicability of evidence for comorbid patients . Current methods for systematic reviews do not include explicit criteria as to how to frame a question and search for and evaluate evidence for a specific comorbid population. For example, should the scientific community value a study with a lower quality of evidence, in terms of rigor of design, because it has a greater inclusion and analytic attention on comorbidity, or is a more rigorous design preferred, even if the study is less focused on comorbidity or reports less of the necessary information. It is also unknown how many existing studies contain the necessary information to address comorbid patients in systematic reviews, and therefore in for clinical decision-making.
Diabetes, congestive heart failure, chronic obstructive pulmonary disease (COPD), and stroke may serve as typical conditions where we commonly see these methodological issues. Both older and younger people with these conditions often suffer from comorbidities –. While the number varies depending on the demographics of the population and the number of comorbidities assessed, less than 20% of people with any these conditions have the condition in isolation –. It is unclear whether the reporting of comorbidities in trials focused on these conditions allows for the extraction of information in order to consider comorbidity in systematic reviews. If reporting is insufficient, clinicians, guideline developers, and policy makers will not be able to make well-informed recommendations for clinical practice. Therefore, the aim of this study was to assess the inclusion of people with comorbidities and the reporting of comorbidities in trials of these four major chronic diseases, and whether comorbidities were considered as potential modifiers of treatment effects.
Study Design and Selection
We conducted a survey of trials reporting on drug and non-drug interventions in patients with the four common chronic diseases: COPD, heart failure, stroke and type II diabetes mellitus. Some results of this survey, addressing a different question, have been published earlier .
Ethics Statement: This work uses data in the published literature, and thus institutional review board approval was not sought.
We chose diseases associated with high morbidity, mortality and significant health care expenditures –. We focused on widely prescribed drug and non-drug therapies, and we wanted to have a complete set of trials. Therefore, we based the selection of randomized controlled trials (RCTs) on 11 Cochrane Reviews that systematically identified and summarized RCTs from 1944–2009 on the effectiveness of diuretics, metformin, anticoagulants, long-acting beta agonists alone or in combination with inhaled corticosteroids, lipid lowering agents, and the non-drug interventions exercise and diet for each of the four diseases –. The Cochrane reviews described the search strategy and eligibility criteria. We retrieved the main reports of included RCTs and retrieved additional papers that described the methods of these trials. We did not consider abstracts and unpublished data used in the Cochrane reviews because they could not provide the level of detail that we needed. We excluded 22 trials (out of 183) for this reason. The bibliography of excluded trials is available on request.
Before systematically extracting data from each trial, we developed a codebook that provided a detailed description of the information to be extracted and how to score it. We pilot tested the data extraction forms and the codebook on a random sample of 10 articles. One reviewer extracted all data and at least one other reviewer checked the data. They recorded and discussed all disagreements. If they were not able to resolve an issue, a third reviewer examined the article and all three reviewers discussed the issue until they resolved it.
Items related to Inclusion and Exclusion Criteria
Reporting and Replicability of Inclusion and Exclusion Criteria.
In order for us to better understand the population for whom researchers assessed treatment effects, the trial must clearly define inclusion and exclusion criteria. This is one criterion used for gauging how to generalize treatment effect estimates to a specific patient or group of patients , , , . We recorded whether or not each trial reported its inclusion and exclusion criteria. Of those studies that had inclusion and exclusion criteria, using a 10-point Likert-type scale [1 = not replicable, 10 = perfectly replicable, 5 = moderate (e.g. replicable for some criteria, irreplicable for others)], two investigators independently rated the replicability of the inclusion and exclusion criteria. More specifically, they rated the difficulty of applying the inclusion and exclusion criteria, as stated in the trial reports, to specific patients. We assessed the inter-rater agreement using the concordance correlation coefficient rho. To aggregate the two independent ratings, we took the mean of the two ratings for trials where the disagreement was <3 points. If disagreements in ratings were ≥3 points for any given trial, we rerated replicability and resolved disagreements through discussion between the two investigators.
Specific Exclusion Criteria.
We recorded whether or not the trials excluded people with the following conditions: pregnancy, recent or acute respiratory tract infections (pneumonia or COPD exacerbations), asthma or atopic conditions, COPD or emphysema, other lung disease (not asthma or COPD), coronary artery disease (history of myocardial infarction or angina), hypertension, arrhythmias, unspecified cardiac conditions (e.g. cardiovascular disease unspecified), renal insufficiency, liver insufficiency, heart failure, New York Heart Association Class IV Heart Failure, valvular disease, type II diabetes mellitus, type I diabetes mellitus, insulin therapy, damage from diabetes mellitus (nephropathy, retinopathy, neuropathy), peripheral vascular disease, oxygen therapy, oral steroid use, unspecified cancer, lung cancer, anemia, musculoskeletal diseases or disabilities, neurologic disabilities, other brain injuries (including hemorrhagic stroke), unable to exercise (unspecified), psychiatric illness, impaired mental status, and serious concomitant disease (unspecified). We chose conditions which are believed to be prevalent or potentially represent important treatment effect modifiers for these conditions , , . We also recorded whether there were specific age exclusions. Disagreements in the categorization of these exclusion criteria were resolved by discussion. If a “severe” version of one of these conditions was an exclusion criterion, we recorded it as an exclusion, even if a mild form of the condition was not an exclusion criterion for the trial. We also recorded if the trial had exclusion criteria other than these listed above or not.
Reporting of Definitions for the Presence or Absence of Comorbidities, Ascertainment Method and Use of Treatments
We were also interested in how often comorbidities were included as part of the description of the characteristics of the trial participants as selection procedures and trial participation also affect the characteristics of the trial participants. We recorded whether or not the study described a definition for those comorbidities for which prevalence was reported, and how the study obtained the information needed for this definition [(patient reported, medical record (includes doctor-reported), not reported, confirmed at baseline (like echo, blood tests, etc), others (e.g. questionnaires)]. We also recorded whether the studies reported any treatments for the comorbidities for which prevalence was reported. The comorbidities assessed were osteoporosis, diabetes mellitus, COPD, hypertension, coronary artery disease (angina or myocardial infarction), sleep apnea, heart failure, cancer, gastro-esophageal reflux, depression, dyslipidemia, osteoarthritis or joint pain, urinary problems, gout, chronic back pain or spinal stenosis, stroke, liver disease, chronic kidney disease, lung disease other than COPD or asthma, peripheral vascular disease, anxiety, arrhythmias, dementia or cognitive impairment, and obesity.
Are Comorbidities Treatment Effect Modifiers? (Subgroup Analyses)
Recognizing that authors have limited power to pre-plan subgroup analyses, we assessed how often and the way the study performed subgroup analyses using recently described criteria , , . We defined a subgroup analysis as the analysis of an effect that changed (or did not change) according to different levels of a variable measured before randomization. For those trials that did report one or more subgroup effects, we recorded whether the subgroup was related to a comorbidity. For those trials that reported a subgroup based on a comorbidity, we recorded whether these trials used an interaction term to compare effects across subgroups by comorbidity status and whether the trials assessed if the interaction terms were statistically independent from other subgroup effects.
Competing Risks and Heterogeneity of Treatment Effect Due to Different Baseline Risks
We recorded whether the trials considering competing risks in the design and analysis of trials. Competing risks are defined as events that occur when the time to a disease-specific endpoint of interest may be precluded by death or a major health event from another cause, and are increasingly recognized as relevant to decision making for older adults and people with multiple chronic conditions , –. We also were interested in whether or not the trials reported on the heterogeneity of treatments effects due to different baseline risks.
We included 161 trials. 91 percent (n = 147) reported inclusion criteria and 85 percent (n = 137) reported exclusion criteria. The replicability of inclusion and exclusion criteria was moderate in general. (See Table 1) Overall, for all four chronic diseases, the replicability of inclusion criteria was mean [standard deviation (SD)]: 6.0 (2.1), range (min-max): 1–9.5. The inter-rater agreement was good, with a concordance correlation coefficient rho of 0.82 [(95% confidence interval (CI) 0.77–0.87)] and a mean difference (SD) between reviewers of 0.13 (1.35). For all four chronic diseases, the replicability of exclusion criteria was slightly worse, with a mean (SD) of 5.3 (2.1), range (min-max): 1–9.5. Again, the inter-rater agreement was good, with a concordance correlation coefficient rho of 0.81 (95% CI 0.76–0.87), and a mean difference (SD) between reviewers of 0.02 (1.38). Replicability did not vary much across diseases. (Table 1).
The proportion of exclusions for comorbidities ranged from 0 to 42 percent for heart failure trials, 0 to 55 percent for COPD trials, 0 to 44 percent for diabetes trials, and 0 to 39 percent for stroke trials. (see Figures 1,2,3, and 4). Twenty-one percent of all trials had an older age exclusion of either age greater than 65, 75 or 80 years of age (13 percent for heart failure trials, 31 percent for COPD trials, 17 percent for diabetes trials, and 18 percent for stroke trials). Serious concomitant illness was a frequent exclusion, with 19 percent of heart failure trials having this exclusion, 33 percent of COPD trials, 23 percent of diabetes trials, and 36 percent of stroke trials) Considering the prevalence of comorbidities among patients with these four index conditions, people with common comorbidities were frequently excluded. For example, 42 percent of heart failure trials excluded people with coronary artery disease, 19 percent of heart failure trials excluded people with renal insufficiency. Similarly, 38 percent of diabetes trials excluded people with coronary artery disease and 44% excluded people with renal insufficiency. Twenty-seven percent of COPD trials excluded people with coronary artery disease, and 35 percent excluded people with musculoskeletal diseases. Among stroke trials, 30% excluded people with coronary artery disease and 24% excluded people with heart failure. The prevalence of the specific exclusions is reported in the Figure.
Seventy of the 161 trials (43.5%) described the prevalence of comorbidities among participants with the index disease. The median number (interquartile range) of comorbidities reported in those 70 trials was 3 (2–4), with 237 total comorbidities reported in these 70 trials. The majority of the 70 trials did not report definitions for these 237 comorbidities; only forty-five (19.0%) of the reported 237 comorbidities in the 70 trials had a definition of the comorbidity. These trials ascertained comorbidity from patient report for 19 (8.0%) comorbidities, from baseline examination for 73 (30.8%), from medical records for seven (3.0%) and from other methods for five (2.1%). The trials did not report the data source they used to ascertain the presence or absence of comorbidities for 133 (56.1%) out of the 237 comorbidities. The studies infrequently described whether or not a person received treatment for a comorbidity. The studies described treatments for 40 (16.9%) of the 237 comorbidities, whereas the studies did not describe treatments for 193 (81.3%) comorbidities and were unclear for 4 (0.2%) comorbidities.
Very few trials used the presence of a comorbidity as a subgroup variable to examine whether it is an effect modifier (n = 5/161, 3.1%). Importantly, none of them used interaction terms to formally test for effect modification. Only one trial (0.6%) considered heterogeneity of treatment effect due to different baseline risks. No trials considered the effects of competing risks .
Our analysis, based on studies of patients with any of four major diseases, found that evidence on the effect of treatments on comorbid patients is limited. The replicability of the inclusion and exclusion criteria is only moderate, thus making it hard to judge whether a specific patient or patient population would have been eligible to participate, and making it hard to replicate the clinical trial. Trials excluded patients with many common comorbidities. The reporting of comorbidities in trials of these four major chronic diseases was very limited, and it was even less common that the studies assessed whether comorbidities were potential modifiers of treatment effects.
Prior work has posited that comorbidities and older age are frequent exclusions for clinical trials –. Published reports of trials frequently modify eligibility criteria when compared to the original trial protocols, and that many of these relate to comorbidity . Our results build on this prior work that identifies a potential problem in the literature, in general, by examining the specific information needed to inform clinical decision-making in people with comorbidities, and highlight the challenges of using the current evidence base to determine how we should treat the rapidly increasing population of people with multiple chronic conditions –. The replicability of inclusion and exclusion criteria was poor in many of these studies, and only moderate on average, making it difficult to judge whether a patient or a specific patient population would have been eligible for the trial. This also makes it more difficult to replicate the study and compare populations across studies. Comparing populations across studies is an essential step for systematic reviews that attempt to synthesize the evidence base for specific clinical questions.
Serious concomitant diseases were very common exclusion criteria for trials, and yet how to replicate these determinations was not clear. The existence of multiple common comorbidities was a reason for exclusion in studies examining each of the four chronic diseases. For example, while 55 percent of COPD patients in a population-based study, the National Health and Nutrition Examination Study (NHANES) have arthritis, 35 percent of the COPD trials excluded people with musculoskeletal diseases . Sixteen percent of 45–64 year old diabetic patients, and thirty percent of 65 and older diabetic patients, have low glomerular filtration rate . However, 44 percent of the diabetes trials excluded patients with renal insufficiency. More than 60% of the congestive heart failure that occurs in the United States general population might be attributable to coronary heart disease, yet more than 40% of heart failure trials excluded people with coronary heart disease . The mismatch of eligibility criteria and the characteristics of a patient or patient population with the disease, affects our confidence in applying the results of the trial to these patients or patient populations. However, it is important to note that inclusion/exclusion criteria is an incomplete approach for determining whether a trial's results apply to a patient or a patient population with comorbidities, because restricting a trial population does not necessarily mean that the results do not apply to those excluded from the trial.
Also important is understanding what the patient characteristics were of people enrolled in a trial . Inclusion and exclusion criteria capture who was eligible, but may not reflect who was actually recruited for the trial. For example, even if older adults are not excluded from a trial, if the mean age is 50 years, with a SD of 15, these results may not apply to an 80-year-old man with multiple chronic conditions. Less than half of the trials reported the prevalence of comorbidities. Among those studies that reported prevalence of any comorbidities, the average number of comorbidities reported on was three. The trials also infrequently reported the definition of comorbidity and how the information needed for these definitions was obtained. Thus, determining the comorbidity burden on average of people in these trials was next to impossible. And even if the trials reported this information, we need additional information to determine whether the results of the trials really apply to people with a specific comorbidity (or comorbidities) or risk profile. Heterogeneity of treatment effects may arise from differences in baseline risk of the primary outcome, risk of harm, competing risks, or relative risk reduction , , , . Frequently, researchers examine subgroup effects to determine whether treatment effects vary across groups, but such analyses should be consistent with criteria for appropriate subgroup analyses . While the trials occasionally examined subgroups based on comorbidity status, they were never examined according to established criteria . The trials also rarely considered heterogeneity of baseline risk and competing risks. The result is that it is rare that we can draw conclusions about the presence, or absence, of different treatment effects in people with comorbidity.
While our results may paint a grim picture of the current ability to draw evidence-based conclusions in evidence syntheses about people with comorbidity, they highlight specific steps for improving the way clinical trials, systematic reviews and resulting clinical practice guidelines might better inform treatments of patients with comorbidities. We could overcome some of the limitations of the current evidence base by improved reporting of the eligibility criteria of trials and the comorbidities of patients included in trials. Greater specificity to the descriptions of inclusion and exclusion criteria would help maximize replicability. Tables of baseline characteristics could include more detailed information about single comorbidities, how frequently they occur in combination and their treatments. And although it is often challenging to assess how treatment effects differ according the extent of comorbidity (sample size requirements may exceed feasibility), trials need to address questions of effect modification for common and important comorbidities for which there are prior hypotheses about potential effect modification. One potential solution could be to use observational studies to investigate effect modification by comorbidity . Although observational studies are more prone to confounding, the evidence base could still be improved overall. Defining multimorbidity clearly in such analyses of trials and observational studies will be essential . Another question is how to best convey the uncertainty arising from an incomplete evidence base to patients and health care providers. One solution would be to develop explicit guidance on how to rate the quality of evidence that is used for comorbid patients, which could then inform the strength of recommendations.
Limitations of our study included our focus on trials included in systematic reviews for four chronic diseases. Information about trials in additional chronic diseases may better guide treatment for people with comorbidity , . Nevertheless, we chose Cochrane systematic reviews as the basis of our included trials in order to address the important challenge of how evidence syntheses can better address the important challenge of developing evidence-based guidance for people with comorbidity. We chose this system for feasibility, and also to ensure replicable sampling of trials included in our study. While the Cochrane reviews are recent, some of the included studies are significantly older, and there may be differences over time in the extent comorbidities are considered and there were no exclusions in the Cochrane reviews for quality of evidence or risk of bias. We did not assess for changes in reporting over time. However, recent work on exclusion and inclusion criteria suggests that there is not variation over the last 15 years among a more selected group of studies from high-impact journals . Our approach of classifying the exclusions was based on categories of exclusion, and may have grouped exclusions with varied definitions. For example, across trials, there was not a standard definition or threshold of “renal insufficiency”, and so some trials may have excluded only more severe renal disease, whereas some may have had a more restrictive cutpoint. Often, these operational details of a definition are not reported by the trials, as shown in our results.
Comorbidities receive little attention in chronic disease trials, which is illustrated by exclusion of patients with comorbidities, poor reporting on the prevalence of comorbidities and description of how comorbidities are ascertained, and few investigations into how comorbidities influence the effects of treatments. With the increasing prevalence and public health importance of comorbid patients,  clinical trials should not only focus on an index disease but should report on comorbidities, and better assess the effect comorbidities have on treatment outcomes. In this way, systematic reviews based on these trials and resulting clinical practice guidelines would be more relevant for comorbid patients, who contribute substantially to the overall disease burden of populations.
The Corresponding Author(CB) has the right to grant on behalf of all authors and does grant on behalf of all authors, an exclusive license (or non exclusive for government employees) on a worldwide basis to PLOS One.
No patient data were collected. The data presented here are available on request from the authors (Dr. Cynthia Boyd email@example.com).
Conceived and designed the experiments: CMB DV MAP. Performed the experiments: CMB DV MAP. Analyzed the data: CMB DV MAP. Wrote the paper: CMB DV MAP. Had the idea for the study: CMB MAP. Designed the study: CMB MAP. Developed the data collection form and collected the data: CMB DV MAP. Interpreted the data: DV MAP. Drafted the report: CMB. Revised the report for final publication: CMB DV MAP. Guarantors of this study CMB MAP. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.
- 1. Boyd CM, Darer J, Boult C, Fried LP, Boult L, et al. (2005) Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance JAMA. 294(6): 716–24.
- 2. US Department of Health and Human Services: Multiple Chronic Conditions – A Strategic Framework: Optimum Health and Quality of Life for Individuals with Multiple Chronic Conditions.
- 3. Parekh AK, Barton MB (2010) The challenge of multiple comorbidity for the US health care system. JAMA 303(13): 1303–4.
- 4. Fortin M, Soubhi H, Hudon C, Bayliss EA, van den Akker M (2007) Multimorbidity's many challenges. BMJ 334(7602): 1016–7.
- 5. Smith SM, O'Kelly S, O'Dowd T (2010) GPs' and pharmacists' experiences of managing multimorbidity: a ‘Pandora's box’. Br J Gen Pract 60(576): 285–94.
- 6. Tinetti ME, Bogardus ST Jr, Agostini JV (2004) Potential pitfalls of disease-specific guidelines for patients with multiple conditions. N Engl J Med 351(27): 2870–4.
- 7. Lugtenberg M, Burgers JS, Clancy C, Westert GP, Schneider EC (2011) Current guidelines have limited applicability to patients with comorbid conditions: a systematic analysis of evidence-based guidelines. PLoS One 6(10): e25987.
- 8. Boyd CM, Fortin M (2011) Future of Multimorbidity Research: How Should Understanding of Multimorbidity Inform Health System Design? Public Health Reviews 32 (2): 451–474.
- 9. Bayliss EA, Ellis JL, Steiner JF (2007) Barriers to self-management and quality-of-life outcomes in seniors with multimorbidities. Ann Fam Med 5(5): 395–402.
- 10. Van Spall HG, Toren A, Kiss A, Fowler RA (2007) Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA 297(11): 1233–40.
- 11. Jadad AR, To MJ, Emara M, Jones J (2011) Consideration of multiple chronic diseases in randomized controlled trials. JAMA 306(24): 2670–2.
- 12. Zulman DM, Sussman JB, Chen X, Cigolle CT, Blaum CS, et al. (2011) Examining the evidence: a systematic review of the inclusion and analysis of older adults in randomized controlled trials. J Gen Intern Med 26(7): 783–90.
- 13. Kravitz RL, Duan N, Braslow J (2004) Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Q 82(4): 661–87.
- 14. Kent DM, Kitsios G (2009) Against pragmatism: on efficacy, effectiveness and the real world. Trials 10: 48.
- 15. Kent DM, Hayward RA (2007) Limitations of applying summary results of clinical trials to individual patients: the need for risk stratification. JAMA 298(10): 1209–12.
- 16. Varadhan R, Weiss CO, Segal JB, Wu AW, Scharfstein D, et al. (2010) Evaluating health outcomes in the presence of competing risks: a review of statistical methods and clinical applications. Med Care 48(6 Suppl). S96–105.
- 17. Owens DK, Lohr KN, Atkins D, Treadwell JR, Reston JT, et al. (2010) Grading the strength of a body of evidence when comparing medical interventions-Agency for Healthcare Research and Quality and the Effective Health Care Program. J Clin Epidemiol May 63(5): 513–23.
- 18. Boyd C, Leff B, Weiss C, Wolff J, Clark R, et al. (2010) Full Report: Clarifying Multimorbidity to Improve Targeting and Delivery of Clinical Services for Medicaid 19. Populations. Center for Health Care Strategies, Inc.
- 19. Boyd CM, Leff B, Wolff JL, Yu Q, Zhou J, et al. (2011) Informing clinical practice guideline development and implementation: prevalence of coexisting conditions among adults with coronary heart disease. J Am Geriatr Soc 59(5): 797–805.
- 20. Weiss CO, Boyd CM, Yu Q, Wolff JL, Leff B (2007) Patterns of prevalent major chronic disease among older adults in the United States. JAMA 298(10): 1160–2.
- 21. Vollenweider D, Boyd CM, Puhan MA (2011) High prevalence of potential biases threatens the interpretation of trials in patients with chronic disease. BMC Med 9: 73.
- 22. Stanton M (2006) The High Concentration of U.S. Health Care Expenditures. Research in Action, vol Issue 19. Rockville, MD: Agency for Healthcare Research and Quality.
- 23. Druss BG, Marcus SC, Olfson M, Pincus HA (2002) The most expensive medical conditions in America. Health Aff (Millwood) 21(4): 105–11.
- 24. Druss BG, Marcus SC, Olfson M, Tanielian T, Elinson L, et al. (2001) Comparing the national economic burden of five chronic conditions. Health Aff (Millwood) 20(6): 233–41.
- 25. Thomas DE, Elliott EJ, Naughton GA (2006) Exercise for type 2 diabetes mellitus. Cochrane database of systematic reviews (Online) 3(Journal Article). CD002968.
- 26. Saunders DH, Greig CA, Mead GE, Young A (2009) Physical fitness training for stroke patients. Cochrane database of systematic reviews (Online) (4)(4): CD003316.
- 27. Saenz A, Fernandez-Esteban I, Mataix A, Ausejo M, Roque M, et al. (2005) Metformin monotherapy for type 2 diabetes mellitus. Cochrane database of systematic reviews (Online) (3)(3): CD002966.
- 28. Orozco LJ, Buchleitner AM, Gimenez-Perez G, Roque I Figuls M, Richter B, et al. (2008) Exercise or exercise and diet for preventing type 2 diabetes mellitus. Cochrane database of systematic reviews (Online) (3)(3): CD003054.
- 29. Nannini LJ, Cates CJ, Lasserson TJ, Poole P (2007) Combined corticosteroid and long-acting beta-agonist in one inhaler versus long-acting beta-agonists for chronic obstructive pulmonary disease. Cochrane database of systematic reviews (Online) (4)(4): CD006829.
- 30. Manktelow BN, Potter JF (2009) Interventions in the management of serum lipids for preventing stroke recurrence. Cochrane database of systematic reviews (Online) (3)(3): CD002091.
- 31. Lacasse Y, Goldstein R, Lasserson TJ, Martin S (2006) Pulmonary rehabilitation for chronic obstructive pulmonary disease. Cochrane database of systematic reviews (Online) (4)(4): CD003793.
- 32. Faris R, Flather MD, Purcell H, Poole-Wilson PA, Coats A (2006) Diuretics for heart failure. Cochrane database of systematic reviews (Online) (1)(1): CD003838.
- 33. Davies EJ, Moxham T, Rees K, Singh S, Coats AJ, et al. (2010) Exercise based rehabilitation for heart failure. Cochrane database of systematic reviews (Online) (4)(4): CD003331.
- 34. Appleton S, Poole P, Smith B, Veale A, Lasserson TJ, et al. (2006) Long-acting beta2-agonists for poorly reversible chronic obstructive pulmonary disease. Cochrane database of systematic reviews (Online) 3(Journal Article). CD001104.
- 35. Algra A, De Schryver EL, van Gijn J, Kappelle LJ, Koudstaal PJ (2006) Oral anticoagulants versus antiplatelet therapy for preventing further vascular events after transient ischaemic attack or minor stroke of presumed arterial origin. Cochrane database of systematic reviews (Online) 3(Journal Article). CD001342.
- 36. Heiat A, Gross CP, Krumholz HM (2002) Representation of the elderly, women, and minorities in heart failure clinical trials. Arch Intern Med 162(15): 1682–8.
- 37. Weiss CO, Boyd CM, Wolff JL, Leff B (2011) Prevalence of Diabetes Treatment Effect Modifiers: The External Validity of Trials to Older Adults. Aging Clinical and Experimental Research. In Press.
- 38. Schnell KM, Weiss CO, Lee T, Krishnan JA, Leff B, et al. (2012) The prevalence of clinically-relevant comorbid conditions in patients with chronic obstructive pulmonary disease: data from the National Health and Nutrition Examination Survey 1999–2008. BMC Pulm Med 13 12(1): 26.
- 39. Sun X, Briel M, Walter SD, Guyatt GH (2010) Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ (Clinical research ed.) 340(Journal Article). c117.
- 40. Oxman AD, Guyatt GH (1992) A consumer's guide to subgroup analyses. Annals of Internal Medicine 116(1): 78–84.
- 41. Walter LC, Covinsky KE (2001) Cancer screening in elderly patients: a framework for individualized decision making. JAMA 285(21): 2750–6.
- 42. Welch HG, Albertsen PC, Nease RF, Bubolz TA, Wasson JH (1996 (Estimating treatment benefits for the elderly: the effect of competing risks. Ann Intern Med 124(6): 577–84.
- 43. Kent DM, Alsheikh-Ali A, Hayward RA (2008) Competing risk and heterogeneity of treatment effect in clinical trials. Trials 9: 30.
- 44. Koller MT, Raatz H, Steyerberg EW, Wolbers M (2012) Competing risks and the clinical community: irrelevance or ignorance? Stat Med 31(11–12): 1089–97.
- 45. Blumle A, Meerpohl JJ, Rucker G, Antes G, Schumacher M, et al. (2011) Reporting of eligibility criteria of randomised trials: cohort study comparing trial protocols with subsequent articles. BMJ 342: d1828.
- 46. He J, Ogden LG, Bazzano LA, Vupputuri S, Loria C, et al. (2001) Risk factors for congestive heart failure in US men and women: NHANES I epidemiologic follow-up study. Arch Intern Med 161(7): 996–1002.
- 47. Kent D, Hayward R (2008) Subgroup analyses in clinical trials. N Engl J Med 358(11): 1199; author reply 1199–200.
- 48. Kent D, Hayward R (2007) When averages hide individual differences in clinical trials. American Scientist 95(1): 60–68.
- 49. Greenfield S, Billimek J, Pellegrini F, Franciosi M, De Berardis G, et al. (2009) Comorbidity affects the relationship between glycemic control and cardiovascular outcomes in diabetes: a cohort study. Ann Intern Med 151(12): 854–60.
- 50. Harrison M, Reeves D, Harkness E, Valderals J, Kennedy A, et al. (2012) A secondary analysis of the moderating effects of depression and multimorbidity on the effectiveness of a chronic disease self-management programme. Patient Educ Couns 87(1): 67–73.
- 51. Ring A, Sestak I, Baum M, Howell A, Buzdar A, et al. (2011) Influence of comorbidities and age on risk of death without recurrence: a retrospective analysis of the Arimidex, Tamoxifen Alone or in Combination trial. J Clin Oncol 29(32): 4266–72.
- 52. Hurria A (2011) Embracing the complexity of comorbidity. J Clin Oncol 29(32): 4217–8.
- 53. Organisation of Economic Cooperation and Development. 50th Anniversary Conference. Available: http://www.oecd.org/document/17/0,3746, en_2649_37407 48127569 1_1_1_37407,00.html. Accessed 2012 Feb 9..