Citation: Ellenberg SS (2012) Protecting Clinical Trial Participants and Protecting Data Integrity: Are We Meeting the Challenges? PLoS Med 9(6): e1001234. doi:10.1371/journal.pmed.1001234
Published: June 12, 2012
Copyright: © 2012 Susan S. Ellenberg. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: No specific funding was received for writing this article.
Competing interests: SE currently serves on 1 industry DMC (BMS) and 2 DSMBs for NIH-funded studies. SE also gets 5% of her salary via a contract to Penn from Pfizer to support statistical leadership on a study funded by Pfizer but designed and carried out by investigators independently of the company. Other companies from which SE has received fees for consulting or lecturing in the past 5 years: Ariad Pharmaceuticals; Shire Pharmaceuticals; Amgen; MGI; Novovax; Lux Bio; GPC Biotech; Astra-Zeneca; Merck; Genentech; J&J; Millenium; Onyx; Tengion; Tarsa; Allergan; and Curagen. SE has also done consulting for a legal firm handling pharmaceutical-related litigation. Most of these except for the data monitoring committees were one-time engagements, either to lecture at an internal scientific meeting or to participate in a mock advisory committee meeting. I do not believe any of these represent competing interests but declare them for completeness.
Abbreviations: DMC, data monitoring committee; FDA, Food and Drug Administration; IRB, institutional review board; NHLBI, National Heart and Blood Institute; NIH, National Institutes of Health
Provenance: Commissioned; externally peer reviewed.
- Although there is substantial consensus regarding the need for interim monitoring of certain types of trials, there is controversy about specific aspects of data monitoring.
- Approaches to ensuring independence of those who perform the interim monitoring and confidentiality of interim data vary substantially by type of trial and trial funder.
- The “independent statistician” model, involving a separate statistician to analyze interim data and report to the data monitoring committee (DMC), remains controversial but provides important protections of data integrity.
- Early stopping guidelines should be clearly understood and accepted by all parties, and only deviated from if there are unexpected findings that confound the overall benefit-risk assessment at interim analysis.
- Liability of DMC members is an important concern that has not been dealt with adequately by either commercial or government trial sponsors.
A long-standing tenet of clinical trial conduct is that the accumulating data must be monitored, if not continuously then at regular intervals during the trial. Such regular review helps to ensure that risks to participants are not greater than anticipated and that the study is being conducted appropriately. Early randomized trials were typically monitored by the investigator(s), the research funder (a government agency or a pharmaceutical company), or by a steering committee appointed by the funder , with no well-accepted quantitative criteria for decision-making and often with the accumulating data widely known.
The practices of data monitoring committees (DMCs; also known as Data and Safety Monitoring Boards [DSMBs]) have evolved substantially over the last 40 years. Statistical methods to guide early termination decisions have been developed –; the principle of confidentiality of interim data has become widely accepted; and the actual conduct of DMC meetings has become more standardized. Many publications, an international US National Institutes of Health (NIH) symposium , and at least three books – have addressed the philosophical and operational issues faced by DMCs; in the United States, the NIH and the US Food and Drug Administration (FDA) have established policies regarding DMC establishment and operation ,, as have international organizations ,. A large study of DMC practices commissioned by the United Kingdom's Health Technology Assessment group has provided substantial detail regarding current DMC practices –.
Despite the increased attention to DMC function, some issues remain controversial. Two of particular importance are 1) the extent to which DMC members and the statistician analyzing interim data and reporting to the DMC should be independent of the trial sponsor and investigators; and 2) the criteria that should be used to guide early termination decisions. A third issue, of increasing concern to scientists serving on DMCs, is that of liability of DMC members.
Independence of the DMC and the Reporting Statistician
When selecting scientists to evaluate research findings, there is inevitable tension in seeking the most knowledgeable experts while minimizing real or perceivable conflicts of interest. The more knowledgeable and experienced a scientist is in a given area of research, the greater the likelihood that s/he will have some connection with any study in that area that could be perceived as a conflict of interest. (Similar tensions apply to selection of members of scientific and regulatory advisory committees, practice guideline committees, and reviewers for medical journals.) Currently, study investigators and employees of the manufacturer, who have clear conflicts of interest, are typically excluded not only from serving on the study DMC but also from attending sessions at which interim results by treatment group are presented and discussed. Other conflicts are more subtle. Should someone holding any amount of stock in a company be permitted to serve on a DMC for a study evaluating that company's products? What about someone serving as investigator in another study of the product, or any product, from the sponsoring company? Or an investigator serving on the company's speaker's bureau? Such individuals are often, but not always, excluded from serving on a DMC because such individuals might favor, or be perceived to favor, an outcome aligned with their personal interests. (When the clinical trials of rofecoxib [Vioxx], a COX-2 inhibitor that was ultimately withdrawn from the market due to adverse cardiovascular effects, were closely examined, it was discovered that the chair of the DMC of one important trial had nontrivial financial relationships with the product's manufacturer . This revelation surely added to the belief of many, fairly or unfairly, that the chair, perhaps subconsciously, might not have evaluated the safety data as rigorously as he otherwise would have.)
Of course, the fact that an investigator holds stock in a company, or that s/he serves as a scientific advisor to the company for other purposes, does not necessarily imply that that investigator will be influenced by that connection to give more weight to protecting the company than study participants, but this can never be proven one way or the other. Concerns about unwanted influence on DMC members have increased to the point that some have advocated removing the selection of DMC members from the hands of companies entirely, placing that authority in an independent public body . This solution may be overly extreme—it is difficult to imagine a single organization being able to effectively constitute appropriately expert DMCs for all trials run by the pharmaceutical industry—but more attention is needed to avoid conflicts of interest. Two possible approaches would be to intensify institutional review board (IRB) or FDA oversight of member selection. It seems unrealistic to expect multiple IRBs to review a trial's DMC membership in sufficient detail to identify conflicts of interest, but if central IRBs became standard in multisite trials, IRB review of the DMC membership, including financial disclosures, with the authority to reject members who appeared too close to the trial sponsor or had other significant conflicts, could be one solution. Alternatively, the FDA could review a company's proposed DMC membership, in the same way it reviews potential members of FDA advisory committees, and disapprove any member with an apparent conflict of interest. Implementation of either of these options would be challenging. Currently, while many have advocated central IRBs for multisite trials we are far from consensus as to whether this would be desirable ,. It could be more straightforward for the FDA to take on this oversight, but it would add substantially to the resources required for regulatory review.
DMCs for trials sponsored by noncommercial funders do not necessarily operate in the same way as DMCs for industry trials. For example, the widely accepted prohibition of sponsor representatives participating in DMC meetings with access to the unblinded interim data has not been applied by most US government sponsors. Wittes et al. , in describing the monitoring of the Women's Health Initiative (WHI) clinical trials, noted the unwillingness of the US National Heart and Blood Institute (NHLBI), whose representatives had access to the interim data and participated fully in the monitoring sessions, to permit the committee to discuss recommendations in an “executive session” without NHLBI representatives. The NHLBI also pressured the committee at times in regard to the direction its recommendations should take. A 2009 paper describing NHLBI monitoring processes stated that when the NHLBI is unhappy with recommendations from a monitoring committee, it works with the committee to develop revised recommendations that NHLBI will accept . If the study sponsor is permitted to pressure its advisors regarding the recommendations they should make, the whole concept of an independent expert committee is undermined. This is particularly true when the outside experts depend on the NIH for their research funding and may therefore be reluctant to persist in advocating recommendations that the participating program staff does not like. Concern about such issues has led the US National Institute of Allergy and Infectious Diseases (NIAID) to exclude staff members with any scientific involvement in a research study from the interim monitoring process . NHLBI has also recently modified its policies to permit executive sessions without the presence of any NHLBI staff, and appears to be more restrictive of NHLBI staff participation in sessions at which confidential interim data are presented .
Other conflicts relate to the statistician performing the interim analysis and reporting to the DMC. Traditionally, this role has been played by the primary study statistician. Difficulties can arise, however, when mid-course changes to the study are contemplated. For example, a study might be focused on improvement in heart failure, but if another study of the same or related drug reported a highly positive survival advantage with little impact on heart failure, the sponsors of the first study might want to consider changing their primary endpoint to survival. If the study statistician (who participates in the decision to change the endpoint) knows that the interim results strongly favor survival, people might question whether, had the results not favored survival, the statistician would have found subtle ways of discouraging the change. Such influences would compromise the interpretability of the final data.
Such situations do not occur frequently, but when they do it is important to ensure that needed changes to trial design can be made without undermining trial integrity. One solution, which has been adopted for most industry-sponsored trials and some government-sponsored trials, is to involve two statisticians in the trial, one to work with the study team on design issues and trial management, and the other to perform interim analyses and report to the DMC. This is sometimes referred to as the “independent statistician” model , and was recommended by the FDA in its 2006 guidance document on clinical trial data monitoring committees . Although this model has worked well for many trials ,, a potential disadvantage is that the statistician doing the interim analysis may not be as knowledgeable about the study as the statistician involved in the design and management, potentially hampering effective communication with DMC members . An alternative approach is to have the study statistician serve as the analyst for interim data, as has traditionally been done, but if changes to the design are contemplated assign a different statistician, who has not seen the interim data, to implement the change. This approach may be more appealing to many as the knowledge base of the primary study statistician remains available to the DMC, and the second statistician is not required as long as no major interim design changes are considered. As additional experience is gained with these models a consensus may emerge regarding the optimal approach, but it is clear that decision-making about mid-course changes in trial design should be done without knowledge of interim results.
Early Termination of Trials
A long-accepted ethical principle for clinical trials is that trials should not continue if interim data provide definitive evidence of the superiority of one of the treatments. A trial whose results showed 200 deaths on one arm and five deaths on the other, for example, would be severely criticized on the grounds that a survival difference would have been evident far earlier, and many of the 200 deaths on the inferior arm should have been averted. On the other hand, in the early days of clinical trials when investigators routinely reviewed interim data, it was not uncommon for trials to stop as soon as interim findings became nominally statistically significant , increasing the likelihood of false positive reports and limiting the credibility of the results. Guidelines were developed during the 1970s and 1980s to permit early termination with valid claims of statistical significance –.
A difficulty, however, is that what constitutes “definitive evidence” is inherently subjective. This issue has been debated for decades , without resolution. Those who are more negative about early termination fear that smaller trials are less reliable and will not be widely persuasive, and therefore will not lead to changes in medical practice. A recent variation on this argument is that early termination of trials should be avoided because it leads to artificially elevated estimates of treatment effect; trials that stop early, of course, show higher estimates of effect than trials of the same regimens that do not stop early ,. Others have countered that there is nothing nefarious about the obvious fact that trials that stop early show higher treatment effects, that any upward bias is small and methods to correct the bias are readily available, and that the ethics of continuing trials of proven life-saving treatments to gain at best minor improvements in estimation of effect are questionable –.
The implication of this debate for a DMC is that the DMC, the study sponsors, and the study investigators must all understand and be comfortable with the criteria for early termination proposed as the basis for trial monitoring, with the understanding that unanticipated issues may always arise that would lead a DMC to ignore the pre-specified criteria. For example, the efficacy boundary could be crossed, but an unanticipated safety concern could have arisen so that further study is needed to determine whether benefit outweighs the risk. Although it is well accepted that a DMC will regard stopping criteria as guidelines rather than strict rules, a DMC should be prepared to apply the agreed-upon criteria unless there are reasons to question the benefit-to-risk assessment. It has been suggested that a DMC might appropriately ignore the pre-stated stopping criteria if its members were simply skeptical that the effect was as large as that being observed ; this would suggest, however, that the stopping criteria were inappropriate and the DMC should have asked for more stringent criteria initially.
Liability of DMC Members
The issue of liability coverage for DMC members has arisen as a concern only in recent years, as litigation has increasingly permeated medical research. It is difficult to cite cases; I have colleagues who have been consulted in such cases but they have all been settled with confidentiality agreements. Nevertheless, concerns on the part of DMC members have increased to the point that several NIH institutes are now providing coverage through contractors to individuals serving on their DMCs. Pharmaceutical companies are often (but not universally) willing to indemnify members of their DMCs; proposed language to be written into DMC contracts for industry studies was provided by DeMets et al. . Such indemnification is probably not the optimal solution, as a DMC member's defense handled by legal staff of a pharmaceutical company could be compromised by the staff's primary responsibility to defend the company . A better approach would be the availability of insurance policies that DMC members could purchase on an individual basis, with the cost of such a policy factoring into the fee a member would negotiate with a company. For an insurer to offer such policies, however, would require information about the risks it would face, and because of the limited information available the development of such policies by insurers would be challenging.
DMCs have become expected components of many clinical trials, and provide an important oversight function. With increased experience, debates about best practices, as well as new issues, have emerged. New approaches are needed to ensure that DMC members do not have unacceptable conflicts of interest. Conflict of interest concerns relate also to the access to interim results. Such concerns have led to exclusion of industry representatives from involvement with interim monitoring, but the potential conflicts of noncommercial sponsors have not been adequately recognized. The “independent statistician” issue, on the other hand, seems to be resolving in favor of a model in which those making day-to-day decisions about trial design and conduct are protected from knowledge of interim results. The key to making such models successful is the presence of statisticians knowledgeable about the trial in both the trial management and the DMC reporting roles. Other issues, such as the appropriate stringency of early termination boundaries, appear less amenable to consensus, at least currently, because the characteristics of particular trials, and the philosophies of trial organizers, are too varied. The best one can hope for is that for any particular trial, the trial sponsor, investigators, DMCs, and IRBs are all in agreement about the monitoring approach taken for that trial. Finally, concerns about potential exposure of DMC members to litigation are relatively recent but need to be taken seriously by DMC members and trial sponsors. The decision of some NIH institutes to establish protection for DMC members serving on institute DMCs is a welcome development.
Wrote the first draft of the manuscript: SE. Contributed to the writing of the manuscript: SE. ICMJE criteria for authorship read and met: SE. Agree with manuscript results and conclusions: SE.
- 1. Heart Special Project Committee (1988) Organization, review and administration of cooperative studies: a report from the Heart Special Projects Committee to the National Advisory Heart Council, May 1967. Control Clin Trials 9: 137–148.
- 2. Pocock SJ (1977) Group sequential methods in the design and analysis of clinical trials. Biometrika 64: 191–199.
- 3. O'Brien PC, Fleming TR (1979) A multiple testing procedure for clinical trials. Biometrics 35: 549–556.
- 4. Lan KKG, DeMets DL (1983) Discrete sequential boundaries for clinical trials. Biometrika 70: 659–663.
- 5. Whitehead J (1983) The design and analysis of sequential clinical trials. New York: Halsted Press.
- 6. Ellenberg SS, Geller N, Simon R, Yusuf S, editors. (1993) Proceedings of “Practical Issues in Data Monitoring of Clinical Trials.” Stat Med 12: 415–616.
- 7. Ellenberg SS, Fleming TR, DeMets DL (2002) Data monitoring committees in clinical trials: a practical perspective. Chichester: John Wiley & Sons, Ltd.
- 8. DeMets DL, Furberg CD, Friedman LM (2006) Data monitoring in clinical trials a case studies approach. New York: Springer.
- 9. Herson J (2009) Data and safety monitoring committees in clinical trials. Boca Raton: Chapman & Hall.
- 10. National Institutes of Health (1998) NIH policy for data and safety monitoring. Available: http://grants.nih.gov/grants/guide/notice-files/not98-084.html. Accessed 15 October 2011.
- 11. Food and Drug Administration (2006) Guidance for clinical trial sponsors: establishment and operation of clinical trial data monitoring committees. Available: http://www.fda.gov/downloads/RegulatoryInformation/Guidances/ucm127073.pdf. Accessed 15 October 2011.
- 12. European Medicines Agency Committee for Medicinal Products for Human Use (2005) Guideline on data monitoring committees. Available: http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500003635.pdf. Accessed 15 October 2011.
- 13. World Health Organization (2005) Operational guidelines for the establishment and functioning of data and safety monitoring boards. Available: http://whqlibdoc.who.int/hq/2005/TDR_GEN_Guidelines_05.1_eng.pdf. Accessed 15 October 2011.
- 14. Grant AM, Sydes M, McLeer S, Clemens F, Altman DG, et al. (2005) Issues in data monitoring and interim analysis of trials (the DAMOCLES study). Health Technology Assess 9: 1–238.
- 15. Sydes MR, Spiegelhalter DJ, Altman DG, Babiker AB, Parmar MK, et al. (2004) Systematic qualitative review of the literature on data monitoring committees for randomized controlled trials. Clin Trials 1: 60–79.
- 16. Sydes MR, Altman DG, Babiker AB, Parmar AK, Spiegelhalter DJ, et al. (2004) Reported use of data monitoring committees in the main published reports of randomized controlled trials. Clin Trials 1: 48–59.
- 17. Presley H.Vioxx and the Merck team effort. The Kenan Institute for Ethics at Duke University. Available: http://www.duke.edu/web/kenanethics/CaseStudies/Vioxx.pdf. Accessed 15 October 2011.
- 18. Drazen JM, Wood AJJ (2010) Don't mess with the DSMB. N Eng J Med 363: 477–478.
- 19. Menikoff J (2010) The paradoxical problem with multiple-IRB review. N Eng J Med 363: 1591–1593.
- 20. Klitzman R (2011) How local IRBs view central IRBs in the US. BMC Medical Ethics 12: 13. http://www.biomedcentral.com/1472-6939/12/13. Accessed 20 April 2012.
- 21. Wittes J, Barrett-Conner E, Braunwald E, Chesney M, Cohen HJ, et al. (2007) Monitoring the randomized trials of the Women's Health Initiative: the experience of the Data and Safety Monitoring Board. Clin Trials 4: 218–234.
- 22. Mondero TH (2009) Data safety monitoring boards: a word from a sponsor (NHLBI). Transfusion 49: 1537–1539.
- 23. Dixon DO, Weiss S, Cahill K, Fox L, Love J, et al. (2011) Data and safety monitoring policy for National Institute of Allergy and Infectious Diseases clinical trials. Clin Trials 8: 727–735.
- 24. National Heart, Lung, and Blood Institute (October 2011) NHLBI policy for data and safety monitoring of extramural clinical studies. Available: http://www.nhlbi.nih.gov/funding/policies/dsmpolicy.htm. Accessed 20 April 2012.
- 25. DeMets DL, Fleming TR (2004) The independent statistician for data monitoring committees. Stat Med 23: 1513–1517.
- 26. Siegel JP, O'Neill RT, Temple R, Campbell G, Foulkes MA (2004) Independence of the statistician who analyses unblinded data. Stat Med 23: 1527–1529.
- 27. Pocock SJ (2004) A major trial needs three statisticians: why, how and who? Stat Med 23: 1535–1539.
- 28. Ellenberg SS, George SL (2004) Should statisticians reporting to a data monitoring committee be independent of the trial sponsor and leadership? Stat Med 23: 1503–1505.
- 29. Wittes J (2004) Playing safe and preserving integrity: making the FDA model work. Stat Med 23: 1523–1525.
- 30. McPherson K (1974) Statistics: the problem of examining accumulating data more than once. N Engl J Med 290: 501–502.
- 31. Pocock SJ (1992) When to stop a clinical trial. BMJ 305: 235–240.
- 32. Discussion in Ellenberg S, Geller N, Simon R, Yusuf S, editors. Proceedings of “Practical Issues in Data Monitoring of Clinical Trials.” Stat Med 12: 521–525.
- 33. Bassler D, Montori V, Briel M, Glasziou P, Guyatt G (2008) Early stopping of randomized clinical trials for overt efficacy is problematic. J Clin Epid 61: 241–246.
- 34. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, et al. (2010) Stopping randomized trials early for benefit and estimation of treatment effects. JAMA 303: 1180–1187.
- 35. Berry SM, Carlin BP, Connor J (2010) Bias and trials stopped early for benefit (letter). JAMA 304: 156.
- 36. Goodman S, Berry D, Wittes J (2010) Bias and trials stopped early for benefit (letter). JAMA 304: 157.
- 37. Korn EL, Friedlin B, Mooney M (2010) Bias and trials stopped early for benefit (letter). JAMA 304: 157–158.
- 38. Ellenberg SS, DeMets DL, Fleming TR (2010) Bias and trials stopped early for benefit (letter). JAMA 304: 158.
- 39. Wheatley K, Clayton D (2003) Be skeptical about unexpected large apparent treatment effects: the case of an MRC AML 12 randomization. Control Clin Trials 24: 66–70.
- 40. DeMets DL, Fleming TR, Rockhold F, Massie B, Merchant T, et al. (2004) Liability issues for data monitoring committee members. Clin Trials 1: 525–531.
- 41. Tereskerz PM (2010) Data safety monitoring boards: legal and ethical considerations for research accountability. Account Res 17: 30–50.