Why Most Published Research Findings Are False: Problems in the Analysis

The article published in PLoS Medicine by Ioannidis [1] makes the dramatic claim in the title that “most published research claims are false,” and has received extensive attention as a result. The article does provide a useful reminder that the probability of hypotheses depends on much more than just the p-value, a point that has been made in the medical literature for at least four decades, and in the statistical literature for decades previous. This topic has renewed importance with the advent of the massive multiple testing often seen in genomics studies. 
 
Unfortunately, while we agree that there are more false claims than many would suspect—based both on poor study design, misinterpretation of p-values, and perhaps analytic manipulation—the mathematical argument in the PLoS Medicine paper underlying the “proof” of the title's claim has a degree of circularity. As we show in detail in a separately published paper [2], Dr. Ioannidis utilizes a mathematical model that severely diminishes the evidential value of studies—even meta-analyses—such that none can produce more than modest evidence against the null hypothesis, and most are far weaker. This is why, in the offered “proof,” the only study types that achieve a posterior probability of 50% or more (large RCTs [randomized controlled trials] and meta-analysis of RCTs) are those to which a prior probability of 50% or more are assigned. So the model employed cannot be considered a proof that most published claims are untrue, but is rather a claim that no study or combination of studies can ever provide convincing evidence. 
 
 
The two assumptions that produce the above effect are: 
 
 
Calculating the evidential effect only of verdicts of “significance,” i.e., p ≤ 0.05, instead of the actual p-value observed in a study, e.g., p = 0.001. 
 
 
Introducing a new “bias” term into the Bayesian calculations, which even at a described “minimal” level (of 10%) has the effect of very dramatically diminishing a study's evidential impact. 
 
 
 
 
 
 
In addition to the above problems, the paper claims to have proven something it describes as paradoxical; that the “hotter” an area is (i.e., the more studies published), the more likely studies in that area are to make false claims. We have shown this claim to be erroneous [2]. The mathematical proof offered for this in the PLoS Medicine paper shows merely that the more studies published on any subject, the higher the absolute number of false positive (and false negative) studies. It does not show what the papers' graphs and text claim, viz, that the number of false claims will be a higher proportion of the total number of studies published (i.e., that the positive predictive value of each study decreases with increasing number of studies). 
 
The paper offers useful guidance in a number of areas, calling attention to the importance of avoiding all forms of bias, of obtaining more empirical research on the prevalence of various forms of bias, and on the determinants of prior odds of hypotheses. But the claims that the model employed in this paper constitutes a “proof” that most published medical research claims are false, and that research in “hot” areas is most likely to be false, are unfounded.

The authors should be commended for a thoughtful and stimulating article [1].However, we wish to clarify the historical record about the use of the term XDR-TB.The concept of XDR-TB as a distinct nosological entity was fi rst developed at the Centers for Disease Control and Prevention (CDC) in March 2005 and introduced into public use in October 2005 at the 36th World Conference on Lung Health in Paris [2,3].At that meeting, data on second-line drug resistance from a global survey of supranational TB reference laboratories conducted by CDC and the World Health Organization, as well as treatment outcomes of XDR-TB patients in Latvia, were fi rst presented.Shortly thereafter, the cluster of TB deaths with resistance to second-line drugs in HIV-infected persons in KwaZulu-Natal was presented at the 13th Conference on Retroviruses and Opportunistic Infections in Denver in February 2006 [4].The original defi nition for XDR-TB published in the Morbidity and Mortality Weekly Report in March 2006 [5] that they have used, however, was revised in October 2006 at an emergency meeting of the Global XDR-TB Task Force.The revised defi nition was published on November 3, 2006 in an MMWR notice to readers [6].Currently, XDR-TB is defi ned as the occurrence of TB in persons whose Mycobacterium tuberculosis isolates are resistant to isoniazid and rifampin plus any fl uoroquinolone and at least one of the three injectable second-line drugs (amikacin, kanamycin, capreomycin).The defi nition was revised because drug susceptibility testing to these drugs produces reliable and reproducible results, and is more accessible in resource-limited settings.In addition, patients meeting the revised defi nition have signifi cantly poorer treatment outcomes.The new defi nition is important for those intending to conduct surveillance for XDR-TB in their setting.
Timothy H. Holtz (tholtz@cdc.gov)Centers for Disease Control and Prevention Atlanta, Georgia, United States of America.[1] highlight safeguards against the spread of XDR-TB and suggest "involuntary detention" as a key infection control measure.Yet several important elements of the current response to XDR-TB may make the application of enforced confi nement ineffective and inappropriate as part of the initial response to this problem.

Singh and colleagues
One irony of this discussion is that patients diagnosed with drug-resistant TB in KwaZulu-Natal are being turned away from the referral hospitals where second-line therapy takes place.There is a waiting list of more than 70 patients for admission to King George V Hospital, where the majority of MDR-TB therapy is provided.Rather than keeping patients "in"-the debate posed in this article-the reality is that health services are unable to accommodate the burden of MDR-TB patients seeking care.
The authors cite United States policies during MDR-TB outbreaks as evidence of the success of detention but fail to note that US confi nement measures were rarely invoked.The New York City Tuberculosis Working Group concluded: "It is unethical, illegal, and bad public health policy to detain 'noncompliant' persons before making concerted efforts to address the numerous systemic defi ciencies that make adherence to treatment virtually impossible" [2].Thus, patients were fi rst offered directly observed therapy as outpatients.Among the few patients cited as non-adherent, less than half were detained.Monetary incentives and transportation vouchers were provided for outpatients, as well as housing to the homeless [3].In contrast, many MDR-TB patients in KwaZulu-Natal must travel several hours monthly to receive treatment.
It is estimated that the South African government will spend 15 billion rand (~US$1.9 billion) for the upcoming World Cup, much of it for building stadiums [4].Yet, while the largest outbreak of XDR-TB ever recorded is unfolding, little appropriate investment has been made.Purchasing trailer homes as isolation facilities, providing particulate respirator masks in all hospitals, and instituting other basic infection control procedures is immediately necessary.
Framing the debate about forced confi nement in terms of individual liberty versus threat to society neglects the true injustice taking place.
While Singh and colleagues discuss the importance of "reciprocity," they fail to mention the most important reciprocity obligation of those instituting confi nement: providing the proper standard of medical care to detained patients.At present, many XDR-TB patients are provided therapy that includes only two active agents-a recipe for amplifi cation of resistance.While XDR-TB patients elsewhere have been successfully treated with other regimens [5], the majority of South African patients have yet to access many second-line drugs, including capreomycin, moxifl oxacin, para-aminosalicylic acid, or adjunctive thoracic surgery.Without these, they are left to die without a fi ghting chance, two years after this outbreak was fi rst reported.The intent of detention in the US was to provide short inpatient stays and curative therapy.The median period of detention was three weeks, and only 2% of patients died from tuberculosis [3].In South Africa, XDR-TB is nearly universally fatal under current treatments, and detention would presumably be sustained until death.Our willingness to respond to the realities of patient needs, rather than to abstract theories, will determine the success of the response to XDR-TB.

Funding:
The authors received no specifi c funding for this article.

Competing Interests:
The authors have declared that no competing interests exist.

XDR-TB in South Africa: Detention Is Not the Priority Eric Goemaere, Nathan Ford, Daniel Berman, Cheryl McDermid, Rachel Cohen
We agree that there is "no time for denial or complacency" when it comes to the spread of MDR-and XDR-TB in South Africa.Unfortunately, the attention the recent PLoS Medicine article [1] generated in South Africa and internationally has overwhelmingly focused on detention of patients.Headlines such as "South Africa urged to isolate 'killer' TB patients" [2] place the blame on patients and divert attention from more urgent priorities.
The TB epidemic in South Africa, as across sub-Saharan Africa, is largely linked to HIV.In Khayelitsha Township near Cape Town, new cases had risen to around 2,000 per 100,000 in 2006, fuelled by the high prevalence of HIV.From our experience in South Africa, a number of challenges must be addressed locally and nationally to curb MDR-TB.Detention does not come high on this list.
Effective MDR-TB management requires improvements in general TB control, but this alone will not remove the need to respond to MDR-TB.The Western Cape has the best TB outcomes in South Africa, thanks to enormous investments in TB control, but despite this MDR-and XDR-TB cases are increasingly being reported.There is an urgent priority for infection control, taking into account the context of limited resources at the primary care level and high HIV prevalence.Data from mid-2006 show that 67% of TB patients in Khayelitsha are HIV positive; in Médecins Sans Frontières' programme in Lesotho the fi gure rises to 92%.Patient triage is one aspect, but the reality is that undiagnosed MDR-and XDR-TB patients with HIV are sitting in overcrowded waiting rooms next to other immunocompromised patients.Personal protection for health staff, starting with basic training on infection control, needs to be improved.Structural improvements to the clinics need to be based on feasible, low-tech solutions-air extractors and windows will be more practical than UV lights and negative pressure rooms [3].
Access to points of care needs to increase.The Western Cape has reported over 800 cases of MDR-TB in the last two years and this is certainly an underestimation.Greater diagnostic capacity and more rapid diagnosis is needed, and diagnosis must be met with better access to treatment.Treating MDR-TB currently relies on hospitalization of patients, but current needs are far greater than hospital capacity-patients can wait up to four months for a hospital bed.The traditional model of leaving MDR-TB management to specialists has incapacitated health-care staff at the primary care level who receive little or no training on how to manage MDR-TB.In other settings in southern Africa the situation is even worse.In Lesotho there is practically no access to reliable culture or drug-sensitivity testing.Given the scarcity of human resources and the overwhelming number of coinfected patients, treatment needs to be delivered in as decentralized a manner as possible.
In settings where clinics are saturated and patient numbers are rising, it is not realistic to rely on a strategy of simply reinforcing directly observed treatment and incarcerating defaulters to respond to MDR-TB.We need to apply the lessons learnt from providing HIV care in resource-poor settings, including decentralization of services to the primary care level, reinforcing adherence through treatment literacy and a patient-centred approach, and community-based support.The reality, though, is that an integrated approach to HIV and TB is far away: around a third of MDR-TB patients in Khayelitsha do not even know their HIV status.
Drug-resistant TB is not a new problem.What is new is the willingness to detect and treat it.The lack of willingness to do so until recently has left us with old drugs and diagnostics that make treating drug-resistant TB at best highly complex and resource intensive, and at worst impossible.Programme-level improvements have to be met with a dramatic increase in efforts to develop new drugs and diagnostics [4].

Funding:
The authors received no specifi c funding for this article.

Competing Interests:
The authors have declared that no competing interests exist.

Adherence to TB Treatment in Ethiopia: Why Do Patients Default? Hundie Tesfaye
Tuberculosis as a disease has been present in humans since antiquity, with the earliest unambiguous detection of Mycobacterium tuberculosis in the remains of bison dated 17,000 years before the present [1].About 90% of those infected with TB have asymptomatic (latent) infection.One in ten latent infections may progress to active disease which, if left untreated, kills more than half of its victims [2].In 2004, 14.6 million people had active TB and there were 8.9 million new cases and 1.7 million deaths [3], mostly in developing countries, including Ethiopia.
Shargie and Lindtjørn explain the physical lack of access to the treatment centre as the main cause of failure to adherence to therapy in 20% of patients [4].Only 52 patients indicated they lived over 10 km from the treatment centre.It is not clear whether all the 52 are among those who failed to complete the doses.It was of interest whether all 64 patients from the urban area (near to the treatment centre) completed the treatment.Nevertheless, it might be worthwhile to compare the rate of completion of treatment of the 64 urban dwellers with those 52 individuals to support the idea that physical access to the treatment centre is the main factor for treatment adherence.
Almost half of the patients enrolled in the study earned 0-99 Ethiopian birr (approximately 0-10 US dollars) per month.However, it is not clear from the report whether all 74 (91%) who failed to complete the treatment belonged to those with low monthly income.Furthermore, the fate of those earning more than 200 birr (approximately US$20) was not clear in terms of treatment completion.The authors reported that income had no infl uence on the outcomes in terms of treatment completion, despite their inclusion of cost of transport among the factors which led to incompletion of therapy.
Unfortunately the study did not evaluate the data of family numbers and its infl uence on treatment adherence in relation to family income.Larger family size with inappropriate income may contribute to malnutrition, leading to drug intolerance and general malaise and fi nally to loss of motivation to continue the treatment course.
The authors stated there was no infl uence of HIV status on the defaulting of treatment, despite the known fact that more than half of the cohort studied did not volunteer for the HIV test.
Drug-resistant strains of TB have emerged and are spreading dangerously.In 2000-2004, 20% of cases were resistant to standard treatments, and 2% were also resistant to second-line drugs [5].In spite of only one treatment failure here, extensively drug-resistant TB may be a possible challenge in Ethiopia.Whether Ethiopia succeeds in the Stop TB Partnership's Global Plan to Stop Tuberculosis, which aims to save 14 million lives between 2006 and 2015 (see http:⁄⁄www.stoptb.org/globalplan),depends on the effectiveness of the national program, infrastructure development, peace, and good governance with sustainable development assistance from donors directed to improving the life condition of the Ethiopian people, so that the population is self-suffi cient and confi dent enough to overcome burning issues like TB.
In conclusion, the study confi rms that TB drug delivery, without implementation of anti-poverty programs and more access to public health facilities, is ineffective.

Funding:
The author received no specifi c funding for this article.
Competing Interests: The author has declared that no competing interests exist.

Steven Goodman, Sander Greenland
The article published in PLoS Medicine by Ioannidis [1] makes the dramatic claim in the title that "most published research claims are false," and has received extensive attention as a result.The article does provide a useful reminder that the probability of hypotheses depends on much more than just the p-value, a point that has been made in the medical literature for at least four decades, and in the statistical literature for decades previous.This topic has renewed importance with the advent of the massive multiple testing often seen in genomics studies.
Unfortunately, while we agree that there are more false claims than many would suspect-based both on poor study design, misinterpretation of p-values, and perhaps analytic manipulation-the mathematical argument in the PLoS Medicine paper underlying the "proof" of the title's claim has a degree of circularity.As we show in detail in a separately published paper [2], Dr. Ioannidis utilizes a mathematical model that severely diminishes the evidential value of studies-even meta-analyses-such that none can produce more than modest evidence against the null hypothesis, and most are far weaker.This is why, in the offered "proof," the only study types that achieve a posterior probability of 50% or more (large RCTs [randomized controlled trials] and metaanalysis of RCTs) are those to which a prior probability of 50% or more are assigned.So the model employed cannot be considered a proof that most published claims are untrue, but is rather a claim that no study or combination of studies can ever provide convincing evidence.
The two assumptions that produce the above effect are: 1) Calculating the evidential effect only of verdicts of "signifi cance," i.e., p ≤ 0.05, instead of the actual p-value observed in a study, e.g., p = 0.001.
2) Introducing a new "bias" term into the Bayesian calculations, which even at a described "minimal" level (of 10%) has the effect of very dramatically diminishing a study's evidential impact.
In addition to the above problems, the paper claims to have proven something it describes as paradoxical; that the "hotter" an area is (i.e., the more studies published), the more likely studies in that area are to make false claims.We have shown this claim to be erroneous [2].The mathematical proof offered for this in the PLoS Medicine paper shows merely that the more studies published on any subject, the higher the absolute number of false positive (and false negative) studies.It does not show what the papers' graphs and text claim, viz, that the number of false claims will be a higher proportion of the total number of studies published (i.e., that the positive predictive value of each study decreases with increasing number of studies).
The paper offers useful guidance in a number of areas, calling attention to the importance of avoiding all forms of bias, of obtaining more empirical research on the prevalence of various forms of bias, and on the determinants of prior odds of hypotheses.But the claims that the model employed in this paper constitutes a "proof" that most published medical research claims are false, and that research in "hot" areas is most likely to be false, are unfounded.
Steven Goodman (sgoodman@jhmi.edu) Jason Andrews (jason.andrews@yale.edu) of Medicine New Haven, Connecticut, United States of America Citation: Andrews J, Basu S, Scales D, Smith-Rohrberg Maru D, Subbaraman R (2007) XDR-TB in South Africa: Theory and practice.PLoS Med 4(4): e163.doi:10.1371/journal.pmed.0040163Copyright: © 2007 Andrews et al.This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Goemaere E, Ford N, Berman D, McDermid C, Cohen R (2007) XDR-TB in South Africa: Detention is not the priority.PLoS Med 4(4): e162.doi:10.1371/journal.pmed.0040162Copyright: © 2007 Goemaere et al.This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Hospital in Motol, Charles University Prague, Czech Republic Citation: Tesfaye H (2007) Adherence to TB treatment in Ethiopia: Why do patients default?PLoS Med 4(4): e165.doi:10.1371/journal.pmed.0040165Copyright: © 2007 Hundie Tesfaye.This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Johns Hopkins School of Medicine Baltimore, Maryland, United States of America Sander Greenland University of California Los Angeles Los Angeles, California, United States of America