How Do US Journalists Cover Treatments, Tests, Products, and Procedures? An Evaluation of 500 Stories

HealthNewsReview.org evaluates US health news coverage of claims made about medical interventions. Gary Schwitzer reports on the project's findings after evaluation of 500 health news stories.


Health in Action
N ews stories about new treatments, tests, products, and procedures appear daily. Such reporting should ideally be accurate, balanced, and complete so that health care consumers are properly informed and ready to participate in decision making about their health care. If reporting is inaccurate, imbalanced, or incomplete, consumers may have unrealistic expectations and demand of their physicians care that would be of little value or even harmful.
Is the news media doing a good job of reporting on new treatments, tests, products, and procedures? Ray Moynihan and colleagues analyzed how often news stories quantifi ed the costs, benefi ts, and harms of the interventions being discussed, and how often they reported potential confl icts of interest in story sources [1]. Of the 207 newspaper and television stories that they studied, 83 did not report the benefi ts of medications quantitatively, and of the 124 stories that did quantify the benefi ts of medications, only 18 presented both relative and absolute benefi ts. Of all the stories, 53% had no information about potential harms of the treatment, and 70% made no mention of treatment costs. Of 170 stories that cited an expert or a scientifi c study, 85 (50%) cited at least one with a fi nancial tie to the manufacturer of the drug, a tie that was disclosed in only 33 of the 85 stories.
Moynihan and colleagues' work was one of the inspirations for the creation of the Australian Media Doctor Web site (http://www.mediadoctor.org.au/) in 2004. That project monitors the health news coverage of 13 Australian news organizations. The project concluded, after its fi rst six months experience, that "Australian lay news reporting of medical advances…is poor" [2].
In Canada, Alan Cassels and colleagues documented similar journalistic shortcomings [3]. Cassels  The Health in Action section is a forum for individuals or organizations to highlight their innovative approaches to a particular health problem.

Summary Points
• The daily delivery of news stories about new treatments, tests, products, and procedures may have a profound-and perhaps harmfulimpact on health care consumers.
• A US Web site project, HealthNewsReview.org (http:// HealthNewsReview.org/), modeled after similar efforts in Australia and Canada, evaluates and grades health news coverage, notifying journalists of their grades.
• After almost two years and 500 stories, the project has found that journalists usually fail to discuss costs, the quality of the evidence, the existence of alternative options, and the absolute magnitude of potential benefi ts and harms.
• Reporters and writers have been receptive to the feedback; editors and managers must be reached if change is to occur.
• Time (to research stories), space (in publications and broadcasts), and training of journalists can provide solutions to many of the journalistic shortcomings identifi ed by the project.
and evening newscasts of the three most watched television networks-ABC, CBS, and NBC. In order to be eligible for review, a story must include a claim of effi cacy or safety in a health care product or procedure (drug, device, diagnostic or screening test, surgical procedure, dietary recommendation, vitamin, supplement).
The rating instrument used (http://www.healthnewsreview.org/ ratings_info.php) includes ten criteria used by the Australian and Canadian Media Doctor sites. All of the criteria are addressed in the Association of Health Care Journalists' Statement of Principles [4]. As seen in the methods of evaluation listed in Box 1, each story is evaluated on whether it has met the ten criteria.
For each criterion, the story is given a rating of "satisfactory," "unsatisfactory," or "not applicable." Three reviewerswith backgrounds in medicine, health services research, public health, or journalism-analyze each article. (All reviewers are listed online at: http:// www.healthnewsreview.org/people. php.) As the publisher of the project, I am always the third reviewer of each story, and I mediate any disagreements between the fi rst two reviewers, gaining consensus before publishing the fi nal evaluation of the health story.
Inter-reviewer reliability was tested using a random sample of 30 stories. Two reviewers coded each story. The average percent agreement between the two reviewers across the ten ratings criteria was 74%.

Results and Impact of the Project to Date
In our evaluation of 500 US health news stories over 22 months, between 62%-77% of stories failed to adequately address costs, harms, benefi ts, the quality of the evidence, and the existence of other options when covering health care products and procedures (Table 1). This high rate of inadequate reporting raises important questions about the quality of the information US consumers receive from the news media on these health news topics.
At a time when US health care spending now represents 16% of the gross domestic product [5], only 23% of the health news stories that we analyzed covered the costs of medical

Adequately discusses costs.
We believe that, in an era when health care spending represents 16% of the US gross domestic product, a story is incomplete if it does not address the costs of an approach, and a comparison with existing alternatives. We also think journalists should explore whether insurers are likely to pay for it.

Quantifi es benefi ts.
Stories should give some sense of the size of the potential benefi ts of the approach being discussed. Stories (and studies, for that matter) should also explain the benefi ts in absolute, not just relative, terms.

Adequately explains and quantifi es potential harms.
Stories should give a complete picture of potential harms of an approach, and quantify those potential harms in absolute terms.

Compares the new idea with existing alternatives.
We expect that a story would put the new approach being discussed into the context of existing alternatives, with some discussion of the possible advantages or disadvantages of the new approach compared with existing alternatives.

Seeks out independent sources and discloses potential confl icts of interest.
We expect, just as the Association of Health Care Journalists does, that journalists should "recognize that most stories involve a degree of nuance and complexity that no single source could provide. To refl ect only one perspective of only one source is not wise; [journalists should] be vigilant in selecting sources, asking about, weighing and disclosing relevant fi nancial, advocacy, personal or other interests of those [they] interview as a routine part of story research and interviews" [4].

Avoids disease mongering.
This criterion is an attempt to help journalists avoid promulgating the medicalization of normal states of or variations in health (e.g., baldness, menstruation, short stature, etc.). We also try to educate journalists about surrogate endpoints and about how risk factors are not diseases. With this criterion, we also remind them not to exaggerate the prevalence or incidence of a condition.

Reviews the study methodology or the quality of the evidence.
The story should refl ect an understanding that not all studies are equal. If a story does not point out some of the limitations of an observational study and does not caution about interpreting uncontrolled data, for example, we will judge it unsatisfactory.

Establishes the true novelty of the idea
Many "new" products or procedures are not really novel. The product reported may be the sixth new member of a well-established class of drugs. It may be a device that has only been judged to be substantially equivalent to other devices already on the market. Journalists should accurately refl ect the novelty (or lack thereof) of "new" products or procedures.

Establishes the availability of the product or procedure.
Many stories report on products or procedures that are still in clinical trials. We expect journalists to explain whether something is only available via limited access in clinical trials, whether something is FDA approved, whether insurability limits availability, etc. Many news stories seem to treat FDA approval of an investigational drug as a fait accompli, making predictions about how the drug "could be" or "should be" approved and on the market within a given time frame. Such stories would be rated as unsatisfactory.

Appears not to rely solely or largely on a news release.
We expect, just as the Association of Health Care Journalists does, that journalists should "Preserve journalistic independence by avoiding the use of video news releases or the use of quotes from printed news releases; label and credit the source whenever a portion of a video or printed news release is used" [4]. We expect a journalist to use a news release for background information only, and to then seek independent experts to comment on a development.

Box 1. HealthNewsReview.org Ratings Criteria and Explanation
treatments adequately. In an era when most news organizations reported on postmarketing problems found with rofecoxib (Vioxx), coronary stents, or hormone replacement therapy, many of those same news organizations still fail to adequately quantify the harms and benefi ts of the products they report on today. Only 28% of the stories we evaluated adequately covered benefi ts, and only 33% adequately covered harms. Many newsrooms across the US have eliminated health and medical reporting positions because of a drop in profi ts [6]. Those cutbacks may be the reason why only 35% of stories we evaluated were rated satisfactory on the criterion of whether the reporter had discussed the study methodology and the quality of the evidence, issues that only a trained health journalist could be expected to understand.
Only 38% of stories were rated satisfactory for putting the intervention under discussion into the context of existing alternative options. The Statement of Principles of the Association of Health Care Journalists [4] urges journalists to disclose relevant confl icts of interests in their sources "as a routine part" of their work, and to "avoid single-source stories." Our analysis shows that journalists are following this professional guideline only about half the time. Examples of problems found in stories scoring poorly are provided in Box 2. Of the fi rst 500 stories reviewed, 41 (8%) received our highest scores. They appear online at http://www. healthnewsreview.org/fi veStars.php.

Next Steps
We are working to communicate these fi ndings to news organizations and to editorial decision makers, and to help them see that they often deliver an imbalanced picture of health care interventions. We believe this imbalance may have a profound impact on the decision making of American consumers who rely on these news stories.
When we evaluate a health news story, we e-mail the evaluation to the journalist who wrote that story, and their responses have been overwhelmingly positive. "It's quite sobering to read the reviews," wrote one journalist. "I imagine you've heard all the laments from reporters, but the lack of both space and research time is enormously frustrating (and will probably drive me out of journalism in the end)." Time and space can provide answers to many of the shortcomings we identifi ed. Reporters who are given more time to research a topic and more space or time in their publications or broadcasts would be better able to address the criteria that we-and their own industry principles-state are important.
Some news organizations choose to offer "health headlines," "medical minutes," or "science briefs"presumably to show the breadth of their coverage. But our analysis suggests that so much is left out in these short items that it is diffi cult to convey any of the nuance that is so important in almost all such stories. We take the position that if a news organization cannot give suffi cient space or time to a story to cover the necessary issues, then it would be better if the story were not published or broadcast at all. Incomplete stories that lack context can cause harm. People may be misled, become anxious, or make ill-informed decisions based on such stories.
Training opportunities have been slashed in many newsrooms [7]. A survey of Midwestern newspapers showed that nearly all reporters covering health news had received no training in this subject matter or in interpreting health statistics; of those, most said that such training would be helpful [8]. Many different forms of health care journalism training opportunities are now available. The National Institutes of Health, the Centers for Disease Control and Prevention, the Knight Foundation, the Association of Health Care Journalists, and several universities now offer specialized training programs of varying lengths, degrees of complexity, and formats.
We observed what we considered to be incomplete coverage in some of the nation's top news organizations, and gave some of our top scores to stories in what might be called second-tier newspapers. These observations suggest that attaining excellence in health news stories is not just an issue of resources.
While our project is in no way a complete analysis of the work done over 22 months by the news organizations we examined, it nonetheless represents the largest dynamic database of content analysis of US health news coverage ever assembled, and it continues to grow daily. It provides the clearest picture yet available of how major newspapers, magazines, and television networks cover treatments, tests, products, and procedures. And this is the fi rst project that gives journalists regular feedback on how they cover health news stories.
We think that our project's fi ndings should be a cause for refl ection by all parties involved in the dissemination of news and information to journalists covering health and medicineincluding medical journals, government agencies, industry, academic medical centers, and individual clinicians and researchers. Some have described sensationalism in medical science stories as the byproduct of a complicit collaboration between journalists and scientists [9], wherein journalists fi nd it easier to get attention for their stories and scientists see "the practical value of media attention to a successful scientifi c career" [9]. Rather than allowing the interests of the two professions to infl uence each another in such an unhealthy manner, we suggest that each should examine its core ethical principles.
One study, by Steve Woloshin and Lisa Schwartz, showed that news releases from medical journals "do not routinely highlight study limitations or the role of industry funding. Data are often presented using formats that may exaggerate the perceived importance of fi ndings" [10]. Vikki Entwistle showed how "the way information fl ows from medical journals to newspapers infl uences the balance of medical topics reported, the quality of the research reported (and its appropriateness for public attention), and the quality of news reporting" [11].
We hope that our evaluation of health news will lead news organizations-and all who engage in the dissemination of health news and information-to reevaluate their practices to better serve a more informed health care consumer population.