News stories about new treatments, tests, products, and procedures appear daily. Such reporting should ideally be accurate, balanced, and complete so that health care consumers are properly informed and ready to participate in decision making about their health care. If reporting is inaccurate, imbalanced, or incomplete, consumers may have unrealistic expectations and demand of their physicians care that would be of little value or even harmful.

Is the news media doing a good job of reporting on new treatments, tests, products, and procedures? Ray Moynihan and colleagues analyzed how often news stories quantified the costs, benefits, and harms of the interventions being discussed, and how often they reported potential conflicts of interest in story sources [1]. Of the 207 newspaper and television stories that they studied, 83 did not report the benefits of medications quantitatively, and of the 124 stories that did quantify the benefits of medications, only 18 presented both relative and absolute benefits. Of all the stories, 53% had no information about potential harms of the treatment, and 70% made no mention of treatment costs. Of 170 stories that cited an expert or a scientific study, 85 (50%) cited at least one with a financial tie to the manufacturer of the drug, a tie that was disclosed in only 33 of the 85 stories.

Moynihan and colleagues' work was one of the inspirations for the creation of the Australian Media Doctor Web site (http://www.mediadoctor.org.au/) in 2004. That project monitors the health news coverage of 13 Australian news organizations. The project concluded, after its first six months experience, that “Australian lay news reporting of medical advances…is poor” [2].

In Canada, Alan Cassels and colleagues documented similar journalistic shortcomings [3]. Cassels heads a team that launched a Canadian Media Doctor Web site (http://www.mediadoctor.ca/) in 2005. That project evaluates health news coverage by 12 Canadian news organizations.

These efforts helped inspire a project to evaluate United States health news coverage of claims made about treatments, tests, products, and procedures. HealthNewsReview.org started publishing evaluations of health news stories in April 2006. The sole support for the project is a grant from the 501c3 nonprofit Foundation for Informed Medical Decision Making (http://www.fimdm.org/about.php), founded in 1989 by Dartmouth's Dr. Jack Wennberg and colleagues, with a mission of “assuring that people understand their choices and have the information they need to make sound decisions affecting their health and well being.”

This article reports on the project's findings after its first 22 months and after evaluation of 500 health news stories.

The HealthNewsReview.org Project

HealthNewsReview.org monitors news coverage by the top 50 most widely circulated newspapers in the US; the most widely used wire service, the Associated Press; and the three leading newsweekly magazines—TIME, Newsweek, and U.S. News & World Report. Each weekday we watch the morning and evening newscasts of the three most watched television networks—ABC, CBS, and NBC.

In order to be eligible for review, a story must include a claim of efficacy or safety in a health care product or procedure (drug, device, diagnostic or screening test, surgical procedure, dietary recommendation, vitamin, supplement).

The rating instrument used (http://www.healthnewsreview.org/ratings_info.php) includes ten criteria used by the Australian and Canadian Media Doctor sites. All of the criteria are addressed in the Association of Health Care Journalists' Statement of Principles [4]. As seen in the methods of evaluation listed in Box 1, each story is evaluated on whether it has met the ten criteria.

For each criterion, the story is given a rating of “satisfactory,” “unsatisfactory,” or “not applicable.” Three reviewers—with backgrounds in medicine, health services research, public health, or journalism—analyze each article. (All reviewers are listed online at: http://www.healthnewsreview.org/people.php.) As the publisher of the project, I am always the third reviewer of each story, and I mediate any disagreements between the first two reviewers, gaining consensus before publishing the final evaluation of the health story.

Inter-reviewer reliability was tested using a random sample of 30 stories. Two reviewers coded each story. The average percent agreement between the two reviewers across the ten ratings criteria was 74%.

Results and Impact of the Project to Date

In our evaluation of 500 US health news stories over 22 months, between 62%–77% of stories failed to adequately address costs, harms, benefits, the quality of the evidence, and the existence of other options when covering health care products and procedures (Table 1). This high rate of inadequate reporting raises important questions about the quality of the information US consumers receive from the news media on these health news topics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Percentage of Satisfactory Reviews for 10 Criteria for 500 Stories

https://doi.org/10.1371/journal.pmed.0050095.t001

At a time when US health care spending now represents 16% of the gross domestic product [5], only 23% of the health news stories that we analyzed covered the costs of medical treatments adequately. In an era when most news organizations reported on postmarketing problems found with rofecoxib (Vioxx), coronary stents, or hormone replacement therapy, many of those same news organizations still fail to adequately quantify the harms and benefits of the products they report on today. Only 28% of the stories we evaluated adequately covered benefits, and only 33% adequately covered harms.

Many newsrooms across the US have eliminated health and medical reporting positions because of a drop in profits [6]. Those cutbacks may be the reason why only 35% of stories we evaluated were rated satisfactory on the criterion of whether the reporter had discussed the study methodology and the quality of the evidence, issues that only a trained health journalist could be expected to understand.

Only 38% of stories were rated satisfactory for putting the intervention under discussion into the context of existing alternative options. The Statement of Principles of the Association of Health Care Journalists [4] urges journalists to disclose relevant conflicts of interests in their sources “as a routine part” of their work, and to “avoid single-source stories.” Our analysis shows that journalists are following this professional guideline only about half the time. Examples of problems found in stories scoring poorly are provided in Box 2. Of the first 500 stories reviewed, 41 (8%) received our highest scores. They appear online at http://www.healthnewsreview.org/fiveStars.php.

Next Steps

We are working to communicate these findings to news organizations and to editorial decision makers, and to help them see that they often deliver an imbalanced picture of health care interventions. We believe this imbalance may have a profound impact on the decision making of American consumers who rely on these news stories.

When we evaluate a health news story, we e-mail the evaluation to the journalist who wrote that story, and their responses have been overwhelmingly positive. “It's quite sobering to read the reviews,” wrote one journalist. “I imagine you've heard all the laments from reporters, but the lack of both space and research time is enormously frustrating (and will probably drive me out of journalism in the end).” Time and space can provide answers to many of the shortcomings we identified. Reporters who are given more time to research a topic and more space or time in their publications or broadcasts would be better able to address the criteria that we—and their own industry principles—state are important.

Some news organizations choose to offer “health headlines,” “medical minutes,” or “science briefs”—presumably to show the breadth of their coverage. But our analysis suggests that so much is left out in these short items that it is difficult to convey any of the nuance that is so important in almost all such stories. We take the position that if a news organization cannot give sufficient space or time to a story to cover the necessary issues, then it would be better if the story were not published or broadcast at all. Incomplete stories that lack context can cause harm. People may be misled, become anxious, or make ill-informed decisions based on such stories.

Training opportunities have been slashed in many newsrooms [7]. A survey of Midwestern newspapers showed that nearly all reporters covering health news had received no training in this subject matter or in interpreting health statistics; of those, most said that such training would be helpful [8]. Many different forms of health care journalism training opportunities are now available. The National Institutes of Health, the Centers for Disease Control and Prevention, the Knight Foundation, the Association of Health Care Journalists, and several universities now offer specialized training programs of varying lengths, degrees of complexity, and formats.

We observed what we considered to be incomplete coverage in some of the nation's top news organizations, and gave some of our top scores to stories in what might be called secondtier newspapers. These observations suggest that attaining excellence in health news stories is not just an issue of resources.

While our project is in no way a complete analysis of the work done over 22 months by the news organizations we examined, it nonetheless represents the largest dynamic database of content analysis of US health news coverage ever assembled, and it continues to grow daily. It provides the clearest picture yet available of how major newspapers, magazines, and television networks cover treatments, tests, products, and procedures. And this is the first project that gives journalists regular feedback on how they cover health news stories.

We think that our project's findings should be a cause for reflection by all parties involved in the dissemination of news and information to journalists covering health and medicine—including medical journals, government agencies, industry, academic medical centers, and individual clinicians and researchers. Some have described sensationalism in medical science stories as the byproduct of a complicit collaboration between journalists and scientists [9], wherein journalists find it easier to get attention for their stories and scientists see “the practical value of media attention to a successful scientific career” [9]. Rather than allowing the interests of the two professions to influence each another in such an unhealthy manner, we suggest that each should examine its core ethical principles.

One study, by Steve Woloshin and Lisa Schwartz, showed that news releases from medical journals “do not routinely highlight study limitations or the role of industry funding. Data are often presented using formats that may exaggerate the perceived importance of findings” [10]. Vikki Entwistle showed how “the way information flows from medical journals to newspapers influences the balance of medical topics reported, the quality of the research reported (and its appropriateness for public attention), and the quality of news reporting” [11].

We hope that our evaluation of health news will lead news organizations—and all who engage in the dissemination of health news and information—to reevaluate their practices to better serve a more informed health care consumer population.