Conceived and designed the experiments: WYYL TL. Performed the experiments: WYYL. Analyzed the data: WYYL TL. Contributed reagents/materials/analysis tools: WYYL TL. Wrote the paper: WYYL TL.
The authors have declared that no competing interests exist.
The placement of medical research news on a newspaper's front page is intended to gain the public's attention, so it is important to understand the source of the news in terms of research maturity and evidence level.
We searched LexisNexis to identify medical research reported on front pages of major newspapers published from January 1, 2000 to December 31, 2002. We used MEDLINE and
Of 734 front-page medical research stories identified, 417 (57%) referred to mature research published in peer-reviewed journals. The remaining 317 stories referred to preliminary findings presented at scientific or press meetings; 144 (45%) of those stories mentioned studies that later matured (i.e. were published in journals within 3 years after news coverage). The evidence-level distribution of the 515 journal articles quoted in news stories reporting on mature research (3% level I, 21% level II, 42% level III, 4% level IV, and 31% level V) differed from that of the 170 reports of preliminary research that later matured (1%, 19%, 35%, 12%, and 33%, respectively; chi-square test,
Only 57% of front-page stories reporting on medical research are based on mature research, which tends to have a higher evidence level than research with preliminary findings. Medical research news should be clearly referenced and state the evidence level and limitations to inform the public of the maturity and quality of the source.
Popular media such as newspapers are commonly the initial source of medical research news for both medical professionals and the public
To medical researchers and health care professionals, the most credible source of medical information is generally accepted to be “mature” research, that is, studies published in peer-reviewed journals
The maturity and evidence level of medical research reported on newspaper front pages have so far not yet been investigated. We therefore aimed to systematically characterize sources of front-page medical research news in terms of journal publication status and level of evidence, and to assess whether these characteristics were reported in news stories.
We searched LexisNexis, an electronic news archive, for medical and health stories published on the front pages of any section of newspapers during a 3-year period from January 1, 2000 to December 31, 2002. In the “Major Papers” domain, which covers more than 50 high-circulation, English-language national and regional newspapers worldwide, we used “pg. 1”, “pg. A1”, “pg. 1A”, or “pg. A01” in the “full-text” field and the keyword “medical” or “health” in the default “headline, lead paragraph(s), terms” field. To target front-page news about medical research studies (e.g. treatments, drugs, medical devices, and public health developments), we refined our search by using the additional keyword “study” or “studies.” Next, the retrieved news items were manually searched to identify the source of medical research news and to exclude stories that were not truly based on studies with research findings (e.g. business and financial reports, medico-legal analyses, medical policies, health insurance stories, animal activist protests, and announcements of upcoming conferences and future studies) and duplicate news stories originating from different daily editions of the same newspaper.
Front-page news stories were classified according to whether or not they mentioned research already published in peer-reviewed journals. We assumed that peer review is the globally accepted quality-control process for biomedical journals and that studies published in journals are mature, reliable sources of medical information. In contrast, studies with preliminary findings were defined as those quoted in newspapers that had not yet been published in a journal but had been presented in scientific or press meetings, such as conferences convened by professional or academic organizations, or those initiated by the general press, private institutions, commercial organizations and public relations companies. Also included in this category were “tip offs” and anecdotal evidence mentioned by individuals within the medical community. We excluded news stories that were based on surveys or reports that had been compiled by government agencies, such as local, regional, state, or national governments and government departments (e.g. US Environmental Protection Agency), by international government-related organizations (e.g. United Nations Children's Fund), and by nongovernment agency and charity groups, because the publication status, peer review status, and maturity were usually unclear or impossible to trace.
For news items quoting mature research but not directly citing a journal, we checked whether the source had been published in a peer-reviewed journal by searching for stated names of researchers or topic keywords in MEDLINE and
The
Next, we used abstracts of retrieved journal articles corresponding to the research reported in the news story to categorize the study design and hence the study's likely level of evidence. If needed, full papers were used to confirm the study design and estimate the evidence level. We modified the Oxford EBM classification system
We classified the topic of each journal article using an expanded range of focus topics used by Bartlett et al
All source classifications according to publication status, level of evidence, study type, and research topic were done by one researcher. A second researcher coded a 20% random sample of news stories. Intercoder reliability (kappa) was calculated with SPSS version 13.0 (SPSS Inc, Chicago, Illinois). A score of 0.7 and higher indicates an acceptable degree of intercoder reliability. Chi-square tests were performed on categorical data with JMP version 5.1 (SAS Institute, Cary, NC). A
Of 734 front-page medical research news stories that mentioned study findings, 417 (57%) were based on mature research and 317 (43%) were based on preliminary findings at the time of news coverage (
The 417 front-page stories reporting on mature research were each based on findings from 1 to 4 published studies, with a total of 567 mentions of research studies published in 435 distinct journal articles. Of the 417 news stories, 251 did not overlap in content and mentioned 333 journal articles once: 190 stories reported on one journal article, 45 reported on 2 different articles, 11 reported on 3 articles, and 5 reported on 4 articles. Among the 166 news stories that showed some overlap in content, 113 reported on one journal article, 38 reported on 2 articles, and 15 reported on 3 articles. Twenty-seven journal articles were quoted once and 75 articles were quoted from 2 to 31 times (total, 234 article mentions). For example, an article on the Women's Health Initiative trial
The majority of the 417 news stories simply reported study findings, but 10 (2%) stories also challenged the findings of the published research. Only 101 (24%) news stories correctly stated the study type, mostly animal or laboratory studies; the rest did not report study type. None explicitly reported evidence level. Among the 567 newspaper mentions of any findings published in journals, 464 (82%) referenced the journal source correctly, 8 (1%) were referenced incorrectly, and 95 (17%) were not referenced.
For the analyses of evidence levels and study types, we used as the denominator all mentions of published findings (i.e. weighted citations) after excluding articles not presenting any evidence (50 commentaries or editorials and 2 nonresearch letters). The 515 mentions of mature findings were most commonly associated with level III and V evidence (42% and 31%, respectively) and least commonly with level I evidence (3%) [
BEFORE, No. (%) (n = 515) | AFTER, No. (%) (n = 170) | ||
I: Systematic reviews of randomized controlled trials | 14 (3) | 1 (1) | (Chi-square test) .0009 |
II: Randomized controlled trials | 106 (21) | 33 (19) | |
III: Prospective/retrospective cohort studies, ecological studies, case-control studies, and systematic reviews not of randomized controlled trials | 217 (42) | 59 (35) | |
IV: Case series | 20 (4) | 21 (12) | |
V: Expert opinions |
158 (31) | 56 (33) | |
Women's health | 126 (24) | 17 (10) | na |
Paediatrics | 57 (11) | 17 (10) | |
Men's health | 4 (1) | 8 (5) | |
Old age | 9 (2) | 1 (1) | |
Cancer | 20 (4) | 21 (12) | |
Heart disease | 48 (9) | 20 (12) | |
Diabetes | 10 (2) | 1 (1) | |
Reproduction | 16 (3) | 1 (1) | |
Mental health | 33 (6) | 14 (8) | |
Public, environmental & occupational health | 96 (19) | 29 (17) | |
Infectious diseases | 33 (6) | 16 (9) | |
Other medical research |
63 (12) | 25 (15) |
Denominators are weighted and are any article used as a front-page news source (Before) and any article published within 3 years of news coverage corresponding to any preliminary study used as a front-page news source (After).
Modified from the Oxford EBM classification system
Expert opinions comprised case reports, descriptive/methodological studies, narrative reviews, qualitative studies, audits, reports of expert committees, research letters, technical notes, and tutorials. Commentaries or editorials, news items, and nonresearch letters from journals were excluded from analysis (52 from “Before” and 7 from “After”).
Studies not classifiable using the other 11 topics (e.g. bionic eye studies, bone cement research).
na = not applicable.
Journal | No. (%) |
104 (20) | |
71 (14) | |
36 (7) | |
28 (5) | |
25 (5) | |
20 (4) | |
17 (3) | |
10 (2) | |
10 (2) | |
7 (1) | |
9 (5) | |
8 (5) | |
5 (3) | |
5 (3) | |
5 (3) |
Denominator is any article used as a front-page news source (weighted).
Denominator is any article published within 3 years of news coverage corresponding to any preliminary findings used as a front-page news source (weighted). Only the 5 most common journals are shown.
A total of 317 front-page stories were each based on preliminary findings from 1 to 5 studies, with a total of 365 mentions of research from 337 distinct studies. There were 274 stories with distinct sources (total, 316 studies) and 43 overlapping stories that were based on 21 studies (3 studies were reported 4 times, one was reported 3 times, and 17 were reported twice; total, 49 mentions). Only 116 (37%) news stories stated the name of the meeting or organizers and only 56 (18%) specifically stated that the findings were preliminary, had been released early, or had limitations because they came from unpublished research. None reported the evidence level.
For more than half (55%; 173/317) of the news stories that were based on preliminary findings, or 24% (173/734) of all stories, the corresponding studies (188 in total) remained unpublished within 3 years. In each of the remaining news stories reporting on preliminary findings (45%; 144/317), at least one of the quoted studies matured within 3 years after news coverage by publication in a peer-reviewed journal. Of these 144 news stories, 114 reported on one study, 21 reported on 2 studies, 6 reported on 3 studies, 2 reported on 4 studies, and one reported on 5 studies. The total number of newspaper mentions of studies with preliminary findings that later matured was 177, but the actual number of distinct studies was 157.
For the analyses of evidence levels and study types of research that matured after newspaper coverage, we again used as the denominator published articles corresponding to all mentions of studies (i.e. weighted citations) after excluding articles not presenting evidence (5 commentaries or editorials, 1 nonresearch letter, and 1 news item). The 170 articles again most commonly contained level III or V evidence (35% and 33%, respectively) and least commonly level I evidence (1%) [
When we examined the study designs of the 515 mentioned studies that had already been published at the time of news coverage, we found that 46% were OSs, 21% were RCTs, 17% were animal or laboratory studies, 14% were expert opinions, and 3% were SRs of RCTs. This profile was similar to that of the 170 cited studies with preliminary findings that later matured (47%, 19%, 20%, 13%, and 1%, respectively; chi-square test,
Among the 515 mentioned studies that had already been published at the time of news coverage, the 3 leading research topics were women's health (126; 24%); public, environmental and occupational health (96; 19%); and “other” medical research (63; 12%). The 3 most common topics among the 170 cited studies with preliminary findings that later matured were public, environmental and occupational health (29; 17%); “other” medical research (25; 15%); and cancer (21; 12%) [
This is the first study to examine the characteristics of a cross-section of front-page medical research news in major newspapers in terms of research maturity and associated level of scientific evidence. Only 57% of front-page news stories reporting medical research can be said to be based on mature research, whereas the remaining 43% cites research based on unpublished medical information, which is less easy to verify, reference, and assess. We recommend that journalists recognize and clearly distinguish between these 2 main kinds of medical information, because studies published in peer-reviewed medical journals have supposedly been scrutinized by experts, whereas unpublished studies are deemed by the scientific community to be preliminary and are not guaranteed to have undergone a rigorous and independent review process. Peer review would also ideally ensure adherence to publishing guidelines such as the “Uniform Requirements” of the International Committee of Medical Journal Editors
We acknowledge that the journalistic process may emphasize being first to report a news story, making conferences a popular information source. Still, Woloshin and Schwartz
We did not detect a difference between the distributions of study types among research that was mature at the time of news coverage and research that matured later. Assuming research articles published after news coverage retain their original study design, the most frequently reported study types on newspaper front pages overall were OSs, followed by RCTs, animal or laboratory studies, expert opinions, and SRs. Bartlett et al
The predominance of OSs among studies reported in front-page news stories may indicate either that the media prefer to report this type of study or that OSs are overrepresented among the information sources available to journalists, both in journals and at meetings, whereas SRs and RCTs are underrepresented. Bartlett et al
Our analysis comparing research that matured before and after newspaper coverage suggests that mature medical research chosen to appear on front pages generally has a higher evidence level than preliminary research. Assuming research articles published after news coverage retain their original evidence level, we found that level IV evidence tends to appear more frequently in front-page news based on preliminary findings than in news based on mature research; level III evidence tends to appear more frequently in the latter. Because both levels III and IV correspond to OSs, including EBM classification as well as study design type in news stories could provide extra, useful information about their sources. In our sample of newspaper front pages, only one-quarter of stories based on mature research evidence stated the study design correctly; none of the stories, whether based on mature or preliminary studies, stated evidence level or referred to any EBM classification. Hence, we recommend that reporters state and explain the level of evidence of any source study, although it is also important to realize that the best available evidence may not necessarily always be from SRs and RCTs (i.e. levels I & II, respectively), for logistic or ethical reasons. Medical professionals could assist the media by helping to put their research studies into context, clarifying risks
Nearly 1 in 5 news stories based on mature research did not give a journal reference and only 2 in 5 news stories based on preliminary findings stated the information source. Clear citation of sources is essential if the media wish to convey accuracy and trustworthiness of medical information. This practice also allows readers to verify the quality of medical research news, especially because news stories may sometimes challenge the findings of published journal papers, as we noted in a small proportion of stories. Healthy debate of peer-reviewed research is to be encouraged, as long as such news stories help readers identify the current best evidence and clarify the context of findings without prioritizing selected opinions over others or over conclusions of published studies
Finally, medical research news dealing with public, environmental and occupational health was often covered on front pages by the press. This trend may be related to the underlying preference for topics of research covered in OSs. Alternatively, perhaps the public tends to be interested in issues of public, environmental and occupational health and the majority of these studies are observational. Also, the widespread news coverage of the Women's Health Initiative trial
An important strength of our study is the systematic selection of a large number of front-page medical research stories reporting on research findings in 47 English-language newspapers over a 3-year period. However, we noted the absence of some high-profile newspapers—
In our sample, only 57% of medical research news stories appearing on newspaper front pages were based on mature research and hence had been scrutinized by peer reviewers and made accessible to others for verification; 24% of news stories were based on preliminary findings that remained unpublished in the following 3 years. We encourage journalists to clearly distinguish whether medical research news is based on mature research or studies with preliminary findings; reference and cite sources well; and become familiar with EBM principles, especially when preparing front-page and hence high-profile medical news stories. Medical reporters and media organizations could enhance their reputations by advocating responsible medical news reporting, including mentioning evidence levels and study limitations
List of newspapers used in this study
(0.06 MB DOC)
List of journals associated with news based on mature research
(0.11 MB DOC)
List of journals associated with news based on preliminary findings
(0.10 MB DOC)