Isabelle Boutron is a member of
Conceived and designed the experiments: AY IB PR. Performed the experiments: AY AB IM PC. Analyzed the data: AY. Contributed reagents/materials/analysis tools: AY IB AB IM PC JM PR. Wrote the first draft of the manuscript: AY IB. Contributed to the writing of the manuscript: AY IB PR.
A study conducted by Amélie Yavchitz and colleagues examines the factors associated with “spin” (specific reporting strategies, intentional or unintentional, that emphasize the beneficial effect of treatments) in press releases of clinical trials.
Previous studies indicate that in published reports, trial results can be distorted by the use of “spin” (specific reporting strategies, intentional or unintentional, emphasizing the beneficial effect of the experimental treatment). We aimed to (1) evaluate the presence of “spin” in press releases and associated media coverage; and (2) evaluate whether findings of randomized controlled trials (RCTs) based on press releases and media coverage are misinterpreted.
We systematically searched for all press releases indexed in the EurekAlert! database between December 2009 and March 2010. Of the 498 press releases retrieved and screened, we included press releases for all two-arm, parallel-group RCTs (
“Spin,” defined as specific reporting strategies (intentional or unintentional) emphasizing the beneficial effect of the experimental treatment, was identified in 28 (40%) scientific article abstract conclusions and in 33 (47%) press releases. From bivariate and multivariable analysis assessing the journal type, funding source, sample size, type of treatment (drug or other), results of the primary outcomes (all nonstatistically significant versus other), author of the press release, and the presence of “spin” in the abstract conclusion, the only factor associated, with “spin” in the press release was “spin” in the article abstract conclusions (relative risk [RR] 5.6, [95% CI 2.8–11.1],
“Spin” was identified in about half of press releases and media coverage. In multivariable analysis, the main factor associated with “spin” in press releases was the presence of “spin” in the article abstract conclusion.
The mass media play an important role in disseminating the results of medical research. Every day, news items in newspapers and magazines and on the television, radio, and internet provide the general public with information about the latest clinical studies. Such news items are written by journalists and are often based on information in “press releases.” These short communications, which are posted on online databases such as EurekAlert! and sent directly to journalists, are prepared by researchers or more often by the drug companies, funding bodies, or institutions supporting the clinical research and are designed to attract favorable media attention to newly published research results. Press releases provide journalists with the information they need to develop and publish a news story, including a link to the peer-reviewed journal (a scholarly periodical containing articles that have been judged by independent experts) in which the research results appear.
In an ideal world, journal articles, press releases, and news stories would all accurately reflect the results of health research. Unfortunately, the findings of randomized controlled trials (RCTs—studies that compare the outcomes of patients randomly assigned to receive alternative interventions), which are the best way to evaluate new treatments, are sometimes distorted in peer-reviewed journals by the use of “spin”—reporting that emphasizes the beneficial effects of the experimental (new) treatment. For example, a journal article may interpret nonstatistically significant differences as showing the equivalence of two treatments although such results actually indicate a lack of evidence for the superiority of either treatment. “Spin” can distort the transposition of research into clinical practice and, when reproduced in the mass media, it can give patients unrealistic expectations about new treatments. It is important, therefore, to know where “spin” occurs and to understand the effects of that “spin”. In this study, the researchers evaluate the presence of “spin” in press releases and associated media coverage and analyze whether the interpretation of RCT results based on press releases and associated news items could lead to the misinterpretation of RCT results.
The researchers identified 70 press releases indexed in EurekAlert! over a 4-month period that described two-arm, parallel-group RCTs. They used Lexis Nexis, a database of news reports from around the world, to identify associated news items for 41 of these press releases and then analyzed the press releases, news items, and abstracts of the scientific articles related to each press release for “spin”. Finally, they interpreted the results of the RCTs using each source of information independently. Nearly half the press releases and article abstract conclusions contained “spin” and, importantly, “spin” in the press releases was associated with “spin” in the article abstracts. The researchers overestimated the benefits of the experimental treatment from the press release as compared to the full-text peer-reviewed article for 27% of reports. Factors that were associated with this overestimation of treatment benefits included publication in a specialized journal and having “spin” in the press release. Of the news items related to press releases, half contained “spin”, usually of the same type as identified in the press release and article abstract. Finally, the researchers overestimated the benefit of the experimental treatment from the news item as compared to the full-text peer-reviewed article in 24% of cases.
These findings show that “spin” in press releases and news reports is related to the presence of “spin” in the abstract of peer-reviewed reports of RCTs and suggest that the interpretation of RCT results based solely on press releases or media coverage could distort the interpretation of research findings in a way that favors experimental treatments. This interpretation shift is probably related to the presence of “spin” in peer-reviewed article abstracts, press releases, and news items and may be partly responsible for a mismatch between the perceived and real beneficial effects of new treatments among the general public. Overall, these findings highlight the important role that journal reviewers and editors play in disseminating research findings. These individuals, the researchers conclude, have a responsibility to ensure that the conclusions reported in the abstracts of peer-reviewed articles are appropriate and do not over-interpret the results of clinical research.
Please access these Web sites via the online version of this summary at
The
The UK National Health Service Choices website includes
The US-based organization
The media play an important role in the dissemination of findings from health research. More than half of US adults report that they follow health news closely
Randomized controlled trials (RCTs) are considered the gold standard for therapeutic evaluation
We aimed to (1) evaluate the presence of “spin” in press releases and associated media coverage and (2) evaluate whether findings of RCTs contained within press releases and media coverage are misinterpreted.
We identified all press releases indexed in EurekAlert! (online free database for science press releases;
The title and full text of all retrieved press releases were screened by one reviewer to exclude any non-eligible press releases.
We obtained a copy of the scientific article related to the press release from (1) the direct link or full reference citation reported in the press release, if available; or (2) the PubMed single citation matcher indicating the year of publication, journal, and author's name. Each retrieved scientific article (abstract and full text) was assessed by the same reader to confirm eligibility.
Finally, for all selected press releases, we systematically searched for related news items in the “general news” library of LEXIS-NEXIS using (1) the name of the disease; (2) the treatment being evaluated, and, if needed, the name of the first or second author. All news related to the articles or press releases were retrieved, and we selected the news that had the highest number of words dedicated to the selected study.
Data were abstracted from the press release, news items, and the related published scientific article. For this purpose, we developed a standardized data-abstraction form using previous work on the same topics
The data-abstraction form was preliminarily tested by two of the reviewers with a sample of 15 press releases and original articles indexed in January 2008. The data that involved some subjectivity, such as the type of “spin” were abstracted by two independent reviewers, with discrepancies resolved by consensus. Other data were evaluated by a single reviewer. The concordance between the two reviewers for the assessment of “spin” is reported in Text S3; the mean kappa coefficient for “spin” was 0.56 (range 0.43–0.69).
We systematically extracted data related to the characteristics of (1) the RCT, (2) the press releases, and (3) the presence of “spin” in the article abstract conclusions, in the press release and, when available, in the news items.
We defined “spin” as a specific reporting (intentional or unintentional) that emphasizes the beneficial effect of the experimental treatment. We used a classification of “spin” described in a previous work
The RCT results were interpreted independently from three different sources: (1) from the full text of the scientific article, (2) from the press release, and (3) from the news items.
For each source, different pairs of assessors independently evaluated the results of the RCT and achieved consensus. Assessment based on the scientific article relied on the results for the primary outcomes, secondary outcomes, and harm. For assessment of press releases, assessors were blinded to the authors of the press release, the content of the scientific article, and the journal of publication. For assessment of news items, assessors were blinded to the content of the press release and scientific article. All results reported represent the consensus of each pair of assessors.
The trial results were interpreted independently by use of the same scale, from 1 to 5
Misinterpretation was defined as the interpretation of the press release or news items differing from that based on the full-text article by at least one class according to the above three-class system of scores. Misinterpretation of the press release or news items could overestimate the treatment beneficial effect or underestimate the treatment effect. For example, an overestimation of the treatment beneficial effect in the press release or news items occurred when reading the published article led to rating the trial results as neutral, whereas reading the press release or new items led to rating the experimental treatment as beneficial.
Data for quantitative variables are expressed with medians and IQRs. Data for qualitative variables are expressed with frequencies and percentages. We planned bivariate and multivariable analysis to identify factors associated with (1) “spin” in the press releases, (2) an overestimation of the beneficial effect of the experimental treatment from press releases, (3) “spin” in the news items, and (4) an overestimation of the beneficial effect of the experimental treatment from news items. For bivariate analysis, we used the chi-square or Fisher exact test for categorical data and the Student
Statistical analysis involved use of SAS v9.1 (SAS Institute).
The search strategy in EurekAlert! between December 1, 2009, and March 31, 2010, retrieved 498 press releases. The selection process resulted in 70 press releases and related scientific articles (
The description of the scientific articles is in
Characteristics | Subcharacteristics | |
Type of journal, |
General medical journal | 32 (46) |
Specialized medical journal | 38 (54) | |
Funding source, |
Profit or both profit and nonprofit | 33 (47) |
None or nonprofit | 33 (47) | |
Not reported | 4 (6) | |
Sample size median; [IQR]; (min–max) | 112; [54–435]; (16–94,370) | |
Experimental treatment, |
Drug | 36 (51) |
Surgery/procedure | 9 (13) | |
Device | 5 (7) | |
Therapeutic strategy | 7 (10) | |
Participative intervention | 12 (17) | |
Other | 1 (1) | |
Comparator, |
Placebo | 29 (41) |
Active treatment | 32(46) | |
Other | 9 (13) | |
Primary outcomes clearly identified, |
61(87) | |
Type of primary outcomes, |
Efficacy | 61 (87) |
Safety | 1 (1) | |
Both | 4 (6) | |
Unclear | 4 (6) | |
Primary outcomes reported adequately, |
56 (80) | |
All statistically significant | 34 (49) | |
All statistically nonsignificant | 24 (34) | |
Some statistically significant/some not | 11 (16) | |
Unclear | 1 (1) | |
At least one “spin” | 28 (40) | |
Type of “spin” |
No acknowledgment of nonstatistically significant primary outcome | 14 (20) |
Claiming equivalence when results failed to demonstrate a statistically significant difference | 5 (7) | |
Focus on positive secondary outcome | 3 (4) | |
Focus on inappropriate subgroup | 4 (6) | |
Focus on within-group (or over-all within) comparison | 6 (9) | |
Nonstatistically significant outcome reported as if they were significant | 3 (4) | |
Ignored data of safety | 1 (1) | |
Inadequate claim of safety | 4 (6) | |
Inappropriate extrapolation | 6 (9) | |
Other | 5 (7) |
Adequately, with effect size and precision or treatment effect in each arm with precision.
Numbers do not add up as the types of “spin” were not mutually exclusive.
The general characteristics of press releases are in
Characteristics | Subcharacteristics | |
Origin, |
Press officer | 40 (57) |
Industry or institution | 30 (43) | |
Easy access to full article, (i.e., direct link or the full reference) |
36 (51) | |
Funding reported, |
25 (36) | |
Design reported, |
70 (100) | |
Sample size reported, |
65 (93) | |
Length of follow-up reported, |
46 (66) | |
Primary outcomes reported, |
In words only | 29 (41) |
Per arms | 30 (43) | |
With effect size | 17 (24) | |
Safety reported, |
Mentioned | 24 (34) |
Quantified | 14 (20) | |
Limits reported, |
10 (14) | |
Interview included, |
Authors only | 40 (57) |
Experts or editorialists only | 6 (9) | |
Both | 12 (17) | |
Article quotation reported, |
22 (31) | |
At least one type of “spin” | 33 (47) | |
Type of “spin” |
No acknowledgment of nonstatistically significant primary outcome | 13 (19) |
Claiming equivalence when results failed to demonstrate a statistically significant difference | 7 (10) | |
Focus on positive secondary outcome | 5 (7) | |
Focus on inappropriate subgroup | 4 (5) | |
Focus on within-group (or over-all within) comparison | 11 (16) | |
Nonstatistically significant outcome reported as if they were significant | 5 (7) | |
Ignored data of safety | 3 (4) | |
Inadequate claim of safety | 5 (7) | |
Inappropriate extrapolation | 6 (9) | |
Other “spin” | 2 (3) |
Numbers do not add up as the types of “spin” were not mutually exclusive.
About half of the press releases (33; 47%) had at least one type of “spin” (
From bivariate analysis (
Characteristics | Subcharacteristics | “Spin” in Press Release |
|
Journal | General | 11/32 (34) | 0.05 |
Specialized | 22/38 (58) | — | |
Funding source | Profit | 15/33 (45) | 0.8 |
Nonprofit or not reported | 18/37 (49) | — | |
Sample size | <112 | 22/35 (63) | 0.008 |
≥112 | 11/35 (31) | — | |
Experimental treatment | Drug | 17/36 (47) | 1.0 |
Other | 16/34 (47) | — | |
All nonstatistically significant | 11/24 (46) | 0.9 | |
Other | 22/46 (48) | — | |
Authors of press release | Press officer | 19/40 (48) | 0.9 |
Other | 14/30 (47) | — | |
“Spin” in abstract conclusion | Yes | 26/28 (93) | <0.001 |
No | 7/42 (17) | — |
For the interpretation based on the full-text scientific articles, for 38 articles (54%), the experimental treatment was considered beneficial, 18 (26%) neutral, and 14 (20%) not beneficial. In contrast, for the interpretation based on press releases, for 55 releases (79%), the experimental treatment was considered beneficial, two (3%) neutral, and 13 (18%) not beneficial. The results were misinterpreted in 22 press releases (31%); for 19 (86%), the assessors overestimated the benefit of the experimental treatment from the press release and for three (14%), they underestimated the benefit of the experimental treatment from the press release.
As shown in
Characteristics | Subcharacteristics | Overestimation of the Benefit of the Experimental Treatment n/Total |
|
Journal | General | 2/32 (6) | <0.001 |
Specialized | 17/38 (45) | ||
Funding source | Profit | 7/33 (21) | 0.3 |
Nonprofit or not reported | 12/37 (32) | ||
Sample size | 16/35 (46) | <0.001 | |
3/35 (9) | |||
Experimental treatment | Drug | 11/36 (31) | 0.5 |
Other | 8/34 (24) | ||
All nonstatistically significant | 10/24 (42) | 0.05 | |
Other | 9/46 (20) | ||
Authors of press release | Press officer | 10/40 (25) | 0.6 |
Other | 9/30 (30) | ||
“Spin” in press releases | Yes | 16/33 (48) | <0.001 |
No | 3/37 (8) |
For a sample of 41 RCTS we retrieved the scientific article, the press release, and any news items. “Spin” was identified in 17 (41%) abstracts, 19 (46%) press releases, and 21 (51%) news items.
Overall, the assessors overestimated the benefit of the experimental treatment from the news for 10 (24%) reports. Factors associated with overestimation of the beneficial effect of the treatment from the news items were small sample size (41% versus 5%,
Our results highlight a tendency for press releases and the associated media coverage of RCTs to place emphasis on the beneficial effects of experimental treatments. This tendency is probably related to the presence of “spin” in conclusions of the scientific article's abstract. This tendency, in conjunction with other well-known biases such as publication bias, selective reporting of outcomes, and lack of external validity, may be responsible for an important gap between the public perception of the beneficial effect and the real effect of the treatment studied.
Previous studies have highlighted the importance of press releases for results communication and dissemination
Unfortunately, as shown in our study, and previous work the quality of media reports is questionable. An assessment of the reporting of medical news in the mainstream media highlighted the inadequate accuracy and balance of the news media in reporting medical science
Of course, press releases are not meant to be condensed versions of scientific papers; they are meant to summarize the most important findings, contextualize these finding for journalists, and provide contact details for authors and quotes. By being condensed, they always lack details that are contained in the papers. The use of “spin” or a particular emphasis could be a way to increase the interest of journalists and subsequent citations in the peer-reviewed literature.
However, this situation becomes problematic if it modifies readers' interpretation of research findings. Our results add to these previous studies by showing the link between the distorted presentation and interpretation of the results in scientific articles and the distorted content and interpretation of press releases. These findings raise the issue of the quality of the peer review process and highlight the importance of this process for disseminating accurate research results.
Our study has several limitations. Firstly, our sample included only published reports of RCTs with a press release indexed in the Eurekalert! database within a 4 mo period, and reported in English; this sample may not be representative of all press releases of RCT results. In fact, half of the press releases selected were written by press officers of medical journals with a high impact factor. Other sources of press releases exist on industry websites, medical journal websites, or other databases for journalists. However, the Eurekalert! database is one of the most important sources of freely available press releases, and most research published on press releases has used this database. Further, there is no reason to believe that the selection of the sample over only 4 mo would bias the results. Secondly, RCTs represent only a small part of the medical literature and the findings may not apply to media reporting of medical or scientific research as a whole. Thirdly, we searched for “spin” only in the article abstract conclusions, not in the entire published article. Consequently, we are not able to determine whether “spin” in the press release was the same as the “spin” in the whole article. We chose the abstract conclusions because it is the most accessible section of an article. Readers often base their initial assessment of a trial on the information reported in an abstract conclusion, and in some geographic areas, the abstract of an RCT report may be all that health professionals have easy access to
In conclusion, previous work showed that exaggerated and inappropriate coverage of research findings in the news media is linked to inappropriate reporting of press releases. Our study adds to these results showing that “spin” in press releases and the news is related to the presence of “spin” in the published article, namely the abstract conclusions. Additionally, our work highlights that this inappropriate reporting could bias readers' interpretation of research results.
Consequently, reviewers and editors of published articles have an important role to play in the dissemination of research findings and should be particularly aware of the need to ensure that the conclusions reported are an appropriate reflection of the trial findings and do not overinterpret or misinterpret the results.
Data abstraction form.
(DOC)
Details related to the method.
(DOC)
Kappa coefficient or agreement percentage for the assessment of “spin” in press releases and in articles.
(DOC)
List of press releases and published articles examined.
(DOC)
Multivariate analysis of factors associated with “spin” in press releases.
(DOC)
Bivariate analysis of factors associated with “spin” in news items (
(DOC)
Bivariate analysis of factors associated with an overestimation of the benefit of the experimental treatment from the news as compared with the interpretation from the article abstract conclusions (
(DOC)
We would like to thank Elodie Perrodeau, Nizar Ahmad, Karima Amazzough, and Guillaume Lonjon for their help in this study.
randomized controlled trial