https://orcid.org/0000-0002-4639-0468
Figures
Abstract
Background
Spin—the misrepresentation of a study’s actual results—has the potential to alter a clinician’s interpretation of the study’s findings and therefore could affect patient care. Studies have shown spin frequently occurs in abstracts of systematic reviews from a variety of other medical disorders and specialties.
Aims
Our primary aim was to evaluate whether the nine most severe types of spin occurred in systematic review abstracts’ concerning diabetic neuropathy treatments. Secondly, we aimed to determine whether spin presence was associated with the methodological quality of a systematic review.
Methods
A search of MEDLINE and Embase collected 1297 articles focused on diabetic neuropathy treatments, of which we included 114 systematic reviews for spin assessment. Each included study was evaluated for the nine most severe types of spin as defined by Yachitz et al. The methodological quality of a systematic review was determined by using the AMSTAR-2 instrument. All screening and data extraction were conducted in a masked, duplicate fashion. Since the final sample size of 114 was not sufficiently powered to do multivariable logistic regression, we calculated unadjusted odds ratios which evaluated relationships between spin presence within abstracts and study characteristics.
Results
From the 114 articles reviewed, spin was present in 7.9% of the studies (9/114), with spin type 5: “conclusion claims the beneficial effect of the experimental treatment despite the high risk of bias in the included primary studies” as the most frequent in our study. Spin types 1, 2, 6, and 8 were not identified. No association was observed between the study characteristics and spin presence, including the methodological quality of a systematic review.
Conclusions
Overall, spin is infrequently observed in abstracts of systematic reviews covering diabetic neuropathy treatments. When comparing our results to other fields of medicine, the field of diabetic neuropathy research publishes systematic reviews whose abstracts mostly portray the findings of the review’s full-text to reflect the results adequately.
Citation: Khan A, Riley H, Ottwell R, Arthur W, Greiner B, Shapiro E, et al. (2022) The presence of spin in systematic reviews focused on diabetic neuropathy: A cross-sectional analysis. PLoS ONE 17(9): e0274744. https://doi.org/10.1371/journal.pone.0274744
Editor: Tim Mathes, University Medical Center Göttingen: Universitatsmedizin Gottingen, GERMANY
Received: July 3, 2021; Accepted: September 5, 2022; Published: September 26, 2022
Copyright: © 2022 Khan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper. Additionally, all raw data forms from the study can be found in the open science frame link (https://osf.io/7e5nd/).
Funding: MV received the award. The development of our protocol was funded by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship Grant. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Nearly 10% of the United States adult population is living with diagnosed diabetes (either type 1 or 2), with an additional 3% estimated to be living with undiagnosed diabetes [1]. Among those living with diabetes, diabetic neuropathy is one of the most common clinical complications, present in over 30% of these individuals [2,3]. Apart from strict glucose control, few therapeutic agents are FDA approved to control the symptoms of diabetic neuropathy [3]. In order to provide the appropriate care to patients with diabetic neuropathy, physicians must stay updated through research and various scientific literature. Systematic reviews and meta-analyses serve as an excellent method for physicians to keep their medical knowledge current.
Systematic reviews are widely recognized as the gold standard for a high level of research evidence and typically represent the most thorough summaries of current literature [4]. Systematic reviews use explicit systematic methods to reduce the risk of bias in the review process, therefore they are considered the highest level of evidence [5]. Given this high standing, they are supposed to report findings free of spin. Yachitz et al. classify spin based on three overarching categories: misleading reporting, misleading interpretation, and inappropriate extrapolation [6]. While the entirety of a scientific work should maintain a high professional standard, there is one section that should be more thoroughly scrutinized—the abstract. As the portion most commonly read by clinicians, the abstract has the greatest potential to alter the course of patient treatment and outcomes [7].
Spin has been identified as a broadly used form of bias in randomized controlled trials within various fields, as well as, specific disease states [8,9]. However, discrepancies have been found between specialties. For example, 70% of otolaryngology abstracts contain spin in comparison to 44.3% of emergency medicine abstracts, and 27.3% of cardiology abstracts [9–11]. Due to the variability in results of past research, further exploration is warranted to examine the presence of spin in additional fields. One unexplored topic is diabetic neuropathy and the presence of spin within treatment options. As diabetic neuropathy is highly prevalent in the US, the evaluation and maintenance of scientific integrity are vital.
The purpose of this study was to analyze the nine most severe types of spin as defined by Yachitz et al. [6] within abstracts of systematic reviews focused on diabetic neuropathy treatment options. Our secondary objective was to determine whether spin was associated with particular characteristics of systematic reviews, including their methodologic quality.
Methods
Protocol and reporting
To ensure our study’s reproducibility and transparency, this study’s protocol and data analysis scripts can be found in Open Science Framework (https://osf.io/7e5nd/) [12]. Additional studies examining the presence of spin in systematic reviews in different medical fields were conducted simultaneously with this study. Additional statisticians repeated analysis to further ensure the reproducibility of our work. We used Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [13], as well as Murad and Wang’s [14] guidelines for the composition of the manuscript.
Search strategy and screening
One of us (DW), a systematic review librarian, systematically searched Embase and MEDLINE to locate systematic reviews limited to the management of diabetic neuropathy treatment (Fig 1). The librarian’s searches were completed on June 2, 2020, with the search returns uploaded to Rayyan [15], a systematic review screening platform. After removing duplicates, in a masked, duplicate fashion, investigators (AK and HR) screened remaining articles by titles and abstracts to determine eligibility. Any discrepancies were noted and resolved by the two researchers through a consensus meeting.
Eligibility criteria
For an article to be included in this study, the following must have been met: (1) the article must be a systematic review with or without a meta-analysis, (2) the review is focused on the management of diabetic neuropathy, (3) the article is retrievable in English, and (4) the article includes human participants. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) definition was used to define a systematic review [16].
Training
An online training course for systematic reviews and meta-analyses by Li and Dickersin [17] was completed by investigators before screening commenced. The investigators also participated in four days of training focused on the definitions and identification of the nine most severe spin types that occur in systematic review abstracts as described by Yavchitz et al. [6] Lastly, investigators completed training over A MeaSurement Tool to Assess Systematic Reviews (AMSTAR-2) [18], an appraisal instrument used to determine the methodological quality of each included systematic review and meta-analysis. A comprehensive outline of author training can be found in our protocol published on Open Science Framework [12].
Spin data extraction
The investigators (AK and HR) individually assessed every included systematic review for the presence of the nine most severe types of spin using a pilot-tested Google form for all data extraction—used previously by our team in another spin study [19]. The definition of these nine spin types can be found in Table 1. Following extraction, the two investigators (AK and HR) reviewed and discussed any inconsistencies within their results. When an agreement could not be reached, an additional author (RO) was available for adjudication.
Data extraction of general characteristics
Additionally, investigators (AK and HR) extracted the following study characteristics from each systematic review: intervention type (pharmacological or non-pharmacological), whether a review discussed compliance with PRISMA [20], whether the associated journal required the compliance with PRISMA, the journal’s 5-year impact factor, the study’s funding source (not funded, industry, not mentioned, private, or public), and date the journals received the systematic reviews. Again, independent data extraction occurred with authors masked to each other’s responses. Following data extraction, authors were unmasked and discrepancies were resolved by group discussion.
AMSTAR-2 rating
The methodological quality of each article was assessed via AMSTAR-2 (https://amstar.ca/), which rates systematic reviews from ‘high’ to ‘critically low’ based on a list of 16 criteria. The investigators (AK and HR) independently evaluated every included article utilizing the AMSTAR-2 tool, entering all data into a Google form. Articles that received different scores from the investigators were reexamined. All conflicts were resolved. The AMSTAR-2 has high reliability and validity compared to other systematic review appraisal tools, such as the original AMSTAR and Sach’s Instrument [18].
Statistical analysis
The study characteristics, spin frequency, and AMSTAR-2 ratings were reported as percentages and counts. In the protocol, we clarified the possibility of a binary logistic regression and assessed a power analysis to determine sample size from a previous study by our team [19]. Since the final sample size of 114 was not sufficiently powered to do multivariable logistic regression, we calculated unadjusted odds ratios which evaluated relationships between spin presence within abstracts and study characteristics. StataCorp 16.1 was used for the analyses [21].
Results
Characteristics
Our searches returned 1,297 articles, with 258 excluded as duplicates. During screening an additional 758 were excluded during abstract screening—578 because they were unrelated to diabetic neuropathy, 96 because they were ineligible by automation tools, 17 because they used animal subjects, and 5 because they were not related to diabetic neuropathy treatment. During the full-text review we excluded an additional 167 articles—68 due to wrong study design, 38 due to no relation to treatment, 18 due to the articles being only published abstracts, 16 due to being unavailable in English, 13 due to no full text being available, 7 due to articles only being protocols, 3 due to using animal studies, 3 due to articles being withdrawn, and 1 due to article being an erratum—resulting in a final sample size of 114 systematic reviews. The exclusion rationale and screening process are illustrated in Fig 2. The primary type of intervention was pharmacological therapies (77/114, 67.5%), followed by non-pharmacological interventions (37/114, 32.5%), and surgical interventions (0/114, 0%). The majority of the included studies did not mention adherence to PRISMA (81/114, 71.1%). Additionally, 74.6% of the journals did not recommend authors adhere to PRISMA guidelines (85/114). Of the 114 systematic reviews, 22 studies did not include a funding statement (21/114, 18.4%). Of the remaining, 77 received funding (67.5%), with public funding being the most common form of funding (44/114, 38.6%). Twenty-one studies did not include a funding statement (21/114, 18.4%), and 16 reported no funding (16/114, 14.04%). The dates the publishing journal received the systematic reviews ranged from 1993 to 2019 (Table 2).
Spin in abstracts
Spin was identified in 9 of the 114 systematic review abstracts pertaining to diabetic neuropathy treatment (7.9%) (Table 1). However, certain abstracts contained multiple types of spin; thus, a total of 12 occurrences of spin were found. The most prevalent spin type was type 5—“conclusion claims the beneficial effect of the experimental treatment despite the high risk of bias in the included primary studies”—which occurred in 5 abstracts (5/114, 4.4%). Spin type 9—“conclusion claims the beneficial effect of the experimental treatment despite reporting bias”—was the second most frequent spin type, present in 3 abstracts (3/114, 2.6%). Nine studies did not mention safety measures in their abstract conclusion; therefore, spin type 4 was only observed in 1 of 105 studies (<1%). Spin types 1, 2, 6, and 8 did not occur in any abstracts. Our logistic regression models showed no significant associations between presence of spin within abstracts and a study’s adherence to PRISMA, intervention type, funding source, nor if the publishing journal recommends PRISMA adherence or its 5-year impact factor (Table 2).
AMSTAR-2 rating
AMSTAR-2 rated 7.9% of studies as ‘high’ quality (9/114), 27.2% as ‘moderate’ quality (31/114), 34.2% as ‘low’ quality (39/114), and 30.7% as ‘critically low’ quality (35/114) (Table 2). Approximately 99.1% of the systematic reviews developed their research question based on the Population, Intervention, Comparator group, Outcome (PICO) method (113/114) (Table 3). Additionally, 86.8% of authors explained the study selection design for inclusion (99/114). In contrast, 7% (7/114) of the systematic reviews used a comprehensive literature search strategy, and 14% (16/114) reported on the sources of funding for their analyzed reviews.
Discussion
General findings
Our study identified five of the nine most severe types of spin in systematic review abstracts involving diabetic neuropathy treatments. The most frequent type of spin was type 5: “conclusion claims the beneficial effect of the experimental treatment despite the high risk of bias in the included primary studies.” One example of spin type 5 is from the abstract of a systematic review covering the efficacy of duloxetine in treating painful neuropathy, chronic pain, or fibromyalgia [22]. In the abstract, the authors claim a daily dose of 60 mg and 120 mg of duloxetine was an effective treatment for individuals with diabetic neuropathic pain. However, an examination of the article’s full text revealed the majority of the primary studies included in this review had either a high risk of bias or the risk of bias was unclear. Omitting the risk of bias assessments from the abstract, as seen in this study, is problematic as studies with a high risk of bias can change the magnitude and direction of the study’s outcomes–studies with a high risk of bias yield greater, and often incorrect, effect sizes as compared to studies with low-risk of bias [23,24]. As healthcare professionals often solely rely on the abstract [7], it becomes apparent a clinician’s interpretation could be distorted. Therefore, we recommend authors of systematic reviews declare whether their included primary studies are at high risk of bias so clinicians can be cautious when interpreting the study’s results.
Investigating spin in medical research has been of growing interest and has been well studied in randomized controlled trials. When comparing our results to other spin studies, spin occurs at a much lower frequency in systematic reviews focused on the management of diabetic neuropathy. The results of this study are encouraging as research over diabetic neuropathy seems to be at lower risk of containing spin compared to other fields. With these encouraging findings, the goal for diabetic neuropathy research should be focused on further eliminating the small amount of spin which occurs and preventing spin from occurring in future studies.
Recommendations
Studies have shown misleading results in the abstract often go unrecognized by peer reviewers. For example, one study found reviewers failing to identify spin in nearly 80% of the studies reviewed [25]. In fact, this same study found 15% of reviewers asked the authors to include some form of spin in their abstract’s conclusion [25]. Based on these findings, we first recommend spin training be provided to reviewers and editors as this would likely increase spin recognition and ultimately lead to its prevention. Additionally, we recommend the establishment of reporting guidelines which directly address spin and its subtypes as current guidelines fail to address spin. By having these guidelines, authors would be able to eliminate spin as it may be unintentionally included in the abstract. Lastly, we recommend clinicians using caution when interpreting the results of a study from only the abstract. Education covering spin may be beneficial to physicians and physicians in training.
Strengths and limitations
Our study’s strengths lie in its transparency, reproducibility, and measures to mitigate the effect of bias. All study materials were uploaded online prior to beginning our study, and any potential deviations from our protocol were recorded in an openly available update on OSF. Additionally, literature was reviewed in a double-masked fashion, the method deemed as the “gold standard” by the Cochrane Collaboration [26]. Limitations of our study include its cross-sectional nature, which reduces its generalizability, and the use of only MEDLINE and Embase to conduct our search, which may have lessened our sample size. Identifying spin is an inherently subjective process. However, to mitigate this subjectivity, investigators underwent intensive training to better characterize and identify spin.
Conclusion
Overall, spin is infrequently observed in abstracts of systematic reviews covering diabetic neuropathy treatments. When comparing our results to other fields of medicine, systematic reviews related to diabetic neuropathy more often accurately portray the findings of the articles’ full-text. Given that nearly 8% of these systematic reviews did contain spin, further reducing spin would lessen potential bias within these articles, which may be beneficial to other researchers, healthcare providers, and their patients.
References
- 1.
National Diabetes Statistics Report Estimates of Diabetes and Its Burden in the United States [Internet]. https://www.cdc.gov/. National Diabetes Statistics Report website; [cited 2021Jun31]. Available from: https://www.cdc.gov/diabetes/pdfs/data/statistics/national-diabetes-statistics-report.pdf.
- 2.
Diabetes and nerve damage [Internet]. Centers for Disease Control and Prevention. Centers for Disease Control and Prevention; 2021 [cited 2021Jun31]. Available from: https://www.cdc.gov/diabetes/library/features/diabetes-nerve-damage.html.
- 3.
Iqbal Z, Azmi S, Yadav R, Ferdousi M, Kumar M, Cuthbertson DJ, et al. Diabetic peripheral neuropathy: Epidemiology, diagnosis, and pharmacotherapy [Internet]. Clinical therapeutics. U.S. National Library of Medicine; 2018 [cited 2021Jun28]. Available from: https://pubmed.ncbi.nlm.nih.gov/29709457/.
- 4.
Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid [Internet]. Evidence-based medicine. BMJ Publishing Group; 2016 [cited 2021Jun27]. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4975798/.
- 5. Impellizzeri FM, Bizzini M. Systematic Review and meta-analysis: A Primer [Internet]. International journal of sports physical therapy. Sports Physical Therapy Section; 2012 [cited 2021May27]. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3474302/.
- 6.
Yavchitz A, Ravaud P, Altman DG, Moher D, Hrobjartsson A, Lasserson T. A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity [Internet]. Journal of clinical epidemiology. U.S. National Library of Medicine; 2016 [cited 2021May25]. Available from: https://pubmed.ncbi.nlm.nih.gov/26845744/.
- 7.
Marcelo A, Gavino A, Isip-Tan IT, Apostol-Nicodemus L, Mesa-Gaerlan FJ, Firaza PN, et al. A comparison of the accuracy of clinical decisions based on full-text articles and on journal abstracts alone: A study among residents in a tertiary care hospital [Internet]. Evidence-based medicine. BMJ Group; 2013 [cited 2021Jun29]. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3607116/.
- 8.
Kinder NC, Weaver MD, Wayant C, Vassar M. Presence of ’spin’ in the abstracts and titles of Anaesthesiology Randomised Controlled Trials [Internet]. British journal of anaesthesia. U.S. National Library of Medicine; 2018 [cited 2021Jun25]. Available from: https://pubmed.ncbi.nlm.nih.gov/30579417/.
- 9.
Reynolds-Vaughn V, Riddle J, Brown J, Schiesel M, Wayant C, Vassar M; Evaluation of spin in the abstracts of emergency medicine randomized controlled trials [Internet]. Annals of emergency medicine. U.S. National Library of Medicine; 2019 [cited 2021May26]. Available from: https://pubmed.ncbi.nlm.nih.gov/31101371/.
- 10.
Cooper CM, Gray HM; Ross AE; Hamilton TA; Bea Downs J; Wayant C; et al; Evaluation of spin in the abstracts of otolaryngology randomized controlled trials [Internet]. The Laryngoscope. U.S. National Library of Medicine; 2018 [cited 2022Mar7]. Available from: https://pubmed.ncbi.nlm.nih.gov/30578543/.
- 11. Roberts WB, Cooper CM, Khattab M; Neff P; Wildes D, Wayant C, et al; Evaluation of "spin" in the abstracts of randomized controlled trial reports in Cardiology [Internet]. The Journal of the American Osteopathic Association. U.S. National Library of Medicine; 2020 [cited 2021May24]. Available from: https://pubmed.ncbi.nlm.nih.gov/32961554/.
- 12. Khan A, Riley H, Ottwell RL, Anderson M. Diabetic neuropathy spin [Internet]. OSF. OSF; 2020 [cited 2022Apr12]. Available from: https://osf.io/7e5nd/.
- 13. Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, et al. Prisma For Abstracts: Reporting Systematic Reviews in journal and conference abstracts [Internet]. PLOS Medicine. Public Library of Science; 2013 [cited 2021Jun26]. Available from: https://journals.plos.org/plosmedicine/article?id=10.1371%2Fjournal.pmed.1001419.
- 14. Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research [Internet]. BMJ Evidence-Based Medicine. Royal Society of Medicine; 2017 [cited 2022Apr12]. Available from: https://ebm.bmj.com/content/22/4/139.
- 15.
Rayyan QCRI [Internet]. Rayyan. [cited 2022Apr12]. Available from: https://rayyan.ai/users/sign_in.
- 16. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement—systematic reviews [Internet]. BioMed Central. BioMed Central; 2015 [cited 2022Apr12]. Available from: https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/2046-4053-4-1.
- 17.
Li T, Dickerson K. Introduction to systematic review and meta-analysis [Internet]. Coursera. John Hopkins University; [cited 2022Apr12]. Available from: https://www.coursera.org/learn/systematic-review.
- 18. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. Amstar 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both [Internet]. The BMJ. British Medical Journal Publishing Group; 2017 [cited 2021Jun21]. Available from: https://www.bmj.com/content/358/bmj.j4008.
- 19.
Ottwell R, Rogers TC, Anderson JM, Johnson A, Vassar M. Evaluation of spin in the abstracts of systematic reviews and meta-analyses focused on the treatment of acne vulgaris: Cross-sectional analysis [Internet]. JMIR Dermatology. JMIR Publications Inc., Toronto, Canada; 2018 [cited 2020Jul14]. Available from: https://derma.jmir.org/2020/1/e16978.
- 20. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: Explanation and elaboration [Internet]. The BMJ. British Medical Journal Publishing Group; 2009 [cited 2022Apr12]. Available from: https://www.bmj.com/content/339/bmj.b2700.
- 21.
Stata. College Station (Tx.): StataCorp; 2007.
- 22. Lunn MPT, Hughes RAC, Wiffen PJ. [Internet]. Duloxetine for treating painful neuropathy, chronic pain or fibromyalgia. 2014 [cited 2022Apr12]. Available from: https://www.cochrane.org/CD007115/NEUROMUSC_duloxetine-treating-painful-neuropathy-chronic-pain-or-fibromyalgia.
- 23.
Fanelli D, Ioannidis J. US studies may overestimate effect sizes in softer research [Internet]. Proceedings of the National Academy of Sciences of the United States of America. U.S. National Library of Medicine; 2013 [cited 2022Apr12]. Available from: https://pubmed.ncbi.nlm.nih.gov/23980165/.
- 24.
Savović J, Turner RM, Mawdsley D, Jones HE, Beynon R, Higgins JPT, et al. Association between risk-of-bias assessments and results of randomized trials in Cochrane Reviews: The robes meta-epidemiologic study [Internet]. UCL Discovery. UCL; 2017 [cited 2022Apr12]. Available from: https://discovery.ucl.ac.uk/id/eprint/10039948/.
- 25. Lazarus C, Haneef R, Ravaud P, Hopewell S, Altman DG, Boutron I; Peer reviewers identified spin in manuscripts of nonrandomized studies assessing therapeutic interventions, but their impact on spin in abstract conclusions was limited [Internet]. Journal of clinical epidemiology. U.S. National Library of Medicine; 2016 [cited 2021Jun26]. Available from: https://pubmed.ncbi.nlm.nih.gov/27164274/.
- 26.
Cumpston M, Li T, Page MJ, Chandler J, Welch VA, Higgins JP, et al. Updated guidance for trusted systematic reviews: A new edition of the Cochrane Handbook for Systematic Reviews of interventions [Internet]. The Cochrane database of systematic reviews. U.S. National Library of Medicine; 2019 [cited 2021Jun12]. Available from: https://pubmed.ncbi.nlm.nih.gov/31643080/.