Recently, concerns have been raised over the potential impacts of commercial relationships on editorial practices in biomedical publishing. Specifically, it has been suggested that certain commercial relationships may make editors more open to publishing articles with author conflicts of interest (aCOI). Using a data set of 128,781 articles published in 159 journals, we evaluated the relationships among commercial publishing practices and reported author conflicts of interest. The 159 journals were grouped according to commercial biases (reprint services, advertising revenue, and ownership by a large commercial publishing firm). 30.6% (39,440) of articles were published in journals showing no evidence of evaluated commercial publishing relationships. 33.9% (43,630) were published in journals accepting advertising and reprint fees; 31.7% (40,887) in journals owned by large publishing firms; 1.2% (1,589) in journals accepting reprint fees only; and 2.5% (3,235) in journals accepting only advertising fees. Journals with commercial relationships were more likely to publish articles with aCOI (9.2% (92/1000) vs. 6.4% (64/1000), p = 0.024). In the multivariate analysis, only a journal’s acceptance of reprint fees served as a significant predictor (OR = 2.81 at 95% CI, 1.5 to 8.6). Shared control estimation was used to evaluate the relationships between commercial publishing practices and aCOI frequency in total and by type. BCa-corrected mean difference effect sizes ranged from -1.0 to 6.1, and confirm findings indicating that accepting reprint fees may constitute the most significant commercial bias. The findings indicate that concerns over the influence of industry advertising in medical journals may be overstated, and that accepting fees for reprints may constitute the largest risk of bias for editorial decision-making.
Citation: Graham SS, Majdik ZP, Clark D, Kessler MM, Hooker TB (2020) Relationships among commercial practices and author conflicts of interest in biomedical publishing. PLoS ONE 15(7): e0236166. https://doi.org/10.1371/journal.pone.0236166
Editor: Omid Beiki, Karolinska Institutet, SWEDEN
Received: February 12, 2020; Accepted: June 30, 2020; Published: July 24, 2020
Copyright: © 2020 Graham et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data are available in a public repository at the following location: 3. Graham, S.S., Majdik, Z., & Clark, D. (2020). Conflicts of Interest: Article XML. [Data file]. Texas Data Repository. DOI: 10.18738/T8/VSWAJY.
Funding: Funding is provided by National Endowment for the Humanities grant HAA-261070. The funder had no role in the analysis or interpretation of data, the writing of this report, or the decision to submit the article for publication.
Competing interests: The authors have declared that no competing interests exist.
For some time now, there has been growing concern about the extent to which financial relationships with industry bias the results of biomedical research. Studies of industry funding and author conflicts of interest (aCOI) in the biomedical sciences have found that these financial relationships can bias choices in experimental design [1–3] as well as clinical decision-making during trial execution [4–6]. In particular, the most recent studies and meta-analyses confirm that these financial relationships and associated practices result in a substantial increase in the likelihood that clinical trial results will be favorable to industry [7–9]. Recent research in these areas also points toward an ever-widening array of potentially biasing practices, including ghost authorship [10–11] and so-called "marketing trials”—i.e., clinical trials that were designed primarily to influence medical decision-making in favor of product use [12–13]. Despite movements toward greater transparency in disclosing aCOI in medical journals, including the International Committee for Medical Journals Editors (ICJME) recommendations for reporting aCOI , inconsistencies still remain in reporting financial and non-financial COI for authors, researchers, and editors. This issue is particularly acute regarding the relative inconsistency and opacity of editorial COI disclosures, a concern that often persists even when author and researcher disclosures become more transparent [15–18].
In addition to worries over personal COI that may be held by journal editors, there are also growing apprehensions over the potential effects of certain commercial publishing practices on biomedical research. Specifically, it has been suggested that journal-level financial relationships such as the acceptance of industry advertising revenue, reprint fees, and additional industry printing contracts held by journal parent companies may impact editorial decision-making, creating an environment more favorable to industry-sponsored research [19–21]. Editor COI and potential commercial publishing biases may be of particular concern given recent fears that non-peer-reviewed publications with aCOI are having significant impacts on biomedical research and clinical practice . Certainly, available anecdotal evidence does suggest there may be cause for concern [23,24]. Two of the most notable cases involve the punitive withdrawal of $1.5 million in advertising revenue from the Annals of Internal Medicine following the publication of an article critiquing multiple industry-funded trials in 1992  and Merck’s dispersal of $836,000 to the New England Journal of Medicine for reprints of the VIGOR study as a part of the Vioxx marketing campaign . Furthermore, there are some data available indicating that commercial publishing biases may lead to editors being less diligent in the execution of journal aCOI policies for article authors [19–20]. As greater attention is paid to the potential adverse consequences of aCOI and industry funding on medical research, it is critical that ongoing discussions regarding potential commercial publishing biases occur in an evidence-rich environment.
Accordingly, this study evaluates the relationships between potential commercial publishing biases and industry favorability (as measured by aCOI likelihood and frequency) in 159 biomedical journals. Research in a variety of subspecialties has demonstrated that aCOI frequently associates with results favorable to industry [26–31], and more recently an analysis across clinical subspecialties indicates aCOI predicts that research will be 2.94 times more likely to return favorable results  Subsequently aCOI can serve as an effective surrogate endpoint for measuring industry favorability more broadly. In what follows, we describe our development of a machine-learning framework for identifying and classifying aCOI. We then compare aCOI likelihood and frequency in journal samples stratified by identified commercial relationships. The results show that the presence of some commercial relationships does appear to create an environment more favorable to scholarship with aCOI, increasing the likelihood that published articles will have aCOI as well as the number of aCOI per article. In particular, the acceptance of reprint orders appears to be the most influential of evaluated potential commercial biases.
In order to enhance the available evidence base regarding the potential influence of commercial publishing practices on editorial decision-making, we collected 128,781 biomedical journal articles indexed with Medline. In 2016 Medline began collecting conflicts of interest information from participating journals. We extracted data for analysis in January 2019, and at that time, approximately 30 million articles were indexed in the database. The population of 128,781 articles was identified first by extracting all MEDLINE-indexed articles (2016–2018) with aCOI disclosure statements (N = 274,246). These articles were published in a total of 1497 journals. Our final sample of 128,781 articles in 159 journals was derived by excluding all articles where the publishing journal was present fewer than 25 times in the full dataset. We evaluated the presence and rate of aCOI across all articles in the dataset using a custom-built automated parser and compared aCOI likelihood and quantity to suggested measures of commercial biases. In what follows, we describe our approaches to 1) aCOI identification and classification, 2) evaluating the reliability of the aCOI parser, and 3) identifying the presence or absence of commercial publishing practices in each journal.
Author COI identification & classification
In order to identify and classify each of the reported aCOIs in these disclosure statements, we developed a metadata assisted, machine-learning enhanced, natural language processing (NLP) tool. In short, the parser uses a trained language model to tag sponsors (e.g., pharmaceutical companies). The parser then uses Medline author metadata to identify named authors in the disclosure statements, matches authors to sponsors, and finally identifies the type of conflict disclosures. Each of these parser stages are described in more detail below.
An NLP method called Named Entity Recognition (NER) uses grammatical and/or statistical techniques to extract and classify entities like persons, locations, dates, or organizations from unstructured text. For example, a sentence such as “Walter Sandulli and Jessica Goldenberg are employees of Akrimax,” when parsed, would produce three “named entities”: Walter Sandulli, PERSON; Jessica Goldenberg, PERSON; and Akrimax, ORG. NER approaches can work accurately on unknown texts, and can achieve high levels of precision when trained using a machine learning approach. But in the case of disclosure statements, the lack of consistent styling in the writing and editing of COI statements means that organization names are presented very differently, sometimes within the same COI statement (e.g., GlaxoSmithKline vs. Glaxo vs. GSK). COI statements are similarly inconsistent in presenting author names; often they use initials, but sometimes last names or other abbreviations will be present. These inconsistencies, coupled with the fact that pharmaceutical company names often resemble proper names, can challenge an out-of-the-box NER model. Using a basic English language model trained on a small sample of human-corrected COI statements (n = 100), we were able to decrease the sponsor identification error rate by 68% compared to the default model.
Our approach used MEDLINE data on author names to further increase recognition accuracy for both author names and organizations. In light of the author naming conventions described above, as well as the fact that organizations in the biomedical field often have names that, to a computer, resemble human names (e.g., the “Smith Kline” of GlaxoSmithKline), automated NER parsing will frequently mischaracterize organizations as names, and vice versa. To counteract this issue, the parser uses author metadata to generate an author-name permutation table with 13 name permutations that correspond to author naming conventions from various journal style guides for disclosure statements. “Jane Alicia Doe,” for example, would be rendered as “J.A.D.,” “J. Doe,” “J Doe,” and ten other permutations of first, middle, and last name and/or initials. Using this metadata-generated list of author permutations instead of relying on the NER to tag both authors and organizations allowed us to not only have a high degree of precision in identifying authors in COI statements, but also to cross-check them against entities tagged as organizations and remove them if they were in the author list.
The aCOI classification dictionary is based loosely on the International Committee of Medical Journal Editors (ICMJE) standardized conflicts of interest disclosure form. Our COI dictionary schema organizes these categories (as well as employment in industry) into a three-level schema based on potential benefit from a product’s success. Specifically, low-level aCOI included personal fees, travel, board memberships, and non-financial support. Mid-level aCOI included grants and research support. Finally, high-level aCOI included stock ownership and employment in industry. The parser assumes a standard syntax that almost all COI disclosure statements follow, where a name (or names) are followed by an aCOI disclosure type (like “is employed by”), which is followed by the aCOI source. The parser extracts aCOI value(s) from each disclosure statement by stitching the three elements described above—NER, author permutations, aCOI classifications—together through a regular expression. This process is repeated for each tagged sponsor in a disclosure statement. Outputs are collated and assigned a numerical weight based on the aCOI classification dictionary. Table 1 provides an example of a fully parsed disclosure statement.
In order to evaluate the reliability of the aCOI parser, a random sample of 1000 disclosure statements was submitted to human evaluation. While the dataset includes 128,781 disclosure statements, the results of our analysis indicate that approximately 94% of these are some version of "The authors report no conflicts of interest." A truly representative sample of 1000 disclosure statements would thus only provide 60 statements for the human or parser to evaluate. Therefore, our sampling protocol excluded disclosure statements of fewer than 50 characters (i.e., those more likely to be some variation of “The authors report no conflicts of interest,” which is 44 characters long. The end result of this approach is that we oversampled disclosure statements where aCOIs were more likely to be present.
In order to compare the human-coded and machine-coded samples, we assessed reliability using the two-way average measure Intra-Class Correlation Coefficient (ICC). The average ICC for low-level conflicts was 0.722, with a 95% confidence interval from 0.69 to 0.751 (F[998,903 [= 6.27, p < .01). The average ICC for medium weight conflicts was 0.773, with a 95% confidence level from 0.747 to 0.797 (F[998,985] = 7.84, p < .01). And, finally, the average ICC for high-level conflicts was 0.618, with a 95% confidence level from 0.578 to 0.656 (F[998,923] = 4.28, p < .001).
Recommendations for appropriate ICC thresholds vary somewhat across disciplines and contexts. The threshold of “low” agreement can be from below ICC = 0.40  to ICC = 0.50 . Fair to moderate agreement thresholds vary the most with recommend ranges from ICC = 0.40 to ICC = 0.75 . Most ICC schemata accept ICC > 0.60 as fair to good and ICC > 0.75 as good to excellent. Since identifying the absence of conflicts is an easier computational task than conflict classification, our approach here invariably resulted in lower ICC scores than would be expected in a truly representative sample. However, the benefit of this approach is that it ensured the parser evaluation would involve a much wider variety of conflict types. Nevertheless, parser reliability scores generally fell within ranges that would be classified as moderate to good.
Commercial relationships identification
Potential sources of commercial bias were identified based on the extant literature. Research and opinion pieces published in biomedical journals regularly identify the acceptance of advertising revenue, the acceptance of reprint contracts, and the parent company’s acceptance of industry publishing contracts (e.g. supplements) as potential sources of editorial bias [20,36]. Therefore, we reviewed journal websites for solicitations of adverting revenue and reprint fees. Additionally, for each journal, the parent company of the journal was identified. This information is typically available in a website header or footer and/or on the “About” page. If the parent company was primarily a publishing firm (e.g. Elsevier, Taylor and Francis, Wiley), the journal was assigned to the large publishing firm category. Every journal in the dataset that was owned by a large publishing firm accepted adverting and reprint fees. Thus we were able to assign each journal to one of the following categories: 1) control group (accepts no advertising, reprint fees, not owned by commercial publishing group); 2) accepts advertising revenue, but not reprint fees, 3) accepts reprint fees, but not adverting revenue, 3) accepts both advertising and reprint fees, but is not owned by a large commercial publisher, and 4) owned by a commercial publishing firm.
We evaluated aCOI rates for 128,781 articles published in 159 journals indexed by MEDLINE. Each journal in the dataset included at least 25 articles, with PLoS One having the most at 22,252 articles. By group, the dataset included 43,630 articles in journals accepting advertising and reprint fees, but not belonging to a commercial publishing firm; 40,887 articles in journals owned by large publishing firms; 1,589 in journals accepting reprint fees but not advertising fees; 3,235 in journals accepting only advertising fees; and 39,440 articles in the control group. Table 2 details these numbers alongside aCOI rates.
aCOI frequency analysis
An initial test for equality of proportions (using Yates' continuity correction) was conducted to assess if articles with aCOI were more likely to be published in journals with potential commercial biases. In order to ensure adequate statistical power (β = .9) for the test, a random sample of 1,000 articles was selected from each set of journals (with vs. without commercial biases). Journals with potential commercial biases published articles with aCOI at a rate of 9.2% whereas those without potential commercial biases published articles with aCOI at a rate of 6.4%. This difference is significant [χ2 (1, N = 2000) = 5.07, p = 0.024].
In order to evaluate whether some commercial relationships were more predictive of these frequency differences, we identified the rate at which articles with aCOI were published in each journal in the data set. These data were fitted to a quasi-binomial multiple regression model with a logit link. Overall, the model was significant at F(23.33, 2496) = 3.51, p = 0.017. Neither advertising revenue nor ownership by a large commercial publishing firm were significant predictors of aCOI likelihood. However, a journal’s acceptance of reprints predicts that the likelihood that a published article will have aCOI increases by a factor of 2.81(95% CI, 1.5 to 8.6, p = 0.0416).
aCOI rates analysis
Shared-control estimation plots were used to compare aCOI rates across journal categories. Shared-control estimation plots are part of the estimation statistics framework recently promulgated as a robust alternative to null-hypothesis significance testing . Estimation plots focus analytic attention on population parameters, mean differences, and effect sizes over p-values. The approach here uses Efron's technique for bias-corrected accelerated bootstrap (BCa) estimation to account for skewed populations . Using a stratified sample of articles in each group, we ran 5,000 BCa iterations at the 95% confidence level in order to derive the effect estimates reported in Table 3.
The results of the estimation plots (Fig 1) indicate that certain commercial practices have a modest effect on the number of aCOI per article. Of course, these effects are not equal across categories or measures. In terms of total aCOI, the effect size is so modest as to be negligible for journals accepting either reprints or ad revenue. However, for journals accepting both reprints and ad revenue the effect size would functionally double the aCOI rate for the average article, whereas ownership by a large commercial publishing firm would increase the aCOI rate of the average article by approximately 60%. However, when aCOI are separated by weight, a more complex picture emerges. Among the aCOI-level specific plots, the largest effect sizes appear on the plot for low-level aCOIs. Interestingly, all the effects are negative on the plot for high-level aCOIs. However, the findings for high-level aCOI should be interpreted cautiously given the more moderate inter-rater agreement rates. 
Nevertheless, certain patterns emerge when looking across tests. The data suggest there may be an aggregation effect whereby an increased number of commercial relationships may result in a greater willingness to publish articles with higher aCOI rates. In each of the tests for total aCOI, low-level aCOI, and mid-level aCOI, journals that accept both ad revenue and reprints account for the greatest effects. A striking finding from this study is that in three of the four tests, the effect of advertising revenue, in isolation, is negative. In the remaining case, the effect size is negligible.
Editorial decision-making is a complex matter driven by a multitude of competing factors, many of which are not accounted for in the literature on the potential impacts of commercial publishing relationships. As biomedical publishing works to address these impacts within the context of broader concerns over COI and industry funding, it is imperative that new and revised policies are reflective of the best evidence available. The potential dangers of adopting new COI policies in the context of a dearth of evidence has become clear following the discovery of unanticipated and pernicious effects of COI disclosure [40,41]. Disclosure statements have been shown to cause audiences to extend more trust to those holding conflicts of interest as disclosure provides an opportunity to display both honesty and expertise. Conflict disclosure can also lead to "moral licensing," a phenomenon whereby those who disclose conflicts become unduly confident in their objectivity because transparency obligations have been fulfilled. In order to mitigate the risks of such unanticipated consequences in future policy proposals, recommendations must be based on a solid evidentiary foundation.
The results presented in this article work toward that end with respect to addressing potential commercial biases in biomedical publishing. One additional benefit of this study, compared to many others in the area, is that it includes non-clinical trial publications. As mentioned above, perspectives, comments, opinion pieces, and recruited articles are often selected solely on the basis of editorial discretion. As such, they may be especially open to commercial publishing biases. Additionally, the available data suggests that readers of biomedical journals are not always sensitive to the differences in peer- or editorially revised content. As such, these non-peer-reviewed publications may be exercising undue influence on practitioner understandings on the state of medical science.
Ultimately, the results presented here indicate that the presence of commercial publishing relationships predicts increases in industry favorability as measured by aCOI frequency and quantity. In particular, the data indicate that accepting reprint fees increases the likelihood that any given article published in a journal will have reported aCOI by a factor of 2.81. Additionally, these data show modest effects on the average quantity of aCOI in conflicted articles. That is, when journals accept both advertising revenue and reprint fees or belong to large commercial publishing firms, we see a modest increase in the average total aCOI per article. Interestingly, however, the results of the aCOI quantity analysis indicate that accepting advertising revenue, in isolation, has a modest negative effect on average aCOI per article. Finally, the data indicate that commercial publishing biases have a negligible, but negative, effect on average number of high-level aCOI per article.
Even though advertising revenue has been subject to the greatest scrutiny in the literature, it may represent the lowest cause for concern among the commercial biases evaluated in this study. This may indicate that something like the journalistic invisible wall is functioning appropriately in biomedical journals. Ultimately, these data indicate that the acceptance of fees for reprints may be the most impactful on commercial bias. In some respects, this makes sense. The potential for reprint revenue is the bias most directly tied to editorial decision-making. That is, the choice to publish a study favorable to industry, especially when that study might suggest new or expanded use of a drug, can be directly traced to reprint revenue.
Despite the suggestive nature of these findings, additional research should be conducted to verify and extend results. Specifically, future studies might further validate aCOI as an effective surrogate endpoint for industry favorability. While a number of studies do indicate that it can be used as such, research design and findings are not entirely uniform [42–44]. However, it has been suggested that some studies that do not indicate that aCOI predicts results favorable to industry may be underpowered . Consequently, one important limitation of this study comes from low participation in Medline’s aCOI reporting program among many of the world’s top medical journals. Indeed, among the ten highest h-index medical journals, only one (BMJ) reports aCOI to Medline. Despite the above-mentioned limitations of disclosure statements, the availability of aCOI data has a real impact on our ability to evaluate potential risks. It would be helpful if more high-profile journals participated in Medline’s program. In the absence of such participation, supplementary research that collects data directly from target publications may be in order.
Additionally, further research should be conducted with respect to the impacts of commercial publishing relationships on other markers of editorial decision making. For example, researchers might take inspiration from the recent study identifying the prevalence of marketing trials across journals . Replicating this study with a data set stratified across journals representing a range of commercial biases would further add to the evidentiary foundation necessary to develop sound policies on commercial biases. New research might also use data on the prevalence of ghost authorship or improperly reported aCOI across journals to evaluate associations with commercial biases. In the meantime, the results presented here suggest that, as these data are being curated, attention should probably be focused on commercial publishing biases that can be tied most directly to editorial decision-making, specifically the collection of reprint revenues.
- 1. Lexchin J. Those who have the gold make the evidence: how the pharmaceutical industry biases the outcomes of clinical trials of medications. Sci Eng Ethics. 2012;18(2):247–261. pmid:21327723
- 2. Lexchin J. 2012b. Sponsorship bias in clinical research. Int J Risk Saf Med. 2012;24(4):233–242. pmid:23135338
- 3. Sismondo S. How pharmaceutical industry funding affects trial outcomes: causal structures and responses. Soc Sci Med. 2008;66(9):1909–1914. pmid:18299169
- 4. Lundh A, Krogsbøll LT, Gøtzsche PC. Sponsors' participation in conduct and reporting of industry trials: a descriptive study. Trials. 2012;13:146. pmid:22920226
- 5. Rochon PA, Sekeres M, Hoey J, Lexchin J, Ferris LE, Moher D, et al. Investigator experiences with financial conflicts of interest in clinical trials. Trials. 2011 Dec;12(1):9.
- 6. Sismondo S. Ghost management: how much of the medical literature is shaped behind the scenes by the pharmaceutical industry? PLoS Med. 2007;4(9):e286. pmid:17896859
- 7. Ahn R, Woodbridge A, Abraham A, Saba S, Korenstein D, Madden E, et al. Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study. BMJ, 2017;356:i6770. pmid:28096109
- 8. Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2017;2: MR000033. pmid:28207928
- 9. Flacco ME, Manzoli L, Boccia S, Capasso L, Aleksovska K, Rosso A, et al. 2015. Head-to-head randomized trials are mostly industry sponsored and almost always favor the industry sponsor. J Clin Epidemiol. 2015;68(7):811–820. pmid:25748073
- 10. Matheson A. Ghostwriting: the importance of definition and its place in contemporary drug marketing. BMJ, 2016;354:i4578. pmid:27576493
- 11. DeTora LM, Carey MA, Toroser D, Baum EZ. Ghostwriting in biomedicine: a review of the published literature. Current medical research and opinion. 2019 Sep 2;35(9):1643–51. pmid:30986084
- 12. Matheson A. Marketing trials, marketing tricks—how to spot them and how to stop them. Trials. 2017;18(1):105. pmid:28270221
- 13. Barbour V, Burch D, Godlee F, Heneghan C, Lehman R, Perera R, et al. Characterisation of trials where marketing purposes have been influential in study design: a descriptive study. Trials. 2016;17(1):31.
International Committee of Medical Journal Editors (ICJME). Conflicts of interest. 2019. http://www.icmje.org/conflicts-of-interest/.
- 15. Shawwa K, Kallas R, Koujanian S, Agarwal A, Neumann I, Alexander P, et al. Requirements of clinical journals for authors’ disclosure of financial and non-financial conflicts of interest: a cross sectional study. PLoS One. 2016;11(3):e0152301. pmid:27030966
- 16. Liu JJ, Bell CM, Matelski JJ, Detsky AS, Cram P. Payments by US pharmaceutical and medical device manufacturers to US medical journal editors: retrospective observational study. BMJ. 2017;359:j4619. pmid:29074628
- 17. Mehlman CT, Okike K, Bhandari M, Kocher MS. Potential financial conflict of interest among physician editorial board members of orthopaedic surgery journals. JBJS. 2017;99(5)Lp.e19.
- 18. Janssen SJ, Bredenoord AL, Dhert W, de Kleuver M, Oner FC, Verlaan JJ. Potential conflicts of interest of editorial board members from five leading spine journals. PloS One. 2015;10(6):e0127362. pmid:26042410
- 19. Easley TJ, Medical journals, publishers, and conflict of interest. JAMA. 2017;317(17):1759–1760. pmid:28464164
- 20. Lexchin J, Light DW. Commercial influence and the content of medical journals. BMJ. 2006;332(7555):1444–1447. pmid:16777891
- 21. Handel AE, Patel SV, Pakpoor J, Ebers GC, Goldacre B, Ramagopalan SV. 2012. High reprint orders in medical journals and pharmaceutical industry funding: case-control study. BMJ. 2012:344:e4212. pmid:22745328
- 22. Fontanarosa P, Bauchner H. Conflict of interest and medical journals. JAMA. 2017;317(17):1768–1771. pmid:28464125
- 23. Dyer O. Journal rejects article after objections from marketing department. BMJ. 2004;328:244
- 24. Tsai AC. Conflicts between commercial and scientific interests in pharmaceutical advertising for medical journals. Int J Health Serv. 2003;33(4):751–768. pmid:14758858
- 25. Smith R. Lapses at the New England Journal of Medicine. J Res Soc Med. 2006;99(8):380–382.
- 26. Amiri AR, Kanesalingam K, Cro S, Casey AT. Does source of funding and conflict of interest influence the outcome and quality of spinal research? Spine 2014;356:308–14.
- 27. Perlis RH, Perlis CS, Wu Y, Hwang C, Joseph M, Nierenberg AA. Industry sponsorship and financial conflict of interest in the reporting of clinical trials in psychiatry. Am J Psychiatry 2005;356:1957–60.
- 28. Perlis CS, Harwood M, Perlis RH. Extent and impact of industry sponsorship conflicts of interest in dermatology research. J Am Acad Dermatol 2005;356:967–71.
- 29. Riechelmann RP, Wang L, O’Carroll A, Krzyzanowska MK. Disclosure of conflicts of interest by authors of clinical trials and editorials in oncology. J Clin Oncol 2007;356:4642–7.
- 30. Friedman LS, Richter ED. Relationship between conflicts of interest and research results. J Gen Intern Med 2004;356:51–6.
- 31. Tibau A, Bedard PL, Srikanthan A, Ethier JL, Vera-Badillo FE, Templeton AJ, et al. Author financial conflicts of interest, industry funding, and clinical practice guidelines for anticancer drugs. J Clin Oncol. 2015 Jan 1;33(1):100–6. pmid:25385736
- 32. Ahn R, Woodbridge A, Abraham A, Saba S, Korenstein D, Madden E,et al. Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study. BMJ. 2017 Jan 17;356.
- 33. Koo TK, Li MY. 2016. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163. pmid:27330520
- 34. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol. Assess. 1994;6(4):284.
Fleiss JL. Design and analysis of clinical experiments. 2011;John Wiley & Sons.
- 36. Lundh A, Barbateskovic M, Hróbjartsson A, Gøtzsche PC. Correction: Conflicts of Interest at Medical Journals: The Influence of Industry-Supported Randomised Trials on Journal Impact Factors and Revenue–Cohort Study. PLoS medicine. 2011 Feb;8(2).
- 37. Ho J, Tumkaya T, Aryal S, Choi H, Claridge-Chang A. 2019. Moving beyond P values: data analysis with estimation graphics. Nat Methods. 201;16,:565–566 pmid:31217592
Efron B, Tibshirani RJ. An introduction to the bootstrap. 1994;CRC press.
- 39. Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8(1):23. pmid:22833776
- 40. Cain DM, Loewenstein G, Moore DA. The dirt on coming clean: Perverse effects of disclosing conflicts of interest. J Leg Stud. 2005;34(1):1–25.
- 41. Loewenstein G, Cain DM, Sah S. The limits of transparency: Pitfalls and potential of disclosing conflicts of interest. Am Econ Rev. 2011;101(3):423–28.
- 42. Aneja A, Esquitin R, Shah K, et al. Authors’ self-declared financial conflicts of interest do not impact the results of major cardiovascular trials. J Am Coll Cardiol 2013;356:1137–43.
- 43. Pang WK, Yeter KC, Torralba KD, Spencer HJ, Khan NA. Financial conflicts of interest and their association with outcome and quality of fibromyalgia drug therapy randomized controlled trials. Int J Rheum Dis 2015;356:606–15.
- 44. Khan NA, Nguyen CL, Khawar T, Spencer H, Torralba KD. Association of author’s financial conflict of interest with characteristics and outcome of rheumatoid arthritis randomized controlled trials. Rheumatology. 2019 May 1;58(5):776–85. pmid:30535001