Background

Animal research plays a crucial role in advancing our knowledge about biological processes and in developing safe and effective therapies, such as antibiotics, insulin, vaccines and organ transplantation [1, 2] [3]. To realize a broad knowledge gain for the wider scientific community and interested public, the results of animal studies need to be published via peer-reviewed journals or other accessible dissemination formats.

However, recent studies have indicated that a substantial proportion of completed animal studies do not publish their results. According to a Dutch survey conducted in 2012, animal researchers thought that approximately 50% of animal experiments are not published [4]. A tracking activity for studies reported in animal research abstracts presented at the 2008 Society of Critical Care Medicine Conference indicated that 38% had not been published by 2017 [5]. An assessment of publication bias in systematic reviews of interventions tested in animal studies of acute ischaemic stroke estimated a non-publication rate of 14% [6]. All of the abovementioned studies, however, applied assessment measures with limited reliability: i) Surveys assess only subjective opinions; ii) sampling conference abstracts as a starting point for assessing publication rates might already imply a bias towards desired outcomes; and iii) statistical and graphical methods to detect bias in meta-analyses, e.g., funnel-plots, bear several risks regarding misleading information [7].

Despite the methodological limitations of the existing assessments of publication rates, opinions regarding publication bias in animal research play an important role in current debates on, for example, the “reproducibility crisis” [8, 9] and “failure rates in clinical research” [10, 11]. A more thorough approach for assessing publication rates would consist of a follow-up of the full sample of protocols for all animal studies conducted at academic research institutions. Several of these studies exist for clinical research [12], but we are not aware of any study following-up animal studies from protocol to publication. German law requires that all animal studies be approved by a governmental body. The approved study protocols, together with amendments, are archived at the site of the regulatory body and at the respective animal research facility. Animal research facilities at German UMCs (university medical centres) provide quality assurance and training for those involved in animal research.

We aim to conduct the first study following-up a representative sample of animal studies from protocol to publication.

Material and methods

The study protocol and amendments provide detailed information on methods and were pre-registered at https://osf.io/az7mt/. In the following, we report the key elements of our methods.

Results

Publication rates

The overall publication rate was 67%. That is, for 106 of the 158 animal studies, we found at least one results publication (journal articles or doctoral theses) for at least one experiment that formed part of the approved protocol. Excluding doctoral theses as results publications, the publication rate decreased to 58%. For studies with publication, the number of publications per protocol ranged from 1 to 5, with a mean of 1.4. The mean time from the end of the study approval to publication was 209 days (median 169). Interrater reliability for the publication search was 78%, i.e., in 78% of all 210 follow-up cases, the raters identified the same publications initially. All other cases were later resolved through discussion (final interrater agreement = 100%). Raw data are provided in S1 Table.

Stratification by year (end of authorization) showed no clear trend for an increase or decrease in publication rates over time (see Table 1). There was also no substantial difference (>10%) in publication with regard to i) the two research sites, ii) the animal type (rodents vs. non-rodents), iii) the scope of research (basic vs. preclinical), or iv) the discipline of the applicant (physicians/dentists vs. life scientists); see Table 1 for detailed information. Comparing proposals by sample size (approved number of animals, n<207 vs. n>207), we found that larger studies were published more often than smaller studies (73% vs. 61%). We chose n = 207 because this number was the median for the number of animals across all 158 protocols.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Publication rates for the full sample.

https://doi.org/10.1371/journal.pone.0223758.t001

As mentioned in the Methods section, we excluded from the analysis those proposals for which their performance was uncertain (n = 52). These excluded proposals had a substantially lower publication rate (46%), indicating that some of these studies were indeed not conducted.

Discussion

This study is the first to assess the rate of results publications in animal research via a follow-up of archived animal study protocols. From the 210 studies conducted at one of two German UMCs, we identified 158 as certainly having been completed. The publication rate for the certainly completed studies was 67%. Excluding theses archived at the respective university libraries, the publication rate decreased to 58%.

In this study, we did not assess why the other 33% of animal studies did not disseminate results. We address this question in an ongoing study. We therefore do not know, for example, whether non-published studies more often had inconclusive results, involved intellectual property issues, or differed in any other way from published results.

However, the non-publication of results from completed animal studies poses several ethical and practical challenges, regardless of the underlying reasons. If animal studies that were approved because of their potential for knowledge gain do not disseminate their results even years after their completion, then their preconditioned knowledge gain for a wider scientific community is highly questionable. In addition to this foundational ethical challenge, the non-publication of results from animal studies poses several further challenges. First, publicly funded animal studies that do not disseminate their knowledge gain waste public resources. Second, if some research questions have already been studied in animals but the results were not reported to the scientific community, then other researchers might conduct the same or very similar studies with further animals or they might include experimental groups that would not be needed in light of previous studies. Non-publication thus contradicts the 3R principle of reduction and further challenges the refinement principles, which could be realized by advancing experimental planning through the integration of previous findings. Third, increasing evidence shows that published results are biased towards study results showing the desired outcome direction. For example, Emily Sena et al. showed that publication bias in reports of animal stroke studies leads to a major overstatement of efficacy [6]. In recent years, several experts, especially from the pharmaceutical industry, have expressed their concerns that this publication bias might contribute to the high failure rate of clinical trials [10, 11]. In light of the major impact of publication bias in animal research, all involved stakeholders should be aware of those implications and should make efforts to antagonize them.How can we improve publication rates in animal research, and what are the related barriers? Properly addressing these questions falls beyond the scope of this paper. In the following, we highlight some core issues that need further discussion and activities by the relevant stakeholders that fund, regulate, host, or otherwise have an impact on animal research. First, results dissemination should become a professional standard for animal research. International guidelines and laws for research with humans, such as the Declaration of Helsinki, already include results dissemination as a core requirement. More recently, internationally leading funders of biomedical research, such as the Wellcome Trust or the Horizon 2020 programme of the European Commission, have published new policies for clinical research, mandating results dissemination as a new requirement for funding [17, 18]. In addition to policy makers and funders, academic institutions must do their part as well [9]. With regard to informing early clinical trials, the challenges go beyond complete results publications in scientific journals as we pointed out in a previous work [19]. Most investigator brochures for phase I or II clinical trials primarily cite internal reports of preclinical studies as evidence base for starting a clinical trial. Preregistration of preclinical studies could provide a helpful tool to prevent selective outcome reporting and facilitate a more balanced decision making [20].

Several dysfunctional incentives for non-publication might erect one important roadblock for the unbiased results publication of animal (and clinical) research. Academic institutions, for example, reward researchers and appoint professorships mostly for publications in journals with high impact factors and for the extent of secured third-party grants. However, it is difficult to publish the results of animal studies that are inconclusive or that did not confirm the hypothesis (“negative studies”) in journals with high impact factors. Even worse, publications in journals with low impact factors reduce a researcher’s average impact factor, which is one reason why some researchers might even actively refrain from publishing inconclusive or “negative” results [21]. From the technical perspective, however, the publication of inconclusive or “negative” results is no longer a challenge. There are many journals that explicitly publish all studies, regardless of the direction of their results. In addition to standard peer-reviewed journals, new tools for results dissemination exist via preprint servers (e.g., bioRxiv) or repositories (e.g., OSF (Open Science Framework)).

Further research should validate and contextualize our results. Other national and international animal research facilities and regulatory authorities should assess the publication rates for representative samples of completed animal studies. For countries that do not have centralized archives of animal study protocols, such tracking efforts are more difficult to realize. Recent developments such as the launch of animal study registries might improve transparency by providing more fine-grained information on animal studies, which, in turn, will allow more effective and efficient tracking activities, as they are common in clinical research (www.animalstudyregistry.org or www.preclinicaltrials.eu). Baseline data for reporting rates are also important for monitoring whether reporting in animal research improves and whether new policies such as animal study registries or new institutional and funding policies at least correlate with such improvements. Further research is also needed to better understand the reasons and potential barriers behind the non-reporting of results and whether non-reported results differ significantly and substantially from reported results. Finally, analyses on the design (PREPARE (Planning Research and Experimental Procedures on Animals: Recommendations for Excellence) guidelines), reporting quality (e.g., adherence to the ARRIVE guidelines) and consistency between the statistical analyses described in the protocol and in the publication (“p-hacking”) are important to complement the picture of reporting practices in animal research. In summary, more evidence generated via meta-research on reporting practices will facilitate an important, practice-oriented discussion on how to increase value and reduce waste in animal research.

Supporting information