Publication pattern.

To assess the publication pattern of the 490 active interventional trials we conducted a publication search two and a half years after our initial data was collected. We identified a total of 139 trials (28,4%) which already published results after 2 years. 84,2% of those publications were already peer-reviewed and 15,8% were only available as preprint articles. Of those 139 trials only 18,7% registration records provided a link or a summary of those results on their original record on their respective source register. 3 (2,2%) trials have submitted a summary of their results to their registry, but those have not yet been published. Detailed results are shown in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Publication pattern per register.

https://doi.org/10.1371/journal.pone.0289193.t004

Additionally, we graphically displayed the time to publication calculated from the date of registration to the date the publications were first published. Peer-reviewed articles were published after a mean time of 11,69 months (SD: 7,78), a median time of 10,59 months, the 25 percentile was 4,66 months and the 75 percentile was 17,59 months. Preprint articles were published after a mean time of 4,12 months (SD: 2,53), a median time of 3,89 months, the 25 percentile was 2,1 months and the 75 percentile was 6,2 months (Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Box-whiskers-plot of the time-to-publication divided into peer-reviewed and preprint articles.

X-axis: Type of publication divided into peer-reviewed and preprint articles, Y-axis: Time to publication expressed in months calculated as the period from the date of registration to the date of first publication. The publication search was conducted by entering each individual trial-ID into 4 major databases: LitCOVID, CochraneLibrary, MedRxiv and preVIEW:COVID-19. The date of registration was extracted from the original database.

https://doi.org/10.1371/journal.pone.0289193.g004

Assessment of transparency.

To assess the transparency of trial registration we excluded all trials which were found as discontinued in the follow-up search (n = 103). The discontinued trials were analyzed separately for their reason of discontinuation and their therapeutic agents in the third objective. This resulted in 490 included trials and only 4 registers with more than 10 registrations: ChiCTR (n = 254), ClinicalTrials.gov (n = 172), IRCT (n = 37) and EU-CTR (n = 13). The rest of 14 trials will be summarized as others. The results for this assessment are displayed in Table 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Assessment of transparency–Analysis of the progress of active interventional trials.

https://doi.org/10.1371/journal.pone.0289193.t005

First, we checked all included trials for prospective or retrospective registration and found out that more than a third were registered retrospectively (37,2%). The highest rate of retrospectively registered trials had the IRCT with almost two thirds (64,9%), followed by the ChiCTR with almost half their trials being registered retrospectively (46,9%).

Then we checked each trial for their original recruitment status to compare it with the findings of our follow-up search after two years. Originally, more than half of the trials were registered as “recruiting” (55,1%) while the rest were registered as “not recruiting”. As of the 11^th of August 2022, the date of our follow-up search more than a third were still labelled as “recruiting” (39,2%), while slightly less than a third were either labelled as “not recruiting” (30,6%) or were already labelled as “completed” (28,8%). The rest had not stated their recruitment status (1,4%). All 490 trials exceeded their date of first enrolment, which means all trials currently labelled as “not recruiting” had not been updated accordingly. 91,4% of the trials also exceeded their estimated or actual completion date of which more than a third (36,1%) were still labelled as “recruiting”. Those trials therefore had not been updated accordingly as well. This adds up to a total of 66,7% trials which had not been updated accordingly.

Furthermore, we found out that nearly a quarter had not been updated once since their date of registration (24,3%). Out of those trials, 12,6% were labelled as “completed” in their latest version of their registration record, which implies that they were already registered as “completed” even though the downloaded database from the WHO stated otherwise.

Additionally, we found out that slightly less than half the trials which already published results were labelled as “completed” (48,9%) with the rest either been labelled as “recruiting”, “not recruiting” or not having stated any status in decreasing order.

Assessment of quality.

To assess the quality of the registration records and the registries itself we checked each record for completeness of the Trial Registration Data Set (TRDS) (S2 File, Table 6). Across all registers item 13 (interventions) was most frequent with 43,9% of all trials displaying insufficient information. Most of those trials have not stated any or insufficient information about the dosage, frequency of application or duration of the treatment. The EU-CTR had the highest rate of missing information on this item with 69,2% of their trials, followed by the ChiCTR, the IRCT and ClinicalTrials.gov. Item 21 (ethics review) was missing second most often with 40,2% of all trials not displaying any information on their ethics review. ClinicalTrials.gov had the highest rate of missing information on this item with 97,1% of their records. By not stating the allocation method, masking or the type of assignment, item 15 (study type) was missing third most often in 22,4% of all trial records. On this item the ChiCTR had the highest rate of missing information with 40,9% of their records. The summarization of results (item 23) was missing for 79,1% of the 139 trials which already published results by not providing a summary of the results or a link to the published results in their original record. ChiCTR and ICRT both had the highest rate of issues on this item with 100% of their published trials not providing a summary or a link to the publication on their record. This was followed by ClinicalTrials.gov with 70,3% and the EU-CTR with 50,0% of their trials. The contact information for public and scientific queries (items 7 and 8) were completely missing in 3,9% across all registers, however for another 15,9% of all trials, the information was only available in the WHO dataset and not in the original record from the source register. Seventy-seven of these trials were registered with ClinicalTrials.gov with a rate of 44,7% of their total 172 registered trials.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Assessment of quality–Completeness of the trial registration data set.

https://doi.org/10.1371/journal.pone.0289193.t006

Assessment of research waste.

For the assessment of research waste, we first investigated the discontinued trials for their reason of discontinuation. During the two-year follow-up process we identified that 17,4% of the interventional trials were discontinued due to several reasons. We sorted all stated reasons for discontinuation into 8 different categories. We also divided those trials by their source register and calculated the discontinued trials per total trials per register by dividing the total number of discontinued trials by the total number of trials from one register. 39,8% of the discontinued trials unfortunately did not state any reason. After that, the most frequent reason was “difficulties with enrolment” with 29,1% trials followed by “new evidence or new treatment guidelines” from other trials and/or publications with 12,6% trials. Divided into registers, ClinicalTrials.gov had the most discontinued trials with 56,3% of all discontinued trials followed by ChiCTR with 37,9% trials and EU-CTR with 12,6%. If we look at the rate of discontinued trials per total trials per register however, we see that out of all registers with more than 10 registered trials, the EU-CTR has the highest rate of discontinued trials with 50,0% followed by ClinicalTrials.gov with 21,8% and ChiCTR with 13,3% (Table 3).

We further evaluated potential research waste by analyzing the study design across the remaining 490 active interventional trials with extensive results shown in Table 7. Overall, randomized-controlled trials (RCT) were most common with more than two thirds of all 490 interventional trials (68,3%), followed by single arm trials and non-randomized-controlled trials (NRCT). In the group of RCTs, 44,4% trials were not masked while 27,5% had not given any information about their masking. Seventy-eight-point-five percent of the RCTs used “standard of care” as the control while the rest used a placebo. In the group of NRCTs 35,4% trials were not masked and 10,4% did not state any information on their masking. Ninety-five-point-eight percent used “standard of care” and 4,2% trials a placebo as their control group. The most frequent study phase over all interventional trials was phase 0 with 23,3% followed by phase 2 trials with 17,3%. The study phase of 24,9% trials was either not applicable or not stated. The mean target size over all 490 trials was 591,56 subjects (SD 3445,83), the median 100 subjects, the 25 percentile 60 subjects and the 75 percentile 255,75 subjects. The target sizes according to each study phase are shown in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Box-whiskers-plot of the target sizes per study phase of all active interventional trials.

X-axis: Active interventional trials separated into the different study phases; N/A, not applicable or not stated. Y-axis: Logarithmic representation of the target sizes. Outliers are titled with their individual trial-ID. Data was extracted from the original dataset downloaded from the ICTRP and from the original registration record of each trial.

https://doi.org/10.1371/journal.pone.0289193.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Assessment of research waste–Analysis of study design of active interventional trials.

https://doi.org/10.1371/journal.pone.0289193.t007

Assessment of redundancies.

To identify possible redundancies among all interventional trials we analyzed the different therapeutic agents being investigated (Table 8). We identified a total of 261 trials which used a therapeutic agent other than complementary or alternative medicine and advanced medicinal products. Of those 261 trials 200 were still active and 61 discontinued. Over all 261 trials we identified 94 different drugs. (Hydroxy)-chloroquine was by far the drug tested most often with 31,8% trials. After that ritonavir, lopinavir and interferons were tested most often in decreasing order. (Hydroxy)-chloroquine also had the highest rate on discontinued trials with 38,6% out of 83 total trials. This was followed by remdesivir with 36,4% and azithromycin with 33,3%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Assessment of redundancies–Analysis of investigated therapeutic agents across all interventional trials.

https://doi.org/10.1371/journal.pone.0289193.t008

First objective: Assessment of spatial and temporal evolution of clinical research of COVID-19 across the globe.

Our key findings for this objective were: 1) The overall research activity was much higher than in any pandemic, epidemic or health crisis before. 2) China was the country with the most registered trials in total and per confirmed cases, however in terms of trials per population size, the Netherlands and France were leading. Looking at the evolution of the numbers from each country though, it appears that the number of trials per cases and population will align over time. 3) Our results suggest that declarations and warnings from institutions like the WHO did not have that much of an impact on the emergence of clinical research. Rather, clinical research might had been triggered by panic due to rising disease incidence.

1) Our initial sampling goal of 1000 registered trials was reached after only approximately 3 months while the current total number of registered trials (accessed 29^th of June 2023) in the ICTRP related to COVID-19 was 15834. For other pandemics and epidemics, the total numbers of registered trials were significantly lower, e.g., H1N1 (n = 509), Ebola (n = 140), SARS (n = 4053), MERS (n = 21) and even HIV/Aids (n = 11764). To put this into perspective, H1N1, Ebola and MERS combined did not have as many registered trials as we found for COVID-19 in the first 3 months [14].

2) The temporal and spatial development of clinical research in the COVID-19 pandemic in general followed the spread of the disease, which was expected. China was the most dominant country with more than two thirds of all trials originating from there and leading on the number of trials per 100 cases with 0,806 vs. Japan being second with 0,512 trials per 100 cases. Considering trials per population though, China was not as dominant as the previous numbers might imply with 0,465 trials per 1 million inhabitants vs. Netherlands with 0,8 and France with 0,682. Another trial which examined the characteristics of published scientific articles in the first 3 months of the COVID-19 outbreak came to similar results with China having published more than half of the articles with 50,5% and 0,422 articles per 1 million inhabitants [15]. However, in terms of articles per 100 disease cases, our study shows different results. In our study China registered third most trials per cases while China was in the bottom end of articles per cases in the opposed study. The opposed study took the number of confirmed cases from the 2^nd of March 2020, before the COVID-19 situation was declared as a pandemic, while our designated date was set one month later, on the 1^st of April 2020, and after the declaration, which could explain the differences. Thus, in our study, the number of confirmed cases per country was significantly higher, especially in countries other than China due to the vast spread of the disease in March 2020, and thereby the differences in terms of trials per cases are smaller. Our results suggest that the numbers of trials per cases will continue to align over time, and thus China will only be standing out in the first few months of the COVID-19 outbreak.

3) Nevertheless, the fact that only 4,9% of all trials before the pandemic declaration by the WHO originated from outside of China was still surprising, since the outbreak was declared “a public health emergency of international concern” one month prior to being officially declared a pandemic. One would have expected a more robust response at this time [1]. The graphical development of the cumulative cases and trials per date of each region could be an explanation for this (Fig 3). We see that in all countries except China the number of cases began to increase dramatically only a few days before the declaration, which was then followed by a rapid growth of registered trials. In China on the other hand, the number of cases stagnated around the same time which was then followed by a flattening curve of registered trials. All in all, this raises the question of whether official declarations by the WHO or other institutions are influencing the start of research or whether panic due to the individual case numbers in each country are the main driver for research. Our findings, however, imply the latter.

Second objective: Assessment of transparency and quality–trial registration.

Our key findings for this objective were: 1) There is a lack of transparency in trial registration expressed by retrospective registration and registration records not kept up to date with conflicting and invalid information being displayed. 2) The overall quality of trial registration seems to have improved since the introduction of the Trial Registration Data Set, yet there is still crucial information missing.

Trial registration is a crucial instrument which aims to ensure transparency, quality and validity of research and therefore prevent publications bias, reproducibility bias, and selective reporting bias [8, 16]. Transparency of trial registration was defined as maintaining the trial registration records and updating them in a timely manner, to ensure that every information extracted from these records can be considered as valid and up to date. The quality of trial registration was assessed by screening each registration record for prospective registration and for completeness of the trial registration data set.

1) Prospective registration is an important tool to prevent selective reporting and publication bias. By making registration records a public matter before recruiting the first patient, it is very unlikely to alter the data to fit the results since every change to the protocol or record is publicly accessible [8, 16, 17]. This topic is also stated in the declaration of Helsinki with the words “Every research study involving human subjects must be registered in a publicly accessible database before recruitment of the first subject” [18]. The ICMJE therefore requires prospective registration from every clinical trial initiated after July 1, 2005 in order to be considered for publication in any of their member journals [19]. Though, several studies concluded that retrospective registration and thus the possibility of publication and selective reporting bias is still present [20, 21]. One study, which analyzed all trials registered in the ANZCTR from 2006 to 2015, came to the result that the number of prospectively registered trials increased from 42% in 2006 to 63% in 2012 and plateaued from thereon [22]. Our study came to similar results with one third of all trials being registered retrospectively. Particularly problematic is the fact that we identified 15 trials which were registered retrospectively after the recruitment was already completed. Although other studies suggest that the number of trials which are being registered retrospectively appear to decrease, there is still a lot of work to do. If retrospective registration exists, selective reporting and publication bias will be a reoccurring problem. Registries and especially journals are in need follow the ICMJE rules more strictly by no longer accepting and publishing studies that were registered retrospectively to overcome this issue.

Keeping trial records up to date is another very important aspect of trial registration. To enforce this, all primary and partner registries in the WHO Registry Network signed a form which endeavors to keep registered information up-to-date, some registries even implemented reminder systems to facilitate this by notifying the authors at least every 6 months if there was no update [23]. To our knowledge this is the first study which conducted a follow-up to track the progression of trial registration records over time to evaluate the accuracy and validity of the information being displayed. We discovered that two thirds of the trials haven’t been updated accordingly and a quarter of all trials have never been updated after two years since their registration. Particularly unfortunate is the fact that more than half of all trials which already published results were not even labelled as completed. One of the goals of trial registration is that the public and scientific community can consider every information and data displayed as accurate and valid. If this is not ensured for arguably one of the most important and fundamental information in trial registration, i.e., the recruitment status, the whole concept of trial registration is to be questioned.

2) The Trial Registration Data Set (TRDS), which was designed by the WHO and is required by the ICMJE, represents another tool to ensure transparency, quality and validity in clinical research and trial registration by providing a minimum set of items each trial record has to display in order to be considered adequate and eligible for publication [24]. To ensure this and to further conquer publication and reproducibility bias, it is crucial that the information of several key items (i.e. interventions, contact information, study type) is informative and complete. Though, our results showed that there were several items with a high rate of missing or insufficient information, e.g., Item 13 (interventions) with 43,9%, Item 21 (ethics review) with 40,2% and Item 15 (study type) with 22,4%. Also, only 26/139 (18,7%) trials displayed Item 13 (summary results) sufficiently and either provided a link or a summarization for already published results. We also detected differences between the registries on which items are missing most often. The ChiCTR for example displayed insufficient information for item 13 (interventions) in 61% and for item 15 (study type) in 40,9% of all their records, while ClinicalTrials.gov was missing Item 21 (ethics review) in 97,1% of their registration records and item 7 and 8 (contact for public and scientific queries) in 44,8%. One study, which took a random sample of 439 records registered between 2008 and 2009 from the ICTRP and assessed those in terms of their contact, intervention and primary outcome information, found that 32,3% of the trials have not provided sufficient contact information and only 44,2% provided sufficient information on interventions [25]. Another study conducted by the same authors 4 years later with the same design found that 15,1% of the trials have not provided sufficient contact information and only 51,9% provided sufficient information on interventions [26]. In our study only 3,9% trials were completely missing the contact information of which all were registered on ClinicalTrials.gov. The differences regarding the contact information can be explained with the fact that in our study ClinicalTrials.gov is underrepresented with 35,1% of all trials being registered on this register compared to 85,9% respectively 57,5% in their studies. Results from another study in 2009 which assessed the whole data set (at that time only 20 items) showed a much higher rate of missing information for nearly every item [27]. Since these studies depict rather the early years after trial registration was implemented, our findings also suggest that there was an overall improvement in registration quality over time, yet some key items like interventions have been and still are missing critical information. These suggestions meet the results from two recent studies which assessed the quality of trial records investigating Traditional Chinese Medicine. The authors concluded that especially the complex items like interventions, study design and primary outcomes were missing crucial information [28, 29]. Although, the implementation of trial registration and therefore compliance with the TRDS was advocated over 17 years ago, there is evidently still a huge lack of quality and transparency. To conquer publication bias, selective reporting bias and reproducibility bias, institutions like the WHO and the ICMJE, must work more closely with the registers, journals, and researchers to overcome these issues. As a possible solution for the differences between registers, lack of quality in trial registration and to ensure that the information is displayed equally, we suggest a unified item mask which each register must implement to be considered as a partner for the WHO Registry Network and be eligible for publication. This, however, must be enforced by the WHO and the ICMJE, as there is no need for registers to change if their trials are still being published.

Third objective: Assessment of research waste and redundancies.

Our key findings for this objective are: 1) The early stages of clinical research about COVID-19 showed clear signs of research waste. Furthermore, research waste appears to be a reoccurring problem in the scientific world. 2) “Hype train phenomenon” in early stages of research assumingly due to a lack of dedicated supervising institutions and coordination which leads to unnecessary redundancies and ultimately to more research waste.

Research waste and redundancies in clinical research are significant problems that can have serious implications for the scientific community, as well as for the patients who participate in clinical trials. Research waste refers to the unnecessary use of resources including time, money, and other valuable resources. Research waste will therefore be measured by the rate of discontinuation and the quality of study design. Redundancies, on the other hand, occur when multiple studies are conducted on the same or similar topics, resulting in duplication of efforts and resources. To assess this issue, we screened all drug trials for their therapeutic agents to depict potential redundancies which would also lead to more research waste.

1) Discontinuation displays one source for wasting money, time, and other valuable resources [30–32]. Nearly 2 years after the last trial of our study was registered, we identified 17,3% discontinued interventional trials. The most common reason for discontinuation was difficulties with enrolment of patients with 29,1% trials. Discontinuation rates ranging from as low as 11% and up to 43% had been displayed in several other studies. Similar to our findings, the most common reason for discontinuation in all of those trials was difficulties with enrolment [30, 31, 33–37]. However, a systematic review from 2016 suggested that most reasons for recruitment failure could be anticipated in the planning phase of a trial and thus are preventable. The authors also provided a checklist to ensure that weak spots in study designs are being identified [38]. Since transparency is important to detect errors in the planning of studies and to prevent them in the future, it is even more problematic if no reason for discontinuation is given. Unfortunately, 39,8% had not stated any reason for discontinuation. Even in discontinued trials, transparency is important to detect errors in the planning of trials and prevent those mistakes in the future by learning from one another.

Another important aspect to avoid research waste is an appropriate and robust study design. Although, more than two thirds of all interventional trials are designed as RCTs, which are mostly ranked on top of the evidence levels and hailed as the “gold-standard” of evidence-based-medicine, almost one third of trials used other designs, often without any control, which is ranked on the bottom of the evidence pyramid [39]. Further, even the RCTs have flaws in their design as nearly half of them were not masked which can lead to several bias such as selection bias, which was supposed to be eliminated by the randomization itself [40]. We speculate that panic and the willingness to help may have driven these “short-cuts” in robustness.

2) To further avoid research waste, it is also important to create awareness for redundancies in clinical research. Trial registration was intended as a tool to diminish this by making planned and ongoing research available for the public and scientific world.

(Hydroxy)-chloroquine (HCQ/QC), as the most prominent example, gained huge popularity in the early months of COVID-19. However, the timeline of HQC/QC showed contradicting information about the pros and cons of its therapeutic use against COVID-19 which was also influenced by the media and even leading politicians in effected countries [41]. The therapeutic use was halted when the US-FDA revoked its emergency authorization on June 15, 2020 [42]. Early reviews from 2020 supported this by indicating weak and conflicting evidence about the efficacy and safety of HQC/CQ [43–46]. Throughout this evolution, there was always an urgent call for better designed and properly powered trials to gain unequivocal data safety and efficacy for HCQ/CQ usage in COVID-19 [44, 47]. Though, our results show that 14% of all interventional trials investigated the usage HQC/CQ with more than a third (38,6%) of those trials being discontinued before completion. This is not only an example for research waste and redundancies but also raises the question on whether numerous trials assessing the same topic operationally fragmented in isolated silo-structures fulfils the purpose of gaining more evidence. The given tools (e.g., trial registration and the registers itself) must be used more thoroughly to detect similar trials and encourage collaboration instead of conducting own trials one after the other without any coordination. This could not only help with the power of trials but could also lower the discontinuation rates by preventing recruitment failure through multicentric designs and thus also prevents unnecessary exposure to clinical trials for patients.