http://orcid.org/0000-0002-8978-458X
Figures
Abstract
Objectives
To examine the risks of bias, risks of random errors, reporting quality, and methodological quality of randomized clinical trials of oral health interventions and the development of these aspects over time.
Methods
We included 540 randomized clinical trials from 64 selected systematic reviews. We extracted, in duplicate, details from each of the selected randomized clinical trials with respect to publication and trial characteristics, reporting and methodologic characteristics, and Cochrane risk of bias domains. We analyzed data using logistic regression and Chi-square statistics.
Results
Sequence generation was assessed to be inadequate (at unclear or high risk of bias) in 68% (n = 367) of the trials, while allocation concealment was inadequate in the majority of trials (n = 464; 85.9%). Blinding of participants and blinding of the outcome assessment were judged to be inadequate in 28.5% (n = 154) and 40.5% (n = 219) of the trials, respectively. A sample size calculation before the initiation of the study was not performed/reported in 79.1% (n = 427) of the trials, while the sample size was assessed as adequate in only 17.6% (n = 95) of the trials. Two thirds of the trials were not described as double blinded (n = 358; 66.3%), while the method of blinding was appropriate in 53% (n = 286) of the trials. We identified a significant decrease over time (1955–2013) in the proportion of trials assessed as having inadequately addressed methodological quality items (P < 0.05) in 30 out of the 40 quality criteria, or as being inadequate (at high or unclear risk of bias) in five domains of the Cochrane risk of bias tool: sequence generation, allocation concealment, incomplete outcome data, other sources of bias, and overall risk of bias.
Conclusions
The risks of bias, risks of random errors, reporting quality, and methodological quality of randomized clinical trials of oral health interventions have improved over time; however, further efforts that contribute to the development of more stringent methodology and detailed reporting of trials are still needed.
Citation: Saltaji H, Armijo-Olivo S, Cummings GG, Amin M, Flores-Mir C (2017) Randomized clinical trials in dentistry: Risks of bias, risks of random errors, reporting quality, and methodologic quality over the years 1955–2013. PLoS ONE 12(12): e0190089. https://doi.org/10.1371/journal.pone.0190089
Editor: Christian Gluud, Copenhagen University Hospital, DENMARK
Received: April 28, 2017; Accepted: December 7, 2017; Published: December 22, 2017
Copyright: © 2017 Saltaji et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This project is funded by Alberta Innovates - Health Solutions (AIHS) through a Clinician Fellowship Award, the University of Alberta through the Honorary Izaak Walton Killam Memorial Scholarship, and the Women and Children’s Health Research Institute (WCHRI) through the Honorary WCHRI Award. The funders had no role in study design, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Randomized clinical trials are the ideal type of clinical research to examine the effectiveness of treatment interventions in health sciences [1]. The value and significance of a randomized clinical trial depends on the control of potential biases, how rigorously it was conducted, and how thoroughly the results were reported. The SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) Statement [2], the Consolidated Standards of Reporting Trials (CONSORT) Statement [3], and recent initiatives such as the International Committee of Medical Journal Editors (ICMJE) statement on clinical trial registration [4], have led to improvements in both the methodological and reporting quality of medical randomized clinical trials [5–7]. Adhering to these initiatives is critical to oral health research and practice, as high quality randomized clinical trials contribute largely to the body of evidence measured in systematic reviews and meta-analyses, especially when assessing therapeutic interventions.
Currently, nearly 50 clinical trials of oral health interventions are estimated to be published every month, and this number is expected to increase over time [8]. Emerging evidence from methodological reports published in various dentistry-related specialties over the last decade (periodontics [9], prosthodontics [10], implantology [11], orthodontics [12], restorative dentistry [13], and six dental specialties [14]) report methodological quality of oral heath randomized clinical trials that is below acceptable levels to adequately lead clinical decision making. Moreover, there is evidence that some trials are biased and, due to weaknesses in their methodological characteristics, they tend to exaggerate the magnitude of the treatment effect [15]. This emerging evidence raises questions about the validity of trial results for oral health interventions, which dental practitioners use when making day-to-day clinical decisions in dental practice, and which policy makers use more generally when developing clinical practice guidelines.
In the context of medical research methodology, the internal validity of a trial is dependent on “risk of bias” (which concerns the internal validity of a trial) [16] and “risks of random errors” (i.e., risks of play of chance). Moreover, while “reporting quality” refers to the reporting of the conduct and design of a trial [17, 18], “methodological quality” is related to the internal validity of a trial and determined by the extent to which the conduct and design of a trial are precisely and rigorously performed to generally acceptable standards so that biases are minimized.
In the field of oral health, to our knowledge, no study has assessed changes over time regarding quality of reporting, methodological characteristics, and risks of bias in randomized clinical trials of oral health interventions. A recent report by Reveiz et al. [19] described the results of an examination of different risk of bias domains in a sample of medical randomized clinical trials (identified from a cohort of Cochrane reviews). Reveiz and colleagues’ report stated that the rate of trials found at low risk of bias consistently increased with time. However, since the randomized clinical trials in Reveiz and colleagues’ sample were performed in the field of medicine and were dependent on the risk of bias assessment performed by investigators presenting published reviews (rather than by conducting standardized data extraction from each trial), the findings cannot be compared to findings from trials in the field of dentistry which tend to have different design characteristics, such as difficulty in applying blinding and common use of the split-mouth design.
Consequently, it is unclear if the increase in number of published randomized clinical trials of oral health interventions over time has been associated with changes in the conduct and reporting of the trials. In this study, we set out to assess these aspects in randomized clinical trials of oral health interventions, and if these aspects have improved over time. Our objectives were to (1) examine the risks of bias, risks of random errors, reporting quality, and methodological quality, and the general trial characteristics of randomized clinical trials of oral health interventions; and (2) determine whether (and to what extent) risks of bias, risks of random errors, reporting quality, and methodological quality have improved over time.
Methods
Study sample
We used the Oral Health Database of Systematic Reviews [20, 21] which includes a comprehensive selection of oral health systematic reviews published in the field of oral health research between 1991 and 2014. From this database, we selected a sample of systematic reviews with meta-analyses and their associated randomized clinical trials that met the following criteria: the meta-analysis was (1) published in any language and (2) conducted in an oral health field that examined an intervention concerning craniofacial, oral, or dental diseases (as defined by the American Dental Association [ADA] scope of practice) [22]. A randomized clinical trial was defined as “an experiment in which two or more interventions (possibly including a control intervention or no intervention) are compared, by being randomly allocated to participants” [16]. Further details regarding the study selection included in the final database of systematic reviews have been published [20, 21] (see Fig 1). Briefly, two reviewers (dentists with oral health research backgrounds) independently selected relevant reports and determined the final eligibility of the full texts (any disagreements were resolved through consensus). Ultimately, 540 randomized clinical trials that met the predefined eligibility criteria were selected and utilized in this study. The list of included randomized clinical trials is provided in S5 Appendix. The study protocol was not registered or published in advance.
Data extraction
A panel of five reviewers from diverse health research areas (dentistry, pediatrics, and physical therapy) extracted the data. To ensure consistency during data extraction, two team members (H.S., S.A.) facilitated a reviewer training process, similar to the process followed in our team’s previous investigations [23, 24]. In this process, the review panel evaluated and discussed 10 randomized clinical trials not included in the final set of trials. Once agreement on data extraction protocols and interpretation was achieved, the review panel performed data extraction in duplicate. Two assessors independently carried out data extraction for each included randomized clinical trial (consensus meetings were employed to resolve any disagreements). One assessor (H.S.) who has a background in oral health research performed a complete data extraction (n = 540, 100%), while other members of the review panel (C.H., J.S., J.F.) who have medical (non-oral health) research backgrounds, acted as secondary assessors. If two assessors could not reach an agreement, then a third assessor (S.A.) assisted with consensus. Only data that received consensus were used for data analyses. We used a structured and pilot-tested data extraction template, designed using Microsoft Office Access, for data extraction. We extracted details from each of the selected randomized clinical trials with respect to publication and trial characteristics, risks of bias, risks of random errors, reporting quality, and methodological quality, as described below.
Publication and trial characteristics
Data elements related to publication and trial characteristics included the following information: publication year, dental speciality as classified by the American Dental Association (ADA) (e.g., periodontics, dental public health, prosthodontics and restorative dentistry, oral medicine and oral pathology, implantology, oral and maxillofacial surgery, orthodontics and dentofacial orthopedics, pediatric dentistry, endodontics [22]), country and continent of first author, number of authors, funding source (e.g., foundation, government, industry, academic), type of journal (e.g., specialty oral health, general oral health, nonoral health), type of intervention (e.g., surgical, drug, dental material, device, psychological, educational, policy), age of participants, number of centres (e.g., multicentres, single centre), design (e.g., parallel, crossover, split-mouth, cluster), type of outcome (e.g., subjective, objective), and sample size.
Risks of bias
We employed the Cochrane risk of bias tool introduced in 2008 [1], which contains six domains and seven items, namely, “sequence generation,” “allocation concealment,” “blinding of outcome assessors,” “blinding of participants,” “incomplete outcome data,” “selective outcome reporting,” and “other sources of bias.” We used the Cochrane Collaboration guidelines to score domains (e.g., high, low, unclear). However, we developed specific rules to make final decisions (see S3 Appendix and S4 Appendix). For “other sources of bias,” we examined baseline comparability, control for cointerventions, whether treatment compliance was acceptable, and funding [25]. For the overall assessment of risk of bias, if one domain was assessed as having a high risk, the overall risk of bias assessment was labelled “high risk.” A randomized trial was considered to be at low risk of bias if it was assessed as “low risk” in all individual domains. If the assessment was “unclear” in at least one domain (and other domains were unclear or low) the overall risk of bias assessment was designated “unclear” [26, 27].When reporting the results of the risk of bias assessment, we combined “high” and “unclear” risk of bias assessments into one type of trials, considering that trials that these types of trials are at risks of overestimating benefits and at risk of underestimating harms [1, 14, 25].
Risks of random error
Risks of random errors involves two criteria: (1) sample size calculation and (2) adequate sample size (see S1 Appendix).
Reporting quality and methodologic quality
Reporting quality and methodological quality are difficult to distinguish and often overlap to some extent. Methodological quality is defined as “the confidence that the trial design, conduct, and analysis has minimized or avoided biases in its treatment comparisons” [17, 18] (e.g., the sequence generation was appropriate). Reporting quality involves the provision of “information about the design, conduct, and analysis of the trial” [17, 18] (e.g., this was a randomized trial). Accordingly, based on preliminary work performed by the research team [28, 29], we obtained 40 quality assessment criteria and their classifications (“reporting” vs. “conduct”) from the most commonly used quality assessment tools in health care research [30–37]. Of the 40 quality criteria selected, 15 criteria assess “reporting” quality, 21 criteria assess “methodological” quality, and four quality criteria assess both reporting quality and methodological quality. We classified the items that evaluated methodological quality according to type of bias as follows [28, 29] (see S1 Appendix): selection bias (6 criteria), performance and detection bias (4 criteria), performance bias (9 criteria), performance and compliance bias (2 criteria), information bias (3 criteria), reporting bias (3 criteria), attrition bias (5 criteria), detection bias (2 criteria), statistical bias (1 criterion), threats to precision (1 criterion), and multiple biases (2 criteria). We also grouped the selected quality criteria according to the following categories [16]: patient selection (inclusion and exclusion criteria, description of participants); assignment, randomization, and allocation concealment; blinding; interventions; attrition, follow up and protocol deviation; outcomes; and statistical analysis. Using original tools as guidelines, the definitions and methods were derived for each criterion, using a three-part answering scheme (yes, no, unclear) for each item. We established decision rules and guidelines to ensure consistency (see S1 Appendix and S2 Appendix).
Data analysis
We conducted descriptive analyses for each trial characteristic, quality assessment item, and risk of bias domain (using means and standard deviations [SD] or median and interquartile range [IQR] for continuous outcomes, and proportions and percentages for categorical outcomes, where appropriate). To evaluate whether the quality of randomized clinical trials has improved over years, trials were grouped according to four periods of publication year: before 1990, 1990–1999, 2000–2006, 2007–2013. We used Chi-square statistics and two-tailed Fisher exact tests to examine the difference in proportion with respect to time periods for all quality assessment items and risk of bias domains. Furthermore, we used a logistic regression to explore the relationship between each criterion and time; we entered time into the logistic regression model as a continuous variable (publication year) and a categorical variable (time period of publication year; < 1990 was used as a reference category). The outcome of each analysis was each methodological criterion dichotomized in low risk vs. others (unclear, high risk of bias), or yes vs. others (no, unclear, not-reported). We reported odds ratios (OR) with 95% confidence intervals (CIs) and we set the statistical significance at P < .05. We performed statistical analyses using Stata Version 14.0 (StataCorp) and the Statistical Package for Social Sciences for Windows (SPSS) Version 18.0 (IBM, Armonk, NY).
Results
Trial publication and characteristics
From the selected 540 trials published between 1955 and 2013 (median year of publication: 2000; IQR: 1990, 2007) (see Fig 2), the majority of trials were published either in periodontics (n = 233; 43.1%), dental public health (n = 124; 23.0%), or prosthodontic and restorative dentistry (n = 54; 10.0%). More than half of the trials were published in journals that specialized in oral health (n = 304; 56.3%) (see Table 1). The trials’ first authors were most frequently from Europe (n = 239; 44.3%) followed by North America (n = 202; 37.4%). Three countries (the United Kingdom, Italy, and the United States) accounted for almost half of all trials (n = 280; 51.9%). Approximately one fifth of the trials were multicenter trials, nearly half of the trials involved four to six authors (n = 249; 46.1%), and one third included two to three authors (n = 169; 31.3%). In approximately half of the trials, the authors did not declare whether they received a source of funding (n = 256; 47.4%), while nearly one third of trials received funding from industry (n = 171; 31.7%).
Approximately one third of the trials were placebo-clinical (n = 204; 37.8%) and two thirds examined nondrug (n = 359; 66.5%) or nonsurgical (n = 370; 68.5%) interventions. One quarter of the trials examined pediatric patients (n = 136; 25.2%), while the majority examined adults (n = 398; 73.7%). The majority of trials used parallel design (n = 372; 68.9%), while almost one quarter used the split-mouth design (n = 126; 23.3%).
Changes in risks of bias, risks of random errors, reporting quality, and methodological characteristics over time
Sequence generation was assessed to be adequate (low risk of bias) in 32% (n = 173) of the trials, while it was inadequate (unclear or high risk of bias) in 68% (n = 367) of the trials. Allocation concealment was inadequate (unclear or high risk of bias) in the majority of trials (n = 464; 85.9%). Blinding of participants was judged to be adequate (at low risk of bias) in 71.5% (n = 386) of the trials, and blinding of the outcome assessment was judged to be adequate (at low risk of bias) in 59.4% of the trials. Other sources of bias—baseline comparability, similarity of cointerventions, and compliance to treatment—were judged at low risk of bias in 77.8%, 40.2%, and 53.5% of the trials, respectively. The influence of the trial sponsor was assessed as being inadequate (unclear or high risk of bias) in 83.3% (n = 450) of the trials, while it was assessed as appropriate in 16.7% (n = 90) of the trials. The overall risk of bias was inadequate (unclear or high risk of bias) in 94.6% (n = 511) of the trials, and an overall low risk of bias was assessed in only 5.4% (n = 29) of the trials (see Table 2).
Regarding risks of random errors, a sample size calculation before the initiation of the study was not performed or reported in 79.1% (n = 427) of the trials, while the sample size was assessed as adequate in only 17.6% (n = 95) of the trials.
Inclusion and exclusion criteria were clearly defined in the majority of trials, while baseline comparability was adequate in 72.8% of the trials. Two thirds of the trials were not described as double blinded (n = 358; 66.3%), while the method of blinding was appropriate in 53% (n = 286) of the trials. Blinding of the principal investigator and statistician was unclear/not reported in 92% and 97.8% of the trials, respectively, that is, in the vast majority of the trials.
The treatment protocol was adequately described for treatment and control groups in the vast majority of trials, with 73.1% (n = 395) having a designated control group, and 38.1% (n = 206) using a placebo group. Whether cointerventions were avoided/comparable was assessed as unclear/not reported in 60.7% (n = 328) of the trials, while 84.1% (n = 424) of the trials did not report cointerventions for each group. Participants compliance to treatment protocol was tested in 61.1% (n = 330) of the trials, with compliance being acceptable (more than or equal to 80%) in 50.9% (n = 275) of the trials. Withdrawals/dropouts were reported in the vast majority (89.4%, n = 483) of trials, with withdrawal/dropout rates being acceptable (less than or equal to 20%) in 73.1% (n = 395) of the trials, and reasons for withdrawals/dropouts reported in 71.1% (n = 384) of the trials. Adverse effects were not described in nearly half (n = 264; 48.9%) of the trials.
Outcome measures were described in the majority of trials, while psychometric properties of main outcome measures—validity, reliability, and responsiveness—were not reported in 96.7%, 93.1%, and 96.7% of the trials, respectively. The statistical analysis was appropriate in 85.7% (n = 463) of the trials, with descriptive measures being reported in the majority of the trials. The clinical significance was not reported in 70.9% (n = 382) of the trials, while the intention to treat analysis was not used/reported in 59.6% (n = 264) of the trials (see Table 2).
We identified a significant decrease (P < 0.001) in the proportion of trials judged as being inadequate (having high or unclear risk of bias) over time in five domains of the Cochrane risk of bias tool: sequence generation, allocation concealment, incomplete outcome data, other sources of bias (including: baseline comparability, similarity of cointerventions, and compliance to treatment), and overall risk of bias (see Table 3). The proportion of trials assessed as being inadequate (having high or unclear risk of bias), with respect to patient blinding, decreased significantly (P < 0.031) in the sample, while change in blinding of outcome assessment and selective outcome reporting were not statistically significant.
The proportion of trials assessed as having inadequately addressed methodological quality items decreased significantly over time in 30 out of the 40 quality criteria (23 quality criteria at P < 0.001, seven quality criteria at P < 0.05). This was not statistically significant in the following items: study described as randomized, method of blinding appropriate, blinding of principle investigator, blinding of assessor, treatment protocol adequately described for the treatment group and for the comparison group, report of withdrawals/dropouts, outcome measures described, validity and responsiveness for main outcome measures reported, descriptive measures reported, and early cessation of a trial (see Table 4).
The results of the logistic regression analyses showed that a significant change over time was evident in 29 out of the 36 quality criteria (that is, 10 risk of bias domains and 26 quality items) of which 26 quality criteria improved over time, while 3 criteria (study described as double blind, blinding of care-provider, and presence of placebo group) worsened over time. Conversely, 8 quality criteria (selective outcome reporting, report of withdraws and dropouts, and 6 blinding-based criteria) did not show a significant change over time (see Table 5).
Discussion
Bias is a threat to the quality of trials [38, 39]; it may impact the reported treatment effect estimates in randomized clinical trials [40–44]. The degree of bias in randomized clinical trials of oral health intervention has decreased over time according to our study. We used the Cochrane Collaboration’s risk of bias tool, in addition to a comprehensive set of reporting and methodological characteristics (selected from seven quality assessment tools reported to be valid), to assess the methodological quality of randomized clinical trials of oral health intervention. Thus, this study provides an in-depth analysis of the methodological characteristics and risks of bias present in dental literature from 1955–2013.
In the majority of the quality items and risk of bias domains, our study showed that the proportion of trials having inadequate quality (or having high or unclear risk of bias) decreased significantly over time. This encouraging trend is similar to what was identified in a recently published report by Reveiz et al. [19]. However, rather than conducting standardized data extraction from each trial, Reveiz used a risks of bias assessment reported by the investigators of reviews; this might be problematic, especially given the documented low reliability of the Cochrane Collaboration’s risk of bias tool [26, 45]. The trend in our study is comparable to that found in a cohort of child related trials [46] and medical randomized clinical trials [5]. A similar trend was identified when the methodological quality of trials of physical therapy interventions was assessed, where an improvement of nearly 0.6 points each decade was found in the total Physiotherapy Evidence Database (PEDro) score [47]. The trend in our report is also similar to a recently published study [48] which analyzed 20,920 randomized clinical trials included in Cochrane reviews; the study concluded that the proportion of trials at unclear risk of bias decreased over time, especially for sequence generation (fell from 69.1% to 31.2%) and for allocation concealment (fell from 70.1% to 44.6%).
Although an improvement over time was identified in our study with respect to randomized clinical trials of oral health intervention, results of risks of bias, risks of random error, and reported methodological quality assessments were still unpropitious, indicating substandard quality and a high potential for bias. We believe that sizable improvements in the conduct and reporting of oral health randomized clinical trials is possible. The fact that the proportion of trials having low risk of bias did not exceed 60% in the majority of risk of bias domains is a significant concern. Because inadequate design and unrigorous conduct of a trial can bias the estimation of the treatment effect size, decisions made in dental practice might not be based on valid findings. For example, allocation concealment (a process of concealing information about which patients are to be assigned to a new treatment versus those to be given a conventional therapy) and sequence randomization (allocation is carried out using a chance mechanism so that neither the participant nor the investigator will know in advance which will be assigned to an intervention) [49–51] were unclear in 84.8% and 66.7% of the trials, respectively, although these factors improved significantly over time. It should be noted that an “unclear” risk of bias results in a trial that may not mirror the actual design and conduct of the trial. Because journals have a word limit that may restrict authors in reporting detailed methods, all of the methodological characteristics used might not be reported [52], thus restricting the accuracy of quality assessment tools. In the field of dentistry, treatment effect size estimates were found to be larger in trials with inadequate trial’s design characteristics which can can affect overestimation of beneficial or underestimation of harmful treatment effects within trials [15, 44].
Our study showed that sample size calculation was not performed or reported in 79.1% of the trials; this finding is concerning as it raises the risks of random errors leading to the risk of having increased numbers of false positive conclusions (type I errors) and false negative conclusions (type II errors) [53] in randomized clinical trials of oral health interventions. It has recently been estimated that randomized clinical trials have to include at least 1000 participants in order for simple randomisation to serve its purpose: allocation a sufficient sample to allow a fair comparison of intervention effects [54].
Moreover, having an appropriate influence of trial sponsor was found to be inadequate or not reported in 83.3% of the trials. This is concerning because sponsorship bias in oral heath randomized clinical trials can potentially benefit the sponsoring company and might lead to inappropriate clinical decisions. Industry sponsorship of trials may lead to favorable findings than sponsorship by other sources [43]. For example, a recently published report examined the influence of industry sponsorship in 41 randomized clinical trials of dental implants and found that the probability of implant failure in sponsored randomized clinical trials was much lower than the probability of implant failure in nonsponsored randomized clinical trials. Also, there is a debate whether the Cochrane risk of bias tool should include funding source as an individual item or not [55–57]. Accordingly, there is a need for large meta-epidemiological studies to quantify the extent of bias associated with influence of funders on the magnitude of treatment effect size in oral heath randomized clinical trials.
Our study revealed that more than half of the trials were published in specialist journals, and that nearly half of the trials were from the United Kingdom and the United States. These trends are similar for medical trials [52]. Possibly the interest of government and public sectors in the aforementioned countries is responsible for facilitating the financial support for such randomized clinical trials [58].
The improvement observed in risks of bias and reported methodological quality of randomized clinical trials over time, could be attributed to efforts made by editors and reviewers of oral health journals, through endorsement of the CONSORT Statement [59, 60], and by the mandatory implementation of trial registration, as recommended by the ICMJE [6, 61].
The CONSORT Statement is an accepted and widely used approach in medical and dental research to assess the reporting quality of randomized clinical trials. This approach covers the fundamental aspects of a trial’s reporting quality; the CONSORT Statement aims to advance the transparency and quality of medical and dental trial reporting through the creation of reporting criteria [62, 63]. It has been endorsed during the last 10 years by several medical journals worldwide, including the majority of high impact oral health journals [59, 63]. Although the CONSORT Statement applies only to reporting quality, it is used commonly and erroneously by many dentistry researchers as a methodological quality assessment tool. Moreover, the SPIRIT Statement has recently become a widely used tool that aims to advance and improve the quality of protocols of randomized clinical trials and involve a set of criteria to address in a trial protocol [64, 65].
In the dental literature, the concepts “reporting quality” and “methodological quality” are often used interchangeably, contributing to conceptual ambiguity. Methodological quality depends mainly on the degree to which the design, conduct, and analysis of a trial follows the highest possible standards (to reduce multiple potential biases) and, hence, suggests that the findings can be based on the used intervention [1, 16, 17]. While the internal validity of a trial (which is closely connected to the risks of bias [16] and the methodological quality) should be the core of quality assessment, “reporting quality” is mistakenly used by researchers as an alternative for methodological quality; this has induced a conceptual ambiguity in the definition of trial “quality” [28]. In the context of medical research, a risk of bias assessment will benefit from an explicit and unambiguous definition of “methodological quality”.
Although endorsement of the CONSORT Statement by dental journal editors and reviewers results in improvement in the reporting quality of trials, it does not guarantee compliance by trialists [60]. Furthermore, reliance on the CONSORT Statement only, may give reviewers, authors, and readers a false sense of security. Transparent reporting is desirable, but it does not necessarily raise methodological quality or lower the risks of bias [39]. For example, good reporting fails to prevent publication bias (i.e., trials of methods that have beneficial and large effects are published rapidly in journals with high impact), and selective outcome reporting (i.e., beneficial findings get publishing priority) [16]. These reported biases can exaggerate the magnitude of treatment effects in clinical trials, and can distort findings in meta-analysis [66, 67]. Implementation of the mandatory trial registration policy [4] could contribute to the improvement of trial quality and lower the risks of bias identified in this study. Implementation of the mandatory trial registration policy started over 10 years ago by 11 leading medical journals, and is currently applied by over 300 medical journals [6], including many leading dental journals [68]. However, recently only 23% of dental randomized clinical trials, published in 15 high impact dental journals, were registered [61, 69].
The results of this study have several implications. Dental trialists need to explicitly report their trials’ results and adhere to published guidelines. Dental journal editors and reviewers should continue to be committed to international initiatives and statements developed to ensure adequate and appropriate conduct and reporting of randomized clinical trials. Adherence to the above guidelines can reduce the risk that inaccurate conclusions will be drawn from the research and, accordingly, will reduce inappropriate recommendations regarding treatment interventions in dental practice. To adequately apply trial findings to care of their patients, clinicians should be aware of the design, conduct, and reporting of a clinical trial. This knowledge will enable a clinician to deliver the best possible results in his or her own dental practice.
Our findings call for oral health policy makers, methodologists, clinicians, and researchers to develop initiatives for improving clinical trials, which would spread such actions within the oral health community. The formation of a global oral health initiative that aims to improve the conduct and reporting of oral health trials, and that prioritizes methodological criteria in oral health research, would be an example of a potentially needed measure to raise standards of randomized clinical trials.
Strengths and limitations
This cross-sectional observational study provided a comprehensive assessment of oral health randomized clinical trials with respect to trial characteristics, reporting quality, methodological characteristics, and risks of bias, and attempts to identify the variation of these factors over time. The range and size of our sample provided a comprehensive evaluation of oral health trials over the 58 year-period of 1955–2013. One of the strengths of our research was the data extraction method, which was performed in duplicate by two independent assessors to ensure high accuracy and avoid potential biases during the data extraction process. We performed a standardized data extraction rather than relying on the risks of bias assessment reported in systematic reviews, which was the case in recent reports by Reveiz et al. where the risks of bias in medical randomized clinical trials were assessed (19) and by Dechartres et al. using data from 20,920 randomized clinical trials included in Cochrane reviews within all disease areas [48].
A potential limitation of our research is that the choice of sample trials is not strictly random. Our sample of trials originated from 64 dental, oral, and craniofacial meta-analyses and was designed to cover the overall spectrum of clinical oral health research during 1955–2013, therefore, we believe that it represents a realistic cohort for that period. However, as the topics of meta-analyses are often focused on topics of high importance to researches and societies, we cannot exclude that the quality of the trials may be better than the average dentistry randomized clinical trial. Also, it is a drawback that we did not publish our protocol for this observational study.
Another potential limitation is that we did not contact the authors of the studied trials for missing data. A large proportion of the trials were published before the year 2000 when an author’s correspondence information was sometimes not current and not always provided in the publication. Moreover, because we extracted data based on the data reported in the published trials, the actual risk of bias potential was not visible in the majority of risk of bias domains studied due to the poor quality of the reporting identified in the studied trials. As our study did not look at factors that contributed to methodological quality improvement over time, these factors must be considered in future research.
We applied an educated judgement to assign each trial to a primary dental specialty (e.g., dental public health), although the trial could be classified under more than one specialty (e.g., both pediatric dentistry and dental public health).
Conclusions
Our study showed a significant increase over time (1955–2013) in the proportion of trials judged to be adequate in risks of bias, risks of random errors, reporting quality, and methodological quality. However, the proportion of trials judged as having low risk of bias did not exceed 60% in the majority of the risk of bias domains. We found the risks of bias and the quality assessment in the studied trials to be generally unfavorable. That is, in the trials of oral health interventions the methodology and reporting quality were substandard, resulting in a high potential for bias. We believe that a commitment to international initiatives by researchers, journal editors, and manuscript reviewers can contribute to the development of more stringent methodology and more detailed reporting of randomized clinical trials of oral health interventions.
Supporting information
S1 Appendix. Tools and items to assess quality of randomized trials.
https://doi.org/10.1371/journal.pone.0190089.s001
(DOCX)
S2 Appendix. Guidelines for the quality assessment of trials based on the tools most commonly used in health research.
https://doi.org/10.1371/journal.pone.0190089.s002
(DOCX)
S3 Appendix. The Cochrane Collaboration’s tool for assessing risk of bias.
https://doi.org/10.1371/journal.pone.0190089.s003
(DOCX)
S4 Appendix. Guidelines for evaluating the risk of bias of trials.
https://doi.org/10.1371/journal.pone.0190089.s004
(DOCX)
References
- 1. Higgins JP, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. pmid:22008217
- 2. Watzlawick R, Sena ES, Dirnagl U, Brommer B, Kopp MA, Macleod MR, et al. Effect and reporting bias of RhoA/ROCK-blockade intervention on locomotor recovery after spinal cord injury: a systematic review and meta-analysis. JAMA Neurology. 2014;71 (1):91–99. pmid:24297045
- 3. Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA. 2001;285(15):1987–91. pmid:11308435
- 4. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Lancet. 2004;364(9438):911–2. pmid:15364170
- 5. Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010;340:c723. pmid:20332510
- 6. Hooft L, Korevaar DA, Molenaar N, Bossuyt PM, Scholten RJ. Endorsement of ICMJE's Clinical Trial Registration Policy: a survey among journal editors. Neth J Med. 2014;72(7):349–55. pmid:25178768
- 7. Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised clinical trials published in medical journals? A Cochrane review. Syst Rev. 2012;1:60. pmid:23194585
- 8. Needleman I, Worthington H, Moher D, Schulz K, Altman DG. Improving the completeness and transparency of reports of randomized trials in oral health: the CONSORT statement. Am J Dent. 2008;21(1):7–12. pmid:18435368
- 9. Montenegro R, Needleman I, Moles D, Tonetti M. Quality of RCTs in periodontology—a systematic review. J Dent Res. 2002;81(12):866–70. pmid:12454104
- 10. Nieri M, Clauser C, Franceschi D, et al. Randomized clinical trials in implant therapy: relationships among methodological, statistical, clinical, paratextual features and number of citations. Clin Oral Implants Res. 2007;18(4):419–31. pmid:17517060
- 11. Vere J, Joshi R. Quality assessment of randomised controlled trials of dental implant surgery and prosthodontics published from 2004 to 2008: a systematic review. Clin Oral Implants Res. 2011;22(12):1338–45. pmid:21418331
- 12. Harrison JE. Clinical trials in orthodontics II: assessment of the quality of reporting of clinical trials published in three orthodontic journals between 1989 and 1998. J Orthod. 2003;30(4):309–15. pmid:14634169
- 13. Sarkis-Onofre R, Poletto-Neto V, Cenci MS, Pereira-Cenci T, Moher D. Impact of the CONSORT Statement endorsement in the completeness of reporting of randomized clinical trials in restorative dentistry. J Dent. 2017;58:54–59 pmid:28159507
- 14. Pandis N, Polychronopoulou A, Eliades T. An assessment of quality characteristics of randomised control trials published in dental journals. J Dent. 2010;38(9):713–21. pmid:20546823
- 15. Saltaji H, Armijo-Olivo S, Cummings GG, Amin M, Flores-Mir C. Methodological characteristics and treatment effect sizes in oral health randomised controlled trials: Is there a relationship? Protocol for a meta-epidemiological study. BMJ Open. 2014;4(2):e004527. pmid:24568962
- 16.
Higgins JP, Green S, eds. (2008) Cochrane Handbook for Systematic Reviews of Interventions version 5.1.0 The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
- 17. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995;16(1):62–73. pmid:7743790
- 18. Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA. The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol. 2001;54(7):651–4. pmid:11438404
- 19. Reveiz L, Chapman E, Asial S, Munoz S, Bonfill X, Alonso-Coello P. Risk of bias of randomized trials over time. J Clin Epidemiol. 2015;68(9):1036–45. pmid:26227423
- 20. Saltaji H, Cummings GG, Armijo-Olivo S, Major MP, Amin M, Major PW, et al. A descriptive analysis of oral health systematic reviews published 1991–2012: cross sectional study. PLoS One. 2013;8(9):e74545. pmid:24098657
- 21. Saltaji H, Ospina MB, Armijo-Olivo S, Agarwal S, Cummings GG, Amin M, et al. Evaluation of risk of bias assessment of trials in systematic reviews of oral health interventions, 1991–2014: A methodology study. J Am Dent Assoc. 2016;147(9):720–728.e1.32. pmid:27155754
- 22.
ADA. Specialty Definitions. Available at: http://www.ada.org/495.aspx. Accessed July 2015).
- 23. Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J, Ha C, Cummings GG, et al. Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials. PloS One. 2014;9(5): e96920. pmid:24824199
- 24. Armijo-Olivo S, Saltaji H, da Costa BR, Fuentes J, Ha C, Cummings GG. What is the influence of randomisation sequence generation and allocation concealment on treatment effects of physical therapy trials? A meta-epidemiological study. BMJ Open. 2015;5(9):e008562. pmid:26338841
- 25. Furlan AD, Pennick V, Bombardier C, Van Tulder M. Updated method guidelines for systematic reviews in the cochrane back review group. Spine. 20019;34(18):1929–1941.
- 26. Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the Risk of Bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2012;66 (9):973–981. pmid:22981249
- 27. Hartling L, Bond K, Vandermeer B, Seida J, Dryden D, Rowe B. Applying the Risk of Bias tool in a systematic review of combination longacting betaagonists and inhaled corticosteroids for persistent asthma. PLoS Med. 2011;6 (2).
- 28. Armijo-Olivo S, Fuentes J, Rogers T, Hartling L, Saltaji H, Cummings GG. How should we evaluate the risk of bias of physical therapy trials?: a psychometric and meta-epidemiological approach towards developing guidelines for the design, conduct, and reporting of RCTs in physical therapy (PT) area: a study protocol. Syst Rev. 2013;2:88. pmid:24070072
- 29. Armijo-Olivo S, Fuentes CJ, Ospina M, Saltaji H, Hartling L. Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: A descriptive analysis. BMC Med Res Methodol. 2013;13(116):1–19.
- 30. Jadad AR1, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12. pmid:8721797
- 31. Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, et al. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol. 1998;51(12):1235–41. pmid:10086815
- 32. van Tulder M, Furlan A, Bombardier C, Bouter L. Updated method guidelines for systematic reviews in the cochrane collaboration back review group. Spine (Phila Pa 1976) 2003;28(12):1290–9. 33
- 33. van Tulder MW, Assendelft WJ, Koes BW, Bouter LM. Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for Spinal Disorders. Spine (Phila Pa 1976). 1997;22(20):2323–30.
- 34. Moseley AM, Herbert RD, Sherrington C, Maher CG. Evidence for physiotherapy practice: a survey of the Physiotherapy Evidence Database (PEDro). Aust J Physiother. 2002;48(1):43–9. pmid:11869164
- 35. Sherrington C, Herbert RD, Maher CG, Moseley AM. PEDro. A database of randomized trials and systematic reviews in physiotherapy. Man Ther. 2000;5(4):223–6.
- 36. Bizzini M, Childs JD, Piva SR, Delitto A. Systematic review of the quality of randomized controlled trials for patellofemoral pain syndrome. J Orthop Sports Phys Ther. 2003;33(1):4–20. pmid:12570282
- 37. Armijo-Olivo S, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008;88(2):156–75. pmid:18073267
- 38. Juni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2011;323 (7303):42–46.
- 39. Saltaji H. Bias is the key challenge in orthodontic research. Am J Orthod Dentofac Orthop. 2015;148 (1):8.
- 40. Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med. 2012;157 (6):429–438. pmid:22945832
- 41. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008;336 (7644):601–605. pmid:18316340
- 42. Page MJ, Higgins JP, Clayton G, Sterne JA, Hrobjartsson A, Savovic J. Empirical Evidence of Study Design Biases in Randomized Trials: Systematic Review of Met Epidemiological Studies. PloS One. 2016;11 (7):e0159267. pmid:27398997
- 43. Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev. 2017 Feb;2:MR000033. pmid:28207928
- 44. Saltaji H, Armijo-Olivo S, Cummings GG, Amin M, da Costa BR, Flores-Mir C. The impact of selection bias on treatment effect size estimates in randomized trials of oral health interventions: A meta-epidemiological study. J Dent Res. 2017 Aug. [Epub ahead of print] pmid:28813182
- 45. Vale CL, Tierney JF, Burdett S. Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ. 2013;346:f1798. pmid:23610376
- 46. Thomson D, Hartling L, Cohen E, Vandermeer B, Tjosvold L, Klassen TP. Controlled trials in children: quantity, methodological quality and descriptive characteristics of pediatric controlled trials published 1948–2006. PLoS One. 2010;5(9).
- 47. Moseley AM, Herbert RD, Maher CG, Sherrington C, Elkins MR. Reported quality of randomized controlled trials of physiotherapy interventions has improved over time. J Clin Epidemiol. 2011;64(6):594–601. pmid:21144705
- 48. Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, et al. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ. 2017;357:j2490. pmid:28596181
- 49. Schulz KF. Randomized controlled trials. Clin Obstet Gynecol. 1998; 41 (2):245–256. pmid:9646957
- 50. Pandis N, Polychronopoulou A, Eliades T. Randomization in clinical trials in orthodontics: its significance in research design and methods to achieve it. Eur J Orthod. 2011;33 (6):684–690. pmid:21320892
- 51. Pandis N. Randomization. Part 3: allocation concealment and randomization implementation. Am J Orthod Dentofac Orthop. 2012;141 (1):126–128.
- 52. Hamm MP, Hartling L, Milne A, Tjosvold L, Vandermeer B, Thomson D, et al. A descriptive analysis of a representative sample of pediatric randomized controlled trials published in 2007. BMC Pediatr. 2010;10:96. pmid:21176224
- 53. Wetterslev J, Jakobsen JC, Gluud C. Trial Sequential Analysis in systematic reviews with meta-analysis. BMC Med Res Methodol. 2017;17 (1):39. pmid:28264661
- 54. Nguyen TL, Collins GS, Lamy A, Devereaux PJ, Daures JP, Landais P, et al. Simple randomization did not protect against bias in smaller trials. J Clin Epidemiol. 2017;84:105–13. pmid:28257927
- 55. Sterne JA. Why the Cochrane risk of bias tool should not include funding source as a standard item. Cochrane Database Syst Rev. 2013;(12):ED000076. pmid:24575440
- 56. Bero LA. Why the Cochrane risk of bias tool should include funding source as a standard item. Cochrane Database Syst Rev. 2013;(12):ED000075. pmid:24575439
- 57. Gøtzsche P. Comment by Peter Gøtzsche, 9 January 2014. Cochrane Database Syst Rev. 2105
- 58. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78. pmid:17388659
- 59. Pandis N, Turpin DL. Enhancing CONSORT compliance for improved reporting of randomized controlled trials. Am J Orthod Dentofacial Orthop. 2014;145(1):1. pmid:24373645
- 60. Turpin DL. Challenges faced by journal editors. Am J Orthod Dentofacial Orthop. 2015;147(3):285–9. pmid:25726386
- 61. Smail-Faugeron V, Fron-Chabouis H, Durieux P. Clinical trial registration in oral health journals. J Dent Res. 2015;94(3 Suppl):8S–13S.
- 62. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996;276:637–9. pmid:8773637
- 63. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010;63(8):e1–37. pmid:20346624
- 64. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gotzsche PC, Krle AJK, et al. SPIRIT 2013 Statement: defining standard protocol items for clinical trials. Am J Public Health. 2015;38 (6):506–514.
- 65. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gotzsche PC, Krleza-Jeric K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Int Med. 2013;158 (3):200–207. pmid:23295957
- 66. Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS One. 2013;8(7):e66844. pmid:23861749
- 67. Bialy L, Vandermeer B, Lacaze-Masmonteil T, Dryden DM, Hartling L. A metaepidemiological study to examine the association between bias and treatment effects in neonatal trials. Evid Based Child Health. 2014;9(4):1052–9. pmid:25504975
- 68. Giannobile WV. Improving clinical trials in dentistry. J Dent Res. 2015;94(3 Suppl):6S–7S.
- 69. Saltaji H, Flores-Mir C, Major PW. Letter to the editor, "Clinical trial registration in oral health journals". J Dent Res. 2015;94(3 Suppl):103S.