Are Japanese Randomized Controlled Trials Up to the Task? A Systematic Review

Objectives Despite increasing numbers of RCTs done in Japan, existing international databases fail to capture them, and detailed information on the quality of Japanese RCTs is still missing. This study assessed the characteristics and quality of Japanese RCTs and analyzed factors related to their quality. Methods All RCTs conducted in Japan, and published as original articles that assessed the effect of healthcare interventions on humans in 2010, were included. We excluded study protocols, conference abstracts, and comments. In addition, quasi-RCTs were excluded. Data were independently abstracted and assessed by two of the authors and disagreements were resolved by consensus. The quality of Japanese RCTs randomly sampled was assessed using the method guidelines for systematic reviews from the Cochrane Back Review Group. The factors affecting RCT quality were analyzed using a logistic regression model. Results A total of 1013 RCTs conducted in Japan were published in 2010. The majority was small-scale (55% of RCTs with sample size less than 50). Eighty percent of RCTs had no information on the funding source and only 8% had been registered before their implementation. RCTs not indexed in international databases were a moderate number (118 RCTs: 37.7% of non-indexed RCTs were of high quality). Surgical intervention studies for external causes of morbidity and mortality with a large sample size, trial registration and a large number of arms were most likely to be of higher quality. Conclusion Despite a considerable number of RCTs conducted in Japan, their quality is not satisfactory in some domains. On the other hand, there are high-quality, non-indexed RCTs. The full disclosure of trial information and quality control of clinical trials are urgently needed in Japan.


Introduction
Randomized controlled trials (RCTs) are widely regarded as providing reliable evidence on the benefits of healthcare interventions.The number of published RCTs is rapidly increasing worldwide [1], but the validity of all RCTs depends on the underlying methodologies (e.g.allocation concealment, blinding, intention-to-treat analysis, and loss to follow-up) which are considered to be important to the quality of RCTs [2].Previous evidences showed that poor-quality trials might lead to the biased estimation of the true effects of interventions [3][4][5].
A systematic review of trends in the methodological quality of RCTs showed there were improvement of quality in some methodological aspects over time, but others did not significantly improve [6].Many methods for improving their quality have been proposed.For example, the CONSORT Statement was developed to provide the guideline to improve the quality of reporting [3,7], and journal endorsement of CONSORT has been shown to be positively related with better quality of reporting of randomized trials [3,7].
The number of clinical trials cited in 2007 and reported during the last 60 years in Japan was ranked 7 th and 9 th in the world, respectively.While RCTs conducted in Japan have been partly reported among international journals and databases, most have been paid little attention from an international perspective [1,8].For example, the majority of Cochrane systematic reviews did not search Japanese databases because of the language barrier [9,10].Therefore, even though Japan has made a relatively large contribution to the evaluation of healthcare interventions, comprehensive information and usefulness on RCTs conducted in Japan is still missing.The major objectives of the present study are therefore to systematically review RCTs conducted in Japan.We identified the number, general characteristics, and quality of all Japanese RCTs published in Japan in 2010, and analyzed factors associated with their quality.

Search strategy
RCTs conducted in Japan and published in 2010 were identified through two broad sources.Firstly, we searched the domestic Japanese database: the Japan Medical Abstract Society Database (January 1 to December 31, 2010).This database includes studies published in Japanese journals, which are written in Japanese sometimes with English summary.Secondly, we searched international databases: MEDLINE (January 1 to December 31, 2010), EMBASE (January 1 to December 31, 2010), CINAHL (January 1 to December 31, 2010) and PsycINFO (January 1 to December 31, 2010).These databases include Japanese papers written in English, as well as only English summaries for a very limited number of papers written in Japanese.We included search terms related to RCTs such as ''single blind'', ''double blind'', ''placebo'', ''randomized controlled trials'', ''randomization'' and ''random* (wild card)'', and articles published from Japan (final search: November 11, 2011).The search terms are presented in Appendix S1.The search strategies were designed by the first author and a librarian and were appropriately modified for each database.This systematic review has no protocol for the study.

Inclusion and exclusion criteria
All papers reporting RCTs conducted in Japan and published as original articles in 2010, which assessed a healthcare intervention, were included.The definition of RCTs followed Cochrane review guidelines.We considered RCTs that took place in Japan and for which the affiliation of the first author was located in Japan.We excluded study protocols, conference abstracts, and comments, as well as quasi-RCTs such as allocation by date of birth, day of the week or medical record number.

Study selection and data extraction
Eligible studies were identified by one reviewer based on available information in the title, abstract, and/or full text.A second reviewer independently checked articles that were judged by the first reviewer to be suitable for inclusion.Discrepancies were resolved by discussion and consensus.The title and abstract were initially scanned for relevance.If it was difficult to determine the trial characteristics, full texts were obtained and checked.The data for the following characteristics of each trial were extracted in this way: type of intervention, disease and conditions, study design, type of control group, number of arms, sample size, registration, funding sources and publisher.

Assessment of study quality
Sixty percent of all RCTs identified were randomly selected to evaluate their methodological quality.This sample size was determined on the basis that it would be sufficient to identify more than a 10% difference in proportions of high quality between indexed and non-indexed groups using a chi-squared test, given the available total number of selected RCTs [11].Two independent reviewers assessed the quality of each trial using the 2009 updated method guidelines for systematic reviews from the Cochrane Back Review Group, which can be regarded as the detailed version of the risk of bias tool created by the Cochrane Systematic Reviews Group, and has confirmed internal/external validity and intra-rater/inter-rater reliability [12,13].The 2009 updated method guidelines for systematic reviews consist of 12 criteria for assessing the quality of an RCT based primarily on published reports.Disagreements were resolved by consensus, and a third reviewer was consulted if disagreements persisted.We did not attempt to contact the corresponding authors to assess the quality of the data and based our review only on reported information.RCTs for which blinding was impossible but outcome measurements were objective bio-marker based assessments (such as mortality) were scored ''low risk of bias'' according to Cochrane review guidelines.RCTs with subjective outcomes (such as quality of life scales) and no blinding were scored ''high risk of bias'' according to Cochrane review guidelines.This categorization is consistent with Cochrane review guidelines, which state that a study can be scored ''low risk of bias'' when there was ''no blinding of outcome assessment, but the review authors judge that the outcome measurement is not likely to be influenced by lack of blinding.''[14].

Data analysis
We first identified the number of RCTs conducted in Japan in 2010 and examined their characteristics.The quality of the randomly sampled Japanese RCTs was assessed using the 2009 updated method guideline for systematic reviews [12,14].To compare the quality of Japanese RCTs according to the type of journals, they were classified into three categories: those indexed in international databases and published in international journals, those indexed in international database with only English summary, and those not indexed in international databases and published in Japanese journals only.In statistical analysis, the chisquare test or Fisher's exact test was used with Holm's multiple Table 1.Diseases and condition of all RCTs conducted in Japan (N = 1,013).

Diseases and conditions Number Percent
Diseases of the circulatory system 161 15.9%  corrections.Finally, the factors affecting RCT quality were analyzed using a logistic regression model.The dependent variable was a binary variable indicating whether the trial was high quality or not.The studies were rated as having a ''low risk of bias'' when at least six of the 12 items were scored as ''yes''.Studies with less than six items having low risk of bias or with serious flaws were rated as having a ''high risk of bias'' following the method of Furlan et al. ( 2009) [12].Under this method, a serious flaw is defined as a drop-out rate in one group of over 80% and a noncompliance rate of over 50%.The independent variables were disease and conditions, type of intervention, comparison group, study design, sample size, registration of the study protocol, number of arms, international indexing and publisher and description of funding sources.Because it was not possible to categorize sample size on the basis of its statistical adequacy for every paper, sample size was instead divided into several categories.Sample size was categorized based on observable breakpoints seen in the distribution of sample sizes, calculated using kernel density estimation.Statistical analyses were performed with R version 2.15.1 (Free Software Foundation's GNU General Public License).A p-value less than 0.05 was considered statistically significant.

Results
Figure 1 shows the flow diagram describing the study identification strategy.The Japan Medical Abstract Society database, MEDLINE, EMBASE, CINAHL, and PsycINFO were searched, and 2957 articles were found (EMBASE: 1174; Japan Medical Abstract database: 903; MEDLINE: 675; PsycINFO: 173 and CINAHL: 32).1013 articles met the inclusion criteria and 1944 articles were excluded because they were duplicates (79.9%), not RCTs (11.0%, of which 5.9% were quasi-RCTs) or had non-Japanese authorship (9.1%).The 1013 articles included consisted of 378 articles (37.3%) indexed in international databases and published in international journals, 105 articles (10.4%) indexed in international databases and published in Japanese journals and 530 articles (52.3%) non-indexed in international databases and published in Japanese journals.
The three most frequent diseases and conditions based on ICD-10 were diseases of the circulatory system (15.9%),diseases of the digestive system (11.5%), and endocrine, nutritional and metabolic diseases (11.2%) (Table 1).Type of medical intervention is shown in Table 2.The proportion of main types was 49.9% for drug trials, 10.5% for physical therapy, and 8.5% for devices.
Table 3 shows the design characteristics.The most frequent type of comparison group, study design and number of arms were the head-to-head trial (45.0%), parallel group design (75.2%) and two arms (77.1%), respectively.The median sample size was 42 (interquartile range: 23 to 101).While the number of subjects in 54.5% of RCTs was less than 50 (and notably, the proportion with sample size #20 was 17.3%), the sample size in 25.9% of studies was more than 100.
Of a total of 605 randomly-selected articles, only 131 articles (21.7%) reported their funding sources.The three most frequent reported funding sources were the Japanese government (41.2%),profit organizations (e.g. pharmaceutical companies) (38.9%), and foundations (9.2%).Other possible sources of funding were selffunding (0.7%), and financing from universities or research institutions (2.6%).Trials in which funding sources were unreported included drug (46.6%),physical therapy (10.8%) and device trials (10.3%).The proportion of description on trial registration was only 8.3%.The proportions of reported funding resources and trial registration among articles indexed in international databases were significantly higher than those nonindexed (p,0.001).
The results of methodological quality of Japanese RCTs are presented in Figure 2. Of all 605 RCTs, 265 RCTs (43.8%) had more than six positive scores and without any fatal flaw (i.e., marked as ''yes'' for low risk of bias).The mean number of high quality item was 5.29 out of 12 domains.The proportion of ''unclear'' was relatively high and varied from 0% to 91.7%.
Table 4 shows the comparison of quality of RCTs classified based on international indexing and publishers (explanations for classification of groups are listed in Table 4).The proportion of studies assessed as high quality that included the following criteria was significantly higher among Group A than Group C: method of randomization (item 1) (p = 0.02), allocation concealment (item 2) (p = 0.02), selective outcome reporting (item 8) (p,0.001) and similarity of baseline (item 9) (p = 0.004).The proportion of RCTs we assessed as high quality was 50.5% in Group A, 50.0% in Group B and 37.7% in Group C, respectively.Statistically significant difference was observed between Group A and C (p = 0.01).
Table 5 shows the result of regression analysis to determine the factors related to the high quality of RCTs.Surgical intervention studies for external causes of morbidity and mortality with a large sample size, trial registration and a large number of arms were most likely to be of a higher quality.As to type of disease and conditions, disease of the respiratory system and external causes of morbidity and mortality were higher quality than ''others'' category (p = 0.02, respectively), while as to type of intervention, device and procedure/surgery were associated with high quality (p = 0.001 and p,0.001, respectively) compared with a drug intervention.Larger sample size (100-200 participants) was correlated to quality (p = 0.004).Registration of the study protocol before trial implementation was positively correlated with quality (p = 0.003), as was number of arms (p = 0.03).However, quality was not related to international indexing in international databases and publishers compared to non-indexed Japanese databases (indexed Japanese journal: p = 0.07 and indexed foreign journals: p = 0.06).

Discussion
This is the first study to assess the characteristics, quality and related factors of RCTs conducted in Japan.Despite a considerable number of RCTs conducted in Japan, their quality is not satisfactory in some domains.A total of 1013 RCTs conducted in Japan were published in 2010.Drug therapy was the most frequent type of intervention in Japan (49.9%).This result is similar to surveys among other countries, although the proportion of drug trials in Japan was higher than that in the UK, where 35% of all trials between 1980 and 2002 were drug trials [15][16][17].
Recently in Japan, clinical trial publications used by pharmaceutical companies as marketing tools for the drug Valsartan were withdrawn due to doubtful claims about the effectiveness of the drug [18,19].As a result, full disclosure and quality control of clinical trials has become a pressing research issue in Japan [20].
Although drugs have been central to many breakthroughs in healthcare, non-drug interventions such as early diagnosis and prevention, diets and nutrition also offer many benefits and tend to be cost-effective.According to the Global Burden of Diseases, Injuries, and Risk Factors Study 2010 in Japan, the top three causes of disability-adjusted life years (DALYs) in 2010 were lower back pain, cerebrovascular disease, and ischemic heart disease.The study strongly recommended the effectiveness of preventive interventions to improve DALYs [21][22][23].Potential efficiency and efficacy of non-drug interventions are largely ignored in Japan.In the US, the Institute of Medicine Committee on Comparative Effectiveness Research Prioritization recommended research priorities by types of intervention, which put emphasis on systems of care, standards of care, behavioral treatment, prevention and procedures, as well as drugs [24,25].In addition, from the perspective of comparative effectiveness, head-to-head assessment, rather than comparison with placebo, is urgently needed to ensure effective decision making for both clinical practice and health policy [25].However, according to this study, the proportion of head-to-head assessment in Japanese trials remained at 45%.
An interesting characteristic of Japanese RCTs is their sample size: more than half had a sample size less than 50 and the proportion with sample size #20 was over 17%.Although the sample size of trials should be decided based on both precision and statistical power, very small RCTs are often insufficient to evaluate benefits and risks of healthcare, their results have little generalizability [26,27] and in large numbers they can introduce bias into meta-analysis [28].Only 8.3% of Japanese trials were registered to a clinical trial registry system.In 2005, the International Committee of Medical Journal Editors required that all clinical trials be registered with a public registry before the enrolment of the first subject.The registry situation in Japan does not conform to this information disclosure rule, which is a key element in medical decision-making and preventing publication bias [29,30].Japan has three main registry systems, but there is no information on their coverage levels or on their relationship to other registry systems, and standardized key search words are not well developed.Also, it is difficult in some systems to search the registration date and more importantly the publication date of trials.Improvement of the Japanese registry system, and registry requirements for clinical trials, is urgently needed.In addition, approximately 80% of Japanese RCTs had no information on funding sources and around half of the studies that did not include funding information were drug trials.A systematic review of conflict of interests showed that systematic biases favored products made by the company funding the research [31].Thus, the result in this study implies that there might be hidden conflicting interests and publication bias in Japan.
The present study showed that 44% of Japanese RCTs were high quality, as indicated by the 2009 updated method guidelines for systematic reviews [12].Although RCTs not indexed in international databases were lower in quality than indexed RCTs in some items such as method of randomization, allocation concealment, selective outcome reporting and similarity of baseline, the overall quality was not significantly different between RCTs in indexed Japanese/foreign journals and non-indexed RCTs after adjusting for other factors in a logistic regression model.The non-indexed RCTs contained a considerable number of studies with high quality RCTs (37.7%).This result suggested that systematic reviewers should carefully examine non-indexed Japanese RCTs and should consider including both international and Japanese databases in their search strategies.
Another aspect of quality of RCTs was the reporting quality.In this study, a high proportion of ''unsure'' results were observed in each item of the 2009 updated method guidelines for systematic reviews.CONSORT statement was developed to respond to the demand for more sophisticated reporting quality of trials [32].Several studies indicated that pre-CONSORT trials were inferior to post-CONSORT studies in several aspects of methodology and found that articles which adapted CONSORT statement showed the remarkable improvement in some specific reporting methods such as allocation concealment [6,33,34].To improve the quality of Japanese RCTs, and increase the Japanese presence in the international academic community, the CONSORT statement should be encouraged for authors and publishers in Japan.Greater efforts are needed to improve the quality of Japanese RCTs as well as to include them in international databases and further research is needed to assess changes in quality longitudinally and conduct country by country comparisons.

Table 4 .
Comparison of quality of Japanese RCTs classified by international indexing and journal publisher's location.The proportion of ''yes'' in each criterion of the 2009 updated method guidelines for systematic reviews.Group A: articles indexed in internationally databases and published in international journals.Group B: articles indexed in internationally databases and published in Japanese journals.Group C: articles non-indexed in international databases and published in Japanese journals.

Table 5 .
Factors related to quality of Japanese RCTs.