All Nations Depend on the Global Knowledge Pool – Analysis of Country of Origin of Studies Used for Health Technology Assessments in Germany

Background Health Technology Assessments (HTAs) are used to inform decision-making and their usefulness depends on the quality and relevance of research and specific studies for health-policy decisions. Little is known about the country of origin of studies used for HTAs. Objective To investigate which countries have made the largest contributions to inform health policy decisions through studies included in HTAs in Germany. Methods The country of origin was extracted from all studies included in HTAs of the German Institute for Quality and Efficiency in Health Care, (IQWiG), published from 2/2006 to 9/2010. Studies were ranked according to the total number of studies per country, adjusted for population size, gross domestic product (GDP), and total health expenditure. Results 1087 studies were included in 54 HTA reports. Studies were assigned to 45 countries. Most of the studies (27%) originated from the United States (USA), 18% were multinational, followed by 7% from the United Kingdom (UK) and 5% from Germany. Nordic countries led the ranking when adjusting for population size/million (ranks 1-3,6,9/45 countries), GDP/billion US$ (1,2,5,9,14/45), or health expenditure/billion US$ (1,3,5,12,13/45). The relative contribution of the UK was stable in the analyses when adjusted for population size (7/45), GDP (7/45), and health expenditure (9/45), whereas the USA (13, 18, and 30/45) and Germany (17, 19, and 21/45) dropped in the ranking. Conclusions More than half of the studies relevant for evidence-informed decision-making in Germany originated from the USA, followed by multinational research and the UK. Only 5% of the studies originated from Germany. According to our findings, there appears to be some discrepancy between the use of globally generated evidence and the contribution to the knowledge pool by individual countries.


Introduction
Health Technology Assessment (HTA) is a multidisciplinary field that systematically investigates the clinical efficacy and effectiveness, safety, cost, cost-effectiveness of health care interventions, as well as organizational implications, social consequences, and legal and ethical considerations [1]. HTA plays a crucial role in health systems throughout the world, supporting decisionmaking on how to access, distribute and implement technologies and innovation. Health technologies include pharmaceuticals, devices, diagnostics and treatments, and other clinical, public health, and organizational interventions. HTA reports include a systematic review of the clinical evidence in a transparent, unbiased, and robust manner in order to quantify the potential benefits and risks of health technologies [2,3].
The Federal Joint Committee (G-BA) and the Institute for Quality and Efficiency in Health Care (IQWiG) are the most relevant HTA institutions in Germany. G-BA is the central selfgoverning body within the German health care system, a committee comprising important stakeholders in the health care system, e.g. health care providers and statutory health insurance funds. 'It issues directives for the benefit catalogue of the statutory health insurance funds for more than 70 million insured persons and thus specifies which services in medical care are reimbursed' [4]. IQWiG is an independent scientific institute and investigates the benefits and harms of medical interventions, usually to inform the decisions made by the G-BA [5].
A variety of publications are available on the analysis of the research activities of different countries. These are mainly bibliometric analyses focusing on Cochrane reviews [6] or specific medical fields such as anesthesia [7,8], dentistry [9], nuclear medicine [10], pharmacological trials [11], primary care [12], radiology [13] or surgery [14].
In addition to these scientific analyses, inclusion in an HTA report emphasizes and measures the impact of clinical research on evidence-informed decision-making and thus the value in research expenditure as well as the return on investment in patient-oriented research. Consideration of studies for HTA reports indicates the quality and relevance of research for health-policy decisions. However, little is known about the country of origin of studies used for HTAs. It is therefore of interest to analyze which countries conduct studies used for HTAs and hence provide research findings relevant to health-policy decisions.
The objective of our study is to investigate the country of origin of clinical studies included in HTAs in a specific country, using Germany as an example.

Search strategy
The IQWiG website www.iqwig.de, which provides an overview of IQWiG projects from 2004 onwards, was searched in September 2010 for completed HTA reports. IQWiG products comprise benefit assessments (full reports and rapid reports), working papers, appraisals of clinical practice guidelines (CPGs), as well as health information. Products pursuant to 1 35a of the German Social Code Book V (assessments of dossiers submitted by pharmaceutical companies) and health economic evaluations were not considered in this analysis, as these types of documents were only published after our project had been completed. Both full reports and rapid reports are HTAs applying the same methods with regard to the actual content of the report. However, they differ in terms of procedures, e.g. in contrast to rapid reports, preliminary versions of full reports are discussed in a public hearing. Working papers provide information on relevant developments in health care or methodological issues. Health information (e.g. feature articles, fact sheets and research summaries) is produced to inform the general public. Appraisals of CPGs aim to describe current health care standards. The latter three types of products are not considered to be HTAs [5].

Inclusion criteria
Eligible HTA reports were benefit assessments (full reports and rapid reports) published on the IQWIG website between 2004 and September 2010.
To be included in an HTA report prepared by IQWiG, studies need to contain data relevant to the specific project, i.e. report patient-relevant outcomes such as mortality, morbidity, adverse events and quality of life related to the health technology under investigation [5]. In our analysis we considered all studies meeting the inclusion criteria of the HTA reports. No further criteria were applied.

Exclusion
We excluded IQWiG products not considered to be HTA reports, such as working papers, health information articles or appraisals of CPGs.

Data extraction
Clinical studies were ascribed to a country as obtained from data extraction tables in published HTA reports. The HTAs reported the country of origin of the studies included based on the location of the study center(s) of the specific study. In cases where this was unclear to the HTA authors, the origin of a study was classified as ''unknown''. A study was classified as ''multinational'' in cases where multiple study centers in various countries (or even a continent) were reported. Neither multinational studies nor those of unclear geographic origin were considered for the adjustments of the subsequent analyses. The overall number of studies included in the HTA reports from one country was used in the analysis and calculated in Excel 2010 (Microsoft Corporation, USA). A ranked order was displayed for further comparisons and analyses.

Quality assessment
Inclusion in an HTA report was used as a quality indicator. These reports are conducted following a rigorous assessment of the risk of bias (e.g. assessment of study design, allocation concealment) and follow recognized standards of evidence-based medicine. They typically included randomized controlled trials (RCTs) for the evaluation of clinical effects, as well as other study designs (e.g. diagnostic accuracy studies) [5]. We did not conduct any additional quality assessments of the primary studies included in the HTA reports.

Analysis
We analyzed how the studies included were distributed across countries adjusted for size of population (studies per 1 million population), gross domestic product (GDP, studies per 1 billion US$ GDP) and national spending on health (studies per 1 billion US$ health expenditure).
Data from the International Monetary Fund (2010) [15] were extracted to adjust for GDP. To adjust for the size of population and total health expenditure, data from the World Health Organization (WHO) report 'World Health Statistics 2010' [16] were obtained. Data for the Republic of China (ROC) were not included in the WHO report and therefore obtained from the Central Intelligence Agency report 'The World Factbook' 2010 [17] and 'Lists of countries by total health expenditure 2007' [18].
For a more detailed analysis we divided the reports into those on drugs and those on non-drug interventions. As an example, the rankings of countries of origin were analyzed according to health expenditure.
Results 81 projects were finalized and published between 06/02/2006 and 11/09/2010. 54 completed HTA reports (full and rapid reports) were included. 26 of these were reports on drugs and 28 on non-drug interventions (table 1), e.g. surgical procedures and diagnostic devices. 27 projects were guideline appraisals, working papers and health information and therefore excluded from the analysis (table 2).
In total, 1087 clinical studies were included in these 54 reports (20.1 studies per report on average). Six reports included more than 50 studies. Of all studies, 843 were assigned to 45 countries while 193 studies (18%) were multinational. 51 studies (5%) were classified as unknown, which was mainly due to the fact that the IQWiG reports lacked information on the country of origin [19,20].    When adjusted for health expenditure, most countries remained stable in their ranking ( figure 8 and 12). Only 6 of 28 countries with sufficient data showed differences of more than 20 places in the ranking of research on drugs and non-drug interventions.

Influence of research topic
China, ROC, Ukraine and USA were more productive in drug research, whereas New Zealand and South Korea were more productive in non-drug research (table 3).

Discussion
HTA reports are important for translating research into policymaking. They aim to inform policy-making comprehensively and with a minimized risk of bias. In order to do so, the global pool of clinical studies, often described as the ''body of evidence'', has to be exploited. The objective of our study is to investigate the country of origin of clinical studies included in HTAs in a specific country, using Germany as an example. The results reflect the national contributions with a particular focus on research relevant to supporting health policy, as HTA is defined as a policy support tool, and within this framework reports are produced to provide answers to relevant questions.

Summary of findings
The findings of our analysis confirm the leading role of the USA and UK as major contributors to the global pool of clinical studies providing relevant information for health-policy decisions. These are followed by a large proportion of multinational studies. Germany contributes only 5% of the research input included in IQWiG reports. When adjusted for population size or economic variables, Nordic countries dominate the ranking, while the relevance of the USA and Germany decreases noticeably. After adjustments, the position of the UK is more stable than that of the USA and Germany.
One limitation in the present analysis is that neither studies of unknown origin nor multinational studies were analyzed in depth, since data on the distribution of countries in the study reports were not reported in detail in the IQWIG reports. We stratified studies from drug and non-drug reports and found that most of the multinational studies were within the pool of drug studies. This might be explained by the fact that to be granted widespread approval for the same drug, a pharmaceutical company must submit approval studies to different regulatory authorities in different countries applying different legislation. It therefore makes sense to conduct large multinational studies in a variety of   Germany performs poorly compared with other countries of similar economic power. This finding is especially surprising as one would expect German HTA reports to include a higher proportion of German studies, as clinical research in a given country is more likely to address the same research questions of relevance as investigated in national HTA reports (e.g. for demographic or epidemiologic reasons). The underrepresentation of German studies might be caused by limited clinical research activity or by a lower output of studies relevant to health policy decision-making.
Overall, our data show vast differences between contributing countries. These differences are particularly striking when adjusted for country population, GDP or health care expenditure, i.e. showing national contributions per capita or per money unit. Rich countries such as Germany show a poor contribution to the global knowledge pool, which is in sharp contrast to the dependence of these countries on global knowledge for decision-making.
It should be noted that regardless of the size of a contribution, all countries are dependent on knowledge generated globally. In countries such as the USA, which contributes a large number of studies to the knowledge pool (in our analysis: 27%), users of information might be tempted to base their decision-making process on their 'own' trials. However, succumbing to this temptation is likely to cause serious problems. First, ignoring large parts of the available evidence is a waste of resources and would introduce bias, as decision-making in health care should be based on all of the available evidence. Second, stratification for medical specialties would change the country league tables considerably, in some fields even dramatically [6]. The obvious           conclusion is that all countries should consider themselves as contributors to and beneficiaries from the global body of evidence.

Research results in context
Comparison of the present analysis to previous ones largely confirmed earlier findings of studies comparing national activities in patient-oriented research. There are many similarities, regardless of whether an analysis was based on studies cited by Cochrane reviews [6,21], pharmacological trials [11], or high-ranking publications in primary care [12], surgery [14], anesthesia [7,8], nuclear medicine [10], or dentistry [9]; Nordic and Anglo-American countries usually take the lead.
Gluud and Nikolova [21] described various factors that have to be taken into account to explain a country's scientific output. Population size, economic wealth and research expenditure are obvious and relatively simple factors to include, while historical and cultural aspects are more difficult to cover. Factors with a strong impact are the research expenditure of pharmaceutical companies, as well as collaboration between researchers and industry. National government budgets play a key role in the funding of clinical research, as well as in the regulation of RCTs on drugs, especially the time taken to obtain regulatory approval [11].
There are numerous other factors possibly explaining the lack of clinical studies; for example, the promotion policies of clinical research, the funding situation, and the specific requirements concerning the availability of specially trained and experienced medical and research staff might be very different from those in other countries [12,[22][23][24].
Numerous hypotheses for future testing can be derived from our analysis. It would be interesting to see whether our findings are supported by similar research in other countries. In addition, the cultural component should be investigated, as well as other questions related to decision science, for example, when to use national or multinational studies or foreign studies from similar or very different countries.

Strengths and limitations
Our analysis differs from existing approaches. We did not merely conduct a bibliometric analysis, but investigated which countries contributed the largest proportion of studies included in HTAs in Germany. Studies had thus undergone a rigorous quality assessment according to IQWiG methods [5]. At the same time our analysis considered the relevance of the research output. All 1087 studies assessed by IQWiG to inform health-policy decisions were included, reflecting the performance of different countries in producing research relevant to decision-making.
One limitation in the present analysis was that neither studies with unknown origin nor multinational studies were analyzed in depth, since data on the distribution of countries in the study reports were not reported in detail in the IQWIG reports.
Some results need to be interpreted with caution. As an example, when weighted by national health expenditures, the Ivory Coast reached sixth place. This is rather misleading, as only one study [25] was included in an HTA report on test accuracy in ultrasound screening in pregnancy [26]. However, due to very low national health expenditure, the Ivory Coast achieved this relatively high ranking.
There might potentially be other factors associated with the output of clinical trials than population size, GDP, or health expenditure, such as promotion policies within departments, the number of universities, their programs, and funding in this field [12,24], which were not analyzed in our study.

Conclusion
According to our findings, there is a discrepancy between the use of globally generated evidence and the contribution to the knowledge pool by individual countries. In absolute numbers, by far the most studies relevant to evidence-informed decision-making in Germany were conducted in the USA, followed by multinational research and the UK.
From the perspective of contributing countries, absolute numbers are misleading as they imply contributions, which do not exist on a per capita level but are merely due to a ''large country effect''. Our study confirms that some small countries have a remarkable input in relation to their population size, health expenditure, or GDP. In contrast, some larger rich countries profit from these imbalances. Germany belongs to this category, with only 5% of the studies in German HTA reports actually conducted in Germany. Even for countries with larger contributions, it would be unwise to ignore the globally available evidence, and even harmful in certain fields because of much richer information outside their own countries.
The often-noted and criticized lack of studies for many relevant clinical questions is a consequence of many countries not taking the responsibility to contribute to global knowledge on the same scale as they are benefitting from it. Limited resources are a crucial issue in all research fields. In the medical field the existing gaps in knowledge are not a pure research problem but have a serious impact on health care decisions on an individual and public health level. A better-balanced contribution of all countries to the generation of global knowledge and its translation into policy and practice are urgently required to eradicate these deficits [27].