Health research is one mechanism to improve population-level health and should generally match the health needs of populations. However, there have been limited data to assess the trends in national-level cardiovascular research output, even as cardiovascular disease [CVD] has become the leading cause of morbidity and mortality worldwide.
Materials and Methods
We performed a time trends analysis of cardiovascular research publications (1999–2008) downloaded from Web of Knowledge using a iteratively-tested cardiovascular bibliometric filter with >90% precision and recall. We evaluated cardiovascular research publications, five-year running actual citation indices [ACIs], and degree of international collaboration measured through the ratio of the fractional count of addresses from one country against all addresses for each publication.
Results and Discussion
Global cardiovascular publication volume increased from 40 661 publications in 1999 to 55 284 publications in 2008, which represents a 36% increase. The proportion of cardiovascular publications from high-income, Organization for Economic Cooperation and Development [OECD] countries declined from 93% to 84% of the total share over the study period. High-income, OECD countries generally had higher fractional counts, which suggest less international collaboration, than lower income countries from 1999–2008. There was an inverse relationship between cardiovascular publications and age-standardized CVD morbidity and mortality rates, but a direct, curvilinear relationship between cardiovascular publications and Human Development Index from 1999–2008.
Cardiovascular health research output has increased substantially in the past decade, with a greater share of citations being published from low- and middle-income countries. However, low- and middle-income countries with the higher burdens of cardiovascular disease continue to have lower research output than high-income countries, and thus require targeted research investments to improve cardiovascular health.
Citation: Huffman MD, Baldridge A, Bloomfield GS, Colantonio LD, Prabhakaran P, Ajay VS, et al. (2013) Global Cardiovascular Research Output, Citations, and Collaborations: A Time-Trend, Bibliometric Analysis (1999–2008). PLoS ONE 8(12): e83440. https://doi.org/10.1371/journal.pone.0083440
Editor: Lutz Bornmann, Max Planck Society, Germany
Received: September 17, 2013; Accepted: November 13, 2013; Published: December 31, 2013
Copyright: © 2013 Huffman et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported by Global Forum for Health Research and the Fogarty International Center. Neither participated in the study design, analysis, writing, or decision to publish. MDH is supported by a National Heart, Lung, and Blood Institute (NHLBI) Pathway to Independence award [1K99HL107749-01A1]. GSB is supported by the Fogarty International Center of the National Institutes of Health [K01TW008407]. PP and VSA are supported by a Wellcome Trust Capacity Strengthening Strategic Award to the Public Health Foundation of India and a consortium of United Kingdom universities. DP receives partial salary support from a contract award [HHS N268200900026C] from NHLBI and a grant award [1D43HD065249] from National Institute of Child Health and Human Development. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Competing interests: The authors have declared that no competing interests exist.
Economic development has been associated with transitions in health, notably shifts from communicable diseases to non-communicable, chronic diseases [NCDs] including cardiovascular diseases [CVD] . Health research is one mechanism to improve population-level health and should generally match the health needs of populations. Even though CVD is the leading cause of death and disability worldwide, there have been limited data to assess the trends in country-level cardiovascular research output , . Bibliometrics, a research evaluation methods used to quantitatively evaluate scientific literature, can be used to measure trends in research productivity over time. Prior cardiovascular bibliometric research has been largely confined to high-income country settings from more than 20 years ago .
Our objective was to evaluate trends in global cardiovascular research publications disaggregated by country from 1999 to 2008, the relative importance of these publications as measured by average number of citations per paper, and the degree to which countries collaborate. An additional aim was to compare cardiovascular research publications against measures of country-level development and CVD burden.
Bibliometric Filter Creation and Testing
We created, tested, and revised a cardiovascular bibliometric filter using a previously published approach to achieve >90% precision (specificity) and recall (sensitivity) to capture cardiovascular research articles, reviews, and conference proceedings from the Web of Knowledge (1999–2008) , . Precision represents proportion of filter output considered to be cardiovascular research, and recall represents the proportion of known cardiovascular research publications retrieved . Testing was undertaken by two cardiovascular research specialists (MDH, GSB). We amended a previously-used definition of cardiovascular research  to develop this filter:
Study of the cardiovascular system (heart and blood vessels) and its functions in health and disease, including heart disease and stroke, invasive/interventional cardiology, cardiac surgery, and vascular surgery, which is the practice of diagnostic and therapeutic procedures that involve entry into the heart and major blood vessels.
With each filter iteration (developed by MDH), the tester (GSB) was given a random sample of 400 titles of publications drawn from the whole retrieval of the cardiovascular filter, mixed with 100 titles of publications from cardiovascular departments. The tester was asked to decide whether titles represented cardiovascular research publications. Iterative amendments were made to the filter. Keywords or headings were included or excluded on the basis of the findings, and tester was given new testing samples. Our initial filter produced precision and recall scores of 0.797 and 0.874, respectively, and after five iterations, our final filter produced precision and recall scores of 0.905 and 0.903.
Bibliometric Filter Application
We applied our final bibliometric filter to the Thomson Reuters Web of Knowledge Science Citation Index Expanded [SCI-EXPANDED] and Social Sciences Citation Index [SSCI] for publication years 1999–2008 . We searched for all articles, reviews, and conference proceedings as citable items over the study period. After removing duplicates, we merged this output with the corresponding citation report from the Web of Knowledge for each publication to create actual citation indices [ACIs], the average of the running five-year citation count post-publication (total citations five years post-publication divided by five), including the year of publication, to provide comparable citation estimates over time through 2012 .
Bibliometric Filter Analysis
We performed geographical analysis through integer counts and fractional counts based on publication years, as we have previously published . In brief, integer counts give a country a score of one for all authors from that country, which overestimates the number of publications by approximately 20%, depending on publication year and subject . Fractional counts divide the number of publication addresses from an individual country (numerator) by the total number of publication addresses (denominator). For example, comparing a manuscript with one Chinese author and 9 US authors with a manuscript with 9 Chinese authors and one US author, the former example would be a 0.1 fractional count for China and 0.9 for the US, and the latter would be a 0.9 fractional count for China and 0.1 for the US. No weight is assigned by author order or reprint addresses . We estimated international collaboration by the ratio of fractional to integer counts to derive a fractional contribution estimate.
Analysis by Income, Population, Development, and Burden of Disease
We estimated the relationship between time and cardiovascular publications through linear regression of log-transformed data within the entire dataset by World Bank Income group, including members of the Organization for Economic Cooperation and Development [OECD] . A list of OECD member countries can be found at http://www.oecd.org/general/listofoecdmembercountries-ratificationoftheconventionontheoecd.htm. World Bank income groups include low-income ($1025 or less), lower middle-income ($1026 to $4035), upper middle-income ($4036 to $12 475), and high-income ($12 476 or more) economies based on per capita gross national income. We created replicate models after removing data from the United States from the dataset as a sensitivity analysis (n = 199 vs. 198 countries). We also performed parametric and non-parametric pairwise comparisons on annual cardiovascular publications among World Bank Income groups.
We compared these data against publicly available country-level data, including income status (1999–2008), population size (1999–2008), total number of scientific and technical articles (1999–2008), and Human Development Index (combination of per capita gross domestic product, life expectancy, and education level for years in which data were available, i.e., 2000; 2005–2008) from the World Bank  and study mid-point (2004) age-standardized cardiovascular disease morbidity and mortality rates from the Global Burden of Disease  to explore their relationships with cardiovascular research output. Novel relationships were explored by creating a Google Motion Chart through R v2.15.1 (R Foundation for Statistical Consulting; Vienna, Austria) for creation of an interactive website. Other analyses were performed using SAS v9.3 (Cary, NC). Statistical significance of the results was set as two-sided P<0.05.
Table 1 outlines trends in the total number of global cardiovascular research articles downloaded from Web of Knowledge each year, number of citations downloaded, and matched number of records. Cardiovascular integer counts increased from 40,661 publications in 1999 to 55 284 publications in 2008, which represents a 36% increase over that decade. Figure 1 shows these trends on a logarithmic scale stratified by World Bank income. High-income countries that are members of the OECD published more articles in aggregate per year than high-income, non-OECD countries, middle-, or low-income countries. However, the proportion of cardiovascular publications from high-income, OECD countries declined from 93% (1999) of the total share to 84% (2008) of the total share over the study period. The rate of publication increase was greatest in upper-middle-income countries over the study period (336%; driven primarily by China), followed by lower-middle-income countries (291%), high-income, non-OECD countries (246%), low-income countries (73%), and high-income, OECD countries (30%). The majority of this growth appears to be driven primarily by increases in the number of publication outlets (i.e., sources increased by 73% over the study period; data not shown).
Trends in log number of cardiovascular publications by integer count, stratified by World Bank income group (1999–2008).
Table 2 shows the linear regression results demonstrating the association between time and cardiovascular research publications by income group. Substantial overlap is seen among the effect sizes (beta coefficients) and measures of imprecision (95% CI) by income group despite different baseline levels (intercepts), even after excluding the largest publisher of cardiovascular research as a sensitivity analysis (United States).
Table 3 demonstrates trends in the median number of citations per publication by year, stratified by income group and shows a direct gradient by income country status. Median citations per publication increased across all country groups over the study period, and the relative difference between poorer and wealthier countries decreased over time. Median (interquartile range) citations per publication were lowest among publications from low-income countries (0 all years; 0 [0, 0] in 2001) and were highest among publications from high-income, OECD countries (18·4 [12·5, 19·7] in 2007). Lower citation counts in 2008 may reflect incomplete citations reported to Web of Knowledge during follow-up period (2012). Trends were statistically significant for all country groups except upper middle-income countries.
The remaining figures evaluate multiple comparisons through a combination of x and y-axes, color, and bubble sizes to identify novel, country-level relationships. Interactive figures whereby users can manipulate combinations of variables are available at: http://188.8.131.52:8080/ccdcgmc-webapp/generate.ccdcgmc. For the remaining figures in this manuscript, the following format is used: each bubble represents an individual country, and the size of each bubble represents a country's population size. By way of orientation, the largest high-income, OECD country represents the United States, the largest upper-middle-income country represents China, and the largest lower-middle-income country represents India. For continuity, the same color scheme is used for stratification of income groups in all figures.
Figures 2A and 2B compare country-level fractional contributions (ratio of fractional to integer counts; scale: 0–1) in 1999 (A) and 2008 (B) against logarithmic annual citation counts (country-level cardiovascular research output) by World Bank country income status to estimate the relative degree of international collaboration. Lower fractional contributions reflect a greater proportion of publications with international collaboration. High-income, OECD countries generally have higher fractional counts than lower income groups, which suggest less international collaboration. There is marked variability in fractional counts among countries with less than 100 annual citations (integer count).
Country-level fractional counts (scale: 0–1) against logarithmic annual citation counts (country-level cardiovascular research output) with color coding representing World Bank country income status. Fractional counts represent the proportion of each country's authors against the total number of authors on each publication. The size of each country's bubble represents the population size of that country. Average citation index (ACI) is calculated by dividing a publication's total citations five years post-publication by five.
In general, the majority of countries appear to publish a similar proportion of cardiovascular publications against total publications with a modest relative increase over the study period (mean over entire study period = 4·3% of total scientific and technical publications) (Figures S1A and S1B).
Figures 3A and 3B demonstrate an inverse relationship between integer counts of cardiovascular research publications in 2004 against age-standardized cardiovascular disease mortality per 100 000 (A) and age-standardized cardiovascular morbidity (as measured through disability adjusted life years [DALYs]) (B) derived from the 2004 Global Burden of Disease dataset. Low- and middle-income countries with the higher burdens of cardiovascular disease measured through both deaths and DALYs have lower research output than high-income countries. Lastly, Figures 4A and 4B demonstrate a curvilinear relationship between integer counts of cardiovascular research publications in 2000 (A) and 2008 (B) against the Human Development Index [HDI]. Countries with low HDI scores (generally considered to be <0·5 ) demonstrate the lowest numbers of cardiovascular research publications in both 2000 and 2008 despite modest gains over the study period.
Color coding representing World Bank country income status. The size of each country's bubble represents the population size of that country.
Summary of Results
In this study of global cardiovascular research output from 1999 to 2008, we demonstrate a 36% increase in cardiovascular research output over a ten-year period by applying a bibliometric filter to the Web of Knowledge. High-income, OECD countries have a 9% lower proportion of cardiovascular publications over the study period (93% to 84%). The relative rate of publication growth was highest among upper and lower middle-income countries, and China had the greatest relative increase in cardiovascular research output over the study period. The wider availability of the Internet over the study period for greater international collaboration, access to journals, and ease of electronic submissions has likely contributed to these gains. The mean number of citations per publication has increased across all income groups, but the relative difference between high-income, OECD countries and low-income countries has decreased from a six-fold difference to a 3·5-fold difference between these groups.
Richer countries tend to have more stable, higher fractional contribution estimates, which reflect a lower degree of international collaboration compared with poorer countries. On the other hand, some middle-income countries have had increases in fractional contributions, which may be associated with concomitant economic development. For example, China's fractional contribution has increased from 0·77 in 1999 to 0·85 in 2008, which reflects a higher proportion of Chinese authors of cardiovascular publications, perhaps due to increased cardiovascular research capacity in China. Cardiovascular research output is also directly associated with Human Development Index rather than cardiovascular disease burden, with which it has an inverse association.
Comparison with Prior Literature
Prabhakaran and colleagues demonstrated a similar distribution of cardiovascular research publications from high-income countries in an analysis of MEDLINE publications in 1994–1995 and 2004–2005 . High-, upper middle-, lower middle-, and low-income countries published 82%, 7%, 7%, and 4% of cardiovascular publications, respectively, although this analysis included only 90 countries. Nevertheless, the ratio of publications by income group was similar between 1994–1995 and 2004–2005. Mendis and colleagues found a similar proportion of cardiovascular research publications from developed market economies in a limited MEDLINE search in 1991 (78%), 1996 (79%), and 2001 (78%) . The differences in these estimates may be due to differences in search strategies and lack of iterative filter testing by Prabhakaran et al. and Mendis et al.
While citations are increasing in the era of reference management software, increases in citations among cardiovascular publications may be a function of larger trends in the topic area under study. Kulkarni and colleagues found in a 2007 analysis of leading general medical journals that cardiovascular publications had, on average, 13 more citations than “other” research articles; however, general medical publications had, on average, nine more citations than “other” articles, suggesting that this reference group may not be particularly useful .
McKee and colleagues have suggested several potential reasons why overall research productivity is low among low- and middle-income countries: limited governmental or non-governmental funding, lack of health research strategy, lack of political will to engage globally or to prioritize health research, geographic isolation, and recent or ongoing conflict . Some have argued that the increasing dominance of English language in scientific writing may be one barrier to cardiovascular research growth, particularly among low- and middle-income countries . The recent increase in cardiovascular research output from China, however, appears to dispute this argument.
The “10/90” gap outlined by the 1990 Commission on Health Research for Development, which suggests that <10% of the research resources are available for 90% of the world's health problems, appears to be waning modestly for cardiovascular research over the past decade ; however, a “16/84 gap” in terms of research productivity remains. The World Health Organization [WHO] has suggested that there is “No Health Without Research” in its planned World Health Report for 2012 in collaboration with PLoS Medicine, with an emphasis on health research system strengthening for better healthcare delivery and policymaking . However, publication of this report was delayed for unclear reasons, and the WHO's World Health Report shifted its focus to “the contributions of research to universal health coverage”  rather than research capacity building in general. Further, the WHO's recent call to reduce premature (<70 years) mortality from NCDs, including CVD, by 25% by the year 2025 (“25×25 goal”) has limited mention of NCD research capacity building as a means to improve health, although prior calls for enhanced research capacity have been made . The inconsistent emphasis on research capacity building may stall sustained efforts required to develop a research workforce.
Mendis and colleagues highlighted the fact that poor research productivity is both “a consequence and a contributory factor” to the widening gap between cardiovascular morbidity and mortality in wealthier countries compared with poorer countries . We argue that future development assistance for health and health research, such as the recent funding for the Global Alliance for Chronic Diseases (gacd.org) and the National Heart, Lung, and Blood Institute/UnitedHealth-supported Centers of Excellence network (www.nhlbi.nih.gov/about/globalhealth/centers/index.htm), should be expanded to fill this gap in research capacity to help poorer countries, which bear a disproportionate burden of CVD morbidity and mortality, as one tool to improve national cardiovascular health. While delays in discovery to application are common in both high-income and low- and middle-income research settings, the pursuit of research may not be solely evaluated by its potential effect on the burden of disease. In addition, the decision to support increases in health investments should be made in parallel with increases in research investments.
Our study has several strengths, including development and iterative testing of a cardiovascular bibliometric filter with high level of precision and recall; long study period with a large number of publications and citations downloaded to evaluate trends over time; and development of an interactive webpage for easy exploration of multivariable relationships capitalizing upon publicly available datasets.
However, our study also has limitations. First, small countries with few cardiovascular research publications had unstable measures of several data points, including fractional count as a measure of international collaboration due to their low sample sizes. These results, in particular, should be interpreted with caution. However, we know of no other method for assessing international research collaboration on a global scale. Second, citations are only one, limited measure of the relative importance of a research article to inform practice and improve care, though they are used widely for comparison across (and within) journals , . Third, the exploration of relationships between country income status, size, cardiovascular research publications and measures such as burden of disease and development are susceptible to reverse causality and confounding and should be considered hypothesis-generating. Fourth, we evaluated publications over the study period that were collected through the Web of Knowledge database, which may have excluded publications from other databases, including others that collect relatively more data from LMICs compared with the Web of Knowledge (Latin American and Caribbean Health Sciences Literature [LILACS], e.g.). However, the Web of Knowledge is the largest bibliometric database currently available and provides standardized author address information for analyses like ours.
Cardiovascular health research has increased substantially in the past decade, with the greatest relative increases in upper middle-income countries such as China. The United States and other high-income, OECD countries remain the dominant producers of cardiovascular research globally, although their share appears to be declining. However, low- and middle-income countries with the higher burdens of cardiovascular disease continue to have lower research output than high-income countries, and thus require targeted research investments to improve cardiovascular health.
Country-level cardiovascular research publications against total scientific and technical publications in 1999 (A) and 2009 (B). Total scientific and technical publications as reported by the World Bank with color coding representing World Bank country income status. The size of each country's bubble represents the population size of that country.
MDH and AB had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Conceived and designed the experiments: DP VSA MDH. Performed the experiments: MDH AB GSB. Analyzed the data: MDH AB GL DP. Contributed reagents/materials/analysis tools: MDH. Wrote the paper: MDH AB GSB LC PP VSA SS GL DP. Obtained funding: DP.
- 1. Callaham M, Wears RL, Weber E (2002) Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 287: 2847–2850.
- 2. Prabhakaran P, Ajay VS, Prabhakaran D, Gottumukkala AK, Shrihari JS, et al. (2007) Global cardiovascular disease research survey. J Am Coll Cardiol 50: 2322–2328.
- 3. Pan RK, Kaski K, Fortunato S (2012) World citation and collaboration networks: uncovering the role of geography in science. Sci Rep 2: 902.
- 4. de Jong JW, Schaper W (1996) The International Rank Order of Clinical Cardiology. Eur Heart J 17: 35–42.
- 5. Lewison G (1996) The definition of biomedical research subfields with title keywords and application to the analysis of research outputs. Res Evaluation 6: 25–36.
- 6. Lewison G (2011) Definition of Cancer Research: Journals, Titles, Abstracts or Keywords? DESIDOC Journal of Library & Information Technology 31: 333–339.
- 7. Lewison G, Purushotham A, Mason M, McVie G, Sullivan R (2010) Understanding the impact of public policy on cancer research: a bibliometric approach. Eur J Cancer 46: 912–919.
- 8. Falagas ME, Kouranos VD, Arencibia-Jorge R, Karageorgopoulos DE (2008) Comparison of SCImago journal rank indicator with journal impact factor. FASEB 22: 2623–2628.
- 9. McVeigh ME, Mann SJ (2009) The journal impact factor denominator: defining citable (counted) items. JAMA 302: 1107–1109.
- 10. Clarke A, Gatineau M, Grimaud O, Royer-Devaux S, Wyn-Roberts N, et al. (2007) A bibliometric overview of public health research in Europe. European Journal of Public Health 17: 43–49.
- 11. Larsen PO, von Ins M (2010) The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index. Scientometrics 84: 575–603.
- 12. Durieux V, Gevenois PA (2010) Bibliometric indicators: quality measurements of scientific publication. Radiology 255: 342–351.
- 13. Mendis S, Yach D, Bengoa R, Narvaez D, Zhang X (2003) Research gap in cardiovascular disease in developing countries. Lancet 361: 2246–2247.
- 14. Kulkarni AV, Busse JW, Shams I (2007) Characteristics associated with citation rate of the medical literature. PLoS One 2: e403.
- 15. McKee M, Stuckler D, Basu S (2012) Where there is no health research: what can be done to fill the global gaps in health research? PLoS Med 9.
- 16. Baussano I, Brzoska P, Fedeli U, Larouche C, Razum O, et al. (2008) Does language matter? A case study of epidemiological and public health journals, databases and professional education in French, German and Italian. Emerg Themes Epidemiol 5: 16.
- 17. Commission on Health Research for Development. Health Research: Essential Link to Equity in Development. New York, NY: Oxford University Press, 1990.
- 18. Editors (2012) The World Health Report 2012 that wasn't. PLoS Med 9.
- 19. Mori H, Nakayama T (2013) Academic impact of qualitative studies in healthcare: bibliometric analysis. PLoS One 8: e57371.
- 20. Bonita R, Magnusson R, Bovet P, Zhao D, Malta DC, et al. (2013) Country actions to meet UN commitments on non-communicable diseases: a stepwise approach. Lancet 381: 575–584.
- 21. Callaham M, Wears RL, Weber E (2002) Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA 287: 2847–2850.