https://orcid.org/0000-0002-3758-3006
Figures
Abstract
Background
The proposed sequential and combinatorial algorithm, suggested as a standard tool for assessing, exploring, and reporting heterogeneity in the meta-analysis, is useful but time-consuming particularly when the number of included studies is large. Metaplot is a novel graphical approach that facilitates performing sensitivity analysis to distinguish the source of substantial heterogeneity across studies with ease and speed.
Method
Metaplot is a Stata module based on Stata’s commands, known informally as "ado". Metaplot presents a two-way (x, y) plot in which the x-axis represents the study codes and the y-axis represents the values of I2 statistics excluding one study at a time (n-1 studies). Metaplot also produces a table in the ’Results window’ of the Stata software including details such as I2 and χ2 statistics and their P-values omitting one study in each turn.
Results
Metaplot allows rapid identification of studies that have a disproportionate impact on heterogeneity across studies, and communicates to what extent omission of that study may reduce the overall heterogeneity based on the I2 and χ2 statistics. Metaplot has no limitations regarding the number of studies or types of outcome data (binomial or continuous data).
Citation: Poorolajal J, Noornejad S (2021) Metaplot: A new Stata module for assessing heterogeneity in a meta-analysis. PLoS ONE 16(6): e0253341. https://doi.org/10.1371/journal.pone.0253341
Editor: Mohammad Asghari Jafarabadi, Tabriz University of Medical Sciences, ISLAMIC REPUBLIC OF IRAN
Received: February 25, 2021; Accepted: June 2, 2021; Published: June 28, 2021
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: The Vice-Chancellor of Research and Technology, Hamadan University of Medical Sciences funded this study (No. 9603161751). However, the funder had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The studies that are brought together in a meta-analysis inevitably differ in many aspects. This variability across studies is called heterogeneity [1]. The between-studies heterogeneity can be assessed by the chi-square test also written as χ2 or Chi2 and can be quantified by I2 statistics [2, 3]. When there is heterogeneity in a meta-analysis, the source of heterogeneity across studies should be carefully investigated on a case-by-case basis [4].
A common approach, which was proposed by Patsopoulos et al, is to perform a sensitivity analysis based on a sequential and combinatorial algorithm [5]. According to this algorithm, one study is excluded from the meta-analysis at a time and the impact of the excluded study on the between-study heterogeneity is evaluated based on I2 statistic and χ2 test. This ‘one-out’ sensitivity analysis tells us to what extent the overall heterogeneity changes by excluding a particular study at a time. Then, the study that is responsible for the largest decrease in I2 value should be dropped out. This process is repeated for a new set of n-1 studies. This sequential and combinatorial algorithm is repeated several times until the I2 statistic drops below the desired threshold value of 50%. In the last step, there is a possibility that more than one omitted study can result in I2 dropping below the intended threshold. In such cases, the algorithm that results in the maximum decrease in the I2 statistic below the desired threshold is selected. There is a chance that two or more studies cause the same reduction in I2 by their exclusion. In this case, the study with the largest reduction in χ2 statistic (the least χ2 statistic) is dropped out.
Based on the aforementioned algorithm, this ‘one-out’ sensitivity analysis must be repeated n-1 times to specify and exclude the outlying study from the meta-analysis. If the desired threshold value of 50% is not achieved in the first step, the algorithm must be repeated n-2, n-3, etc. Therefore, this algorithm may be boring and time-consuming when the number of included studies is large and the between-studies heterogeneity is substantial.
In this study, we aimed to introduce a novel Stata graph that performs the ‘one-out’ sensitivity analysis for n-1 studies and identifies immediately the studies responsible for substantial heterogeneity across studies by executing "metaplot.ado" Stata command.
Methods
Metaplot is a Stata module based on Stata’s commands, known as "ado". Metaplot produces a two-dimensional (x, y) Stata graph. The x-axis represents the included studies. The studies are shown on this axis by an ID code. The y-axis represents the values of I2 statistics based on ‘one-out’ (n-1 studies) sensitivity analysis indicating to what extent the overall heterogeneity changes by excluding a particular study at a time.
Furthermore, the "metaplot" command generates a table in the “Results window” of the Stata including more details about ‘one-out’ sensitivity analysis in terms of the I2 and χ2 statistics and their P-values. In addition to study codes, the studies’ identifications can be presented in the table.
The "metaplot" command is flexible and works with any measurement option including binary data (effect size + standard error or effect size + confidence intervals) and continuous data (sample + mean + standard deviation). The full form of the "metaplot" command is as follows
metaplot varlist [if] [in] [, id(study) tr(#)]
where
- “varlist” can be “a b c d” or “lnes se” or “es lles ules” or “n1 mean1 sd1 n0 mean0 sd0”
- “id(study)” option displays studies identifications (the first authors and the year of publication) specified by the variable “study” in the dataset.
- “tr(#)” option specifies the desired threshold values for example: 0.4, 0.5, 0.6, 0.65, 0.8, etc.
The abbreviations in the above command represent the following terms.
- “a b c d” represents “events” and “non-events” in the intervention (exposure) and control groups, respectively.
- “lnes” represents the “Naperian logarithm” of the effect size that may be risk ratio (lnrr) or odds ratio (lnor).
- “se” represents the standard error of the effect size.
- “es” represents the effect size that may be risk ratio (rr) or odds ratio (or).
- “lles” represents the lower limit of the confidence interval for the effect size.
- “ules” represents the upper limit of the confidence interval for the effect size.
- “n1” and “n0” represent the sample size for the intervention (exposure) and control groups, respectively.
- “mean1” and “mean0” represent the mean for the intervention (exposure) and control groups, respectively.
- “sd1” and “sd0” represent the standard deviation for the intervention (exposure) and control groups, respectively.
The relevant files including “metaplot.ado” and “metaplot.hlp” are attached to this paper as (S1 and S2 Files).
Results
To show the capability and flexibility of the ’metaplot" command we used various datasets (S1–S3 Datasets) related to our previous published meta-analyses [6–8].
The first dataset (S1 Dataset), which was used to introduce the "metaplot" module, related to a published meta-analysis addressed the risk factors for stomach cancer [6]. This is a dataset with a “binomial” outcome (stomach cancer). In this meta-analysis, 15 studies addressed the association between stomach cancer and drinking black tea. The heterogeneity across studies was high (I2 = 64.23%). To perform sensitivity analysis using the “metaplot” command for this dataset, we executed the following command in the Stata software.
- metaplot es lles ules, id(study)
The result of the above command is given in Fig 1. This figure shows the results of the ‘one-out’ sensitivity analysis using the "metaplot" command. According to this figure, all values of I2 statistics excluding one study at a time (n-1 studies) were above the desired threshold value of 50% except for study #5. By omitting study #5 from the meta-analysis, the heterogeneity fell below the desired threshold value of 50%. That means this study was an outlier and the main reason for heterogeneity across studies. Table 1 shows the results of ‘one-out’ sensitivity analysis in detail including I2 and χ2 statistics and their P-values omitting one study at a time. Based on this table, the overall heterogeneity across studies was high (I2 = 64.23%). However, the heterogeneity decreased to 38.93% after omitting study #5.
The second dataset (S2 Dataset), which was used to introduce the "metaplot" module, related to a published meta-analysis addressed the effect of oral potassium supplementation on the management of essential hypertension [7]. This is a dataset with a “continuous” outcome (blood pressure). In this meta-analysis, 22 studies addressed the effect of oral potassium supplementation on diastolic blood pressure. The heterogeneity across studies was high (I2 = 81.88%). To perform sensitivity analysis using the “metaplot” command for this dataset, we executed the following command in the Stata software.
- metaplot n1 mean1 sd1 n0 mean0 sd0, id(study)
The result of the above command is given in Fig 2. This figure shows the results of the "metaplot" command based on a ‘one-out’ sensitivity analysis. According to this figure, all values of I2 statistics excluding one study at a time (n-1 studies) were above the desired threshold value of 50%. However, the effect of omitting one study at a time was not similar across studies. For example, studies #14, #3, and #5 were responsible for the largest decrease in I2 values, respectively. Although heterogeneity decreased significantly, particularly by omitting study #14, it did not reach below the threshold value of 50%. Therefore, this process should be repeated for a new set of n-1 studies after omitting study #14. According to the results of Table 2, the overall heterogeneity across studies was high (I2 = 81.88%). However, the heterogeneity decreased to 67.76%, 75.19%, and 79.85% after omitting studies #14, #3, and #5, respectively.
The third dataset (S3 Dataset), which was used to introduce the "metaplot" module, related to a published meta-analysis addressed the preventable factors for primary prevention of childhood obesity [8]. This is a dataset with a “binomial” outcome (stomach cancer) and multiple studies. In this meta-analysis, 84 studies addressed the association between physical activity and childhood obesity. The heterogeneity across studies was high (I2 = 96%). We used the sequential and combinatorial algorithm and performed a ‘one-out’ sensitivity analysis and repeated the process several times. For this purpose, we executed the following command in the Stata software for n-1 studies several times.
- metaplot lnor se, id(study)
The result of the above command is given in Fig 3. This figure shows the last step when the I2 statistic dropped below the desired threshold value of 50% by omitting just one more study. By looking at Fig 3 one can realize that there are at least 5 options to reduce the I2 statistic below the value of 50%. By omitting any of the studies #13, #16, #25, #37, and #57 the I2 statistic drops below the value of 50% and reaches 49.25%, 48.35%, 49.95%, 49.16%, and 47.25%, respectively (Table 3). When there is a possibility that more than one omitted study can result in I2 dropping below the intended threshold, the study that results in the maximum decrease in the I2 statistic below the desired threshold is selected. Accordingly omitting study #57 is the best choice. There might have been a chance that two or more studies caused the same reduction in I2 by their exclusion. In that case, the study with the largest reduction in χ2 statistic (the least χ2 statistic) would have been dropped out.
Discussion
The idea of Metaplot, which was first introduced in 2010 [9], is a simple graphical approach to identify outliers and their effects on overall heterogeneity across studies. Patsopoulos et al. [5] suggested the sequential and combinatorial algorithm for performing sensitivity analyses. This algorithm is a useful method for assessing, exploring, and reporting the between-study heterogeneity in the meta-analysis but is time-consuming when the number of included studies is large and heterogeneity is substantial. For example, as noted in the results section, 84 studies addressed the association between physical activity and childhood obesity [8]. In this case, the sequential and combinatorial algorithm needs to be repeated hundreds of times particularly when the heterogeneity across studies is substantial. While by executing the "metaplot" command we can perform ‘one-out’ sensitivity analysis across several studies, no matter how many they are, and identify immediately to what extent the overall heterogeneity changes by excluding a particular study at a time. Another capability of the "metaplot" command is its flexibility. It is possible to execute this command for meta-analysis of different types of outcome data (e.g. binary, continuous, or time to event) and different types of summary measures (e.g. odds ratio, risk ratio, rate ratio, or hazard ratio).
The I2 threshold value of 50% usually depends on the type of research we are performing. The threshold value of 50% is not rigid in the "metaplot" command. A rigid threshold value for the interpretation of I2 can be misleading since the importance of inconsistency depends on several factors [1]. The "metaplot" command has the option "tr(#)" that establishes different threshold values.
Care must be taken in the interpretation of the chi-squared test since it has low power in the situation of a meta-analysis when studies have a small sample size or are few in number. This means that while a statistically significant result may indicate a problem with heterogeneity, a non-significant result must not be taken as evidence of no heterogeneity [1]. This is also why a P-value of 0.10 is sometimes used, rather than the conventional level of 0.05. Another problem with the test is that when there are many studies in a meta-analysis, the test has a high power to detect a small amount of heterogeneity that may be clinically unimportant.
Huedo-Medina et al. [10] examined and compared the performances of the Q test and the I2 index for assessing homogeneity across individual studies in meta-analysis. They confirmed that the Q test only reports the presence or absence of homogeneity across studies but does not specify the extent of such heterogeneity. On the other hand, the I2 index can quantify the degree of heterogeneity. Although the I2 index has the same problems of low statistical power with a small number of studies, they suggested the I2 index as a complement to the Q test.
The raw idea of “metaplot” was first introduced in 2010 [9]. This preliminary idea was never implemented actually at that time because the package had not been generated yet. The new design of the “metaplot” presented in this paper is very different from the original one introduced in 2010. The original design was a complicated three-dimensional graph with x, y, and z axes including unnecessary information. It was rather hard to understand. The new design of “metaplot” is a two-dimensional graph with x and y axes. Furthermore, we added a table including details of information (I2 and χ2 statistics and their P-values omitting one study in each turn) to simplify the interpretation of the ‘metaplot’ graph. In the current paper, we explained the capability of the “Metaplot” module and how to use the Stata command and its options. We examined this module on different real datasets and reported the results.
There are several graphical methods for the exploration of heterogeneity in the meta-analysis. One of these methods is the traditional Galbraith plot [11, 12]. This plot provides a graphical display to get a visual impression of the amount of heterogeneity from a meta-analysis. For each study, the observed effect sizes on the vertical axis are plotted against the reciprocal standard errors on the horizontal axis. The regression line projects through the origin, with its 95% confidence interval positioned 2 units over and below the regression line, has a slope equal to the overall log rate ratio. In the absence of heterogeneity, we could expect all the points to lie within the confidence bounds. The L’Abbé plot is another useful method for assessing heterogeneity in the meta-analysis [13, 14]. It is a scatter plot with the risk in the control group on the x-axis and the risk in the experimental group on the y-axis. The visual inspection gives a quick and easy indication of the studies having different results from other studies. These studies are considered outliers and hence potential sources of heterogeneity. Although these graphical procedures are useful and their interpretations are straightforward, they have a major limitation. When only one study causes extreme heterogeneity, these methods point to the same study as Metaplot suggests. However, in situations where the heterogeneity is resulted from several studies, the above graphical procedures are impractical to indicate to what extent a particular study influences the overall heterogeneity. Our proposed graphical method has overcome this problem. According to Metaplot method, one study is excluded from the meta-analysis at a time and the impact of the excluded study is evaluated on the overall heterogeneity. This ‘one-out’ approach tells us to what extent the overall heterogeneity changes by excluding a particular study at a time.
The Metaplot has a limitation. When the number of studies is very large (more than 35) as shown in Fig 3, the study codes in the x-axis come together and even may collapse due to space constraints. In such cases, the identification of the study codes may be difficult. Fortunately, the properties of the “metaplot” module solved this problem. In addition to the "Metaplot", this module generates a table in the “Results window” of the Stata and gives more details of ‘one-out’ sensitivity analysis including the I2 and the χ2 statistics and their P-values as well as the studies codes and the studies identifications. Therefore, by turning back to the “Results window” we can realize which study has the greatest impact on the overall heterogeneity based on the I2 and χ2 statistics.
Conclusion
Metaplot is a visual complementary approach for testing between-study heterogeneity. This plot is a simple graphical approach that gives a quick and easy identification of the studies having substantial influences on overall heterogeneity as fast as possible. This method is based on ‘one-out’ sensitivity analysis and provides information both graphically and quantitatively about the extent of the overall heterogeneity changes by excluding a particular study at a time in terms of I2 and χ2 statistics. It is possible to implement this graph for the meta-analysis of different types of outcome data.
References
- 1.
Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions. Chichester: John Wiley & Sons, Ltd; 2019.
- 2.
Michael Borenstein V. Hedges L, Higgins JPT, Rothstein Hannah R. Introduction to Meta-Analysis. Chichester: John Wiley & Sons Ltd; 2009.
- 3. Higgins JPT, Thompson SG, Deeks JJ, Altman D. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60. pmid:12958120
- 4. Glasziou PP, Sanders SL. Investigating causes of heterogeneity in systematic reviews. Stat Med. 2002;21:1503–11. pmid:12111916
- 5. Patsopoulos NA, Evangelou E, Ioannidis JPA. Sensitivity of between-study heterogeneity in meta-analysis: proposed metrics and empirical evaluation. Int J Epidemiol. 2008;37:1148–57. pmid:18424475
- 6. Poorolajal J, Moradi L, Mohammadi Y, Cheraghi Z, Gohari-Ensaf F. Risk factors for stomach cancer: a systematic review and meta-analysis. Epidemiol Health. 2020;42:e2020004. pmid:32023777
- 7. Poorolajal J, Zeraati F, Soltanian AR, Sheikh V, Maleki A. Oral potassium supplementation for management of essential hypertension: A meta-analysis of randomized controlled trials. PloS One. 2017;12(4):e0174967. pmid:28419159
- 8. Poorolajal J, Sahraei F, Mohamdadi Y, Doosti-Irani A, Moradi L. Behavioral factors influencing childhood obesity: a systematic review and meta-analysis. Obes Res Clin Pract. 2020;14:109–18. pmid:32199860
- 9. Poorolajal J, Fotouhi A, Majdzadeh R, Mahmoodi M. MetaPlot: a novel Stata graph for assessing heterogeneity at a glance. Iran J Public Health. 2010;39(2):102–4. pmid:23113013
- 10. Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J. Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychological methods. 2006;11(2):193–206. 16784338. pmid:16784338
- 11. Galbraith RF. Graphical display of estimates having differing standard errors. Technometrics. 1988a;30(3):271–81.
- 12. Galbraith RF. A note on graphical presentation of estimated odds ratios from several clinical trials. Stat Med. 1988b;7:889–94.
- 13. Deeks JJ. Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary outcomes. Stat Med. 2002;21:1575–600. pmid:12111921
- 14. L’Abbé KA, Detsky AS, O’Rourke K. Meta-analysis in clinical research. Ann Intern Med. 1987;107:224–33. pmid:3300460