Reader Comments

Post a new comment on this article

Clarity in the reporting of health equity issues requires addressing measurement issues.

Posted by jscanlan on 03 Nov 2012 at 18:29 GMT

In this editorial,[1] the authors commend the PRISMA guidelines on the reporting of systematic reviews with a specific focus, described in the article by Welch et al.,[2] for bringing clarity to this subject. In fact, however, by overlooking crucial measurement issues, the guidelines contribute to confusion in discussions of health equity.

The overwhelming majority of health equity research to date has been fundamentally flawed due to the reliance on standard measures of differences between outcome rates to appraise the size of health disparities without consideration of patterns by which such measures tend to be systematically affected by the prevalence of an outcome. The most notable such pattern is that whereby the rarer an outcome the greater tends to be the relative difference in experiencing it and the smaller tends to be the relative difference in avoiding it.[3-5] Thus, for example, as mortality declines relative differences in mortality tend to increase while relative difference in survival tend to decrease. Similarly, as reflected in published life tables, relative differences in mortality will tend to be larger among the young than the old, while relative differences in survival will tend to be larger among the old than the young.[6,7] On the other hand, as beneficial healthcare procedures like immunization or cancer screening generally increase, relative differences in receipt of such procedures tend to decrease, while relative differences in rates of failure to receive such procedures tend to increase. Illustrating how poorly these patterns have been understood, however, published studies, especially those involving racial differences in cancer outcomes, commonly purport to be examining relative differences in survival when in fact examining relative differences in the mortality. Invariably they do so without recognizing that the two relative differences tend to change in opposite direction studies as mortality and survival rates change generally.[8]

Absolute differences between rates are the same whether one examines the favorable or the adverse outcome. But absolute differences tend also to be systematically affected by the prevalence of an outcome. Roughly, as uncommon outcomes (less than 50% for all groups being compared) become more common, absolute differences between rates tend to increase; as common outcomes (greater than 50% for all groups being compared) become even more common, absolute differences tend to decrease. Further, as the prevalence of an outcome changes, absolute differences tend to change in the same direction as the smaller relative difference.[9] An illustration of the patterns by which the two relative differences, absolute differences, and odds ratios tend to be affected by the prevalence of an outcome may found in Figure 5 of reference 10.

In 2005, the United States National Center for Health Statistics (NCHS) recognized that determinations of whether health and healthcare inequalities were increasing or decreasing could turn on whether one examined relative differences in favorable outcomes or relative differences in adverse outcomes.[11] But rather than address the fact that relative differences in either favorable or adverse outcomes cannot offer meaningful information on whether the comparative situation of two groups has changed without taking overall prevalence into account, NCHS merely chose henceforth to measure all health and healthcare disparities in terms of relative differences in adverse outcomes. The decision thus caused many healthcare disparities that were deemed to be decreasing to instead to be deemed to be increasing, and researchers who had previously been looking for explanations for why such disparities had been decreasing now had to look for explanations for why such disparities were increasing. But the decision did nothing to cause health disparities research to be any more statistically sound than it had been in the past.

With respect to the issue of clarity, it also warrant note that the NCHS recommendation is not universally followed even in the United States and may not even be widely known. Thus, in a study published in Pediatrics in 2008 Morita et al.,[12] relying on relative differences in vaccination rates as a measure of disparity, found that a school-entry Hepatitis B vaccination requirement that dramatically increased overall vaccination rates also dramatically reduced racial and ethnic vaccination disparities. By contrast, NCHS would have found dramatic increases in disparities. On the other hand, researchers relying on absolute differences between rates would have reached different results for different points in time, as in fact would commonly occur when an outcome goes from being very uncommon to being very common. See Table 4 of reference 10. Of course, there is little reason to expect researchers in Europe to follow the NCHS recommendation (as reflected by the study by Osborn et al.[14] cited in the editorial, which examined cancer screening inequalities in terms of relative differences in screening rates).

Apart from the NCHS document, there has been increasing recognition of these issues in recent years, including in works that more explicitly recognize the ways standard measures tend to be affected by the prevalence of an outcome than the NCHS document did.[15-18] Nevertheless, the PRIMSA guidelines fail to show a recognition that the two relative differences might yield contrary conclusions much less than that they tend systematically to do so.

Rather, the guidelines merely advise (checklist Item 14) that reviews of studies of health equity should discuss whether, for example, the study examined both relative and absolute differences (“both” referring to relative and absolute, not to the two relative differences). Increasingly studies do report a relative difference and the absolute difference, especially when the relative difference and the absolute difference yield different conclusions as to the direction of changes in disparities over time. Yet anytime an observer notes that a relative difference and the absolute difference have changed in different directions over time, the unmentioned relative difference will have changed in the opposite direction of the mentioned relative difference and in the same direction as the absolute difference. Any discussion of relative differences and absolute differences without recognition of such fact is a recipe for confusion and, more important, for ignoring the patterns by which the measures being discussed tend to be systematically affected by the prevalence of an outcome.

The textual discussion by Welch et al. of absolute differences also reflects several misconceptions. By noting with respect to Checklist Item 14 that “the absolute impact [of an intervention] is likely to be higher in disadvantaged groups who are likely to have worse baseline health status,” the authors suggest that in the normal course a factor would cause equal relative changes in baseline rates with resultant larger change in the larger baseline rate. But any expectation that a factor would cause equal proportionate changes in different baseline rates is illogical given that a factor cannot cause equal proportionate changes for groups with different baseline rates while causing equal proportionate changes in the two groups rates of experiencing the opposite outcomes. Rather, the sound expectation is that a factor will cause a larger proportionate change for the group with the lower baseline rate of experiencing an outcome while causing a larger proportionate change in the opposite outcome for the other group.[19]
But the expectation that an intervention will cause a larger absolute change in the larger baseline rate is unjustified irrespective of any assumption about patterns of relative changes. For the absolute change is the same number of percentage points for the favorable outcome as for the adverse outcome, and, while the disadvantaged group will have the larger baseline rate for the adverse outcome, the advantaged group will have the larger baseline rates for the favorable outcome.

Whether the absolute difference changes more for one group than another will depend on the rate ranges as issue. But where favorable outcomes are uncommon, beneficial interventions will tend to increase absolute differences in those outcomes, as shown in the top part of Table 4 of reference 10 and Tables 1 and 4 of reference 20.
Finally, while Welch et al. may well be correct that policy makers may be principally interested in absolute effects, discussion concerning relative and absolute differences commonly involve situation where the question to be addressed is whether a disparity is changing over time. As discussed in Section D of reference 21, there can exist only one reality as to whether the forces causing rates at which advantaged and disadvantaged groups experience favorable or adverse outcomes to differ have increased or decreased over time. But neither the absolute difference nor any of the other standard measures of differences between outcome rates can reveal that reality unless appraised with recognition of the patterns by which the measure tends to be affected by the prevalence of an outcome.

As with guidance on the conduct of studies of health equity, useful guidance on the reporting of studies of focused on health equity must be informed by a sound understanding of these patterns. But the PRISMA guidance fails to recognize that the patterns exist.

References:

1. The PLOS Medicine Editors (2012) Bringing Clarity to the Reporting of Health Equity. PLoS Med 9(10): e1001334. doi:10.1371/journal.pmed.1001334

2. Welch V, Petticrew M, Tugwell P, Moher D, O'Neill J, et al. (2012) PRISMA-Equity 2012 Extension: Reporting Guidelines for Systematic Reviews with a Focus on Health Equity. PLoS Med 9 (10) e1001333 doi:10.1371/journal.pmed.1001333

3. Scanlan JP (2006). Can We Actually Measure Health Disparities? Chance;19(2):47-51: http://www.jpscanlan.com/...

4. Scanlan JP (2000) Race and Mortality. Society 2000;37(2):19-35: http://www.jpscanlan.com/...

5. Scanlan JP (1994) Divining Difference. Chance;7(4):38-9,48: http://jpscanlan.com/imag...

6. Life Tables Illustrations sub-page of Scanlan’s Rule page of jpscanlan.com:
http://jpscanlan.com/scan...

7. Life Table Information Document: http://jpscanlan.com/imag...

8. Mortality and Survival Page of jpscanlan.com: http://jpscanlan.com/mort...

9. Scanlan’s Rule Page of jpscanlan.com: http://jpscanlan.com/scan...

10. Scanlan JP (2012) Applied Statistics Workshop, presented at the Institute for Quantitative Social Science at Harvard University, Cambridge, MA, Oct. 17: http://jpscanlan.com/imag...

11. Keppel K., Pamuk E., Lynch J., et al. (2005) Methodological Issues in Measuring Health Disparities. Vital Health Stat;2 (141): http://www.cdc.gov/nchs/d...

12. Morita JY, Ramirez E, Trick WE (2008) Effect of School-Entry Vaccination Requirements on Racial and Ethnic Disparities in Hepatitis B Immunization Coverage among Public High School Students. Pediatrics;121:e547.

13. Subgroup Effects sub-page of the Scanlan’s Rule page of jpscanlan.com; http://www.jpscanlan.com/...

14. Osborn DPJ, Horsfall L, Hassiotis A, Petersen I, Walters K, et al. (2012) Access to Cancer Screening in People with Learning Disabilities in the UK: Cohort Study in the Health Improvement Network, a Primary Care Research Database. PLoS ONE 7(8): e43841. doi:10.1371/journal.pone.0043841

15. Eikemo TA, Skalicka V, Avendano M (2009) Variations in Health Inequalities: Are They a Mathematical Artifact? International Journal for Equity in Health;8:32: http://www.equityhealthj....

16. Bauld L, Day P, Judge K (2008). Off Target: A Critical Review of Setting Goals for Reducing Health Inequalities in the United Kingdom. Int J Health Serv;38(3):439-454: http://baywood.metapress....

17. Houweling TAJ, Kunst AE, Huisman M, Mackenbach JP (2007). Using Relative and Absolute Measures for Monitoring Health Inequalities: Experiences from Cross-National Analyses on Maternal and Child Health. International Journal for Equity in Health 2007;6:15: http://www.equityhealthj....

18. Carr-Hill R, Chalmers-Dixon P (2005) The Public Health Observatory Handbook of Health Inequalities Measurement. Oxford: SEPHO: http://www.sepho.org.uk/e...

19. Subgroup Effects sub-page of the Scanlan’s Rule page of jpscanlan.com: http://www.jpscanlan.com/...

20. Scanlan JP (2011). Perverse Perceptions of the Impact of Pay for Performance on Healthcare Disparities,” presented at the 9th International Conferences on Health Policy Statistics, Cleveland, Ohio, Oct. 5-7: http://jpscanlan.com/imag...

21. Scanlan JP (2012) Harvard University Measurement Letter. Oct. 9: http://jpscanlan.com/imag...

No competing interests declared.