https://orcid.org/0000-0002-1544-5884
Figures
Abstract
This study aims to advance the Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy (MAIHDA) approach by addressing two key questions. First, it investigates the impact of using increasingly complex combinations of variables to create intersectional strata on between-stratum variance, measured by the variance partitioning coefficients (VPCs). Second, it examines the stability of coefficients for fixed effects across models with an increasing number of hierarchical levels. The analysis is performed using data from a survey of over 42,000 respondents on the prevalence of gender-based violence in European research organisations conducted in 2022. Results indicate that the number of intersectional strata is not significantly related to the proportion of the total variance attributable to the variance between intersectional strata in the MAIHDA approach. Moreover, the coefficients remain relatively stable and consistent across models with increasing complexity, where levels about organisations and countries are added. The analysis concludes that the MAIHDA approach can be flexibly applied for different research purposes, either to better account for structures of power and inequality; or to provide intersectionality-sensitive estimates. The findings underscore the need for researchers to clarify the specific aims of using MAIHDA, whether descriptive or inferential, and highlight the approach’s versatility in addressing intersectionality within quantitative research. The study contributes to the literature by offering empirical evidence on the methodological considerations in applying the MAIHDA approach, thereby aiding in its more effective use for intersectional research.
Citation: Humbert AL (2024) Methodological insights into multilevel analysis of individual heterogeneity and discriminatory accuracy: An empirical examination of the effects of strata configurations on between-stratum variance and of fixed effects across hierarchical levels. PLoS ONE 19(3): e0297561. https://doi.org/10.1371/journal.pone.0297561
Editor: Wen-Jun Tu, Chinese Academy of Medical Sciences and Peking Union Medical College, CHINA
Received: September 11, 2023; Accepted: January 8, 2024; Published: March 18, 2024
Copyright: © 2024 Anne Laure Humbert. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data underlying the results presented in the study are available from the Leibniz Institute for the Social Sciences (GESIS) at https://doi.org/10.7802/2475. The research data are accessible for academic research and teaching with the written permission of the data depositor.
Funding: This research has been supported by the European Commission under the Horizon 2020 Framework Programme (Award Number 101006261). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The author has declared that no competing interests exist.
Introduction
Reconciling intersectionality with a quantitative approach is not without challenges. At the heart of the concept is the idea that different social identities intersect in unique ways to create structural inequalities and discrimination [1]. How power and privilege are experienced, via the lens of these myriads of intersecting identities, is regarded as incompatible with the somewhat crude categorisation that is invariably involved in any quantitative approach. The criticisms of quantitative methods in relation to intersectionality theory revolve around six main issues. First, it must be recognised that quantitative approaches tend to reflect dominant narratives, which shape the categories that are captured by data [2]. As a result, while more salient groups tend to be measured, this is at the expense of other minoritised groups that are not represented in data. Second, quantitative methods often default to assuming uniform experiences across a certain demographic group, the so-called ‘tyranny of the averages’ [3], that conflates ‘unity’ with group ‘uniformity’ [4]. This assumption overlooks the variance within these categories, posing the question of whether it is desirable to even consider that experiences should be regarded as universally identical within a particular group. Third, is the issue of the over-simplification of what are complex identities. Quantitative methods are inherently reductionist, often distilling complex realities into simpler, measurable variables [2, 5]. This leads to a decontextualisation of unique, individual identities and experiences. For instance, the interpretation of womanhood may vary substantially from one individual to another. In particularly, this asks the questions of whether it is possible to assume that being a woman will mean the same from one woman to another, even where they share the same other diversity traits. Fourth is the need to regard identities as being more than the sum of their parts. Identities can encapsulate additive traits, i.e. being a woman, being white. Identities can also encapsulate multiplicative traits, which is then concerned with being a white woman. However, as argued by Weldon [6], intersectionality is also about an additional ‘qualitative’ layer that goes beyond these additive or multiplicative aspects. What is certain is that identities rely of a process that is layered, raising questions about the capacity of quantitative methods to capture this complexity entirely. Fifth are concerns about statistical limitations. While certain quantitative methods attempt to incorporate interaction effects among different variables, there are inherent limitations related the number of interactions that can be effectively accounted for within a single model [7]. Sixth, this is exacerbated by the problem of achieving sufficient representation within intersectional groups. Where there are limited numbers, this can result in insufficient statistical power, thereby reducing the voice and visibility of these individuals in the data and, consequently, in the findings themselves [8].
These challenges underscore the need for more nuanced and flexible research strategies that can accurately portray the richness of intersectional identities and experiences. The latest advancements in quantitative intersectional research suggest integrating the interplay of various social determinants (such as age, gender identity, disability, etc.) into multi-level models using random effects, as opposed to the traditional method of including them as fixed effects in single-level regression models [7, 9–11]. Multi-level modelling approaches have been developed to deal with complex structures, including those where it is possible to identify atomic units nested into higher level units [12]. This new approach uses a multi-level model structure where intersectional strata combining all non-empty intersections of socio-demographic and functional diversity variables are placed at level 2 (Fig 1). The use of this complex structure is necessary to account for the non-independence of experiences of gender-based violence within different sets of social relations. In this article, we use the terminology ‘sets of social relations’, which Walby and colleagues [5] recommend instead of ‘categories’, to draw greater attention to power relations and inequalities between categories, and the actions of the powerful within categories. This terminology is thus more aligned to an understanding of intersectional inequalities that is structural, rather than merely linked to individual identity categories. Intersectional strata representing these sets of social relations are formed by creating membership groups along a predetermined set of characteristics. This intersectional multi-modelling approach, often referred to as multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) [9] allows for an assessment of between-stratum variance; within-stratum heterogeneity; discriminatory accuracy by looking at the magnitude of within- and between-stratum variance relative to each other; how much between-stratum variance is explained by additive terms; and as with other modelling approaches, intersectional MAIHDA can also be used to estimate predicted probabilities or expected (mean) values of dependent variables for each stratum [8].
The use of intersectional multi-level modelling can be regarded as aligned with both the inter- and anti-categorical approaches to intersectionality outlined by McCall [2]. The creation of intersectional strata responds to the anti-categorical approach, which criticises the use of categorisation in the first place as it is seen to reify existing inequalities and fail to capture the complexity of different identities. Main effects provide information that is aligned with an inter-categorical approach by providing information on differences between groups on the basis of one or more individual membership criteria [13, 14]. In addition, the use of intersectional multi-level modelling is useful to shift the focus from inequalities located at the individual level, and towards inequalities stemming from an organisational/societal level imbalance of power at the structural level [5, 15].
What is not clear, however, is the number of variables that can be used for the construction of these intersectional strata. On the one hand, intersectionality theory would suggest, or even dictate, that it is desirable to create as many strata as possible, to attempt to reflect the complex reality of various intersecting identities. On the other hand, this presents a mathematical challenge in that where a stratum has too few observations, by construction, estimates of group parameters tend to the grand mean, providing little information about the group. This is due to the fact that higher level residuals are calculated as a mean raw residual (, i.e. the mean of
for group j), and then multiplied by a shrinkage factor
[16]. The shrinkage factor will adopt lower values where
is large relative to
(that is when variation is located more within individual heterogeneity than between strata) or where nj (strata size) is small. Intersectional quantitative researchers using MAIHDA approaches agree that the method performs best with larger samples [10]. Yet, few guidelines are available on the desirable floor value to choose for nj, though Persmark and colleagues [13] imply adopting a minimum threshold of 50. Evans [8] for example also emphasises the need to ensure a minimum number of observations across most, if not all, stratum but does not specify any rule of thumbs. Milliren and colleagues [17] also suggest a lower bound of about 5 to 20, though they note that including some groupings with fewer observations is not problematic due to the shrinkage factor.
In addition to these concerns about the number and size of intersectional strata, Persmark and colleagues [13] note as a limitation that they cannot take into account geographical location, which would a clear level to be used in modelling. Their concern is that it would necessarily reduce numbers within each stratum. However, this contextual information can be important, conceptually [15] as well as empirically [8]. To incorporate this additional level (or even more levels than one), it is necessary to consider that a nested structure is no longer sufficient to represent a complex reality, and that a cross-classified model is needed instead [17, 18]. This cross-classified structure (Fig 2) incorporates not only intersectional strata, but also organisational and national levels in which individuals are nested. The use of a cross-classified model, as opposed to a nested model, not only allows for the simultaneous analysis of multiple contexts but also mitigates the effects of potential ‘omitted context bias’ [17, 19]. While the recommendations for stratum size have been discussed in relation to nested model, these have seldom been examined in the context of cross-classified models. Exceptions are the work of Milliren and colleagues [17], though this work focuses on random effects rather than fixed effects, or Evans [20] which focuses on the robustness of the model after adding a contextual level.
The contribution of this paper is thus a methodological one by considering two questions in the application of the MAIHDA approach. The first aim is to test the extent to which increasingly complex combinations of variables used to compute intersectional strata, and thus increasing the number of intersectional strata while decreasing the number of observations within each stratum, has an effect on between-stratum variance measured by computing the variance partitioning coefficient (VPCs). The question of interest is whether between-strata variance is affected by the choice of different configurations of intersectional strata. The second aim is to examine whether coefficients, and thus information about the fixed effects of different traits used to construct intersectional strata, remain stable across models using an increasingly large number of levels, thus both increasing the number of strata and diminishing the number of observations within each stratum. The question of interest for this second aim is whether inferences about different sets of social relations are robust when not only considering intersectionality, but also contextual levels such as countries, and whether it is therefore feasible to also incorporate other classification levels (e.g. organisational, national) in a cross-classified structure given that this increases the number of strata, but decreases the size of each strata as each intersectional strata are then subdivided into groupings from other levels.
Methods and sample
This paper contributes to literature on the MAIHDA approach by (1) illustrating whether adopting an increasing number of intersectional strata has an effect on between stratum and (2) whether coefficients change when using a cross-classification structure to also consider how individuals are nested both in institutions and countries. It relies on data on the prevalence of gender-based violence in research organisations, using a survey of n = 42,000+ respondents, which was designed to capture differences in the experiences of gender-based violence between different groups characterized by socio-demographic or functional diversity. The survey focused on experiences of gender-based violence in an academic context, and data were collected in 46 universities and other research organisations across 15 European countries between 17 January and 1 May 2022 [21]. The data collection was approved by the Leibniz Institute for the Social Sciences (GESIS) Ethics Committee (Approval number: 2021–7). Respondents to the survey provided written consent, and their answers were recorded anonymously with no possibility for reidentification. The data were accessed between 1 July and 15 November 2023 for the purpose of this analysis. As the dependent variable is dichotomous (having experienced any form of violence asked about or not), a logit link function is used.
The creation of intersectional strata is performed on the basis of eight variables: gender identity (women; men; non-binary or another gender identity not listed), trans status (current gender aligned with sex at birth; not aligned), sexual orientation (asexual; bisexual; heterosexual; homosexual; queer; another sexual orientation not listed), ethnic minoritised group (no; yes), disability or chronic illness (no; yes), age (up to 20 years; 21–25; 26–30; 31–35; 36–40; 41–45; 46–50; 51–55; 56–60; 60 and above); target group (staff; student); mobility (domestic; international). Intersectional strata are built iteratively, progressively building up to more combinations of these categories, and resulting in a total of n = 255 scenarios (the different combinations are listed in Table 1). When all variables are used in combination, this represents a total of 5,760 (3×2×6×2×2×10×2×2) distinct intersectional strata, though removing all empty strata yields n = 1,127 intersectional strata in the analysis. While there are on average about 34 individuals per intersectional strata in this scenario, it is important to note that this distribution is heavily skewed to the right. Only 61% of intersectional strata consist of more than a single individual, 22% have 10 or more individuals, 11% have 30 or more, and 8% have 50 or more. Using this high number of intersectional strata thus produces numbers in each stratum that are well below those used in the literature [see for example 7, 8].
The analytical approach consists of two parts. The first part examines the effects of increasingly complex intersectional strata on VPCs, using two related models. The first model is a null model (also known as the variance components model) which measures the extent to which differences in prevalence are located at the level of intersectional strata, i.e. how much difference there is between the strata. This is measured via the variance partitioning coefficient, calculated as , where
is the variance between strata and where the variance between individuals within strata is estimated by
which corresponds to the variance of the logistic distribution, under the assumption that prevalence can be regarded as a latent response. In the case of a two-level model, the variance partitioning coefficient corresponds to the intraclass correlation coefficient. The higher the VPC, the more variation is located between different groups. The second model is an additive main effects model (also known as random intercept model) which extends the first model by integrated the variables used to calculate intersectional strata as fixed effects for all models with at least two variables used in the intersectional strata (where there is only one variable, it is not desirable for it to be included both in the fixed and random part of the model), allowing VPC values to be computed. The first and second models are run iteratively, using increasing complex combinations for the intersectional strata. The results for each of the 255 combinations are provided in Table 1. The VPCs generated are then used in a simple OLS regression on the variables used to construct intersectional strata, and on intersectional strata characteristics (number, minimum size, average size, maximum size). Potential multicollinearity is checked by examining Variance Inflation Factors.
In the second part of the analysis, only the model with the greatest number of intersectional strata is retained, i.e. making use of the intersections provided by all the variables described above, and extended to a cross-classified model [18]. The complexity of the model is progressively increased, from a single-level logistic model (M1); to multi-level models including respectively a level for intersectional strata (M2), organisational level (M3) and national level (M4); and finally including all three levels in a cross-classified model (M5). These models are first estimated without any fixed effects (M1a to M5a), and subsequently with all the variables used to construct the intersectional strata as fixed effects (M1b to M5b). In the case of M1, the null model corresponds to the odds, while the additive main effects model consists of a logistic regression model. This analysis provides an examination of the stability of the estimates for the coefficients across increased complexity of the social structure considered in the model, and thus about whether their interpretation is similar.
All models are fitted through the external software package ‘runmlwin’ [22, 23] within Stata v17. The MCMC algorithm is used with a 5,000 iterations burn-in period followed by a monitoring period of 50,000 iterations and thinning every 50 iterations, with initial values provided by the IGLS (PQL2 method) parameter estimates [24].
Results
The variance partitioning coefficients generated from the simulation of null models using increasingly granular intersectional strata have a mean of 7%, and range from 1% to 21%. As expected, the interclass correlation coefficients for the additive main effects models are lower overall, and range from 0% to 12%, with an average of 4%. The results for all possible combinations of intersectional strata are provided in Table 1. The results show that the variation in VPCs is related to the stratum specification used, with some of the characteristics used to construct the intersectional strata associated with higher VPCs at the intersectional level, as illustrated in Table 2.
To further explore how the variance partitioning coefficients relate the characteristics of these intersectional strata, simple regression models with log-transformed VPCs (expressed as percentages on a scale from 0 to 100) as the response variable are used. The VPCs were log-transformed due to a small skew in their distribution (Figs 3 and 4). Regression models were fitted using VPCs with and without this log transformation, with similar results. Only the log-transformed results are presented.
The models (Table 3) examine how different characteristics of intersectional strata relate to variance partitioning coefficients, to understand whether the size of intersectional strata affects VPCs. The results show that average size and maximum size are statistically significant. However, what is worth noting more than this statistical significance is the magnitude of these effects, as these are so small as to lose any practical significance. The number of intersectional strata is not related to the magnitude of variance partitioning coefficients.
The aim of the second part of the analysis is to understand whether the use of an increasing number of levels in a cross-classified structure affects conclusions that could be reached, owing to increasingly small strata when intersectional strata are broken down further by country and institution (Tables 4 and 5). All models suggest that there is very little variation in the level of gender-based violence at country or institutional level. Some variation is present across intersectional strata, when covariates are not included. However, once fixed effects are added, there remains very little variation at the intersectional of the sets of social relations included in the analysis.
The analysis examines whether the use of increasingly complex strata, here not only by intersectional sets but also countries and institutions, affects the results. This might be the case as doing so means using more numerous groups, hence relying on strata with small membership. In M5b, for example, the model incorporates 6,205 strata, which range from size 1 to 923, and with an average of just 6.1. An examination of the coefficients across M1b to M5b demonstrates that interpretations about the odds of experiencing gender-based violence for different group characteristics (considering all other variables, strata membership, and noting possible omitted variables bias) remain relatively stable. It can be concluded that for the purpose of inferences about a population, the use of very granular strata is thus appropriate. Going even further, it is possible to argue that it is not only appropriate but in fact desirable, because this approach is most aligned to the principles of analysing data both intersectionally and in context [15].
Conclusion
The paper contributes to scholarship on the Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy (MAIHDA) approach by exploring two main questions. The first question examines how using increasingly complex combinations of variables to create intersectional strata affects between-stratum variance, as measured by the variance partitioning coefficient (VPCs). The results show that the number of intersectional strata used appears to be unrelated to between stratum variance. The second question investigates the stability of coefficients for the fixed effects of different characteristics used in constructing intersectional strata across models in a cross-classified model with additional levels. This aims to assess the robustness of inferences about social relations when considering not just intersectionality but also other contextual levels like organisations and countries. The results suggest that increasing the number of levels, and as a result increasing the number of strata and diminishing the number of individuals therein, does not hinder the potential conclusions that can be reached about the effects of individual strand of identities. Further work should consider whether the results obtained in this analysis can be replicated using a simulated dataset, as this would provide insights from knowing with certainty about underlying relationships and how the results would appear in different parameter spaces. In addition, these results were obtained on the basis of a binary outcome, and ought to be replicated for other types of response variables including continuous normally distributed outcomes as well as ordinal or non-normally distributed outcomes.
What can be concluded more generally from these results? These results do not mean that the advice given in the literature [8, 13] to pay attention to the structure of intersectional strata should not be heeded. Instead, it points to the importance of stressing the ‘and’ in the MAIHDA approach: Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy. There are different purposes for which the approach can be used, and which should therefore be clarified explicitly at the outset of any analysis. The results presented here show that if the purpose is to include interaction terms to better account for structures of power and inequalities, then a greater number of intersectional strata is possible, and indeed desirable in line with the transformational aim intersectionality theory [4, 5]. Using fewer intersectional strata is a loss of information that, from an intersectional perspective, is hard to justify as the aim should be to provide data about all categories, and particularly the most minoritised ones, which also tend to be the smallest strata. However, if the aim is to provide intersectionality-sensitive estimates, for example here of the prevalence of gender-based violence across different sets of social relations, then as suggested in the literature, a large number of intersectional strata may not be informative because of the shrinkage property, whereby small intersectional strata’s estimates revert to the grand mean. As Bell [25] emphasises, the two main aims of MAIHDA–that is either to understand which grounds and intersectional matter, and provide specific measures for these intersections–have yet to be fully worked out in relation to their implications for policy and practice. This analysis reminds us of the need to clarify and remember the purpose for which MAIHDA is used, and whether the aim is descriptive (classification to understand if intersectionality matters) or inferential (how to address inequalities through understand the axes along which they operate).
References
- 1. Crenshaw K. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Policies. The University of Chicago Legal Forum. 1989;1989:139–67. pmid:21675331.
- 2. McCall L. The Complexity of Intersectionality. Signs: Journal of Women in Culture and Society. 2005;30(3):1771–800.
- 3. Merlo J, Mulinari S, Wemrell M, Subramanian SV, Hedblad B. The tyranny of the averages and the indiscriminate use of risk factors in public health: The case of coronary heart disease. SSM—Population Health. 2017;3:684–98. pmid:29349257
- 4. Hancock A-M. When Multiplication Doesn’t Equal Quick Addition: Examining Intersectionality as a Research Paradigm. Perspectives on Politics. 2007;5(1):63–79.
- 5. Walby S, Armstrong J, Strid S. Intersectionality: Multiple Inequalities in Social Theory. Sociology. 2012;46(2):224–40.
- 6. Weldon SL. The Structure of Intersectionality: A Comparative Politics of Gender. Politics & Gender. 2006;2(2):235–48. Epub 2006/08/30.
- 7. Evans CR, Williams DR, Onnela J-P, Subramanian SV. A multilevel approach to modeling health inequalities at the intersection of multiple social identities. Social Science and Medicine. 2018;203:64–73. Epub 20171130. pmid:29199054.
- 8. Evans CR. Reintegrating contexts into quantitative intersectional analyses of health inequalities. Health & Place. 2019;60:102214. pmid:31563833
- 9. Merlo J. Multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) within an intersectional framework. Soc Sci Med. 2018;203:74–80. Epub 20171226. pmid:29305018.
- 10. Mahendran M, Lizotte D, Bauer GR. Quantitative methods for descriptive intersectional analysis with binary health outcomes. SSM—Population Health. 2022;17:101032. pmid:35118188
- 11. Mahendran M, Lizotte D, Bauer GR. Describing Intersectional Health Outcomes: An Evaluation of Data Analysis Methods. Epidemiology. 2022;33(3):395–405. 00001648-202205000-00013. pmid:35213512
- 12.
Rasbash J. Multilevel structures and classifications. LEMMA (Learning Environment for Multilevel Methods and Applications): Centre for Multilevel Modelling, University of Bristol; 2008.
- 13. Persmark A, Wemrell M, Evans CR, Subramanian SV, Leckie G, Merlo J. Intersectional inequalities and the U.S. opioid crisis: challenging dominant narratives and revealing heterogeneities. Critical Public Health. 2020;30(4):398–414.
- 14. Axelsson Fisk S, Mulinari S, Wemrell M, Leckie G, Perez Vicente R, Merlo J. Chronic Obstructive Pulmonary Disease in Sweden: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy. SSM—Population Health. 2018;4:334–46. pmid:29854918
- 15. Humbert AL, Strid S, Tanwar J, Lipinsky A, Schredl C. The role of intersectionality and context in measuring gender-based violence in universities and research performing organisations in Europe for the development of inclusive structural interventions. Violence Against Women. forthcoming.
- 16. Steele F. Introduction to Multilevel Modelling Concepts. LEMMA VLE Module 5. Centre for Multilevel Modelling, 2008.
- 17. Milliren CE, Evans CR, Richmond TK, Dunn EC. Does an uneven sample size distribution across settings matter in cross-classified multilevel modeling? Results of a simulation study. Health & Place. 2018;52:121–6. pmid:29885555
- 18. Leckie G. Cross-Classified Multilevel Models—Concepts. LEMMA VLE Module 12 2013. 1–54]. Available from: https://www.cmm.bris.ac.uk/lemma/.
- 19. Barker KM, Dunn EC, Richmond TK, Ahmed S, Hawrilenko M, Evans CR. Cross-classified multilevel models (CCMM) in health research: A systematic review of published empirical studies and recommendations for best practices. SSM—Population Health. 2020;12:100661. pmid:32964097
- 20. Evans CR. Adding interactions to models of intersectional health inequalities: Comparing multilevel and conventional methods. Social Science & Medicine. 2019;221:95–105. pmid:30578943
- 21. Lipinsky A, Schredl C, Baumann H, Humbert AL, Tanwar J, Bondestam F, et al. UniSAFE Survey–Gender-based violence and institutional responses: GESIS—Leibniz-Institute for the Social Sciences. Data File Version 1.0.0; 2022. Available from:
- 22. Charlton C, Rasbash J, Brown WJ, Healy M, Cameron B. MLwiN version 3.06: Centre for Multilevel Modelling, University of Bristol; 2022.
- 23. Leckie G, Charlton C. runmlwin: A Program to Run the MLwiN Multilevel Modeling Software from within Stata. Journal of Statistical Software. 2013;52(11):1–40.
- 24. Browne WJ. MCMC Estimation in MLwiN v3.06: Centre for Multilevel Modelling, University of Bristol; 2022.
- 25. Bell A, Holman D, Jones K. Using Shrinkage in Multilevel Models to Understand Intersectionality. Methodology. 2019;15(2):88–96.