Round 1: Development of measurement framework.

The first round of the modified Delphi study consisted of developing a framework for the measurement of structural stigma and discrimination in healthcare systems through a literature review and expert consultation workshop.

Identification of literature and experts. Literature related to structural stigma and discrimination in mental healthcare was identified through a non-systematic literature review. A search of the literature was carried out using the terms “structural” AND “stigma” OR “discrimination” AND “mental illness” OR “mental health” in multiple databases: Web of Science, PsychINFO, and Google Scholar. The search was limited to articles published in English and the top 40 hits in each database were reviewed by DG and BS and their bibliographies were checked for cross-references.

PWLEs and stigma researchers/academics were recruited for the consultation workshop as expert panellists. The focus on researchers/academics and PWLEs for the consultation was mainly due to the aim of the workshops to explore conceptual understanding of structural stigma and identify indicators that mattered most. The panellists for the consultation workshop were identified through the INDIGO Partnership Research Programme [19] and Reducing Stigma among Healthcare Providers (RESHAPE) study [20]. The two projects were specifically chosen because both the projects focused on mental health-related stigma reduction and had the involvement of stigma researchers/academics and PWLEs.

Process of Round 1. Identified literature from databases and cross-references were reviewed by authors (DG & BS) to list indicators that were suggested and/or used to assess mental health-related structural stigma and discrimination. The list of indicators was then categorized into groups based on common themes and sorted into the domains in the framework proposed by Livingston on the measurement of structural stigma and discrimination in mental healthcare systems [14].

Four consultation workshops were carried out between April and June 2023. Three were online consultation workshops with 22 international experts including stigma researchers, program implementers, policy makers, and PWLE advocates (3 to 7 participants per workshop, discussions lasting between 1–1.5 hours). Workshops were recorded and conducted in English. Workshops consisted of free listing exercises of indicators for measurement of structural stigma and discrimination in the mental healthcare system using an online tool for free-listing and rating (https://easyretro.io/). Experts were then asked to rank the list of indicators identified through the free listing and sort them into the domains proposed by Livingston [14]. Understanding of structural stigma was explored during Round 1 of the Delphi exercise. Participants were asked to critique and disaggregate the definition of structural stigma provided by Hatzenbuehler and colleagues [10]–“societal level conditions, cultural norms, and institutional policies that constrain the opportunities, resources, and wellbeing of the stigmatized” and provide examples of mental health related structural stigma they have experienced or seen within the healthcare system settings.

A separate face-to-face workshop with 8 PWLEs from Nepal was conducted on June 25^th, 2023 after approval from local ethical body. The discussions focused on their understanding of structural stigma and discrimination, experiences of structural stigma in healthcare systems, and how they think this form of stigma and discrimination can be measured. The discussions carried out in Nepali were audio-recorded and transcribed/translated by an experienced Nepali researcher in English.

Adaptations were made to the measurement Livingston framework based on the insights from Round 1, after which, the revised framework was shared with expert panellists in Round 2.

Round 2: Selection and ranking of indicators.

Expert selection and recruitment. The criteria to participate in the Delphi study as expert panels were:

1. Leading health-related stigma researcher; AND/OR
2. Anti-stigma program implementer; AND/OR
3. Policymaker involved in the drafting or implementing of healthcare systems-related policies; AND/OR
4. A person with lived experiences of mental health conditions

The expert panel members were first identified from the mailing list of stigma researchers, anti-stigma program implementers, and policymakers who were part of the INDIGO network [19, 21] and the Lancet Commission on Ending Stigma and Discrimination in Mental Health [22]. They were approached via email to participate in the Delphi Survey after getting permission from the Principal Investigator and commission chairs. Additionally, they were asked to nominate individuals within their network who met the above criteria for inclusion. Thus 110 experts were identified and approached through this method. This also included panellists involved in Round 1 of the study.

PWLEs of mental health-related stigma were sent a link for the Delphi survey to which they could respond anonymously. The emails were sent by the president of the Global Mental Health Peer Network (GMHPN) to their network members. The GMHPN is an international organization led by people with lived experiences and consists of more than 300 network members from 45 countries around the globe [23]. Round 2 survey was active from 20^th June to 12^th July 2023 and completed by 53 panellists.

Development of the survey and administration process. The Qualtrics XM platform [24] was used for the development of the online survey. The survey consisted of study consent, demographic questions, and a list and description of indicators for the domains identified from Round 1 within each of the 5 domains. Panellists could also add up to 3 further indicators to the list if they felt these aspects were missing. Panellists were asked to select up to 5 indicators (from the list of indicators, including any new ones they added) in each domain. These were then ranked according to three criteria: 1) Appropriateness (whether the indicators were meaningful or appropriate in capturing structural stigma in a mental healthcare setting); 2) Usefulness (whether the indicators were sensitive to change and useful in comparing from one setting to another); and 3) Feasibility (whether the information for the indicators was feasible to collect). At the end of each domain, any additional comments could be added via an open-ended field. The participants were given options to complete the survey anonymously or provide email address for further communications regarding the study if required.

Round 3: Re-ranking of indicators.

For the third round, we compiled the summary results of Round 2 that included the mean rank and average Smith’s Salience (of 3 criteria- appropriateness, usefulness, and feasibility) of the top 5 indicators for each domain. Everyone approached for Round 2 was sent the survey link to Round 3 irrespective of their participation in Round 2. Panellists were asked if they agreed with the summary results, and if not, they were asked to re-rank the indicators from the full list of indicators (including the indicators added by participants in Round 2). Additionally, panellists were asked an open-ended question on any comments regarding each domain. A specific question was added on the appropriateness of measurement domain 3 (‘aggregate stigmatizing attitude and practices of individuals affiliated with the health systems’) to measure structural stigma, in response to queries raised in Round 2 feedback. Round 3 survey was active from 10^th September to 4^th October 2023. Similar to Round 2, the participants were provided with option to complete survey anonymously or opt to provide their email for future correspondence on the topic.

Round 1: Development of measurement framework.

The indicators identified through the literature review were listed in an Excel sheet and the frequency of citation of the indicators was listed. The qualitative transcripts from the consultation workshops with PWLEs and stigma experts were coded and charted in the Excel sheet using broad a priori themes: i) understanding of structural stigma and its definition; ii) barriers and facilitators to the measurement of structural stigma; iii) measurement domain specific comments and feedback; and iv) indicator-specific comments and feedback.

The free-listed indicators during the consultation workshops were sorted into the domains provided by Livingston [14] and the experts themselves. The domains were modified based on the feedback from experts during the workshops. Any additional indicators identified during discussions were supplemented by the indicators identified during the literature review. These indicators too were then sorted into the adapted domains after the consultation workshops.

Round 2: Selection and ranking of indicators.

For the second-round survey, as panellists were asked to select up to 5 indicators from a list of indicators and rank them, panel consensus was not just limited to the ranking of indicators but also the item selection. Hence, Smith’s Salience Index (Smith’s S) was used as a measure of consensus for the survey. Smith’s S is a function that considers the frequency of item selection and its average rank and can be defined as: S = (∑ (L_i − R_ij + 1) / L_i) / N where R_ij is the rank of the item “j” in the list given by respondent i, L_i is the length of respondent i’s list, and N is the total number of lists in the sample [25]. Smith’s S provides a value between 0 and 1 and has been used as a measure of consensus in previous Delphi studies where values closer to 1 were considered higher consensus [26]. Smith’s S above 0.5 were “highly salient”, 0.3–0.49 was “moderate salience”, and below 0.3 was “low salience” [26].

FLARES software for cultural analysis [27] was used to analyse the data. The dataset from Qualtrics software was downloaded and entered in the FLARES software, where mean rank, frequency of selection, and Smith’s S were analysed statistically for each of the 3 criteria (appropriateness, usefulness, feasibility) in each domain. A similar analysis was conducted on the dataset with responses only from panellists identifying as PWLEs. The Smith’s S across the 3 criteria were aggregated using average mean giving equal weights to all 3 criteria. A list of the top 5 ranked indicators (as determined by the average salience) in each domain was created and shared with the panellists for re-ranking in Round 3. Open-ended narrative feedback on the survey was reviewed and synthesized, based on which further follow-up questions were added in Round 3.

Round 3: Re-ranking exercise.

At the end of Round 3, rather than the frequency of item selection, panellists were asked to re-rank the list of full indicators (including the ones added by panellists in Round 2) if they disagreed with the ranking from Round 2. This was done because of moderate to low salience seen in Round 2 due to variability in item selection. Hence, the calculation of Smith’s S considered the ranking of the indicators as the frequency of selection for all indicators in this round was 1. Similarly, to assess the agreement between panellists in the ranked order of items, Kendall’s concordance of coefficient (Kendall’s W) was calculated. Kendall’s W is a non-parametric test for rank correlation that ranges from 0 (no agreement) to 1 (complete agreement) [28]. The Kendall’s W test was conducted in IBM SPSS for Windows version 29.0 [29]. Similar to Round 2, a sub-analysis of responses from PWLEs was conducted to highlight any differences in opinions. Open-ended feedback on the survey was reviewed thematically and reported in a narrative format.

Understanding of structural stigma and its definition.

The experts during the consultation workshops commented on the vagueness of societal-level conditions as not capturing anything substantial while some reflected that ‘cultural norms’ and ‘societal conditions’ could mean the same thing which is focused on stereotypes and their use in institutional practices. Everyone agreed that the ‘institutional policies’ in the definition were more specific and understandable, which may be why it is widely used as an indicator of structural stigma. However, they also cautioned that the definition does not include ‘lack of policy’, which in many cases reflects structural stigma.

Among the PWLEs in Nepal, it was difficult for them to comprehend the definition and concept of structural stigma, as they viewed stigma in terms of their personal experiences and negative interactions in the healthcare setting. However, almost all the PWLEs highlighted the lack of availability of medications and lack of trained mental health personnel in local healthcare settings as a key form of discrimination. Similarly, when talking about negative experiences, they shared the fear and confusion they faced due to the lack of information provided by staff at hospitals during their involuntary admission. “I thought I was imprisoned in the hospital ward forever without any prospect of going back home…No one told me how long I was admitted for. I thought- I wasn’t like other ‘crazy’ people around me, I didn’t show such symptoms. So, I couldn’t understand why I was imprisoned with them. The fear and confusion made my mental health even worse” (Male, PWLE from Nepal).

Other examples of structural stigma and discrimination shared during expert consultations were mainly policy-specific (e.g. lack of mental health policy or discriminatory policies), resource-related (e.g. lack of private space, unavailability of mental health-related medications, and lack of funding for mental health), and service-related (unavailability of mental health services at local health facilities and poor quality of mental healthcare compared to other health conditions). A psychiatrist shared how medical personnel such as nursing staff were reprimanded for bad behaviour by placing them in the mental hospital. Another expert highlighted structural discrimination in the health education of medical students. “In medical examinations of emergency medicine, even if medical students fail mental health-related competency evaluations for mental health, you could still pass. However, that is not the case for other medical conditions”. (Female, Psychiatrist/stigma researcher). Similarly, district nurses refusing to visit homes of people with severe mental health conditions because they can walk and come to General Practice and only visit homes of people who are physically unable to walk was described as a discriminatory practice that could fall under structural stigma.

Barriers and facilitators to the measurement of structural stigma.

One of the key barriers highlighted by the experts was the difficulty in measuring structural stigma due to the broadness of the concept and the difficulty in operationalization of the concept. Experts struggled to operationalize the ‘cultural norms’ aspect of the structural stigma. Experts mentioned that structural stigma takes a long time to translate into action and contrarily, any action targeting structural stigma takes a long time to show outcomes, which could be the reason for the dearth of research in the field. An expert suggested that the difficulty in operationalizing structural stigma could be because of the structural stigma itself. “Structural stigma in itself means there is a lack of indicators and studies conducted on having the right measures. Or the workforce being limited in terms of time to do research or even think about structural forms of stigma and discrimination within healthcare systems or healthcare policies.” (Female, Stigma researcher)

Another challenge highlighted in the measurement of structural stigma was the comparative aspect of mental health-related structural stigma. An expert commented- “For me, structural stigma needs to have a sense of ’differential’ impact compared to something else. I chose ’unavailability’ of medicines/mental health indicators because we know that other health conditions have those and so unavailability indicates something that is differential” (Female, Stigma researcher). Some experts discussed possibilities of comparison with chronic or long-term health conditions while others cautioned against such medical comparisons and advocated for social disability framework.

Domain-specific feedback.

One of the key points raised by experts in the consultation workshops (Round 1) was the inter-connectedness of the Livingston domains. The participants reflected that it was difficult to sort indicators into domains because they were interrelated as causes and consequences. Indicators within one domain could lead to another and might fit into both domains. Experts reflected that it would make better sense to merge some of the domains such as “denial of care” and “coercive approaches” to make it simpler and add other domains to accommodate indicators like lack of mental health policies or discriminatory policies.

During the online surveys (Rounds 2 and 3), most domain-specific comments were provided in domain 3 (aggregate stigma attitude and practices of individuals within healthcare systems). Panellists commented on the domain not being reflective of structural stigma and asked for the removal of the domain as it “refers to attitudes and behaviours of individuals, and to call this structural stigma seems self-contradictory” (Male, Stigma researcher). In Round 3, participants were asked if they agreed to the removal of the domain from the framework and to provide reasons for their response. Sixty-four percent of the panellists (80% PWLEs) disagreed with the statement that the domain did not reflect structural stigma. Almost all the experts stated that the individuals make up the structure and so are the enablers of discriminatory culture and practices within the structure. A panellist remarked- “Attitudes and behaviour of professionals reflect organizational culture so while they are measured at the interpersonal level can be seen as an aspect of structural stigma; the same people might behave quite differently in another organization and given better training” (Female, stigma researcher). Alternatively, some experts suggested that the domain could be mentioned as supplementary to the other domains as a cause or consequence of structural stigma rather than fitting it within the framework, while others (mostly PWLE panellists) advocated for fitting it within the framework as a pathway that reflects structural forms of discrimination.

Indicator-specific feedback.

Feedback on making indicators specific, adding indicators that are from PWLE’s perspective, and feedback on the relevance or irrelevance of certain indicators in the domains were some examples of indicator-specific feedback provided by panellists.

Most panellists commented on the indicator “involuntary/compulsory treatment of PWLEs” in domain 4. Some panellists reflected that the indicator might not be reflective of structural forms of discrimination, as it may be necessary in some cases to treat people involuntarily. A PWLE panellist mentioned “Involuntary admission should be nuanced. As a PWLE, I’m grateful to my parents for involuntarily getting me to treatment when I was in a manic episode. Involu ntary treatment is sometimes the solution than the problem” (Female, PWLE).

Similarly, feedback was received on the importance of indicators such as “unavailability of medications”, “underfunding of the mental health sector”, and “differential quality of space for mental health”, especially for LMICs. A panellist mentioned- “It will be difficult to measure the quality of the space, but stock-outs or non-availability of essential medications for mental health is easy to measure. Also, medication access is highly valued by people with lived experience in studies from Africa” (Female, Stigma researcher).