https://orcid.org/0000-0001-7221-7645
Figures
Abstract
The majority (40%) of the world’s under-five mortality burden is concentrated in nations like Nigeria (16.5%), India (16%), Pakistan (8%), and the Democratic Republic of the Congo (6%), where an undetermined number of under-five deaths go unrecorded. In low-resource settings throughout the world, the Verbal Autopsy-Social Autopsy (VASA) technique may assist assess under-five mortality estimates, assigning medical and social causes of death, and identifying relevant determinants. Uncertainty regarding missing data in high-burden nations like Pakistan necessitates a valid and reliable VASA instrument. This is the first study to validate Child Health Epidemiology Reference Group-CHERG’s VASA tool globally. In Pakistan, data from such a valid and reliable tool is vital for policy. This paper reports on the VASA tool in Karachi, Pakistan. Validity and reliability of the CHERG VASA tool were tested using face, content, discriminant validation, and reliability tests on one hundred randomly selected mothers who had recently experienced an under-five child death event. Data were computed on SPSS (version-21) and R software. Testing revealed high Item-content Validity Index (I-CVI) (>81.43%); high Cronbach’s Alpha (0.843); the accuracy of between 75–100% of the discriminants classifying births to live and stillbirths; and I-CVI (>82.07% and 88.98% respectively) with high accuracy (92% and 97% respectively) for assigning biological and social causes of child deaths, respectively. The CHERG VASA questionnaire was found relevant to the conceptual framework and valid in Pakistan. This valid tool can assign accurate medical and non-medical causes of child mortality cases occurring in Pakistan.
Citation: Siddiqui MB, Ng CW, Low WY, Abid K (2023) Validation of CHERG’S Verbal Autopsy-Social Autopsy (VASA) tool for ascertaining determinants and causes of under-five child mortality in Pakistan. PLoS ONE 18(12): e0278149. https://doi.org/10.1371/journal.pone.0278149
Editor: Ritesh G. Menezes, Imam Abdulrahman Bin Faisal University, SAUDI ARABIA
Received: August 17, 2021; Accepted: November 11, 2022; Published: December 18, 2023
Copyright: © 2023 Siddiqui et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The datasets used and analysed during the current study are available on the following link. https://figshare.com/s/883fe45ea7da71483acd.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: C.A., Cronbach’s Alpha; C.C., Canonical correlation; C.V., Content validation; CHERG, Child Health Epidemiology Reference Group; CoD, Cause of Death; CRVS, Civil Registration and Vital Statistics; D.V., Discriminant Validation; E.V., Eigenvalue; HMIS, Health Management Information System; I-CVI, Item-content Validity Index; LR, Likelihood-Ratio (LR) test; MCoD, Medical Cause of Death; PCVA, Physicians certified VA technique; R.A., Reliability Analysis; SA, Social Autopsy; SCoD, Medical Cause of Death; SPSS, Statistical Package for Social Sciences; TPtoSCF, The Pathway to Survival Conceptual Framework; VA, Verbal Autopsy; VASA, Verbal autopsy/Social Autopsy; W.L., Wilk’s Lambda; WHO, World Health Organization
Introduction
Verbal autopsy (VA) is a technique that involves structured questions focusing on the signs and symptoms the deceased case faced during the illness but before the mortality incident. VA is used to record various relevant biological determinants (BDs) linked with child mortality incidents, and based on such information, VA helps in assigning the medical cause of death (MCoD). Similarly, social autopsy (SA), a relatively new technique with a questionnaire structure similar to VA, helps record non-biological determinants (especially the health-seeking process, and issues and barriers in the process of accessing health care during the child illness process but before the mortality event) so that the social cause of death (SCoD) of any child death can be concluded. VA has been used in low-resource settings when the cause of any child death (CoD) has not been ascribed by a health care provider [1–3]. In contrast to VA, SA is not typically used for child mortality in the majority of developing countries, especially Pakistan. Both VA and SA not only provide the MCoD and SCoD to the child mortalities respectively but also assist in updating epidemiological estimates on the mortality which have been missed to be recorded by the health system in these geographies [4, 5]. Although VA has been used in nations with little resources, there is still a need to investigate a wider range of data on the societal factors that contribute to child mortality [3, 6]. Hence, in a more recent timeframe, a concept of running through the missed mortality case with a VA and SA tool has been attempted in some countries (Cameroon [6, 7], Mozambique [8], Niger [9–11], Uganda [12–15], Rwanda [16, 17], Malawi [6], Nigeria [18], Guinea [19]) which helped to get detailed evidence of social causes of different medically assigned CoDs. However, these attempts made use of different VA and SA tools in combinations [20].
The Child Health Epidemiology Reference Group (CHERG) was established in 2001 with the support of WHO and UNICEF to provide technical guidance to relevant institutions and organisations for collecting evidence on determinants and causes of child mortality and improving global child survival epidemiological estimates [7, 21]. The agency developed an integrated VASA tool based on a most holistic approach for recording different biological determinants (child-specific, maternal-specific, determinants across the CoC, and preventive measures taken before and during the pregnancy and child illness) and non-biological determinants (NBDs) identified through several models (The Three Delay Model), conceptual framework (TPtoSCF); and several recommendations for VA suggested by the World Health Organization [22] to record different biological and non-biological determinants affecting the under-five mortality [20, 23–25]. In addition to recording biological determinants, this tool also focuses on all the barriers which come across as hindrances throughout the health-seeking process when the child was ill [21]. This tool is one of the assets that can be utilized to acquire information on the relevant biological and non-biological determinants of child mortality for every missed child mortality case on a single visit to the household and has already supported several countries to gather evidence on the medical and social causes to child deaths [10, 20, 26]. The extensive data obtained from CHERG VASA tool based investigations provides highly elaborated evidence and information on extended determinants of child mortality which is of great public health importance [4, 10, 26].
Similar to any data collection tool, it’s always been a concern whether the data collected through VA and SA are correct, reliable, and represent the information that the researcher intends to collect. The process of validation involves testing the questionnaire on validity and reliability parameters. To obtain the best possible data, many efforts have been undertaken globally in refining the methodology of VA [27–33]. In continuation of such efforts, a few validation studies have been undertaken globally as well in Pakistan [34] to assess the capacity of VA tools in providing precise and reliable evidence on the causes of child mortality [27, 28, 32, 33]. However, no attempts have been made to validate CHERG’s integrated VASA tool. Considering the current status of child mortality estimates in Pakistan [34–36], CHERG’s integrated VASA tool was validated to gather evidence on a comprehensive list of medical and social determinants and medical and social causes of under-five mortality in Pakistan, where child mortality cases may be missed to be registered under national CRVS and Health Management Information System. This is the first time a VASA integrated tool has been validated so thoroughly, concentrating on most of the approaches used to evaluate validity and reliability. We expect the validated tool would provide comprehensive data on under-five mortality in Pakistan. This paper presents the results of the validation of the CHERG VASA tool. The validated instrument will be used to conduct a future VASA child mortality survey in Pakistan’s underprivileged community.
Materials and methods
This validation study is a component of the Karachi VASA child mortality study [37] and is designed to objectively assess the validity and reliability estimates of the CHERG’s VASA questionnaire, which was later utilized in Karachi VASA child mortality study.
The sample size
Although there are no specific guidelines for the sample size of respondents required for pre-survey validation of the VASA tool [38], the only available guideline suggests [39] that, in general, the sample size required for any questionnaire validation should be a minimum of 20% of the total sample size of the study. Since 400 under-5 died cases were included in the Karachi VASA child mortality study, 20% of 400 cases (80 cases) were selected from the study sites to participate in the questionnaire validation study. The original sample size of 80 was increased to 100 to overcome the issue of potential data entry errors. Mothers of these 100 under-five deceased cases (n = 100) were selected from 12 selected sites i.e., Cheneser goth (CG), Mujahid colony (MC), Haryana Colony (HC), Sultanabad (SD), Bilal colony (BC), Sherabad (SA), Yaseenabad (YS), Landhi slums (LS), Khuda ki Basti (KKB), Safora goth (SG), Malir slum (MS), and Sultanabad (SD), based on the number of deaths occurred in these sites during the past one year (from June 1, 2015, to May 3, 2016) before the data collection. These 12 study sites were located in five of the city’s districts, into which the city was subdivided at the time of the study, namely, District Malir, District Central, District South, District West, District East, and District Korangi. Two study sites (one urban and another one urban-slum) were randomly selected from each of the six districts, making a total of 12 study sites for research purposes. The urban sites and the urban-slum sites were classified based on the definition of a slum from UN-Habitat [40]. To ensure that the sample was representative of the entire population, the sample (100 mothers of deceased under-five deaths, falling under strict inclusion and exclusion criteria) was drawn from all six districts of Karachi.
Sampling methodology
The estimates of all the under-five deaths (occurred between 01st June 2015 till 03rd May 2016) from each of the study sites were identified from each site through multiple sources (household visits i.e., snowball sampling; local graveyards; local town health and union council offices). To fulfil the sample size of 100 respondents for the validation study, each identified study site was allocated a quota based on the proportionate quota sampling technique (proportionally of the overall sample size required for the study). The details (quota calculation) are available in Table 1.
Inclusion and exclusion criteria
To reduce response bias, only women whose children under the age of five passed away during the previous year (from 01st June 2015 till 3rd May 2016) were included in this validation study.
After being informed of the study’s purpose and procedures, all participants were asked if they would be willing to participate and informed written consent was obtained. Those who did not wish to participate in the study were given the option to withdraw themselves from the study, and this was done with respect. All responses were treated as strictly confidential. It was ensured that, if a respondent experienced any emotional disturbance during the data collection process, the respondent would be referred to a health professional; however, no such case was encountered.
Structure of the CHERG’s VASA questionnaire
The CHERG’s VASA questionnaire [23] is a semi-quantitative tool for collecting data on the signs and symptoms of illness (including varied BDs) and the relevant history of social circumstances (NBDs) right from the conception of the mother until the death of the ill child including few open-ended questions i.e., the open narratives (which specifically asks for any additional notes to be mentioned by the respondents in relation to the Verbal autopsy component as well as the summary from the interviewer of the whole case). It also quantitatively captures the signs and symptoms a child encountered during the illness and the social history of the deceased child before the death of the child. The initial pre-validated CHERG VASA questionnaire consisted of VA and SA components having 14 different modules (sections) focusing on the different variables ranging from the demographic profile of the respondent, family of the deceased child, parents of the deceased child, healthcare-seeking process and socioeconomic status to variables encompassing the entire continuum of care during the pregnancy till the death of the child.
After the section on the VASA general information (which collects information on the general variables), the VA component has 06 modules altogether, while the SA component has 07 modules (Table 2, columns 1&2). These VA, SA, and pertinent modules have been organized in such a way that as the respondent’s interview progresses, the chronology of events becomes clear to the researcher (Table 2, column 3). To record relevant information on relevant age groups (i.e., neonatal deaths, and post-neonatal deaths), specific skip patterns exist in the tool.
Translation & back translation
The CHERG VASA tool is freely available [23]. The tool is available either in the integrated version of VASA or in different sections separately. Before validation, the questionnaire was forward translated into the local language, Urdu, and then back-translated to English based on the “Brislin Back-translation” technique [41]. The process involved a professional language translator and a panel of professional bilingual subject experts (paediatricians, neonatologists, gynaecologists, obstetricians, social scientists, and psychologists). The final translated version was processed for the validation process.
Data collection for the validation study
The respondents (mothers of deceased children) were recruited, and their written consents were obtained. A total of 100 VASA interviews were conducted. Data was recorded on both the platforms i.e., Paper-And-Pencil Interviews (PAPI)(i.e., paper-based VASA tool); and Computer Assisted Personal Interview software (CAPI)(an electronic platform format for data collection with skip patterns using CSPro software application) and were compared to check any inconsistencies. However, no inconsistencies were encountered between the PAPI and CAPI platforms. The completion rate of the questions on both PAPI and CAPI was the same (100%), however, there were some issues by the interviewee and interviewer on the implementation of these two methods which are elaborated in a qualitative study [42] The only difference between the two platforms was the ease of data collection and less time utilized for data collection by the data collectors [42]. Data quality was checked by a quality management team and researcher. None of the mothers refused or rejected the interview. Extensive training to the data collectors and data entry operators (with a minimum educational background of having Intermediate level (Higher Secondary School Certificate i.e., Grade-12) was given on communication skills, details of the project, questionnaire design and data entry process. Collected data were shared with a panel of subject specialists (five Paediatricians with expertise in Neonatology) trained for PCVA for reviewing the data and assigning the MCoD and SCoD (five sociologists) through VA and SA components of the integrated VASA tool. Medical CoD was assigned by the physicians (five Paediatricians with expertise in Neonatology) (through Physician Certified Verbal Autopsy-PCVA), where the PCVA trained physicians reviewed the mortality case reports (complete VASA data) including signs and symptoms, open narratives, medical history and sequence of events prior to death of deceased children and assigned the most probable MCoD. Initially the data was shared with one reviewer, who assigned a MCoD. Once received, the data was then shared with second reviewer and the level of agreement was checked. In case of any disagreement between the initial two reviewers, a third reviewer was assigned until the agreement between the two different reviewers was achieved. The MCoD assigned by the reviewers were kept confidential by the other reviewers. While conducting PCVA, reviewers were kept blinded with the MCoD assigned on the death certificates of those cases (under-five mortalities) with a record of death certificates. Similarly, the SCoD were assigned by the social scientists through the same process after reviewing the integrated VASA tool.
Tool’s validation procedure
As illustrated in Table 3 and Fig 1, the CHERG VASA tool was evaluated on its validity and reliability through face validation, content validation, discriminant validation, and reliability analysis and the details are described in the respective sections below.
Results
The sample of 100 respondents and the included households had the following demographic characteristics (Table 4). Only key demographics appear in Table 4. Full details are available can be accessed upon request.
Characteristics of included deceased children
The Table 5, mentions the details of some of the important characteristics of the deceased children included in the validation study.
(a) Face validity assessment.
For assessing the face validity of the tool, the questionnaire was reviewed and critically evaluated by a panel of 6 experts—a researcher and 5 expert(s) in the relevant field of interest (e.g., paediatrician, gynaecologist, obstetrician, social scientist, and psychologist [44, 45] on the tool’s format to find out, whether all the sections and items be easily readable, clear, and straightforward to understand and comprehend by the interviewees and respondents; and on the need of rephrasing, omitting, or adding any questions. The panellists found no need for any modifications in terms of rephrasing questions, omitting any option of any item, adding an option of any item, etc. The team identified that the items of the tool were phrased in such a way that the respondent would need to think before responding, thereby minimizing the chances of response bias. The translated questionnaire which went for face validation was then named “VASA Questionnaire-version-1”. This version of the questionnaire was utilized for content validation as well as to collect data for undertaking discriminant validation.
(b) Content validity assessment.
It involved a panel of five subject specialists (paediatrician, gynaecologists, obstetrician, social scientist, and psychologist) who gave their ratings on different parameters (especially on the comprehensiveness of the tool and the relativity of the items on the tool with the background conceptual framework of the topic). Each member from a panel was requested to rate every item on the VASA Questionnaire-version-1 on a 5-point Likert scale based on the relevancy to the conceptual framework (1 = not relevant, 2 = somewhat relevant, 3 = relevant, 4 = quite relevant, 5 = highly relevant). The ratings were calculated using Item-content Validity Index (I-CVI). For each item, the I-CVI was computed as the number of experts who rated an item on the top two boxes (i.e., quite relevant or highly relevant), divided by the total number of experts. The level of agreement between experts should be as high as possible to make the I-CVI acceptable. The data on content validation is presented in Table 6 under different themes/constructs. These themes/constructs were clustered either as pre-birth themes/constructs or post-birth themes/constructs, sections 1 through 8 cover pre-birth constructs, and sections 9 through 16 cover during and post-birth constructs. The average rating of each section was more than 4 (on a scale of 1 to 5), and the I-CVI is more than 80% for all the sections of the VASA questionnaire. Relatively lower I-CVI was observed for “symptoms before pregnancy” (81.43%) and “Birth History of Child” (81.82%) sections.
(c) Discriminant validity (construct validity) assessment.
It shows the distinctiveness of different constructs within the questionnaire and helps to identify whether the items on the questionnaire that are not supposed to be related to each other are unrelated or not. It is one of the types of construct validity. The “VASA Questionnaire-version-1” (after the face validation exercise) of the tool was utilized for data collection from the respondents who fall within the inclusion and exclusion criteria. All the mothers identified with a recent history of child mortality, and none of the mothers refused or rejected the interview. Collected data were shared with verbal autopsy reviewers (physicians). The reviewers used their clinical reviewing expertise (a process called “Physicians certified VA technique-PCVA”) in assigning the medical cause of death. The complete data (including the CoD assigned by the physicians) were computed on SPSS (version 21) and R for the validation process. For pre-birth constructs, “birth status of the child” and for during and post-birth constructs “diseases or conditions directly leading to death” are the decisive elements with potential discriminating power. Clear clusters and grouping in response patterns on VASA elements were found for the pre-birth constructs. Therefore, valid response differences between the live birth group and stillbirth group (on the elements of each section) were used for the discriminant validity of the pre-birth constructs of the tool. Similarly, valid response differences among different diseases or conditions directly leading to death were used for the discriminant validity of during or post-birth constructs of the tool.
For pre-birth constructs of the questionnaire, discriminant analysis was performed using birth status as a dependent variable, which was defined in such a way that birth status was coded as “1” (one) in case of stillbirth and “0” (zero) in case of live birth, and elements of each section as discriminating or classification variables. The goodness of fit summary of discriminant analysis models, such as eigenvalue (E.V.) of canonical discriminant function, canonical correlation (C.C.), and Wilks’ Lambda (W.L.) statistics was used for the discriminant validity measures of the constructs. To analyze the discriminant validity of post-birth constructs of the VASA questionnaire, multinomial logistic regression was performed using “disease or condition directly leading to death” as dependent variables and elements of each section of the questionnaire as explanatory variables. Likelihood-Ratio (LR) test and Pseudo R-Square measures (Cox and Snell, Nagelkerke, and McFadden) were used as discriminant validity measures of the constructs.
Out of 100 cases, we had 33 stillbirths and 67 live birth cases. For the analysis, unequal prior probabilities of stillbirth and live birth were assumed and assigned prior probabilities as per the group size in the data. The deaths included in the study were both, i.e. home-based and institutional (formal and informal health care facilities). The Backward variable elimination method was used to compute the discriminant analysis model. It is vital to comment here at this point that the variables which are not included in the final model do not necessarily mean they are not important, but it means these variable(s) do not provide any additional statistical evidence for the classification of live and stillbirth.
The goodness of fit summary of discriminant analysis models of each of the pre-birth constructs of the VASA questionnaire is presented in Table 7. The eigenvalue (E.V.) of canonical discriminant function is the ratio of the between-groups sum of squares to within groups sum of squares. A high value of eigenvalue indicates that the canonical function explains sufficiently higher variation in data. Similarly, the canonical correlation (C.C.) coefficient shows the predictive ability of the canonical discriminant function. The value of C.C. ranges from 0 to 1, with a value closer to 1 showing the strong predictive ability of canonical discriminant function. The third goodness of fit statistics i.e., Wilks’ Lambda (ranging from 0 to 1), was calculated to be the proportion of the total variance in the discriminant scores not explained by differences among the groups. A small value of W.L. and statistically significant (p-value of <0.05) shows a good fit of the discriminant model.
The goodness of fit summary of multinomial logistic regression models of each of during & post-birth constructs of the VASA questionnaire is presented in Table 7. Statistically significant Likelihood-Ratio (LR) test result infers statistically significant improvement over the null model. Pseudo R-Square measures (Cox and Snell, Nagelkerke, and McFadden) range from 0 to 1, with a value closer to “1” (one) indicating a good model fit.
A high eigenvalue of canonical discriminant function (>1), a high value of canonical correlation (>0.8), and a statistically significant Wilks’ Lambda, with a p-value of <0.001, validate the good fit of discriminant models for pre-birth constructs of VASA questionnaire. A high value of Pseudo R-Square measures (>0.8) and a statistically significant LR test, with a p-value <0.001, validate the good fit of multinomial logistic regression models for during & post-birth constructs of the VASA questionnaire. Hence, underlying constructs of the VASA questionnaire, in all sections (Table 7), were found to be distinctive and well-established to address the conceptual framework. More than 85.07% accuracy of the discriminant functions and the multinomial logistic regression model was observed in all sections of the VASA questionnaire.
(d) Reliability assessment.
Reliability analysis measures the tool’s capacity to provide a consistent result. Use of Cronbach’s Alpha statistics was utilized to measure the reliability of constructs. Due to the dominantly dichotomous nature of data, the calculation of Cronbach’s Alpha based on the Pearson correlations matrix would be incorrect or potentially very biased. Therefore, “hetcor” function of “polycor” package of R was used to compute the appropriate heterogeneous correlation matrix. The ‘hetcor’ function is capable of calculating Pearson correlations (for numeric data), polyserial correlations (for numeric and ordinal data), and polychoric correlations (for ordered or non-ordered factors). Reliability for the different sections of the VASA questionnaire is presented in Table 7. The universal rule of thumb is that, if the value of Cronbach’s alpha (C.A.) is 0.70 and more, the reliability is good, if the value of C.A. is between 0.80 and 0.90 the reliability is better, and if the value is 0.90 and beyond, the reliability is best. In our study, the Cronbach’s Alpha values are above 0.90 across all the constructs of pre-birth, during & post-birth constructs of the VASA questionnaire (except “Demographic and social constructs” and “Care Seeking behaviour for symptoms of last trimester” which are 0.843 and 0.876 respectively) (Table 8).
Classification of sensitivity, specificity, and accuracy of the models developed based on the elements of different sections of the VASA questionnaire are presented in Table 8 below. 100% specificity, 100% sensitivity and 100% accuracy of the discriminant functions in classifying births to live and stillbirths were observed in all sections, except, symptoms before pregnancy and symptoms during the initial two-trimester sections of the VASA questionnaire. For which 100%, 72.73% and 91% of specificity, sensitivity, and accuracy were observed, and for these two sections lowest content validity was observed (see Table 8). Cronbach’s Alpha values were found higher than 0.80, indicating the reliability and internal consistency of the underlying constructs.
Discussion
This research validated CHERG’s integrated VASA tool using rigorous methodology and substantially all the fundamental methodologies for determining questionnaire validity and reliability. The validation exercise showed good I-CVI (81.43%), Cronbach’s Alpha reliability values between 0.843 and 0.973, 81%–100% discriminant accuracy in categorising live and stillbirths, and 97% in assigning medical and social causes of child deaths.
After this current study, this validated tool will be helpful for future researches to collect valid and reliable data pertaining to child mortality determinants operating in Pakistan. Such a tool was highly required to overcome the data shortage-related issues in the country, where the death registration is highly compromised [3, 20, 46]. When it comes to the reasons for having scanty data on death registration, one of the potential bottlenecks is the delayed and compromised notification of child deaths to the National Civil Registration and Vital Statistics Agency (NCRVS). Because the process of notification is time-consuming, there is no evidence of strict compliance among the general population in death notification [46, 47]. There are no financial benefits or assistance being provided to the community. The death registration process in the NCRVS is severely hampered, and this is true at all levels, from the local community to the national level. The existing data is also very limited and with questionable validity and accuracy [3, 20, 46], this is particularly due to no implementation of standardized methods of data collection and reporting for death notification [46, 48]. Utilization and implementation of this validated tool in identifying mortality-related determinants and in assigning medical and social causes of child deaths, especially of those deaths which have been missed from the National Civil Registration and Vital Statistics will surely help us in overcoming data-related issues pertaining to child mortalities.
Validation studies usually encounter daunting challenges. Most of them include recall biases, information biases, small sample size estimation, etc. To overcome such challenges, the methodology involved in the validation studies should be very concrete and categoric, ensuring that the responses from the validated tool should be able to give information with acceptable accuracy. It is highly essential to undertake such relevant validation studies in developing countries so that it will make the validated version of the VASA tool available for conducting VASA investigations.
Literature shows that there are several versions of VA tools developed by multiple agencies and research teams. Literature reflects several attempts at validation studies of different VA tools that have utilized different methodologies to overcome such challenges and identified the precise CoD with much accuracy [30, 31, 49, 50]. The most commonly used VA tool which underwent validation is the World Health Organization’s (WHO) VA tool [2, 27].
Having precisely calculated and appropriate sample size, the results of any validation study show the true validity and reliability of the tool. Compared to our VASA validation study, which included 100 deceased under-five cases, all the other VA validation studies undertook validation on a comparatively larger sample size and implemented only the VA component [27, 28, 32, 33]. We feel that having a smaller sample size is one of the limitations of our validation study. However, our sample size is purely based on general recommendations from Hertzog [39] (this was purely due to the reason that there are no specific guidelines for the sample size of respondents required for pre-survey validation of the VASA tool, hence this study followed the general guidelines from Hertzog guidelines), which suggests that the minimum sample size for any validation study should be 20% of the sample size calculated for the survey the tool will be utilized for. Aggarwal et al. [27] included 313 neonatal deaths; Nausheen et al. [32] undertook validation on 204 neonatal deaths; the sample size in Soofi et al. [33] was 626 neonatal deaths, and Marsha et al. [28] used 137 neonatal deaths.
Conducting interviews after a long delay subsequent to the death event may affect the quality and reliability of data. This is most likely because the respondent needs to recall the events and happenings of different events that occurred before and around the death event. It is highly recommended that VASA interviews should be undertaken somewhere between 01 months till 06 months provided the family has come out of the mourning period [51]. Conducting interviews within the mourning period may cause distress and influence the respondent’s disposition and ability to engage in a VASA interview [51]. P Serina et al. [51], suggest that a long recall period may limit the respondent’s ability in recalling and recollecting pertinent facts. Events (or symptoms) with extraordinary severity (or implications) remain in the recall for a much more extended period compared to those with mild to moderate severity [51]. Likewise all the other studies [28, 32, 33, 52], our VASA validation study undertook interviews within six (06) weeks after the mortality event. In our sample size, none of the respondents had any difficulty recalling the events.
The capacity of any questionnaire to correctly assign the proper CoD to any mortality with a degree of precision is called accuracy. When dealing with individual-level data, the diagnostic accuracy of the VASA tool is considered to be satisfactory when the specificity and sensitivity of the questionnaire are a minimum of 90% [53]. However, at a much larger, i.e. national level, if the validation attempt of the VASA tool shows sensitivity and specificity of at least 50% and 90% respectively, the tool is usually considered to be having acceptable diagnostic accuracy [53]. This study found an accuracy of between 97% and 100% in assigning medical and social causes of child deaths respectively. In comparison, the accuracy of the WHO’s VA tool validated in Aggarwals’ study [27] in identifying different neonatal causes of mortalities was found to be up to between 78% to 92% (except for birth asphyxia-16%); however, the kappa statistics was moderate from 0.46 to 0.55. Similarly, in Soofi et al. [33] the WHO VA tool showed a sensitivity of more than 83.5% for diagnosing different causes of neonatal mortality (except congenital malformation, which was 57%), while the specificity of all the major causes of neonatal deaths was found to be more than 90%. Marsha et al. [28] found the sensitivity of the process of assigning the cause of death ranging from 39% (in diagnosing infection) to 90% (in diagnosing causes related to too early/too small syndrome) and specificity ranging from 67% (in diagnosing infection) and 99% (in diagnosing Neonatal Tetanus).
Strengths of our validation study
Our study is first in its kind as it validates the CHERG’s integrated VASA tool, which is based on the most holistic conceptual framework, i.e. TPtoSCF (that addresses the barriers and limitations involved in accessing health care services), that have the capacity to record information required to develop policy in preventing under-five death incidents and improving the child survival estimates in developing countries (like Pakistan) where an unknown, but a large number of child mortality events have known to be missed and their medical causes have not been assigned.
Moreover, this study is important as it attempted a detailed validation of CHERG’s integrated VASA tool by incorporating rigorous methodology and utilizing almost all the core methods suggested for identifying the validity and reliability evidence of any questionnaire [54]. The validation exercise yielded high I-CVI (>81.43%), with Cronbach’s Alpha reliability statistics being 0.843 for “Demographic and social constructs” and highest of 0.973 for “Preventive Measures between childbirth and illness”; the accuracy of between 81% (for “Care seeking behaviour for symptoms of last trimester”) and 100% of the discriminants classifying births to live and stillbirths. The tool showed an accuracy of between 97% and 100% in assigning medical and social causes of child deaths respectively.
Although different VA questionnaires have been validated in Pakistan, however, some of them are specifically designed for the neonatal population [33], while others are for stillbirths [32]. The CHERG’s VASA integrated questionnaire encompasses three age groups and has not been validated for the Pakistani population.
An electronic CAPI (Computer Assisted Personal Interview) notebook format for data collection with skip patterns using CSPro software application has also been developed for this validated tool which was utilized for data collection during the Karachi VASA child mortality study. The primary purpose for transforming the lay paper-based form into a notebook application was to minimize the possibility of data collection and data entry errors. No data collection and data entry errors were encountered possibly due to extensive training of the field staff on the data collection and entry.
Conclusion
Based on our results, we confirm that the CHERG VASA integrated questionnaire is valid, reliable, and relevant to the conceptual frameworks for the Pakistani Population. The validated CHERG VASA tool can be utilized for establishing medical and social causes of Under-five child deaths in Karachi, Pakistan, and is one of the assets for Pakistan’s child health policy with the potential to record relevant data and assign causes of child mortality cases occurring in Pakistan. The evidence extracted from the data and information obtained from the validated VASA tool will surely complement healthcare professionals and policymakers in improving the practice and modifying the policy for enhancing the survival of under-five children in Karachi, Pakistan.
Supporting information
S1 File. VASA original English questionnaire and translated version.
https://doi.org/10.1371/journal.pone.0278149.s001
(PDF)
S2 File. VASA translated version questionnaire.
https://doi.org/10.1371/journal.pone.0278149.s002
(PDF)
Acknowledgments
We would like to acknowledge the key personnel from the relevant communities for assisting the data collection process. The authors would like to express their sincere gratitude to all the officials who provided invaluable assistance in data collection during this research. We extend our deepest appreciation to the healthcare professionals, administrators, and staff who generously shared their time, expertise, and resources with us. We would like to inform that an unreviewed preprint of this manuscript is already available on Research Square at the link (https://www.researchsquare.com/article/rs-35489/v1) and Europe PMC (https://europepmc.org/article/ppr/ppr191893) with DOI (10.21203/rs.3.rs-35489/v1).
References
- 1. Moyer CA, Johnson C, Kaselitz E, Aborigo R. Using social autopsy to understand maternal, newborn, and child mortality in low-resource settings: a systematic review of the literature. Glob Health Action [Internet]. 2017;10:1413917. Available from: https://www.tandfonline.com/doi/full/10.1080/16549716.2017.1413917 pmid:29261449
- 2. Nichols EK, Byass P, Chandramohan D, Clark SJ, Flaxman AD, Jakob R, et al. The WHO 2016 verbal autopsy instrument: An international standard suitable for automated analysis by InterVA, InSilicoVA, and Tariff 2.0. PLoS Med [Internet]. Public Library of Science; 2018;15:e1002486. Available from: pmid:29320495
- 3. Siddiqui MB, Ng CW, Low WY. Social Autopsy is a Dire Need for Investigating Child Mortality in Pakistan. Int J Endorsing Heal Sci Res [Internet]. International Journal of Endorsing Health Science Research; 2016;4:01. Available from: http://tinyurl.com/SocialAutopsyEN
- 4. Gouda HN, Flaxman AD, Brolan CE, Joshi R, Riley ID, AbouZahr C, et al. New challenges for verbal autopsy: Considering the ethical and social implications of verbal autopsy methods in routine health information systems. Soc. Sci. Med. 2017. pmid:28501755
- 5. Thomas Lisa-Marie, Lucia D’Ambruoso DB. Verbal autopsy in health policy and systems: a literature review. BMJ Glob Heal. 2018;3. pmid:29736271
- 6. Koffi AK, Mleme T, Nsona H, Banda B, Amouzou A, Kalter HD. Social autopsy of neonatal mortality suggests needed improvements in maternal and neonatal interventions in Balaka and Salima districts of Malawi. J Glob Health [Internet]. 2015;5:010416. Available from: http://www.jogh.org/documents/issue201501/jogh-05-010416.XML pmid:27698997
- 7. Koffi AK, Libite P, Moluh S, Wounang R, Kalter HD. Social autopsy study identifies determinants of neonatal mortality in Doume, Nguelemendouka and Abong–Mbang health districts, Eastern Region of Cameroon. J Glob Health [Internet]. 2015;5. Available from: http://www.jogh.org/documents/issue201501/jogh-05-010413.pdf pmid:26171142
- 8. Källander K, Counihan H, Cerveau T, Mbofana F. Barriers on the pathway to survival for children dying from treatable illnesses in Inhambane province, Mozambique. J Glob Health [Internet]. 2019;9:1–14. Available from: www.jogh.org
- 9. Kalter HD, Roubanatou AM, Koffi A, Black RE. Direct estimates of national neonatal and child cause-specific mortality proportions in Niger by expert algorithm and physician-coded analysis of verbal autopsy interviews. J Glob Health [Internet]. 2015;5:10415. Available from: http://www.jogh.org/documents/forthcoming/jogh-05-010415.pdf pmid:25969734
- 10. Kalter H, Yaroh AG, Maina A, Koffi AK, Bensaïd K, Amouzou A, et al. Verbal/social autopsy study helps explain the lack of decrease in neonatal mortality in Niger, 2007–2010. J Glob Health [Internet]. 2016;6. Available from: http://www.jogh.org/documents/issue201601/jogh-06-010604.pdf pmid:26955474
- 11. Koffi AK, Maina A, Yaroh AG, Habi O, Bensaïd , Kalter HD. Social determinants of child mortality in Niger: Results from the 2012 National Verbal and Social Autopsy Study. J Glob Health [Internet]. 2016;6:10603. Available from: http://www.jogh.org/documents/issue201601/jogh-06-010603.pdf pmid:26955473
- 12. Waiswa PK, Kallander S, Peterson G, Tomson G, Pariyo . Using the three delays model to understand why newbornbabies die in eastern Uganda. Trop Med Int Heal [Internet]. 2010;8:964–72. Available from: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-3156.2010.02557.x/pdf
- 13. Källander K, Kadobera D, Williams TN, Nielsen RT, Yevoo L, Mutebi A, et al. Social autopsy: INDEPTH Network experiences of utility, process, practices, and challenges in investigating causes and contributors to mortality. Popul Health Metr [Internet]. 2011;9. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-79961125058&partnerID=40&md5=e2dc10af4279b24950af30811791371b pmid:21819604
- 14. Hildenwall H, Tomson G, Kaija J, Pariyo G, Peterson S. “I never had the money for blood testing”—Caretakers’ experiences of care-seeking for fatal childhood fevers in rural Uganda—A mixed methods study. BMC Int Health Hum Rights [Internet]. 2008;8. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-58149335516&partnerID=40&md5=2fbff59e64ae076eb06f7b6684d4800e pmid:19055738
- 15. Källander K, Hildenwall H, Waiswa P, Galiwango E, Peterson S, Pariyo G. Delayed care seeking for fatal pneumonia in children aged under five years in Uganda: a case-series study. Bull World Health Organ. 2008;86:332–8.
- 16. Navale S, Habumugisha L, Amoroso C, Sayinzoga F, Gupta N, Hirschhorn LR. Exploring Drivers of Infant Deaths in Rural Rwanda Through Verbal Social Autopsy. Ann Glob Heal. 2017;83:756–66. pmid:29248092
- 17. Kagabo DM, Kirk CM, Bakundukize B, Hedt-Gauthier BL, Gupta N, Hirschhorn LR, et al. Care-seeking patterns among families that experienced under-five child mortality in rural Rwanda. PLoS One. 2018;13. pmid:29320556
- 18. Koffi AK, Kalter HD, Loveth EN, Quinley J, Monehin J, Black RE. Beyond causes of death: The social determinants of mortality among children aged 1–59 months in Nigeria from 2009 to 2013. PLoS One [Internet]. Public Library of Science; 2017;12:e0177025. Available from: pmid:28562610
- 19.
Schumacher R, Diallo MO, Keita DR, Kalter HD, Pasha O: SE. Mortality study in Guinae: Investigating the causes of death for children under 5 The Basic Support for Institutionalizing Child Survival Project (BASICS II). Arlington, VA; 2002. 2002; http://www.basics.org/documents/pdf/guinea_mort.pdf
- 20. Siddiqui MB, Ng CW, Yun Low W. Methodologies of Verbal autopsy and Social autopsy tools, adopted by VASA studies in exploring under-five mortality determinants. Int J Endorsing Heal Sci Res. 2018;6:43.
- 21. Siddiqui MB, Ng CW, Low WY. To Identify the Non-biological Causes of Child Mortalities in Developing Countries, Social Autopsy Tools should be Based on “The Pathway to Survival Conceptual Framework”. Int J Endorsing Heal Sci Res [Internet]. Karachi, Pakistan; 2017;5:01. Available from: http://aeirc-edu.com/wp-content/uploads/Download-full-Paper-B.pdf
- 22.
WHO, Department of Measurement and Health Information Systems evidence and information for policy. WHO Technical Consultation on Verbal Autopsy Tools: Review of the literature and currently-used verbal autopsy tools. 2004;1–45. http://www.who.int/healthinfo/statistics/mort_verbalautopsy.pdf
- 23.
CHERG. Projects: CHERG Verbal and Social Autopsy Survey Tools and Information [Internet]. CHERG; 2018 [cited 2021 Jul 18]. http://cherg.org/projects/pathways.html#socialautopsy
- 24. Kalter HD, Salgado R, Babille M, Koffi AK, Black RE. Social autopsy for maternal and child deaths: A comprehensive literature review to examine the concept and the development of the method. Popul Health Metr [Internet]. 2011;9. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-79961122036&partnerID=40&md5=54293a56a08e344174924aa944cb2512 pmid:21819605
- 25.
Koffi AK, Perin J, Kalter HD, Monehin J, Adewemimo A, Black RE. How fast did newborns die in Nigeria from 2009–2013: A time-to-death analysis using Verbal /Social Autopsy data. J. Glob. Health. 2019.
- 26. Bensaïd K, Yaroh AG, Kalter HD, Koffi AK, Amouzou A, Maina A, et al. Verbal/Social Autopsy in Niger 2012–2013: A new tool for a better understanding of the neonatal and child mortality situation. J Glob Health [Internet]. 2016;6:10602. Available from: http://www.jogh.org/documents/issue201601/jogh-06-010602.pdf pmid:26955472
- 27. Aggarwal AK, Kumar P, Pandit S, Kumar R. Accuracy of WHO Verbal Autopsy Tool in Determining Major Causes of Neonatal Deaths in India. PLoS One [Internet]. 2013;8. Available from: pmid:23372783
- 28. Marsha DR, Sadruddin S, Fikree FF, Krishnan C, Darmstadt GL. Validation of verbal autopsy to determine the cause of 137 neonatal deaths in Karachi, Pakistan. Paediatr Perinat Epidemiol [Internet]. 2003;17:132–42. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-0346997048&partnerID=40&md5=be143424f4660df2ca7611dfb8170011 pmid:12675779
- 29. Midhet F. Validating the verbal autopsy questionnaire for maternal mortality in pakistan. Int J Health Sci (Qassim) [Internet]. 2008 [cited 2018 Nov 14];2:91–6. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3068724&tool=pmcentrez&rendertype=abstract pmid:21475477
- 30. Mpimbaza A, Filler S, Katureebe A, Quick L, Chandramohan D, Staedke SG. Verbal autopsy: Evaluation of methods to certify causes of death in Uganda. PLoS One [Internet]. 2015;10:e0128801. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26086600 pmid:26086600
- 31. Murray CJL, Lozano R, Flaxman AD, Serina P, Phillips D, Stewart A, et al. Using verbal autopsy to measure causes of death: The comparative performance of existing methods. BMC Med [Internet]. 2014;12:5. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24405531 pmid:24405531
- 32. Nausheen S, Soofi SB, Sadiq K, Habib A, Turab A, Memon Z, et al. Validation of Verbal Autopsy Tool for Ascertaining the Causes of Stillbirth. PLoS One [Internet]. 2013;8:e76933. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24130814 pmid:24130814
- 33. Soofi SB, Ariff S, Khan U, Turab A, Khan GN, Habib A, et al. Diagnostic accuracy of WHO verbal autopsy tool for ascertaining causes of neonatal deaths in the urban setting of Pakistan: A hospital-based prospective study. BMC Pediatr [Internet]. 2015;15:1–9. Available from: pmid:26438252
- 34.
IGME U. Levels & Trend in Child Mortality: Estimates Developed by the Estimates developed by the UN Inter-agency Group for UN Inter-agency Group for Child Mortality Estimation Child Mortality Estimation. UNICEF; 2020.
- 35. Afzal U, Yusuf A. The State of Health in Pakistan: An Overview. Lahore J Econ. 2013;18:233–47.
- 36. Nisar Y Bin Dibley MJ. Determinants of neonatal mortality in Pakistan: Secondary analysis of Pakistan Demographic and Health Survey 2006–07. BMC Public Health [Internet]. 2014;14. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-84902916386&partnerID=40&md5=2a323c97d87a9ed046088bf1b54e3f1a pmid:24972633
- 37. Siddiqui MB, Ng CW, Low WY, Syed S, Ahmed S, Noushad S, et al. A Verbal/Social Autopsy (VASA) Child Mortality Inquiry to Investigate Under-Five Mortality Determinants in Slums of Karachi, Pakistan:A Mix Methods Interventional Study. Int J Endorsing Heal Sci Res. 2016;4:01.
- 38. Anthoine E, Moret L, Regnault A, Sbille V, Hardouin JB. Sample size used to validate a scale: A review of publications on newly-developed patient reported outcomes measures. Health Qual Life Outcomes [Internet]. London: BioMed Central; 2014;12:2. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4275948/
- 39. Hertzog MA. Considerations in determining sample size for pilot studies. Res Nurs Heal. 2008/01/10. 2008;31:180–91. pmid:18183564
- 40.
UN-Habitat. Slums of the World: the face of urban poverty in the new millennium. UN-Habitat, Nairobi. 2003.
- 41.
Bennet AE, Ritchie K. Questionnaires in medicine A guide to their design and use [Internet]. London New York; 1975. http://trove.nla.gov.au/work/11045163?selectedversion=NBD793226
- 42. Siddiqui MB, Ng CW, Low WY, Noushad S, Ahmed S, Syed S. Challenges and acceptance of the use of computer-assisted personal interviewtechnology for verbal autopsy/social autopsy child mortality survey in urban slums of Karachi, Pakistan. Int J Endorsing Heal Sci Res [Internet]. 2014;6:30. Available from: http://aeirc-edu.com/ojs14/index.php/IJEHSR/article/view/296/478
- 43.
WHO/UNICEF. WHO/UNICEF Joint Monitoring Programme for water supply and sanitation: Sanitation tables [Internet]. Who/Unicef. 2015. http://www.wssinfo.org/data-estimates/tables/
- 44. Lebet RM, Asaro LA, Zuppa AF, Curley MAQ. Face and content validity of variables associated with the difficult-to-sedate child in the paediatric intensive care unit: A survey of paediatric critical care clinicians. Aust Crit Care. 2018;31:167–73.
- 45. Trockel M, Bohman B, Lesure E, Hamidi MS, Welle D, Roberts L, et al. A Brief Instrument to Assess Both Burnout and Professional Fulfillment in Physicians: Reliability and Validity, Including Correlation with Self-Reported Medical Errors, in a Sample of Resident and Practicing Physicians. Acad Psychiatry. 2018;42:11–24. pmid:29196982
- 46. Abouzahr C, Azimi SY, Bersales LGS, Chandramouli C, Hufana L, Khan K, et al. Strengthening civil registration and vital statistics in the Asia-Pacific region: Learning from country experiences. Asia-Pacific Popul J. 2014;29:39–73.
- 47.
Asad Mahmood Alvi B. National Registration System in Pakistan Director General of Registration Government of Pakistan [Internet]. 1993. https://unstats.un.org/unsd/demographic/meetings/wshops/1993_China_CRVS/docs/1993_Doc.11_Pakistan.pdf
- 48. Zakar MZ, Zakar R, Mustafa M, Jalil A, Fischer F. Underreporting of stillbirths in Pakistan: Perspectives of the parents, community and healthcare providers. BMC Pregnancy Childbirth [Internet]. 2018;18. Available from: https://bmcpregnancychildbirth.biomedcentral.com/articles/10.1186/s12884-018-1924-9 pmid:30012104
- 49. Kahn K, Tollman SM, Garenne M, Gear JSS. Who dies from what? Determining cause of death in South Africa’s rural north-east. Trop Med Int Heal. 1999;4:433–41. pmid:10444319
- 50. Rodriguez L, Reyes H, Tome P, Ridaura C, Flores S, Guiscafre H. Validation of the Verbal Autopsy Method to Ascertain Acute Respiratory Infection as Cause of Death. Indian J Pediatr. 1998;65:579–84. pmid:10773908
- 51. Serina P, Riley I, Hernandez B, Flaxman AD, Praveen D, Tallo V, et al. What is the optimal recall period for verbal autopsies? Validation study based on repeat interviews in three populations. Popul Health Metr [Internet]. 2016;14. Available from: https://pophealthmetrics.biomedcentral.com/articles/10.1186/s12963-016-0105-1 pmid:27833459
- 52. Dikid T, Gupta M, Kaur M, Goel S, Aggarwal AK, Caravotta J. Maternal and Perinatal Death Inquiry and Response project implementation review in India. J Obstet Gynecol India [Internet]. 2013;63:101–7. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-84879506441&partnerID=40&md5=f25652099104439c504096bbc6f93d81 pmid:24431614
- 53. Soleman N, Chandramohan D, Shibuya K. Verbal autopsy: current practices and challenges. BullWorld Heal Organ [Internet]. 2006;84:239–45. Available from: http://www.scopus.com/inward/record.url?eid=2-s2.0-33644795635&partnerID=40&md5=09ffcd11e7668e66416b1919445bb9d2 pmid:16583084
- 54. Taherdoost H. Validity and Reliability of the Research Instrument; How to Test the Validation of a Questionnaire/Survey in a Research. SSRN Electron J. 2018;5:28–36.