The WHO Maternal Near-Miss Approach and the Maternal Severity Index Model (MSI): Tools for Assessing the Management of Severe Maternal Morbidity

Objectives To validate the WHO maternal near-miss criteria and develop a benchmark tool for severe maternal morbidity assessments. Methods In a multicenter cross-sectional study implemented in 27 referral maternity hospitals in Brazil, a one-year prospective surveillance on severe maternal morbidity and data collection was carried out. Diagnostic accuracy tests were used to assess the validity of the WHO maternal near-miss criteria. Binary logistic regression was used to model the death probability among women with severe maternal complications and benchmark the management of severe maternal morbidity. Results Of the 82,388 women having deliveries in the participating health facilities, 9,555 women presented pregnancy-related complications, including 140 maternal deaths and 770 maternal near misses. The WHO maternal near-miss criteria were found to be accurate and highly associated with maternal deaths (Positive likelihood ratio 106.8 (95% CI 99.56–114.6)). The maternal severity index (MSI) model was developed and found to able to describe the relationship between life-threatening conditions and mortality (Area under the ROC curve: 0.951 (95% CI 0.909–0.993)). Conclusion The identification of maternal near-miss cases using the WHO list of pregnancy-related life-threatening conditions was validated. The MSI model can be used as a tool for benchmarking the performance of health services managing women with severe maternal complications and provide case-mix adjustment.


Introduction
An estimated 287 000 maternal deaths occurred in 2010 around the world. Despite the substantial reduction as compared to 1990, much has to be done for achieving the relevant target of the Millennium Development Goals [1]. Most of the burden of maternal deaths is carried by low-income countries, but maternal mortality is still a relevant public health problem among middleincome countries. In this context, strengthening health systems and services to provide optimal care for women during pregnancy and childbirth is crucial, particularly to those women experiencing acute pregnancy-related complications [2][3][4][5].
Confidential enquiries of maternal deaths have been used for many years to understand health systems and services failures in the provision of appropriate maternal health care. Based on these enquiries, lessons can be learned and used to strengthen health systems and improve quality of care [6]. Despite the positive contribution of this approach, it has limitations, particularly in low mortality settings or at the health service level, where the amount of maternal deaths is generally insufficient to provide useful information. In the last 20 years, the concept of maternal near miss has been explored in maternal health as an adjunct to maternal-death confidential enquiries. Women who nearly died but survived complications have been studied as surrogates of maternal deaths. Among other positive characteristics, maternal near-miss cases can directly inform on problems and obstacles that had to be overcome during the process of health care. Maternal near-miss audits have been considered as useful approaches to improve maternal health care [7,8].
In 2008, the World Health Organization (WHO) has adopted a maternal near-miss definition and established standard criteria for the identification of women presenting pregnancy-related lifethreatening conditions. Women surviving any of the life-threatening conditions listed in the Table 1 during pregnancy, childbirth, or postpartum are considered as maternal near-miss cases by WHO. The WHO definition enables a common ground for the implementation of maternal near-miss assessments across countries and allows international comparisons to be carried out [9].

Validation of the Set of the WHO Criteria for Identifying Women with Life-threatening Conditions
The WHO maternal near-miss definition and identification criteria have been developed using the collective wisdom of a group of international experts gathered by WHO in an evidenceinformed process [9]. A systematic review summarizing the range of previous related experiences provided the scientific basis for the development of the WHO definition and identification criteria [10]. Intensive-care prognostic and severity assessment scores, such as the APACHE II, SAPS, MODS and particularly the SOFA score, were reviewed as possible sources of severity markers for identifying women with life-threatening conditions [11][12][13]. A secondary analysis of a previous multicountry study was conducted to explore pragmatic identification criteria for maternal near misses [14]. In addition, a small preliminary study was carried for piloting and pre-validating the criteria proposed by the WHO working group [15].
In spite of the evidence-informed process that led the development of the WHO criteria for identifying women with life-threatening conditions, an actual validation of such criteria is required. The validation of such criteria depends on the similarity between maternal deaths and maternal near-miss cases. From the conceptual standpoint, near-miss cases should be as similar to maternal deaths as possible. However, the development of criteria to identify near-miss cases is challenged by the absence of a goldstandard for near-miss cases. In addition, the identification of near-miss cases is always retrospective, i.e. the woman needs to survive the life-threatening complication in order to be considered as a near-miss case. In this context and considering a set of criteria as a diagnostic test, it is assumed that a set of criteria able to accurately ''identify'' maternal deaths would have as false positives the maternal near-miss cases. Truly-positive cases (maternal deaths) would be similar to false positive cases (maternal nearmiss cases) except the vital status [14].

Development of a Benchmarking Tool
Populations of critically ill patients may differ in their mortality risks, depending on the severity of individual cases, the case-mix, and the quality of the therapeutic management, among other factors. Prognostic/risk-scoring systems have been used in the assessment of medical and surgical critically ill patients. Most of these systems are based on mathematical models and aim at determining the position of individual patients in the spectrum of severity. By providing a judgment-free assessment of individual risks, these tools enable neutral case-mix adjustments for groups of patients. Some of these tools also enable an assessment of health service performance in the provision of care, as they allow an estimation of expected mortality considering the specific population case-mix. By comparing the observed mortality to the expected mortality in the population, a sense of appropriateness of care can be made [11,12,16,17]. This is a benchmark approach, which uses the population that generated the model as the standard for comparison [17]. Some attempts to use these generic systems in populations of severely-ill obstetric populations have been made, but the accuracy of these systems has been considered as suboptimal, possibly due to the mismatch between the reference population (general, non-obstetric) and the target population (obstetric) [18][19][20][21][22]. The WHO criteria are a package of 25 severity markers portraying a comprehensive range of life-threatening conditions. Each of these severity markers is associated with a specific relationship with mortality. In addition, women experiencing pregnancy-related complications can present more than one of these severity markers. Thus, combinations of severity markers would be associated with different mortality risks. For instance, a woman that presents postpartum hemorrhage and undergoes a hysterectomy due to uterine atony has a different risk of death as compared to a woman presenting several markers of severity denoting multiple organ dysfunctions. Similarly, depending on the case-mix, a population of women presenting life-threatening conditions may differ in its relationship with mortality in comparison to other populations of women presenting lifethreatening conditions. Thus, a benchmark tool able to minimize severity bias by providing case-mix adjustment and enable comparisons to a reference population would improve the applicability of the maternal near-miss concept and enable more appropriate comparisons between populations.

Objectives
The present analysis aims at validating the WHO criteria for identification of women with pregnancy-related life-threatening conditions and developing a benchmark tool for severe maternal morbidity assessments.

Ethics Statement
This project has been reviewed and approved by the National Council for Ethics in Research (CONEP, Brazilian Ministry of Health) and by the Institutional Review Boards of each site (listed below). All data was obtained from medical records and did not identify participants. The National Council for Ethics in Research and the Institutional Review Boards of each site granted a waiver of individual informed consent. The

Study Design
Methodological details of the Brazilian Network for Surveillance of Severe Maternal Morbidity Study have been published elsewhere [23,24]. Briefly, this is a multicenter, cross-sectional study implemented in 27 Brazilian referral maternity hospitals. A convenience sampling strategy was used to build the network of health facilities. Aiming at lessening the impact of the non-random sampling and optimizing the representativeness of Brazilian referral maternity hospitals, an adequate mix of health facilities was sought: public and private health facilities, university and nonuniversity hospitals, at least one health facility from each of the five country macro-regions. The number of health facilities was determined based on the number of maternal deaths and maternal-near miss cases required to validate the use of the WHO criteria for pregnancy-related life-threatening conditions and carry out other relevant analysis. Based on previous studies [10,25], a total of approximately 75,000 deliveries would have to be screened in order to identify around 100 maternal deaths and 600 maternal near miss cases.
In each hospital, a research team was responsible for carrying out prospective surveillance on severe maternal morbidity. During the one-year data collection period (between July 2009 and June 2010), all women admitted to the participating health facilities were screened against the inclusion criteria by research assistants. The inclusion criteria were: presence of any of the conditions listed in the Table 1 (i.e. potentially life-threatening conditions, WHO life-threatening conditions), maternal deaths and referral to another health facility due to severe ill health. The prospective surveillance was implemented by research assistants through daily visits to obstetric wards and other relevant facilities (e.g. intensive care units and emergency rooms). During the daily visit the attending staff was contacted and the medical charts of hospitalized women were screened for the study inclusion criteria. Medical records of eligible cases were retrieved for thorough review at hospital discharge, transfer to another hospital or death. Data was collected into a previously coded form and included demographic, medical and obstetric characteristics, primary determinants of lifethreatening conditions (the first complication in the chain of events), duration of hospitalization (prior to delivery, following delivery and total time), the occurrence of life-threatening conditions, maternal and perinatal outcomes and information related to the occurrence of delays in the provision of care. All severity markers (i.e. WHO life threatening conditions) present in each case were recorded. Institutional capacity was assessed using an adapted version of the hospital complexity index developed for the WHO Global Survey project [26]. The hospital complexity index was used to determine the range of services available in each of the facilities and to summarize an institution's capacity to provide obstetric care. This index comprised eight categories reflecting the: standard of building/basic services, maternal intrapartum care and human resources; availability of general medical care, anaesthesiology, emergency obstetric services; and provision of screening tests and academic resources and clinical protocols. The original hospital complexity index was used by WHO in Latin America and Asia and adapted for use in Africa. We implemented minor adaptations to reflect the Brazilian context. An open-access, web-based, good-clinical-practice com-pliant database solution and study management system was used (OpenClinica, Akaza Research, LLC, 2009, Waltham, MA, USA, www.openclinica.org).

Data Quality
Several procedures were adopted to ensure high quality data and reliable information, including preparatory meetings, site visits, close monitoring of data collection and data entry, concurrent query management, inconsistency checks, double data collection for selected medical charts, and use of a detailed manual of operation. In the preparatory meetings, study coordinators and data collectors from each site were trained in the study procedures. The study protocol, manual of operation and study forms were thoroughly reviewed; training was provided on the use of the webbased data management system. During data collection the participating hospitals received continuous support including site visits. In the site visits, the study implementation was assessed and selected medical records were checked against data entered in the data management system. In addition, the web-based data management system used in this study is compliant with Good Clinical Practice (GCP) and regulatory guidelines (e.g. 21 CFR Part 11), allowing differentiated user roles and privileges, password and user authentication security, electronic signatures, SSL encryption, de-identification of Protected Health Information (PHI). Auditing to record and monitor access and data changes aligned with a set of validation and cross checking rules were implemented as part of the online data-management. Through this comprehensive package of data quality procedures reliable and high quality data were obtained.

Statistical Analysis
The Maternal Mortality Ratio (MMR) with 95% Confidence Intervals (CI) was determined based on the total of live births that took place in the participating hospitals during the data collection period. The study maternal mortality data was compared with the WHO maternal mortality estimates for 2010 in Brazil (together with its range of uncertainty) [1]. Sensitivity, specificity, positive and negative likelihood ratios were used to determine the accuracy of the WHO criteria in the identification of women with lifethreatening conditions. Vital status at discharge was considered as the gold standard for identifying women with life-threatening conditions. The prevalence of women with life-threatening complications, maternal deaths and maternal near-miss cases was calculated. In addition, the prevalence and mortality estimates together with crude relative risks and 95% CI were calculated for each severity marker.
It was hypothesized that the number of severity markers present in each case would be correlated with mortality. The study population was categorized according with the total number of severity markers per case. In each category, the frequency of cases and mortality with 95% confidence interval was determined. The correlation between number of markers per case and mortality was determined by calculating the Pearson's correlation coefficient. In each case, the total number of severity markers is hereby defined as the maternal severity score.
Two binary logistic regressions models were developed and tested to describe the relationship between (severe) morbidity and mortality. In order to enable this, the study population was split in two: the subpopulation ''A'' (used for model development), and the subpopulation ''B'' (used for model testing) [27]. The size of the subpopulation ''B'' was determined considering that the area under the Receiver Operating Characteristic curve (AUROC) would be used as an estimator of model validity. The following parameters were used in the subpopulation ''B'' sample size calculation: 0.05 as the probability of making a Type I error, 0.20 as the probability of making a Type II error, AUROC 0.500 as null hypothesis value, and 0.800 as the minimum expected AUROC. Based on these parameters, a total of 28 maternal deaths would be required in the subpopulation ''B'' for the comparison of the AUROC with the null hypothesis value. Considering the total number of maternal deaths included in the study database, 80% of the study population was randomly allocated to the population ''A'' and the remainder (20%) was allocated to the population ''B''.
The model I consisted in a univariate analysis including only the maternal severity score (i.e. the number of WHO severity markers). The model II tested the maternal severity score, distal predictors of maternal deaths (e.g. obstetric and demographic variables, direct and indirect causes of maternal deaths) and lifethreatening conditions categorized as presented in the Table 1. In the multivariate analysis (model 2), a stepwise approach was used and only predictors that contributed significantly to the model were retained (the probabilities for stepwise were set as 0.05 (entry) and 0.10 (removal)). These logistic regression models estimate the probability of maternal death based on the presence or absence of severity markers and other relevant characteristics [27]. The Hosmer-Lemeshow goodness-of-fit test was used to assess the ability of these models in adequately describing the data. The Nagelkerke R square test was used to estimate the proportion of variance in maternal mortality associated with the models' predictors. The area under the ROC curve was determined for the Maternal Severity score and the models I and II. The best performing model was selected to generate the Maternal Severity Index (MSI), hereby defined as the estimated probability of maternal death.

Results
A total of 82,388 women were admitted to the 27 health facilities during the one-year data collection period. These women gave birth to 82,144 born-alive infants. The study population comprised 9,555 women presenting pregnancy related complications and meeting the study inclusion criteria. Of this population with pregnancy-related complications, 910 women presented at least one of the severity markers classified as life-threatening conditions by WHO, including 140 maternal deaths and 770 survivors. The MMR in the screened population was was 170 maternal deaths per 100,000 live births (95% CI 144-201 maternal deaths per 100,000 live births). The maternal deaths included in this study represent about 8% of all maternal deaths that are estimated to occur in Brazil in 2010 (range of uncertainty: 5-13%) [1]. Table 2 shows the performance of the WHO set of criteria in two populations: all women giving birth during the datacollection period and those presenting pregnancy-related complications. All women that died presented at least one of the listed life-threatening conditions. Table 3 presents the relationship between markers of severity (WHO criteria) and maternal deaths. The prevalence of each of these life-threatening conditions ranged from 0.19 to 3.55 cases per 1000 deliveries and the condition specific mortality ranged between 12.9% and 85.0%. All lifethreatening conditions were highly associated with maternal deaths, but heterogeneity was observed: 5 life-threatening conditions presented relative risks between 10 and 20, 14 presented relative risks ranging from 20 to 60, and 6 presented relative risks over 60. The amount of severity markers per case (the maternal severity score) presented a very high positive correlation with mortality as illustrated in the Figure 1 and Table 4.
The logistic regression models were developed in the subpopulation ''A'', which included a total of 7,674 randomly selected cases (111 maternal deaths). The models were tested in the subpopulation ''B'', which included a total of 1881 cases (29 maternal deaths). The size of each subpopulation complied with the predetermined study sample size requirements. Table 5 summarizes the performance of the models and the maternal severity score. The model II presented a better death prediction performance as evaluated by the Hosmer-Lemeshow test, Nagelkerke R 2 test, and the percentage of maternal deaths with a modelestimated death probability greater than 50%. Based on these findings, the model II was selected to become the MSI model. Table 6 presents the covariates that were retained in the model II (i.e. the MSI model) together with the coefficients that correspond to each covariate. The death probability can be obtained using the formula presented in the Table 5. A downloadable calculator has been developed to facilitate the use of the MSI (Appendix S1).

Discussion
The WHO criteria for pregnancy-related life-threatening conditions were found to be highly associated with maternal deaths. Survivors of the WHO pregnancy-related life-threatening conditions can be accurately classified as maternal near-miss cases. A severity score representing the total number of life-threatening conditions present in each case and a mathematical model describing the relationship between severity markers and maternal deaths have been developed.
Scoring systems have been used to evaluate severity and outcome of critically ill patients since many years. APACHE, SAPS, and SOFA systems are among the most used ones [11]. Some of these systems are extensively used for intensive care benchmarking and quality of care assessment [16,17]. Notwithstanding, these systems have been developed based on general critical care populations from developed countries. In these reference populations, obstetric patients were largely omitted or underrepresented, either because pregnant women were excluded, or because maternal deaths are rare events in the countries where these systems were developed. Another issue that should be acknowledged is that the physiological changes of pregnancy affect some of the markers used by general severity scoring systems leading to overestimation of severity [18]. Also, diseases that are exclusive to this period of life (e.g. eclampsia, HELLP syndrome, acute fatty liver of pregnancy, amniotic fluid embolism), have peculiar characteristics that may not have been adequately addressed by scores designed for general populations. Owing to this, the performance of these systems in obstetric population is challenged, particularly in developing countries. Limitations are observed in the discriminatory power and the calibration of generic scoring systems when applied to obstetric population [12,15,[19][20][21][22].
The maternal severity score and the MSI model, developed in this study, may contribute for a better assessment of severity of obstetric populations and enable a benchmark approach to quality of care of women experiencing severe complications related with pregnancy. As part of the strengths of this analysis, the MSI model was developed in a large multicenter study, which had an appropriate sample both in terms of number of critically ill obstetric patients and maternal deaths. The study population was large enough to allow adequate model development and testing in different subsets of the study population as methodologically recommended [27]. In the end, the MSI model was found to be robust, presenting good performance and discriminatory power. There are also two additional potential advantages that should be noted: the MSI model was developed based on the WHO criteria for pregnancy-related life-threatening conditions and the study database reflects the standard of care provided in obstetric referral hospitals from a developing country.
The WHO criteria for pregnancy-related life-threatening conditions are part of a strategy promoted by WHO for assessing and improving quality of maternal health care [8,9,28]. These criteria are used in the identification of maternal near-miss cases in clinical audits and other near-miss studies. Together with routine implementation, there are several near-miss research projects being currently conducted around the world using these criteria. In addition to external validation, these initiatives may favor further dissemination of the maternal near miss concept and enable the use of the MSI model as a benchmark tool.
As in other severity models, the MSI model reflects the characteristics and standards of care received by the population Table 2. Accuracy of the WHO set of severity markers in the prediction of maternal deaths among all women and women with pregnancy-related complications *.  that provided data for its development. Brazil is the largest country in Latin America and the world's fifth largest country (both by geographical area and population). It is an upper-middle income country which overall MMR is estimated as 56 maternal deaths per 100,000 live births in 2010 by WHO [1]. This study includes a substantial proportion of all maternal deaths that are estimated to have occurred in the country during the data-collection period (8%, range of uncertainty 5-13%). The fact that the MSI model has been developed in an obstetric population is relevant for the applicability of this tool in other populations. This is particularly important in the context of other scoring systems (e.g. the APACHE family) that were developed in generic populations of developed countries. We believe that the application of the MSI to obstetric populations of other developing countries is more direct as compared to extrapolating results from non-obstetric populations from developed countries. However, it should be noted that the standard of care provided by the participating facilities is being used as the reference for the MSI estimates. One would expect that there may be still some limitations and constraints in the quality of care provided by the participating facilities. Thus, the underlying aim of using the MSI as a benchmark is to assess the health service performance against a standard and, through interventions to improve quality of care, achieve a superior performance. This study has some limitations that are worth noting. First, the non-random nature of the facility sampling process may have introduced some level of selection bias, potentially impairing the country representativeness of this study. On the other hand, the convenience sampling approach was realistic and made this study feasible. Precautions have been taken to maximize country representativeness. An analysis based on the intra-cluster correlation coefficients provided some evidence supporting the success of these precaution measures [29]. Second, this study is largely based on information obtained from medical records. In order to reduce the chances of recording bias, information from medical records was complemented with information obtained directly from the assisting staff (if relevant information was missing and in case of doubt). In addition, several procedures to optimize quality of data have been put in place. Third, the study population is essentially provided by referral hospitals which tend to concentrate the more severe cases: the MMR observed in this study is about three times the overall MMR estimated for the country. Another aspect that deserves noting is the relationship between the various covariates within the MSI model. The maternal severity score (i.e. the total number of severity markers present in each case) is positively correlated with maternal mortality and as the number of lifethreatening conditions increase, the death probability increases. If a life-threatening condition is identified at hospital arrival or within the first 24 hours of hospital stay, there is an increase in the risk of death, possibly denoting the fact that the woman has arrived in the hospital already in a very severe condition. Cancer and a cardiovascular or respiratory failure substantially increase the death risk. Two covariates (i.e. severe pre-eclampsia and hysterectomy) have negative coefficients denoting a ''protective'' association within the model. At the first glance this may seem counterintuitive, but these negative coefficients have to be considered in the context of severe maternal morbidity. Our  interpretation to the negative coefficient associated to preeclampsia is that women presenting a severe health condition due to pre-eclampsia have the potential of a better outcome as compared to women in the same level of severity having other complications. This can be due to the fact that severe preeclampsia tend to be a transient complication and effective strategies to manage women with pre-eclampsia exist (e.g. fetal delivery, magnesium sulfate and anti-hypertensive drugs) and may have been used in the population that provided data to this model, resulting in reduced death risks. Similarly, hysterectomy plays a game-changing role in the outcome of women with uterine-related haemorrhage and infection. There are several potential applications to both Maternal Severity Score and MSI. A primary application is determining the level of complexity and severity of a certain obstetric population. For example, a district hospital treating a population with an average maternal severity score = 0.5 is expected to require much more material and human resources than another district hospital treating a population with a maternal severity score = 0.1. These two hypothetical district hospitals receive two different case-mix and the maternal severity score can be used to put the health service in context and support decision making for resource allocation. Another primary application is the health impact evaluation, as part of quality of care assessment. The average MSI can provide an estimation of the expected number of maternal deaths for a selected population. For example, a hypothetical obstetric population being treated in an intensive care unit or in a high dependency facility in a tertiary hospital has an average MSI of 10%. It means that in a group of 100 women treated in this facility it would be expected the occurrence of 10 maternal deaths. If 20 maternal deaths have taken place in this population, one could conclude that there may be some opportunities being missed in this facility and a strategy to improve care is needed. The MSI allows also inter-hospital and over-time comparisons. Another possible application is in research. In a randomized controlled trial, for instance, it is worthwhile determining if both trial arms  Table 4. Relationship between the number of severity markers (maternal severity score) and mortality*. are comparable in terms of severity. Severity can functions as a major confounder: trial results can differ because of unbalances in the severity of the study populations (e.g. populations containing more severe cases tend to present worse health outcomes in comparison to populations with less severe cases). The maternal severity score and the MSI can be used for adjusting for the casemix, through stratification or as a covariate in statistical modeling. In summary, the maternal severity score provide an estimation of the overall severity associated with a specific women or a selected population. Similarly, the MSI provides an estimate of the death risk. To maximize usability, the Appendix S1 contains a maternal severity score and maternal severity index calculator. As a final remark, the estimates derived from the MSI model are to be used with caution. A MSI with 95% of death risk means that among 100 women with similar conditions, 95 women may die. However, the model is not able to differentiate if the specific woman is among the 5 that will survive. Thus, MSI estimates should not directly guide the management of critically ill patients.

Conclusions
The identification of maternal near-miss cases using the WHO list of pregnancy-related life-threatening conditions is valid, as these conditions are accurately associated with maternal deaths. The MSI model adequately describes the relationship between severity markers and maternal deaths. The MSI model can be used as a tool for benchmarking, population severity assessment and case-mix adjustment. The use of the MSI model within a maternal near-miss approach has the potential of contributing to the assessment and improvement of maternal health care, particularly that required by women experiencing severe maternal morbidity. Further studies assessing the performance of the MSI model in other populations are welcome.