Monitoring quality of obstetric care from hospital discharge databases: A Delphi survey to propose a new set of indicators based on maternal health outcomes

Objectives Most indicators proposed for assessing quality of care in obstetrics are process indicators and do not directly measure health effects, and cannot always be identified from routinely available databases. Our objective was to propose a set of indicators to assess the quality of hospital obstetric care from maternal morbidity outcomes identifiable in permanent hospital discharge databases. Methods Various maternal morbidity outcomes potentially reflecting quality of obstetric care were first selected from a systematic literature review. Then a three-round Delphi consensus survey was conducted online from 11/2016 through 02/2017 among a French panel of 37 expert obstetricians, anesthetists-critical-care specialists, midwives, quality-of-care researchers, and user representatives. For a given maternal outcome, several definitions could be proposed and the indicator (i.e. corresponding rate) could be applied to all women or restricted to specific subgroup(s). Results Of the 49 experts invited to participate, 37 agreed. The response rate was 92% in the second round and 97% in the third. Finally, a set of 13 indicators was selected to assess the quality of hospital obstetric care: rates of uterine rupture, postpartum hemorrhage, transfusion incident, severe perineal lacerations, episiotomy, cesarean, cesarean under general anesthesia, post-cesarean site infection, anesthesia-related complications, postpartum pulmonary embolism, maternal readmission and maternal mortality. Six were considered in specific subgroups, with, for example, the postpartum hemorrhage rate assessed among all women and also among women at low risk of PPH. Implications This Delphi process enabled us to define consensually a set of indicators to assess the quality of hospital obstetrics care from routine hospital data, based on maternal morbidity outcomes. Considering 6 of them in specific subgroups of women is especially interesting. These indicators, identifiable through codes used in international classifications, will be useful to monitor quality of care over time and across settings.


Introduction
For several years, safety at birth and the quality of care in the perinatal period have been a topic of concern to public health officials, care providers, and patient groups around the world. Accordingly, several sets of indicators assessing quality of care in obstetrics have been proposed but no consensus has emerged around any of them [1][2][3][4][5][6][7][8][9][10][11]. They have three principal limitations. First, many define the quality of care by indicators not directly associated with health (e.g. organization of care [5,6] or process and practice [8,9,12]). Nonetheless, the final objective of evaluation of the quality of care is to improve health and reduce the frequency of adverse health outcomes. Another limitation of quality indicators based on health outcomes is that they often rely on vague or heterogeneous definitions that can lead to various interpretations. For example, some postpartum hemorrhage definitions are very precise (threshold of blood loss >1000 mL in the first 24 hours) [8], while others do not mention any threshold [4,7]. Lastly, most indicators are defined in the general population of women giving birth and not assessed within specific subgroups where they could best reflect the quality of care.
Severe acute maternal morbidity (SAMM), associated with the most pathological forms of maternal health, complicates around 1% of deliveries. Its events are considered avoidable in a large fraction of cases, for reasons often linked to inadequate or faulty quality of care [13]. An evaluation tool based on the severe maternal morbidity outcomes that best reflect quality of care might therefore constitute a useful contribution to quality surveillance in obstetrics.
Permanent hospital discharge databases that synthesize information on each hospital stay exist in the great majority of high-resource countries, usually for initial billing purpose. They are coded in a standardized manner and contain diagnostic codes derived from the international ICD9 or ICD10 classifications used worldwide, and procedure codes derived from classifications that may be more country-specific but for which equivalences between countries can be established. They are therefore a potential common source for the continuous surveillance of quality of care in the great majority of high-income countries. In addition, two aspects of this data source are of particular interest for the identification of severe maternal outcomes. First, it allows exhaustive coverage, as mothers with serious complications are systematically hospitalized, in high-resource settings. Second, the validity of the reporting of some of these severe maternal outcomes in this database has already been assessed [14,15].
The objective was to propose a set of indicators of the quality of hospital obstetric care chosen from severe maternal morbidity and mortality outcomes that reflect the quality of care, can be modified by improving practices, are reliably identifiable in routine hospital discharge data and do not depend exclusively on individual characteristics.

Organization of the delphi process
We used a formal Delphi expert consensus method to reach a consensus in the selection of indicators to characterize the quality of hospital obstetric care. This method took place in several stages: i) first, we conducted a systematic review of the literature of the different indicators proposed internationally to assess the quality of obstetric care, ii) next, we set up a panel of French experts representing the specialists/professionals concerned: gynecologists-obstetricians, midwives, and anesthetists-critical-care specialists, together with representatives of service users, and iii) finally, we applied Delphi techniques to consult this panel. The process was supervised by a Scientific Committee composed of AAC, CCD, CDT, FG, HK and PS, who represented the professions consulted and are also researchers in epidemiology and social sciences or specialists in the quality of care.
This study was approved by the Advisory Committee on health research information (CCTIRS) on October 9, 2014, and by the CNIL (n˚DR-2015-233, April 17, 2015. Because the Momassi project is not an interventional research and does not collect individual information, it did not require a formal ethics agreement, in accordance with the French law (Jardé legislation).

Literature review
Based on a systematic review of the literature about the quality of obstetric care, we drafted an initial list of indicators. To review the literature, we searched PubMed with the key words "quality indicators" and "obstetrics" for the period beginning January 1, 2003, andending August 31, 2016 (327 articles). Review of the bibliographies in these articles led to the addition of 17 more articles to the corpus.
A review of the abstracts of these 344 articles enabled us to classify 186 as not relevant, because they dealt with oncology, assisted reproductive technology, or gynecology. Another 42 articles concerned low-resource countries and were not transposable to settings in rich countries; 15 concerned solely fetal/neonatal outcomes, and 9 had either no abstract at all or no abstract in English or French.
We therefore selected 92 articles for a full reading and discussion by the Scientific Committee. After reading, 27 were eliminated from consideration because the quality indicators that they mentioned were indicators of practices, and 10 more articles because they were thought pieces about quality of care rather than analyses of its indicators. The remaining 55 articles mentioned indicators of quality of care based on maternal morbidity outcomes (listed in Appendix 1). This review of the literature finally produced 15 indicators that met the eligibility criteria we had defined: a maternal morbidity event proposed as a quality indicator and for which we had verified the availability in French routine hospital discharge data. Fig 1 synthesizes the process of the literature review and selection. The Prisma checklist is available in supporting information section (S1 File).Consultation of the websites and reports of various national and international learned societies did not identify any additional indicators not already mentioned in the literature we had reviewed.

Constitution of a panel of experts
The experts recruited were either designated by three learned societies (the French National College of Gynecologists-Obstetricians, the National College of Midwives and the Club of Anesthetists-Critical-Care specialists in obstetrics) as experts in severe maternal morbidity or had worked on guidelines for the quality of care issued by French institutions (French National Authority for Health or the French National College of Gynecologists-Obstetricians). A balance between the various types of maternity units (of differing size and status) and the places (regions) of practice was sought. We also recruited user representatives through 3 associations of patients. Of the 49 individuals approached, 37 agreed to participate in this study. The panel included 16 obstetricians, 6 midwives, 11 anesthetists-critical-care specialists, 2 representatives of user associations, one specialist in the quality of care, and one person who preferred not to be identified.

Organization of the consultation and analytic strategy
Initiated during the 1950s in the United States, the Delphi-type consensus approach makes it possible to organize a consultation of experts on a specific subject [16] by using iterative questionnaires (3 or 4 rounds); the panel must have feedback about all of the opinions stated to reach new positions, and the exchanges remain anonymous, which encourages free expression and avoids opinion leader effects. Delphi consensus processes have a proven track record for selecting indicators of quality of care in obstetrics [17]. This consultation was organized by a Delphi process in three rounds, completed online with Survey Monkey1 polling software, from November 2016 through February 2017.
Each questionnaire was sent by individual emails. A reminder was sent to all participants 48 hours before the deadline, and another to those who had not responded at the deadline. The experts were to assess whether or not each of the 15 indicators was a good indicator of the quality of obstetric care, that is, was rarely expected except in cases of inadequate quality of care or variations of which reflected various levels of quality of care. The first questionnaire also proposed that the indicator be considered for some specific subgroups, because these morbid outcomes might reflect quality of care particularly when they occur among these subgroups, such as women in principle at low risk of the morbid complication considered. The experts were asked to assess the relevance of each indicator for this population (or these populations). It was clearly stated that the objective was not to combine the indicators selected into a global rate but rather to consider them one by one. Finally, the experts could suggest additional indicators if they wished.
Each of the proposed indicators was rated by the participants on a 4-point Likert-type scale: 1 = Disagree strongly, 2 = Disagree, 3 = Agree, and 4 = Agree strongly.
All of the analyses were conducted anonymously. For the first round, a threshold of 75% of consistent responses between the participants-of agreement or disagreement for each question (indicator, definition, or population)-was chosen to define consensus. Items scored as 1 and 2 were added together for disagreement and those coded scored 3 or 4 for agreement. For the second and third rounds, a threshold of 70% of consistent responses was chosen. To avoid a fourth round, the Scientific Committee decided to include indicators or populations very close to the threshold (>66%) in the third round. Those not reaching a consensus in the third round were not selected.

Results
Among the panel of 37 participants in the first round, the participation rate was 92% in the second round (34 respondents) and 97% in the third (36 respondents). Table 1 recapitulates the indicators submitted to the panel as well as the precisions of definitions or groups of women concerned, and those selected at the different rounds.
The panel excluded four indicators during the Delphi process. These were the rates of maternal pyelonephritis (89% agreement to exclude it in round 1), laparotomy after delivery (consensus not reached, exclusion at the end of round 3), eclampsia (consensus not reached, excluded at the end of round 3), and maternal ICU admissions. The latter had the particularity of receiving its highest vote count during the first round rather than any of the following rounds (not selected in the third round, with only 44% agreement). This change resulted from feedback from experts, who insisted that this indicator depends strongly on the local organization of care Some indicators required adjustments. Accordingly, a proposal that the indicators of the rates of severe perineal lacerations and of episiotomies should be monitored concomitantly rather than separately was submitted to the panel and approved. To specify that it is most relevant in particular populations, it was finally decided to consider the cesarean rate in 3 different populations: among all women, among those at low risk of cesarean (singleton in cephalic presentation at term, with no abnormal placental insertion, no history of cesarean, and no preeclampsia) and among women who were at low risk of a cesarean but nonetheless had a cesarean before labor. These populations were specified only during the second and third rounds of the consultation.
At the conclusion of the first round, the panel considered that the indicator concerning the complications of anesthesia was too unspecific. The complications were then separated into subindicators, three of which were selected during the consensus process: headaches induced by spinal or epidural anesthesia and for which a blood patch was performed, anesthesia-related Table 1. Momassi delphi process: Indicators of quality of care considered and selected or excluded at different rounds of the consultation.

Indicators
Consensus to include or exclude the indicator Agreement % (Round)  (2) P_ Among all women Included 79% (3) P_ Among women who had a planned cesarean before labor Excluded 93% (1) P_ Among women at low risk of PPH � Included 66% (3)

Maternal readmission rate after hospitalization for delivery
pulmonary aspiration, and "other pulmonary complications" of anesthesia. Moreover, the panel proposed 13 additional indicators. The Scientific Committee endorsed two of them because they concerned maternal health, were proposed by at least 2 experts, and could be identified in the hospital data. These were the rate of cesareans under general anesthesia and an item entitled "other complications associated with care," to be entered by medical staff at discharge, when applicable. Both of these received consensus approval in the second round. The other indicators proposed were not selected, because they concerned either neonatal health status or items related to practices rather than health status. The restriction of indicators to one or several groups of women in which they might be considered more pertinent than in the general population was suggested for eight of the indicators proposed to the panel, and finally kept for six. A consensus was reached for 3 subgroups of women at the first round, rate of uterine rupture among women with a previous cesarean (77% agreement, note this is the only subgroup of women at higher risk than the general population), rate of episiotomy among all women (73% agreement), and rate of postpartum hemorrhage among women who had a planned cesarean before labor (93% agreement to exclude it in round 1).
The third round of consultation finally resulted in a set of 13 indicators of the quality of hospital care, reported in Table 2.

Principal results
Through a Delphi-type process that included a national panel of French experts, it was possible to define a set of 13 indicators to characterize the quality of hospital obstetric care chosen from maternal morbidity outcomes that can be identifiable in routine hospital discharge databases. One original aspect of the process was to consider assessing some indicators in particular subgroups of women in whom these outcomes are rarely expected; this choice makes it possible to identify perspectives to improve clinical practices.

Complications of anesthesia: 6 subindicators first specified in the second round
Included 84% (1)

I_ Anesthesia-related complications involving the central nervous system
Not selected 65% (3)

I_ Toxic reaction to local anesthesia
Not selected

Include
69% (3) 1 The panel decided that these indicators should be monitored concomitantly rather than separately � Population at low risk of PPH: singleton at term, with no abnormal placental insertion, no history of cesarean, and no preeclampsia �� Population at low risk of cesarean: singleton in cephalic presentation at term, no abnormal placental insertion, no history of cesarean, and no preeclampsia ��� : Population at low risk of maternal mortality: singleton at term, no abnormal placental insertion, no history of cesarean, no preeclampsia, and no chronic maternal disease https://doi.org/10.1371/journal.pone.0211955.t001

Strengths and limitations
The panel had thorough knowledge of the topic of quality of care and of maternal morbidity and represented 4 professions and different modes of practice. The excellent participation rate is evidence simultaneously of good panel selection, interest in the subject, and an effective email reminder strategy. Participation by users' representatives was one of the Scientific

Antenatal Birth Postpartum
Gray boxes = selected groups of women � Item reflecting "major" transfusion as coded in the French hospital discharge database �� Population at low risk of PPH: singleton at term, no abnormal placental insertion anomaly, no history of cesarean, and no preeclampsia ��� Population at low risk of cesarean: singleton in cephalic presentation at term, no abnormal placental insertion, no history of cesarean, and no preeclampsia ���� : Population at low risk of maternal mortality: singleton at term, no abnormal placental insertion, no history of cesarean, no preeclampsia, and no chronic maternal disease https://doi.org/10.1371/journal.pone.0211955.t002 Committee's priorities and was achieved, although their participation in all three rounds of the consultation is rare in Delphi processes.
Although several of the sets of indicators of quality of care found in the literature include some components of maternal morbidity, they are generally mixed within composite indicators that include indicators of practices, organization of care, and/or neonatal outcomes [5][6][7][8][9]. To our knowledge and based on our systematic review of the literature on this topic, this set of indicators is the first that considers only indicators based on maternal morbidity outcomes. The combination of indicators assessing the quality of obstetric and anesthetic care in the same tool is also a strength and highlights the multidisciplinary approach to the management of obstetrical patients.
The choice of indicators identifiable in routine hospital discharge databases will allow regional and national comparisons and benchmarking of performance as well as ongoing surveillance of changes over time. The "diagnosis" indicators chosen in this process (e.g. uterine rupture, postpartum hemorrhage, perineal laceration, infection) are defined with precision by the panel and can be described by their codes in ICD9 or ICD10, the classifications used for coding medical information worldwide. They should accordingly enable international comparisons of these quality of care indicators. Finally, our proposal to consider some quality indicators in one or more specific subgroup of women is an originality of this project and suggests the utility of developing research in this direction.
This method of consultation nonetheless has limitations. Even though the panel was selected mainly by nominations from learned societies who identified experts on the subject matter, some respondents nonetheless found it difficult lend themselves entirely to this exercise without a feeling of defensiveness about their practices. Moreover, the consensus process led investigators/respondents to introduce components on the frontier between practices and maternal morbidity. This explains the less than severe character of some of the events included in the final set of indicators. This is also related to the well-known difficulty of defining a threshold or borderline for SAMM [18]. Furthermore, the choice of indicators by the panel may have been different in another national context; however, this appears unlikely since the main causes of maternal mortality are similar in France and in other high-income countries [19]. Finally, this set of indicators is not intended to cover all types of obstetric care but rather to focus mainly on delivery and its immediate consequences, although two indicators do explore the 42-day postpartum period (maternal readmission rate after hospitalization for delivery and maternal mortality rate). Indicators for the quality of prenatal care are also necessary and can be built applying the same procedure but must also include aspects of outpatient care and therefore other data sources.
While we have proposed a tool to assess the quality of hospital obstetric care with indicators available in hospital discharge databases, the translation of this set into an algorithm of codes will be followed by a validation stage. The validity of the reporting in hospital databases of some of the selected maternal outcomes has already been assessed, in France and internationally [14,15,20,21]. We will complete this assessment through a comparison with data from specific French epidemiological population-based studies. It is also important to note that we have not assigned specific targets or thresholds to each event. This is probably the most complicated aspect of performance monitoring and could be considered in subsequent research.

The participating clinicians and the quality of care
As Bailit et al. recently underlined in a recent SMFM report, the presence of clinicians within the research teams and expert groups working on indicator development, for example, both organizing and participating in Delphi procedures, is essential, to avoid misinterpretations that literature review alone can induce [22]. Nonetheless, quality procedures are more often perceived as coercive or constraining than as steps leading to favorable changes in outcomes, and a "culture of quality" or a quality mindset does not yet appear to be fully integrated into the culture of French hospitals. In the comment areas available, some respondents expressed reactions expressing an analysis based more on clinical reasoning than on quality of care. Accordingly, about the cesarean indicator, one obstetrician remarked: "I don't really understand the question, if there are fetal heart rate abnormalities [in a woman at] low risk of cesarean delivery, the cesarean is not an indicator of inadequate quality of care" (excerpt from questionnaire O16). Another obstetrician said about the episiotomy indicator: "I do not understand how an episiotomy constitutes defective care" (excerpt from questionnaire O20).
These comments point to the physicians' feelings of loss of autonomy, in response to the demands they face to standardize their practices, as Eliot Freidson's work shows [23].

Point of discussion about the delphi method
Over the course of this Delphi consultation, some of the respondents' comments expressed their progressive ownership of the concept that some indicators can be usefully considered in some specific contexts or populations. This led us to wonder about the need for a conference (virtual or in person) to explain and train participants in the process before it starts. The Delphi method normally foresees, at least in one of its branches, the Delphi-Rand method, a final meeting to reach consensus. A preliminary training meeting may have two disadvantages: eliminating the panel members' anonymity and thus promoting opinion leader effects. It might also, however, have the advantage of clarifying the objectives.

Conclusion
The consultation of clinicians who were also experts in quality of care, representing the professions involved, as well as user representatives, enabled us to create a set of indicators to assess the quality of obstetric care from hospital discharge databases. The set is composed of 13 indicators, including 6 which will be considered in subgroups of women. The translation of this tool into an algorithm of codes will be followed by a validation stage.
This set is composed of outcomes identifiable in routine databases through codes available in international classifications; it may then be applied across or within countries to reveal themes for improving obstetric care.

Appendix 1: list of articles selected through the review of the international literature
Asch DA, Nicholson S, Srinivas S, Herrin J, Epstein AJ. Evaluating obstetrical residency programs using patient outcomes. Jama. 2009 Sep 23;302 (12)