Introduction

The adoption of the Sustainable Development Goals (SDGs) in September 2015 set out an ambitious new agenda for global development. Whereas the Millennium Development Goals (MDGs) had 8 goals and 21 targets, the SDGs have 17 goals and 169 targets [1,2]. The unprecedented increase in the amount of data needed for SDG monitoring has inspired calls for a ‘data revolution’, which necessitates innovative approaches to data collection, analysis and use [3]. The SDGs have also drawn attention to the need to expand the envelope of options to collect monitoring data but also the critical role that key data sources such as household surveys will continue to play in tracking progress and monitoring inequalities.

Household surveys are a rich source of data on major development goals such as the MDGs, and for sectoral data on water, sanitation and hygiene (WASH). Indeed, in the final assessment of the MDGs, 84 percent of data on WASH indicators came from household surveys [4]. The UNICEF-supported Multiple Indicator Cluster Surveys (MICS) is a key global data source and was established in the mid-1990s to support countries to collect representative data on the situation of children and women [5]. Close to 300 MICS surveys have been conducted in over 100 countries and MICS is entering the sixth round of surveys (MICS6). Without doubt, MICS and other survey programmes, such as the USAID-supported Demographic and Health Surveys (DHS) and the World Bank-supported Living Standards Measurement Study (LSMS) which routinely collect data on WASH, will continue to be instrumental in monitoring progress against national and international development goals for WASH and other sectors [6].

During the MDG era, the WHO/UNICEF Joint Monitoring Programme for Water Supply and Sanitation (JMP), which sets global standards for monitoring progress on water, sanitation and hygiene, estimated the use of ‘improved’ drinking water sources and sanitation facilities based on national data sources. Improved drinking water sources are those designed to protect against contamination, especially faecal matter, while improved sanitation facilities are those designed to safely separate excreta from human contact [7]. These terms have been widely used with the known limitation that improved facilities are not always ‘safe’. For example, water from a piped source or borehole, which is classified as ‘improved’, may still contain faecal contamination [8] while use of improved sanitation facilities may not prevent sewage and faecal sludge from being discharged into the environment without treatment [9]. The new global indicators for SDG targets 6.1 and 6.2 address these concerns by setting a higher benchmark of ‘safely managed’ drinking water and sanitation services.

‘Safely managed drinking water’ builds on the established indicator ‘use of an improved drinking water source’ and introduces three additional criteria: 1) the improved source is located on premises, 2) water should be available when needed, and 3) water supplied should be free from faecal and priority chemical contamination. While previous MICS surveys have included information on the type and location of drinking water sources used by households, questions on water availability and quality are new. Escherichia coli (E.coli) is the recommended indicator of faecal contamination which is a major concern for human health in all countries [10]. Evidence from a systematic review suggests that 1.8 billion people use a source of drinking water contaminated with faecal indicator bacteria [11].

‘Safely managed sanitation’ also builds on the established ‘improved/unimproved’ facility classification used in MICS and other surveys and addresses the subsequent management of excreta including safe treatment and disposal of excreta, which is an emerging issue in varied settings [12,13].

SDG Target 6.2 also addresses hygiene. The global SDG indicator focusses on access to a handwashing facility with water and soap at home, as a proxy for individual hygiene practices [14,15]. This indicator was developed by the MICS programme and has been used in surveys supported by MICS and DHS since 2009. Another emerging priority issue for the WASH community is management of menstruation. The current literature suggests that menstrual hygiene management (MHM) can be a challenge for women and girls, especially in low and middle-income settings though there have been few large-scale studies of the impacts on health, education and gender equality [16–19]. As this is a potential area for large-scale measurement, there is a need to test new questions on this topic that can be recommended to countries.

While the new SDG indicators for WASH build on established indicators, thereby providing continuity for monitoring, new elements place considerable demands on national household surveys to develop new data collection methodologies that respond to emerging data needs. The MICS survey programme, as part of its ongoing work to ensure that appropriate tools are available to countries for collecting household survey data, periodically updates all survey tools, including standard questionnaire modules and survey protocols. We designed a mixed-methods field test of MICS tools in Belize to assess the suitability and performance of new SDG WASH indicators for the MICS surveys. The objectives of this study are to examine the process of testing drinking water, both in the household and at the source, for E. coli as part of a MICS survey, assess a minimum set of new questions on the availability of drinking water, test respondent’s knowledge about faecal sludge management and evaluate questions on menstrual hygiene. The findings of this study will inform the uptake of questions and protocols into the MICS surveys and help shape WHO/UNICEF JMP recommendations for enhanced WASH monitoring.

Data and methods

This mixed-methods study evaluated questions and protocols implemented in a MICS field test in Stann Creek district, Belize from November to December 2015. Data were collected using quantitative and qualitative methods, which is a widely recommended practice in survey-development and questionnaire evaluation [20]. Prior to each element of data collection, interviewers explained the informed consent process, providing the purpose and content of the study and explaining that participation was optional and could be revoked by the participants at any time during the process. Participants then provided verbal consent to begin the interview or declined to participate. No personal identifiers were documented. The Statistical Institute of Belize (SIB) as the national authority on household data collection provided ethical oversight, reviewed all data collection protocols and approved all study protocols. Belize was selected for the field test as the country provides wide variation in respondents’ background such as education, level of rurality and wealth, which are desirable when testing questions across different domains and which reflects the variation which exists in the different countries that implement MICS. Further, as SIB had recently completed a national MICS survey, the field test used the same field workers for the field test, being able to take advantage of their intimate knowledge of the questions and protocols.

Results

Characteristics of households and women

Table 1 shows the characteristics of the households that were interviewed using the household survey. Overall, there sample was somewhat more rural than urban. A large majority of the heads of household were male. Three in four households had piped water while close to one in five relied on packaged water for drinking. The majority (69%) had a toilet facility that flushed to a septic tank while close to 20 percent relied on an improved latrine (see Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of households interviewed in the household survey.

https://doi.org/10.1371/journal.pone.0189089.t001

In total, 267 women were interviewed for the household survey, with slightly more in sample 1 (57 percent) than sample 2 (43 percent, see Table 2). The majority were from rural areas (55 percent), married (66 percent) and had secondary or higher education (55 percent).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of women in the household survey.

https://doi.org/10.1371/journal.pone.0189089.t002

Respondents to the cognitive interviews were 24 percent male and 76 percent female. They ranged in age from 18 to 65. About 41 percent lived in an urban area and 59 percent in a rural area. Cognitive interviews lasted a median of 28 minutes (range: 12–58 minutes). We included text data from the shorter interviews in the analysis to the extent that they provided information on question quality.

Questions on water availability

The two questions produced similar results; in sample 1, 12 percent of households did not have water at any time in the last month while in sample 2, 10 percent of households said that water was not always available. However, behaviour coding data showed that the first formulation (WS5C) consistently had higher percentages of interviewer and respondent problems (see Table 3), indicating more difficulty to implement WS5C. For example, in the first formulation of the question, close to 25 percent of cases needed further clarification or probing by the interviewer while this occurred in 15 percent of cases in the second formulation of question.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of behaviour coding.

https://doi.org/10.1371/journal.pone.0189089.t003

Respondents tended to correctly interpret “unable to access water in sufficient quantities when needed” (WS5C) as an interruption in the piping system or insufficient water pressure, as shown in this discourse:

“I: When she asked if you have enough drinking water, how do you come up with your answer of what is sufficient water?

R: If we have enough to drink, cook, clean, bathe.

I: And when she asked about not enough, how did you make the determination?

R: When your water system isn’t working properly or low water pressure, then you can say you do not have enough water to cook.

I: Even when you have water, sometimes it might not be enough.

R: Yes, say, for example, you need to do laundry. The water pressure is too low so it does not full up the machine enough to wash. Same if you need to bathe.” (Dangriga, 28 year old female)

Measuring the SDG indicator of safely managed drinking water

While virtually all household members used an improved source of drinking water or had drinking water located on their premises (98 and >99 percent respectively), fewer had water available when needed (87 percent) and only about two in three had water free from E. coli (64 percent, see Table 4). These sub-indicators, when combined, show that 57 percent of household members had a safely managed water service, in sharp contrast to the near universal level given by the MDG indicator (98 percent). By type of water source, the SDG value differed, with packaged water scoring higher than other sources, because of better water quality (see Table 4). Expected patterns of lower coverage of safely managed drinking water services among rural households are noted though patterns by wealth are not clear.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Safely managed drinking water services in Stan Creek, Belize ^a.

https://doi.org/10.1371/journal.pone.0189089.t004

The proportion of samples with detectable E. coli levels in drinking water increased between the point of collection and the point of consumption in the household (Fig 1). Overall, one third (36 percent) of household members collected water from a drinking water source with detectable E. coli compared with almost half (47 percent) at the point of consumption. The difference was smaller for higher levels of E. coli: 23 percent of samples were over 10 E. coli per 100 mL at the point of collection compared with 27 percent at the point of consumption. Contamination was higher in piped than packaged water, and much higher in ‘other sources’ (Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Levels of E. coli in drinking water collected at the source and consumed in the household.

https://doi.org/10.1371/journal.pone.0189089.g001

Emptying of pit latrines and septic tanks

Table 5 shows that a majority of pit latrines and septic tanks had never been emptied (53 percent), and about 20 percent were emptied in the previous five years. About one in four respondents did not know whether their facility had been emptied or could not provide an answer. The non-response to the question on “the last time emptied” (WS8A) is partly explained by qualitative findings. Interviewers in one focus group identified that respondents found the question inapplicable to pit latrines as the usual practice is to cover the hole when it is full and make a new one. Respondents renting accommodation also did not know when the septic tank was last emptied as this is the responsibility of the landlord. Behaviour coding indicated that in about one quarter of the cases, the interviewer had to provide clarification or probe further (see Table 3). We observed little to no heaping of data on expected categories in the distribution of the data on timing of emptying.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Emptying of sanitation facilities in households with latrines or septic tanks.

https://doi.org/10.1371/journal.pone.0189089.t005

In households where respondents reported that the sanitation facility had been emptied, the vast majority had been emptied mechanically by a service provider (see Table 5). Behaviour coding showed that in about 20 percent of cases, additional interviewer probing and clarification was provided, and in about 5 percent of cases the interviewers modified the question.

Menstrual hygiene management

Virtually all women age 15–49 interviewed had a private place to wash (95 percent), nearly always referring to the toilet or bathroom, and nearly all had access to materials for managing their periods (98 percent). About half of women used a ‘special bin with a lid’ to dispose of the menstrual materials, while 14 percent disposed of the materials in a latrine and 16 percent burned the materials. Rural women were more likely to dispose of menstrual materials in a latrine or burn the materials than urban women (see Table 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Menstrual hygiene management.

https://doi.org/10.1371/journal.pone.0189089.t006

In general, these questions were acceptable as part of the household survey; few women refused to answer these questions (less than 2 percent) and women, during cognitive interviews, showed few inhibitions discussing MHM with the exclusively female interviewers.

Behaviour coding, however, indicated that in 38 to 52 percent of cases for the three question, interviewers needed to clarify the questions or probe to elicit responses. Of the three questions, UN15 (on access to materials for the period) posed the greatest challenge to the interviewers in terms of clarifications and probing. Cognitive interviews provide further data on this. Words such as “washing” in UN14 evoked from some women, the idea of a bath tub. However, for most women interviewed, this could only mean a bathroom. The term ‘private’ seemed to cause some confusion as this respondent explains her thought process,

“Well, that one [referring to “private”] was a little strange, but I just think about the norm like the bathroom. When I hear private, I say all people will be the same, because private means, it will be just you. When I hear private place, it sounds like something other than the bathroom. It made it seem like you have other options.” (Dangriga, 28 year old female)

Still other women thought that “private place”, because it can only be a bathroom for menstrual hygiene, was a private room or bathroom exclusively for one individual woman to use:

“I told her no, because we only have one bathroom that everyone uses.” (Pamona, 40 year old female)

The focus groups also pointed out that the term “materials” (UN15) was not well-understood, in agreement to the behaviour coding. One focus group cites,

“The women did not understand that question. When the women hear materials, they think about materials to wash.” (Focus Group 1).

All women interviewed saw no problem with disposal of materials (UN16). They described the various ways of disposing, from wrapping in toilet paper, to placing in trash to burning. Interviewers concurred, citing that women often provided an elaborate response of how they disposed,

“They gave us a full briefing of how they disposed.” (Focus Group 1)

However, consistent with the high levels of probing seen in the behaviour coding (38 percent), most interviewers agreed that the place of disposal prompted considerable probing.

Discussion

This study evaluated new protocols and questions for enhancing WASH monitoring in the MICS surveys in order to track progress towards the SDG targets for drinking water, sanitation and hygiene. This is one of the few studies to examine new protocols on WASH monitoring for the SDGs using household surveys. The study tested different questionnaire approaches and protocols and indicated that some specific refinements to these are needed for the next round of MICS surveys.

The study found that drinking water consumed in the household was somewhat more likely to be contaminated than water collected at the source, similar to other studies of contamination of drinking water after collection from the main source [28,29]. It also highlights that the proportion of population using sources which meet the new SDG criteria for ‘safely managed’ drinking water is likely to be substantially lower than estimates based on the old MDG indicators [30]. These results underscore the value of indicators which go beyond access to infrastructure and reveal inequalities in service levels.

Respondents understandably want to know the results of water quality tests and future survey should clearly explain to respondents whether or not they will receive test results. Logistically, this is difficult as results are currently available 24 hours or later after the sample has been processed, at which time, field teams may have already left the area. Returning results also raises additional questions: does a single test provide enough evidence to indicate the need to change water treatment and storage practices? Are results enough for households to change behaviours? Or are governments better able to act on the results at the aggregate level? The Medical Research Council (UK) Wellcome Trust points out that further evidence on the utility of returning results on health-related findings is still needed [31]. We also consider that in many countries, providing drinking water is the responsibility of governments, which are better placed to identify and address sources of contamination than household surveys which can provide only minimal information to households safe handling and storage. The MICS programme recommends that national governments who own and implement the MICS surveys devise strategies for disseminating results to fit the needs of their contexts.

Testing water inside households interrupted the flow of the interview and increased the time in the household. Based on this finding, in the current round of MICS, we propose that field teams collect water samples and immediately perform testing after leaving the household. We also note that we tested water in all households to boost the sample size for analysis. In actual MICS surveys, countries are advised to test water in 3–5 households per cluster, which reduces the overall implementation cost for equipment and time in the field. Our results also showed that water availability can be measured in MICS, as is done in other household surveys [23]. Though the subjective measure produces somewhat more difficulty for interviewers and respondents, respondents were able to understand key elements of the question, such as having sufficient water and having it when needed, which is essential to having a broader understanding of the issue. For the latest round of MICS, we include this subjective measure and will closely monitor its performance across countries.

Safely managed sanitation services represent an ambitious new benchmark for global monitoring and tracking progress requires data from both household surveys and regulatory authorities or service providers [32]. In order for services to be considered safely managed excreta must be safely disposed of in situ or transported and treated offsite([25]. This study sought to understand the extent to which information for onsite sanitation facilities could be captured in MICS surveys–we focused on emptying of onsite sanitation facilities which is the first step in the faecal sludge management chain [33] and who was responsible for emptying the facilities. In Stann Creek, Belize, emptying of these facilities is not prevalent, and when done, it is performed mechanically by a service provider. This poses important questions about how the emptying is conducted and whether these service providers are treating the wastes appropriately or disposing of them inappropriately. A study of 12 cities suggests that in many settings faecal sludge is not being managed effectively [34]. We found significant numbers of respondents were not aware of emptying practices and this may be partly due to the fact that the person interviewed may not be responsible for construction and maintenance of sanitation facilities. This lack of knowledge poses challenges for monitoring emptying practices through household surveys and in itself may reflect systems that are not safely “containing” human waste. While MICS6 has adopted these questions as part of routine data collection and added a further question to attempt to establish where the excreta are disposed to (see MICS6 questionnaires at mics.unicef.org/tools), further exploratory work on emptying in settings with higher prevalence and more varied practices is planned for several countries as part of the Global Expanded Monitoring Initiative [35]. Moreover, there is a need to establish approaches for the collection of data on the rest of the faecal sludge management chain (transport, treatment, disposal/reuse).

The ability to manage menstruation safely and with dignity is fundamental for achieving gender equality [36]. A recent UN Women discussion paper describes many of the challenges women and adolescent girls face–ranging from lack of privacy and knowledge to ingrained taboos and social restrictions [37]. In consultation with MHM experts, we selected a short set of questions to examine whether women and girls (15–49) were able to wash in private during their last period, had access to menstrual materials and a suitable location to dispose of these. Only a small minority of women and girls in this study had difficulties with these issues, and this contrasts sharply with other studies including in India and Kenya, where access to materials, disposal and privacy for washing was much less common [38,39]. While some terms in the MHM questions were difficult to understand for respondents, we show that discussion of this topic was acceptable in a household survey in Stann Creek, Belize. To improve understanding of the concepts, MICS6 questions focus on being able to wash and change in privacy (rather than asking about a ‘private place’) and list types of menstrual hygiene management materials rather than referring to ‘absorbent materials’. These data collected in MICS6 will be the first global attempt to quantify the challenge that women and girls have to manage menstruation safely and with dignity and will complement information collected on facilities at institutional settings, including schools [40].

This study has several limitations. The survey data are representative of the Stann Creek district, Belize. As MICS surveys are conducted in a wide variety of contexts, we may not have identified all challenges with the new topics. WASH services in Stann Creek are better than many parts of the world, limiting the generalisability of the lessons learnt from this pilot. In particular, water was found to be available in almost all cases and most women had access to improved sanitation and used disposable materials. Further work is needed to ensure that the questions are appropriate for other contexts and understood by respondents in these contexts. Survey programmes are encouraged to “pre-test” questionnaires, an approach that is always applied in MICS surveys. The cognitive interviews and two focus groups covered a wide range of topics new to MICS. With limited numbers, the study may not have reached saturation on every topic explored. Nevertheless, patterns across the various evaluation techniques provided coherent, logical and explainable patterns. We advocate for a continued agenda of testing survey questions across different settings which would ultimately build a more comprehensive body of evidence on the performance of survey questions and protocols.

Collectively, these findings have been used to improve the protocols and guidelines of the MICS programme (available at mics.unicef.org/tools) which supports roughly 50 to 60 countries per round and covers all regions of the world. These are also useful to inform measurement guidelines from the WHO/UNICEF JMP and can potentially be adapted and adopted by other household survey programmes such as the DHS and LSMS. These actions will ultimately contribute to increasing the availability of more harmonized and comparable data for monitoring the new elements of SDG targets 6.1 and 6.2 at the country and global levels.

Supporting information

Acknowledgments

We would like to thank the Statistical Institute of Belize for their help with study implementation, the survey teams for their data collection efforts and respondents for taking part in this survey, and MHM experts who contributed to design of survey questions: Marni Sommer, Bethany Caruso, Sue Cavill and Penelope Phillips-Howard. We also thank Gabrielle Berman who advised on ethical issues.