Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Development and validation of a time and motion guide to assess the costs of prevention and control interventions for nosocomial infections: A Delphi method among experts

  • Eric Tchouaket Nguemeleu ,

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Methodology, Project administration, Validation, Writing – original draft, Writing – review & editing

    Affiliation Université du Québec en Outaouais, Department of Nursing Research, Saint-Jérôme, QC, Canada

  • Sandra Boivin,

    Roles Conceptualization, Data curation, Methodology, Project administration, Resources, Software, Writing – review & editing

    Affiliation Centre Intégré de Santé et de Services de Sociaux des Laurentides, Direction de la Santé Publique, Saint-Jérôme, Québec, Canada

  • Stephanie Robins,

    Roles Data curation, Methodology, Writing – original draft, Writing – review & editing

    Affiliation Université du Québec en Outaouais, Department of Nursing Research, Saint-Jérôme, QC, Canada

  • Drissa Sia,

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – review & editing

    Affiliation Université du Québec en Outaouais, Department of Nursing Research, Saint-Jérôme, QC, Canada

  • Kelley Kilpatrick,

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Writing – review & editing

    Affiliation Ingram School of Nursing, McGill University, Montréal, Québec, Canada

  • Sylvain Brousseau,

    Roles Conceptualization, Data curation, Methodology, Project administration, Resources, Writing – review & editing

    Affiliation Université du Québec en Outaouais, Department of Nursing Research, Saint-Jérôme, QC, Canada

  • Bruno Dubreuil,

    Roles Conceptualization, Data curation, Methodology, Software, Validation, Writing – review & editing

    Affiliation Institut de Cardiologie, Montreal Heart Institute, Montréal, Québec, Canada

  • Catherine Larouche,

    Roles Conceptualization, Data curation, Methodology, Software, Validation, Writing – review & editing

    Affiliation Centre Intégré Universitaire de Santé et de Services Sociaux du Saguenay, Lac-Saint-Jean, Québec, Canada

  • Natasha Parisien

    Roles Conceptualization, Data curation, Methodology, Software, Validation, Writing – review & editing

    Affiliation Institut National de Santé Publique du Québec, Québec, Canada



Nosocomial infections place a heavy burden on patients and healthcare providers and impact health care institutions financially. Reducing nosocomial infections requires an integrated program of prevention and control using key clinical best care practices. No instrument currently exists that measures these practices in terms of personnel time and material costs.


To develop and validate an instrument that would measure nosocomial infection control and prevention best care practice costs, including estimates of human and material resources.


An evaluation of the literature identified four practices essential for the control of pathogens: hand hygiene, hygiene and sanitation, screening and additional precaution. To reflect time, materials and products used in these practices, our team developed a time and motion guide. Iterations of the guide were assessed in a Delphi technique; content validity was established using the content validity index and reliability was assessed using Kruskall Wallis one-way ANOVA of rank test.


Two rounds of Delphi review were required; 88% of invited experts completed the assessment. The final version of the guide contains eight dimensions: Identification [83 items]; Personnel [5 items]; Additional Precautions [1 item]; Hand Hygiene [2 items]; Personal Protective Equipment [14 items]; Screening [4 items]; Cleaning and Disinfection of Patient Care Equipment [33 items]; and Hygiene and Sanitation [24 items]. The content validity index obtained for all dimensions was acceptable (> 80%). Experts statistically agreed on six of the eight dimensions.


This study developed and validated a new instrument based on expert opinion, the time and motion guide, for the systematic assessment of costs relating to the human and material resources used in nosocomial infection prevention and control. This guide will prove useful to measure the intensity of the application of prevention and control measures taken before, during and after outbreak periods or during pandemics such as COVID-19.


Nosocomial infections (NIs), also known as healthcare associated infections (HCAIs), are infections acquired during an episode of care in a healthcare facility [1]. These infections are directly related to care and are considered among the most common and preventable adverse events [2]. NIs place a significant burden on patients and hospital staff as they result in medical complications, prolonged hospital stays, high rates of morbidity and mortality, and reduced quality of life; they also place a significant burden on healthcare systems as they result in extra costs related to the extended hospitalization or readmission of patients, patient’s care-related expenses and costs involved in limiting further contagion [37]. Similar to other nations, Canada has not solved the problem of NIs. In 2013, the Public Health Agency of Canada reported that more than 200,000 patients contract a NI each year, resulting in more than 8,000 deaths [8]. In 2016, the Canadian Patient Safety Institute estimated that one NI occurs in every 41 hospitalizations, at an estimated cost of 281 million dollars, or 41% of the total cost of adverse events [9]. In the province of Québec, the Comité des infections nosocomiales du Québec (CINQ) estimated, based on American studies, that the annual number of NIs reached somewhere between 80,000 and 90,000 patients in 2005, with an estimated mortality rate of between 1 and 10% [10].

To address the issue of NIs, in 2004 the Québec Ministry of Health and Social Services (MSSS) implemented a mandatory program for monitoring NIs in all hospitals [11]. Since then, this program has become a mainstay for healthcare institutions in managing risks, quality, and patient safety. The program currently tracks different pathogens including: methicillin-resistant Staphylococcus aureus (MRSA), Clostridium difficile-associated diarrhea (CDAD), colonization and infection caused by multidrug-resistant bacteria such as vancomycin-resistant enterococci (VRE) and carbapenemase-producing enterobacteria (CPGNB). Along with NI tracking, target 22 of the MSSS 2015–2020 action plan, On the prevention and control of NIs for safe delivery of healthcare in the province of Quebec, is to “evaluate NIPC [Nosocomial Infection Prevention and Control] measures, taking into account the organizational model, the burden of disease, as well as their clinical and epidemiological impacts” [12].

In order to evaluate costs of core NIPC measures, it is helpful to refer to the framework set out by Resar and colleagues of the United States Institute for Healthcare Improvement (IHI) that describes an adverse-events intervention framework of clinical best practices (CBPs) [13]. The authors propose the implementation of “bundles” of evidence-based CBPs in order to ensure the best possible outcomes for: infection prevention, a reduction of complications and the highest level of quality and safety with regards to health care. This framework previously informed Canadian and Quebec healthcare infection prevention and control strategies as well as their respective safety care campaigns [1416]. Importantly, bundles are defined by the authors as: “A small set of evidence-based interventions for a defined patient segment/population and care setting that, when implemented together, will result in significantly better outcomes than when implemented individually.” Bundles were conceived from a collaborative effort between the IHI and the Voluntary Hospital Association who established a working group to obtain feedback from clinical experts working in 13 hospital intensive care units. The team also undertook an analysis of the medical evidence surrounding clinical processes that were considered to be both the most costly and potentially harmful. From this, design guidelines included 6 key concepts: a) Bundles must have 3–5 interventions and be clinician endorsed; b) Elements in bundles are relatively independent; c) Bundles are specific to a defined patient population; d) Multidisciplinary care teams are involved in the development; and e) Bundle elements should be locally customizable, and descriptive in nature and f) Compliance is measured as an all or none measurement.

According to the IHI, the application of bundled care produces better outcomes and has been shown to bring about monetary savings in the millions. This occurs when bundle elements are measured and accurate is obtained data, either confirming or challenging the assumption that care is being reliably delivered.

Bundles of care, however, do not usually evaluate CBPs that are routinely performed to control NIs. Despite this, four essential and overlapping CBPs are commonly associated with all levels of patient care: a) hand hygiene; b) hygiene and sanitation including the cleaning and disinfecting of surfaces and equipment; c) screening on admission and during hospitalization of carriers and patients who are at risk; and d) application of basic and additional precautions. As has been seen with the COVID-19 pandemic, these four CBPs are very important prevention and control practices to reduce NIs in hospitals [17, 18]. Any bundle designed for improved patient outcomes would be expected to include no less than one of these CBPs and potentially all four. To better understand how these transverse CBPs might impact the costs of care and associated patient outcomes, our team searched for a standardized or systematic way to assess costs.

Studies have reviewed costs and cost-effectiveness related to NIPC management in hospitals. Some studies have assessed the costs and cost benefits of a hand hygiene campaign on the reduction of the incidence of NIs, specifically as they related to MRSA [19], while others focused on the direct costs and cost savings of contact isolation precautions [2023]. The costs of active surveillance of VRE or MRSA have been estimated in regards to: systematic screening, hygiene procedures, wearing gowns, gloves, single use materials, hydroalcoholic solutions, disinfection procedures, and sanitation [24, 25].

To the best of our knowledge, and after a review of the literature, we found no formal systematic instrument or guide exists that captures the costs required to prevent and control NIs using the four CBPs presented above. Specifically, such a guide should capture, in real time, (i) the time healthcare workers spend on any sequence of actions related to the CBPs of NI prevention and control (hand hygiene, wearing protective barrier apparel, cleaning and disinfecting healthcare equipment and the environment, screening (with or without cultures), additional precautions (by contact, droplet or airborne), isolation, education, training and awareness campaigns); (ii) the materials used for these CBPs; (iii) the products required for these CBPs.


To fill this gap, this study aimed to develop and validate a time and motion guide that would be used to measure the costs of the four CBPs of NI prevention and control.


The research team formed a working group to establish the steps required to build the time and motion guide. It was composed of: nurses (SB, NP, CL) in the field of infection prevention and control; a professional expert in hygiene and sanitation (BD); and researchers with expertise in time and motion studies (KK); nursing administration (SB); public health (DS, ET); and, health economics (ET).

Literature review and meetings

Between March 2018 and July 2018, a review of the literature was undertaken in order to appraise and synthesize the data on the costs of the four CBPs, as well as the human resources, material and product costs related to NI prevention and control. We searched the scientific and grey literature using Medline, Cinhal, Pubmed, and Cochrane databases as well as Canadian and Québec government and World Health Organisation (WHO) websites, limiting our search from 2010 to 2018 inclusive. We used the following research strategy and key words: (tool OR instrument OR scale OR measure) AND (“nosocomial infections” OR “hospital acquired infections” OR “health care associated infections”) AND (prevention OR intervention OR treatment OR program OR screening OR assessment OR test OR diagnosis OR “screening tool” OR strategy OR management) AND (cost*). After screening the title and abstract of the articles, we selected and reviewed 22 articles [1941] and more than ten international and national reports [4256]. Five meetings of the working group occurred between March and July of 2018. As noted above, the literature was examined to better understand how to measure the costs of CBPs, however a comprehensive cost-accounting tool did not emerge. In order to accurately measure costs, we considered incorporating the methodology from time-driven activity-based costing (TDABC) [57]. TDABC functions by dynamically allocating costs related to the consumption of resources across human-driven processes, with the purpose of adding up costs throughout a supply chain to determine overall expenses. Adapted from a business model, TDABC has been used in healthcare to measure human resource costs related to specific medical conditions such as paediatric surgery, ear nose and throat medicine, dermatology and psychiatry [for a review see 58]. For our purposes, instead of measuring a medical condition we focused on our four CBPs and began breaking down each one in granular fashion, considering the start and end of each process. The activity and location of each CBP was determined. Process maps and all resources used for each step of any CBP. This process revealed that each CBP actually had layers of elements that resulted in the emergence of sub dimensions (later defined as items) that were not previously considered.

Further discussions helped to establish models of workflow of different healthcare staff, and brought to light which CBPs might be used to accomplish individual tasks. In order to capture the time required for CBPs, it was proposed that a chronometer be integrated that could measure at least one, but preferably more than one simultaneously occurring action. This proposal was accepted, and it was decided that external observers would use the guide and chronometer to capture data continuously. These two decisions fall in line with a recent schema of time and motion studies [see 59 for a review]. Members of the team proposed that this detailed and direct continuous observation approach to data collection may result in the tendency of staff to improve their performance [for an example in healthcare see 60]. This point was discussed, and accepted as a limitation of the instrument. Sessions were iterative, with issues being revisited as the guide was developed. By July of 2018, these sessions had proven to be key in helping to establish a consensus between all members as to the necessary components of an algorithm that would serve as a framework for the time and motion guide.

Creation of an algorithm and time and motion guide

To inform our process, the working group first drafted an initial version of the time and motion algorithm (see Fig 1). The algorithm outlines a series of sequential steps observers would follow when recording the time necessary for CBPs and for the materials and products used in daily prevention and control of NIs. The algorithm first included observer and location-specific measures to be captured including: identification of the observer; the date and site where measurements would be taken; the hospital unit (medicine or surgery), and the title of the professional being observed (patient-care attendant, nurse or housekeeping staff). The algorithm was further divided into seven main dimensions: 1) area being evaluated, 2) hand hygiene, 3) personal protective equipment, 4) screening, 5) disinfection of patient care equipment, 6) overall disinfection, hygiene and cleanliness, and 7) additional precautions. Each dimension contained sub-categories of items that described, in a more granular fashion, the time and/or material to capture. This information would provide a more precise measurement of the amount of personnel time and material costs used, and where these resources were spent. For example, the first sub-category of ‘area’ would clarify where (i.e., a private, semi-private or multi-bed room) staff spent time in NIPC actions. In the category of hand-washing, the sub-categories to identify included: when personnel washed their hands (before, during, or after patient intervention) along with the product they used: soap or hydroalcoholic solution. The final and most essential level of the algorithm notes that for each infection and control opportunity, time taken (in seconds) and products used will be systematically recorded.

The algorithm then helped guide the research team to develop the second phase: the creation of the time and motion guide, effectively operationalizing the constructs set out in the research question. The time and motion guide contains items in each of the seven dimensions. The guide was created as a paper questionnaire to be used for the standard measurement of NIPC costs. In order to optimally integrate the chronometer, the questionnaire was incorporated into an online application using the software platform Dataformz ( This digital version of the questionnaire allows for access from a variety of devices (tablet, phone or computer) in order to capture all actions in real time. It also provides a tool that facilitates nurse observer training and pilot testing province-wide. Users access the Dataformz platform using a secure URL and identify themselves by choosing their name from a list of user names. The dashboard first displays information that is deemed essential, such as the health care facility location, unit, personnel job title, and area being assessed; here users are presented with mandatory fields. At the completion of these fields, users are prompted to ‘submit’ their answers, effectively registering the collected data for that individual in the online database. The nurse observer following this professional then opens pages of the app where appropriate categories of the CBP are measured. For time measurements, users make use of the online chronometer to quantify movements in real time, and this, even if two or more actions are underway, as overlapping actions can be recorded. Recordings of time can be saved or, if there was an error, discarded. Similarly, all materials used in NIPC are listed on the platform as radio buttons or check-all boxes to facilitate data collection. Open text boxes (‘other’) are also provided in sections of the app where the description of a product is not predetermined. When an observation is complete, the user clicks the ‘end questionnaire’ button to submit their record.

Finally, a user manual of the guide was developed to facilitate training of nurse observers and ensure all observations are entered in a standardized fashion. The manual provides a description of the research team and a brief summary of the study’s purpose; the time and motion algorithm is annexed for reference purposes. The user manual describes in detail how to navigate each page of the online platform and how to correctly input observations. For each category of the guide, written explanations are accompanied by screenshots of the interface that users should expect to encounter on the app.

Construction of the time and motion guide questionnaire

The time and motion guide questionnaire to be sent to Delphi panelists included an initial version with 9 dimensions: (1) Identification [11 items]; (2) Personnel [4 items]; (3) Zone [2 items]; (4) Additional Precautions [1 item]; (5) Hand Hygiene [2 items]; (6) Personal Protective Equipment [2 items]; (7) Screening [4 items]; (8) Cleaning and Disinfection of Patient Care Equipment [5 items]; (9) Hygiene and sanitation [5 items]. The questionnaire was designed to elicit both qualitative and quantitative responses for the guide items and dimensions, and to obtain a qualitative evaluation of the web app functioning. Panelists were asked to rate all items in all dimensions of the guide, using a 4-point Likert scale from 1 = “not relevant” to 4 = “very relevant without corrections” as shorter Likert scales have produced stable findings in Delphi studies previously [61, 62]. Panelists were also provided an open-ended question about each item’s relevance, should they have a suggestion for improvement or other opinion.

Content validity of time motion guide

The content validity was assessed using the Delphi Technique among experts. The Delphi Technique is a well described group survey technique widely used in health research that results in the narrowing of a consensus on a topic of public importance [63]. The Delphi process allows for a variety of methodological approaches, including qualitative and quantitative data gathering and analysis that can be revised. It also allows for the investigation of a solution that may not yet exist. Due to its anonymity and iterative process, the Delphi method is considered to be less prone to bias as expert participants have the opportunity to provide their honest opinion or creative solution without feeling the need to agree with dominant personalities [64].

Our team used this systematic and interactive forecasting method as has been done by others in the field of health care [6567], to obtain consensus from a panel of experts, consulted over two or more rounds, for feedback on the time and motion guide. Experts were invited to two successive rounds of review. After each round, quantitative and qualitative data collected were analysed in order to assess the agreement and consensus. In each step, the panelists received the time and motion guide questionnaire, access to the Dataformz website and the user manual for revision and resubmission. Prior to the second round, panelists received their own feedback along with anonymized qualitative feedback from others (see S1 File for the final questionnaire provided to experts). Once two rounds were complete, the working group addressed any remaining discrepancies and produced a final version of all tools for pilot testing. Fig 2 presents the Delphi process for this study.

Fig 2. Validation process of the time and motion guide using the Delphi technique.

Delphi technique: Selection of expert panel and procedure

In order to consider a variety of opinions from participants having lived and/or professional experience in NI prevention and control, 18 experts [as recommended by 68] from two distinct viewpoints were selected: those who incorporated NIPC measures directly in their workday and those who could be considered content experts as a result of their professional training. The first group (12 experts) was composed of professionals that worked directly in infection prevention and control and were considered to have experiential expertise that complemented their professional training. A group of six was chosen from two healthcare facilities that differed in their structure, one being associated with a university and thus serving as a teaching facility, the other not. The time motion guide was analyzed by: a manager in charge of NIPC, two nurses, a microbiologist/infectious disease expert, and two health and sanitation staff. The second group of experts (n = 6) were chosen for their knowledge and expertise in the field, their active involvement in infection prevention and control via their role in government and public policy implementation, or because they were key opinion leaders. This group was made up of two microbiologist/infectious disease experts, two clinical nurse specialists in NIPC and two hygiene and sanitation advisors. We purposely sourced male and female panelists from different geographic regions of Quebec, Canada, to ensure the feedback would be valid for, and representative of, practices across the province. All experts who expressed interest in the study (n = 18) were sent a consent form to be returned by e-mail and all then provided informed consent.

The two rounds of the Delphi survey were conducted between July 2018 and January 2019. All experts were sent the time and motion guide questionnaire and asked to rate each item and dimension between 1 “not relevant”, 2 “relevant with major corrections”, 3 “relevant with few corrections”, and 4 “very relevant without corrections”, based on its relevance for the assessment of costs related to the four CBPs. The content validity was assessed using the content validity index (CVI) [69], which was set as a percentage, counting each item and dimension rated 4 “very relevant without corrections” over the total number of replies by experts for the item or dimension.

Three levels of review were considered to assess the content validity or agreement between experts. First, when the CVI for a dimension (CVI_D) was less than 0.80 (80%), the dimension was considered not acceptable. The dimension was then completely reviewed and revised according to the proposed corrections. The resultant items and dimension were then resubmitted to the same experts in the next Delphi round. When the CVI_D was equal to or greater than 0.80, the dimension was considered acceptable. Second, the CVI for individual items (CVI_I) of all dimensions was assessed. As above, when CVI_I was less than 0.80, the item was considered not acceptable and revised according to the proposed corrections. The item was then resubmitted to the same experts in the second round. When the CVI_I was equal to or greater than 0.80, the item was kept. In a third and final level of review, items that were revised but did not improve to the level of 0.80 following two rounds of Delphi review were revisited by the working group in order to finalize the items and address concerns expressed by the experts in round two.

Reliability of time motion guide

Finally, in order to ascertain reliability, (i.e., how much of the variance in observed ratings was due to a true difference in score, and not measurement error between groups of expert panelists), we performed non-parametric Kruskall-Wallis ANOVA of rank tests for each dimension, with 5% threshold, to demonstrate any effect of group. When the Kruskall-Wallis tests of a dimension was not statistically significant, it meant that all the responses were acceptably constant between the panel experts, or reliable. Otherwise, the dimension and its items were revisited in its entirety and revised according to the proposed corrections by experts. The dimension would then be resubmitted to the same experts in another round. All the revisions were discussed and validated by the research team during working group meetings. Data were analysed using SPSS version 26 (Armonk, NY, USA: IBM Corp). Ethical approval was obtained from the Centre intégré de santé et services sociaux des Laurentides (reference #: 2017MP-364-É and MP-28-2018-002) and the Université du Québec en Outaouais. All participants in the Delphi panel provided written informed consent.


Delphi: Rounds 1 & 2

In an initial round, all 18 experts were invited to complete the time and motion survey using the first version of the algorithm, user manual and web platform; tools were provided in French versions as all panelists were Francophones. Of these, 16 experts (88.9%) completed the questionnaire: 6 from the non-university affiliated hospital (group 1), 5 from the university-affiliated hospital (group 2), and 5 content experts (group 3). For all response rates, see Table 1. For quantitative responses, only one dimension of the guide, (cleaning and disinfection of patient care equipment) met our CVI_D acceptable limit of greater than 80%, at 83.4%. However, only two items within this dimension met a CVI_I of 80%. Reliability for all dimensions was acceptable (i.e., the groups did not significantly differ) with the exception of the dimension hygiene and sanitation, where the effect of expert group was significant, X2(2) = 9.02 p = 0.01. Thus, all the dimensions were reviewed and analyzed based on the comments and suggestions of experts. The guide was reworked and the questionnaire adapted for round 2, with all anonymized responses from the first round provided to experts.

Table 1. Delphi round 1 content validity index for all dimensions and items and panelist reliability by site.

In round 2, all 16 panelists who had completed round 1 were invited to respond to the second version of the questionnaire in order to evaluate the guide using the user manual and web platform. In this round, 14 experts (87.5%) completed the survey with 5 responses each from the university and non-university site, and 4 from content experts. Content validity improved markedly on round 2, with all dimensions surpassing the threshold of 80% (see Table 2). However, one item each from the dimension Identification and Zone as well as three within the dimension of Hygiene and sanitation, had CVIs of below 80%, and were flagged for further review. In this second round, reliability was acceptable for all dimensions except two. In Zone, the Kruskall Wallis test revealed a significant difference between groups of experts: X2(2) = 10.20 p = 0.006, as it did in the dimension Hygiene and Sanitation, X2(2) = 9.95 p = 0.007. Comments were reviewed in order to determine the source of disagreement.

Table 2. Delphi round 2 content validity index for all dimensions and items and panelist reliability by site.

Description of time and motion guide (final version)

The final version of the guide contains eight of the nine original dimensions, with the dimension Zone merged with Hygiene and Sanitation. These eight dimensions include: Identification [83 items]; Personnel [5 items]; Additional Precautions [1 item]; Hand Hygiene [2 items]; Personal Protective Equipment [14 items]; Screening [4 items]; Cleaning and Disinfection of Patient Care Equipment [33 items]; and Hygiene and Sanitation [24 items].

The guide and online platform have been substantially adapted. For example, the first section, Identification, was expanded from 12 items in round 2, to 83 items in the final version. This increased level of detail provides the user with the ability to measure items such as the brand and format of hydroalcoholic solution being used (gel or foam) and the count of disposable items. It also includes newly considered items that have a profound impact on NICP costs, such as staff attendance at NIPC training and information sessions as well as awareness campaigns. Other sections that were vastly revised included both sections that enumerated equipment (either personal protective equipment or patient monitoring equipment), to include detailed lists of all equipment used. The guide includes chronometers at the level of all items that require personnel time; the digital stopwatch can be set to measure time, either sequentially or simultaneously with CBPs, as needed. The complete paper version of the guide can be seen in S2 File. All changes to the time and motion guide were implemented in the final version of the web app and user manual in preparation for pilot testing.


The goal of this study was to develop and validate a time and motion guide to facilitate the objective assessment of the costs of four CBPs of NI prevention and control, using a systematic approach. Our process first defined an original set of items chosen from an extensive review of the literature. These were later expanded upon to create a final version of the time and motion guide in three successive waves of review, two of these undertaken by a panel of experts. By using the Delphi technique, preliminary evidence of validity and reliability of the guide were gathered.

Content validity is an important step in the development of questionnaires and quantifies the level of agreement between experts; we decided a priori to evaluate the assessments of our panelists using the CVI, both at the item and dimension level. According to Polit [69] the index is useful in questionnaire development as it provides a focus on consensus rather than consistency estimates, ease of computation, understandability and ease of communication, provision of both item diagnostic information and scale validity information”. By the end of two rounds of Delphi review, the CVI was deemed acceptable in all dimensions, and only failed to meet the 80% cut-off for six individual items. The Kruskall Wallis tests of reliability revealed disagreement between the groups in two dimensions: Zone, and Hygiene and Sanitation.

However, divergent opinions are an expected outcome of the Delphi process, and have been considered as representing smaller groups who hold a “clustered consensus” within the panel [70]. Therefore, along with the commentary provided by the panelists these items and dimensions were conceptually reviewed and extensively modified. The dimension of Zone was merged within Hygiene and Sanitation. This change reflected not only the poor statistical reliability between expert groups but also the context of sanitation procedures and how they might differ in terms of time and intensity, between physical spaces in clinical settings. For example, the cleaning of an isolation room may require more time or products as all surfaces and equipment are meticulously disinfected or discarded when patients are discharged. This intense terminal cleaning is not comparable to the hygiene and sanitation practices in a common area such as a hallway, where floors and hand railings receive daily disinfection.

The Delphi technique can provide invaluable insight for the development of new solutions; this is especially true within evolving disciplines such as healthcare. For example, Njuangang and colleagues [71], via the Delphi technique, established 11 novel critical success factors and performance measures within hospital maintenance to reduce the risk of maintenance-associated NIs. The ability to identify salient performance measures is also reflected in our study; over two rounds of review, 16 experts expanded and enriched the guide’s functionality and precision to create a tool with 8 dimensions and over 175 items that would assess cost. The strength of our process rested in part on our choice of panelists- a heterogeneous mix of stakeholders who were wholly invested in the innovation of the guide, from its development to its indicators of credibility and quality. From the original group of 18 invitees, 16 members completed the first wave of review, and all but two members completed the second. Those who recused themselves joined our research team in an advisory capacity to further the development and pilot testing of the guide, thus providing expert feedback as part of the working group in the final iteration.

Implications for research and practice

The time and motion guide, in its current version, is an instrument that can effectively allow health care managers and clinicians to assess the costs of nosocomial infection prevention and control practices. This information will complement epidemiological data that tracks the incidence of infection in health care facilities. Importantly, the guide provides a way to measure the cost effectiveness of CBPs in relation to the pathology of NIs. At the time of writing of this manuscript, measuring cost effectiveness in relation to infection control is exceptionally fitting as the world is struggling with the COVID-19 pandemic, where unprecedented infection control and prevention measures are being undertaken.

Aside from cost estimates, the time and motion guide is strategically detailed and can easily be implemented as an online instrument to serve as a training tool for personnel involved in NI control. Along with enumerating, for the user, materials and products used to control the spread of infection, the chronometer of the guide provides an objective measurement of the time taken for CBPs, such as hand washing. While subjectively, healthcare staff may consider their hand hygiene techniques adequate, they may not meet hand hygiene guidelines set out by the WHO that suggest 20–30 seconds for hydro-alcoholic solutions, or 40–60 seconds for soap and water [56]. It could thus serve in training and continuing education. The guide could also help us to better understand how personnel adapt their clinical best care practices to their working conditions. An analysis of the intensity of nosocomial prevention and control measures in terms of costs (time, materials and products) could be undertaken to compare practices and their related costs before, during and after outbreaks or pandemic events such as COVID-19.

Strengths and limitations

A strength of this study is that the time and motion guide represents a new and completely original approach to nosocomial infection prevention and control cost assessment. By adapting the questionnaire to a web-based application, the guide can span geographic separation of health care facilities where it can be tested, and may deliver a framework upon which to structure other health care associated costing tools. The time motion guide is innovative in that it simultaneously measures more than one action at a time, compared to other time motion guides that normally only measure a single action at a time [72, 73]. A further strength of our process is that the Delphi technique was assiduously conceived, administered and reported, an underlying prerequisite for credible conclusions. We attempted to fulfill all sixteen recommendations set out in the Conducting and REporting of DElphi Studies (CREDES) framework [63]. However, this study has limitations. The acceptability of the time and motion guide, and feasibility of use, will have to be pilot-tested by users in different health care facilities and in real-life settings under normal work conditions. Only then will the efficiency and effectiveness of the guide become evident. The study has certain other limitations, notably, that our guide is based on a health care system in one province and country. Health care settings, NI rates and NIPC practices will differ between developed and developing nations, limiting the generalizability of the guide.


We developed and validated a time and motion guide for the measurement of costs of clinical best practices used for nosocomial infection control and prevention in medical and surgical hospital units. Future research should include pilot testing of the guide with a view to how the instrument could be adapted for use in other healthcare settings.

Supporting information

S1 File. Time motion guide questionnaire for experts.


S2 File. Time motion guide paper version.



The authors would like to thank all our funders as well as Patrick Finken for his assistance with the web-based mobile platform.


  1. 1. Allegranzi B, Nejad SB, Combescure C, Graafmans W, Attar H, Donaldson L, et al. Burden of endemic health-care-associated infection in developing countries: systematic review and meta-analysis. The Lancet. 2011;377(9761):228–41. pmid:21146207
  2. 2. Schwendimann R, Blatter C, Dhaini S, Simon M, Ausserhofer D. The occurrence, types, consequences and preventability of in-hospital adverse events–a scoping review. BMC Health Services Research. 2018;18(1):521. pmid:29973258
  3. 3. Arefian H, Hagel S, Heublein S, Rissner F, Scherag A, Brunkhorst FM, et al. Extra length of stay and costs because of health care-associated infections at a German university hospital. Am J Infect Control. 2016;44(2):160–6. pmid:26521700
  4. 4. Bohlouli B, Jackson T, Tonelli M, Hemmelgarn B, Klarenbach S. Health care costs associated with hospital acquired complications in patients with chronic kidney disease. BMC Nephrol 2017;18(375). pmid:29282006
  5. 5. Cassini A, Plachouras D, Eckmanns T, Abu Sin M, Blank HP, Ducomble T, et al. Burden of Six Healthcare-Associated Infections on European Population Health: Estimating Incidence-Based Disability-Adjusted Life Years through a Population Prevalence-Based Modelling Study. PLoS Med. 2016;13(10):e1002150. pmid:27755545
  6. 6. McGarry SA, Engemann JJ, Schmader K, Sexton DJ, Kaye KS. Surgical-site infection due to Staphylococcus aureus among elderly patients: mortality, duration of hospitalization, and cost. Infect Control Hosp Epidemiol 2004;25(6):461–7. pmid:15242192
  7. 7. Serra-Burriel M, Keys M, Campillo-Artero C, Agodi A, Barchitta M, Gikas A, et al. Impact of multi-drug resistant bacteria on economic and clinical outcomes of healthcare-associated infections in adults: Systematic review and meta-analysis. PLOS ONE. 2020;15(1):e0227139. pmid:31923281
  8. 8. Public Health Agency of Canada. The Chief Public Health Officer’s report on the state of public health in Canada 2013: infectious disease–the never-ending threat. Ottawa, ON: PHAC. 2013.
  9. 9. Chan B, Cochrane D. Measuring Patient Harm in Canadian Hospitals. What can be done to improve patient safety? Canadian Institute for Health Information, Canadian Patient Safety Institute. 2016.
  10. 10. Comité d’examen sur la prévention et le contrôle des infections nosocomiales. D’abord, ne pas nuire…Les infections nosocomiales au Québec, un problème majeur de santé, une priorité. Gouvernement du Québec, Ministère de la Santé et des Services sociaux. 2005.
  11. 11. MSSS. La prévention et le contrôle des infections nosocomiales: Cadre de référence à l’intention des établissements de santé du Québec. Ministère de la Santé et des Services sociaux-Direction générale de la santé publique; 2006.
  12. 12. MSSS. Plan Stratégique du Ministeère de la Santé et des Services sociaux du Québec 2015–2020 (mise à jour 2017). La Direction des communications du ministère de la Santé et des Services sociaux: Ministère de la Santé et des Services sociaux du Québec 2015–2020; 2017 Décembre.
  13. 13. Resar R., Griffin FA., Haraden C., Nolan TW. Using Care Bundles to Improve Health Care Quality. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement (Available on; 2012.
  14. 14. Canadian Patient Safety Institute. Canadian Patient Safety Institute. New approach to controlling superbugs. Safer Healthcare Now! Campaign; Ottawa, ON: CPSI. 2010.
  15. 15. Groupe de travail en hygiène et salubrité. Guide de gestion intégrée de la qualité en hygiène et salubrité. Gouvernement du Québec, Ministère de la Santé et des Services sociaux. 2013.
  16. 16. Groupe Hygiène et salubrité au regard de la lutte aux infections nosocomiales. Lignes directrices en hygiène et salubrité: analyse et concertation. Québec, QC: Gouvernement du Québec, Ministère de la Santé et des Services sociaux, Direction générale de la coordination, du financement et de l’équipement. 2006.
  17. 17. First Affiliated Hospital of Zhejiang University. Handbook of COVID-19 Prevention and Treatment 2020. Available from:
  18. 18. U.S. Department of Health and Human Sevices, Centers for Disease Control and Prevention. Comprehensive Hospital Preparedness Checklist for Coronavirus Disease 2019 (COVID-19) 2020. Available from:
  19. 19. Chun JY, Seo HK, Kim MK, Shin MJ, Kim SY, Kim M, et al. Impact of a hand hygiene campaign in a tertiary hospital in South Korea on the rate of hospital-onset methicillin-resistant Staphylococcus aureus bacteremia and economic evaluation of the campaign. Am J Infect Control. 2016;44(12):1486–91. pmid:27658769
  20. 20. Roth JA, Hornung-Winter C, Radicke I, Hug BL, Biedert M, Abshagen C, et al. Direct Costs of a Contact Isolation Day: A Prospective Cost Analysis at a Swiss University Hospital. Infect Control Hosp Epidemiol. 2018;39(1):101–3.
  21. 21. Bessesen MT, Lopez K, Guerin K, Hendrickson K, Williams S, O'Connor-Wright S, et al. Comparison of control strategies for methicillin-resistant Staphylococcus aureus. Am J Infect Control. 2013;41(11):1048–52. pmid:23663857
  22. 22. Wassenberg MW, Kluytmans JA, Bosboom RW, Buiting AG, van Elzakker EP, Melchers WJ, et al. Rapid diagnostic testing of methicillin-resistant Staphylococcus aureus carriage at different anatomical sites: costs and benefits of less extensive screening regimens. Clinical microbiology and infection: the official publication of the European Society of Clinical Microbiology and Infectious Diseases. 2011;17(11):1704–10. pmid:21595786
  23. 23. Herr CE, Heckrodt TH, Hofmann FA, Schnettler R, Eikmann TF. Additional costs for preventing the spread of methicillin-resistant Staphylococcus aureus and a strategy for reducing these costs on a surgical ward. Infect Control Hosp Epidemiol. 2003;24(9):673–8. pmid:14510250
  24. 24. van Rijen MM, Kluytmans JA. Costs and benefits of the MRSA Search and Destroy policy in a Dutch hospital. Eur J Clin Microbiol Infect Dis. 2009;28(10):1245–52. pmid:19618223
  25. 25. Hassan K, Koh C, Karunaratne D, Hughes C, Giles SN. Financial implications of plans to combat methicillin-resistant Staphylococcus aureus (MRSA) in an orthopaedic department. Ann R Coll Surg Engl. 2007;89(7):668–71.
  26. 26. Escaut L, Bouam S, Frank-Soltysiak M, Rudant E, Saliba F, Kassis N, et al. Eradication of an outbreak of vancomycin-resistant Enterococcus (VRE): the cost of a failure in the systematic screening. Antimicrob Resist Infect Control. 2013;2(1):18. pmid:23738672
  27. 27. Chowers M, Carmeli Y, Shitrit P, Elhayany A, Geffen K. Cost Analysis of an Intervention to Prevent Methicillin-Resistant Staphylococcus Aureus (MRSA) Transmission. PLoS One. 2015;10(9):e0138999. pmid:26406889
  28. 28. Conterno LO, Shymanski J, Ramotar K, Toye B, Zvonar R, Roth V. Impact and cost of infection control measures to reduce nosocomial transmission of extended-spectrum beta-lactamase-producing organisms in a non-outbreak setting. J Hosp Infect. 2007;65(4):354–60.
  29. 29. Montecalvo MA, Jarvis WR, Uman J, Shay DK, Petrullo C, Horowitz HW, et al. Costs and savings associated with infection control measures that reduced transmission of vancomycin-resistant enterococci in an endemic setting. Infect Control Hosp Epidemiol. 2001;22(7):437–42. pmid:11583213
  30. 30. Birgand G, Leroy C, Nerome S, Luong Nguyen LB, Lolom I, Armand-Lefevre L, et al. Costs associated with implementation of a strict policy for controlling spread of highly resistant microorganisms in France. BMJ Open. 2016;6(1):e009029. pmid:26826145
  31. 31. Creamer E, Galvin S, Dolan A, Sherlock O, Dimitrov BD, Fitzgerald-Hughes D, et al. Evaluation of screening risk and nonrisk patients for methicillin-resistant Staphylococcus aureus on admission in an acute care hospital. Am J Infect Control. 2012;40(5):411–5. pmid:21962934
  32. 32. Gagnaire J, Gagneux-Brunon A, Pouvaret A, Grattard F, Carricajo A, Favier H, et al. Carbapenemase-producing Acinetobacter baumannii: An outbreak report with special highlights on economic burden. Med Mal Infect. 2017;47(4):279–85. pmid:28343727
  33. 33. Gardam MA, Burrows LL, Kus JV, Brunton J, Low DE, Conly JM, et al. Is surveillance for multidrug-resistant enterobacteriaceae an effective infection control strategy in the absence of an outbreak? The Journal of infectious diseases. 2002;186(12):1754–60. pmid:12447761
  34. 34. Nelson RE, Samore MH, Jones M, Greene T, Stevens VW, Liu CF, et al. Reducing Time-dependent Bias in Estimates of the Attributable Cost of Health Care-associated Methicillin-resistant Staphylococcus aureus Infections: A Comparison of Three Estimation Strategies. Med Care. 2015;53(9):827–34.
  35. 35. Scott RD 2nd, Sinkowitz-Cochran R, Wise ME, Baggs J, Goates S, Solomon SL, et al. CDC central-line bloodstream infection prevention efforts produced net benefits of at least $640 Million during 1990–2008. Health Aff (Millwood). 2014;33(6):1040–7. pmid:24889954
  36. 36. Raschka S, Dempster L, Bryce E. Health economic evaluation of an infection prevention and control program: are quality and patient safety programs worth the investment? Am J Infect Control. 2013;41(9):773–7. pmid:23993762
  37. 37. Slayton RB, Scott RD, Baggs J, Lessa FC, McDonald LC, Jernigan JA. The cost-benefit of federal investment in preventing Clostridium difficile infections through the use of a multifaceted infection control and antimicrobial stewardship program. Infect Control Hosp Epidemiol. 2015;36(6):681–7. pmid:25783204
  38. 38. Slover J, Haas JP, Quirno M, Phillips MS, Bosco JA 3rd. Cost-effectiveness of a Staphylococcus aureus screening and decolonization program for high-risk orthopedic patients. J Arthroplasty. 2011;26(3):360–5. pmid:20452175
  39. 39. Sewell B, Rees E, Thomas I, Ch'ng CL, Isaac M, Berry N. Cost and Impact on Patient Length of Stay of Rapid Molecular Testing for Clostridium difficile. Infect Dis Ther. 2014;3(2):281–93. pmid:25183400
  40. 40. Barkatali BM, Heywood N, White R, Paton RW. MRSA screening in orthopaedic surgery: clinically valuable and cost effective? A prospective analysis of 8,867 patients. Acta Orthop Belg. 2013;79(4):463–9. pmid:24205779
  41. 41. Hardy KJ, Szczepura A, Davies R, Bradbury A, Stallard N, Gossain S, et al. A study of the efficacy and cost-effectiveness of MRSA screening and monitoring on surgical wards using a new, rapid molecular test (EMMS). BMC health services research. 2007;7:160 pmid:17915008
  42. 42. Institut national de la santé publique du Québec. Surveillance provinciale des diarrhées à Clostridium difficile au Québec: protocole. Montréal, QC: INSPQ [Internet]. 2018. Available from:
  43. 43. Institut national de la santé publique du Québec. Surveillance provinciale des bactériémies nosocomiales dans les centres hospitaliers de soins aigus du Québec: protocole. Montréal, QC: INSPQ; 2018. 2018.
  44. 44. Institut national de la santé publique du Québec. Surveillance provinciale des infections à entérocoque résistant à la vancomycine au Québec: protocole. Montréal, QC: INSPQ. 2018.
  45. 45. Institut national de la santé publique du Québec. Surveillance provinciale des bactériémies à Staphylococcus au Québec: protocole. Montréal, QC: INSPQ. 2018.
  46. 46. Des soins de santé plus sécuritaires maintenant! Cadres de travail de l’amélioration de la qualité. Trousse en avant! 2015.
  47. 47. Ministère de la Santé et des Services sociaux, Plan d'action 2010–2015. Prévention et contrôle des infections nosocomiales 2011 Available from:
  48. 48. Ministère de la Santé et des Services sociaux, 2010–2015. Prévention et contrôle des infections nosocomiales, Plan d’action 2010–2015, État d’avancement des travaux, Bilan et faits saillants 2013. Available from:
  49. 49. Agence de la santé publique du Canada. Rapport de l’administrateur en chef de la santé publique sur l’état de la santé publique au Canada, 2013 –Les infections associées aux soins de santé–une diligence raisonnable 2013. Available from:
  50. 50. U.S. Department of Health & Human Services, Centers for Disease Control and Prevention. Guidelines for Environmental Infection Control in Health-Care Facilities, Recommendations of CDC and the Healthcare, Infection Control Practices Advisory Committee (HICPAC) 2003; 52(RR 10):[1–42 pp.]. Available from:
  51. 51. U.S. Department of Health & Human Services, Centers for Disease Control and Prevention. Guideline for Hand Hygiene in Health-Care Settings, Recommendations of the Healthcare Infection Control Practices Advisory Committee and the HICPAC/SHEA/APIC/IDSA Hand Hygiene Task Force 2002; 16(51):[1–44 pp.]. Available from:
  52. 52. Organisation Mondiale de la Santé. Le Guide pédagogique pour la sécurité des patients. 2011.
  53. 53. Organisation Mondiale de la Santé. Modèle pour l’auto-évaluation de la promotion et des pratiques d’hygiène des mains au niveau de l’établissement de soins. 2010.
  54. 54. World Health Organization. Prévention des infections nosocomiales 2002.
  55. 55. U.S. Department of Health & Human Services, Center for Disease Control and Prevention. Hand Hygiene in Healthcare Settings [Available from:
  56. 56. World Health Organization. WHO Guidelines on hand hygiene in health care, first global patient safety challenge clean care is safer care 2009. Available from:
  57. 57. Kaplan RS, Anderson SR. Time-driven activity-based costing. Harvard Business Review. 2004; 82(50):131–138. pmid:15559451
  58. 58. Keel G, Savage C, Rafiq M, Mazzocato P. Time-driven activity-based costing in health care: A systematic review of the literature. Health Policy. 2017;121(7):755–63. pmid:28535996
  59. 59. Lopetegui M, Yen P-Y, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking about? J Biomed Inform. 2014;49:292–9. pmid:24607863
  60. 60. Eckmanns T, Bessert J, Behnke M, Gastmeier P, Ruden H. Compliance with antiseptic hand rub use in intensive care units: the Hawthorne effect. Infect Control Hosp Epidemiol. 2006;27(9):931–4. pmid:16941318
  61. 61. Lange T, Kopkow C, Lützner J, Günther K-P, Gravius S, Scharf H-P, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability. BMC Med Res Methodol. 2020;20(1):28. pmid:32041541
  62. 62. Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:37-. pmid:15670334
  63. 63. Junger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative medicine. 2017;31(8):684–706. pmid:28190381
  64. 64. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLOS ONE. 2011;6(6):e20476. pmid:21694759
  65. 65. Knight S, Aggarwal R, Agostini A, Loundou A, Berdah S, Crochet P. Development of an objective assessment tool for total laparoscopic hysterectomy: A Delphi method among experts and evaluation on a virtual reality simulator. PLoS One. 2018;13(1):e0190580. pmid:29293635
  66. 66. Vogel C, Zwolinsky S, Griffiths C, Hobbs M, Henderson E, Wilkins E. A Delphi study to build consensus on the definition and use of big data in obesity research. International Journal of Obesity. 2019;43(12):2573–86. pmid:30655580
  67. 67. Havers SM, Martin E, Wilson A, Hall L. Implementation of government-directed policy in the hospital setting: a modified Delphi study. Health Research Policy and Systems. 2019;17(1):91. pmid:31775771
  68. 68. Turoff M. The Policy Delphi. (Eds.) HLMT, editor2002.
  69. 69. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in nursing & health. 2007;30(4):459–67.
  70. 70. Birko S, Dove ES, Ozdemir V. Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation. PLoS One. 2015;10(8):e0135162. pmid:26270647
  71. 71. Njuangang S, Liyanage C, Akintoye A. Application of the Delphi technique in healthcare maintenance. International Journal of Health Care Quality Assurance. 2017;30(8):737–54. pmid:28958204
  72. 72. Kai Zheng, Guo Michael H, Hanauer David A. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc. 2011;18:704–10. pmid:21527407
  73. 73. Ogunfiditimi F., Takis I., Paige V. J., Wyman J. F., Marlow E. Assessing the Productivity of Advanced Practice Providers Using a Time and Motion Study. Journal of healthcare management. 2013;58(3):173–85.