Figures
Abstract
Premature birth and neonatal mortality are significant global health challenges, with 15 million premature births annually and an estimated 2.5 million neonatal deaths. Approximately 90% of preterm births occur in low/middle income countries, particularly within the global regions of sub-Saharan Africa and South Asia. Neonatal hypothermia is a common and significant cause of morbidity and mortality among premature and low birth weight infants, particularly in low/middle-income countries where rates of premature delivery are high, and access to health workers, medical commodities, and other resources is limited. Kangaroo Mother Care/Skin-to-Skin care has been shown to significantly reduce the incidence of neonatal hypothermia and improve survival rates among premature infants, but there are significant barriers to its implementation, especially in low/middle-income countries (LMICs). The paper proposes the use of a multidisciplinary approach to develop an integrated mHealth solution to overcome the barriers and challenges to the implementation of Kangaroo Mother Care/Skin-to-skin care (KMC/STS) in LMICs. The innovation is an integrated mHealth platform that features a wearable biomedical device (NeoWarm) and an Android-based mobile application (NeoRoo) with customized user interfaces that are targeted specifically to parents/family stakeholders and healthcare providers, respectively. This publication describes the iterative, human-centered design and participatory development of a high-fidelity prototype of the NeoRoo mobile application. The aim of this study was to design and develop an initial (“A”) version of the Android-based NeoRoo mobile app specifically to support the use case of KMC/STS in health facilities in Kenya. Key functions and features are highlighted. The proposed solution leverages the promise of digital health to overcome identified barriers and challenges to the implementation of KMC/STS in LMICs and aims to equip parents and healthcare providers of prematurely born infants with the tools and resources needed to improve the care provided to premature and low birthweight babies. It is hoped that, when implemented and scaled as part of a thoughtful, strategic, cross-disciplinary approach to reduction of global rates of neonatal mortality, NeoRoo will prove to be a useful tool within the toolkit of parents, health workers, and program implementors.
Author summary
We aim to leverage the promise of digital health to overcome barriers and challenges to neonatal care, particularly thermal care, vital signs monitoring, and Kangaroo Mother Care/Skin-to-skin care (KMC/STS), in low/middle-income countries. Our multidisciplinary team has developed an integrated mHealth platform, composed of a wearable biomedical device called NeoWarm, which is a self-warming swaddling pouch plus baby carrier with sensors for automated vital signs monitoring, combined with a mobile app called NeoRoo. NeoRoo is a permission-based app with two user interfaces, customized for parents/family stakeholders and healthcare providers of premature and low birthweight babies. The app provides information, support, and resources designed to equip and empower adult stakeholders of prematurely born babies to more effectively adopt and implement KMC/STS, an evidence-based neonatal intervention that has been shown to significantly reduce the incidence of neonatal hypothermia and improve survival among premature infants. Using a human-centered and participatory design approach, we performed iterative development efforts for the open-source NeoRoo app from August 2020 to February 2021. This resulted in a high-fidelity prototype with features and functions for automated vital signs monitoring, communication between parents and health workers, shared goal-setting and tracking for key KMC/STS metrics, and educational resources.
Citation: Bucher SL, Young A, Dolan M, Padmanaban GP, Chandnani K, Purkayastha S (2023) The NeoRoo mobile app: Initial design and prototyping of an Android-based digital health tool to support Kangaroo Mother Care in low/middle-income countries (LMICs). PLOS Digit Health 2(10): e0000216. https://doi.org/10.1371/journal.pdig.0000216
Editor: Haleh Ayatollahi, Iran University of Medical Sciences, IRAN (ISLAMIC REPUBLIC OF)
Received: March 1, 2023; Accepted: August 12, 2023; Published: October 25, 2023
Copyright: © 2023 Bucher et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Our code is available on GitLab (https://gitlab.com/librehealth/incubating-projects/mhbs).
Funding: This project was funded with support from the Indiana Clinical and Translational Sciences Institute, through an IU Center for Global Health Equity/Indiana CTSI Reciprocal Innovation Demonstration Grant to Dr. Sherri Bucher (Indiana University School of Medicine) and Dr. Jacqueline Linnes (Purdue University Weldon School of Biomedical Engineering). Indiana CTSI is funded in part by Award Number UL1TR002529 from the National Institutes of Health, National Center for Advancing Translational Sciences, Clinical and Translational Sciences Award. Open-source manuscript publication costs were supported by the IU School of Medicine Department of Pediatrics, Division of Neonatal-Perinatal Medicine to SLB. The funders had no role in study design, data collection, and analysis, decision to publish, or preparation of the manuscript.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: Dr. Sherri Bucher has been awarded intellectual property protection for invention of the NeoWarm biomedical device. This includes US patent US10390630B2, Nigeria NG/PT/C/2018/2802 and ARIPO patent PT/C/2018/2802. The authors declare no other potential conflicts of interest with respect to research, authorship, financial relationships, and/or publication of the article.
Introduction
High rates of premature birth (defined as delivery prior to 37 weeks gestation), and neonatal mortality (NMR; defined as the death of liveborn infants from the day of delivery to 28 days postnatal), are urgent global challenges. Annually, there are an estimated 2.5 million neonatal deaths. Fifteen million babies are born prematurely every year, and complications from prematurity are the leading cause of death among children under the age of five years. [1] Despite significant global progress over the past 30 years in reducing deaths due to prematurity [2], there remains a disproportionate burden of mortality due to complications of prematurity within low/middle-income countries (LMICs), particularly within Sub-Saharan Africa and Asia. [3,4]
Neonatal hypothermia–the inability of infants to maintain their body temperature above 36.5°C (97.7°F)—is an extremely widespread complication among prematurely born and low birthweight (small) babies [5], and a significant cause of morbidity (illness) and contributing factor to mortality (death) among newborns. [6,7] Globally, it is estimated that around 17 million newborns in LMICs each year suffer from hypothermia. [8] In the sub-Saharan Africa setting, rates of neonatal hypothermia as high as 60–85% have been reported. [9,10] Within high-income settings, prematurely born infants are typically managed within incubators, by highly skilled healthcare providers (HCPs) with years of specialized training in neonatal care. Within LMIC settings, however, lower availability of incubators and radiant warmers [11,12], high patient volumes (especially within newborn units), [13] and a paucity of physicians and nurses [14–16] contribute to unacceptably high rates of neonatal hypothermia and hypothermia-related newborn mortality.
Kangaroo Mother Care/Skin-to-skin care (KMC/STS) is an evidence-based method of newborn care involving skin-to-skin contact between an adult caregiver and infant, which has been shown to significantly reduce the incidence of neonatal hypothermia within LMICs [17], and greatly improve rates of survival among premature infants. [18–24]Challenges and barriers to adopting and implementing KMC/STS have been identified to exist for all the key adult stakeholders within a premature baby’s environment, including among parents. [25–27] and healthcare providers. [28,29]
Affordable digital health solutions that can be adopted at the health facility level might hold promise to improve care for premature babies, including overcoming barriers to adoption of KMC/STS, and strengthening implementation. Over the past decade, mobile phone ownership, including of smart phones, and access to cellular networks has surged. [30–32] Acceptance of, and access to, a wide variety of digital health interventions using mobile devices (mHealth) has skyrocketed among all stakeholders, including healthcare providers. [33–37] Implementation of mHealth and digital tools in LMICs have been demonstrated to have numerous benefits, including improving access to care for patients, [38,39] increasing knowledge retention and competencies among community health workers, [40,41] and improving training outcomes, confidence/satisfaction, and reporting of key indicators among health providers. [35,42–44]
Landscape for newborn care
Within many LMICs, including Kenya, midwives and nurses provide most of the care for mothers and babies; unfortunately, there is a profound global shortage of nurses, particularly those specialized in neonatal care. [13,45] As a result, nurse:newborn patient ratios in some health facilities in Kenya can range from 1 nurse:7–15 newborns, [16] or even exceed 1:25+. [15] By contrast, in resource-rich settings, neonatal intensive care unit (NICU) nurse:patient ratios are typically 1:1 to 1:3. [46] A NICU nurse:patient ratio of 1:4 in the United States has been demonstrated to result in missed nursing care. [47] The nurse:newborn patient ratios observed in Kenya and other LMIC health systems are significantly higher than that recommended within guidelines for staffing ratios from organizations such as the National Association of Neonatal Nurses. [48] Evidence-based neonatal care recommendations from international partners such as WHO indicate that some key vital signs should, ideally, be measured in hospitalized newborns around every 4 hours, around the clock. However, at one large referral hospital in Kenya, it was demonstrated that only 63% of premature babies had at least 1 temperature taken and only 53% had at least one respiratory and heart rate measurement recorded. [49]
Description of prior mobile app development
From 2016 to present, our Indianapolis-based team, in conjunction with partners at Moi University, Kenya and a variety of international stakeholders, have designed and developed five Android apps, purposively integrated with the District Health Information System 2 (DHIS2; [50]) for newborn care education, training, data collection, and clinical decision support, as digital tools to augment the American Academy of Pediatrics’ Helping Babies Survive programs. These mobile Helping Babies Survive powered by DHIS2 apps (mHBS/DHIS2) include digital tools to support the Helping Babies Breathe (HBB) and Essential Care for Every Baby (ECEB) programs. [51]
Three mHBS/DHIS2 apps, to support training and education for neonatal resuscitation (HBB), were recently utilized in a randomized control trial among 274 maternity nurses from 20 healthcare facilities in Kenya and Nigeria, and were found to be feasible, acceptable, and exceptionally robust. [52,53] An award-winning clinical decision support app developed by our team, ECEB/mHBS, provides comprehensive support for healthcare providers in LMICs to deliver evidence-based essential newborn care interventions from birth through 24 hours after delivery. [54,55] We have also built a digital health tool, “NeoLinkID,” that is designed to support birth registration and secure portability of mother-baby health records in LMICs. [56]
NeoRoo mobile applications expand the mHBS/DHIS2 suite to equip healthcare workers and parents of premature/small babies with digital tools to support an integrated continuum of support for KMC/STS. With NeoRoo, we seek to alleviate barriers to KMC/STS, a proven method of reducing mortality in premature infants, and empower parental caregivers, families, and healthcare workers with education and tools to deliver high-quality, evidence-based care.
We propose to leverage the promise of digital health to overcome known barriers, challenges, and gaps in the care of premature/small babies and to the implementation of facility-based KMC/STS in LMICs. Our multidisciplinary team has developed a suite of innovations to provide automated thermal support and vital signs monitoring for premature/low birthweight infants while they are engaged in KMC/STS with adult caregivers. Our innovation is an integrated mHealth platform that features a wearable, sensor-enabled, self-warming, biomedical device (swaddling baby carrier) with automated vital signs monitoring for KMC/STS adult-baby dyads (“NeoWarm”; [57,58]) and connected Android-based mobile applications which are customized, respectively, for parents/family stakeholders and healthcare providers (“NeoRoo”).
Within the scope of this paper, we describe the human-centered design and iterative development of high-fidelity prototypes of an initial (“A”) version of the NeoRoo mobile application and highlight key features and functions that have resulted from these efforts. This paper describes iterative app design and high-fidelity prototype development activities from August 2020 –February 2021.
Materials and methods
Ethics approval
This study was determined to be “Non-Human Subjects Research” (#15181) by the Indiana University Institutional Review Board based on the following criteria: (1) neither the biomedical device (NeoWarm) nor the app (NeoRoo) are currently being studied according to the FDA’s definition; (2) this is not a safety/efficacy study; (3) this was development of a mobile app prototype that would not be tested in the clinical setting within the current study; (4) the study was conducted purely for product design and development purposes. Because this study was determined by the IU IRB to be “Non-Human Subjects Research,” formal consent was not obtained.
Study setting
The use case for the NeoRoo app is to support adult caregivers (parents/families and healthcare providers) of premature or small babies within KMC/STS wards in Kenyan health facilities to provide evidence-based neonatal care more effectively.
Information, communication, and technology (ICT) ecosystem in Kenya.
Kenya, located in East Africa, is ideally suited for digital health and mHealth solution studies. It has a very robust ICT ecosystem, with governance and policy development lead by the Ministry of Information, Communications, and Technology and the ICT Authority Kenya. [59] Kenya boasts a 70% internet penetration rate [60] and has been dubbed “The Silicon Savannah,” [61] because it is a global leader in mHealth and digital innovation. [60] Of particular interest, one of the stated priorities of The Silicon Savannah is to support rapid acquisition toward 2030 Sustainable Development Goals, including reduction of neonatal mortality. Thus, the overall goals of the NeoRoo initiative, reduction of morbidity and mortality among prematurely born and low birthweight babies, also overlaps with the stated purposes of national partners in Kenya.
IU-Kenya Global Health Equity Partnership.
The NeoRoo app design and development efforts are performed within the context of a long-term research and educational partnership between Indiana University School of Medicine, in Indianapolis, Indiana, and Moi University College of Health Sciences Eldoret, Kenya. From 1989 to the present, the Indiana University School of Medicine and Moi University College of Health Sciences have participated in the IU-Kenya Partnership. [62–66] The IU-Kenya Partnership also supports the Academic Model Providing Access to Healthcare (AMPATH), which, with USAID support, serves a population of 13 million persons at over 300 Ministry of Health facilities across a 17-county catchment area. Thus, for purposes of iterative participatory design and development of mHealth innovations, the NeoRoo team has access to a wide variety of stakeholders and potential end-users across all levels of the health system and health service delivery continuum within Kenya.
Study design
Overall methodological approach.
To develop mHealth and biomedical device innovations, our collaborative team utilizes a combined human/User-centered [67] and participatory design approach, [68–70] and incorporates principles of Design Thinking. [71] These activities, in turn, are firmly nested within a comprehensive, multi-faceted, integrated effort to improve the adoption, uptake, and delivery of proven, evidence-based interventions for premature/small babies. We employ iterative, agile development techniques. [72] Across our iterative design and development efforts, we utilize best practices in regards to user interface (UI) and user experience (UX) design [see also, Table 3 in Horsky et al.,2012 [73]], and, in close collaboration with our international partners, employ successive cycles of participatory design, underpinned by heuristic evaluation, to ensure that the digital tools and platforms we build are feasible, acceptable, and effective, within the settings where they are deployed (Fig 1).
We utilize a variety of evaluation and analysis methods, as further described in the text.
Data security and privacy protections are paramount; the mHealth apps that we build are GDPR compliant. We build our mHealth solutions to maximize the potential for wide adoption and sustainable scale-up; as such, we align our efforts with those currently on-going among key global partners across various landscapes and ecosystems, such as the World Health Organization’s Every Newborn Action Plan (ENAP) metrics group and WHO Quality, Equity, Dignity Network, for the capture of indicators regarding KMC/STS, quality-of-care, and newborn mortality and morbidity. [74] Our mHealth solutions are built to align with the 9 Principles for Digital Development, endorsed by a wide variety of partners within the international development and global health space. [75] Our mobile app development efforts are incubated within the open-source LibreHealth collaborative community, according to FLOSS (Free/Libre and Open-Source Software) development principles. We partner hand-in-glove with LMIC partners, to ensure that we “Design for the User,” “Understand the Existing Ecosystem,” and “Be Collaborative” (Principles for Digital Development 1, 2, 9).
Description of the connected biomedical device.
A complete description of the biomedical device is provided in Bluhm et al., 2020, [57] and within the US 1039063 B2 patent application, Infant thermoregulation and monitoring support system. [76] Briefly, the biomedical device is a carrier (worn by an adult) and integrated self-warming, sensor-enabled swaddling pouch with 4 flaps (worn by the premature baby). When an adult caregiver and infant engage in KMC/STS, the flaps open to allow for skin-to-skin contact. When the dyad is not engaged in KMC/STS, if the adult caregiver wishes to lay the baby on a cot or other surface (“stand-alone” mode), the flaps of the swaddling pouch snap closed to prevent thermal loss in the newborn via conduction and convection. Regardless of the mode of use (i.e., whether during “KMC/STS” or “stand-alone” mode), sensors in the device provide continuous vital signs monitoring of 4 key physiological parameters in the newborn (body temperature, respiration, heart rate, and blood oxygen saturation). There are also visual and audio alarms integrated within the biomedical device, to alert caregivers, for example, when a baby’s body temperature drops too low. Bluetooth low energy technology is used to transmit information from the biomedical device to Android devices that are loaded with the NeoRoo mobile app. [57]
Data collection and analysis
Composition of the app development team.
The NeoRoo design and development team, comprised of Faculty, students, and technical advisors, was highly interdisciplinary, representing a broad range of collective expertise across continents (North America; sub-Saharan Africa; India) and scientific and technological domains, including: health informatics, human-computer interaction, UI/UX development, clinical care (pediatrics; neonatology; midwifery), biomedical engineering, public health, maternal-newborn-child (MNCH) health, monitoring and evaluation, and implementation science. The app design and development activities reported in this paper include: (1) user requirements analysis; (2) development of information architecture (whiteboarding); (3) user interface and user experience design (wireframes); (4) iterative development of high-fidelity prototypes (2 cycles).
User requirements analysis.
We utilized 3 primary methods to ascertain key pain points within the existing cascade for neonatal thermal care and KMC/STS in LMICs and generate an initial list of desired user requirements (Box 1). First, a sub-set of co-authors (SLB; AY; MD) conducted a thorough literature review of both published (PubMed) and gray literature (Google Scholar) to obtain background information, on behalf of the larger team, regarding the existing landscape for KMC/STS, thermal care/neonatal hypothermia, and existing digital health applications for newborn care. Particular focus was directed toward obtaining and collating published literature regarding barriers, challenges, gaps, facilitators, and enablers for KMC/STS in LMICs, with specific emphasis on global regions with the highest neonatal mortality rates, namely, sub-Saharan Africa (especially Kenya) and Asia.
Box 1. NeoRoo user design requirements
- Off-line functionality, combined with automated syncing when app is on-line
- Locally adaptable & customizable
- Ability to integrate with OpenMRS and DHIS2
- Automated, “at a glance” vital signs monitoring
- Augmented decision support
- “First, do no harm:” align app content, functions, and features with existing international, national, and local guidelines and recommendations for facility-based KMC/STS and care of premature/small babies
- Scaffold improved information exchange and shared goal-setting between parents and HCPs
- Create space for potential task-sharing (between various cadres of HCPs; between HCPs and family stakeholders)
- Reduce cognitive burden
- Another useful tool in the toolkit of care for some premature or small babies, as determined by local end-users and stakeholders – not an attempt at a “silver bullet” solution
- One permission-based app, 2 targeted user interfaces
Second, our team drew upon our collective experience in global health and health informatics implementation, and prior mHealth development efforts, to inform the user requirements analysis. Advisors also included US-based neonatologists, Kenyan nurses, nurse-midwives, physicians, and public health experts. As previously described, NeoRoo is part of a suite of mobile newborn care apps that, from 2016 to the present, have collectively undergone extensive contextual analysis, human-centered design, heuristic evaluation, and randomized control testing. [51–56,77] Thus, lessons learned from this prior work also directly informed design and development efforts for the NeoRoo mobile application, an approach that adheres to the “Reuse and improve” and “Be Data Driven” Principles for Digital Development. [75]
The final method used to perform user requirements analysis was to assess results from a separate feasibility, acceptability, and user-design feedback study that was conducted independently for the NeoWarm biomedical device (IU IRB #1602698245) Those findings indirectly informed the NeoRoo team’s app development efforts by providing information confirming some of the challenges and pain points specifically faced by Kenyan adult stakeholders of premature babies. For that qualitative study (manuscript in preparation), parents, healthcare providers, and community stakeholders of small/premature babies reported that, similar to findings described in other published studies from LMIC settings, the barriers, gaps, and challenges to KMC/STS and care for premature/small babies specific to the Kenyan setting span multiple implementation domains, including sociocultural expectations, perceived gender roles, [78] lack of resources, [79] and worries about safety. [28,80,81]
Development of NeoRoo information architecture.
After determining initial user requirements, the NeoRoo design and development team engaged in whiteboarding to develop an information architecture schema for the NeoRoo mobile application (Fig 2). The most important desired overall functions of the NeoRoo app are: (a) provide users with essential, up-to-date information regarding the baby’s health and well-being, primarily via real-time vital signs monitoring and automated alerts, and (b) enable the users to utilize the app to act upon this information, as needed. In addition, we also wished for the app to serve as a tool to scaffold targeted information sharing around key evidence-based educational messages for essential newborn care and care of small/premature babies. It was hypothesized that the most effective architecture would be one that allowed for targeted customization of information dissemination to two equally important stakeholders within the premature baby’s ecosystem–parents and healthcare providers.
An important assumption that underpins the design of NeoRoo’s information architecture is that, unlike educational or quality improvement apps, for which “behavior change” might be a desired direct outcome, the NeoRoo approach is that the primary end-users of the apps are already highly motivated and incentivized to engage in activities that maintain and improve the health and well-being of premature babies within their care. Put simply, none of the key stakeholders within a small baby’s environment (parents; healthcare providers; family members; larger communities) wish for the baby to fall ill, or to die. [82] Rather than targeting a need for “behavior change,” then, we hypothesize that the barriers and challenges to the provision of evidence-based newborn care interventions, including KMC/STS within health facilities in LMICs, have less to do with a lack of motivation among families and HCPs to keep babies healthy, and more to do with a paucity of knowledge, empowerment, agency, skills, or access to key tools or resources. The NeoRoo information architecture is built to reflect this hypothesis. Thus, our default position during the whiteboarding process was that adult caregivers of newborns are already highly motivated and incentivized to engage in activities that will keep babies alive and well. The primary purpose of the connected NeoWarm device + NeoRoo app mHealth platform, then, is to collect pertinent information, in real time, about the baby’s health status (e.g., low or high body temperature; apneic episodes), accurately convey this information to parents and healthcare providers in a timely fashion, via color-coded iconography and alerts in the user interface, and utilize the apps’ architecture to equip and empower adult stakeholders to act upon this information, as needed (e.g., parents and healthcare providers can message one another; healthcare providers can utilize the information to ascertain relative acuity of the newborns under their care).
The app is designed with one permission-based access point to achieve this type of information architecture. However, different user interfaces are available depending on what type of stakeholder (parent/family member or HCP) is using the app. In some instances, such as in vital signs monitoring, information is presented to parent/family end-users in a simpler, more straightforward manner. In contrast, the information viewed by HCPs is more detailed, to support clinical decision making. In other instances, the same information is presented identically to both user groups to scaffold shared goal-setting (e.g., skin-to-skin activity module), or to maintain the fidelity of key evidence-based educational and training messages across all stakeholder groups.
Wireframes
Upon completing the whiteboarding activities and developing the NeoRoo information architecture, the team generated wireframes to visualize an end-to-end workflow for the NeoRoo Family and NeoRoo Healthcare Provider apps. These wireframes served as the basis for the first iteration mock-up of the NeoRoo Family and NeoRoo HCP high-fidelity prototype.
Iterative prototype development
Figma was utilized to conduct 2 iterations of development for NeoRoo high-fidelity prototypes. During these iterative cycles, we performed internal reviews with subject matter experts and obtained feedback from interaction designers and researchers.
Iteration 1: Internal reviews with subject matter experts
As previously described, the NeoRoo app design and development team is composed of highly experienced content and subject matter experts. We reviewed the wireframes internally and renamed labels, to ensure clinical accuracy, such as in reporting vital signs monitoring. At this stage, revisions were also made, based on feedback from subject matter experts, to more effectively align the workflow of the apps with the processes and procedures by which daily tasks are typically conducted in a Kenyan KMC/STS ward (NeoRoo Family) and during nursing shifts (NeoRoo Healthcare Provider). A search bar was added to the NeoRoo HCP app, to allow nurses and physicians to search more efficiently through lists of babies under their care. For graphics that demonstrated trends, feedback was received to revise labeling in both the X and Y axes, for improved clarity and accuracy.
Iteration 2: Feedback from interaction designers and researchers
After seeking functional feedback from internal subject matter experts, we shared the NeoRoo app prototypes with industry-based design professionals and sought their expertise from a visual standpoint. The primary feedback input received from interaction designers during this iterative cycle was in regards to the background color (made it difficult to read the text), the bottom navigation bar (was “clunky” and “heavy,” needed to be streamlined), and depiction of information regarding infant body temperature (placement of the toggle switch for local adaptation by users to the preferred temperature scale of “Celsius” or “Fahrenheit” needed to be shifted).
Results
Throughout an intensive 7-month iterative design and development phase, we successfully developed a high-fidelity prototype of one permission-based app with two unique user interfaces, each targeted toward specific adult caregivers within a premature/small baby’s care environment. The apps include “NeoRoo Family,” for parents and family stakeholders of small babies in LMICs, and “NeoRoo Healthcare Provider,” for nurses, physicians, and other clinical care staff of premature/small babies.
NeoRoo Logo
The NeoRoo app is purposively designed to be feasible and acceptable to any family stakeholder providing care for a premature baby, not solely the mother. Evidence of user requirements analysis emphasized the importance of maintaining gender-neutral colors and iconography, to improve the potential adoption and uptake of the technologies by male stakeholders and the potential for broader acceptance within communities. Thus, in order to remain gender-neutral, no extra cues were added to the NeoRoo logo. Instead, it was constructed with simple geometrical shapes. The “N” within the NeoRoo logo (Fig 3) employs iconography that mimics an image reminiscent of that which might be found on an electrocardiogram trace. This is a nod to the fact that our connected mHealth platform (NeoWarm + NeoRoo) automatically monitors 4 key newborn vital signs, including heart rate. We also hope that, for health care workers, this symbolism will convey that the app will equip them with digital tools to provide evidence-based healthcare for premature babies.
The selected logo version is depicted with a green tick.
Shared UI and UX features
In the completed NeoRoo version “A” high-fidelity prototype, many UI/UX design choices, described below, are identical between the NeoRoo HCP and NeoRoo Family apps.
App landing page and permission-based access
The NeoRoo Version “A” app landing page has screens for permission-based access to user interfaces customized for parents and for healthcare providers. This feature can be customized to require unique passwords or pin codes, as desired by local implementors.
Iconography
The high-fidelity prototype (“A” version) developed for the NeoRoo app utilizes permission-based access to direct users toward user interface and experience features that are customized for parents/family members and healthcare providers, respectively. As possible, shared iconography is deployed within the UI. Information regarding the continuous monitoring of 4 key neonatal vital signs in premature babies wearing the NeoWarm biomedical device is displayed on the NeoRoo app. The vital signs include body temperature, breathing rate and pauses (e.g., apnea detection), heart rate, and blood oxygen saturation.
Strategic use of color to depict key information
A number of newborn care educational, training, and clinical care curricula that have been developed by international partners utilize color-coding to emphasize key educational concepts. In particular, the American Academy of Pediatrics Helping Babies Survive curricula, and more recently, the interim WHO ENCC courses, creatively employ a “traffic light” color-coding schema to indicate “routine/normal” (green), “caution/warning/needs close monitoring” (yellow) and “danger signs/high risk/emergency” (red). The color-coding schema is a common graphical theme employed across many curricular materials such as facilitator flip charts, action plans, provider guides, and job aids. In developing prior digital health tools within the mHBS/DHIS2 suite, we maintained this color-coding schema across apps. During participatory design feedback and heuristic evaluations of the ECEB/mHBS app, users reported that use of color-coding for particular features and functions facilitated recognition, rather than recall, and reduced cognitive burden, particularly in regards to organizing clinical decision-support information on the user interface. [54,55]
Participants of the user-design sessions for the NeoWarm biomedical device also indicated that color-coded diodes on the strap of the device, which provide information about the baby’s body temperature, are highly desirable. Within the Kenyan setting, there was automatic recognition, among surveyed stakeholders, that a blue LED light indicated “baby too cold,” a green LED light indicated “baby body temperature ok” and a red LED light indicated “baby too hot.” Thus, for NeoRoo, we iterated ways to most effectively augment and display vital signs information and alerts with color-coding. The intent is to provide highly accurate and nearly instantaneous messaging, even to low literacy audiences, regarding the baby’s real-time health status.
Fig 4A and 4B demonstrates successive iterations of the NeoRoo Family app user home page, combining common iconography and strategic use of color. In the first iteration (Fig 4A), color-coded iconography depicting an infant and thermometer indicates the baby is too cold. In a successive iteration (Fig 4B), three types of color-coded rings were displayed, to show the baby’s body temperature status. For infants with body temperature readings that are outside the normal range, the most recent body temperature reading is augmented with color-coded arrows to further indicate if this reading is “too high” or “too low.” The latter iteration also reflects alignment of app UI features with the color-coding schema of the LED diodes on the NeoWarm device itself.
A-B: Iterations, across successive cycles of prototype development, of the user interface design for NeoRoo Family home page, which utilizes combined features of color-coding and iconography to depict the real-time body temperature of a simulated premature baby wearing a NeoWarm biomedical device. 4A is iteration 1, and 4B is iteration 2. Please note that all photos and images of persons depicted in the high-fidelity prototype are stock images from Pexels and Flaticon, used under Creative Commons license (additional information and links in Acknowledgements).
Vital signs trends over time
An intended function of NeoRoo apps is to provide adult caregivers of premature babies with the ability to see vital signs in real-time, as depicted in the previous section, but also to provide them with digital tools to examine trends in vital signs readings over time. We explored various ways to deploy the color-coding schema to assist in clearly transmitting trend information, even to lower literacy audiences. On the Y (vertical) axis, we experimented with various iterations of a stacked color bar, to demonstrate whether particular body temperature readings are within the normal (green) or abnormal (blue; red) ranges. This feature is reminiscent of a thermometer. It aligns with human-computer interaction principles of the mental model law. The intention was to strategically deploy a common cognitive representation that would, in turn, allow users to have a more intuitive understanding of the information visualized within the graph.[83] The trend line is also color-coded, indicating, over time, how often the baby’s body temperature is normal, too hot, or too cold.
Trend line visualizations are augmented by tool tips. When users click on the “i” icon, they are able to view additional information regarding abnormal body temperature fluctuations. This information can be triangulated with data regarding the trends and duration of KMC/STS, which are depicted in similar graphical displays accessed via the “STS” tab in the toolbar.
Shared goal-setting for duration of KMC/STS
In order to encourage shared goal-setting for duration of KMC/STS, we provide a feature, within both the NeoRoo Family and NeoRoo Healthcare Provider interfaces, where users can set, and edit, KMC/STS goals by number of hours and minutes, per day, week, or month. This objective is then represented as a “goal line” on the STS activity tracking graph. Duration and trends, over time, of both STS and non-STS care are automatically plotted within the app.
Aligned educational and training resources
Similar to the other Trainer apps within the mHBS/DHIS2 suite of digital health innovations, the NeoRoo Trainer module allows for linkage with digitized educational training resources from important MNCH partners such as WHO, UNICEF, Save the Children, American Academy of Pediatrics, and Ministries of Health. Within the NeoRoo app, we provide an opportunity for parents and HCPs to develop a small, curated video repository on their Android device. The Global Health Media Project (GHMP) provides access to 226 educational and training videos across a wide range of reproductive health and MNCH topics that are freely downloadable and available in 109 languages. [51] Currently, this on-line library includes 42 videos regarding newborn care, and 27 videos specific to care of small and prematurely born babies, as well as 18 videos regarding breastfeeding.
The NeoRoo app allows end-users to download, store, and watch off-line, around 3 videos at a time. Limiting the video repository within NeoRoo to only a handful of videos at a time is for three primary reasons. First, to ensure that there is not too much storage space taken up on the Android mobile device as a result of accrued videos, which might lead to slow or poor functioning of the device and downloaded apps. Second, because internet connectivity is required to download videos (although, they all play off-line after being loaded on the device), and videos often require a lot of bandwidth, we wanted to reduce the potential frustration, in areas of low internet connectivity, related to slow or poor video download. Third, by allowing for careful curation of a limited, user specific video library, NeoRoo apps adhere to a “just-in-time” concept for educational messaging. Newborn care recommendations can change rapidly over time, as the baby ages, gains weight, and moves through the health service delivery pipeline, from admission to discharge. Thus, different educational content is appropriate for parents and families at various periods. Utilizing the NeoRoo app, HCPs, social workers, or peer mentors can suggest videos and other resources that are the most pertinent resources for each dyad.
In addition to supporting the curation of educational resources related to newborn care and KMC/STS from key partners, the NeoRoo trainer module will also contain a pre-loaded short demo and instructional videos specific to the NeoWarm device. These will be regarding instructional or safety topics such as donning/doffing, safe use, cleaning and disinfection, and troubleshooting technical challenges encountered during use.
NeoRoo Family vs. NeoRoo Healthcare Provider app customizations
Home pages.
The respective home pages of the NeoRoo Family and NeoRoo HCP apps (Fig 5) have been designed to present pertinent information to different adult stakeholders in a targeted fashion. Users of NeoRoo Family are allowed to view only the information about their particular infant (or infants, in the case of multiple-gestation deliveries). They receive instant visual feedback about the health status of their baby, and have the ability, within the app, to send an alert and message to the nurse’s NeoRoo HCP app if they have any concerns. The NeoRoo home page for healthcare providers provides information about any high-priority alerts and parental messages that might require immediate attention. Healthcare providers can generate task lists on the home page to help organize their workflow.
Home pages for NeoRoo Family (left) and NeoRoo Healthcare Provider (right) apps. Key features of the respectively targeted UI designs are highlighted.
Skin-to-Skin activity
Fig 6 shows iterations of the NeoRoo user interface depicting information regarding trends and total duration of hours spent by an adult-baby dyad in KMC/STS. NeoRoo Family app users are only able to view information about their own baby; NeoRoo HCP end-users can view information about all babies under their care. As previously mentioned, both apps have a shared feature in which parents and healthcare providers can collaborate to set collective goals for the number of desired hours of daily, weekly, or monthly hours in which the KMC dyad engages in STS care.
Functions and features only available to healthcare providers
List of babies and access to limited clinical history.
Through the NeoRoo Healthcare Provider app, healthcare providers (nurses; physicians) can access lists of all the babies under their care during a shift. In order to facilitate clinical management, they are also able to access a clinical history summary, such as a very brief description of each baby’s birth history (Fig 7).
Discussion
Our research initiative aims to develop feasible, acceptable, affordable, and locally-adaptable health innovations that our partners in LMICs can utilize to more effectively implement known, evidence-based interventions to significantly reduce maternal and neonatal morbidity and mortality. We hypothesize that low-cost, open-source digital health innovation (permission-based, customized Android apps for health workers and families, augmented by clinical-decision support algorithms), coupled with a wearable, sensor-enabled, self-warming, swaddling carrier for KMC/STS adult-baby dyads (“NeoWarm”), can be utilized, as part of a thoughtfully integrated mHealth platform, to improve the ability of overburdened adult caregivers in LMICs to care for premature and low birthweight babies more effectively.
In support of this objective, we utilized human/user-centered design techniques and best practices from human-computer interaction to successfully develop a high-fidelity prototype of one permission-based app with two unique user interfaces, each targeted toward specific adult caregivers within a premature/small baby’s care environment.
NeoRoo is being incubated within a global ecosystem that continues to evolve toward increased acceptance and utilization of mHealth and digital health tools. Over the past decade, mobile phone ownership and access to cellular networks have surged. [30–32] In 2022, there were 7.26. billion mobile phone users; 6.64 of these devices are “smart;” with internet access, data storage, and other computing capabilities. This means that over 91% of the global population owns a mobile phone, and 83% own “smart” devices.[84] Acceptance of, and access to, a wide variety of digital health interventions has skyrocketed among a wide range of stakeholders within the mHealth innovation ecosystem, including healthcare providers. [33–37]
The potential advantages of connected mHealth platforms include increased opportunities for more efficient delivery of healthcare services both within, and outside the health facility setting. [32,85,86] The increased penetration of mobile devices among a wide variety of potential end-users and stakeholders, combined with rapid innovation in connected devices, paired with mHealth, eHealth, or telehealth solutions, brings additional opportunities and challenges into this landscape. [87–92] Connected device + app solutions also offer intriguing opportunities for the application of machine learning and artificial intelligence algorithms [93–95] to potentially augment the accuracy and speed of diagnosis, [96] clinical decision -making, [97,98] and secure linkage to portable digital health records via unique user IDs or biometrics. [99]
Challenges include those related to the management of information collected within connected device and diagnostic platforms, and issues around data safety and security, privacy, and management. [88,89,100–103] In addition, an important limitation of our approach is that mobile device access and connectivity still remains out-of-reach for some economically disadvantaged populations, especially in rural settings. To address this particular challenge, the NeoWarm + NeoRoo integrated platform is currently proposed for use within health facility settings, where hospital-purchased hardware bundles (NeoWarm biomedical device + Android smart phones or tablets pre-loaded with the NeoRoo apps) could be provided to KMC/STS dyads during hospitalization and re-used for subsequent patients. Another limitation is that, as described in this paper, the UI/UX choices may not be suitable for low-literacy audiences. We attempted to address this challenge through the development of color-coded iconography, but some features and functions, such as trend graphs, might not be useful for family stakeholders. These are key aspects that we are currently exploring, via participatory design interviews with key stakeholders.
On-going work and future directions
Currently, our team is performing additional participatory design testing and heuristic evaluation for the “A” version of the NeoRoo app described in this paper. Simultaneously, we have also developed a “B” version of the NeoRoo app. We plan to perform a formal “A/B” split test within Kenyan health facilities in Western Kenya. Eventually, our goal is to conduct a large-scale, global, multisite clinical effectiveness trial, to explore whether, and how, mHealth tools and integrated technologies, such as the NeoWarm biomedical device + NeoRoo mobile apps, might augment the ability of health workers to perform routine monitoring of key vital signs in premature/small babies, and improve compliance, among KMC/STS dyads, to the recommended number of hours skin-to-skin care each day. We seek to understand the manner by which machine learning algorithms, integrated within NeoRoo can assist overburdened healthcare workers in LMICs to more effectively prioritize the healthcare needs of premature babies under their care, and manage workflows in more efficiently. In short, our overarching goal is to develop integrated mHealth solutions that will help to reduce the “know-do” gaps [104] that are often observed when evidence-based interventions, such as KMC/STS, move from the laboratory or tightly controlled research setting, into complex “real world” health ecosystems.
Conclusions
Utilizing best practices for human-centered, open-source digital health design and mobile app development, our multidisciplinary team used successive iterative cycles to produce high-fidelity mobile app prototypes to equip families and HCPs of premature babies in LMICs to provide evidence-based newborn care, including KMC/STS. On-going participatory design interviews and heuristic usability evaluation will provide key information for additional design iterations.
Acknowledgments
We are grateful to our Kenyan-based collaborators and technical advisors at Moi University, Moi Teaching and Referral Hospital, and Alupe University. In particular, we are grateful for the support of Professor Fabian Esamai. We acknowledge the steadfast support of the IU Center for Global Health Equity in our efforts to develop reciprocal innovations to improve the health and well-being of mothers, babies, and communities around the world. Thank you to Dr. Osayame Ekhaguere for serving as the Neonatology technical advisor on this project, and to biomedical engineering colleagues at IUPUI School of Engineering and Technology and Purdue University Weldon School of Biomedical Engineering for technical assistance related to the NeoWarm biomedical device.
This manuscript was developed in partial fulfillment of requirements for Allison Young and Madison Dolan to complete the Scholarly Concentration in Public Health Certificate program, jointly offered by the IU School of Medicine and IUPUI Richard M. Fairbanks School of Public Health. AY and MD contributed equally to this effort. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Images depicted within Figs 4–7 are stock images from Pexels and Flaticon, utilized under Creative Common License. These include (Pexels): Mother - https://www.pexels.com/photo/a-woman-carrying-her-baby-11369185/; Nurse - https://www.pexels.com/photo/woman-in-medical-frontliner-uniform-6098047/; Babies
https://www.pexels.com/photo/baby-wearing-red-dress-161593/; https://www.pexels.com/photo/white-blanket-on-top-of-an-adorable-baby-6061680/ and Flaticon: Infant - https://www.flaticon.com
References
- 1. Perin J, Mulick A, Yeung D, Villavicencio F, Lopez G, Strong KL, et al. Global, regional, and national causes of under-5 mortality in 2000–19: an updated systematic analysis with implications for the Sustainable Development Goals. Lancet Child Adolesc Health. 2022;6(2):106–15. Epub 2021/11/21. pmid:34800370; PubMed Central PMCID: PMC8786667.
- 2. Bello M, Pius S, Ibrahim BA, Ibrahim HA. Preterm Babies: Management and Challenges Associated with Survival in a Resource-Limited Setting. West Afr J Med. 2020;37(4):402–6. Epub 2020/08/25. pmid:32835403.
- 3. Lunze K, Bloom DE, Jamison DT, Hamer DH. The global burden of neonatal hypothermia: systematic review of a major challenge for newborn survival. BMC Med. 2013;11:24. Epub 2013/02/02. pmid:23369256; PubMed Central PMCID: PMC3606398.
- 4. Walani SR. Global burden of preterm birth. Int J Gynaecol Obstet. 2020;150(1):31–3. Epub 2020/06/12. pmid:32524596.
- 5. Nyandiko WM, Kiptoon P, Lubuya FA. Neonatal hypothermia and adherence to World Health Organisation thermal care guidelines among newborns at Moi Teaching and Referral Hospital, Kenya. PLOS ONE. 2021;16(3):e0248838. pmid:33755686
- 6. Amadi HO, Olateju EK, Alabi P, Kawuwa MB, Ibadin MO, Osibogun AO. Neonatal hyperthermia and thermal stress in low- and middle-income countries: a hidden cause of death in extremely low-birthweight neonates. Paediatr Int Child Health. 2015;35(3):273–81. Epub 2015/05/06. pmid:25936414.
- 7. Onalo R. Neonatal hypothermia in sub-Saharan Africa: a review. Niger J Clin Pract. 2013;16(2):129–38. Epub 2013/04/09. pmid:23563449.
- 8. Mathur NB, Krishnamurthy S, Mishra TK. Evaluation of WHO Classification of Hypothermia in Sick Extramural Neonates as Predictor of Fatality. Journal of Tropical Pediatrics. 2005;51(6):341–5. pmid:16014762
- 9. Byaruhanga R, Bergstrom A, Okong P. Neonatal Hypothermia in Uganda: Prevalence and Risk Factors. Journal of Tropical Pediatrics. 2005;51(4):212–5. pmid:15917265
- 10. Ogunlesi TA, Ogunfowora OB, Adekanmbi FA, Fetuga BM, Olanrewaju DM. Point-of-admission hypothermia among high-risk Nigerian newborns. BMC Pediatrics. 2008;8(1):40. pmid:18837973
- 11. Brambilla Pisoni G, Gaulis C, Suter S, Rochat MA, Makohliso S, Roth-Kleiner M, et al. Ending Neonatal Deaths From Hypothermia in Sub-Saharan Africa: Call for Essential Technologies Tailored to the Context. Front Public Health. 2022;10:851739. Epub 2022/04/26. pmid:35462801; PubMed Central PMCID: PMC9022947.
- 12. Amadi HO, Mokuolu OA, Adimora GN, Pam SD, Etawo US, Ohadugha CO, Adesiyun OO. Digitally recycled incubators: better economic alternatives to modern systems in low-income countries. Ann Trop Paediatr. 2007;27(3):207–14. Epub 2007/08/25. pmid:17716449.
- 13. English M, Gathara D, Nzinga J, Kumar P, Were F, Warfa O, et al. Lessons from a Health Policy and Systems Research programme exploring the quality and coverage of newborn care in Kenya. BMJ Glob Health. 2020;5(1):e001937. Epub 2020/03/07. pmid:32133169; PubMed Central PMCID: PMC7042598.
- 14. English M, Strachan B, Esamai F, Ngwiri T, Warfa O, Mburugu P, et al. The paediatrician workforce and its role in addressing neonatal, child and adolescent healthcare in Kenya. Arch Dis Child. 2020;105(10):927–31. Epub 2020/06/20. pmid:32554508; PubMed Central PMCID: PMC7513261.
- 15. Gathara D, Serem G, Murphy GAV, Obengo A, Tallam E, Jackson D, et al. Missed nursing care in newborn units: a cross-sectional direct observational study. BMJ Qual Saf. 2020;29(1):19–30. Epub 2019/06/07. pmid:31171710; PubMed Central PMCID: PMC6923939.
- 16. Murphy GAV, Gathara D, Abuya N, Mwachiro J, Ochola S, Ayisi R, English M. What capacity exists to provide essential inpatient care to small and sick newborns in a high mortality urban setting?—A cross-sectional study in Nairobi City County, Kenya. PLoS One. 2018;13(4):e0196585. Epub 2018/04/28. pmid:29702700; PubMed Central PMCID: PMC5922525.
- 17. Ibe OE, Austin T, Sullivan K, Fabanwo O, Disu E, Costello AM. A comparison of kangaroo mother care and conventional incubator care for thermal regulation of infants < 2000 g in Nigeria using continuous ambulatory temperature monitoring. Ann Trop Paediatr. 2004;24(3):245–51. Epub 2004/10/14. pmid:15479575.
- 18. Montealegre-Pomar A, Bohorquez A, Charpak N. Systematic review and meta-analysis suggest that Kangaroo position protects against apnoea of prematurity. Acta Paediatr. 2020;109(7):1310–6. Epub 2020/01/10. pmid:31916621.
- 19. Bilal SM, Tadele H, Abebo TA, Tadesse BT, Muleta M, F WG, et al. Barriers for kangaroo mother care (KMC) acceptance, and practices in southern Ethiopia: a model for scaling up uptake and adherence using qualitative study. BMC Pregnancy Childbirth. 2021;21(1):25. Epub 2021/01/09. pmid:33413193; PubMed Central PMCID: PMC7789316.
- 20. Cai Q, Chen DQ, Wang H, Zhang Y, Yang R, Xu WL, Xu XF. What influences the implementation of kangaroo mother care? An umbrella review. BMC Pregnancy Childbirth. 2022;22(1):851. Epub 2022/11/20. pmid:36401193; PubMed Central PMCID: PMC9675107.
- 21. Cho YC, Gai A, Diallo BA, Samateh AL, Lawn JE, Martinez-Alvarez M, Brotherton H. Barriers and enablers to kangaroo mother care prior to stability from perspectives of Gambian health workers: A qualitative study. Front Pediatr. 2022;10:966904. Epub 2022/09/13. pmid:36090565; PubMed Central PMCID: PMC9459153.
- 22. Kinshella MW, Salimu S, Chiwaya B, Chikoti F, Chirambo L, Mwaungulu E, et al. "So sometimes, it looks like it’s a neglected ward": Health worker perspectives on implementing kangaroo mother care in southern Malawi. PLoS One. 2020;15(12):e0243770. Epub 2020/12/18. pmid:33332395; PubMed Central PMCID: PMC7746165.
- 23. Hadush MY, Gebremariam DS, Beyene SA, Abay TH, Berhe AH, Zelelew YB, et al. Barriers and Enablers of KMC Implementation in Health Facility and Community of Tigray Region, Northern Ethiopia: Formative Research. Pediatric Health Med Ther. 2022;13:297–307. Epub 2022/09/16. pmid:36106330; PubMed Central PMCID: PMC9467444.
- 24. Blomqvist YT, Frölund L, Rubertsson C, Nyqvist KH. Provision of Kangaroo Mother Care: supportive factors and barriers perceived by parents. Scand J Caring Sci. 2013;27(2):345–53. Epub 2012/07/24. pmid:22816503.
- 25. Anderzén-Carlsson A, Lamy ZC, Tingvall M, Eriksson M. Parental experiences of providing skin-to-skin care to their newborn infant—part 2: a qualitative meta-synthesis. Int J Qual Stud Health Well-being. 2014;9:24907. Epub 2014/10/17. pmid:25319747; PubMed Central PMCID: PMC4197398.
- 26. Lewis TP, Andrews KG, Shenberger E, Betancourt TS, Fink G, Pereira S, McConnell M. Caregiving can be costly: A qualitative study of barriers and facilitators to conducting kangaroo mother care in a US tertiary hospital neonatal intensive care unit. BMC Pregnancy Childbirth. 2019;19(1):227. Epub 2019/07/06. pmid:31272398; PubMed Central PMCID: PMC6610951.
- 27. Artese C, Paterlini G, Mascheroni E, Montirosso R. Barriers and Facilitators to Conducting Kangaroo Mother Care in Italian Neonatal Intensive Care Units. J Pediatr Nurs. 2021;57:e68–e73. Epub 2020/11/16. pmid:33189484.
- 28. Maniago JD, Almazan JU, Albougami AS. Nurses’ Kangaroo Mother Care practice implementation and future challenges: an integrative review. Scand J Caring Sci. 2020;34(2):293–304. Epub 2019/10/28. pmid:31657039.
- 29. Abdulghani N, Edvardsson K, Amir LH. Health care providers’ perception of facilitators and barriers for the practice of skin-to-skin contact in Saudi Arabia: A qualitative study. Midwifery. 2020;81:102577. Epub 2019/11/30. pmid:31783230.
- 30. Jemere AT, Yeneneh YE, Tilahun B, Fritz F, Alemu S, Kebede M. Access to mobile phone and willingness to receive mHealth services among patients with diabetes in Northwest Ethiopia: a cross-sectional study. BMJ Open. 2019;9(1):e021766. Epub 2019/01/27. pmid:30679284; PubMed Central PMCID: PMC6347931.
- 31. Endehabtu B, Weldeab A, Were M, Lester R, Worku A, Tilahun B. Mobile Phone Access and Willingness Among Mothers to Receive a Text-Based mHealth Intervention to Improve Prenatal Care in Northwest Ethiopia: Cross-Sectional Study. JMIR Pediatr Parent. 2018;1(2):e9. Epub 2019/09/14. pmid:31518334; PubMed Central PMCID: PMC6715064.
- 32. Madanian S, Parry DT, Airehrour D, Cherrington M. mHealth and big-data integration: promises for healthcare system in India. BMJ Health Care Inform. 2019;26(1). Epub 2019/09/07. pmid:31488497; PubMed Central PMCID: PMC7062344.
- 33. Noordam AC, Kuepper BM, Stekelenburg J, Milen A. Improvement of maternal health services through the use of mobile phones. Trop Med Int Health. 2011;16(5):622–6. Epub 2011/02/24. pmid:21342374.
- 34. Garner SL, Sudia T, Rachaprolu S. Smart phone accessibility and mHealth use in a limited resource setting. Int J Nurs Pract. 2018;24(1). Epub 2017/11/22. pmid:29159919.
- 35. Källander K, Tibenderana JK, Akpogheneta OJ, Strachan DL, Hill Z, ten Asbroek AH, et al. Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low- and middle-income countries: a review. J Med Internet Res. 2013;15(1):e17. Epub 2013/01/29. pmid:23353680; PubMed Central PMCID: PMC3636306.
- 36. Amoakoh-Coleman M, Borgstein AB, Sondaal SF, Grobbee DE, Miltenburg AS, Verwijs M, et al. Effectiveness of mHealth Interventions Targeting Health Care Workers to Improve Pregnancy Outcomes in Low- and Middle-Income Countries: A Systematic Review. J Med Internet Res. 2016;18(8):e226. Epub 2016/08/21. pmid:27543152; PubMed Central PMCID: PMC5010646.
- 37. Tuti T, Winters N, Edgcombe H, Muinga N, Wanyama C, English M, Paton C. Evaluation of Adaptive Feedback in a Smartphone-Based Game on Health Care Providers’ Learning Gain: Randomized Controlled Trial. J Med Internet Res. 2020;22(7):e17100. Epub 2020/07/07. pmid:32628115; PubMed Central PMCID: PMC7380991.
- 38. Oyeyemi SO, Wynn R. Giving cell phones to pregnant women and improving services may increase primary health facility utilization: a case-control study of a Nigerian project. Reprod Health. 2014;11(1):8. Epub 2014/01/21. pmid:24438150; PubMed Central PMCID: PMC3898403.
- 39. Budrionis A, Wynn R, Marco-Ruiz L, Yigzaw KY, Bergvik S, Oyeyemi SO, Bellika JG. Impact of the Use of Electronic Health Tools on the Psychological and Emotional Well-Being of Electronic Health Service Users (The Seventh Tromsø Study—Part 3): Population-Based Questionnaire Study. J Med Internet Res. 2020;22(3):e13118. Epub 2020/03/07. pmid:32134396; PubMed Central PMCID: PMC7082736.
- 40. Early J, Gonzalez C, Gordon-Dseagu V, Robles-Calderon L. Use of Mobile Health (mHealth) Technologies and Interventions Among Community Health Workers Globally: A Scoping Review. Health Promot Pract. 2019;20(6):805–17. Epub 2019/06/11. pmid:31179777.
- 41. Modi D, Dholakia N, Gopalan R, Venkatraman S, Dave K, Shah S, et al. mHealth intervention "ImTeCHO" to improve delivery of maternal, neonatal, and child care services-A cluster-randomized trial in tribal areas of Gujarat, India. PLoS Med. 2019;16(10):e1002939. Epub 2019/10/28. pmid:31647821; PubMed Central PMCID: PMC6812744 following competing interests: Argusoft India Ltd is the owner of the core IT platform which was used to develop ImTeCHO application. RG and SV are co-investigators for this study and employed by Argusoft India Ltd. SQ is retired medical officer at the WHO. AS is senior scientist at the ICMR; WHO and ICMR funded the trial.
- 42. Thomsen CF, Barrie AMF, Boas IM, Lund S, Sørensen BL, Oljira FG, Tersbøl BP. Health workers’ experiences with the Safe Delivery App in West Wollega Zone, Ethiopia: a qualitative study. Reprod Health. 2019;16(1):50. Epub 2019/05/11. pmid:31072399; PubMed Central PMCID: PMC6506934.
- 43. Lund S, Boas IM, Bedesa T, Fekede W, Nielsen HS, Sørensen BL. Association Between the Safe Delivery App and Quality of Care and Perinatal Survival in Ethiopia: A Randomized Clinical Trial. JAMA Pediatr. 2016;170(8):765–71. Epub 2016/06/21. pmid:27322089.
- 44. Bolan NE, Sthreshley L, Ngoy B, Ledy F, Ntayingi M, Makasy D, et al. mLearning in the Democratic Republic of the Congo: A Mixed-Methods Feasibility and Pilot Cluster Randomized Trial Using the Safe Delivery App. Glob Health Sci Pract. 2018;6(4):693–710. Epub 2018/12/29. pmid:30591577; PubMed Central PMCID: PMC6370362.
- 45. Fitzgerald L, Gathara D, McKnight J, Nzinga J, English M. Are health care assistants part of the long-term solution to the nursing workforce deficit in Kenya? Hum Resour Health. 2020;18(1):79. Epub 2020/10/22. pmid:33081790; PubMed Central PMCID: PMC7576771.
- 46. Guidelines for Professional Registered Nurse Staffing for Perinatal Units Executive Summary. Nursing for Women’s Health. 2011;15(1):81–4. pmid:21332964
- 47. Tubbs-Cooley HL, Mara CA, Carle AC, Mark BA, Pickler RH. Association of Nurse Workload With Missed Nursing Care in the Neonatal Intensive Care Unit. JAMA Pediatrics. 2019;173(1):44–51. pmid:30419138
- 48.
Kopischke K, Armstrong L, Stropyra Deeley A, Gilhousen K, Rogowski J. RN Staffing in the NICU. Chicago, IL: National Association of Neonatal Nurses, Directors Bo; 2021 September, 2021. Report No.
- 49. Mitchell EJ, Qureshi ZP, Were F, Daniels J, Gwako G, Osoti A, et al. Feasibility of using an Early Warning Score for preterm or low birthweight infants in a low-resource setting: results of a mixed-methods study at a national referral hospital in Kenya. BMJ Open. 2020;10(10):e039061. Epub 2020/10/30. pmid:33115899; PubMed Central PMCID: PMC7594348.
- 50. Byrne E, Sæbø JI. Routine use of DHIS2 data: a scoping review. BMC Health Serv Res. 2022;22(1):1234. Epub 2022/10/07. pmid:36203141; PubMed Central PMCID: PMC9535952.
- 51. Bucher SL, Cardellichio P, Muinga N, Patterson JK, Thukral A, Deorari AK, et al. Digital Health Innovations, Tools, and Resources to Support Helping Babies Survive Programs. Pediatrics. 2020;146(Suppl 2):S165–s82. Epub 2020/10/03. pmid:33004639.
- 52. Ezenwa BN, Umoren R, Fajolu IB, Hippe DS, Bucher S, Purkayastha S, et al. Using Mobile Virtual Reality Simulation to Prepare for In-Person Helping Babies Breathe Training: Secondary Analysis of a Randomized Controlled Trial (the eHBB/mHBS Trial). JMIR Med Educ. 2022;8(3):e37297. pmid:36094807
- 53. Umoren R, Bucher S, Hippe DS, Ezenwa BN, Fajolu IB, Okwako FM, et al. eHBB: a randomised controlled trial of virtual reality or video for neonatal resuscitation refresher training in healthcare workers in resource-scarce settings. BMJ Open. 2021;11(8):e048506. Epub 2021/08/27. pmid:34433598; PubMed Central PMCID: PMC8390148.
- 54.
Nuthakki S, Bucher S, Purkayastha S, editors. The Development and Usability Testing of a Decision Support Mobile App for the Essential Care for Every Baby (ECEB) Program2019; Cham: Springer International Publishing.
- 55.
Rajapuri AS, Ravindran R, Horan K, Bucher S, Purkayastha S, editors. Essential Care for Every Baby: Neonatal Clinical Decision Support Tool2020; Cham: Springer International Publishing.
- 56. Freytsis M, Barclay I, Radha SK, Czajka A, Siwo GH, Taylor I, Bucher S. Development of a Mobile, Self-Sovereign Identity Approach for Facility Birth Registration in Kenya. Frontiers in Blockchain. 2021;4.
- 57. Bluhm NDP, Hoilett OS, Walters BD, Pickering AS, Bucher SL, Linnes JC. NeoWarm: Kangaroo Mother Care with Continuous Temperature Tracking and Heating. Annu Int Conf IEEE Eng Med Biol Soc. 2020;2020:4514–7. Epub 2020/10/07. pmid:33018997.
- 58. Ummel JD, Hoilett OS, Walters BD, Bluhm NDP, Pickering AS, Wilson DA, Linnes JC. Kick Ring LL: A Multi-Sensor Ring Capturing Respiration, Electrocardiogram, Oxygen Saturation, and Skin Temperature(1). Annu Int Conf IEEE Eng Med Biol Soc. 2020;2020:4394–7. Epub 2020/10/07. pmid:33018969.
- 59. Kenya IA. The Information and Communication Technology Authority: Ministry of Information Communication and Technology; 2022 [cited 2022 August 15]. Available from: https://icta.go.ke/.
- 60.
KenInvest. Information Communication Technology 2022. Available from: http://www.invest.go.ke/information-communication-technology/.
- 61. Hruby A, Bright J. TechCrunch+ [Internet]: TechCrunch+. 2015. Available from: https://techcrunch.com/2015/07/23/the-rise-of-silicon-savannah-and-africas-tech-movement/.
- 62. Inui TS, Nyandiko WM, Kimaiyo SN, Frankel RM, Muriuki T, Mamlin JJ, et al. AMPATH: living proof that no one has to die from HIV. J Gen Intern Med. 2007;22(12):1745–50. Epub 2007/11/01. pmid:17972138; PubMed Central PMCID: PMC2219843.
- 63. Inui TS, Sidle JE, Nyandiko WM, Yebei VN, Frankel RM, Mossbarger DL, et al. ’Triangulating’ AMPATH: demonstration of a multi-perspective strategic programme evaluation method. Sahara j. 2009;6(3):105–14. Epub 2010/05/21. pmid:20485850.
- 64. Litzelman DK, Gardner A, Einterz RM, Owiti P, Wambui C, Huskins JC, et al. On Becoming a Global Citizen: Transformative Learning Through Global Health Experiences. Ann Glob Health. 2017;83(3–4):596–604. Epub 2017/12/10. pmid:29221534; PubMed Central PMCID: PMC5726429.
- 65. Tierney WM, Nyandiko WN, Siika AM, Wools-Kaloustian K, Sidle JE, Kiplagat J, et al. "These are good problems to have…": establishing a collaborative research partnership in East Africa. J Gen Intern Med. 2013;28 Suppl 3(Suppl 3):S625–38. Epub 2013/06/26. pmid:23797916; PubMed Central PMCID: PMC3744278.
- 66. Turissini M, Mercer T, Baenziger J, Atwoli L, Einterz R, Gardner A, et al. Developing Ethical and Sustainable Global Health Educational Exchanges for Clinical Trainees: Implementation and Lessons Learned from the 30-Year Academic Model Providing Access to Healthcare (AMPATH) Partnership. Ann Glob Health. 2020;86(1):137. Epub 2020/11/13. pmid:33178558; PubMed Central PMCID: PMC7597575.
- 67. Holeman I, Kane D. Human-centered design for global health equity. Inf Technol Dev. 2019;26(3):477–505. Epub 2020/09/29. pmid:32982007; PubMed Central PMCID: PMC7484921.
- 68. Saparamadu A, Fernando P, Zeng P, Teo H, Goh A, Lee JMY, Lam CWL. User-Centered Design Process of an mHealth App for Health Professionals: Case Study. JMIR Mhealth Uhealth. 2021;9(3):e18079. Epub 2021/03/27. pmid:33769297; PubMed Central PMCID: PMC8088861.
- 69. Purkayastha S, Addepally SA, Bucher S. Engagement and Usability of a Cognitive Behavioral Therapy Mobile App Compared With Web-Based Cognitive Behavioral Therapy Among College Students: Randomized Heuristic Trial. JMIR Hum Factors. 2020;7(1):e14146. Epub 2020/02/06. pmid:32012043; PubMed Central PMCID: PMC7055853.
- 70. Purkayastha S, Allam R, Maity P, Gichoya JW. Comparison of Open-Source Electronic Health Record Systems Based on Functional and User Performance Criteria. Healthc Inform Res. 2019;25(2):89–98. Epub 2019/05/28. pmid:31131143; PubMed Central PMCID: PMC6517630.
- 71. Voorheis P, Zhao A, Kuluski K, Pham Q, Scott T, Sztur P, et al. Integrating Behavioral Science and Design Thinking to Develop Mobile Health Interventions: Systematic Scoping Review. JMIR Mhealth Uhealth. 2022;10(3):e35799. Epub 2022/03/17. pmid:35293871; PubMed Central PMCID: PMC8968622.
- 72. Wilson K, Bell C, Wilson L, Witteman H. Agile research to complement agile development: a proposal for an mHealth research lifecycle. NPJ Digit Med. 2018;1:46. Epub 2019/07/16. pmid:31304326; PubMed Central PMCID: PMC6550198.
- 73. Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: A targeted review of best practices for clinical prescribing interventions. Journal of Biomedical Informatics. 2012;45(6):1202–16. pmid:22995208
- 74. Manu A, Arifeen S, Williams J, Mwasanya E, Zaka N, Plowman BA, et al. Assessment of facility readiness for implementing the WHO/UNICEF standards for improving quality of maternal and newborn care in health facilities—experiences from UNICEF’s implementation in three countries of South Asia and sub-Saharan Africa. BMC Health Serv Res. 2018;18(1):531. Epub 2018/07/11. PubMed Central PMCID: PMC6038273 country-specific adaptations of the study protocol have been approved by the Ethics Committee and Institutional Review Boards of the International Centre for Diarrhoeal Diseases Research (Bangladesh); Navrongo Health Research Centre of the Ghana Health Service (Ghana) and the National Institute of Medical Research (Tanzania). The approval also included references to allow for pooling of data across all three countries for analysis. This is an observational study in which pregnant women presenting in labour within the health facilities, health facility staff and managers are being observed or interviewed. There are no conceivable direct safety concerns. Mothers having a still birth or newborn death will be excluded from interviews to avoid distress to already traumatized individuals. Observation of care might involve invading the privacy of the patient and might be distressing to the care provider but participants will be reassured before the observations begin. Each respondent was individually consented for participation in the study. Facility staff were also assured that the study is not a fault-finding mission and assured that any information they provide or that will be collected on their practices will be confidential and will not affect their position within the facility or in the health services. Data collectors read out a standard study information sheet and the consent form to all respondents in a language of their preference and checked for understanding before a request was made for consent. Participation will be voluntary with assurance of participants’ rights to withdraw from the assessment at any stage, even after initial consent. Agreement to participate in the study was indicated by a signature or a thumbprint. Confidentiality of all data collected is being maintained, always, and accessible only to senior project staff and to the study coordinators. The database is being stored on a security protected server, with password access only by senior project staff. The data forms are being stored in secure storage to be kept for a set minimum period agreed between UNICEF and the respective countries after the end of the study. Analyses will present aggregate results without identifier information. All interviews are being conducted in the respondents’ preferred language. For the qualitative interviews, consent is being taken for recording the interview. CONSENT FOR PUBLICATION: Not applicable. COMPETING INTERESTS: One co-author, Debra Jackson, was an editor of this journal till April 2018 but the decision and process of submission was completely transparent and all the authors declare no conflict of or competing interests. PUBLISHER’S NOTE: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. pmid:29986692
- 75.
USAID, WHO, Program UD, UNICEF, Bank W, Agency SID, Foundation BaMG. Principles for Digital Development 2015. Available from: https://digitalprinciples.org/.
- 76. Bucher S, inventorInfant thermoregulation and monitoring support system. United States patent US10390630B2. 2019 August 27, 2019.
- 77. Bucher S, Meyers E, Agnikula Kshatriya B, Avanigadda P, Purkayastha S. Bucher S., Meyers E., Agnikula Kshatriya BS., Avanigadda PC., Purkayastha SP. In: Abraham A, Gandhi N, Pant M, editors. Innovations in Bio-Inspired Computing and Applications—Proceedings of the 9th International Conference on Innovations in Bio-Inspired Computing and Applications IBICA 2018; May 21: Advances in Intelligent Systems and Computing; 2019. p. 361–4.
- 78. Dawar R, Nangia S, Thukral A, Chopra S, Khanna R. Factors Impacting Practice of Home Kangaroo Mother Care with Low Birth Weight Infants Following Hospital Discharge. J Trop Pediatr. 2019;65(6):561–8. Epub 2019/02/16. pmid:30768199.
- 79. Chan G, Bergelson I, Smith ER, Skotnes T, Wall S. Barriers and enablers of kangaroo mother care implementation from a health systems perspective: a systematic review. Health Policy Plan. 2017;32(10):1466–75. Epub 2017/10/04. pmid:28973515; PubMed Central PMCID: PMC5886293.
- 80. Charpak N, Ruiz-Peláez JG. Resistance to implementing Kangaroo Mother Care in developing countries, and proposed solutions. Acta Paediatr. 2006;95(5):529–34. Epub 2006/07/11. pmid:16825131.
- 81. Smith ER, Bergelson I, Constantian S, Valsangkar B, Chan GJ. Barriers and enablers of health system adoption of kangaroo mother care: a systematic review of caregiver perspectives. BMC Pediatr. 2017;17(1):35. Epub 2017/01/27. pmid:28122592; PubMed Central PMCID: PMC5267363.
- 82. Målqvist M. Neonatal mortality: an invisible and marginalised trauma. Glob Health Action. 2011;4. Epub 2011/03/23. pmid:21423597; PubMed Central PMCID: PMC3060998.
- 83. Liu Z, Stasko JT. Mental models, visual reasoning and interaction in information visualization: a top-down perspective. IEEE Trans Vis Comput Graph. 2010;16(6):999–1008. Epub 2010/10/27. pmid:20975137.
- 84.
Group EatR. Bank My Cell [Internet]2022. Available from: https://www.bankmycell.com/blog/how-many-phones-are-in-the-world.
- 85. Michelini E, Calabretta MM, Cevenini L, Lopreside A, Southworth T, Fontaine DM, et al. Smartphone-based multicolor bioluminescent 3D spheroid biosensors for monitoring inflammatory activity. Biosens Bioelectron. 2019;123:269–77. Epub 2018/09/20. pmid:30227987.
- 86. Reading M, Baik D, Beauchemin M, Hickey KT, Merrill JA. Factors Influencing Sustained Engagement with ECG Self-Monitoring: Perspectives from Patients and Health Care Providers. Appl Clin Inform. 2018;9(4):772–81. Epub 2018/10/12. pmid:30304745; PubMed Central PMCID: PMC6179719.
- 87. Clark KD, Woodson TT, Holden RJ, Gunn R, Cohen DJ. Translating Research into Agile Development (TRIAD): Development of Electronic Health Record Tools for Primary Care Settings. Methods Inf Med. 2019;58(1):1–8. Epub 2019/07/06. pmid:31277082; PubMed Central PMCID: PMC6823924.
- 88. Colbert AJ, Co K, Lima-Cooper G, Lee DH, Clayton KN, Wereley ST, et al. Towards the use of a smartphone imaging-based tool for point-of-care detection of asymptomatic low-density malaria parasitaemia. Malar J. 2021;20(1):380. Epub 2021/09/27. pmid:34563189; PubMed Central PMCID: PMC8466697.
- 89. Colbert AJ, Lee DH, Clayton KN, Wereley ST, Linnes JC, Kinzer-Ursem TL. PD-LAMP smartphone detection of SARS-CoV-2 on chip. Anal Chim Acta. 2022;1203:339702. Epub 2022/04/02. pmid:35361434; PubMed Central PMCID: PMC8905050.
- 90. Moehling TJ, Lee DH, Henderson ME, McDonald MK, Tsang PH, Kaakeh S, et al. A smartphone-based particle diffusometry platform for sub-attomolar detection of Vibrio cholerae in environmental water. Biosens Bioelectron. 2020;167:112497. Epub 2020/08/25. pmid:32836088; PubMed Central PMCID: PMC7532658.
- 91. Smith S, Koech R, Nzorubara D, Otieno M, Wong L, Bhat G, et al. Connected diagnostics: linking digital rapid diagnostic tests and mobile health wallets to diagnose and treat brucellosis in Samburu, Kenya. BMC Med Inform Decis Mak. 2019;19(1):139. Epub 2019/07/25. pmid:31331394; PubMed Central PMCID: PMC6647279.
- 92. Swanson JO, Nathan RO, Swanson DL, Perez KM, Bresnahan BW, Mirza W, Goldenberg RL. Use of ultrasound and mHealth to improve perinatal outcomes in low and middle income countries. Semin Perinatol. 2019;43(5):267–72. Epub 2019/04/21. pmid:31003635.
- 93. Kang J, Hanif M, Mirza E, Khan MA, Malik M. Machine learning in primary care: potential to improve public health. J Med Eng Technol. 2021;45(1):75–80. Epub 2020/12/08. pmid:33283565.
- 94. Peiffer-Smadja N, Rawson TM, Ahmad R, Buchard A, Georgiou P, Lescure FX, et al. Machine learning for clinical decision support in infectious diseases: a narrative review of current applications. Clin Microbiol Infect. 2020;26(5):584–95. Epub 2019/09/21. pmid:31539636.
- 95. Ridgway JP, Lee A, Devlin S, Kerman J, Mayampurath A. Machine Learning and Clinical Informatics for Improving HIV Care Continuum Outcomes. Curr HIV/AIDS Rep. 2021;18(3):229–36. Epub 2021/03/05. pmid:33661445; PubMed Central PMCID: PMC8215683.
- 96. Tariq A, Purkayastha S, Padmanaban GP, Krupinski E, Trivedi H, Banerjee I, Gichoya JW. Current Clinical Applications of Artificial Intelligence in Radiology and Their Best Supporting Evidence. J Am Coll Radiol. 2020;17(11):1371–81. Epub 2020/11/07. pmid:33153541.
- 97. Sanchez-Martinez S, Camara O, Piella G, Cikes M, González-Ballester M, Miron M, et al. Machine Learning for Clinical Decision-Making: Challenges and Opportunities in Cardiovascular Imaging. Front Cardiovasc Med. 2021;8:765693. Epub 2022/01/22. pmid:35059445; PubMed Central PMCID: PMC8764455.
- 98. James C, Ranson JM, Everson R, Llewellyn DJ. Performance of Machine Learning Algorithms for Predicting Progression to Dementia in Memory Clinic Patients. JAMA Netw Open. 2021;4(12):e2136553. Epub 2021/12/17. pmid:34913981; PubMed Central PMCID: PMC8678688 from the National Institute on Aging of the National Institutes of Health, Alzheimer’s Research UK, Mind over Matter Medtech, and SharpTx outside the submitted work. No other disclosures were reported.
- 99. Kasiiti N, Wawira J, Purkayastha S, Were MC. Comparative Performance Analysis of Different Fingerprint Biometric Scanners for Patient Matching. Stud Health Technol Inform. 2017;245:1053–7. Epub 2018/01/04. pmid:29295262.
- 100. Cohen DJ, Wyte-Lake T, Dorr DA, Gold R, Holden RJ, Koopman RJ, et al. Unmet information needs of clinical teams delivering care to complex patients and design strategies to address those needs. J Am Med Inform Assoc. 2020;27(5):690–9. Epub 2020/03/07. pmid:32134456; PubMed Central PMCID: PMC7647291.
- 101. Biswas SK, Chatterjee S, Bandyopadhyay S, Kar S, Som NK, Saha S, Chakraborty S. Smartphone-Enabled Paper-Based Hemoglobin Sensor for Extreme Point-of-Care Diagnostics. ACS Sens. 2021;6(3):1077–85. Epub 2021/02/27. pmid:33635650.
- 102. Bowles JKF, Mendoza-Santana J, Vermeulen AF, Webber T, Blackledge E. Integrating Healthcare Data for Enhanced Citizen-Centred Care and Analytics. Stud Health Technol Inform. 2020;275:17–21. Epub 2020/11/24. pmid:33227732.
- 103. Dehnavieh R, Haghdoost A, Khosravi A, Hoseinabadi F, Rahimi H, Poursheikhali A, et al. The District Health Information System (DHIS2): A literature review and meta-synthesis of its strengths and operational challenges based on the experiences of 11 countries. Health Inf Manag. 2019;48(2):62–75. Epub 2018/06/15. pmid:29898604.
- 104. Theobald S, Brandes N, Gyapong M, El-Saharty S, Proctor E, Diaz T, et al. Implementation research: new imperatives and opportunities in global health. Lancet. 2018;392(10160):2214–28. Epub 2018/10/14. pmid:30314860.