1. Introduction

As global focus shifts from ensuring children survive to enabling them to thrive, millions of children living in low-resourced settings are found to be at-risk of not attaining their full developmental potential [1]. This is due to their disproportionately high exposure to risk factors for poor development during critical periods in childhood [2], including malnutrition, iron-deficiency anemia, infectious diseases, exposure to violence, exposure to toxins, extreme poverty, maternal depression, and inadequate cognitive stimulation [3]. While interventions and programs are being implemented to address these risk factors [4,5], we need to understand how best to optimize neurocognitive outcomes within the context of scarce resources.

Healthy child development comprises a complex interplay of rapid physiological, psychological, and physical changes in response to early environmental experiences, with lasting effects on multiple domains of development, including cognition. Cognitive development in preschool years, through which children develop the skills to acquire, assimilate and apply knowledge, has been demonstrated to be predictive of later outcomes such as IQ and academic achievement [6,7]. Executive function, a key component of cognitive development, is our ability to temporarily manipulate information mentally (working memory), generate different solutions to a problem (cognitive flexibility), and maintain impulse control (inhibition). Executive function enables us to plan, focus attention, set and achieve goals [8], and it predicts math, reading, and science achievements in school [9–11]. Deficits in cognitive ability and executive function have long-term impacts across the life-course, starting with poor educational attainment and resulting in loss of adult income, thus impacting the livelihood of families, communities, and countries [4]. Therefore, high-quality measures of cognitive development, including executive functioning, are essential to identify children at-risk of not attaining their full developmental potential, triaging them into timely interventions, and finally, monitoring their impact on developmental trajectories within clinical, research, or community-based settings.

Many tools have been used in measurement of cognition and in some cases, executive function, in low-resource settings. Assessing cognitive ability, and in particular, executive function, in children within these settings, may be impeded by a number of challenges. These challenges include using tools with appropriate cultural and language adaptations, training a workforce for high-quality administration, and utilizing tools with adequate reliability and validity, while maintaining affordability and scalability [12,13].

Currently, many neurocognitive assessments depend on observations of child behavior conducted by child development and neuropsychology specialists, who undergo substantial training to become proficient in high quality and consistent administration, scoring and interpretation of assessment tools. Although capacity building and infrastructure in psychological testing has improved in recent years, many countries still have few trained workers qualified to implement these tools [14]. Examples of tools used for measuring cognition are the Kaufman-ABC or the Wechsler Preschool and Primary Scale of Intelligence (WPPSI) [15,16] and in younger children, the Bayley Scales for Infant and Toddler Development [17]. Measurement of executive function has been measured with tools including, but not limited to, the NEuroPSYchological Asssessment (NEPSY) [18,19], the knock-tap tests [20], the Spin the Pots task [21], or parent report measures such as the Behavior Rating Inventory of Executive Function (BRIEF) [22]. These all vary in reliability and validity [13]. Most of these tools have been developed and validated within high-resource settings, although some attempts have been made to develop versions for other settings. This can present further challenges, as most items in neuropsychological assessment tools are contextually related to settings where tools were developed and validated, therefore requiring lengthy and expensive adaptation process for use cross-culturally. This has been attempted in some settings, e.g., Kilifi, Kenya, where a battery was made for an African context; however, difficulties remain in capacity to scale-up training, application, and measurement [23]. Furthermore, tools created in high-resource contexts are often proprietary and the cost to administer in low-resourced settings can be prohibitive. There is, thus, a paucity of contextually relevant scalable cognitive assessment tools developed, validated, and used within low-resourced settings [24]. We define low-resourced settings as those with a population facing health inequities that negatively impact child development, typically in low- and middle-income countries.

Advances in technology, including the emergence of tablet computers, could be leveraged to aid in the scalable evaluation of cognition and particularly executive function, across a wide range of settings. Tablet computers are readily available, inexpensive, portable, and have functionality without internet. Paper-based childhood developmental assessment tools are increasingly being deployed on tablet computers using basic open data kits or similar programs. This includes the Malawi Developmental Assessment Tool (MDAT) [25] and the Global Scales of Early Development (GSED) [26]. While this standardizes administration and scoring by minimizing errors, these tools either rely on parent report, assuming parental knowledge of developmental milestones, or behavioral observations of children by non-specialists, which itself is resource-intensive and only partially addresses the workforce challenge.

In recent years, computerized tasks within tablet computers are increasingly being used to evaluate child performance directly. This has the potential to overcome many current challenges of measuring cognition in children globally. These tools permit the administration and scoring of a broad array of tasks that measure specific domains of cognitive functioning with minimal potential for error, while also having the potential to be gamified [27], increasing children’s interest in these cognitive tasks. Furthermore, with cameras (to capture images and videos during task performance), accelerometers and gyroscopes (to estimate motion and force on the screen) [19,20], timers (to assess latency in responses) [28], and microphones (to capture audio), developers can tap into a wide range of child responses to assess their cognitive abilities [29]. Such nuanced variables are not feasibly evaluated with traditional pen-and-paper administration of cognitive assessment tools [30,31]. Because the cognitive tasks are administered with a tablet computer rather than skilled observation of child behavior, it requires very little workforce support for administration or scoring, enabling non-specialists with minimal training to administer tasks. While smartphone technology has similar strengths, the smaller screen size presents some limitations and, at this point, only a few cognitive tools are under development for smartphone use. In sum, tablet computers provide opportunities to perform high-quality cognitive assessments of children in a manner that is easily scalable and practical in low-resourced settings.

While this technology appears to be the future of cognitive testing, no known summary of these tools exists. This overview aims to summarize the current landscape of tablet-based cognitive assessment tools used in low-resourced settings that directly measure child performance. We pose considerations for use of these tools with the existing state of the evidence and make recommendations for ways in which this novel field can move forward. This overview is for those who may want to use tools for programmatic and research evaluation and therefore specifically targets information on the feasibility and validity of the tools in their current format, when used in low-resource settings.

2. Methodology for compiling tools

We identified tablet-based cognitive assessment tools by performing a general scoping literature search as well as through reaching out to international content experts working in the field. We aimed to identify peer-reviewed articles providing information on tablet-based cognitive assessment tools from electronic databases that included PubMed and Google Scholar identifying any publications from January 2000 to September 2021. The search terms used included “cognition”, “executive function,” “pediatrics,” “tabled-based assessments,” and “children.” Inclusion criteria required that the assessment had to measure cognition was used in children under 18 years of age, was used in low-resourced settings, and was used on a tablet computer. Tools were excluded if they were only used in well-resourced settings or performed only on smartphones or laptop/desktop computers. Much of the information gathered required direct interviews with the developers and was not available through peer-reviewed articles. We therefore supplemented our literature search through general search engines, such as Google and Bing, and sought topic experts (i.e., those who lead and consult on projects measuring cognition in low-resourced settings, but do not themselves develop tools) to identify further tools. We emailed each application developer requesting an interview and further information about any other known tablet-based cognitive assessment tools used in low-resourced settings. Through our review of published papers, websites, and interviews with developers, we collected data on domains of cognition and executive function measured, time for training and administration, country of use and present adaptations, psychometric properties of each tool (as published to date), and feasibility of use.

3. Current landscape of tablet-based cognitive assessment tools for children

Numerous commercial and non-commercial tablet-based cognitive assessment tools have been developed in recent years. From our search, we identified 16 tools and described characteristics of each in Table 1 and Fig 1. Most tools were developed in North America, Europe, or Australia and initially used the English language, which was subsequently translated into other languages. Two tools had significant development work performed in India (DEEP and START), which included formative work in the community and iterative development of the user-interface through testing with non-specialist administrators and the target age range of children [32]. One tool has been further developed in Brazil (Educational Neuroscience App-Based Learning Environment (ENABLE). From interviews with the tool developers, the primary users of these tools, to date, are researchers, with administration being performed by research staff or community health workers. However, some tools, such as the Early Years Toolbox and NIH Toolbox, have also been used by educators and clinicians as well.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of tablet-based cognitive assessments.

https://doi.org/10.1371/journal.pdig.0000196.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Synthesis across tablet-based cognitive assessments.

The names of the tools are listed in the Y-axis, with non-commercial tools listed first, followed by commercial tools. The X-axis is the range of ages, in years, for the intended population for the tool’s use. The numbers encompassed by () by the tool names indicates the number of languages for which the tool is translated. The numbers within the bands indicate the number of minutes (min) required to complete each test, with the number of tests listed in []. The color of the bands indicates the amount of training required to administer, with blue indicating <4 h of required training, yellow indicating an estimated 4–12 h of required training, pink indicating an estimated >12 h of required training.

https://doi.org/10.1371/journal.pdig.0000196.g001

While nearly all commercial tools have age ranges that extend into adulthood, the non-commercial tools tend to have more narrow ranges, likely due to a target study population for whom the initial tool was developed. Most assessment tools contain batteries that consist of individual tests, and the length of the assessment is modifiable based on the number of tests selected. For example, Cantab and Cogstate each have over 16 tests that can be administered, depending on the needs of the user [33,34]. Most tools contain tests that each evaluate specific domains of cognition, often focusing on memory, attention, visual-spatial, and inhibition tasks (Table 2). Babyscreen and the Minnesota Executive Function Scale (MEFS)/EFgo test the youngest age range of children (as young as 18 months of age), but only have 1 test administered measuring multiple sub-domains of cognition [35,36]. The MEFS/Efgo tool administers a test based similar to the Dimensional Change Card Sort task, aiming to measure inhibition, cognitive flexibility, and memory [36]. This approach is linked to the challenges of tapping into specific, differentiated dimensions of cognitive tasks within these early ages [37].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Domains measured by tablet-based cognitive assessments.

https://doi.org/10.1371/journal.pdig.0000196.t002

4. Considerations for identifying the tool(s) of choice

While the concept and aims of these tablet-based cognitive assessment tools are often similar, their strengths and limitations are highlighted for potential users to consider while identifying an appropriate tool for their setting. Some limitations are inherent to the way tools are funded and designed. Non-commercial tools are often developed by academicians with grant funding. These are therefore typically dependent on this funding or other program fees to maintain and update applications along with the tablet operating system. While these tools are often open-source, and hence costs are generally low, researchers often must directly contact developers to arrange for an agreement for tool use. While this benefits smaller projects with budgetary constraints, the lack of available support can cause delays in their implementation or any adjustments required.

Specific tools also have their own strengths and limitations. The Early Years Toolbox is the only tool that is freely downloadable and has instructions on administration on its website, a great strength in its accessibility. However, due to the “open access” nature of the tool, this results in limited knowledge on the full extent on locations of use and the degree of success of cross-cultural administration. This can only be gleaned from published articles and reports of its use, which may be subject to publication bias. A summary of the strengths and limitations for each tool is included in Table 3. Associated psychometrics and additional information about these tools are detailed in the S1 File.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. General strengths and limitations of tablet-based cognitive assessments.

https://doi.org/10.1371/journal.pdig.0000196.t003

We identified a number of factors we consider important when choosing a tool for use in a low-resourced setting for measuring cognition in early childhood. These include (a) ease of using technology to measure cognition in the early childhood; (b) validity in varying cultural contexts; (c) ease of adaptability for different settings; (d) overcoming workforce challenges; and (e) accessibility.

Validity of tools for varying socioeconomic and cultural settings

An important first step in choosing a cognitive tool is to critically review its measurement properties (such as validity and reliability). The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) guidelines provide a framework to guide this [43] and supports the use of both qualitative (acceptability, face, and content validity) and quantitative aspects of validity (construct, cultural, and structural validity) alongside reliability (Fig 2). While nearly all the tablet-based cognitive tests included in this review (14/16) have some preliminary or published validity data (S1 File), the detail and rigor of this psychometric testing differs among tools. Some tools have attempted validation against a “gold standard.” However, this is complex, particularly in international low-resourced settings since “gold standard assessment tools” are typically developed and validated in high-resourced western settings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Assessing the measurement properties of a tool.

Domains of psychometric properties adapted from the following publication: Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Medical Research Methodology. 2010;10(1):22.

https://doi.org/10.1371/journal.pdig.0000196.g002

Evaluating validity of tools can be particularly challenging within settings where children have limited daily exposure to technology or the items contained within the assessment tool. Tools are often developed for children with high exposure to technology, indicating that limited exposure may impact a child’s ability to interact with the tool’s content in the same meaningful way. However, evidence is emerging that suggests the lack of prior exposure to smart devices may not impact the accuracy with which tablet-based tools can measure cognition [28]. Indeed, a “rights-based perspective” would argue that technology coupled with culturally neutral content can benefit children globally. However, our review has identified rigorous psychometric testing as a gap in this field, which warrants additional consideration as tablet-based tools are disseminated widely.

Notably, caution should be taken when interpreting the assessment scores for children who differ significantly, either by language, age, or culture, from the original normative population of the assessment tool. Certain aspects of cognition are more adaptive and highly prioritized in some settings compared to others, and thus, the normative range of scores is likely to differ across settings. Thus, issues will arise when one depends on the normative population data to determine cutoffs for deficits in cognition or to function as a comparison or control group.

5. Future directions for tablet-based pediatric cognitive assessment tools

This review aimed to summarize the current landscape of tablet-based assessment tools that measure cognition in children, particularly those used in low-resourced settings, as a potential solution to poor healthcare infrastructure and workforce-related barriers. We identified 16 tools that ranged the full spectrum of possibilities: from open-sourced to proprietary, and from those in their early stages of piloting in one region to those with extensive validation in multiple countries. As this novel area of digital pediatric cognitive assessment tools emerge and are pushed up the global mHealth agenda [29,46], it is critical for users to consider a tool’s psychometric properties, such as validity and reliability, before integrating it into clinical practice, research, or public health developmental surveillance systems. The COSMIN checklist provides useful guidance, not only to evaluate the validity and utility of a novel digital assessment tool, but also for developers to keep in mind while planning validation studies, or when describing the strengths and limitations of their tools to potential users [43].

In the absence of a true “gold-standard,” tool developers should aim to generate local normative scores for these novel tools in the target populations, instead of benchmarking them to the available “gold-standards.” The developers of the ENABLE and DEEP tool hope to develop a cloud database from their users, so that global sample “norms” can become available in open-source platforms (verbal communication from N. Pitchford and S. Bhavnani, respectively (February 2021)).

Additionally, while this review focuses on tablet-based assessment tools, we are keenly aware that smartphones represent the ultimate game changer in terms of achieving scale. The use of tablet-based assessment tools still requires a trained individual to go into the community or household for administration, whereas a smartphone-based assessment tool could potentially be downloaded onto a phone and then self-administered or administered by a parent within the home. Lead investigators of the NIH Toolbox are involved with the development of self-administered smartphone-based tests, Mobile Toolbox, which they hope will be available for public dissemination in 2023 [47]. Given the ubiquity of smartphones globally—over 5 billion subscribers, 70% of those residing in low-resourced settings—and cellular networks connecting 85% of the world’s population [29], we believe the potential for these cognitive assessment tools to scale will vastly improve with smartphones administration. Tablet-based tools were the focus on this review as there are currently a number of well-studied and validated options available for use in low-resources settings. They also form a useful “bridge” between more expensive and potentially fragile laptop computers and a larger screen than mobile devices. It is clear, however, that smartphones are more accessible and cheaper. Presently, less information is available about applications on these devices for measuring cognition in children, smartphone-based cognitive assessments for other populations, such as adults with dementia, have recently been developed [48,49]. It is likely that this review will need updating considering the fast-moving field and will need to include a focus on smartphone-based tools once data on their validity are available. Over 80% of World Health Organization (WHO) member states use at least 1 mHealth initiative operationalized through smartphones [29], with features such as the use of videos and decision support systems, that have proven useful in improving maternal and child health in a variety of low-resourced settings [50]. Therefore, the integration of cognitive assessments in smartphones may be the next leap towards optimizing child development at scale across global settings.

The digital administration of these tools also makes it possible to sync data collected from different modalities, such as eye tracking and electroencephalography, to provide a deeper, more integrated level of evaluation. With advances in technology, further spurred by substantial increases in use of telemedicine during the Coronavirus Disease 2019 (COVID-19) pandemic, these assessment tools may facilitate the possibility of virtual, parent-led, in-home well-baby checks in the future [51].

A forum to house the available digital tools, regularly updated to reflect the latest progress and associated data, would be valuable to stakeholders with interest in early childhood development. Such an effort has been initiated by the World Bank [52]. For this review, we aim to use an example of this and consolidate the information gleaned within the IMPACT Measures Tool database [53]. This online and open-source database is based on a research-driven scoring system that allows for the comparison of early childhood and parenting measures based on 4 categories (i.e., cost, usability, cultural relevance, and technical merit) [54]. Attending to the current landscape and new demands in low-resourced settings, we thought that the IMPACT database would benefit from the addition of digital measures, but other digital databases, as they emerge, should be considered.

This review has some limitations. Because we did not perform a formal systematic review, it is possible that we have missed some existing literature regarding tools currently being used in low-resourced settings outside of the network of investigators with whom we connected. Further, the dynamic and fast-paced nature of this emerging field would imply that new and updated tools are being added to the existing pool rapidly. Some have not yet been used in low-resourced settings and therefore will have not been included in this review. However, despite these limitations, we believe that this review of the existing tablet-based cognitive assessment tools adds great value, since the current landscape of available tools and possible future directions have not yet been summarized in this field.

Supporting information

Acknowledgments

We would like to thank all of the developers of these tablet-based applications, who willingly answered numerous questions as we strived to understand their tool’s function and purpose. Many of these individuals created these tablet-based tools to improve the manner in which cognition is assessed around the world. We appreciate their commitment and dedication to finding effective ways of measuring pediatric cognition globally.