A Systematic Review and Meta-Analysis of a Measure of Staff/Child Interaction Quality (the Classroom Assessment Scoring System) in Early Childhood Education and Care Settings and Child Outcomes

The quality of staff/child interactions as measured by the Classroom Assessment Scoring System (CLASS) in Early Childhood Education and Care (ECEC) programs is thought to be important for children’s outcomes. The CLASS is made of three domains that assess Emotional Support, Classroom Organization and Instructional Support. It is a relatively new measure that is being used increasingly for research, quality monitoring/accountability and other applied purposes. Our objective was to evaluate the association between the CLASS and child outcomes. Searches of Medline, PsycINFO, ERIC, websites of large datasets and reference sections of all retrieved articles were conducted up to July 3, 2015. Studies that measured association between the CLASS and child outcomes for preschool-aged children who attended ECEC programs were included after screening by two independent reviewers. Searches and data extraction were conducted by two independent reviewers. Thirty-five studies were systematically reviewed of which 19 provided data for meta-analyses. Most studies had moderate to high risk of bias. Of the 14 meta-analyses we conducted, associations between Classroom Organization and Pencil Tapping and between Instructional Support and SSRS Social Skills were significant with pooled correlations of .06 and .09 respectively. All associations were in the expected direction. In the systematic review, significant correlations were reported mainly from one large dataset. Substantial heterogeneity in use of the CLASS, its dimensions, child outcomes and statistical measures was identified. Greater consistency in study methodology is urgently needed. Given the multitude of factors that impact child development it is encouraging that our analyses revealed some, although small, associations between the CLASS and children’s outcomes.


Introduction
For preschool aged children, enrollment in Early Childhood Education and Care (ECEC) in many countries is now the norm [1,2].Some studies have demonstrated that higher-quality ECEC is associated with improved language development, cognitive functioning, social competence, and emotional adjustment [3][4][5].However, others have failed to report such linkages [6][7][8].
Structural quality indicators (e.g., staff/child ratios, staff education) generally refer to aspects of quality that are more easily quantifiable.As a result, governments tend to focus on regulation of these characteristics.Structural quality is thought to impact children indirectly by setting up the conditions in which children's direct experiences with the environment take place [9].For example, when staff-child ratios are better, staff may have more opportunities to have positive interactions with children.Process quality is harder to assess as it focuses on the quality of interactions that children experience directly such as the warmth and responsiveness of the staff that care for children.At least in part due to increasing calls for accountability in education systems, substantial attention is being paid to identifying the kinds of environments that support young children's learning.Recent studies have focused attention on the quality of interactions children experience and are, therefore, thought to influence children more directly [8,10].This focus on interactions is driven by some of the most fundamental theories in developmental psychology.These include, but are not limited to, the importance of children's early social exchanged as outlined by attachment theory [11], Ecological Systems theory's focus on the child's interactions with his/her most immediate environment [12] and Vygotsky's emphasis on learning through social exchanges by supportive "experts" [13].In keeping with these theories, a key aspect of process quality is the warmth and responsiveness of interactions between staff and children, and the extent to which the interactions scaffold children's learning and development.The Classroom Assessment Scale (CLASS) is a relatively new measure of the emotional tone and the quality of developmentally appropriate "instruction" (e.g., providing a language/literacy rich environment, opportunities for hands on exploration of natural phenomena, etc.) of ECEC classrooms in a systematic way.In this paper we present a systematic review of the CLASS, which is increasingly influential in the field and child outcomes.Hamre et al. (2013) [14] present, and provide empirical support for Teaching Through Interactions.This is a conceptual framework that identified three key domains in teacherchild interactions.These domains are Emotional Support, Classroom Organization and Instructional Support.While this work largely discusses instruction within the formal education system it is based on theories that apply equally well to the preschool years.Specifically, in keeping with attachment [11], Emotional Support focuses on staffs' ability to create warm, safe, responsive and predictable environments that help children because secure, autonomous learners.Classroom Organization captures staffs' ability to structure the room in a way that minimizes behavioral problems and allows children to focus on their attention on learning.Finally, in keeping with Vygotskian thinking, Instructional Support focuses on the extent to which staff builds on children's existing knowledge and scaffold leaning.The importance of these domains is supported by a body of research that links them to a host of child outcomes, broadly construed (see Cappella et al. [15] for a comprehensive review of child outcomes associated with these domains of interactions between teachers and children).
The CLASS is a key measure in this important research area as it is specifically designed to capture these three domains.Furthermore, the CLASS has also been central in the development of professional development programs for early childhood educators (e.g., MyTeaching-Partner [16]) that are built around this fundamental view of good caregiving/instruction for young children.Thus, the contribution of this systematic review and meta-analysis lies within the current theoretical discussion about the importance of teacher-child interactions as a key driver of teacher effectiveness.
The CLASS is one of several classroom level measures of ECEC process quality.The most frequently used of these is the Early Childhood Environment Rating Scale-Revised (ECERS-R [17]).The ECERS-R is a global measure of quality that includes both structural aspects of the environment (e.g., the physical environment) and process quality (e.g., the kind of language that staff direct to children).There is a substantial body of research linking the ECERS-R to other measures of quality as well as to child outcomes [18][19][20].The Observational Record of the Caregiving Environment (ORCE [21]) is another measure but research using the ORCE is largely based on one sample of approximately 1000 participants.As a result, it lacks the evidence base for extensive use or for a systematic review/meta-analysis.While the CLASS is a relatively new measure (certainly compared to the ECERS-R), it is supported by a growing body of evidence and grounded in a substantial theoretical/conceptual framework.
The growing importance of the CLASS is evidenced by the fact that it is now used in statewide ECEC program quality assurance systems in Arizona, Illinois, Maryland, Massachusetts, Minnesota, New Mexico, Oklahoma, Vermont, Virginia and Washington [22].These assessments are used to make key funding and quality improvement decisions.Some quality assurance programs also make ratings public so that parents can use ratings in selecting placement for their children.Having access to empirically based measures of ECEC quality that are known to be associated with child outcomes is important for a range of stakeholders including parents and the people parents turn to for advice (e.g., their child's pediatrician), funders, policymakers and others.Despite rapid increases in use of the CLASS in research and in high stakes accountability settings, associations between the CLASS and child outcomes have not been systematically reviewed.
We chose to cast a wide net regarding the child outcomes we included in this review.This reflects an understanding of the classroom context as having an impact on children that goes "beyond achievement tests" [15].In keeping with this view, we set out to compile studies that used a broad range of outcomes and included information about young children's very early academic outcomes (including language and early literacy skills); their health/wellbeing and their social-emotional development.
The objective of this review was to evaluate associations between CLASS domain and dimension scores in classrooms that served preschool aged children and children's concurrent or subsequent social, emotional and cognitive outcomes.A secondary aim was to explore associations between specific CLASS domains and dimensions and specific child outcomes (e.g., Emotional Support to children's social outcomes, Instructional Support for Learning to language and mathematics or Language Modeling to language outcomes).

Selection of Studies
Searches for potential studies were conducted in three ways.First, the electronic databases, PsycINFO, Medline, and ERIC, were searched for papers that were published from the year each journal began until July 3, 2015.Search terms are provided in Tables A-D in S1 File.Second, websites for datasets that provide key data on ECEC quality and child outcomes were searched.Specifically, websites for the following databases were searched: Cost, Quality and Outcomes Study (CQO) [23]; Early Childhood Longitudinal Study (ECLS) [24]; Effective Provision of Pre-School Education (EPPSE) Project [25]; Head Start Impact Study (HS) [26]; National Center for Early Development and Learning (NCEDL) Multi-State Study of Pre-Kindergarten [27]; State-Wide Early Education Program Study (SWEEP) [27]; Family and Child Experiences Survey (FACES) [28]; and the National Institute of Child Health and Human Development (NICHD) Study of Early Child Care and Youth Development [29].Finally, reference lists of identified studies were searched to locate additional studies.Searches were limited to English language articles only.

Type of Participants
ECEC programs are generally divided into classrooms that serve children of different ages.We focused on research on classrooms that served preschool aged children (i.e., children approximately between the ages of 30 and 72 months) because much of the research has focused on this group [30,31] as it is the largest group of children cared for in a single type of ECEC setting [32].The inclusion criteria used in this study and the rationale for each criterion are provided in Table 1.

Study Selection and Data Extraction
The selection of eligible documents was done in two steps that involved pairs of independent raters.First, the titles and abstracts of documents were screened and then full text articles were reviewed to determine eligibility.Discrepancies between raters were resolved through discussion and consensus.Relevant child and family characteristics, study characteristics and reported measures of association were extracted using standardized forms by two reviewers and discrepancies were resolved by consensus.

CLASS Domains and Dimensions
The CLASS consists of three Domains: Emotional Support, Classroom Organization and Instructions Support.Each domain is made up of several dimensions.Specifically, Emotional Support is made up of Negative Climate, Positive Climate, Teacher Sensitivity and Regard for Student Perspective).Classroom Organization is made up of Behavior Management, Productivity and Instructional Learning Formats.Instructional Support is made up of Concept Development, Quality of Feedback, Language Modeling, along with Literacy Focus.Factor analysis of the earlier version of the CLASS yielded two domains: Emotional Climate and Instructional Climate [33].Emotional Climate includes all of the dimensions of the Emotional Support domain and the Behavior Management dimension that later became part of the Classroom Organization domain.Instructional Climate is a composite of Concept Development and Quality of Feedback.Eligible studies used in this paper operationalized the instrument in various ways (see Table 2).When reported, findings for individual dimensions were gathered.This was done to explore the possibility that associations might be higher between specific dimensions that mapped onto specific outcomes more closely than between the broader domains and specific outcomes (e.g., that language modeling might have a stronger association to children's language skills than Instructional Support).Including associations with dimensions also increased the comprehensiveness of our review of findings about the CLASS.

Outcomes
In an effort to include the many outcomes that have been associated with high quality "instruction" for children [15] we defined "outcomes" broadly.We included any measure of children's academic competence (e.g., cognitive, language, mathematics) as well as health/wellbeing and social and emotional functioning.

Meta-Analyses
Studies that could be meta-analyzed were drawn from the pool of studies that were eligible for the systematic review.To be meta-analyzed studies had to use identical child outcome measures and had to use an identical operationalization of the CLASS.We adopted a minimum requirement of three studies to conduct a meta-analysis on a particular child outcome.To increase homogeneity among studies that were meta-analyzed, only studies that ensured children's exposure to the program were included.To do this, we only included studies in which the authors explicitly stated that children had been in the program for a period of time prior to their assessment.We also included analyses in which children's pre-scores were used as a

Child Care Type
Only studies that examined the impact of the quality of centre-based programs on children's outcomes were included.Centre-based programs included daycare and preschool programs, nursery schools, pre-kindergarten programs, and Head Start programs.Studies that only examined home-based child care, or those in which home-based and centre-based could not be separated were excluded.
Center-based child care settings differ from home daycare in many ways such as ratios, group size, physical environment, curriculum, age range of children, and caregiver qualifications.As a result, quality is often measured differently for these two settings (e.g., ECERS versus FCCERS).

Age Served
Studies that included preschool-aged children as the majority of participants were included.For the purposes of the meta-analysis, preschool-age was defined as ranging from 30 to 72 months.
Preschool-aged classrooms are different from infant/ toddler classrooms due to the developmental stage and needs of the children in these two age groups.
As a result, regulations and standards of care (e.g., ratios, physical environment, etc.) as well as daily activities (e.g., curriculum) differ between infant/ toddler and preschool-aged classrooms.

Child Outcomes
Studies that provided information about the association between CLASS on children's cognitive, academic, social-emotional, health, or motor outcomes were included.Data could have been gathered from teachers, parents, and/or children themselves.Measures that focus on dyads (e.g., attachment) were excluded.
Cognitive, academic, social-emotional, health, and motor outcomes were selected because they are key predictors of children's developmental trajectories.
Measures that focus on staff-child or peer dyads were not included given that these outcomes often reflect an aspect of child care quality.

Study Design
Cross-sectional and longitudinal designs were included.When multiple child outcome assessments were reported the earliest time-point following the measurement of quality were extracted.
To increase the homogeneity across the extracted data from eligible studies (i.e., increase the likelihood of meta-analysis), we focused on the earliest timepoint in which child outcomes were measured following the measurement of quality in instances where multiple waves of outcome data were presented.

Outcome Reporting
Studies must have presented statistical data quantifying the association between CLASS and a child outcome measure.
Studies only reporting qualitative results were not considered for this review as the domains of assessment could vary markedly between studies.

Language
To be extracted studies had to be in English.
We did not have resources to systematically translate material written in other languages.
Abbreviations  This paper is one of a series of "Meta-Analyses and Systematic Reviews" assessing the relationship between child care quality and children's outcomes; therefore, uppercase superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.covariate, or in which gain scores were used in analyses because such analyses required that children have spent a period of time in the program.In addition, only statistics that accounted for covariates (e.g., child and family characteristics) were combined within a single meta-analysis.Finally, to insure independence of samples, when multiple papers drew from the same dataset, only the study with the largest sample size was selected for inclusion in meta-analyses.Thus, within a given meta-analysis, only one coefficient from each sample was included in any one analysis.Statistical models with quadratic terms assume non-linear associations between the variables.Given the statistics extracted for most studies only tested for linear relationships (correlation coefficients and linear regression coefficients), associations in models using quadratic terms were excluded and only results examining linear relationships alone were used in the meta-analyses.We used random-effects models for meta-analyses.All meta-analyses were conducted using the software Comprehensive Meta-Analysis Version 3 (See S3 File) [68].We used the I 2 statistic [69] as an indicator of statistical heterogeneity across studies.Low I 2 values suggest homogeneity between studies.

Assessment of Risk of Biases Among Included Studies
We planned to assess risk of bias in included studies using Newcastle-Ottawa scale [22,70].However, initial review revealed that because of the correlational nature of research in this area, the standard questions on this checklist did not enable us to discriminate differences in study quality in our sample of studies.All included studies had moderate to high risk of bias due to issues with sample selection, exposure assessment and confounder adjustment.However, the inclusion criteria we used for the meta-analysis selected for the better quality studies.This means that the studies in the meta-analyses used well-researched, standardized measures, they tended to use covariates in their analyses and to ensure that the children in these studies had at least some exposure to the program before exploring whether program quality was associated with their outcomes.Finally, the majority of the papers we included were published in peer-reviewed journals.

Search Results
Details of the search and study selection are provided in Fig 1 .Thirty five studies were included in this review (consisting of 39 samples) [7,8,30,31,34,36,, which included 28 journal articles, 6 reports and 1 book chapter.
Of the independent samples, (i.e., excluding overlapping datasets by retaining only the largest sample size), the total sample size was 15,167 preschool-aged children.Sample sizes ranged from 129 to 3584 children (Median = 398).All of the studies had similar numbers of boys and girls (46% to 62% males).All but one study [58] sampled at-risk children, with 47% to 100% of the children coming from low-income families.Children were primarily Caucasian, Hispanic, or Black.

Outcomes
As noted above, outcomes were grouped into cognitive (for which Backward Digit was the most frequently used measure), language (where Peabody Picture Vocabulary Test, PPVT; Woodcock-Johnson Letter-Word Identification, WJ-LWI and the Oral and Written Language Scales, OWLS, are used most often), mathematics (where the Woodcock-Johnson Applied Problems, WJ-AP and Early Childhood Longitudinal Study Birth Cohort Math, ECLS-B math, are used most frequently) and social/emotional outcomes (where the Social Skills Rating System, SSRS, and Teacher Child Relationship Scale, TCRS, are used most often).The 55 different child outcomes identified through this process are listed in S2 File.

Systematic Review
Data for 55 different child outcome measures (see Table 2 where all studies are described and S2 File where all outcomes are listed) were reported in included studies.All of this data (which is presented in Tables A-D, S4 File) was included in the systematic review.To simplify this heterogeneity and help with visual interpretation, child outcomes that were reported in 3 or more samples are reported in Figs 2-7.Of the 35 eligible articles, 29 studies [7,8,30,31,34,36,[38][39][40][41][42][43][44][45]47,48,50,51,53,54,56,58,[60][61][62][63][64][65]67] provided data for at least one outcome variable that was also included in at least two other samples.Six studies [46,49,52,55,57,59] are not included in Figs 2-7 because they report only outcomes found in less than three samples.show relationships between domains and dimensions of the CLASS and outcomes in individual studies.Data is presented for 5 language outcomes, 3 mathematics outcomes and 5 social/emotional outcomes.For language outcomes, a high number of significant associations for the OWLS, PPVT, WJ Letter Word ID (and letter identification measures that were not part of the WJ) and WJ Rhyming were identified in the included studies.However, these results mostly came from studies using NCEDL database.Few significant associations remained when the studies that used the NCEDL [7,8,30,31,[39][40][41]43,44,47,48,50,56,58,62,63] dataset were excluded.Results from studies that used NCEDL data included in this systematic review should be interpreted with caution as the samples in these publications are drawn from a single dataset and therefore overlap.There were a few positive associations between CLASS scores and the math outcome of WJ Applied Problems but none with the ECLS-B Mathematics measure.Few positive significant associations were reported between the various CLASS domains and dimensions and measures of child outcomes.
We also explored whether any one domain or dimension of the CLASS had a stronger pattern of associations with any of our outcomes.We paid particular attention to CLASS domains and dimensions that were conceptually linked to specific child outcomes (e.g., Emotional Support to children's social outcomes, Instructional Support for Learning to language and mathematics or Language Modeling to language outcomes).Across domains and dimensions there were few associations identified and most of them used from NCEDL data.Overall a consistent lack of relationships between the CLASS and the various outcomes reported in the literature was observed.
We also explored whether accounting for covariates influenced the pattern of results.We did this by comparing results in studies that provided both a Pearson's correlation and either partial correlations, betas or unstandardized regression coefficients.Five studies [7,38,42,43,63] presented these statistics for the same operationalization of the CLASS and child outcome.Findings across these five studies were highly consistent indicating that when covariates were added there were fewer significant associations compared to the Pearson's correlations.The one exception to this was Sabol, 2013 [63] in which the significance level of most of the Pearson correlations was not reported.As a final check, we compared studies that used overlapping samples and found that results were highly consistent among them.
Classroom Organization.The meta-analysis (Fig 8) revealed significant but small correlations between the Classroom Organization domain and Pencil Tapping (N = 3757, pooled correlation coefficient 0.06; 95% CI: 0.04 to 0.09, I 2 = 0%).).The relationships between the  2 in this manuscript.a To improve the readability of this complex table, six papers [46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.Classroom Organization domain and PPVT Vocabulary (N = 3172), WJ LWI (N = 3826), WJ Applied Problems (N = 4976), and SSRS Social Skills (N = 3115) subscales were not significant.
Instructional Support.The meta-analysis (Fig 10) revealed significant but small correlation between the Instructional Support domain the SSRS Social Skills subscale (N = 3556, 0.09; 95% CI: 0.02 to 0.12, I 2 = 3.6%).The relationships between the Instructional Support domain and PPVT (N = 7740) and WJ Letter Word ID (N = 7146) were not significant.The relationship between Instructional Support and the WJ Applied Problems (N = 7531) was also not significant.

Discussion
The CLASS is a relatively new measure of ECEC quality that focuses on constructs that are supported by a long history of theory (e.g., attachment theory, ecological systems theory, etc.) and research on the kids of environments that support child development.The field has embraced the measure as evidenced by its growing use in Quality Ratings and Improvement Systems in the US.One of the appeals of the CLASS is that the measure developers have paid To improve the readability of this complex table, six papers [46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table 2 in  close attention to testing its psychometric properties.Thus, despite being a relatively new measure we found 35 studies that could be used in this review.The quick uptake of this measure in research studies allows us to draw some conclusions about the extent to which it is associated with child outcomes.
In this systematic review and meta-analyses, we found a limited number of significant associations between the CLASS and the language social outcomes available in the literature.This was especially true when the large number of studies that used the NCEDL data was accounted for.Meta-analyses of studies that used the same operationalization of the CLASS, had identical outcome measures and had statistics that could be converted to r revealed some relationships between the different domains of the CLASS and child outcomes.Of the 14 meta-analyses we conducted, showed significant, although small effect sizes.The low I 2 values found across these analyses suggest that the studies within each meta-analysis were homogeneous and support our conservative approach of only meta-analyzing studies that were quite similar to one another in terms of operationalization of the CLASS and child outcome.
The lack of relationships is especially surprising in cases in which the CLASS domains maps onto a specific child outcome measure most closely (e.g., Emotional Support and children's Social Skills measured using the SSRS).However, because the studies we were able to To improve the readability of this complex table, six papers [46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table 2 in  meta-analyze were limited, we did not have an opportunity to test some of these relationships (e.g., we were not able to meta-analyze the relationship between language modeling and children's vocabulary).Our finding that associations between the CLASS and child outcomes are limited likely reflects the multitude of family and other factors the impact the outcomes we explored in this paper.Below we discuss some of the methodological limitations of the research we covered.We then provide alternative explanations for the weak associations.
The literature in this area is problematic because: 1) it is heterogeneous and 2) many studies have methodological limitations.First, potential for selection bias in the included studies in this review is non-trivial as a number of studies did not randomly select programs and many programs and parents decline to participate.For example, 16 studies [7,8,30,31,[39][40][41]43,44,47,48,50,56,58,62,63] included in this review used data from the NCEDL database.The NCEDL is a large-scale study that draws on randomly selected programs from a number of diverse states.While this study is clearly an important one, it consists of a single sample.Approximately 78% of randomly selected programs in this study agreed to participate while 55-61% of parents from these programs agreed for their children to take part in the study.Thus, even in this relatively methodologically strong study, the final sample may not be representative of the general To improve the readability of this complex table, six papers [46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table 2 in  population of children.Moreover, sample self-selection may contribute to the low variability in CLASS scores observed.With few exceptions, the standard deviations reported for the CLASS domains and dimensions were smaller than 1.This self-selection may result in an over estimate of the quality of existing programs as better programs may be more willing to participate in research on program quality.It may be possible to achieve higher response rates in studies that are part of government initiatives to improve regulation of ECEC programs.Thus, researchers and government or other agencies involved in oversight of ECEC programs need to work together towards improving the quality of research in this area.In general parents who agree to participate in research tend to be better educated.This parent self-selection likely skews the types of children who participate in published studies.Another limitation of the samples used in these CLASS studies is that they all come from the United States.Clearly more research is needed to explore the role of the process quality, as captured in the CLASS, in other jurisdictions.
Second, scores on the domains and dimensions of the CLASS; though consistent across programs, were not always favorable.Scores for Instructional Support were consistently low with values at around "2" whereas Classroom Organization and Emotional Support scores were higher at around "5" on this 7-point scale.This indicates that Instructional Support is an area in particular need of quality improvement across programs.The low scores observed for some domains and dimensions are especially worrisome given our speculation that these are likely to be overestimates of quality due to sample self-section issues.
Finally, there was significant heterogeneity in the outcomes included in these studies.There are two versions of the CLASS, some researchers reported a total score, others created their own profiles from different CLASS codes.To maximize the number of studies we could draw To improve the readability of this complex table, six papers [46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table 2 in this manuscript.d Acronyms for child outcomes are listed in S5 File.e Identifying Letters (also referred to as Letter Knowledge, Letter-Naming, Naming Letters).A National Center for Early Development and Learning Dataset (NCEDL).doi:10.1371/journal.pone.0167660.g006from we attempted to "streamline" the CLASS scales as much as possible by equating domains that were made up of very similar dimensions and including studies that provided data on some dimensions only.There was also substantial variability in the statistics that were reported which further limited our ability to meta-analyze findings across papers.Finally, there is enormous variability in the covariates used in different studies.For example, some studies reported zero order correlations using no covariates [31], while other studies reported only partial correlations or regressions that included anywhere between 5 [39] and 22 [34] covariates controlling for child, family and program factors in their analyses [32].Clearly studies in this area need to be conducted and reported more consistently so that findings can be integrated more effectively in the future.
Studies about the effects of child care generally have some methodological limitations.For example, for ethical as well as logistical reasons, it is not possible to randomly assign children to programs of differing quality.As a result, all of the studies we included are observational/ correlational and causal conclusions about the link between the CLASS and child outcomes cannot be drawn.Reporting of study methodologies and results is inconsistent and at times incomplete.For example, a measure of variability was not reported in a number of studies [7,55,[65][66][67].However, it is worth noting that the studies included in our meta-analyses used psychometrically strong measurements, had large sample sizes, included many covariates in the analyses in an effort to mitigate the program and parent self-selection issues discussed To improve the readability of this complex table, six papers [46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table 2  above, and ensured that children who were assessed had been exposed to the programs.Thus, while the studies we included suffered from substantial limitations, they also had many strengths.
One major drawback of this heterogeneity and the number of studies available for metaanalysis is that we could not test the effects of possible moderators on the relationship between the CLASS and child outcomes.Variables of particular interest are whether the duration of exposure and socioeconomic backgrounds of children influence associations between the CLASS and child outcomes.In general, ECEC programs are expected to make the greatest impact on children from low SES backgrounds.While we were not able to test SES as a moderator, the studies we included generally over represented lower income children.Future reviews based on additional studies to be published in the future in this area, will hopefully be able to test these issues more directly.
Other explanations for the small effect sizes found in this review include the possibility that higher quality programs only impact children who require especially supportive environments.The differential susceptibility hypothesis [74] posits that children who are more vulnerable for genetic, temperamental or other reasons, may be differentially impacted by the extent to which their surroundings are supportive.Thus, overall effects may be eliminated/reduced when aggregating across diverse children within a given sample.Another possible explanation is that the CLASS measures quality at the classroom level, while classrooms in the early childhood sector are staffed by multiple adults.Measures like the Caregiver Interaction Scale [75], which are collected at the staff level, are worth exploring.However, these generally focus only on warmth and sensitivity without the emphasis on developmentally appropriate instruction offered by the CLASS.Discrepancies in the unit of analysis may be contributing to the lack of associations reported in this review.In addition, it is possible that other aspects of interaction should be attended to.For example, Hamre et al. (2013) [14] mention the "use of emotion words and emotion coaching" as "unique elements of classroom interactions not measured by the CLASS" (p.464).Alternatively, child outcomes might be better predicted by more comprehensive measures such as the ECERS-R described earlier, that capture both structural and process quality.In addition, while we set out to collect a broad range of child outcomes, we could only report on outcomes that were found in the literature.Some outcomes (e.g., gross and fine motor abilities) are notably absent and could explain the lack of associations observed in this study.
Future studies should look at the presence of non-linear relationships between quality characteristics and child outcomes, as there may be some threshold value in ECEC quality that changes the direction of the association.Such relationships cannot be adequately captured by statistical indices that assume linear relationships reported here.The possibility that there are thresholds of quality in terms of impacts on children is gaining attention from researchers.This is illustrated by the recent publication of a special issue on this topic [76].However, one study that tested this possibility directly with the CLASS did not provide support for the threshold hypothesis for this measure [77].Finally, one of the strengths of this study is that we cast a very wide net in terms of our initial searches.This resulted in many "hits" that were not, in the end, relevant to the goals of this study.We then developed and systematically applied a set of selection criteria with the aim of having a relatively homogenous sample of studies for subsequent analysis.The downside of this approach is that it was very labour intensive.The upside is that we can be more confident that our searches were comprehensive.
As described earlier, the CLASS is being used increasingly as part of quality assurance programs.It is also used as an integral part of professional development (PD) programs where with the help of a coach, caregivers use the CLASS as a lens through which staff co-view videos of their own interactions with children to improve their practice.The logic of such PD programs stems from the rich literature that formed the basis for approaches like Teaching Through Interaction and informed the development of the CLASS.Yet, given the results of the current systematic review and meta-analysis, clearly more research about what aspects of the classroom environment are associated with child outcomes is needed.

Conclusion
The CLASS operationalizes difficult constructs such as scaffolding and contingent responding by ECEC staff, constructs that are thought to be key in supporting children's development.The CLASS domain that was most closely linked to child outcomes was Instructional Support.Sadly, it is also, by far, the lowest scoring domain across studies.Very few effects were found for Classroom Organization and Emotional Support.However, it is important to note that based on these findings we cannot conclude that programs of lower or higher quality than what we found in the literature would not impact children.As noted above, it may well be that there are threshold/levels of quality that must be met for quality to impact children.One of the most important conclusions of this study is that more research, using stronger methodologies and a variety of samples is needed.Research with samples from outside the US is especially important.However, despite all of the methodological limitations discussed above, there is a growing body of research about associations between the CLASS and child outcomes and it is important to systematically review the existing literature about this influential measure.Unfortunately, based the existing literature, it appears that associations between the CLASS and child outcomes are quite limited.
Abbreviations: NR = Not Reported; C = Caucasian, B = African American, H = Hispanic, A = Asian, M = Mixed, O = Other.For all other acronyms, please refer to S5 File. a Descriptives provided reflect characteristics (actual or estimates) of the sample/research design for which data was extracted for the current study and therefore may represent a subsample/analysis of the larger study.b

Fig 2 .
Fig 2. Systematic review of associations between the CLASS Total Score or Dimension and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.Total Score (Two Factor Solution) = Total Score for Emotional Climate and Instructional Climate; Total Score (9 Scale Version) = Total Score with Language Modeling and Literacy Focus dimension not included.For more details, see Table2in this manuscript.a To improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.

c
Samples within papers are described in more detail in Table2in this manuscript.d Acronyms for child outcomes are listed in S5 File.e Identifying Letters (also referred to as Letter Knowledge, Letter-Naming, Naming Letters).f SSRS/ SSIS problem behaviour also includes individual scales: internalizing and externalizing for Hestenes et al., 2015 [54].A National Center for Early Development and Learning Dataset (NCEDL); B Head Start Family and Children Experiences Survey (FACES 2006 Cohort); M Head Start Family and Children Experiences Survey (FACES, 2009 Cohort); U Preschool Curriculum Evaluation Research (PCER, 1999-2003); X Ohio Virginia (2005-2006, 2006-2007).doi:10.1371/journal.pone.0167660.g002

Fig 3 .
Fig 3. Systematic review of associations between the CLASS domain of Classroom Organization and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2in this manuscript.d Acronyms for child outcomes are listed in S5 File.
Fig 3. Systematic review of associations between the CLASS domain of Classroom Organization and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2 inthis manuscript.d Acronyms for child outcomes are listed in S5 File.e Identifying Letters (also referred to as Letter Knowledge, Letter-Naming, Naming Letters).f SSRS/SSIS problem behaviour also includes individual scales: internalizing and externalizing for Hestenes et al., 2015 [54].A National Center for Early Development and Learning Dataset (NCEDL); M Head Start Family and Children Experiences Survey (FACES, 2009 Cohort).doi:10.1371/journal.pone.0167660.g003

Fig 4 .
Fig 4. Systematic review of associations between the CLASS domain of Emotional Support and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2in this manuscript.d Acronyms for child outcomes are listed in S5 File.
Fig 4. Systematic review of associations between the CLASS domain of Emotional Support and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2 inthis manuscript.d Acronyms for child outcomes are listed in S5 File.e Identifying Letters (also referred to as Letter Knowledge, Letter-Naming, Naming Letters).f SSRS/SSIS problem behaviour also includes individual scales: internalizing and externalizing for Hestenes et al., 2015 [54].A National Center for Early Development and Learning Dataset (NCEDL); M Head Start Family and Children Experiences Survey (FACES, 2009 Cohort).doi:10.1371/journal.pone.0167660.g004

Fig 5 .
Fig 5. Systematic review of associations between the CLASS domain of Instructional Support and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2in this manuscript.d Acronyms for child outcomes are listed in S5 File.
Fig 5. Systematic review of associations between the CLASS domain of Instructional Support and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2 inthis manuscript.d Acronyms for child outcomes are listed in S5 File.e Identifying Letters (also referred to as Letter Knowledge, Letter-Naming, Naming Letters).f SSRS/SSIS problem behaviour also includes individual scales: internalizing and externalizing for Hestenes et al., 2015 [54].A National Center for Early Development and Learning Dataset (NCEDL); B Head Start Family and Children Experiences Survey (FACES 2006 Cohort); M Head Start Family and Children Experiences Survey (FACES, 2009 Cohort).doi:10.1371/journal.pone.0167660.g005

Fig 6 .
Fig 6.Systematic review of associations between the CLASS domain of Emotional Climate and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2in this manuscript.d Acronyms for child outcomes are listed in S5 File.e Identifying Letters (also referred to as Letter Knowledge, Letter-Naming, Naming Letters).A National Center for Early Development and Learning Dataset (NCEDL).

Fig 7 .
Fig 7. Systematic review of associations between the CLASS domain of Instructional Climate and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2in this manuscript.d Acronyms for child outcomes are listed in S5 File.
Fig 7. Systematic review of associations between the CLASS domain of Instructional Climate and child outcomes.a Abbreviations: Symbols bolded are significant and positive, symbols bolded and italicized are significant and negative, and symbols in grey are non-significant.Star = Zero Order Pearson's Correlation, Unfilled circle = Beta, Filled square = Unstandardized Coefficient, Black diamond minus white X = T-Test, Key clover = Partial Correlation, Downward arrow = Effect Size, Filled circle = F-Ratio.aTo improve the readability of this complex table, six papers[46,49,52,55,57,59] that had an outcome that appeared in only one or two samples were omitted from this figure.Several analyses from other papers that had idiosyncratic outcomes were also excluded.For a comprehensive display of all of the data for all of the child outcomes see Tables A-D, S4 File.b This paper is one of a series of Meta-Analyses and Systematic Reviews assessing the relationship between child care quality and children's outcomes; therefore, superscript letters below are in reference to various large databases that samples in these papers were drawn from.These letters have been kept consistent across the series of papers for our readers.c Samples within papers are described in more detail in Table2 inthis manuscript.d Acronyms for child outcomes are listed in S5 File.e Identifying Letters (also referred to as Letter Knowledge, Letter-Naming, Naming Letters).A National Center for Early Development and Learning Dataset (NCEDL).doi:10.1371/journal.pone.0167660.g007

Fig 8 .
Fig 8. Meta-analyses of associations between the CLASS domain of Classroom Organization and child outcomes.doi:10.1371/journal.pone.0167660.g008

Fig 9 .
Fig 9. Meta-analyses of associations between the CLASS domain of Emotional Support and child outcomes.doi:10.1371/journal.pone.0167660.g009