Introduction

Systematic research on the diagnostic assessment of mental health in preschool children has lagged considerably behind that in school-age children, adolescents, and adults [1]. However, longitudinal studies indicate that early psychopathology can show meaningful continuity across development and is associated with adverse outcomes later in life [2–5]. Therefore, early identification of clinically relevant symptoms is important for timely diagnostics and age-appropriate evidence-based treatment [1].

Structured clinical interviews are important diagnostic instruments that can also be used as part of the diagnostic process in preschool children. Such interviews not only represent the gold standard for diagnosing mental disorders in different age groups [6–8] but also foster the accuracy and reliability of diagnoses [9–11]. They are employed both in research and increasingly also in clinical practice as part of a comprehensive and standardized diagnostic process [7,9,11]. Furthermore, as these interviews systematically cover diagnostic criteria, they are also beneficial for clinicians in training [9–11].

Clinical interviews can be categorized into unstructured, semi-structured, and highly structured. Typically, highly structured interviews follow a categorical approach, given their purpose of establishing categorical diagnoses [9–11]. While requiring little training, they offer limited flexibility for the interviewer to explore and rate the patient’s symptoms, and usually assess symptoms dichotomously (i.e., present or absent) using closed-ended questions [9–11]. Semi-structured interviews, by contrast, allow the interviewer to seek information about symptoms, make informed judgments, and score the patient’s responses more flexibly (e.g., using Likert-type scales). This scoring format necessitates more extensive training but enables a dimensional approach by considering the severity of symptoms [9,10,12].

For the assessment of mental disorders in childhood and adolescence, both clinical parent and patient interviews are available, with the former generally used for preschool children. Preschool versions are modified and adapted for young children, with age-appropriate questions to identify symptoms typical for this age group [13]. The NIMH Diagnostic Interview Schedule for Children Version IV (DISC-IV) [14] is a highly structured clinical interview that was originally developed for children and adolescents. The DISC-IV follows a categorical approach and is designed to assess more than 30 psychiatric disorders. However, due to its closed-ended questions, the interview provides limited flexibility. A developmentally adapted parent version for young children, the Diagnostic Interview Schedule for Children–Young Child Version (YC-DISC-IV) [15], was developed for children aged 3–8 years based on the parent version of the DISC-IV. The YC-DISC-IV includes developmentally appropriate adaptations, such as modified wording and a shorter assessment timeframe. To date, the YC-DISC-IV does not appear to have been formally published as a finalized instrument, and published psychometric evidence remains limited to preliminary validation of the section assessing DSM-IV [15] criteria for major depression in a sample of preschool children [16]. The Preschool Age Psychiatric Assessment (PAPA) [17] is a structured clinical parent interview based on the Child and Adolescent Psychiatric Assessment (CAPA) [18], which covers a wide range of mental disorders and follows both a categorical and a dimensional approach. The PAPA was evaluated in a sample of 307 caregivers of children aged 2–5 years, who were recruited from a general pediatric clinic [19]. The test-retest reliability was found to be substantial for attention-deficit/hyperactivity disorder (ADHD; κ = .74) and conduct disorder (CD; κ = .60), and moderate for oppositional defiant disorder (ODD; κ = .57).

Regarding semi-structured clinical interviews, the Diagnostic Infant and Preschool Assessment (DIPA) [20] covers symptoms of thirteen disorders as well as disorder-specific functional impairment. The DIPA was evaluated in a sample of 50 caregivers of 1–6-year-old children recruited from mental health clinics specialized for this age group [20]. The test-retest reliability was found to be moderate for ADHD (inattentive type: κ = .58; hyperactive-impulsive type: κ = .42) and fair for ODD (κ = .38). Convergent validity was supported by overall moderate correlations (ADHD inattentive type: r = .56, ADHD hyperactive-impulsive type: r = .59, ODD: r = .55) between the DIPA scores and the corresponding symptom scales of the DSM-based Child Behavior Checklist (CBCL/1½−5) [20]. Moreover, the DIPA scale scores showed good to excellent internal consistency in preschool children (ADHD inattentive type: α = .87, ADHD hyperactive-impulsive type: α = .84, ADHD combined type: α = .91, ODD: α = .89) [21]. The interrater agreement for categorical diagnoses (i.e., present vs. absent) was substantial to perfect (ADHD inattentive type: κ = .75, ADHD hyperactive-impulsive type: κ = 1.00, ADHD combined type: κ = .75, ODD: κ = 1.00), and the interrater reliability (IRR) for continuous variables (i.e., number of symptoms) was almost perfect (intraclass correlation coefficient [ICC]: ADHD inattentive type: ICC = .98, ADHD hyperactive-impulsive type: κ = .98, ADHD combined type: κ = .99, ODD: κ = .98). Furthermore, the DIPA scale scores showed good convergent and adequate divergent validity [21]. Although the Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version (K-SADS-PL) [22] does not have an explicit preschool version, Birmaher, Ehmann [23] evaluated the K-SADS-PL in a sample of 204 caregivers of 2–5-year-old children. The analyses yielded good convergent validity between the CBCL/1½−5 subscales and corresponding categorical DSM-IV diagnoses assessed using the K-SADS-PL. Additionally, ODD and ADHD diagnoses based on the K-SADS-PL showed significant correlations with the Externalizing Problems scale of the CBCL (rho = .41 and  .43). Divergent validity was supported by negligible/small correlations between the CBCL Externalizing Problems scale and anxiety disorders (rho = .15), mood disorders (rho = .19), and emotional disorders (rho = .20). Furthermore, the IRR was good to excellent (.70 ≤ κ ≤ .86) for all diagnoses [23].

Overall, existing semi-structured clinical interviews provide a reliable and valid tool for diagnosing mental disorders in preschool children. However, debate about whether mental disorders should be viewed as categorical or dimensional concepts is still ongoing [12,24–26]. Consequently, state-of-the-art assessment instruments should be sufficiently flexible to allow for both a categorical and a dimensional assessment, thus providing further information on symptom severity or subclinical symptoms, which can serve as potential prevention targets [12,24]. Moreover, assessment instruments should assess not only clinical symptoms but also functional impairment and psychological stress associated with these symptoms [27,28]. Importantly, as it is unlikely that one single informant is sufficiently privy to a child’s situation-specific behavior, e.g., at home or in preschool, a key component of best practice in evidence-based clinical child assessment is the integration of reports by multiple informants (e.g., clinicians, parents, teachers) [12,25,29–31]. Therefore, diagnostic instruments that systematically combine the assessment of different rater perspectives within the framework of a multi-informant approach should be preferred [12].

To meet the above criteria, we developed a comprehensive set of clinical parent and patient interviews, the Interviews according to the DSM-5 Diagnostic System of Mental Disorders for Children and Adolescents (DISYPS-ILF; German: Interview-Leitfäden zum Diagnostik-System für psychische Störungen nach DSM-5 für Kinder und Jugendliche) [32]. The DISYPS-ILF are part of the German Diagnostic System of Mental Disorders in Children and Adolescents based on the ICD-10 and DSM-5 (DISYPS-III; German: Diagnostik-System für psychische Störungen nach ICD-10 und DSM-5 für Kinder und Jugendliche) [33]. The DISYPS-III also includes corresponding parent-, teacher-, and (for children older than 10 years) self-report questionnaires, allowing for a comparison of ratings from parents, caregivers/teachers, and patients with clinical judgments based on interviews.

One of the DISYPS-ILF interviews covers the diagnostic criteria for externalizing disorders according to the DSM-5 (see Materials and Methods for further details): the Clinical Parent Interview for Externalizing Disorders in Children and Adolescents (ILF-EXTERNAL; German: Interview-Leitfaden für Externale Störungen). Along with a categorical assessment, this interview also enables a dimensional perspective on clinical symptoms. The psychometric properties of the ILF-EXTERNAL were comprehensively evaluated in a clinical sample of children aged 6;0–11;11 years within the ESCAschool multicenter study, with the results revealing good to excellent IRR on both the item and scale level (.83 ≤ ICC(1,1) ≤  .95) and satisfactory to good internal consistencies (.60 ≤ α ≤ .86) [34]. Moreover, overall agreement on DSM-5 diagnoses was assessed using Fleiss’ kappa, which yielded substantial to almost perfect values (.38 ≤ κ ≤ .94). Please note that κ coefficients reported for the ILF-EXTERNAL reflect interrater reliability (IRR), whereas κ coefficients reported for the PAPA reflect test-retest reliability. Therefore, these coefficients should not be directly compared as indicators of the instruments‘ reliability. With regard to the factor structure of the ILF-EXTERNAL, the basic factorial configuration of the scale scores were confirmed [35]. In terms of measurement invariance, the same organization of factor and loading patterns, but not of item thresholds, was found across clinician, parent, and teacher ratings, underlining cross-situational variability in child behavior [35]. Furthermore, as functional impairment is often considered as a prerequisite for making a diagnosis, Thöne, Dose [36] examined associations between individual symptoms and global functional impairment, and found significant variations; for example, the ADHD symptom Easily Distracted explained approximately five times more variance than the ADHD symptom Careless, even though both of these symptoms originate from the ADHD inattention dimension. A recent study, which was conducted online via video chat as part of the INTEGRATE-ADHD research project (N = 202, age: M = 12.87 years, SD = 3.04, 28.2% female) [37], examined the psychometric properties of the ADHD section of the ILF-EXTERNAL, and found good to excellent internal consistency for the ADHD symptom scales (α = .89 to  .93) and high IRR for both the categorical and dimensional analyses (κ = .78 to  .81; ICC(1,1) =  .97 to  .98). Moreover, convergent validity was confirmed by strong correlations between the parent interview and both the German Symptom Checklist for Attention-Deficit/Hyperactivity Disorder (FBB-ADHS) (r = .79 to  .85) and the child interview (r = .60 to  .71). However, the extent to which these psychometric results can be replicated in a clinical sample of preschool children remains unclear.

The overall aim of this study was to evaluate the psychometric properties and factor structure of the clinical parent interview ILF-EXTERNAL in a clinical sample of 3–6-year-old preschool children with ADHD symptoms and externalizing behavior problems. This approach includes (1) testing the underlying factor structure using confirmatory factor analysis (CFA), (2) descriptive statistics for all scales, (3) internal consistencies and item-total correlations, (4) IRR, (5) overall agreement on DSM-5 diagnoses, and (6) convergent and divergent validity.

Materials and methods

Participants and procedure

Data collection was based on the ESCApreschool intervention study, which is part of the research consortium ESCAlife (Evidence-Based Stepped Care of ADHD along the Lifespan) and involved nine study centers located in Germany (Marburg, Aachen, Cologne, Göttingen, Hamm/Bochum, Mannheim, Neuruppin, Tübingen, Würzburg). Recruitment commenced on February 1, 2016, at the study centers in Marburg, Mannheim, Bochum, Tübingen, and Würzburg. The Cologne site initiated recruitment slightly later, on March 14, 2016. Subsequently, three additional centers, Neuruppin (June 26, 2017), Aachen (October 26, 2017), and Göttingen (November 30, 2017), also began recruitment. The final day of recruitment across all study centers was January 31, 2020. The ESCApreschool study aims to investigate the efficacy of an individualized, stepwise-intensifying treatment program based on behavioral and pharmacological interventions for 3–6-year-old preschool children with an ADHD diagnosis according to the DSM-5 or with a diagnosis of ODD plus additional substantial ADHD symptoms [38].

Families were eligible for participation in the ESCApreschool study if the children fulfilled the following inclusion criteria: (1) age between 3;0 and 6;11 years, (2) attending preschool, (3) exhibiting externalizing behavior problems, that is, either meeting DSM-5 criteria for ADHD on the ILF-EXTERNAL or meeting DSM-5 criteria for ODD on the ILF-EXTERNAL plus substantial attention or hyperactivity/ impulsivity problems (score ≥ 0.70 on either the Inattention or the Hyperactivity-Impulsivity scale of the ILF-EXTERNAL), (4) informed consent of the parents (and, where applicable, of the preschool teachers), and assent of the child. The following exclusion criteria were applied: (1) child’s IQ < 80, (2) a diagnosis of pervasive developmental disorder, schizophrenia, bipolar disorder, or severe depressive episode, (3) insufficient German-language skills of parents, (4) psychotropic medication of the child (except for ADHD medication). and (5) child’s participation in a regular intensive behavior therapy. Further details on the background and procedures are outlined in the ESCApreschool study protocol [38]. All participating parents and preschool teachers provided written informed consent and participating children gave their assent for participation.

In the present study, we used the screening dataset of the ESCApreschool study. This dataset comprised all children for whom screening data were available and whose families provided informed consent for the scientific use of their data. The eligibility criteria described above applied to participation in the ESCApreschool study and were not used as selection criteria for the present analyses. The ILF-EXTERNAL was evaluated using data from 211 children (age range 3–6 years, M = 5.12, SD = 0.84, 62% males; for more details see Table 1), including 190 who were enrolled in the ESCApreschool study [cf. 38]. The sample additionally included 21 children who did not meet all eligibility criteria of the ESCApreschool study (i.e., so-called screening negatives; an overview of violated inclusion and exclusion criteria can be found in S1 Table). These children were retained because the present analyses were based on all available screening data and were not restricted to children eligible for participation in the intervention study. Their inclusion resulted in a more heterogeneous sample, including a broader range of externalizing symptom severity and greater variability in cognitive functioning, as children with IQ scores below 80 were not excluded. Table 2 presents descriptive statistics (M, SD) for all ILF-EXTERNAL scales. Clinical diagnoses of ADHD and comorbid externalizing disorders were based on the results of the ILF-EXTERNAL. The ILF-EXTERNAL baseline data were collected during screening procedure of the ESCApreschool study. The screening included an assessment of eligibility for participation in the intervention study and took place during two appointments conducted no more than eight weeks apart. Interviewers typically performed the first part of the interview (assessment of ADHD and ODD symptoms) during the first appointment and the second part (assessment of CD symptoms) during the second appointment. The ILF-EXTERNAL was conducted with the parents and was recorded by audio or video. In order to assess comorbid symptoms, clinicians applied a clinical diagnostic checklist (DCL-SCREEN) from the DISYPS-III [33].

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Scale characteristics, Cronbach’s alpha (α), McDonald’s omega (ω), and Range of Item-Total Correlations of the ILF-EXTERNAL in the ESCApreschool Screening Dataset.

https://doi.org/10.1371/journal.pone.0357785.t002

Subsample for the analysis of interrater reliability

To assess IRR, a subset of 129 interview recordings of the ILF-EXTERNAL conducted at baseline was randomly drawn (see Table 1 for the characteristics of this subsample). To be included in this subset, an audio or video recording for both the ADHD and ODD/CD parts of the ILF-EXTERNAL had to be available, with sufficient audio quality.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Sample characteristics.

https://doi.org/10.1371/journal.pone.0357785.t001

Results

Confirmatory factor analysis

The goodness-of-fit indices for the ILF-EXTERNAL factor models are summarized in Table 3. For our unidimensional model, the model fit indices did not meet the aforementioned cut-off criteria, resulting in a relatively poor model fit. However, the model with four correlated factors showed a generally acceptable model fit (RMSEA = .04, SRMR = .09), all standardized CFA factor loadings were significant, and λ ≥ 0.27 (Fig 1). Examining the factor correlations of the first-order CFA model with four correlated factors revealed moderate to high significant correlations between the Inattention and the Hyperactivity-Impulsivity factors (r = .51), and between ODD and CD (r = .76). ODD also showed a significant but low correlation with Inattention (r = .32) and with Hyperactivity-Impulsivity (r = .36). The correlations between CD and Inattention and between CD and Hyperactivity-Impulsivity remained non-significant. All remaining factor models tested revealed model fit indices that were inferior to those of the model with four correlated factors and did not meet the criteria for an acceptable fit. Although the bifactor CFA showed slightly lower AIC than the first-order CFA model with four correlated factors (Δ = 9.27), the latter showed a lower BIC (Δ = 79.23), indicating a more parsimonious solution. Given its acceptable model fit and direct correspondence to the DSM-5-based symptom domains assessed by the ILF-EXTERNAL, the first-order model with four correlated factors was considered the most appropriate representation of the data. Additional findings regarding the factor loadings and (co-)variance structure of the remaining factor models can be found in the supporting information (S1-S4 Figs).

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Goodness-of-fit statistics and information criteria for the CFA models.

https://doi.org/10.1371/journal.pone.0357785.t003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Standardized factor loadings and covariance structure of the first-order CFA with four correlated factors.

Factor descriptions: ina = Inattention; hi = Hyperactivity-Impulsivity; odd = Oppositional Defiant Disorder; cd = Conduct Disorder. Item descriptions: ad_a1 = Careless; ad_a2 = Difficulties Sustaining Attention; ad_a3 = Does not Listen; ad_a4 = Does not Finish Work; ad_a5 = Organizational Skills; ad_a6 = Concentration; ad_a7 = Loses Things; ad_a8 = Easily Distracted; ad_a9 = Forgetful; ad_b1 = Fidgets; ad_b2 = Leaves Seat; ad_b3 = Runs, Climbs; ad_b4 = Difficulties Playing Quietly; ad_b5 = Driven, on the go; ad_b6 = Blurts out Answers; ad_b7 = Waits for Turn; ad_b8 = Interrupts, Intrudes; ad_b9 = Talks Excessively; ss_a1 = Loses Temper; ss_a2 = Touchy, Easily Annoyed; ss_a3 = Angry, Resentful; ss_a4 = Argues with Adults; ss_a5 = Refuses to Comply with Requests; ss_a6 = Annoys; ss_a7 = Blames Others; ss_a8 = Spiteful, Vindictive; ss_b1 = Physical Fights; ss_b2 = Bullies, Threatens, or Intimidates; ss_b3 = Cruel to Animals; ss_b4 = Lies; ss_b5 = Steals without Confrontation; ss_b6 = Uses Weapons in Fight; ss_b7 = Cruel to People. Due to floor effects, eight items that measured aggressive and antisocial behavior were excluded from the analyses. *p < .05, ***p < .001.

https://doi.org/10.1371/journal.pone.0357785.g001

Interrater reliability

Table 4 provides the IRR of the ILF-EXTERNAL ADHD and ODD scales, based on the ICC one-way random-effects, absolute agreement model for single ICC(1,1) and average measures ICC(1,2) as well as their respective 95% confidence intervals. Regarding the scales ADHD Symptoms [ICC(1,1) =  .52] and ADHD Inattention [ICC(1,1) =  .58], the ICC(1,1) indicate fair or moderate IRR according to Cicchetti [55] or Koo and Li [53]. For the ADHD Hyperactivity-Impulsivity scale [ICC(1,1) =  .60] and the ODD Symptoms scale [ICC(1,1) =  .74], the IRR was good [55] or rather moderate [53].The ICC(1,2) of the ADHD Symptoms scale [ICC(1,2) =  .68] and the ADHD Inattention scale [ICC(1,2) =  .73] indicated good IRR according to Cicchetti [55] and moderate IRR according to Koo and Li [53]. For the ADHD Hyperactivity-Impulsivity scale [ICC(1,2) =  .75] and the ODD Symptoms scale [ICC(1,2) =  .85], the IRR was excellent [55] or rather good [53].

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Interrater reliability of the ILF-EXTERNAL scales.

https://doi.org/10.1371/journal.pone.0357785.t004

Agreement on DSM-5 diagnoses

Table 5 presents the overall agreement regarding DSM-5 diagnoses assessed using Cohen’s kappa values and their corresponding 95% confidence intervals. According to the benchmarks of Landis and Koch [57], agreement on categorical diagnoses was moderate (ADHD combined type & ADHD inattentive type: κ = .44; ADHD hyperactive-impulsive type: κ = .52; ODD: κ = .52).

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Interrater agreement on DSM-5 diagnoses in the subsample (n = 129) for the analysis of interrater reliability.

https://doi.org/10.1371/journal.pone.0357785.t005

Convergent and divergent validity

Table 6 presents the correlations between the ILF-EXTERNAL scales and the corresponding scales of the parent forms (FBB-ADHS-V; FBB-SSV). Pearson correlations were moderate to high and significant (.46 ≤ r ≤ .69, p < .001), indicating acceptable to good convergent validity between clinical judgment and parent ratings. Ratings on the ILF-EXTERNAL scales ADHD Functional Impairment, DMDD, LPE, and ODD/CD Functional Impairment differed significantly from ratings on the corresponding scales of the parent forms (p ≤ .01), with higher mean scale scores on the parent forms for the LPE, ODD/CD Functional Impairment, ADHD Functional Impairment, and DMDD scales compared to clinician ratings.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Comparisons of the ILF-EXTERNAL scales and the corresponding parent forms (FBB-ADHS-V and FBB-SSV).

https://doi.org/10.1371/journal.pone.0357785.t006

Table 7 presents the correlations between the ILF-EXTERNAL scales and the corresponding scales of the proxy rating forms completed by preschool teachers (FBB-ADHS-V; FBB-SSV). Pearson correlations were negligible to moderate (.05 ≤ r ≤ .33). Furthermore, the mean scale scores on the preschool teacher ratings differed significantly (p < .01) from the corresponding clinician ratings (with the exception of Inattention, ODD/CD Symptoms-Short Version, and ODD Symptoms-Short Version, DMDD). Overall, clinicians’ mean scale scores were significantly higher for ADHD hyperactivity-impulsivity symptoms (p < .01), whereas preschool teachers provided higher ratings for the majority of the disruptive behavior symptoms (p < .01).

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Comparisons of the ILF-EXTERNAL scales and the corresponding preschool teacher forms (FBB-ADHS-V and FBB-SSV).

https://doi.org/10.1371/journal.pone.0357785.t007

Table 8 provides a summary of the Pearson correlations of the ILF-EXTERNAL scales with the seven CBCL/1½−5 syndrome scales and with the CBCL/1½−5 broadband scales Externalizing Problems and Internalizing Problems. On the whole, the ILF-EXTERNAL scales showed significant, moderate to high correlations with the CBCL Externalizing Problems scale (.38 ≤ r ≤ .56). Comparison of the dependent correlations using Steiger’s test revealed that all ILF-EXTERNAL scales, except the Inattention Symptoms subscale, showed significantly stronger associations with the CBCL Externalizing Problems scale than with the Internalizing Problems scale (.001 ≤ p ≤ .018). Moreover, the CBCL Externalizing Problems scale showed the highest correlations with the ILF-EXTERNAL scales ODD/CD Symptoms – Short Version (r = .40) and CD Symptoms – Short Version (r = .56).

Download:

• PNG
  larger image
• TIFF
  original image

Table 8. Correlations of the ILF-EXTERNAL scales and the child behavior checklist (CBCL/1½−5)scales.

https://doi.org/10.1371/journal.pone.0357785.t008

Discussion

This study evaluated the psychometric properties and factor structure of the DSM-5-based, semi-structured, clinical parent interview ILF-EXTERNAL using data from the ESCApreschool screening data set, which comprised preschool children aged 3–6-years screened for participation in the ESCApreschool multicenter study targeting preschool children with externalizing symptoms [38]. A notable strength of the ILF-EXTERNAL is its ability to provide both categorical and dimensional assessments.

In terms of the factor structure of the ILF-EXTERNAL, which has rarely been analyzed in previous studies using interviews, a first-order factor model with four correlated factors (Inattention, Hyperactivity-Impulsivity, ODD, CD) showed the best model fit and was straightforward to interpret. Notably, the intercorrelations between the factors in this model were fairly high, especially between the ODD and the CD factor both in the presented study (r = .76) and in a conceptually similar study with elementary school children [35; r = 0.75]. Despite this high correlation, 42% of the variance remained unshared, suggesting that the CD and ODD factors represent two closely related but distinct factors. The better fit of the first-order four-correlated-factor model compared to the unidimensional and the first-order two-factor model further supports this interpretation. Moreover, the consideration of CD and ODD as distinct dimensions aligns with their theoretical conceptualization: While ODD is primarily understood as a disorder of emotional regulation (irritability, anger, frustration tolerance), CD is a disorder characterized by serious antisocial and aggressive behavior [59,60]. Longitudinal data from the Great Smoky Mountains Study showed that although ODD in school age is a risk factor for later CD in boys, transitions from ODD to CD were less common than assumed [61]. In young adulthood, CD symptoms primarily predicted later delinquent and antisocial behavior, while ODD symptoms were more strongly associated with later anxiety and depressive disorders [61]. Population samples with a longitudinal design tended to show parallel, only partially interrelated developmental trajectories [61,62]. In summary, the amount of unshared variance, the superiority of the four-correlated-factor model over alternative models, and the alignment with theoretical considerations and empirical findings from previous studies suggest that ODD and CD, despite their high correlation and shared risk factors, represent two distinct, albeit closely related, disorders. In terms of the factor structure of ADHD, the presented findings correspond to previous literature and the DSM-5 / ICD-11, insofar as ADHD inattention and hyperactivity-impulsivity dimensions are considered as two different but related constructs [63]. On the whole, the item-factor configuration of the ILF-EXTERNAL was confirmed in a clinical sample of preschool children.

With regard to scale reliability, Cronbach’s alpha and McDonald’s omega coefficients for the ILF-EXTERNAL scales were generally acceptable to good, similarly to the findings for the DIPA scale [21]. An exception is the CD Symptoms – Short Version scale (α = .69, ω = .73), where Cronbach’s alpha fell just below the threshold of acceptance, but McDonald’s omega showed acceptable reliability. Similar internal consistencies of CD Symptoms scales have been reported by Frick, Barry [9] for the DISC-IV version 2.3 [α =  .59, 9] and in our own study examining psychometric properties of the ILF-EXTERNAL in school-age children [34]. We believe that this rather low internal consistency is unsurprising for several reasons: First, we excluded the items B08 to B15, which assess antisocial symptoms, as they are typically only relevant for older children. This resulted in a shorter scale comprising only seven items, with less variance. Second, the scores of the remaining items of this shortened scale showed a low mean (M = 0.44; SD = 0.43) and a skewed distribution. Third, the heterogeneity of these symptoms might have impaired the reliability of the scale [9]. While the ADHD Functional Impairment scale also showed low internal consistency (α = .65, ω = .68), which may likewise be explained by the small number of items and their heterogeneity, the ODD/CD Functional Impairment scale showed very good internal consistency despite comparable item heterogeneity (α = .84, ω = .73). It should be noted that the individual items of both scales capture distinct domains of functional impairment (family, preschool, etc.). Accordingly, only moderate inter-item correlations are to be expected and may be diagnostically informative. Furthermore, especially in preschool-aged children, developmental asynchrony across domains should be taken into account [e.g., 64]. With some exceptions, the item-total correlations were generally satisfactory. Interestingly, the exclusion of any of these items with item-total correlations below r_it = .30 did not lead to a noticeable change in the Cronbach’s alpha of the respective scales. Taken together, these findings indicate that the internal consistency of some ILF-EXTERNAL scales should be interpreted with caution and that further refinement and evaluation of individual scales in larger samples are warranted. Nevertheless, internal consistency represents only one aspect of reliability, and the present findings should be considered together with the evidence from interrater reliability (IRR) and validity analyses.

After calculating the ICC one-way random-effects model for single ICC(1,1) and average ICC(1,2) measurements, the ICC revealed “moderate” to “good” IRR in line with Koo and Li [53] and “fair” to “excellent” IRR in line with Cicchetti [55]. To the best of our knowledge, only one previous study has provided information on interrater reliabilities pertaining to semi-structured clinical interviews in preschool children. Gigengack, Hein [21] assessed IRR for categorical diagnoses in the seven DIPA modules using Fleiss’ κ for n = 14 randomly selected interviews, and further evaluated the continuous scores on these modules using ICC. The results indicated almost perfect IRR for ADHD inattentive type (ICC = .98), ADHD hyperactive-impulsive type (ICC = .98), ADHD combined type (ICC = .99), and ODD (ICC = .98). However, as it remains unclear which ICC model the authors calculated and whether they relied on single or average measurements, the ability to interpret their findings is limited. In our own study on the IRR of the ILF-EXTERNAL in school-age children [34], IRR values were predominantly very good to excellent. However, despite administering the same interview and using the same training procedures for interviewers, the IRR values in the present study are considerably lower than those reported in our previous study Thöne, Görtz-Dorten [34]. One possible explanation for these diverging results might lie in the challenge of differentiating between clinically relevant behavior and transient problem behavior within the normative range in preschool-aged children [e.g., 13,65,66]. For clinical practice, limited interrater agreements lead to the conclusion that diagnoses can indeed be made as early as preschool age, but that a particularly comprehensive and careful clinical assessment is necessary. There must be significant impairments in several areas of life and alternative explanations must be carefully ruled out [58].

Diagnostic agreement between the interviewers and the blind raters were moderate and was similar to, albeit somewhat lower than, the kappa agreements for externalizing disorders in the DIPA [21], the K-SADS [23], and the ILF-EXTERNAL in school-age children [34] as well as the ILF-EXTERNAL conducted online via video chat [37]. Interestingly, these findings reflect the categorial-dimensional debate [24,67] and the frequently reported lack of agreement between different clinicians [68–70]. The meta-analyses by Markon, Chmielewski [69], for instance, suggested an expected 15% increase in reliability and a 37% increase in validity when using dimensional rather than categorical measures of psychopathology.

With regard to convergent and divergent validity, our findings revealed moderate to strong correlations between the ILF-EXTERNAL scales and the German CBCL/1½−5 scales, which cover similar symptoms. Moreover, the ILF-EXTERNAL scales showed significantly stronger associations with the CBCL Externalizing Problems scale than with the Internalizing Problems scale, thus supporting construct validity. Furthermore, the ILF-EXTERNAL scales generally correlated more strongly with the corresponding CBCL scales than with the non-corresponding CBCL scales. As such, our results are largely consistent with previous research, which reported small to moderate associations between the CBCL and clinical diagnoses from semi-structured interviews to assess clinical symptoms in children and adolescents [23,34,71–73]. Importantly, we additionally included proxy rating forms (FBB-ADHS-V/SSV) to be completed by parents and preschool teachers, which cover the same DSM-5 symptoms as the ILF-EXTERNAL, uniquely allowing us to specifically compare ratings between clinicians, parents, and preschool teachers in a multi-source approach. The results revealed the highest bivariate correlations between clinicians and parents, followed by very weak to small correlations between clinicians and preschool teachers. On the one hand, the considerable correlations between parents and clinicians can be explained by the fact that the clinical assessment is based on an interview with the parents. On the other hand, while informant discrepancies were initially attributed to measurement error, invalidity, or rater bias [e.g., 74], there is growing recognition that children’s behavior varies considerably depending on the situation and that this contextual variability should be considered clinically significant information [e.g., 31]. Consistently, Overgaard and Oerbeck [63] demonstrated in a large-scale cohort study that considering ratings of preschool teachers alongside parent ratings at the age of 3 has the potential to increase the predictive validity of an ADHD diagnosis in boys at the age of 5. Two important clinical implications can be derived from these findings: First, since it is apparent that no single informant can fully capture the important variations in children’s functioning across different interaction partners and contexts, such as in preschool or at home, multi-informant assessment is particularly important [26,29,31,35,75–78]. Specifically, it seems advisable, in addition to a clinical interview with the parents, for example based on the ILF-EXTERNAL, to also explore the perspectives of (preschool) teachers, for example through questionnaires. Second, as the clinician is responsible for integrating these diverse informant assessments and forming his/her own clinical judgment, our results emphasize the crucial role of informed clinical judgment [cf. 35].

Several strengths and limitations of this study should be noted. A major strength lies in the systematic comparison of ratings by clinicians, parents, and teachers, thus applying a multi-source approach in a large multicenter study. A distinctive characteristic of the DISYPS-III is that all of the diagnostic instruments within a specific mental disorder domain share identical diagnostic scales, and largely consist of identical items, enabling an easy assessment and fostering a systematic comparison of different rater perspectives in a multi-informant approach [12].

With regard to limitations, it should be acknowledged that the analytical sample was derived from the ESCApreschool screening dataset [38], which comprised preschool children who were screened in the context of a multicenter study evaluating evidence-based stepped-care treatments for children with externalizing symptoms. Accordingly, the sample was enriched for externalizing symptoms and predominantly comprised children with ADHD (82.5%) and/or ODD (57.8%). Although all available screening cases were included, irrespective of whether children subsequently fulfilled the intervention eligibility criteria, the sample does not represent the full spectrum of externalizing behavior in the general preschool population. Nevertheless, the inclusion of all available screening cases resulted in a heterogeneous sample with variability in externalizing symptom severity and cognitive functioning beyond that represented by children fulfilling the ESCApreschool eligibility criteria. However, the relatively small sample size (N = 211 or n = 129) may further limit the generalizability of the findings. In terms of the factor structure of the ILF-EXTERNAL, while we endeavored to incorporate the symptoms from the DMDD scale and the LPE scale into our CFAs, this was not possible due to the rather small sample size (N = 211). Furthermore, the small sample size of the female subgroup did not allow for an investigation of measurement invariance across gender. A further limiting factor is that we had to exclude eight items from the CD scale analysis due to substantial floor effects, which is in line with prevalence rates and considerations of developmental psychology. The excluded items assessed more severe forms of antisocial behavior and their removal resulted in a limited coverage of the DSM-5 CD construct. Consequently, findings related to the CD scale should be interpreted with caution, as the retained items may not fully capture the breadth of CD symptoms, thereby limiting the comparability and interpretability of the findings. Furthermore, we tested a higher-order factor model with two second-order factors (ADHD and disruptive behavior symptoms) but found no convergence. Accordingly, future research should include more externalizing symptoms and potentially also functional impairment domains, as well as larger samples, in order to further explore the hierarchical nature of the externalizing spectrum [cf., 79] and examine possible gender differences.

Overall, the present findings should be considered as initial evidence supporting the reliability and validity of the ILF-EXTERNAL. Further studies are needed to replicate these findings in independent samples and to extend the psychometric evaluation of the instrument, including additional aspects of validity and clinical utility.

Conclusions

The present study provided a comprehensive psychometric evaluation of the clinical parent interview for diagnosing externalizing disorders in children and adolescents (ILF-EXTERNAL) in clinically referred preschool children. More generally, the study contributes to the debate about categorical versus dimensional concepts of diagnostic classification, through a consideration of both. Overall, we were able to confirm the a priori assumed DSM-5-based factor structure of the ILF-EXTERNAL, as demonstrated by the correlated factors model with four factors (Inattention, Hyperactivity-Impulsivity, ODD, CD). With regard to categorical vs. dimensional agreement, IRR estimates at the dimensional level were generally higher than those at the categorical level. This highlights that mental disorders are not characterized by abrupt thresholds and are rather better described as continuous phenomena. Our findings provide evidence for the convergent and divergent validity of the ILF-EXTERNAL, as indicated by moderate to high correlations between the ILF-EXTERNAL scale scores and corresponding externalizing symptom scales, and by significantly lower correlations with divergent symptom scales from the CBCL/1½−5. This study revealed a complex picture of how associations between different informants can vary depending on the context (e.g., preschool vs. home), highlighting the need to systematically integrate multiple informants into clinical decision making. In conclusion, the present findings provide preliminary evidence that the ILF-EXTERNAL is a factorially sound, reliable, and valid semi-structured clinical interview for the categorical and dimensional assessment of externalizing disorders in both clinical practice and research.

Supporting information

Acknowledgments

Members of the ESCAlife consortium are: Tobias Banaschewski (lead author: tobias.banaschewski@zi-mannheim.de), Anna Kaiser (Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty, Mannheim/Heidelberg University, Mannheim, Germany; German Center for Mental Health (DZPG), Partner Site Mannheim-Heidelberg-Ulm, Germany); Nina Christmann, Brigitta Gehrig, Monja Groh, Christine Igel, Karina Janson, Sabina Millenet, Marie-Therese Steiner, Matthias Winkler, Mirjam Ziegler (Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty, Mannheim/Heidelberg University, Mannheim, Germany); Daniel Brandeis (Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty, Mannheim/Heidelberg University, Mannheim, Germany; Department of Child and Adolescent Psychiatry and Psychotherapy, University Hospital of Psychiatry Zurich, University of Zurich, Zurich, Switzerland; Neuroscience Center Zurich, University and ETH Zurich, Zurich, Switzerland); Sarah Hohmann (Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty, Mannheim/Heidelberg University, Mannheim, Germany; Department of Child and Adolescent Psychiatry, Psychotherapy and Psychosomatics, University Medical Center Hamburg-Eppendorf, Hamburg, Germany); Katja Becker (Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University of Marburg & University Hospital Marburg (UKGM), Marburg, Germany; Center for Mind, Brain and Behavior (CMBB), University of Marburg and Justus Liebig University Giessen, Marburg, Germany), Nicole Grau, Wiebke Haberhausen, Alisa Hiery (prev. Samel), Katja John, Daria Kasperzack, Johanna Ketter, Jennifer Krämer, Christopher Mann, Tanja Mingebach, Anna Maria Paulus, Bastian Schrott, Linda Weber, Anne-Kathrin Ohlsen (prev. Wermter) (Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University of Marburg & University Hospital Marburg (UKGM), Marburg, Germany); Manfred Döpfner, Christopher Hautmann (Center for Child and Adolescent Cognitive Behavior Therapy (CEKiP), Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany; Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany); Elena von Wirth (Center for Child and Adolescent Cognitive Behavior Therapy (CEKiP), Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany; Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany; Department of Clinical Psychology and Psychotherapy for Children and Adolescents, Trier University, Trier, Germany); Christina Dose, Julia Kellner, Lea Teresa Kohl (prev. Jendreizik), Katrin Krugmann, Karen Schulze-Husmann, Ann-Katrin Thöne, Anne-Katrin Treier, Paula Vetter (prev. Altenberger) (Center for Child and Adolescent Cognitive Behavior Therapy (CEKiP), Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany); Marcel Romanos, Julia Geissler, Thomas Jans (University Hospital Würzburg, Centre of Mental Health, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Würzburg, Germany); Tobias Johann Renner, Ute Dürrwächer (Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Center of Mental Health, University Hospital of Psychiatry and Psychotherapy Tübingen, Tübingen, Germany; German Center for Mental Health (DZPG), Partner Site Tübingen, Germany; LEAD Graduate School and Research Network, University of Tübingen, Tübingen, Germany); Lena Flik, Melinda Mross, Anja Pascher, Natalie Richardt, Priska Schneider, Ida Steinacker (Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Center of Mental Health, University Hospital of Psychiatry and Psychotherapy Tübingen, Tübingen, Germany); Martin Holtmann, Inken Kirschbaum-Lesch, Tanja Legenbauer, Franziska Martin (LWL University Hospital Hamm for Child and Adolescent Psychiatry, Psychotherapy and Psychosomatics, Ruhr University Bochum, Hamm, Germany); Michael Kölch (Department of Child and Adolescent Psychiatry and Psychotherapy, University Medical Center Ruppin-Brandenburg; Faculty of Health Sciences Brandenburg, Brandenburg Medical School, Germany; Department of Child and Adolescent Psychiatry and Psychotherapy, University of Rostock, Rostock, Germany); Stefanie Bienioschek (Brandenburg Medical School Theodor Fontane (MHB), Neuruppin, Germany; Department of Child and Adolescent Psychiatry and Psychotherapy, University Hospital Ruppin-Brandenburg, Neuruppin, Germany; Faculty of Health Sciences, Joint Faculty of the University of Potsdam, the Brandenburg University of Technology Cottbus-Senftenberg and the Brandenburg Medical School, Potsdam, Germany); Luise Poustka (Department of Child and Adolescent Psychiatry, University Medical Center Göttingen, Göttingen, Germany; Department of Child and Adolescent Psychiatry, Center for Psychosocial Medicine, University Hospital Heidelberg, Heidelberg, Germany); Henrik Uebel-von Sandersleben, Fatmeh Al-Deb’i (Department of Child and Adolescent Psychiatry, University Medical Center Göttingen, Göttingen, Germany); Kerstin Konrad (Section Child Neuropsychology, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Uniklinik RWTH Aachen, Aachen, Germany; JARA-BRAIN Institute II, Molecular Neuroscience and Neuroimaging (INM-11), Forschungszentrum Jülich GmbH and RWTH Aachen University, Germany); Vanessa Reindl (Section Child Neuropsychology, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Uniklinik RWTH Aachen, Aachen, Germany); Jochen Seitz, Tamara Novak, Miriam Remy, Jasmin Wenning (Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, LVR-University Hospital Essen (AöR), University of Duisburg-Essen, Essen, Germany); Michael Huss (Department of Psychiatry and Psychotherapy, University Medical Center Mainz, Mainz, Germany); Christine Margarete Freitag, Nadine Friedrich (Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital, Goethe University Frankfurt, Frankfurt am Main, Germany); Michael Rösler, Steffen Barra, Petra Retz-Junginger, Birgit Leipnitz, Florence Philipp, Toivo Zinnow (Institute for Forensic Psychology and Psychiatry, Saarland University, Homburg, Germany); Esther Sobanski (Department of Psychiatry and Psychotherapy, Medical Faculty of Mannheim, Central Institute of Mental Health, University of Heidelberg, Mannheim, Germany; Department of Child and Adolescent Psychiatry Lucerne, Lucerne, Switzerland); Barbara Alm, Melanie Bleich, Oliver Hennig, Peter Praus, Barbara Scharnholz (Department of Psychiatry and Psychotherapy, Medical Faculty of Mannheim, Central Institute of Mental Health, University of Heidelberg, Mannheim, Germany); Andreas Jochen Fallgatter, Julia Becker-Sadzio, Saskia Deppermann, Thomas Ethofer, Lydia Weber (Department of Psychiatry and Psychotherapy, University Hospital Tübingen, University of Tübingen, Tübingen, Germany; German Center for Mental Health, Tübingen, Germany); Michael Erb (Department of Biomedical Magnetic Resonance, University Hospital of Tübingen, Tübingen, Germany; Core Facility MRI, University Hospital of Tübingen, Tübingen, Germany; Max Planck Institute for Biological Cybernetics, Tübingen, Germany); Alexandra Phillipsen, Marcel Schulze, Behrem Aslan, Meike Lingen, Maria Steffens (Department of Psychiatry und Psychotherapy, University Hospital Bonn, Bonn, Germany); Sina Klüver, Samira Groß, Vivien Groß (Medical Campus University of Oldenburg, School of Medicine and Health Sciences, Psychiatry and Psychotherapy, University Hospital, Oldenburg, Germany); Johannes Thome, Beate Krecklow (Clinic and Policlinic for Psychiatry and Psychotherapy, Rostock University, Rostock, Germany); Wolfgang Retz (University Medical Center, Johannes Gutenberg University Mainz, Mainz, Germany; Institute for Forensic Psychology and Psychiatry, Saarland University Medical Center, Saarland University, Homburg, Germany); Lea Bensing, Sergiy Davydenko, Priscilla Gregório-Hertz, Daniel Turner (University Medical Center, Johannes Gutenberg University Mainz, Mainz, Germany).

Further we would like to thank Christina-Lara Bangert, Raphael Gutzweiler, Lisa Leinberger, Angelina Samaras, Anne Krieger (prev Schreiner), Anne Schröter, Lisa Wagner, Theresa Weil (prev. Nickel), Jennifer Krämer (prev. Karl), Thomas Mooz, Jana Langhammer, Doreen Blume, Jens Pfeiffer, Claudia Freitag, Maximiliane Werther, Charlotte Finger, Silas Deutsch, Isabell Schulz, Rebecca Stein, Viktoria Birkenstock, Svenja Katharina Weber, Lea Isabel Bach, Anna Fronhofen, Annabelle Warschburger, Carolin Böhning, Carolina Hast, Corinna Smits (prev. Broekmans), Dirk Alfer, Dominique Deli, Ellen Settegast, Elvan Dogan, Felix Oswald, Franziska Frenk, Friederike Waschau (prev. Neugebauer), Hedda Körner, Imke Partzsch, Jana Rausch, Jeannine Pawliczek, Johanna Kießling, Johanna Spichert (prev. Schnitzler), Judith Mühlenmeister, Jutta Pace (prev. Heermann), Katharina Szczecko, Katrin Eiben (prev. Floß), Kerstin Koppenborg, Laura Wähnke, Lea Ostrowski, Louise Seidl, Marcia Domanski, Marie-Theres Klemp, Moritz Hepperle, Mrijam Brüning, Nathalie Bastian, Nina Geldermann, Ruth Beckmann, Sara Zaplana-Labarga, Sarah Meier, Sophie Niestroj, Susanne Stollewerk, Yvonne Buntrock, Katharina Peters, Nadja Becker, Christoph Biohlawek, Lisa Haber, Julia Gläser, Verena Hartlieb, Silke Hauck, Christin Heim, Corinna Otte, Isabel Paul, Carolin Steinmetz, Stefanie Fekete, Zuzana Fouskova, Miriam Thierfelder (prev. Davids), Natalie Deux, Mareen Krüger (prev. Dökel), Silvia Eißing, Regina Herdering, Lara Kaffke, Nina Emshoff (prev. Müller), Nicola Nolting, Daniela Pingel, Carina Graf (prev. Schulz), Karen Schumann, Birthe Wagner, Daniela Walkenhorst, Daria Nolkemper, Anne Schüller, Lea Jahnen, Johannes Hebebrand, Jutta Mayer, Karin Bopp, Dr. Ann-Christine Ehlis, Katharina Bachmann, Bernd Ströhlein, Helge Müller, Alexandra Lam, Roland Wandschneider, and the Clinical Trials Unit, Medical Center – University of Freiburg (Dr. Lydia Herbstritt, Dr. Carolin Jenkner) for their contribution to ESCAlife. We thank Martin Hellmich for his statistical advice and Sarah Mannion for English language proofreading. We would also like to thank all patients and their families for their participation.