Prior research on HOPS program

In an early study of HOPS, the program was implemented by school mental health professionals in participating schools to 47 students identified as having ADHD and academic problems [10]. The study showed that school counselors demonstrated a high level of fidelity to intervention content. In comparison to a treatment-as-usual, waitlist (TAU/WL) control condition, caregivers in HOPS reported their children made significant improvements in organized actions (Cohen’s d = 0.88), task planning (d = 1.05), and homework completion behaviors (d = 0.85). Outcomes based on teacher ratings did not show improvement. Gains based on caregiver ratings generally were maintained at 3-month follow-up. In addition, HOPS participants demonstrated a superior grade point average (GPA) in comparison to TAU/WL.

HOPS was evaluated subsequently in a large-scale randomized controlled trial (RCT, n = 280, 7 schools; [4]). The trial was conducted in authentic practice settings (schools). Intervention providers were school counselors who had recently graduated, but they were hired and supervised by the research team. The findings revealed that, compared to TAU/WL, students receiving HOPS demonstrated significant improvements with large effect sizes in caregiver ratings of homework performance (d = 1.29), task planning (d = .79), materials management (d = .81), and organized actions (d = 1.14). Improvements based on teacher ratings were significant with medium effect sizes in materials management (d = .53) and organized actions (d = .55). Findings showed evidence of sustained effects when assessed about 3 months after intervention. An examination of implementation factors revealed that counselor-reported working alliance was a significant predictor of favorable effects of HOPS on outcomes [11].

Gaps in research on HOPS program

Although the HOPS program was designed primarily for students with ADHD, it has the potential to be effective for students with OTMP deficits who do not have this disorder [9]. Indeed, HOPS may be more useful to school professionals if it can be provided to the broader population of students with OTMP problems, not just those with ADHD [8]. Deficits in OTMP skills are highly common among students with ADHD [12], and they also occur among students with autism spectrum disorder [13], learning disabilities [14], and adverse psychosocial childhood experiences [15]. As such, this study will include students in general education classrooms demonstrating OTMP deficits that may be impacting their academic functioning, regardless of diagnostic status.

HOPS is unique among psychosocial interventions in that it was developed in partnership with schools with a focus on provider usability and feasibility of implementation. However, the manner in which HOPS was applied in the large-scale RCT differs in some important ways from how it likely is applied in routine practice conditions. When schools make the choice to implement HOPS, they need to make use of existing resources. For example, school administrators need to identify one or more school staff to provide the intervention to students. Research is needed to inform school administrators about whether HOPS works in routine practice when the program is offered by school staff. In the original design of the current study, the plan was for HOPS to be provided solely by school professionals working in the school. Because the initial years of this study were implemented during the COVID-19 pandemic, which placed limits on the capacity of schools to participate in this trial, the study design was modified in response to input from school professionals and funding sponsors so that HOPS will be provided by both school professionals (school team providers) as well as educational professionals contracted by the research team who do not work in the schools (research team providers). This shift in design affords the opportunity to test the effectiveness of HOPS when delivered under conditions similar to the original trial (i.e., implementation by members of the research team, [4] as well as conditions more similar to actual practice (i.e., implementation by school providers).

Importance of intervention implementation and relational processes

Intervention effectiveness depends to a substantial extent on how well the intervention is implemented (provider fidelity) and the extent to which participants are engaged or involved in the intervention [16,17]. Provider fidelity refers to adherence to the steps of the intervention curriculum as well as level of competence in providing intervention. Participant engagement includes attendance, active participation in sessions, and adherence to recommended intervention practices [16]. Another important implementation factor is the working alliance between the provider and student, or the ability of the provider to work with the student in a responsive, collaborative manner [18]. Monitoring implementation processes (provider fidelity, student and parent engagement) and assessing working alliance are especially important when interventions are provided under routine practice conditions, because there is less control over provider fidelity and often less emphasis on promoting intervention in a replicable, standardized manner. In this study, our team will carefully examine provider fidelity, student and parent engagement, and working alliance between the intervention provider and student participants and evaluate the extent to which these factors are associated with improvements in student outcomes.

Conceptual framework of HOPS

The HOPS program is based primarily on principles of behavior therapy and incorporates elements of cognitive behavioral therapy. Behavioral strategies include verbal instruction, prompting skill use, shaping, and contingency management (with an emphasis on positive reinforcement for effort and performance of targeted OTMP behaviors). HOPS includes parent consultation, and often teacher consultation, to foster generalization of skills across settings and maintenance of strategy use after the intervention has ended. In addition, cognitive-behavioral strategies are used to identify and modify negative, defeatist attitudes that prevent the child from making progress [9].

Several factors, including provider intervention fidelity, as well as student and caregiver engagement in HOPS likely contribute to changes in students’ OTMP skills and homework performance (proximal outcomes). In turn, based on previous research [19,20] changes in OTMP skills likely lead to improved academic (distal) outcomes. Based on research from the previous evaluation of HOPS [4], we expect that severity of student organization problems, ADHD symptoms, and oppositional defiant behaviors at baseline will moderate the effect of intervention on OTMP skills and homework performance. In addition, based on prior research on child organizational skills training at the elementary level, we expected severity of child internalizing symptoms (anxiety and depression) baseline will moderate the effect of intervention [20]. We will also examine whether child medication status (i.e., taking or not taking medication to treat ADHD) and family socioeconomic status modify the effect of intervention. (See Fig 1 for the Spirit Checklist and Fig 2 for the HOPS theory of change.)

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Participant timeline: SPIRIT schedule of enrollment, interventions, and assessments.

^a -t₁ = screening, 0 = baseline, t₁ = post-intervention, t₂ = 6-month follow up, t₃ = 12-month follow up. ^b HOPS refers to the Homework, Organization, and Planning Skills program. ^c Provider refers to school and research provider unless otherwise indicated (e.g., school or research provider). ^d Arrows indicate continuous delivery of HOPS intervention or fidelity/engagement/cost data collection for each HOPS session. ^e For students in the TAU/WL condition, fidelity, engagement, and cost data are collected when they receive the intervention (after post-intervention data collection).

https://doi.org/10.1371/journal.pone.0343894.g001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. HOPS study theory of change.

The figure depicts HOPS intervention components and factors that may impact the effect of HOPS on proximal and distal outcomes. OTMP = organization, time management, and planning skills; ODD = oppositional/defiant behavior; SES = socioeconomic status.

https://doi.org/10.1371/journal.pone.0343894.g002

Study purpose and aims

The purpose of this study is to conduct a replication (i.e., replication with design elements that differ from the original trial) of a randomized controlled trial that showed the HOPS intervention provided by counselors employed by the research team improved organizational, time management, and planning (OTMP) skills for middle school students diagnosed with ADHD in relation to students on a waitlist receiving treatment-as-usual (TAU/WL; [4]). The trial will be conducted by an independent research team; the developer, Dr, Langberg, and his team will provide training, technical assistance and follow-up consultation on request to HOPS providers but will not be involved in any intervention evaluation activities. Similarly, the research team evaluating the intervention will not be involved in training or supervising HOPS intervention implementation. The study will test whether HOPS compared to TAU/WL yields similar benefits when implemented by providers on the research team (HOPS-Research Team; HOPS-RT) and school-based professionals (HOPS-School Team; HOPS-ST) for the full range of students who have OTMP skills deficits, not only those with a formal diagnosis of ADHD. The study will examine six aims:

• Aim 1: Evaluate the difference between HOPS-ST and TAU-WL on improvement in proximal outcomes (organization skills, homework performance) and distal outcomes (academic performance) at post-treatment (4-months after baseline) and explore the intervention’s sustainability (6 and 12-months after baseline). This aim will enable our team to determine the effectiveness of HOPS when provided by school professionals. We hypothesize that students in the HOPS-ST group will demonstrate significantly fewer OTMP skills deficits and homework problems and will show significantly better academic performance at post-treatment compared to students in TAU-WL. In addition, we expect the benefits of HOPS-ST to be maintained at follow-up.
• Aim 2: Examine the effectiveness of HOPS-RT in relation to TAU-WL. This aim will enable our team to replicate the findings of the Langberg et al. [4] trial. We hypothesize that students in the HOPS-RT group will demonstrate significantly fewer OTMP skills deficits and homework problems and would show significantly better academic performance at post-treatment compared to students in TAU-WL. In addition, we expect the benefits of HOPS-ST to be maintained at follow-up.
• Aim 3: Explore differences in effect for HOPS-ST and HOPS-RT at post-treatment and follow-up.
• Aim 4: Explore mechanisms of action, specifically whether improvements in OTMP skills and homework performance mediate the effect of HOPS on improvements in academic performance.
• Aim 5: Explore whether student grade level, severity of organization problems, severity of ADHD, oppositional defiant behavior, and internalizing symptoms, and student medication status at baseline moderate the effect of HOPS on outcomes.
• Aim 6: Explore the extent to which treatment fidelity, student engagement, parent engagement, teacher engagement, and provider-student working alliance predicts outcomes for students in HOPS.
• Aim 7: Estimate the cost of providing HOPS under authentic practice conditions in schools.

Trial design

Collection of outcome data.

Outcome data for students in HOPS and TAU-WL will be collected at baseline and post-treatment (about 4 months after baseline). Follow-up data will be collected 6 months after baseline for students assigned to the HOPS condition. In addition, follow-up data will be collected 12 months after baseline for students in HOPS who remain in their school of enrollment the following school year. We will not collect 6-month and 12-month follow-up data for students in TAU-WL because of ethical concerns about further delaying the delivery of HOPS to those in the control group. However, we will collect post-treatment outcome data after students in the TAU/WL condition receive the HOPS program. Treatment fidelity, student engagement, and parent engagement will be assessed for participants in the HOPS condition during the intervention period. Provider-student working alliance will be assessed after the intervention period. Severity of organization problems, ADHD symptoms, disruptive behavior symptoms, and anxiety symptoms, as well as school organizational factors that may contribute to intervention implementation, will be assessed at baseline. (See Fig 3 for the study plan and flow diagram and see Fig 1 for a list of measures and the timelines for administering measures).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Study plan and flow diagram.

The figure depicts random assignment to HOPS and TAU/WL, random assignment to school vs. research provider for participants in HOPS, and data collection at four time periods. HOPS-ST refers to HOPS implemented by a school team member, and HOPS-RT is HOPS implemented by the research team.

https://doi.org/10.1371/journal.pone.0343894.g003

Research settings

The study will be conducted in Philadelphia, PA greater metropolitan region. Our team will purposely recruit schools that are highly diverse in subsidized lunch rates of the schools and racial and ethnic composition of the students to maximize generalizability. Our team will attempt to recruit schools with varying levels of urbanicity (i.e., rural, suburban, and urban). We expect the racial and ethnic minority composition across schools to range from 15% to 99% and subsidized lunch rates to range from 15% to 100%. For each school, we plan on recruiting one or two school providers to implement the HOPS intervention.

Student participants

Students will be eligible for this study if: (a) they attend a school participating in the study, (b) they are enrolled in grades 6–8 and placed in general education for a majority of the day, and (c) they are identified as having OTMP deficits that are impairing academic performance, as identified by the four interference items on the Children’s Organizational Skills Scale-Teacher version (COSS-T; [22]). Students classified with emotional/behavioral disorders and/or learning disabilities will be included if it is determined that the students’ disabilities are being addressed adequately in their current educational program and OTMP skill deficits are a primary concern. Students with a one-to-one aide will be excluded because the presence of an aide substantially alters how an organizational intervention is implemented. In addition, students from families in which both caregivers do not speak English will be excluded because the program has not yet been developed for non-English speakers. Parents or legal caregivers must give consent, and students must provide assent for student and family participation, which can be withdrawn at any time during study participation. The potential for harm to participants is minimal, but they are informed that it is possible they may feel uncomfortable in completing measures. Participants are not mandated to complete items on measures used in the study.

Recruitment

Two primary methods will be used to recruit schools. First, emails will be sent to the principals of schools serving students in grades 6–8 located in the large metropolitan area in which the study will be conducted. The selection of schools will be inclusive of districts across the region to achieve balance across subsidized lunch rate levels of the schools (low [0%−30%], middle [31%−60%], and high [61%−100%]). In addition, there will be an intentional effort to recruit schools in urban, suburban, and rural areas. Second, emails will be sent to principals of schools from districts in which we have previously conducted research or those referred to us by school administrators or other researchers. Approval will be obtained from the school principal, school superintendent, and school board (as necessary) prior to conducting research in the school. In addition, principals will be asked to identify one or more school providers to implement the HOPS intervention. School provider(s) must provide consent to participate in the study for the school to be enrolled in the study. Once all approvals and consents are obtained from school professionals, the school will be randomly assigned to a study condition (HOPS vs. TAU/WL).

Students will be referred to the study by their classroom (general education) teachers. School principals will identify the specific strategies to be used in convening teachers, informing them about the study, and obtaining referrals. Teachers will be informed that referral of a student to the study indicates their willingness to complete forms for outcome evaluation. The pool of teachers selected for inclusion in the referral process will be asked to nominate students from their general education classroom: (a) who are struggling the most with OTMP skills, (b) whose OTMP skill deficits interfere with their academic performance, and (c) who have at least one caregiver who speaks English. Each teacher who agrees to participate will complete a nomination form for each referred student for the purpose of determining student eligibility.

After a student is referred and determined to be eligible, a professional from the school, typically the school provider, will attempt to contact the parent or legal guardian to explain the study, indicate their child has been referred for HOPS through the study, and determine if the family is willing to be contacted by the study team. The school professional will often send a letter to the parent/guardian that includes a link to an electronic survey to offer permission for the study team to contact the family and provide contact information. The school provider may also contact (phone or email) parent(s)/guardian(s) of identified students to introduce the study. When the school gets permission from a family to share contact information with the study team, they will pass that information to the study team. With caregiver approval, a member of the study team will contact the family to determine interest, obtain caregiver consent, and obtain student assent.

Intervention conditions

The study has two intervention conditions: HOPS intervention implemented by a research or school provider and TAU/WL.

Training and technical support for HOPS providers

Procedures for offering training and technical support for intervention providers were adapted from those used in prior studies of HOPS [4,10] and reflect the strategies used in actual practice by Langberg and colleagues when school professionals request training to implement HOPS. School and research providers will receive a copy of the HOPS manual [9] and 1.5–2 hours of training before providing HOPS. The training will include a description of: (a) the conceptual model and core principles of HOPS, (b) principles for promoting behavior change, (c) major components of the program, (d) procedures for providing each student session, (e) procedures for providing parent consultation (and teacher consultation when feasible), and (f) guidelines for selecting reinforcers for students. The training is provided virtually by Dr. Langberg or a trainee working under his direction.

Intervention providers will be able to login to a website (hopsintervention.com) that will give them access to videos of Dr. Langberg providing HOPS training and responding to common treatment challenges, access to PowerPoint slides for Dr. Langberg’s training, and responses to Frequently Asked Questions (FAQs). Through the website, intervention providers will be able to ask brief logistical/technical questions about HOPS, which will be answered regularly by the training team (Dr. Langberg and trainees). In addition, intervention providers will be given the opportunity to receive consultations on demand via telephone, virtual communication, or email with this trainee. The training team will log each consultation call to record the name of the provider, topics discussed, and duration of communication.

Outcome measures – Proximal and distal

The Children’s Organizational Skills Scale – Parent and Teacher versions (COSS-P, COSS-T) will be used to assess changes in OTMP functioning at home and school. COSS Total scores (26 items for COSS-P, 28 for COSS-T) have strong construct and discriminant validity [23] and treatment sensitivity [4]. Each item is rated on a 4-point scale. The total score will be used in analyses, with higher scores being less favorable.

The Homework Problem Checklist (HPC) is a 20-item caregiver-report measure that indicates the frequency of child homework problems. The HPC has been shown to have strong construct validity [24] and treatment sensitivity [4]. Each item is rated on a 4-point scale. The total score will be used in analyses with higher scores indicating more problems.

The Homework Performance Questionnaire – Teacher version (HPQ-T) assesses students’ homework behavior during the past 4 weeks. Each item is rated on a 7-point scale indicating how often each behavior occurs, with higher scores indicating better performance. The 17-item General factor that assesses overall homework competence has strong construct validity [25] and will be used in analyses.

The Academic Proficiency Scale (APS) is a teacher-report measure assessing proficiency in six academic subjects. Each item is rated on a 5-point scale (1 = well below standard expected at this time of year; 3=at standard; 5 = well above standard). This measure has high internal consistency [8] and strong treatment sensitivity [5]. The mean item score will be used in analyses, with higher scores indicating better performance.

Student academic report card grades will be used to examine the impact of HOPS vs. TAU/WL on academic performance. Student grades will be recorded for the last marking period of the year prior to enrollment (baseline), the year of enrollment (post-treatment), and the year following enrollment (follow-up). Grades will be harmonized across participating school districts to a uniform grading scale (ranging from a low of 1 to a high of 4). A composite academic grade score will be computed for each student based on their grades in math and reading/language arts (possible range = 1–4). Higher scores indicate better outcomes.

Participant, family, and school characteristics

Caregivers will report information about student age, gender, race and ethnicity, caregiver education, family composition, and academic support services received by students outside of school. School staff will report on students’ special education status, amount of special education services received, Section 504 services, and other services being provided to address student organizational problems. Publicly-available, online data about the free and reduced-price lunch percentage of each school will be accessed. A geo-mapping tool (Area Deprivation Index, ADI, [26]) will be used to determine each family’s neighborhood level of disadvantage; higher scores on this measure indicate greater disadvantage.

Medication and educational intervention status will be assessed at baseline and each assessment point by asking caregivers to indicate whether the child is currently on psychotropic medication, name of medication, and date of initiation. We will monitor change in medication status over the course of the intervention and follow-up periods. Caregivers will also be asked to indicate whether their child is receiving intervention (e.g., therapy, skills training, coaching) related to organizational skills/executive functioning.

Caregivers will report on their support of their child’s organizational skills related to materials management, homework management, and planning (i.e., how much time and support they provide to the child to maintain these skills).

The Vanderbilt Rating Scales will be used to assess severity of ADHD, oppositional defiant behavior, and internalizing symptoms. The caregiver- and teacher-report versions of the Vanderbilt scales will be used [27,28]. Informants rate how frequently each symptom occurs on a 4-point Likert scale (0 = never, 3 = very often). Both scales consist of a 9-item Inattention factor, a 9-item Hyperactivity-Impulsivity factor, an 8-item Opposition-Defiant factor, and a 7-item Internalizing factor. The internal consistency of each of these factors with middle school students has been shown to be high (>.90) [4].

HOPS providers, including school and research providers, will report on their professional roles, years of experience, gender, race and ethnicity, educational level, and years of experience. School providers will report aspects of the organizational (school) context that are relevant for the implementation of evidence-based practices in schools. The Implementation Climate Scale has 15 items; each item is rated on a five-point Likert scale (0 = not at all; 4 = very great extent). Higher scores indicate a more favorable climate. Research has supported the factor integrity and internal consistency of this scale [29].

Measures of intervention implementation

Measures of acceptability, usability, working alliance, and cost

Caregiver acceptability will be assessed using a 13-item caregiver satisfaction measure developed for use with HOPS [32]. The Children’s Intervention Rating Profile (CIRP), a 7-item measure, will be adapted to assess student views of acceptability [33]. Intervention provider (both school provider and research provider) usability will be examined by the Usage Rating Profile–Intervention–Revised (URP-IR), a 29-item measure assessing factors influencing the usage of school interventions [34]. The URP-IR will only be completed once by each provider.

The working relationship between the intervention provider and student will be assessed at the end of intervention using an adaptation of the Working Alliance Inventory-Short Version (WAI-S; [35]). The WAI-S consists of 12 items rated on a 7-point Likert scale (1 = never; 7 = always) assessing agreement on tasks and goals and the student-provider bond); higher scores are more favorable. The internal consistency (alpha coefficient) of this measure among middle school students was demonstrated to be.92 for intervention providers and.90 for students. In previous research, scores on the WAI-S have been shown to be associated with intervention outcomes [11]. The WAI-S will be filled out by the student (“Working Alliance Inventory-Student”) and intervention provider (“Working Alliance Inventory-Counselor”).

Total costs (labor and non-labor) associated with the implementation of HOPS as well as training and technical assistance for providers in each school will be calculated according to the ingredients method [36]. To do so, we will document intervention implementation activities of providers, trainers, and project assistants using time diaries, administrative records, and interviews. We will conduct time use interviews with providers at the end of each intervention period. Materials costs will be determined using acquisition costs. The costs then will be determined from the total effort on each activity and per hour wage rate of the person conducting the activity using local price levels (average salary and benefits for those in job categories matching the staff providing services). Costs for HOPS will include: (a) time to establish and maintain the relationship with each school, (b) time spent conducting the initial training for providers; (c) provider time spent implementing each student, parent, and teacher session, (d) time spent providing technical support to each provider, and (e) materials costs. We will distinguish two types of costs: cost of physical materials used for training and interventions (e.g., training manual for each provider) and costs associated with time spent for training, technical support, and intervention implementation [37]. In addition, costs for initial training and tech support of providers will be differentiated from the ongoing costs of implementation to provide separate estimates of costs for start-up in each school and costs for ongoing implementation. Cost estimates will not include time and materials needed for research activities (e.g., participant recruitment and enrollment, data collection).

Intervention, training, and technical support time will be derived from administrative records of HOPS sessions, diaries completed by Dr. Langberg and technical support assistants, and training website usage data. Study staff will interview intervention providers (both school providers and research providers) after intervention to assess their experience delivering the intervention, including how much time they have spent on session preparation, delivery, and follow-up.

Ethics approval and consent to participate

The study was approved by the Institutional Review Board of Children’s Hospital of Philadelphia (CHOP, IRB#19–016865). Caregivers will be required to provide consent for their child to participate in the study, and students will be required to provide assent. A waiver of consent was obtained for the screening of student participants to obtain referrals and determine eligibility. Consent for the participation of teachers (who nominate students for participation and provide data on participating students) in this study will be indicated by their submitting referrals for students to participate in the study. Only authorized study staff will obtain consent from study participants. Plans to modify the study protocol must be reviewed and approved by the Institutional Review Board.

Data analyses