Feasibility.

Information related to the feasibility of the intervention were gathered using a combination of quantitative and qualitative methods. All students participating in the study completed a paper-based feedback survey in the final week of the intervention. Consent from all parents of students in participating classrooms was obtained at the schools’ discretion. The survey asked students to respond to statements about the intervention such as “I enjoyed In My Shoes” and “In My Shoes activities were interesting” using a 4-point Likert scale (1= strongly disagree to 4 = strongly agree). Teachers, parents, and school leadership staff involved in supporting teachers were sent a link to an anonymous online feedback survey, individualised to their role, post intervention. These surveys asked participants to respond to statements such as “In My Shoes was a positive experience” and “the content of In My Shoes was relevant” using a 5-point Likert scale (1 = strongly agree to 5 = strongly disagree). Participants were prompted to provide reasoning if they responded ‘neutral’, ‘disagree’ or ‘strongly disagree’ to any of the statements. Responses to feedback surveys were supplemented with qualitative interview data through specific lines of questioning relating to the implementation and practicality of the intervention.

Fidelity.

The fidelity protocol for this study was based on the behaviour change consortium treatment fidelity recommendations [25] (see S1 Table). The primary researcher observed one lesson in each classroom and scored teachers on a fidelity checklist, which included questions relating to adherence, duration, quality of delivery, student responsiveness and programme specificity (i.e., whether teachers adhere to activities as designed and show knowledge of content and intervention strategies). Teachers were also required to complete weekly online fidelity checklists to ensure intervention was being delivered as stated in manual. Parents’ receipt and response to weekly parent information handouts was evaluated via an online feedback survey post-intervention and following interviews. Schools’ adherence to the whole school component of the intervention (e.g., implementation of whole school activity ideas such assembly items, newsletter inserts) was evaluated via online teacher and school leadership feedback surveys and in interviews with teachers.

Preliminary effectiveness.

To evaluate the effectiveness of the intervention in targeting identified intervention outcomes (see page 7), several measures were administered pre-post intervention. Some of these measures were conducted with students on the autism spectrum only and others with all students participating in the study.

Classroom participation.

The Behaviour Assessment System for Children – Third Edition Student Observation System [BASC-3 SOS; 35] was used to conduct direct observations of the classroom behaviour of students on the autism spectrum. The SOS uses the technique of momentary time sampling (i.e., systematic coding during three second intervals spaced 30 seconds apart over a 15-minute period) to record a range of student behaviours including positive (e.g., teacher-student interaction) and negative behaviours (e.g., inappropriate movement or inattention). It also includes a 71-item observer rating scale that is completed after the time sampling procedure that gives in-depth information about students’ behaviours that may impede or promote learning and adjustment in the classroom (Part A). Two 15-minute video recordings of students on the autism spectrum, participating in similar classroom activities, were taken in the first and final week of the intervention and sent to the primary researcher. The primary researcher recorded students’ behaviour on the BASC-SOS. An independent rater who is a qualified occupational therapist with experience working in schools and with students on the autism spectrum, was trained in the use of the SOS and scored a 40% random sample of the pre-post video observations to establish inter-rater reliability. The independent rater was blinded to all aspects and purposes of the study, and scored students on the BASC-3 independently to the primary researcher to minimise bias.

Subjective school experiences.

There is not one single measure that adequately captures all factors that contribute to students sense of school connectedness. Therefore, a battery of measures was used to evaluate self-reported changes in this construct and related intrinsic factors (i.e., preferences, sense of self, activity competence) pre-post intervention. Due to various terms used to describe the concept of school connectedness (e.g., belonging, bonding, engagement), we have collectively labelled these measures under the heading ‘subjective school experiences’ for the purposes of this paper.

Experience Sampling Method (ESM) was used to evaluate changes in the school connectedness of students on the autism spectrum, by exploring the nature and quality of their experience while participating at school. ESM, an ‘in-the-moment’ technique that is “…commonly used for the examination of the context and content of individuals’ daily life from their own perspective” [36] and has been found to be a reliable and valid tool to self-report the participation experiences of children on the autism spectrum aged 8 to 10 years [36]. This methodology was chosen as it captures the influence of context on experiences, which allows for the examination of individual values relating to school participation and identifies fluctuations in perceptions of everyday experiences [36]. Collecting ESM data at multiple moments throughout the day also minimises error due to recall, distortion and rationalisation and allows for exploration of the dynamic relationship between subjective experiences and everyday contexts [37, 38].

Students on the autism spectrum were required to complete a survey that had been loaded onto the mEMA app [39], which is an ESM platform designed for IOS devices. The mEMA app prompts participants to complete the ESM survey, time stamps the response and stores data for analysis. The survey was adapted from a version developed by Chen and colleagues [40] and included closed questions and scaled items related to the student’s participation in school occupations, including the specific place (e.g., where were you when you were beeped?), the specific activity (e.g., what was the main thing you were doing?) and interaction status (e.g., who were you with? Were you talking with someone? Who were you talking to?). The ESM survey also explored the quality of their experiences relating to enjoyment, difficulty, interest, degree of involvement and importance. Emotions were explored on a continuous scale across five domains: anxious-relaxed, lonely-sociable, sad-happy, angry-friendly, and bored-excited. The format of the questions included multiple choice, yes/no and visual analogue scales for items relating to emotions. The mEMA platform was chosen over others as it allows researchers to use images to supplement text as a visual support for students on the autism spectrum.

The ESM survey was piloted with two typically developing students aged between 7 and 11 years to ensure the questions were clear and developmentally appropriate, and the device was easy to use. Adjustments were made to the survey based on observations and students’ feedback. The Flesch Kincaid readability test [41] showed that the grade level of survey questions was 3; lower than participants reading comprehension levels identified by the WRMT-III. Prior to data collection, students on the autism spectrum, their parents and educators were provided training in the use of the device and the mEMA app. Training involved the researcher asking students to read and respond to survey items, clarifying questions and troubleshooting students’ responses.

Students were provided with an iOS phone with the mEMA app installed in the first and final week of the intervention. The device randomly prompted students to respond to the survey 5 times a day for a week between 7am and 5pm. This sampling period enabled researchers to capture students’ experiences while at school, as well as their transition to-and-from school. Students were informed that they could skip prompts that occurred at inconvenient times; that they would be reminded three times every five minutes to complete the survey and that the survey would become inactive after 15 minutes. Students were encouraged to seek help from their parents, educators or contact the researcher directly if they required assistance during sampling periods.

All students in participating classrooms were required to complete a battery of outcome measures, including the Student Engagement Instrument – Elementary Version [SEI-E; 42], Belonging Scale [43] and four scales developed by the research team.

The SEI-E and Belonging Scale were used to evaluate changes in students school connectedness. The SEI–E assessed students’ self-report levels of cognitive and affective engagement in school. The SEI–E includes 31 items and four subscales (i.e., teacher student relationships, peer support for learning, future goals and aspirations and family support for learning); scored on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree), with higher scores indicating a higher level of engagement. For example, ‘other students here like me the way I am’, ‘students at my school are there for me when I need them’ and ‘teachers at my school care about the students’. The SEI–E was adapted from Appleton and colleagues [44] original SEI to ensure items addressed all relevant engagement constructs and were developmentally appropriate for primary school students. The SEI was found to have the strongest psychometric properties of 15 measures in a recent systematic review [1]; with the SEI–E showing promising psychometrics from preliminary studies for students in years 3 to 5 [42, 43].

The Belonging Scale assessed students’ sense of school belonging. It is a 12-item adapted version of the Psychological Sense of School Membership scale [45] designed for use with students from 8 years of age to. The 12 items include six that focus on students’ general feelings towards school and sense of belonging (e.g., I feel really happy at my school) and six that focus on their perception of support, help and acceptance from adults and peers at school (e.g., there is an adult in school I can talk to about my problems). Students respond using a 3-point Likert scale (1= no not true, 2 = not sure, 3 = yes). The Belonging Scale has been validated with students aged 8 to 11 years and has a Cronbach’s alpha of 0.79, within the range of values (α = 0.77 – 0.88) commonly reported for the 18-item Psychological Sense of School Membership [45].

Four-scales developed by the research team, entitled ‘In My Shoes’, ‘in the past week’, ‘involvement’ and ‘learning about the autism spectrum’, were used to evaluate changes in students’ interpersonal empathy, self-perceived confidence and involvement at school, and understanding of autism (see intervention outcomes listed on page 7).

In My Shoes, was a situation-based scale that presented 10 social situations that commonly occur at school. Students were required to select how they would respond to each social situation from a multiple-choice list (e.g., you are playing a game of four square with your friends. You see Johnny is sitting on his own in the playground. Do you: A: ask Johnny to come and play; B: ignore Johnny. He’s not good at four square or C: leave Johnny alone. You know that he likes playing by himself). The purpose of the scale was to assess changes in a student’s ability to identify pro-social behaviours that would lead to the inclusion of their peers (see intervention outcome (e) on page 7). Multiple choice responses that involved higher level of interpersonal empathy were scored higher (e.g., 2 = ask Johnny to come and play; 0 = ignore Johnny. 1 = leave Johnny alone). Each social situation was directly related to intervention lesson content.

‘In the past week’ included 12-items that assessed students’ self-perceived confidence in: asking for help, knowing when a peer needs help, helping a peer, encouraging a peer, inviting a peer to play, starting, or joining in conversation and sharing with a peer; all skills that were targeted in the intervention (see intervention outcome (d) and (e) on page 7). Items were rated on a 4-point Likert scale (1 = not confident at all to 4 = very confident), with higher scores indicating higher levels of confidence.

The ‘involvement’ scale included 8-items assessing students’ self-perceived involvement in classroom, school, and extracurricular activities. Items (e.g., most mornings, I look forward to going to school, I work hard at school, and I am an active participant in classroom activities) were scored on a 3-point Likert scale (1= no not true, 2= not sure, 3=yes true), with higher scores indicating higher levels of self-perceived involvement.

Finally, ‘learning about the autism spectrum’ included eight statements about autism that students were required to identify as true or false (e.g., children on the autism spectrum brains work differently and only boys have autism). This scale was administered at the beginning and end of the second lesson, to evaluate changes in students understanding of autism (see intervention outcome (a) on page 7). This lesson focused on increasing students understanding of autism using a documentary style video of students on the autism spectrum. All self-developed questionnaires were reviewed by a speech therapist for language comprehension and trialled with two typically developing elementary school students.

Semi-structured interviews were conducted by the primary researcher with teachers, parents, and students on the autism spectrum pre-post intervention to verify and enrich quantitative data. Interview guides were specifically designed to gather more information about the feasibility and perceived benefits of the intervention from different participants perspectives. For example, teachers were asked questions like ‘how easy was it to implement In My Shoes in your classroom?’, and ‘do you think peers have experienced any benefits as a result of participating in In My Shoes? If so, can you please share some specific examples with me?’

Preliminary effectiveness.

The parent-report Home & Community Social Behaviour Scale (HCSBS; 31) and the teacher-report School Social Behaviour Scale [SSBS; 46] were used to describe and evaluate changes in the social competence and behaviour (see intervention outcome (e) on page 7) of students on the autism spectrum in the home and school environment. The HCSBS has excellent internal consistency (social competence, α = 0.96; antisocial behaviour, α = 0.98) and good to excellent (α = 0.82 – 0.91) test-retest reliability [31]. The SSBS has excellent internal consistency (social competence, α = 0.91; antisocial behaviour α = 0.98) and acceptable (α = 0.68 – 0.80) test-retest reliability [46]. A maximum of five parents of typically developing students from each classroom were asked to complete the HCSBS for their child, to evaluate differences and changes in samples pre-post intervention.

Recruitment capability and sample characteristics.

Descriptive characteristics of students on the autism spectrum. Ten students on the autism spectrum aged between 8 and 10 years participated the study. Most students had at least one sibling and 90% of students were male and the only child in their family with a diagnosed disability. Ninety percent of students had an additional diagnosis with anxiety and ADHD being reported in more than 50% of the sample. Half of students changed schools at least once due to parent reports of inadequate support or bullying at their previous school. All student participants were on an Individual Education Plan and had access to an Education Assistant. All students accessed services outside of school including occupational therapy, speech therapy and psychology. All students had at least a grade 1 reading level so were able to comprehend survey items (see Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of students on the autism spectrum.

https://doi.org/10.1371/journal.pone.0269098.t001

Descriptive characteristics of participating teachers and schools. Eight teachers represented six mainstream independent co-educational elementary schools in the Perth metropolitan area. Four of the teachers had more than 10 years teaching experience, with only one teacher newly graduated. All teachers had experience teaching students on the autism spectrum and most teachers had experience working in other grade levels (70%) and schooling sectors (50%). The average SEIFA decile was 9.3 (SD: 1.06; range: 7 – 10). A high SEIFA decile reflects a relative lack of disadvantage rather than relative advantage; for example, few households with low incomes, few people with no qualifications or in low skilled occupations [32]. As shown in Table 2, most schools came from higher decile regions of Western Australia and were large in student size. Only one teacher taught in a classroom with less than 25 students. Five out of eight teachers reported moderate to high levels of teaching self-efficacy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of teachers and schools.

https://doi.org/10.1371/journal.pone.0269098.t002

Examining recruitment capability and resulting sample characteristics was important in determining whether In My Shoes was relevant to study participants and if future efficacy studies would be successful [23]. Students on the autism spectrum appeared to have characteristics that were consistent with what was reported literature as students who would be appropriate for the intervention. While the recruitment process was time consuming, we were able to recruit an adequate sample for the purposes of the feasibility study. Several schools expressed interest in participating, however, parents of students on the autism spectrum declined to participate mostly because their child was not aware of their diagnosis. Schools that declined to participate in the study attributed this to not meeting eligibility criteria, lack of time and resources, and pressure in meeting curriculum requirements due to COVID-19 school closures in the previous term.

Data collection procedures and outcome measures.

Teachers reported several challenges relating to the collection of data during the feasibility study. Data were collected in the first and final week of term, which are often the busiest, most unstructured weeks of term which can be highly charged with emotion for students. Teachers felt students may have experienced fatigue, altered their responses due to the teacher’s presence, and were concerned students’ responses may have varied depending on the time of day. Maria (teacher) said,

It would depend on the kind of day they are having, and this can change so quickly…even the time of day you do it, they might have had a fight with mum before they got out of the car. You never know with kids. You could literally do the same thing on two different days and they get totally different answers.

The timing of data collection during this study may have impacted results and emphasises the importance of strategic timing of data collection when conducting research in schools. Authors suggest future pilot studies consider collecting data mid-term to mid-term to minimize the impact of these contextual factors on study findings.

Overall, ESM proved a useful tool in capturing students’ in-the-moment lived experiences at school. Students on the autism spectrum reported that they found it easy to use the ESM device and complete the surveys. Most students reported enjoying the responsibility of having the device and responding to ESM surveys. Only one student reported that she did not like the attention the device brought to her in the classroom. The primary researcher supported this student and school to implement strategies to minimize student anxiety. Students’ response rate to ESM surveys was relatively stable over time, reducing slightly from an average of 75.5% at pre-intervention to 72.8% post-intervention. Some participants reported that their device did not prompt consistently every day, which may have impacted response rate. The primary researcher supported these students and schools to troubleshoot technical issues and substituted these survey instances with paper-based surveys to maximize response rates.

Appropriateness.

Overall, the intervention was well received with most teachers, school leadership staff, and parents reporting In My Shoes was a positive experience and that it was relevant, important, and beneficial to students on the autism spectrum and their peers (see S2 Table). Given the intervention only targeted students in grade 3 and 4 and ran over a term, it is a positive outcome for the feasibility of the intervention, that 25% of school leadership reported the intervention made sustainable changes to the whole school and half of teachers thought it increased the participation of students on the autism spectrum. Most teachers (87.5%), school leadership (100%) and parents (90%) reported they would recommend the intervention to another teacher or school. Students’ feedback was overwhelmingly positive with more than 80% of students reporting they enjoyed participating in In My Shoes and that it was fun, interesting, made sense to them, and that they learnt something new (see S3 Table). Qualitative comments from 200 student feedback surveys highlighted a preference for ‘hands on’ activities such as role play; suggested minimizing time spent sitting on the mat engaging in whole class discussion, and completing surveys related to data collection.

Qualitative data revealed that the intervention has several strengths. Teachers reported experiencing several benefits including improved understanding of autism, ways to support students in the classroom, and their ability to reflect on their teaching and practice. Amanda said,

To be honest there is only so much you can learn at university, your lessons were able to sort of break it all down for me – like how to support someone who has autism or someone who struggles with change – just to know some more strategies for myself has been very helpful.

Teachers valued the ‘ready-to-go’ nature of intervention resources including a detailed interactive PDF manual, online professional learning, and video resources. Teachers reported students engaged particularly well with lessons that involved role play and video modelling, benefited from access to the power-point resources as an additional visual support, and related well to the diversity of characters presented in lesson plans. These findings were supported in student interviews; students added by suggesting researchers incorporate more technology into lesson plans using game-based learning platforms (e.g., Kahoot) and iPads to complete worksheets to maximize student engagement.

Overwhelmingly, a significant benefit of the program has been increased peer understanding and acceptance of autism, according to parents and teachers. Teachers reported specific examples of instances where post intervention students had recognised when a peer (with or without autism) needed help and actively supported peers in the classroom or playground. For example, Lachlan (teacher) said,

She’s had meltdowns in the past and the kids sort of just stared. They did not know what to do. They were worried. Whereas now if she starts blocking her ears, there’s been a few occasions where they’ve come to me and said I think Jessica needs a break. That is straight from the program. She benefits because the kids know how to better help her…the kids have benefit because they have a better knowledge and understanding and so do I.

Jess (teacher) described an incident at lunch where a student on the autism spectrum was standing on their own and the peers approached the student and asked him if he wanted to join in, saying “I don’t know if that would have happened without the program to be honest”.

Implementation and practicality.

Teachers reported some 45-minute lesson plans took longer than expected to deliver (mean: 65 minutes; range: 30 – 90 minutes). Jessica said, “The biggest issue is just, it’s time. By the time we do protective behaviours and everything else, it is finding the time. Some of the lessons were spot on, some of them were too long.” Teachers acknowledged, however, time management is highly dependent on teachers’ skills and experience. Teachers suggested condensing content to enable teachers to deliver the lesson within a 45-minute time frame, simplifying worksheets and, in some cases, substituting worksheets movement-based activities to maximize student engagement.

Teachers reported challenges implementing the whole school and parent component of the intervention, attributing this to a lack of time and resources, lack of priority placed on health in the curriculum, and COVID-19 restrictions. At the time of the pilot, due to COVID-19 restrictions, parents were not permitted to be onsite at schools and whole school events were limited due to social distancing requirements. School events that had been cancelled in the previous term had been pushed forward, which limited time available for whole school activities. Loretta said, “I think it was in the too hard basket, to be honest. COVID has had a lot to do with it and it just kind of got put on the back burner… it’s not the top priority”. Most teachers recommended that to make an impact at a whole school level, the intervention needs to target more grade levels so that there is common terminology and a shared understanding within the school.

Teachers acknowledged the importance of parent involvement but were doubtful about parent uptake even if COVID-19 was not a barrier. Toby (teacher) said,

To be fair I email parents lots of things that they do not even read. I feel like even if it was less, there is just a chunk of parents that are going to read it and there is a chunk that won’t, but I think it has the potential to really get the parents involved in a very positive way.

Maria (teacher) said,

At this particular school, I honestly think health would be brushed off as something that is not the most important subject and certainly not one where parents would feel like they need to come on their workday… especially if they think my kid’s not autistic so it doesn’t apply to me.

Parents of students on the autism spectrum expressed frustration that information was not sent home as they felt it limited their ability to generalize their child’s learning and identify whether changes in their child’s behaviour were due to the intervention or other reasons. Parents expressed a desire to be informed and involved in their child’s learning, but acknowledged the way in which this is implemented needs to be realistic. Jackson (parent) said,

There are some parents who are barely surviving themselves. And it is just a case of pushing the kids out the door… you can throw as much information at them but that is not going to get in because they can’t even take care of themselves. And there’s other people which are probably that proactive, that anything that you bring up, they have probably already considered because they like to be ahead of the curve.

Teachers suggested adapting the program to include children with other neurodiversity’s, such as ADHD, to make the intervention more applicable to a broader student population, which may increase parent buy in and uptake of the intervention at a whole school level. Parents and teachers also suggested condensing written parent information and utilizing other forms of media (e.g., uploading work pieces or videos to school portal) where possible to maximize parent engagement.

Fidelity.

Teachers reported a 25% improvement in confidence in implementing In My Shoes after they completed standardized online professional learning; with 87.5% of teachers reporting, they were ‘fairly confident’ or ‘very confident’ in delivering the intervention in their classroom. There was a 98.8% response rate to online fidelity surveys, which took teachers approximately 4 minutes to complete every week. Teachers reported sending parent information handouts home 74% of the time; attributing this to lack of time and school policies relating to the amount of information that can be sent to parents. Teachers reported conducting activities in the lesson plan as specified in the manual 90% of the time and that students were actively engaged in lessons more than 95% of the time. The primary researcher observed at least one lesson in every classroom, either in person or via video recording. Teachers were observed to deliver lesson plans as specified in the manual on average 90.9% of the time (range: 77.8 – 100%).

Only fifty percent of parents reported reading parent information handouts, with the remaining parents reporting that they did not receive information from their school. The majority of school leadership staff reported their school did not implement whole school activity ideas due to lack of time and resources and COVID-19 restrictions.

Preliminary effectiveness.

Changes in the classroom participation of students on the autism spectrum reported on the BASC-3 SOS. There was a positive trend in student behaviour in the classroom including increased peer interactions and responsivity to their teacher and less inattentive behaviours (see Tables 3 and 4). Students were observed to display significantly less inattentive behaviours and were observed to talk more to peers post intervention. Qualitatively, there were more reports of peers prompting students on the autism spectrum (e.g., to re-engage in an activity, to locate materials, or to help complete a task) and of students appearing happier and more engaged during observations post intervention. Inter-rater reliability was deemed excellent with a weighted kappa of 1.0 (100% agreement) for Part A and an intra-class correlation coefficient greater than 0.90 for all sections in Part B (i.e., α >0.90 response to teacher, α = 0.992; peer interaction, α = 0.997; work on school subjects, α = 0.996; transition, α = 0.998; inappropriate interactions, α = 0.963; inattention, α = 0.994; inappropriate vocalisations α = 0.999).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Difference in BASC-3 SOS Part A observations pre-post intervention for students on the autism spectrum.

https://doi.org/10.1371/journal.pone.0269098.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Difference in BASC-3 SOS Part B observations pre-post intervention for students on the autism spectrum.

https://doi.org/10.1371/journal.pone.0269098.t004

Changes in the subjective school experiences of students on the autism spectrum reported using ESM. There was a statistically significant reduction in students on the autism spectrum reporting difficulties in the classroom post intervention. Several ESM findings trended in a positive direction but did not reach significance. For example, students reported higher levels of enjoyment; needing less help; being with classmates more; finding classwork less difficult; being more interested in classwork; and feeling more sociable and excited when in the classroom post intervention. When students reported that they needed help, classmates helped them more post intervention. Students also reported feeling increased enjoyment and interest and feeling more sociable when with their teacher and reported feeling happier when listening to their teacher post intervention, but the change was not statistically significant.

Some results relating to students’ emotions were inconsistent. Although students reported feeling more interested in classwork, they also reported feeling more worried when participating in classwork post intervention. Students also reported increased enjoyment when with their classmates’, but at the same time reported feeling angrier when with their classmates’ post intervention. Refer to S4 and S5 Tables for differences in ESM data pre-post intervention.

Benefits to students on the autism spectrum were reported in interviews. Parents and teachers reported increased: student self-awareness of their diagnosis and differences; feelings of self-confidence and empowerment; peer connections and sense of belonging. There were also reports of less friendship challenges and improved social (e.g., ability to join in a game and work in groups) and self-regulation skills. One of the teachers, Maria said,

He seems more confident in himself and the fact that people weren’t thinking that because he couldn’t do stuff or that he got upset easily was because there was something wrong with him… the fact that the whole class had an understanding of [autism] and were openly talking about it and accepting of it, made him feel more confident.

Amanda (parent) said, “Its boosted his confidence… it has really made him feel more accepted and that it’s okay to be a bit different”. These notions were supported in interviews with some students on the autism spectrum reporting an increased sense of confidence and feelings of empowerment when sharing their experience of autism with their peers. Some students also reported that they had formed new friendships and that their peers seemed “a little nicer” in that they would play with them more in the classroom and playground post intervention. In some instances, teachers and parents reported increased student participation in the classroom and other school related activities such as assemblies and extra-curricular sport.

Changes in the subjective school experiences of students across the sample. Students across the sample reported statistically significant higher levels of engagement and intrinsic motivation at school post intervention. Students also reported improved peer support while learning, but this was not found to be statistically significant. SEI-E scores at or below the 10^th percentile are most significant indicators of low student engagement. Pre-intervention, students who scored 89 or lower were deemed at risk of low engagement. Post intervention, students who scored 93 or lower were deemed at risk of low engagement. The increase in the cut off for the 10^th percentile from 89 to 93 (4.49%) indicates there was an improvement in the average engagement of students participating in the program post intervention. When analysing students on the autism spectrum data in isolation, no statistically significant differences were found in students self-report school engagement and belonging; with scores for SEI-E and Belonging measures staying the same or reducing slightly (see S6 Table).

A statistically significant improvement in students’ responses to the In My Shoes situation-based scale was noted, which indicates an improvement in students understanding of intervention content; selecting responses that demonstrate behaviour that would lead to the inclusion of their peers in various social situations. Students’ confidence in asking for help and helping others in the classroom and playground reduced slightly post intervention. No change was reported in students’ self-perceived school involvement. A statistically significant improvement in students’ understanding of autism was reported following the second lesson of the intervention. Refer to Table 5 for differences in HCSBS, SEI-E, Belonging, and self-developed scales pre-post intervention across the sample.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Difference in HCSBS, SEI-E, Belonging, In My Shoes scales pre-post intervention across whole sample.

https://doi.org/10.1371/journal.pone.0269098.t005

No statistically significant differences were found between students on the autism spectrum and typically developing peers post intervention across all measures. SEI-E total scores declined for students on the autism spectrum but improved for peers. While differences between students on the autism spectrum and peers SEI-E scores did not reach significance, scores for both samples moved in a positive direction for most subscales. Refer to S7 Table for differences between samples in HCSBS, SEI-E, Belonging and self-developed scales post intervention.

Changes in social skills and behaviour in the home, community and school environment reported using HCSBS and SSBS. Parents of students on the autism spectrum and typically developing peers reported an improvement in their child’s self-management and compliance and reported less defiant/disruptive and anti-social behaviour on the HCSBS post intervention. No statistically significant changes were reported in the social skills of students on the autism spectrum on the SSBS by teachers post intervention.

Recruitment capability and sample characteristics.

The recruitment process for this study was time consuming and eligibility criteria was restrictive. The primary researcher was required to contact school principals in the first instance to determine interest, eligibility and to gain written informed consent. Due to school principals limited availability, this resulted in several phone calls and emails before the primary researcher was able to communicate with students on the autism spectrum, their parents and teachers and prepare for data collection. The fact that we received several enquiries about our study from schools with students experiencing school participation restrictions, without a formal diagnosis of autism, indicates the need for an intervention that focuses on improving students school participation and feelings of connectedness. Although we were able to recruit sufficient participants for this study, we anticipate challenges recruiting large numbers of schools in future studies without broadening eligibility criteria to include students with social challenges without formal diagnosis of autism. This, however, needs to be considered carefully, as broadening eligibility may impact scientific rigor as we will not be able to differentiate intervention effects for different student populations.

Data collection procedures and outcome measures.

We received consistent feedback from students and teachers that the timing and quantity of outcome measures were burdensome. Selecting outcome measures in intervention research is challenging [54]. Several outcome measures are used in feasibility studies to identify the most appropriate measure to use in future efficacy studies [24]. Striking the balance between thorough data collection procedures and feasibility can be particularly challenging in busy school environments. We selected measures that addressed constructs of interest and that had been validated with elementary school students. The measures available, however, had limited psychometric evaluations which may have impacted findings. As we move forward, we may need to develop new measures that align with the theoretical perspectives and hypothesized mechanisms of change reflected in the intervention [23, 54].

Several objective changes were observed in classroom participation of students on the autism spectrum post intervention. Students were observed to display more on task behaviour and interacted more with peers post intervention; a proximal intervention outcome, specifically targeted in the In My Shoes intervention. Students also reported via ESM surveys, that when they needed help and classmates helped them more post intervention. This is an important finding, as it suggests peers have an improved ability to demonstrate pro-social behaviour; an intended outcome of the intervention. Social skills, however, such as the ability to adjust to different behavioural expectations explored using the teacher-report SSBS, were not overtly targeted in the intervention and did not change post intervention. The SSBS was recommended in a recent systematic review evaluating the psychometric properties of social skills measures [55]. This distal outcome measure was important to include in this study as it served an important function in determining if intervention effects transcended immediate intervention targets. These findings suggest that for this intervention and sample size, there were no effects in relation to distal outcomes.

Several factors may have contributed to lack of significant change in the self-report school engagement of students on the autism spectrum. For example, students may have misinterpreted survey items or may have experienced difficulty understanding and applying key concepts of the program specifically relating to perspective taking. While interventions that adopt whole class approaches have their advantages, students on the autism spectrum may benefit from additional individualised support throughout the duration of the intervention to specifically apply concepts and practice skills with peers to support change in intervention outcomes over time. This adaptation, however, would need to be tested to evaluate if it is feasible in the school environment.

Some interesting findings arose from ESM data. For example, students reported feeling more interested in classwork, but also reported feeling more worried when participating in classwork post intervention. This may suggest that students care more about their classwork and therefore feel more worried about their performance in the classroom post intervention. Several inconsistencies, however, were noted in data relating to students’ emotions from ESM surveys. For example, students reported increased enjoyment when with their classmates’, but at the same time reported feeling angrier when with their classmates post intervention. Although students had appropriate reading comprehension and were provided with training in the use of the ESM survey, inconsistencies suggest more training is needed to support students to interpret emotion-specific items.

Appropriateness, implementation, and practicality.

The components of the intervention that were most valued was the whole class program. Teachers valued detailed lesson plans and interactive pre-prepared resources. The whole school and parent component of the intervention, that were less prescriptive and provided schools with flexibility in the way they were delivered, were less valued and therefore not implemented as stated in the manual. This raises important questions about how to best support learning between the classroom and school and between school and home.

All schools and teachers felt that to make a difference at a school level, the intervention needed to be embedded across the school; tailored to as many grade levels as possible. This would help to develop a shared set of values within the school about how students should respond to and support each other and equip schools with the tools they need to facilitate these behaviour transactions. These findings are consistent with school connectedness literature that suggest whole school approaches targeting school organisational environments are effective in promoting a sense of belonging [56, 57]. It is not reasonable to expect systemic change if content is only delivered to a small number of students. Future research should aim to expand In My Shoes across grade levels and provide additional resources to support schools to implement whole school activity ideas. Additional emphasis should also be placed in pre-intervention professional learning on the importance of whole school and parent involvement so that teachers understand the potential impact this could have on intervention outcomes and therefore be more invested in delivering these intervention components. Identifying school leadership staff who will be accountable for implementing whole school activity ideas from the outset would also help to improve the fidelity of this intervention component.

Parents oscillated between wanting to be provided with information and not wanting to be provided with too much information. COVID-19 social distancing restrictions made parent engagement particularly challenging in this study, highlighting how quickly the disconnect between home and school can occur and the amount of effort required in building relationships and sharing knowledge between school and home. Innovative ways to maximise parent engagement, such as presenting written information in functional formats (e.g., condensing weekly information handouts to present key concepts on an A4 sized fridge magnet) and using videos on school portals to demonstrate student learning, should be incorporated in the future.