Establishment of the IHC-Barcelona working group.

The IHC-Barcelona Working Group included a coordination team (which led and coordinated the study), and a multidisciplinary Advisory Group (which reviewed and advised on the different steps of the study). We aimed for profile representativeness of the Working Group members (researchers, teachers, family members, paediatric primary care providers, and education and health stakeholders).

Selection of schools.

We selected a convenience sample of three schools in Barcelona. We used the following eligibility criteria: 1) schools included in the school directory from the Regional Ministry of Education from the Government of Catalonia (2018–2019) [35]; 2) schools that had previously participated in a health promotion programme (2016–2017) [36]; and 3) schools that had previously participated in the initiative Escola Nova 21 (alliance of schools and civil society institutions for updating the Catalan education system during 2016–2019) [37]. We also took into consideration whether the schools included students that were representative of the neighbourhood, if they were in different neighbourhoods of the city, their type of funding (public, publicly-funded private, or private schools), and schools that previously participated in the IHC resources translation into Spanish. Some of these criteria were established to identify schools that had previously collaborated on related initiatives (health promotion programmes, initiative Escola Nova 21, or the IHC project), and thus guarantee participation in the study and intervention adherence.

Selection of the students and teachers.

We included 4th and 5th-grade students (9 to 11-year-olds) from all the classes of the participating schools, and one teacher from each included class.

Intervention in the schools.

We scheduled the intervention for 2019–2020 school year (first intervention or intervention pre COVID-19 pandemic), but in mid-March 2020 all schools closed due to the COVID-19 pandemic. We finished the data collection of the first intervention during the COVID-19 pandemic lockdown. We restarted the intervention in the 2020–2021 school year (second intervention or intervention after COVID-19 pandemic lockdown) and adapted the methods to schools’ COVID-19 prevention and control measures.

The intervention in the schools included: 1) a workshop with the teachers, 2) lessons taught to the students, and 3) assessment of CLAIMs about treatments by the students at the end of the lessons. Each of the activities is summarised below:

• Workshop with the teachers: Before starting the lessons, we ran a workshop to introduce and review the IHC project, the IHC resources, and the study with the teachers. The workshop program included the following sessions: 1) an opening session about Evidence-Based Medicine, 2) the presentation of the IHC project and the IHC resources, 3) the presentation of the study, 4) a mock lesson to the teachers, and 5) teachers’ presentation about their plan to teach the lessons to the students (the workshop program is available in S3 File). The workshop lasted five hours.
• Lessons to the students: The IHC primary school resources include a textbook, an exercise book, a teachers’ guide, some activity cards, a poster, and a song (Fig 1) [10–12, 27–29]. The textbook includes nine lessons that explain 12 IHC key concepts (S4 File). The textbook tells a story, narrated as a comic, about a brother and a sister, John and Julie, who meet two teachers and health researchers, professor Compare and professor Fair. The professors teach the children: 1) what questions they should ask when someone says something about a treatment; 2) what questions health researchers ask to find out more about treatment effects; and 3) what questions they should ask when deciding to use a treatment or not. The IHC resources were designed to be managed by teachers, used over nine weeks, with one lesson per week, during a single term, and one hour to complete a test at the end of the term [9].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The informed health choices learning resources for primary school children (English and Spanish version) [10–12, 27–29].

https://doi.org/10.1371/journal.pone.0288082.g001

We provided schools with the IHC resources. Each teacher received the following printed copies: a textbook, a teacher’s guide, a packet of activity cards and a poster for the class. Each student, if requested by the teacher, received a printed copy of the textbook. We did not provide printed copies of the exercise book for students, instead we asked the teachers to print the exercises if needed.

We asked teachers to at least read and discuss the story for each lesson. Beyond that, they could adapt the lessons to their students depending on the education plan of each school (e.g., continuity and duration of the lessons, completion of the activities/exercises, development of extra activities, resource format and language).

• Assessment of CLAIMs about treatments by the students at the end of the lessons (CLAIM test): After completing all the lessons, the students completed a self-administered test (CLAIM test) to evaluate their ability to apply the 12 IHC key concepts taught during the lessons. The test included 24 questions (two questions for each IHC key concept) from the CLAIM Evaluation Tools item bank [7]. During the first intervention (and under the COVID-19 pandemic lockdown), the students completed the CLAIM test at home and sent it by email. During the second intervention, the students completed the CLAIM test during school time.

Quantitative sample.

All the teachers who participated in the intervention completed ad hoc self-administered questionnaires (before the lessons, after each lesson, and at the end of the lessons) to explore the users’ experience with the IHC resources.

Qualitative sample.

We selected a convenience sample of lessons to conduct non-participatory observations (NPOs) during the lessons and semi-structured interviews (SSIs) with students after the lessons. We expected to perform at least two NPOs from each lesson in randomly assigned classes (18 observations) and two SSIs of each lesson on randomly assigned students (18 interviews).

Quantitative data.

• Assessment of the IHC resources by the teachers before the lessons: Before staring the lessons, the teachers completed an ad hoc self-administered questionnaire that assessed: 1) teacher’s impression of understandability (easy for users to comprehend), desirability (something the users like), suitability (something the users feel is for “someone like me”), and usefulness (helpful to users in achieving their needs) of the IHC resources for their students [38]; 2) understandability, desirability, suitability, and usefulness for themselves; and 3) examples of treatment claims (open-ended questions) (S5.1 File in S5 File). In this assessment, and in the following ones, the understandability, desirability, suitability, and usefulness of the IHC resources were evaluated using a 5-Point Scale (1 meaning completely disagree and 5 meaning completely agree).
• Assessment of the lessons by the teachers after a lesson: After teaching each lesson, the teachers completed an ad hoc self-administered questionnaire that assessed: 1) teacher’s impression of understandability, desirability, suitability, and usefulness for students; 2) understandability, desirability, suitability, and usefulness for themselves; 3) technique used to teach the lessons (close and open-ended questions), 4) facilitators and barriers to teach the lessons, and 5) suggestions to improve the lessons (open-ended questions) (S5.2 File in S5 File).
• Overall assessment of the IHC resources by the teachers at the end of the lessons: After completing all the lessons, the teachers completed an ad hoc self-administered questionnaire that assessed: 1) teacher’s impression of overall understandability, desirability, suitability, and usefulness for students; 2) overall understandability, desirability, suitability, and usefulness for themselves; and 3) suggestions to improve the IHC resources (open-ended questions) (S5.3 File in S5 File).

Qualitative data.

• Non-participatory observations during the lessons (NPOs): Two researchers from the IHC-Barcelona Working Group made the NPOs during the lessons (NPOs were audio-recorded). The researchers conducted the NPOs using an ad hoc guide that assessed: 1) understandability, desirability, suitability, and usefulness for students and teachers, 2) technique used to teach the lessons, 3) facilitators and barriers to teach the lessons, and 4) examples of treatment claims (S5.4 File in S5 File). The researchers made detailed notes during NPOs; these were collected as direct quotes, paraphrases, or researcher hypotheses. They cross-checked quotes and solved discrepancies through discussion; if necessary, they reviewed the audio-recordings. During the second intervention, due to COVID-19 restrictions, we held the NPOs on-line.
• Semi-structured interviews with the students after a lesson (SSIs): After each NPO, two researchers from the IHC-Barcelona Working Group made the SSIs (SSIs were also audio-recorded). The researchers conducted the SSIs using an ad hoc guide that assessed: 1) understandability, desirability, suitability, and usefulness for students, 2) examples of treatment claims, and 3) suggestions to improve the lessons (S5.5 File in S5 File). The researchers made detailed notes during SSIs; these were collected as direct quotes, paraphrases, or researcher hypotheses. They cross-checked quotes and solved discrepancies through discussion; if necessary, they reviewed the audio-recordings. During the second intervention, due to COVID-19 restrictions, we held the SSIs on-line.

Quantitative analysis.

We conducted a descriptive analysis of the categorical variables (absolute and relative frequencies), and the continuous variables (median and range) (S1 File). We only considered the CLAIM test results for the students at schools that completed all the lessons. We calculated the proportion of students with a passing score (basic knowledge of the concepts and how to apply them, 13 points or more over 24), and the proportion of the students with a mastery score (clear knowledge of the concepts and how to apply them, 20 points or more over 24) [39].

Qualitative analysis.

We analysed qualitative data derived from lessons’ NPOs, students’ SSIs, and free-text responses of teachers’ questionnaires (S1 File). We conducted a framework deductive analysis [40] for qualitative data related to several domains of an adapted version of a user-experience honeycomb framework: understandability, desirability, suitability, and usefulness [38]. We applied the following steps: 1) categorisation of quotes using the framework’s themes, and 2) proposal of subthemes under the themes. We conducted a thematic inductive analysis for qualitative data not suitable for the framework: technique used to teach the lessons, facilitators and barriers to teach the lessons, examples of claims about treatment effects, and suggestions to improve the lessons. We applied the following steps: 1) codification of quotes, 2) proposal of descriptive themes, and 3) identification of analytic themes. One researcher categorised and coded quotes, and proposed themes and subthemes independently. A second researcher cross-checked codes, corresponding quotes, themes, and subthemes. Disagreements were initially solved by consensus; if necessary, a third reviewer was consulted.

Mixed methods integration analysis.

We merged quantitative and qualitative results using a joint display to compare and validate the findings [41]. We applied the following steps: 1) mapping quantitative and qualitative results by outcome into a summarized matrix, 2) exploring the convergence (findings from quantitative and qualitative approach agree), complementarity (findings from each approach offer complementary information), or discrepancy (findings from each approach appear to be contradictive) [42], and 3) narrative synthesising of the integration findings. One reviewer conducted the integration analysis independently. A second reviewer cross-checked the summarised results matrix and the narrative synthesis. Disagreements were initially solved by consensus; if necessary, a third reviewer was consulted.

Participants.

The IHC-Barcelona Working Group included 21 participants (researchers, teachers, family members, and paediatric primary care providers) (Table 1), most of them were women (18/21; 85.7%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Participants of the study.

https://doi.org/10.1371/journal.pone.0288082.t001

We identified 947 schools in Barcelona [35]; of those, 34 fulfilled the eligibility criteria and three agreed to participate in the study (School A, B, and C) (S6 File). Two primary schools in Barcelona (one public [School A], and one publicly-funded private school [School B]) conducted the intervention. The school that dropped out (School C) was a public school, included a more disadvantaged population, younger students (9 to 10-years-old), only one teacher agreed to participate, and planned to include the lessons in the “Spanish language” subject.

We included a total of 143 students in the study, 100 during the first intervention (pre COVID-19 pandemic) and 43 during the second intervention (after COVID-19 pandemic lockdown) (Table 1). A higher proportion of students were 10 to 11-year-olds (100/143; 69.9%), and there were more boys (81/143; 56,6%) than girls. The students were divided in six classes, with 21 to 26 students in each class. Six teachers participated in the study; all of them were women. The teachers presented the study to the family members during school meetings, and most families agreed to participate (143/147; 97.3%).

Workshop with the teachers.

The teachers and other members of the IHC-Barcelona Working Group participated in the workshop (15 participants) (Table 1). The workshop took place in November 2019 at Hospital de la Santa Creu i Sant Pau (Barcelona).

Lessons to the students.

One school (School A) completed all the lessons during the first intervention (January-March 2020) (S7.1 File in S7 File). Each teacher from School A taught the lessons to one class. They scheduled one lesson per week, with approximately 60 minutes per lesson. The lessons were included in two different subjects, “Social and civic values” and “Spanish language”, and were conducted in Spanish. They used printed IHC resources translated into Spanish, and we provided a printed copy of the textbook for each student.

The other school (School B) only completed two lessons during the first intervention (February-March 2020), and 3–5 lessons during the second intervention (April-June 2021) (S7.1 File in S7 File). Each teacher from School B taught the lessons to one class. They adapted the lessons to include them in their extensive project-based learning methodology (each lesson was divided into different sessions to work together with other related content in the curriculum) [44]. The lessons were included in the subject “Science” and were conducted in English as it is the vehicular language of this subject. They used digital IHC resources in English, and we provided some printed copies of the textbook translated into Spanish for students with English comprehension difficulties.

CLAIM test.

A total of 37 students completed the CLAIM test (37/51 students from School A that completed all the lessons, response rate 72.5%, Table 1). Almost all the students obtained a passing score on the test (36/37, 97.3%), and more than half of the students achieved a mastery score (23/37, 62.2%) (S7.2 File in S7 File).

Quantitative findings.

We collected quantitative data from the following data collection activities (S7.1 File in S7 File): six assessments of the IHC resources by the teachers before the lessons, 26 assessments of the lessons by the teachers after a lesson, and two overall assessments of the IHC resources by the teachers at the end of the lessons.

Understandability, desirability, suitability, and usefulness of the IHC resources. Before starting the lessons, teachers expected that students would understand and would be able to apply the content of the lessons in their daily lives (median score 4 [1 meaning completely disagree and 5 meaning completely agree] in assessment before the lessons for both items), but they thought that students would be slightly less interested in them (median score 3.5 in assessment before the lessons) (Table 2). During the lessons, students understood (median score 5 in assessment after a lesson), were interested (median score 4 in assessment after a lesson), and were able to apply the content of the lessons (median score 4 in assessment after a lesson). Teachers understood the content of the IHC resources (median score 5 in all the assessments), found them interesting (median score 4.5 in assessment before the lessons, and median score 5 in assessment after a lesson and at the end of the lessons) and suitable (median score 4 in assessment before the lessons, 5 in assessment after a lesson, and 4.5 at the end of the lessons).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Students’ and teachers’ experience with the IHC resources.

https://doi.org/10.1371/journal.pone.0288082.t002

In terms of usefulness of the IHC resources, teachers thought that the IHC resources would be useful for the students (median score 4 in assessment before the lessons, 5 for the textbook in assessment after a lesson, and 5 at the end of the lessons). Before starting the lessons, the teachers mainly expected the textbook to be useful (median score 4.5 in assessment before the lessons); after the lessons and at the end of the lessons, they reported that both, the textbook and the teachers’ guide, were useful (median score 5 for the textbook and the teachers’ guide in both assessments). Teachers found the poster not useful, neither for the students nor for themselves.

Technique used to teach the lessons. The teachers used the techniques of the proposed plan in the IHC resources to teach the lessons, including a review of the previous lesson, reading the lesson’s comic, a discussion, and the completion of the activity/exercises (S7.3 File in S7 File). They also used other methods or strategies.

Facilitators and barriers to teach the lessons. The most frequent facilitators for teaching the lessons that teachers identified were related to themselves (their ability to adapt the instructions to their own style and context, their feeling of self-efficacy, and their understanding of the content), the IHC resources (credibility of the material), and the students (attitudes, beliefs, and motivation to learn) (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Facilitators and barriers to teach the lessons.

https://doi.org/10.1371/journal.pone.0288082.t003

Teachers identified fewer barriers than facilitators (Table 3). Most of them were related to the students (peer influence and differentiated instruction) and the school system and environment (time constraints).

Qualitative findings.

We collected qualitative data from the following data collection activities (S7.1 File in S7 File): 14 NPOs and 14 SSIs (nine on-site during the first intervention and five on-line during the second intervention, one or two observations/interviews for each lesson, except for Lesson 6 which we did not attend). We also added the answers from open-ended questions included in the questionnaires. Two researchers from the Working Group conducted the NPOs and SSIs (both were female, medical doctors and one of them had experience in qualitative research). A higher proportion of interviewed students were 10 to 11-year-olds (9/14; 64.3%), and the majority were girls (8/14; 57.61%).

Understandability, desirability, suitability, and usefulness of the IHC resources. During the lessons, students and teachers understood, were interested, and were able to apply the content of the IHC resources. Relevant quotes about the understanding of the content were noted (Fig 2). Students defined some of the terms to work on (e.g., definition of health or treatment) (Fig 2, quotes 1 and 2). They also addressed other concepts not included in the IHC resources (e.g., placebo effect, washout period, or cumulative effect) (Fig 2, quotes 3–5). Moreover, they established relationships between different concepts (e.g., unreliable claims could be based on more than one bad basis, or positive effects versus disadvantages of a treatment) (Fig 2, quotes 6 and 7). Teachers also reinforced the meaning of the concepts by repeating their definition, giving examples, or transferring the concepts to other areas (Fig 2, quotes 8–10). In terms of usefulness of the IHC resources, we observed that students used the textbook (printed or digital format); however, teachers rarely used their printed copy of the textbook or teachers’ guide during the lessons.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Relevant quotes.

Abbreviations: ICT: Information and Communications Technologies; IHC: Informed Health Choices.

https://doi.org/10.1371/journal.pone.0288082.g002

All the interviewed students were able to explain what they had learned after the lessons (Fig 2, quote 11–13). A few students identified some concepts that were difficult to understand (e.g., random allocation or how to make a decision about treatments) (Fig 2, quote 14 and 15). Most of the interviewed students enjoyed the lessons and thought they were interesting. Some of them explained the aspects of the lessons that they enjoyed the most (e.g., learning something new, the comic, and the animated version of Lesson 2) or disliked the most (e.g., explanations of concepts and repetitive activities) (Fig 2, quote 16–20). More than half of the students expressed that they could apply what they learned in their daily life; when we asked about the reason, they were able to identify their treatments (e.g., vaccines, bandages, or ibuprofen) and the advantages and disadvantages of those (Fig 2, quote 21). Moreover, some of them considered that what they learned could be helpful for their family (Fig 2, quote 22). Only one of the interviewed students noted that the textbook included examples from other settings (Fig 2, quote 23).

Technique used to teach the lessons. Teachers from School A almost always followed the proposed plan in the IHC resources, including a review of the previous lesson, reading the lesson’s comic, a discussion, and the completion of the activity/exercises. Teachers from the School B implemented the lessons as project-based learning, and they substantially changed the proposed plan in the IHC resources (S7.4 File in S7 File). Nevertheless, they maintained some steps of the plan, such as reading the lesson’s comic, the discussion, and completion of exercises.

In both schools, teachers used other methods or strategies to teach the lessons, which we classified into two main themes: adaptation of the IHC resources, and use of Information and Communications Technologies (ICT) tools.

• Adaptation of the IHC resources: All teachers developed other activities/exercises and work materials based on the IHC resources to promote students’ participation. Some of the resources developed by teachers from School A included: a worksheet with concepts and definitions (Fig 2, quote 24; S8.1 File in S8 File), cards with the treatment illustrations from the textbook (Fig 2, quote 25; S8.2 File in S8 File), a diagram with treatment advantages and disadvantages (Fig 2, quote 26; S8.3 File in S8 File), green/red response cards for true or false exercises (Fig 2, quote 27), or a summary activity (Fig 2, quote 28).

Teachers from School B also developed resources to work on IHC contents, for example, a worksheet with examples of claims (S8.4 File in S8 File), a worksheet with concepts and definitions (S8.5 File in S8 File), an exercise to identify bad bases for claims (Fig 2, quote 29; S8.6 File in S8 File), and a worksheet with exercises (S8.7 File in S8 File). Furthermore, they developed resources ad hoc to work on other related content, including how to develop and analyse surveys (health survey activity [Fig 2, quote 30; S8.8 File in S8 File]), the comic format (a questionnaire about the comic [S8.9 File in S8 File], a worksheet to create a new comic character [S8.10 File in S8 File]), or English language (English vocabulary exercise about common symptoms [S8.11 File in S8 File], a worksheet to develop a health word search [S8.12 File in S8 File], a reading comprehension worksheet [S8.13 File in S8 File], and an exercise to create or listen to dialogues between doctors and patients [S8.14 and S8.15 Files in S8 File]).

• Use of Information and Communications Technologies (ICT) tools: Teachers and students used different ICT tools (including equipment [hardware] and computer programs [software]) during the lessons. For example, teachers from School A created PowerPoint presentations for each lesson to explain the content and present the activities/exercises (Fig 2, quote 31). Teachers from School B projected the comic on a screen to read the comic aloud, and students from School B had personal computers and they used the application Google Classroom to download the textbook and lesson materials.

Facilitators and barriers to teach the lessons. Teachers were deeply engaged to teach the lessons (attitudes, fit to the teacher’s teaching style and context, self-efficacy, motivation, and positive learning environment). Students also contributed to the adequate progress of the lessons (literacy, motivation to learn, peer influence, and attitudes).

We identified few barriers, some of these were related to the students (peer influence and attitudes) and the school system and environment (the schedule of the lessons affected the teaching, as the students’ attention was lower in the afternoon lessons than in the morning ones [Fig 2, quote 32]).

We observed circumstances that were both, a facilitator and a barrier. For example, sometimes students’ attitude and peer influence facilitated the classroom dynamics (e.g., a student asked a question, and a classmate answered it), but other times they hindered the progress of the lesson (e.g., a noisy classroom).

Examples of claims about treatment effects. During the first intervention, students and teachers identified examples from their daily lives, mainly based on personal experiences and non-serious illnesses (e.g., bites, colds, specific pain, or aesthetic treatments) (Fig 3). During the second intervention, students and teachers gave several examples of treatment claims related to COVID-19.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Examples of claims about treatment effects.

https://doi.org/10.1371/journal.pone.0288082.g003

Suggestions to improve the lessons. Teachers suggested ideas to improve the lessons based on the activities that they implemented to teach the lessons or their teaching experience. We used the two themes above to classify the suggestions: 1) adaptation of the IHC resources to promote students’ participation (Fig 2, quotes 33–39), and 2) use of ICT tools to promote students’ participation (Fig 2, quote 40) or to support the teaching of the lessons (Fig 2, quotes 41–44). Some of the interviewed students also made suggestions to improve the lessons and highlighted that the comic was what interested them the most (Fig 2; quotes 45 and 46).

Mixed methods integration findings.

When integrating the qualitative and quantitative findings into a summarized matrix, we observed almost a complete convergence of the results (S7.5 File in S7 File). Although, two integration concepts emerged:

• Discrepancy in usefulness of the IHC resources for teachers: We observed discordance in qualitative and quantitative findings about the usefulness of the IHC resources for teachers. Before starting the lessons, they expected the textbook to be useful (median score 4.5); after the lessons and at the end of the lessons, they reported that the textbook and the teachers’ guide were useful (median score 5). However, during the NPOs we observed that they rarely used their printed copy of the textbook or the teachers’ guide. This discrepancy could be due to the teachers rating the IHC resources as highly useful because they used them to prepare the lessons (adaptation of the IHC resources); instead, observers rated the IHC resources as less useful because teachers made many adaptations to them.
• Complementarity in techniques used by teachers to teach the lessons: We observed that qualitative findings complemented the quantitative findings about how teachers delivered the lessons. Teachers used the techniques of the proposed plan in the IHC resources (review of the previous lesson, reading the lesson’s comic, a discussion, and the completion of the activity/exercises), and other methods or strategies. During the NPOs, we observed differences in the implementation of the techniques of the proposed plan in the IHC resources between participating schools, and we captured in detail how other strategies and methods were implemented.

Implications for practice.

We formulated recommendations to support the use of the IHC resources in our setting, which involve teachers, the IHC resources, and ICT tools. Before implementing the IHC resources, each school should consider these recommendations to improve the end users’ experience.

Additionally, further efforts are needed to disseminate and promote the use of the IHC resources in schools in our setting.

Implications for research.

It is necessary to continue exploring the barriers to use of IHC resources in our setting, especially the barriers that limit the access and participation of schools. In parallel, other strategies to teach IHC key concepts outside school contexts should be explored (e.g., summer camps).

We are currently working on other contextualization activities to ensure the relevance and appropriateness of the IHC resources for Spanish primary school children 1) systematic assessment to identify and describe relevant educational documents and resources that support teaching of critical thinking about health in Spanish primary schools [58], 2) translation of the IHC learning resources into other local language (Catalan), and 3) development and validation of an interactive test to measure the ability of Spanish primary school children to assess treatment claims and make informed health choices (the interactive CLAIM Test) [59].