https://orcid.org/0000-0001-5963-8314
Figures
Abstract
Background
The Informed Health Choices Key Concepts are principles for thinking critically about healthcare claims and deciding what to do. The Key Concepts provide a framework for designing curricula, learning resources, and evaluation tools.
Objectives
To prioritise which of the 49 Key Concepts to include in resources for lower secondary schools in East Africa.
Methods
Twelve judges used an iterative process to reach a consensus. The judges were curriculum specialists, teachers, and researchers from Kenya, Uganda, and Rwanda. After familiarising themselves with the concepts, they pilot-tested draft criteria for selecting and ordering the concepts. After agreeing on the criteria, nine judges independently assessed all 49 concepts and reached an initial consensus. We sought feedback on the draft consensus from other stakeholders, including teachers. After considering the feedback, nine judges independently reassessed the prioritised concepts and reached a consensus. The final set of concepts was determined after user-testing prototypes and pilot-testing the resources.
Results
The first panel of judges prioritised 29 concepts. Based on feedback from teachers, students, curriculum specialists, and members of the research team, two concepts were dropped. A second panel of nine judges prioritised 17 of the 27 concepts that emerged from the initial prioritisation and feedback. Based on feedback on prototypes of lessons and pilot-testing a set of 10 lessons, we determined that it was possible to introduce nine concepts in 10 single-period (40-minute) lessons. We included eight of the 17 prioritised concepts and one additional concept.
Citation: Agaba JJ, Chesire F, Mugisha M, Nandi P, Njue J, Nsangi A, et al. (2023) Prioritisation of Informed Health Choices (IHC) key concepts to be included in lower secondary school resources: A consensus study. PLoS ONE 18(4): e0267422. https://doi.org/10.1371/journal.pone.0267422
Editor: Sara Rubinelli, Universitat Luzern, SWITZERLAND
Received: April 7, 2022; Accepted: March 21, 2023; Published: April 7, 2023
Copyright: © 2023 Agaba et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: This research was funded by the Research Council of Norway (https://www.forskningsradet.no/en/). Project number 284683, grant no:69006 awarded to ADO. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Dewey noted the importance of teaching concepts over a century ago [1]. “It follows that it would be impossible to overestimate the educational importance of arriving at conceptions: that is, of meanings that are general because applicable in a great variety of different instances in spite of their difference; that are constant, uniform, or self-identical in what they refer to, and that are standardized, known points of reference by which to get our bearings when we are plunged into the strange and unknown.” [2].
We have identified concepts that people need to understand and apply when deciding what to believe about health actions and what to do. “Health actions” are things that individuals or groups can do (“interventions” or “treatments”) to care for their health or the health of others. The “Informed Health Choices (IHC) Key Concepts” provide a framework for curriculum planning and designing learning resources [3]. As a first step towards developing learning resources for students in the first two years of secondary school (lower secondary school) in Kenya, Rwanda, and Uganda, we needed to decide which of the 49 IHC Key Concepts (Table 1) should be taught in this context. In these countries, this is the 8th or 9th year of school, and the normal starting age is 13 or 14.
Teachers may be overwhelmed by the amount of content they are expected to cover, especially when standards are viewed as discrete and disconnected. Marzano and Kendall reviewed 160 national and state-level documents listing standards in various subject areas in the USA and synthesized the material to avoid duplication. They identified 255 content standards and 3,968 discrete benchmarks that delineate what students should know and be able to do [4]. They estimated that if teachers devoted 30 minutes of instructional time to teach each benchmark, they would need an additional 15,465 hours (nine school years).
This is consistent with findings of a process evaluation we conducted to explore barriers to scaling up use of the IHC primary school intervention in Uganda [5]. The intervention consisted of providing the IHC primary school resources, as well as teacher training workshops. It was shown to have a large effect on primary school children’s ability to think critically about health claims [6], which was sustained after one year [7]. Teachers who used the primary school intervention in the trial said: the IHC Key Concepts were important; they were motivated to teach the concepts; and the children were enthusiastic about the lessons. The main barrier we identified to scaling up use of the intervention was the need to incorporate the lessons in the national curriculum. The IHC lessons were viewed as an addition to what was already a packed primary school curriculum.
It is essential to prioritise what to include in school curricula. Wiggins and McTighe argue that prioritising should focus on “big ideas” and “core tasks”; “A big idea is a concept, theme, or issue that gives meaning and connection to discrete facts and skills,” while a core task is “the most important performance demands in any field” [8]. Priorities should be established by building upon the big ideas and by focusing schoolwork on core tasks or “transfer tasks” derived from authentic challenges. In the same vein, Bruner writes: “For any subject taught in […] school, we might ask [is it] worth an adult’s knowing, and whether having known it as a child makes a person a better adult. A negative or ambiguous answer means the material is cluttering up the curriculum” [9].
The basic principle underlying Wiggins’ and McTighe’s approach to curriculum design–“backward design”–is to begin with the desired, final outcomes and to focus on the learner’s needs. Rather than building a curriculum around the logic of the content, a curriculum should be designed around the needs of learners trying to understand the big ideas and to perform the core tasks. This principle is consistent with Bruner’s idea of a “spiral curriculum” based on recurring, deepening inquiries into big ideas and important tasks, helping students learn in a way that is developmentally sensible and effective; “The basic ideas at the heart of all science and mathematics and the basic themes that give form to life and literature are as simple as they are powerful. To be in command of these basic ideas and use them effectively requires a continual deepening of one’s understanding of them that comes from learning to use them in progressively more complex forms.
This paper describes a process in line with the thinking of Wiggins and McTighe and of Bruner, in which we prioritised and ordered IHC Key Concepts to be included in lower secondary school resources for Kenya, Rwanda, and Uganda. Our intention was to develop a spiral curriculum for a series of resources.
Objective
To prioritise and order IHC Key Concepts to be included in the IHC lower secondary school resources.
Methods
We used an iterative, structured consensus process built on Wiggins’ and McTighe’s “backward design” approach [8], the Nominal Group Technique consensus process [10], and Feinstein’s criteria for sensibility [11]. The process, as described in the protocol for this study [12], included six steps. A seventh step that was not described in the protocol was added to the process. Each of these steps is described below.
1. Selecting and training the judges
Ten judges were purposively selected to prioritise the IHC Key Concepts to be included in the IHC secondary school resources (Table 2). We aimed to include judges from all three countries who were familiar with the context, including curriculum specialists and teachers, in addition to researchers. Three of the judges were curriculum specialists or teachers, one was a health promotion officer, and the other six were health researchers who were members of the project team and were familiar with the IHC Key Concepts.
At the first online meeting of the panel in November 2019, ADO presented an overview of the research project. He then introduced the IHC Key Concepts, the protocol for prioritising the concepts, and plans for pilot-testing proposed criteria for prioritising the concepts [12]. Three criteria were used in the pilot to make two judgements about each of nine Key Concepts:
Criteria
- Are the learners likely to be able to understand and use the concept?
- How important is the concept to the learners?
- Are there sufficient time and resources to help them learn the concept?
Judgements
- Should the concept be included?
- When should it be included?
2. Deciding on criteria and response options for judgements
Following the pilot, the panel discussed the criteria and response options and agreed on six revised criteria (Table 3). These included the importance of each concept for the “central ideas” and the “core tasks” (Table 4). Nine of the judges then independently assessed each of the 49 Key Concepts using those criteria. ADO summarised their assessments and facilitated a discussion that resulted in an initial prioritisation of the Key Concepts. For each concept, people from each end of the range of judgements (between 1 and 7) were invited to provide the reasons for their judgements, before others were invited to comment. The outcome of the meeting was summarised, fed back to the judges and discussed. This resulted in a consensus regarding the criteria that would be used and a shared understanding of the judgements the panel was to make.
3. Prioritising and ordering concepts
Each judge independently assessed the 49 IHC Key Concepts in the 2019 version (Table 1), using the revised criteria. Their judgements were again summarised by ADO, including each judgement made by each judge, and the mean and the median judgements. As was done in the pilot, the judges reviewed this summary and then met. They discussed each concept and reached a consensus on whether each concept should be prioritised for inclusion in the secondary school resources. For each concept, judges from each end of the range of judgements (between include (1) and exclude (7) (Table 3) were invited to provide the reasons for their judgements, before others were invited to comment. The results of the discussion were summarised by ADO and fed back to the judges.
4. Reaching initial consensus
The judges met again to discuss the summary and make any necessary changes to the prioritisation and ordering of the concepts. This resulted in an initial consensus.
5. Collecting feedback
We collected informal feedback on the initial prioritisation from teacher and student networks and advisory groups in each country, from our international advisory group, and from other members of our research team [13].
6. Prioritising and ordering concepts and reaching a consensus
The nine judges who participated in the final prioritisation are shown in Table 5. The second panel included a curriculum specialist and an education researcher from Rwanda who did not participate in the initial prioritisation. A health promotion officer from Kenya and a Public health researcher from Rwanda that participated in the first panel did not participate in the final panel. In addition to prioritising the concepts, both panels also made judgements about when each included concept should be taught (Table 3).
7. Agreeing on a subset of concepts to be included in a single set of resources
Our original plan was to develop two sets of learning resources to be used during two school terms. However, due to the Covid-19 pandemic, school closures, and project delays, it was only possible to produce one set of resources for a single school term. Therefore, after collecting feedback on prototypes of the learning resources, the research team proposed which concepts to include in that set of resources. The second consensus panel approved the proposed concepts.
Ethical considerations
The judges who were not members of the project team consented to participate in the introductory meeting by email. They consented to participate in the consensus process verbally at that meeting after being informed about the overall project and the methods that would be used. The teachers who were members of the teacher networks were informed about the overall project, including their role in providing input and feedback, and provided written consent. The students in the student networks also were informed about the overall project including their role in providing input and feedback. They provided written assent and their parents provided written consent.
Approval for the entire Informed Health Choices project was obtained from Masinde University of Science and Technology Institutional Ethics Review Committee, the Kenya National Commission of Science and Technology Institute (License number: NACOSTI/P/19/1986), the Rwandan National Ethics Committee (RNEC) (approval number 916/RNEC/2019), the Makerere University School of Medicine Research Ethics Committee, and the Uganda National Council of Science and Technology (Reference number HS91ES). The project protocols are updated, and ethics approval is renewed annually in Kenya, Rwanda, and Uganda. A data privacy impact assessment for the project was conducted in accordance with the European Union General Data Protection Regulation and internal approval obtained from the Norwegian Institute of Public Health (Reference number 30713).
Results
In the initial prioritisation, the average score for including each of the 49 concepts ranged from 1.2 to seven (S1 File). It was 1.2 for these two concepts:
- Failure to consider unpublished results of fair comparisons may result in estimates of effects that are misleading.
- Treatment claims based on models may be sensitive to underlying assumptions.
It was seven for these three concepts:
- Treatments can cause harms as well as benefits.
- Your existing beliefs may be wrong.
- Personal experiences or anecdotes alone are an unreliable basis for most claims.
Seventeen concepts had an average score greater than six. Nine concepts had an average score between five and six, eight had an average score between four and five, and 15 had an average score less than four. After discussing the scores, the panel agreed on prioritising 29 of the 49 concepts (“Consensus 1” in Table 6).
Based on feedback from teachers, students, curriculum developers, and other members of the research team, two of the 29 concepts were not considered in the second consensus process:
- If possible, people should not know which of the treatments being compared they are receiving.
- Average differences between treatments can be misleading.
In the final prioritisation, the average score for including each of the 27 concepts that were considered (Consensus 2a in Table 6) ranged from 4.3 to seven (S2 File). It was 4.3 for this concept:
- The results of one study considered in isolation can be misleading.
It was seven for this concept:
- Treatments can cause harms as well as benefits.
After discussing the scores, the panel agreed on prioritising 17 of the Key Concepts (Consensus 2b in Table 6). After collecting feedback on prototypes of the learning resources, the research team suggested including nine concepts in the IHC secondary school resources (Final consensus in Table 6). The second consensus panel agreed on those nine concepts.
One concept that was not one of the 17 prioritised concepts, was included as one of the nine concepts:
- Small studies may be misleading.
That concept had the same average score (5.4) as the two included concepts with the lowest score. That concept was included because only one other concept was included from the second group of concepts (about comparisons):
- Comparison groups should be as similar as possible.
We felt that it was important to include the additional concept that addresses the risk of being misled by the play of chance (Small studies may be misleading) as well as the concept that addresses the risk of bias (Comparison groups should be as similar as possible).
Ordering of the concepts
Based on the second panel’s average scores, the prioritised concepts were ordered as shown in Table 7. However, based on feedback on early lesson prototypes and pilot-testing of a complete version of the lessons, the order in which the concepts are introduced was modified. The final order in which the concepts were introduced in the 10 lessons is shown in the last column in Table 7, and an overview of the 10 lessons is shown in Table 8. Because students and teachers, as well as others, frequently understand “treatment” narrowly to only include medical care given to a patient, we have used “health action” in the secondary school resources.
Discussion
Using an iterative process that engaged curriculum planners, teachers and researchers, we prioritised 17 key concepts to teach lower secondary school students in Kenya, Rwanda, and Uganda. The prioritised and ordered Key Concepts were the starting point for the IHC secondary school resources that we developed for lower secondary schools in East Africa. In parallel, we conducted a context analysis in each country [14–16]. The context analyses explored where teaching the Key Concepts best fits in the curriculum. They also explored conditions for introducing the learning resources into schools. These included the availability of time, the availability and use of digital learning resources, who decides what learning resources are used, and what influences their decisions.
Based on feedback on prototypes of the lessons, we prioritised eight of the concepts and one concept not prioritised by the second consensus panel. There were several reasons for this. First, because of limitations to our funding and the pandemic, we decided to develop only a single set of lessons that could be taught in a single school term. We had originally planned to develop two sets of learning resources. Based on feedback from curriculum specialists and teachers in the three countries, we determined that it would be difficult to find time in the curriculum for more than 10 single-period (40-minute) lessons in a single school term. Moreover, based on feedback on prototypes of the lessons, we determined that it was necessary to introduce some basic terminology in the first two lessons, such as “health action”, “effect”, and “claim”. In addition, the last lesson was used as a review of the first nine lessons. Based on findings from piloting the first complete set of lessons, we decided to include a second review lesson (Lesson 5).
Judgements of the consensus panel regarding the order in which the concepts should be taught were based on Wiggins’ and McTighe’s argument that a curriculum should be designed around the needs of learners trying to understand the big ideas and to perform the core tasks, rather than around the logic of the content. However, when designing and revising the lessons, we chose not to introduce the concepts in the order suggested by the consensus panel. One reason for this was that the logic underlying the organisation of the Key Concepts into three groups (claims, comparisons, and choices) corresponds directly to the needs of learners trying to understand the big ideas and to perform the core tasks (Table 4). Thus, it was both logical and consistent with learners’ needs to group the lessons into claims, comparisons, and choices, as shown in Table 8. We also felt it was important to include concepts from each of the three groups (claims, comparisons, and choices) in the lessons. Another explanation might be that the judgements of members of the consensus panel were not informed by experience teaching the concepts.
What this study adds to what was previously known
A systematic review that included 415 studies found that other educational interventions to improve people’s ability to think critically about healthcare claims included only a handful of the key concepts [17], and it is unclear how those concepts were prioritised.
In a previous project where we prioritised concepts for primary school resources [18], the judgements were made by the research team. At that time, there were 32 Key Concepts in the framework [19]. A network of teachers in Uganda assessed the relevance of the 32 concepts and determined that 24 of those concepts were relevant to primary school children in Uganda [18]. However, all 24 concepts proved to be too much to learn in a school term. The early prototypes of the primary school resources that we created had too many concepts per lesson and took too long to teach in a normal school period (40 minutes). We also observed that the teachers needed time to repeat material from previous lessons. After each round of prototyping, we eliminated more Key Concepts from our list. We decided which ones to eliminate by considering the importance of the concepts and the difficulty that the children had learning them. The importance of the concepts was based on judgements made by members of the research team by:
- Each person individually identifying which of the 24 Key Concepts they considered most important
- Compilation and discussion of those judgements
- Voting on the concepts
- Reaching a consensus by informal discussion
The research team reached agreement that eight of the concepts were most important for our target population in Uganda. Three members of the research team also reviewed data from our piloting and user-testing and identified concepts that appeared to be too difficult to teach to 10- to 12-year-old children. Later, based on feedback from piloting the resources, we considered how the concepts were grouped in the lessons and the number of concepts being taught in each lesson. We ended up including 12 of the 24 concepts in the final version of the resources (Table 9) and reorganised them into three groups to simplify and clarify their purpose:
- Claims: “questions you should ask when someone says something about a treatment”
- Comparisons: “questions that health researchers ask to find out more about the effects of treatments”
- Choices: “questions that you should ask when you are choosing whether to use a treatment”
The main reason for including fewer concepts in the secondary school resources was that 10 double periods (80 minutes) were used for the primary school resources–twice as much time as for the secondary school resources. In addition, the primary school resources included printed materials for the students (including textbooks), whereas the secondary school resources are mainly digital resources for teachers, with optional handouts for students. Thus, the students are more dependent on teachers being well-prepared to teach the lessons. The reason for using digital learning resources and limiting each lesson to a single 40-minute period was to increase the likelihood that use of the resources will be scaled up if the intervention is shown to be effective.
Strengths and limitations of this study
Strengths of this study include:
- involvement of curriculum specialists, teachers, and researchers in the three countries for which the IHC Key Concepts were prioritised,
- use of explicit criteria,
- independent judgements by a panel of judges, and
- feedback from teachers and other stakeholders before finalising the priorities.
Changes to the priorities based on feedback on prototypes and judgements made by the research team could be viewed as either a strength or a limitation. It is important to be pragmatic, and we view the changes made based on user-testing prototypes and pilot-testing all 10 lessons as essential. The nine concepts were not being taught in any of the three countries and teachers had no prior experience teaching the concepts.
An important limitation of this study is that the concepts were prioritised independently of the rest of the curriculum. Ideally, IHC Key Concepts could be prioritised together with other important concepts in the curriculum. Rwanda implemented a new competence-based curriculum in 2016. Uganda introduced its new competence-based curriculum for lower secondary schools in 2020, and Kenya has plans to introduce a new competence-based curriculum by 2024 [14–16]. The new curricula in all three countries include critical thinking as a core competence and they include health topics. However, critical thinking about health is not explicitly included in any of the curricula, and both critical thinking and health are taught across subjects. This limited our ability to integrate the IHC Key Concepts into the curricula. In addition, teaching is exam-oriented in all three countries. Since IHC Key Concepts are currently not assessed in national examinations, this may further limit implementation of teaching the prioritised concepts.
An additional limitation is that we were not able to design resources to support a spiral curriculum, as originally intended. Ideally IHC Key Concepts would be introduced in primary school [6]. What was learned in primary school could then be reinforced and built upon in secondary school over several school terms. We also did not prioritise competences or dispositions [3]. However, these overlap substantially with the concepts, as reflected in the learning goals for the 10 lessons (Table 8).
Implications
There is substantial overlap between the IHC Key Concepts that were prioritised for our primary school and secondary school resources. The included concepts appear to be a good starting point for teaching children and adolescents to think critically about health actions. There remains a need to develop a spiral curriculum and a series of resources that can be used in primary and secondary schools, and that can be easily adapted to different contexts.
Conclusions
Young people are confronted with an excess of health information, including a large amount of misinformation. The ability to distinguish between reliable and unreliable information about the effects of health actions depends on understanding and applying concepts in the IHC Key Concept framework. Many people have not learned many of those concepts [6, 20, 21]. It is not practical to teach or learn all the IHC Key Concepts in a single school term. Thus, is important to prioritise which concepts to teach. Using an iterative process with explicit criteria, we have prioritised nine concepts as a starting point for lower secondary school students in Kenya, Rwanda, and Uganda [3].
Supporting information
S1 File. First consensus prioritisation of key concepts.
https://doi.org/10.1371/journal.pone.0267422.s001
(XLSX)
S2 File. Second consensus prioritisation of key concepts.
https://doi.org/10.1371/journal.pone.0267422.s002
(XLSX)
Acknowledgments
We are grateful to members of the teacher and student networks in Kenya, Rwanda, and Uganda who provided feedback on the results of the first consensus and feedback on prototypes of the lower secondary school resources, to members of the international and national advisory groups who provided feedback, and to other members of the research team for their input.
References
- 1.
Dewey J. How We Think. Boston: DC Heath and Company; 1910.
- 2.
Dewey J. The educational significance of concepts. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. 2nd ed. Boston: DC Heath and Company; 1933. p. 153–4.
- 3. Oxman AD, Chalmers I, Austvoll-Dahlgren A, Informed Health Choices Group. Key Concepts for assessing claims about treatment effects and making well-informed treatment choices. F1000Res. 2019;7:1784. pmid:30631443
- 4.
Marzano RJ, Kendall JS. A Comprehensive Guide to Designing Standards-Based Districts, Schools, and Classrooms. Alexandria, VA: Association for Supervision and Curriculum Development; 1996.
- 5. Nsangi A, Semakula D, Glenton C, Lewin S, Oxman AD, Oxman M, et al. Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation. BMJ Open. 2019;9(9):e030787. pmid:31511291
- 6. Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, Oxman M, Rosenbaum S, et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet. 2017;390(10092):374–88. pmid:28539194.
- 7. Nsangi A, Semakula D, Oxman AD, Austvoll-Dahlgren A, Oxman M, Rosenbaum S, et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects, 1-year follow-up: a cluster-randomised trial. Trials. 2020;21(1):27. pmid:31907013
- 8.
Wiggins GP, Wiggins G, McTighe J. Understanding by Design. Alexandria, VA: Association for Supervision and Curriculum Development; 2005.
- 9.
Bruner JS. The Process of Education. Boston: Harvard University Press; 2009.
- 10. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80. pmid:7640549
- 11.
Feinstein A. The theory and evaluation of sensibility. Clinimetrlcs. New Haven: Yale University Press; 1987. p. 141–66.
- 12. Oxman AD, Oxman M, Semakula D, Chalmers I, Chelagat F, Kaseje M, et al. Prioritising and ordering Informed Health Choices (IHC) Key Concepts to be included in secondary school resources: Protocol. IHC Working Paper. 2019.
- 13. Nsangi A, Oxman AD, Oxman M, Rosenbaum SE, Semakula D, Ssenyonga R, et al. Protocol for assessing stakeholder engagement in the development and evaluation of the Informed Health Choices resources teaching secondary school students to think critically about health claims and choices. PLoS One. 2020;15(10):e0239985. Epub 2020/10/13. pmid:33045009
- 14. Mugisha M, Uwitonze AM, Chesire F, Senyonga R, Oxman M, Nsangi A, et al. Teaching critical thinking about health using digital technology in lower secondary schools in Rwanda: A qualitative context analysis. PLoS One. 2021;16(3):e0248773. Epub 2021/03/23. pmid:33750971.
- 15. Ssenyonga R, Sewankambo NK, Mugagga SK, Nakyejwe E, Chesire F, Mugisha M, et al. Learning to think critically about health using digital technology in Ugandan lower secondary schools: a contextual analysis. PLoS One. 2022;17(2):e0260367. pmid:35108268
- 16. Chesire F, Ochieng M, Mugisha M, Ssenyonga R, Oxman M, Nsangi A, et al. Contextualizing critical thinking about health using digital technology in secondary schools in Kenya: a qualitative analysis. Pilot and Feasibility Studies. 2022:8(1):227. Epub 20221006. pmid:36203201
- 17. Austvoll-Dahlgren A, Nsangi A, Semakula D. Interventions and assessment tools addressing key concepts people need to know to appraise claims about treatment effects: a systematic mapping review. Syst Rev. 2016;5(1):215. Epub 20161229. pmid:28034307
- 18. Nsangi A, Semakula D, Rosenbaum SE, Oxman AD, Oxman M, Morelli A, et al. Development of the informed health choices resources in four countries to teach primary school children to assess claims about treatment effects: a qualitative study employing a user-centred approach. Pilot Feasibility Stud. 2020;6:18. Epub 2020/02/15. pmid:32055405
- 19. Austvoll-Dahlgren A, Oxman AD, Chalmers I, Nsangi A, Glenton C, Lewin S, et al. Key concepts that people need to understand to assess claims about treatment effects. J Evid Based Med. 2015;8(3):112–25. Epub 2015/06/25. pmid:26107552.
- 20. Semakula D, Nsangi A, Oxman AD, Oxman M, Austvoll-Dahlgren A, Rosenbaum S, et al. Effects of the Informed Health Choices podcast on the ability of parents of primary school children in Uganda to assess claims about treatment effects: a randomised controlled trial. Lancet. 2017;390(10092):389–98. Epub 2017/05/26. pmid:28539196.
- 21. Dahlgren A, Furuseth-Olsen K, Rose CJ, Oxman AD. The Norwegian public’s ability to assess treatment claims: results of a cross-sectional study of critical health literacy. F1000Research. 2021;9(179).