Participants and recruitment.

Twenty-three participants from Zambia and twenty-five participants from Ethiopia were recruited by purposive sampling by co-authors familiar with the vector control stakeholders in each country (DY and BH). Participants were employees of government, universities, or research institutes, whose work related to vector control. Participants were introduced to the study in the first instance via email. One day prior to the start of the course, we gave participants an information sheet and a copy of the consent form. We verbally described why the study was being conducted, the methods used, and benefits and risks to the participants. Any questions or concerns were discussed. Participants were then allowed to consider their participation before providing written consent the following morning.

Evaluation instruments.

To assess knowledge acquisition, we developed a multiple-choice questionnaire. It was designed to assess different levels of cognitive learning based on Bloom’s taxonomy, from recalling factual knowledge to synthesising different pieces of information to solve a problem [12]. The knowledge test included 22 questions about insecticide resistance and managing insecticide-based interventions (16 questions connected to ResistanceSim and 6 to Resistance101) and was administered before and after the training (S1 Appendix).

Self-efficacy is “one’s belief in one’s ability to succeed in specific situations or accomplish a task” [13, p. 155]. Self-efficacy is typically assessed by asking participants to rate on a Likert scale how strongly they believe they can perform specific tasks. We developed a self-efficacy assessment tool containing 11 tasks focusing on key IRM areas in accordance with accepted guidelines [14] (S2 Appendix). The self-efficacy questionnaire was administered before and after the training. To evaluate if participants’ attitudes towards games changed after the training, we asked participants to rate on a seven-point scale, if games could be used as a learning tool for IRM.

We developed an interview guide to assess the training itself, as well as the impact of the training on participants’ work. Interviews were centred on five major open-ended questions: 1) Can you please tell me a bit more about your job? 2) How, if at all, did the topics covered within the training relate to your job? 2) Can you try to give an example of an occasion, if any, where you acted differently in your work, after the training? 3) Was there any aspect of the training which was particularly useful? If so, which one and why? 4) Was there an aspect of the training which was less useful than others? If so, which one and why? 5) Did you play the game(s) after the training had finished? Why/why not?

Ethics approval was obtained from the Liverpool School of Tropical Medicine, Research Ethics Committee (17–004), as well as from the Institute of Health IRB at Jimma University in Ethiopia (IHRPG/8/3/17) and the ERES Converge IRB in Zambia (2017-Jul-011). Written informed consent was obtained from all participants.

Data collection.

At the start of the training, we collected socio-demographic data and information on participants’ gameplay patterns (how often they play and which games) in the form of questionnaires. Participants completed the knowledge test and self-efficacy assessment tool at the beginning and end of the training. We conducted semi-structured interviews with participants one month after the training. Of the forty-eight participants in the training, thirty were available for the interviews. Interviews were conducted by one of the co-authors (CD or KD). Both interviewers had been trained in qualitative research and interviewing techniques, and CD had prior experience conducting similar research. Each interview was held in a private meeting room with just the interviewer and the participant. Interviews lasted approximately one hour, were audio-recorded and then transcribed verbatim. Notes were taken by the interviewer when feasible while still allowing them to concentrate on the conversation. Two participants were excluded from the analysis due to poor quality of the audio recording.

Data analysis.

We compared knowledge test and self-efficacy scores before and after participation in the course using paired t-tests. If multiple comparisons were made, we used Bonferroni-adjusted significance thresholds. Where tests and rating scales were not completed in entirety in both pre and post course, participant answers were excluded from the analysis.

Guided by an inductive thematic analysis approach, interviews were coded following the guidelines from Braun and Clarke [15], which provide comprehensive steps for researchers to describe, organise and report patterns in qualitative data. This approach, based on the emergence of meaning, is very useful for capturing the richness of the data, such as nuances of participants’ experiences and perspectives.

We first familiarised ourselves with the data by reading all transcripts and noting ideas and observations. We then began generating codes to help organise and interpret the data. The first four interviews were coded and discussed by two researchers (CD and KD) to establish a common understanding of the coding framework. All the interviews were then coded by a researcher (CD) using NVivo version 11 (QSR International Pty Ltd) as a supporting tool. Codes were reviewed iteratively as new ones emerged. Duplicate codes were eliminated, and codes were deleted or regrouped as necessary. During this interpretation process, we focused on what participants said explicitly, but also looked for implicit meanings. We reviewed the data iteratively to make sure that emergent codes were applied consistently.

We then grouped the codes into categories. Maps of categories were drafted to support this process and to refine categories. A final analytic step involved searching for and defining themes and sub-themes. The data were systematically reviewed to ensure that each theme and sub-theme were identified correctly. Considerable effort was made at each stage to ensure that the emerging analysis came from the available data and reflected the experiences of the participants. The researchers strived to stay open-minded and free of preconceived ideas. Furthermore, reflective notes and memos were written and refined at every stage of the process to support the interpretation of data and emerging patterns.

Theme: Game-based training.

Participant interviews provided insight on participants’ reactions to and perceptions of three main areas: the use of games as training tools, the training, including perceived strengths and weaknesses, and verisimilitude of ResistanceSim compared to IRM in the field. Participants’ reactions to the training and the games were overwhelmingly favourable, and participants used a range of positive affective terms in relation to both the games and the training.

Sub-theme: Games as learning tools. Participant reflections on the games and their experience included what they learned and an awareness of the specific game characteristics that supported their learning. Participants’ attitudes toward the use of the games as a learning tool were enthusiastic, expressing that they enjoyed playing the games. Besides being enjoyable, participants described the games positively, primarily in two emotional dimensions: pleasantness and excitement, using descriptive words like “friendly”, “nice”, “good”, or “interesting”, “stimulating”, “exciting”.

Their game experience with ResistanceSim and Resistance101 was distinctly different. One participant noted that Resistance101 was more like a game while ResistanceSim was more like a professional tool. Resistance101 was liked by all, and strongly seen as a “play and learn” tool, a simple game with strong clear mechanics, “choosing the insecticide and shooting all the mosquitoes”. Participants mentioned the animations of Resistance101 were enjoyable, and a good way to learn through presentation of concepts visually and vividly: “the video is easy to grasp compared to just lectures.” One participant stressed how he could visualise the biological aspect of resistance:

“To just learn about how the neurons work and you are looking at the pictures and you are seeing what is happening, the positive charges, the negative charges and all that, it really helps one to understand.”

Participants saw ResistanceSim as more complex and demanding, but interesting and motivating. The game required more concentration and problem-solving skills.

“You have to play with different parameters. There are insecticide of different classes that are acting on different ways, so that part impressed me and it’s like playing a game, like when you are playing chess you have to think in many ways.”

The engaging nature of the games and their efficacy as learning tools can be explained by several game characteristics like scaffolding, feedback and replay value. Participants reported that the sequential structure of Resistance101 and the ResistanceSim RoadMap facilitated the acquisition of knowledge. Participants praised the progression of Resistance101, including the sequential introduction of scientific concepts in each animation, as well as levelling up during gameplay (going from the easiest to the most complex). Participants mentioned feedback as a positive feature of the game. Feedback is given in the game in tutorials, during gameplay, and after failures. Participants felt that visual and vivid qualities of the games facilitated retention and allowed flexibility and freedom in learning, which participants mentioned: “learning at your own pace”, or allowing you to “know every detail” and “I repeated the game several times to capture the science as well as the practical things.”

The participants recognised the key messages from the game-based training. The game mechanic in Resistance101, where the player has to shoot mosquitoes with the right insecticide at the right time, was notable in effecting participant understanding.

“Because it always also shows you that if you do an intervention at the right time with the right dosage and the right chemical, you’re going to get a proper result and you won’t have that resistance.”

Although participants acknowledged the value of continuing to play after the training, half of them mentioned that finding the time to do so was a major issue.

“That game is not just actually playing and making fun. Yeah, in the game, the information behind the game is very useful and that one is an actual target for this course. So, having the time is one of the key factors.”

Participants described having many demanding tasks and activities, and some participants were often out of their offices conducting field work. Network and infrastructure limitations were also mentioned, including access to and availability of PCs. Mobile phones were clearly the preferred platform.

Sub-theme: Training appreciation. Participants reflected more generally on the training as a whole, rather than just the games themselves, by comparing the game-based course with other courses, highlighting its strengths and weaknesses. They assessed the training in slightly different ways than the games themselves, referring to coverage and content in terms such as “comprehensive, interesting, useful and helpful”. Participants differentiated the course from previous courses by referring to it as “unique” and “awesome”.

“For me? It was the first time I think I attended such a training which was interesting from the beginning to the end.”

Participants compared the game-based training to traditional teaching, based on “presentation and PowerPoint”. As someone stated, “he preferred to interact than to read”. Participants appreciated the training structure and flow, mini-lectures, gameplay, discussions and the group exercise. They placed high value on the practical or hands-on aspects of the training embodied through the games. Participants contrasted the training’s theoretical and practical aspects. Participants were first introduced with information on IRM concepts and processes, which was followed by their immediate revision and application through gameplay. Participants found the group work, and the coaching during the gameplay sessions, very helpful, as they were monitored to observe progress and help was provided as needed.

Participants specified that the whole training was relevant to them and increased their knowledge, and they also stressed that the training was directly related to their job: “is all about insecticide management, resistance management, it all involves our job”.

Despite the success of the training, some aspects need to be strengthened. According to participants, the training was too short, too fast, and there was not enough time for participants to play the games. Participants felt they were rushing too much between the training components, but most of all, participants wanted more time to play, especially for ResistanceSim.

“Playing ResistanceSim unfortunately for me, I really needed much time on it, because it is a game alright, but it is more a technical part than just a game”.

This is quite understandable, participants first needed enough time to become familiar with the game environment and tasks, and then to assimilate the different concepts and procedures. This is especially true for a multi-layered simulation game like ResistanceSim. Consequently, as a participant highlighted, it might be difficult to re-engage with the game if they had not mastered the game well enough during their initial exposure.

Some participants requested additional support with the game-based training, especially reading material, a game manual / documentation, and feedback on their progress at the end of the course. Participants also made several suggestions to make the training more effective, consolidating it by adding follow-up components such as reminders to play, incentives to boost their interest such as an online quiz, and new versions of the games. Participants also agreed that a short refresher training could be useful:

“Maybe, after six months so that we review and see how the course has impacted on our work. Or maybe again, as a reminder. So that we play the games and then get to know the games better.”

Finally, some participants requested adding a field component to complement the course, going over some of the steps from the game-based training, to experience in real-life those aspects covered in the game and for problem-solving in real situations, as needed.

Sub-theme: Degrees of verisimilitude in the games. Resistance101 is an introductory educational game, while ResistanceSim is a simulation game. Participants noted that current operational settings were not as ideal as those depicted in the games, especially in ResistanceSim, as not all options are available to them.

“Yeah. Because the differences are that in IRM we are trying to equate the theory of the game, the application and then also the changing the chemicals, rotating them and rotating more of the chemicals than just coming from the centre. We don’t have those options at the same time.”

More than half of the participants mentioned resources (budget, human, products and material) as the most important barrier to the success of IRM. Limited budgets have a strong impact on the choice and quantity of tools for vector control. The logistics of IRM can also be a challenge including transportation, equipment maintenance, or delivery of chemicals on time. To compound this, access to scientific equipment, “inadequate tools to use”, might be lacking.

“For example, we know that morphological identification, the PCR identification is much better as a morphological identification, but changing this will require additional resources, so we are using, we are just testing samples, subsamples of the mosquitoes that we collected from the field with the PCR.”

Some of these operational challenges were embedded in the game mechanics: a set budget restricts the strategy that players can implement e.g. a limited amount of insecticide/bed nets can be procured. In addition, the efficiency of some operations in the games, such as spraying, was decreased by lack of adequate training. Potential future versions of ResistanceSim could include other operational barriers to reflect conditions in a particular region or country.

Theme: Transformative impact.

The training increased participants’ knowledge of insecticide resistance, and procedural knowledge of IRM. The game-based training seems flexible enough to accommodate a range of needs, filling knowledge gaps, providing detailed information, or refreshing their knowledge.

Sub-theme: Knowledge acquisition. Many participants needed to gain new knowledge in insecticide resistance or IRM. One participant stipulated he had previously found it difficult to understand some of the issues when he had tried to self-learn. Some participants had moved from other specialisations and were missing expert knowledge in some areas, in particular in the area of entomology. Due to the nature of their jobs or / and because some procedures are centralised, participants were also lacking vital information.

“I would think maybe basic susceptibility tests like I said, I never really observe how it’s done but the course was able to bring that one out.” “So, it can guide me to have the appropriate tool for surveillance, to use the appropriate technique for species identification.”

Some participants at the district level stated that they appreciated getting insights on “what happened at the higher level”, as they are not involved in the choice of insecticide. The training assisted meaning-making: it helped participants to integrate knowledge or information and gain new insights that enhanced their understanding.

“Like I got a grip of it, I could understand what they really mean when an insect is, like the mosquitoes are resistant to a specific type of insecticide and what not” but he added that “this course adds a lot of things to me, because you see, I learnt these two areas in a way that can be conceptualised”.

Some more experienced participants felt that the training refreshed knowledge and skills that they had not used for a while. One participant summed this up as “Okay the course that I took with you has mostly, kind of, enhanced the knowledge that I had”.

The game-based training, and especially Resistance101, gave participants an opportunity to get an overview of insecticide resistance, and fill gaps in their understanding. Many participants highlighted the specific topics they learned such as insect biology, mechanisms of insecticide resistance, insecticide modes of action, as well as strategies to manage resistance.

“It is changed positively in that I now fully understand why it is important not to use one type of chemical for IRS and how cross-resistance works.”

Participants noted that the course information was directly applicable to their jobs. As a participant emphasised, they work with insecticide, and they need to know how to strategically use it to prevent resistance from developing, “it was very useful actually, as we know, IR is very problematic and a barrier to malaria elimination”. As a participant noted, knowledge of insecticide resistance will enhance their job performance, “know how this works, how I will just apply this and it will work”.

ResistanceSim enhanced participants’ procedural knowledge. Some participants indicated that the game provided a good overview of how to manage a vector control programme and guided them through the steps of the IRM process. This overview consolidated participants’ understanding.

“I found the RoadMap interesting because it was a map, it was like a process on vector management, so you are going from this step to that step. So even as you are doing your work, you are supposed to start from here, collection of data, so I found that very interesting.”

Quite a few participants lacked knowledge of advanced techniques. As a participant noted, he now knew the purpose of a synergist assay and which steps to follow to apply it effectively.

“Even the advanced procedures also mentioned in the course, you know the PCR, the synergist also, which is not implemented even in this country and … but we didn’t implement the synergist form of the insecticide here. But I have got the detailed information from the course.”

Sub-theme: Competencies. The training enhanced participants’ understanding of systematic decision-making at different stages of the process. Participants recognised the importance of making evidence-based decisions, and thus collecting appropriate and quality data at key points. One the most important outcomes of the game-based training course relates to a greater realisation that the quality of the intervention was key to the effectiveness of IRM. Consequently, it led to some changes in work practices.

Participants acknowledged the importance of data collection to inform IRM. The game-based training highlighted the importance of collecting baseline data and monitoring interventions. After the training, participants reiterated a key point from the course that collecting baseline data will help participants to make comparisons after an intervention and make data-driven decisions to ensure effective IRM. A participant elaborated:

“The same species can act differently in different geographical areas. So, based on your baseline data you need to choose appropriate insecticide or intervention in general for which species, for which study sites and then the likes. This is what I learnt from the programme. Yeah.”

The importance of monitoring was highlighted as follows:

“So, it’s better, according to this expert training, according to this part it is better to monitor that part of the intervention regularly. So, if there are any matters of resistance it is better to look for another option either with integrating another intervention or using techniques like mosaics, rotation, mixture of the insecticide.”

For some participants, the training changed their outlook on IRM, they now recognise the importance of making evidence-based decisions “I need to look at my data first before I make a decision.” The training led to differences in the decision-making process and resulted in changes in insecticide spraying strategies.

“And those were the ones that had the most occupying malaria incidences. So, that was informed by the training where we said okay, let’s look at the baseline data. We just don’t say look what’s the structures. The structures in this and how many you target like that, but you say no, let’s record the incidence and I think that’s what will inform us where we will spray. Even within the hotspots we had less chemicals. We had to request for some more, about 862 bottles extra and they extended our spray number of days from 17 to 25 days and then I think we covered. So, we looked at the baseline information that was existing to make decisions. I think that also came from the course.”

The game-based training helped learners to visualise what will happen in the field from the operations and decisions that are made and thus can have some practical implications on their work. ResistanceSim and the game-based training more generally were seen as tools that could support decision-making.

“Oh yes. I do think the course can make a difference, yes, encourage our decisions and can even improve the way programmes are being done. Yeah. So that we make decisions based on scientific data that on the risk points of the vector to whatever type of insecticide the characteristics of the vector it can only help in changing a lot of things.”

Not only did the training improve their decision-making, but it appeared to motivate participants to raise the quality of the insecticide-based intervention.

“This programme can tell them to follow the right protocol, to follow and use the right tool for monitoring, to follow the standard operating procedure, to have good quality data as well as to apply the appropriate intervention.”

Participants reflected that resistance could be caused through their negligence, or misinterpreting instructions or information. For example, IRM is not always fully implemented or IRS conducted adequately “missing hotspots with high cases of malaria”.

“Before we were just saying okay, if one could use a mosquito net then they could as well substitute that net with the spray, so it’s either this or that. Whereas this time we understood that we needed to combine. If we just say okay, I have a mosquito net, I don’t need the spraying, then we are kind of promoting resistance because we are just using one insecticide, so that’s how it’s actually connected very well.”

Thus, the game-based training also helped participants to recognise some flaws in their actions or reasoning thus helping to correct them, “to have the right thing done, so that can help us to see where we are going wrong”. Quite pragmatically, as a participant added, if they do the wrong thing, they will not get any results, so they may as well do the right thing.

Subtheme: Sharing knowledge. Sharing knowledge was one of the primary actions that participants reported taking. Indeed, participants recognised the importance of the knowledge they gained and its impact on IRM. One participant mentioned, “When people have knowledge, they act different”.

Participants better appreciated the importance of involving and exchanging information with major IRM stakeholders, including policymakers dealing with malaria elimination, “how we can seek assistance or advice from other ministries”. Participants felt that a greater cooperation was needed with everyone concerned: trust would be secured by informing stakeholders, establishing consensus about decisions, and fulfilling responsibilities as planned.

“All the district heads of departments were invited to a meeting and then also other key stakeholders and we ensured that they came. But previously, maybe we would have just got whoever came and we went ahead. By this time, we wanted to make sure that we explained this to all at once and then when we came out of the meeting, whatever we agree is what is going to be done. Yes. And we stuck to the dates. When we say we are going to spray this for maybe three days, we would be there three days, this facility four days, it would be four days as agreed. So, I’m sure even the stakeholders were pleased.”

Several participants told us that they had or would share knowledge with their closed social networks, primarily colleagues in similar positions, but also subordinates and supervisors. Besides knowledge, participants also mentioned wanting to share the games with their closed and specialist networks like the Ethiopian Malaria Research Network, or at malaria workshops. Note we specified at the time that participants could only share the games once they were officially released.

“I think it would really help so I was thinking of the officers in the district. I think it would be helpful to introduce the game to them and make sure they know how to play them. People would be able to build on their own knowledge and capacity in that way”

Training seems quite a regular activity that at least half the participants are carrying out. Because of a high turnover of personnel, the most common training related to insecticide resistance and spray operations. After the course, training field staff adequately and more thoroughly was seen as quite important to improve the quality of the intervention.

“They will tell me just look at the odd spray operators and just go into the field for a day or two then … but this time I acted differently because I refused, you are doing shoddy work. They will just tell you do two days and then go and … I refused this time, so I did adequate training and the job is … I think it’s going one better than before.”

Participants identified that existing training of spray operators did not include much about insecticide resistance, so participants started sharing knowledge of IR with them and noted that it should become integral to part of their training. Participants felt that they too should understand the importance of doing the right thing at the right time, and the consequences of not doing it. “So it gave them some insight, some ideas, for them to do the correct thing if we are to prevent malaria.” By doing so, the operators can also feel more valued, as they see the importance of their role. It would also allow spray operators to communicate better with communities, gaining their trust and cooperation.

Community issues in IRM are well-known including bed net use, and acceptance of IRS spraying. There was a strong feeling from participants that a greater social mobilisation was needed with communities, otherwise “there would be no solidarity.” Thus, informing and sharing knowledge with the community was seen as quite important, as it could affect engagement and cooperation, “and the fight now becomes their fight”. There was a realisation that they in turn would benefit from knowing the scientific evidence about insecticide resistance, especially community health workers, acting as intermediaries between communities and malaria programme managers.

“If you know nothing about vector management, you won’t even know the importance it has in malaria elimination. So at least it brings about awareness and you also feel like you are part of the team.”

The training also provided participants with the information to effectively communicate with communities to increase acceptance of IRS.

“But you just gave us mosquito nets. Why are you spraying again?” So, we acted differently by saying, what we are doing is we are trying to manage resistance. Previously, I think I would have had no proper reason to give, but this time I was able. It was just after the training; the spraying came, and we were able to explain like that.”

Sub-theme: Facilitating empowerment. The game-based training seems to have empowered vector control professionals. Besides gaining new knowledge and enhancing competencies, the training also increased participants’ motivation, confidence and dedication to overcome IRM challenges. The training increased participants’ motivation for their work, as they learn how to overcome the many challenges that they are facing. One participant mentioned, "I got more interested in doing what I was doing,” and another that he would continue learning:

“Therefore, the point that you have raised encouraged us to know in detail insecticide resistance management is very important for our country”.

The training also reinforced their motivation to engage in problem-solving and learning from their experiences:

“What we learn is just to prime us, to prepare us to what we are going to find in the field and it’s up to you to develop that capacity to overcome what you want. Because not everything we are going to be taught anyway during the training. But it is your drive and your inner motivation which is going to make you like everything.”

The training raised participants’ confidence, as they were given the tools to confidently share their expertise and participate in meetings more authoritatively, thus raising self-esteem.

“So, my experience with the training was able to comment in that particular meeting. I think I was speaking more with some authority. I understood what I was talking about and what the other people were talking about.”

Based on his new understanding about entomology, a participant felt better integrated in his professional field, as he was now able to participate in discussions. Similarly, others mentioned being able to share knowledge, inform and advise others as “when you’re speaking from an informed point of view”. Thus, participants felt more empowered in their work; it is not just the policy makers that can tackle that issue, but everyone can do something at their level. By making insecticide resistance and its management clear, the training stimulated action to sustain malaria control.

“I appreciate this innovative approach in making resistance very clear to people and encourage people to take action and contribute for resistance management strategy. One thing that might be a priority in the future.”

Empowerment in this context also means the participation of malaria professionals in the IRM decision-making process, but some participants highlighted that this was not the case. National strategies were not always seen as effective, as the situation in the field varies greatly from one end of the country to the other.

“If we can establish entomological and insecticide resistance studies and monitoring at district level, it would help us, because we would be sharing the scientific evidence, finding out what is the best insecticide to use, and what to avoid, as opposed to what is happening now”

Decisions regarding the choice of insecticide could be influenced by the private sector, which might not be in line with the national strategic plan, or by economic criteria “cost effectiveness, might be more important than scientific evidence”. The lack of involvement of malaria professionals in the decision-making process could render IRM less effective.

Sub-theme: Supporting scientific capacity. As a participant noted, the success of the game-based training, and its impact on IRM and malaria control in general, can only occur if knowledge is propagated. Not only should knowledge be shared in an operational context, but the training should be widely distributed.

This participant suggested that the training be disseminated at national and regional levels, strengthening the capacity of all malaria professionals including programme district managers, environmental health officers, and related government staff, especially policy makers.

“I have a team of environmental health officers who help supervise the spray operators so they need to understand.”

Several participants mentioned that the training should be given for newcomers, everyone coming to work at the malaria programme. It should also be given to students in colleges and universities, especially new graduates involved in research. It was also suggested that all partners and stakeholders involved in the malaria programmes would benefit from the training, such as those in the private sector and medical staff at health centres treating malaria cases. Indeed, medical staff are in a unique position to raise community awareness. One participant also suggested involving traditional healers.

“Because those are the people who will help us in common behavioural change management communication, for them to accept the programme”.

Participants also advocated that the game-based training, or some of its components, suitably modified, should be used as part of community sensitisation and mobilisation.

“But if they were able to work through that and understand the whole process, that just makes life easier for all of us and that even motivates them”.