Figures
Abstract
Background
Continuing professional development (CPD) activities are one way that new knowledge can be translated into changes in practice. However, few tools are available for evaluating the extent to which these activities change health professionals’ behavior. We developed a questionnaire called CPD-Reaction for assessing the impact of CPD activities on health professionals’ clinical behavioral intentions. We evaluated its responsiveness to change in behavioral intention and verified its acceptability among stakeholders.
Methods and findings
We enrolled 376 health professionals who completed CPD-Reaction before and immediately after attending a CPD activity. We contacted them three months later and asked them to self-report on any behavior change. We compared the mean rankings on each CPD-Reaction construct before and immediately after CPD activities. To estimate its predictive validity, we compared the median behavioral intention score (post-activity) of health professionals reporting a behavior change three months later with the median behavioral intention score of physicians who reported no change. We explored stakeholders’ views on CPD-Reaction in semi-structured interviews. Participants were mostly family physicians (62.2%), with an average of 19 years of clinical practice. Post-activity, we observed an increase in intention-related scores for all constructs (P < 0.001) with the most appreciable for the construct beliefs about capabilities. A total of 313 participants agreed to be contacted at follow up, and of these only 69 (22%) reported back. Of these, 43 (62%) self-reported a behavior change. We observed no statistically significant difference in intention between health professionals who later reported a behavior change and those who reported no change (P = 0.30). Overall, CPD stakeholders found the CPD-Reaction questionnaire of interest and suggested potential solutions to perceived barriers to its implementation.
Citation: Légaré F, Freitas A, Turcotte S, Borduas F, Jacques A, Luconi F, et al. (2017) Responsiveness of a simple tool for assessing change in behavioral intention after continuing professional development activities. PLoS ONE 12(5): e0176678. https://doi.org/10.1371/journal.pone.0176678
Editor: Tanya Horsley, Royal College of Physicians and Surgeons of Canada, CANADA
Received: July 13, 2016; Accepted: April 14, 2017; Published: May 1, 2017
Copyright: © 2017 Légaré et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This study was funded by a Partnership for Health System Improvement grant from the Canadian Institutes of Health Research (CIHR; 2010-2013; grant # 200911PHE-216868-PHE-CFBA-19158) and by the Ministère de la Santé et des Services Sociaux du Québec (MSSS), QC, Canada. F Légaré holds a Tier 1 Canada Research Chair. The following continuing medical education organizations provided in kind support for this study: Vice-décanat à la pédagogie et au développement professionnel continu - Secteur DPC, Faculty of Medicine, Université Laval, Quebec City, QC, Canada; Practice Enhancement Division, Collège des médecins du Québec, Montreal, QC, Canada; Fédération des médecins spécialistes du Québec, Montreal, QC, Canada; Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, QC, Canada; Université de Montréal, Montreal, QC, Canada; and Centre for Continuing Health Professional Education, McGill University, Montreal, QC, Canada.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Participation in continuing professional development (CPD) training programs for health professionals forms a central component of quality improvement efforts [1–4]. A 2016 scoping review of national programs to validate physician competence and fitness for practice showed an emerging concern that, owing to the rapid increase in evidence relevant to practice, continuing education and assessment should be a requirement for today’s physicians [5]. In a survey of 10 European countries, all of them viewed CPD as an important mechanism for demonstrating that doctors continue to meet key standards [6]. CPD training programs are now expected to transmit new knowledge in ways that enable health professionals not only to gain new knowledge but to change their behavior in the clinic so that patient care is optimal and patient outcomes improve [7,8].
However, a lack of theory hampers these efforts. Studies included in the 2016 scoping review of national programs frequently cited goals and aims, for example, but none described any theoretical framework used to guide setting up the program or reported on evidence used to inform program design [5]. Indeed, there are few relevant conceptual models for understanding behavior change in health professionals and thus little is known about how to design effective interventions for CPD programs [9–11]. Some conceptual CPD frameworks have been developed in reaction to evidence that a) CPD activities should be based on assessed need, b) passive approaches do not generally change physician behavior, and c) interactive activities (with practice and feedback) are more effective. Moore et al., for example, developed their “expanded outcomes framework” for planning and assessing continuous learning [12]. Taking six assessment criteria from an earlier model (participation, satisfaction, learning, performance, patient health and community health) they expanded its learning/performance components into Miller’s 4-level formula: “knows”, “knows how”, “shows how”, and “does” [13]. Few conceptual models for CPD have integrated the knowledge base of socio-cognitive behavior change theories, however [14–16], and there are few valid and reliable methods for identifying the factors influencing health professionals’ behavior in this context [17]. Together, these gaps in knowledge may partially explain why so few CPD activities have proven effective for promoting behavior change among health professionals [18,19].
In 2009, CPD decision makers and knowledge translation researchers in Canada met to begin the process of developing a short, theory-based questionnaire that could be used as a tool for routinely assessing the impact of CPD activities on clinical practice [19,20]. The CPD-Reaction questionnaire shows adequate validity and reliability, with Cronbach’s coefficients for the constructs varying from 0.77 to 0.85 [19]. It is a 12-item instrument based on an integrated model combining a number of social cognitive theories for explaining health professionals’ clinical behavior through the proxy of intention (e.g. the Theory of Planned Behavior and Triandis’ theory) [11]. Briefly, this integrated model proposes that three categories of variables predict the behavior of health professionals: 1) their intention to adopt a particular behavior or not; 2) their beliefs about their capabilities; and 3) their past behavior and habits. The model also suggests that the first category, behavioral intention, is influenced by several factors as described in the original account of the model[11].
This study describes the validity of the CPD-Reaction questionnaire for assessing the impact of CPD activities on health professionals’ clinical behavioral intentions [19,20]. Specifically, we sought to evaluate the responsiveness of the CPD-Reaction questionnaire (i.e., its ability to measure changes in behavioral intention after a CPD activity) and its predictive potential for subsequent behavior change. We also explored its acceptability among CPD providers to develop an implementation plan for its use in CPD activities.
Methods
Ethics statement
The Research Ethics Committee of the Centre Hospitalier Universitaire de Québec (CHU de Québec) approved this project on 30 June, 2010 (project # S10-06-033). Participants gave written informed consent to participate in the study. The Ethics Committee approved the consent procedure.
Study design
We conducted a prospective mixed-methods study. First, we performed a before-and-after study with health professionals attending accredited CPD activities to evaluate the CPD-Reaction questionnaire’s responsiveness to change in behavioral intention and its predictive validity (subsequent self-reported behavior change). We also conducted a qualitative study with CPD decision-makers and providers to verify its acceptability. A detailed protocol of the entire research project is published elsewhere [20].
Quantitative study
Participants and recruitment strategy.
We recruited participants who attended eligible CPD activities offered by the CPD providers and collaborators in this project [19]. Eligibility criteria for CPD activities included the following: (a) be accredited by one of Quebec’s CPD providers; (b) be group-based, with live activities; (c) be focused on one main behavior change as stated in the learning objectives established for that activity, (d) take place in any setting; (e) use any materials and teaching methods or combination thereof; and (f) are one-time interventions. Activities offered as part of a larger program, such as a medical conference held over several days, were also eligible. Eligibility criteria for participating health professionals were: (a) attending an eligible live CPD activity; (b) is still in clinical practice six-months after the indexed CPD activity; (c) agrees to be contacted for a phone interview three months after the indexed CPD activity; (d) speaks French or English; and (e) has not participated in this study before.
Data collection.
Research assistants (RAs) attended those CPD activities that were eligible from among all the activities available [19]. As soon as attendees arrived at the CPD activity, RAs invited them to participate in the study. Those who agreed to participate were asked to complete a consent form and the CPD-Reaction questionnaire, a self-administered paper-based questionnaire adapted to match each eligible CPD activity. The generic CPD-Reaction questionnaire was adapted by replacing the word “behavior” in each item with an observable or measurable learning objective from the CPD activity in question (e.g., “handling an automated external defibrillator”, see S1 Table). Participants completed the CPD-Reaction questionnaire just before and immediately after the CPD activity. Data was collected anonymously. Participants were then contacted three months later by email to collect information about their self-reported clinical behavior. At that moment, we ascertained if the participants had adopted the clinical behavior targeted in the CPD activity. If so, we asked them to estimate the percentage of relevant clinical cases in which they adopted the clinical behavior. We sent three reminders to all those who agreed to participate in this phase of the study. Participants returned their responses by email to the project manager.
Data analysis.
Descriptive statistics such as mean, standard deviation, median and frequencies were calculated to summarize participant characteristics and CPD-Reaction questionnaire item responses. Table 1 presents a summary of the score calculations for each item and construct of the CPD-Reaction questionnaire. Briefly, each item response format was pre-coded with Likert-type scale values (Table 1). The item score for each participant ranges from 1 to 7. A score for each construct was obtained by calculating the mean score for the construct (e.g., if the construct includes 2 items, the item scores were summed and divided by 2, yielding a score between 1 and 7).
The responsiveness of the CPD-Reaction questionnaire to change in behavioral intention was examined by comparing the mean ranks of each construct obtained before and after CPD activities using the Wilcoxon signed-rank test. Data collected three months after the CPD activities were used to estimate how effective the tool is for predicting behavior. Based on the results of a systematic review of factors predicting behavior in health professionals [11], self-reported behavior can be used as a proxy of behavior change. Predictive validity was estimated by comparing the intention (median intention construct score) of health professionals who reported a behavior change three months after the CPD activities with the intention score of those physicians who reported no change, using a Mann-Whitney test. We considered a p value < .05 as statistically significant. Analysis was performed using SAS version 9.3 software (SAS Institute Inc., Cary, North Carolina).
Qualitative study
Participants and recruitment strategy.
To assess the barriers and facilitators for implementing the CPD-Reaction questionnaire in CPD settings, we conducted semi-structured phone interviews with CPD providers involved in the development of these activities in the province of Quebec, Canada. Members of the research team identified key opinion leaders (or key individuals to contact). An invitation was sent by email to all potential participants.
Data collection.
The interview guide was pre-tested with a member of the research team (ML) who has many years of experience as a CPD trainer. Those who had agreed to participate received the semi-structured interview guide by email, a sample of the generic CPD-Reaction questionnaire, and an example adapted to the context of a specific CPD activity. Participants were instructed to review these documents before the interview. Phone interviews were conducted individually in English or French. Before the beginning of each interview, a brief summary of the research project was presented to each participant to contextualize the CPD-Reaction questionnaire and explain the purpose of the interview. All interviews were audio recorded and transcribed in full. We conducted the interviews until data saturation was reached [21].
Data analysis.
We used NVIVO 9 software (QRS International, Melbourne, Australia) to analyse the interview data. Two independent and experienced RAs trained in qualitative analysis identified the themes and sub-themes emerging from the data using a deductive approach. After independent analysis of six interviews, a common coding structure was established through discussion and then applied to the remaining ones. We performed thematic analysis of the content by reducing data, displaying data, and then drawing conclusions [22]. Specifically, we coded the transcripts using three broad categories concerning the instrument: 1) characteristics; 2) usability; and 3) acceptability. The strength of a particular viewpoint was determined by tabulating the number of individuals who expressed it during the interviews. Participants also completed a questionnaire providing sociodemographic data, the characteristics of their CPD organizations, and their function within these organizations.
Results
Quantitative study
Participants and CPD activities characteristics of the before-and-after study.
Out of 110 eligible activities evaluated, representing a total of 404 learning objectives from the previous study [19], we found 37 activities with at least one learning objective that focused on behavior change for any topic or content. A total of 1819 attendees registered for these 37 activities, of whom 611 were enrolled to participate in this study by a RA just before the activity begun. Most activities were lectures lasting an average of 45 minutes (n = 24), offered within the context of a symposium offering several activities in one setting. Other participants were attending interactive workshops lasting at least three hours (n = 13). Half of the eligible CPD activities were conducted in groups of fewer than 50 attendees (n = 19; range: 11–50) while the other half were conducted in groups of more than 50 attendees (n = 18; range: 61–130). Overall, 376 participants (62%) completed the questionnaire before and after the CPD activity (Fig 1). Half of the participants who completed the CPD-Reaction questionnaire were female. The majority of respondents were family physicians (62%), while 8% were specialists, 5% were residents, and 8% were health professionals other than physicians (Table 2). The characteristics of the CPD activities are described elsewhere [19].
Responsiveness to change and predictive validity of the CPD-Reaction questionnaire.
Table 3 summarizes the descriptive statistics of the items and constructs of the CPD-Reaction questionnaire. Mean and median values were very high before the CPD activities for each construct. The score distributions were non-normal because skewed to the right. Overall, we observed an increase in score for all constructs post-activity (P < 0.001) with the most appreciable increase for the construct beliefs about capabilities (the median increase of 5 to 6).
Of the 376 participants in the before-and-after study, 313 participants agreed to be contacted three months later to self-report their clinical behavior. We contacted all 313 participants but only 69 (22% participation rate at follow-up) reported back, of whom 43 (62%) self-reported a clinical behavior change as targeted by their CPD activity. Table 4 (predictive validity) shows the results of comparing the post-activity intention scores of the 43 physicians who reported a change three months later with the scores of the 26 physicians who did not. We did not observe a statistically significant difference between these two groups (P = 0.30). However, we observed that in both the pre and the post assessment period, the group who self-reported that they had changed their behavior had higher scores than the group that did not. Also, the group who reported a behavior change three months later presented a potential ceiling effect concerning their intention to adopt the behavior proposed by the CPD activity.
Qualitative study
Participants’ characteristics.
Of the 31 CPD decision-makers and providers who were emailed invitations, 16 replied and accepted the invitation. Data saturation was reached with eight interviews. Then we conducted three more interviews (n = 11). At this point, as we did not observe any new ideas emerging from the interviews, we ceased contacting participants. The majority of participants were physicians (n = 10), male (n = 10), aged 51 years and older, and had been involved in CPD activities, on average, for 23 years. Most participants were representatives of CPD organizations (n = 7) from either a medical regulatory body (n = 2), or academic institutions (n = 5) in the province of Quebec, Canada. Three participants were keynote speakers invited by CPD organizations to develop activities concerning their medical speciality. One participant worked directly with CPD evaluations in a medical school.
Acceptability of the CPD-Reaction questionnaire among CPD decision-makers and providers.
Most respondents (n = 7) found the CPD-Reaction questionnaire relevant for the evaluation needs of their organizations. It was considered to be a helpful tool for understanding how a specific clinical behavior is adopted as a result of a CPD activity. Four respondents criticized the repetitive nature, or redundancy, of the CPD-Reaction questionnaire items.
While most respondents (n = 8) found the CPD-Reaction questionnaire to be adaptable to the CPD activities offered by their organizations, the usability of the instrument in different CPD contexts polarized respondents’ opinions. Half of respondents (n = 5) were convinced that the instrument could be used in all CPD contexts. Some participants (n = 3) even suggested other uses than the one initially proposed, (i.e., the assessment of the impact of a CPD activity on clinical practice). Some mentioned that their CPD organizations could use the instrument to validate the learning objectives of a new CPD activity and when needed, modify the objectives in order to indicate the behavior to adopt after the activity. However, the other half (n = 6) found that the usefulness of the CPD-Reaction questionnaire would be restricted only to CPD activities focusing on behavior change.
Although most people involved with the planning and organization of CPD activities had a positive attitude towards the use of the CPD-Reaction questionnaire, respondents were unanimous in affirming that participants attending CPD activities would not be favorable to regular use of the instrument. The main reason for this would be physicians’ habitual resistance to completing evaluation forms and the time consumed to complete it (Table 5).
As for strategies that could be used for overcoming identified barriers, respondents said it was important to convince participants to complete the instrument. Some of them (n = 4) suggested that the completion of the CPD-Reaction questionnaire should be mandatory to receive credit for attending the CPD activity. Some (n = 4) suggested that the research team advertise the CPD-Reaction questionnaire to encourage its widespread use among representatives of CPD organizations.
Discussion
This study investigated the responsiveness of the CPD-Reaction questionnaire to behavioral intention change after completing a CPD activity and its predictive validity for actual behavior change in clinical practice. It also verified its acceptability among CPD stakeholders to help the research team to plan for an implementation strategy. Overall, our results indicate that the CPD-Reaction questionnaire is sensitive to changes in behavioral intention and its determinants before and after CPD activities. We observed no statistically significant results pertaining to its predictive validity for actual behavior change, but observed that the median intention scores tended to be higher among health professionals who reported they had adopted the behavior proposed by the CPD activity than among those who had not adopted it. In addition, diverse CPD stakeholders said that CPD-Reaction was of interest to them but that a future implementation strategy would need to address identified barriers, mostly time to complete it. These results lead us to make four main observations.
First, our analysis indicated that the CPD-Reaction questionnaire is responsive enough to detect a change in the behavioral intention of health professionals attending CPD activities. In general, results confirmed that health professionals tend to stay within their “comfort zones” when selecting their CPD activities, aiming most of the time to confirm or verify that their practice is the same (or similar enough) to what their peers are doing [23,24]. However, this does not necessarily mean that CPD activity persuades health professionals to actually change how they practice medicine. Indeed, most CPD activities do not target clinical behavior change and/or patient outcomes [19].
Second, the integrated conceptual model upon which the CPD-Reaction is based proposes that behavioral intention is a predictor of behavior. Interestingly, both pre- and post-activity median intention scores tended to be higher among health professionals who reported that they had adopted the behavior proposed by the CPD activity than among those who had not adopted it, which suggests a confirmation of the integrated conceptual model. However, we did not find statistically significant results pertaining to predictive validity of the CPD-Reaction questionnaire between the groups. This lack of statistical significance could be explained by a considerable ceiling effect in the behavioral intention construct scores and a very low response rate at follow-up for which we obtained an effect size (the rank-biserial correlation) of 0.18, a negligible effect size for adequate power [25]. Future research should take into account reproducibility of the methods but also consider proper rewards and compensations for participants in the longitudinal portion of the study to maximise sample size and increase statistical power.
Third, we found that the CPD-Reaction questionnaire was accepted well by CPD stakeholders interviewed, who believed that the CPD-Reaction questionnaire was relevant for the evaluation needs of their organizations. This relevance may be due to trends in the CPD field, mainly new accreditation regulations (such as the Mainpro+ quality criteria requirements of the College of Family Physicians of Canada and forthcoming National Accreditation standards) that emphasize evaluation and outcome assessment and knowledge transfer tools. CPD stakeholders did, however identify various barriers to its implementation in CPD settings. These barriers were mostly participants’ limited interest in completing evaluation forms after a CPD activity, which could have been aggravated by the apparent redundancy of some of the CPD-Reaction items: few CPD participants may be familiar with social cognitive theories and the measurement needs of this type of instrument. These barriers are not surprising as they are observed in other types of innovative implementation in clinical practices [26,27]. Future research is necessary to develop efficient strategies for overcoming these barriers and for increasing participants’ motivation to complete the questionnaire.
Fourth, the CPD-Reaction questionnaire has provoked a new interest in assessing CPD activities. Such assessments may result in a new form of CPD design among CPD stakeholders. Producing effective CPD activities will require identifying relevant determinants of behavior change and formulating learning objectives that focus on the higher cognitive skills [28] in order to promote a change in professional practice and in patient outcomes. This could lead to the development of more efficient CPD activities that focus more directly on results in changes in practice and improvements in patient care. Moreover, it would also allow for the target audience to display different levels of readiness to change their practice [29]. A user manual containing all necessary information for adapting the instrument to different CPD activities, and how to interpret the results, is available from the authors.
This study has some limitations. First, because we did not have a control group we could not be sure that the behavior change was due to the CPD activity, or due to a measurement effect (i.e., the act of completing the questionnaire) [30]. Second, the low three-month follow-up of 22% of the initial group of participating doctors is a significant limitation of our study. Third, the small sample size used for the qualitative study may have been insufficient; however, according to the literature, data saturation justifies confidence in the results [21]. Finally, we could not standardize the CPD-Reaction questionnaire scores because the scores were not normally distributed.
Conclusion
This study represents the last phase of a research program to support the validity of the CPD-Reaction questionnaire for assessing the impact of CPD activities on health professionals’ clinical behavioral intentions. The CPD-Reaction questionnaire is a brief and valid tool that it is able to detect a change in the behavioral intention of health professionals attending CPD activities. The instrument seems to be appropriate for assessing the impact of CPD activities on behavioral intention. CPD providers found it to be relevant for evaluating the activities that they offer to health professionals, although future implementation will require addressing the barriers they identified.
Supporting information
S1 Table. The generic CPD-Reaction questionnaire.
https://doi.org/10.1371/journal.pone.0176678.s001
(DOCX)
Acknowledgments
The authors wish to thank Louisa Blair for editing this article and Brian J. Hess for some constructive feedback on an earlier version.
Author Contributions
- Conceptualization: F. Légaré FB AJ F. Luconi GG AB JS ML.
- Data curation: AF ST F. Légaré.
- Formal analysis: ST AF GG.
- Funding acquisition: F. Légaré FB AJ F. Luconi GG AB JS ML.
- Investigation: F. Légaré AF FB ML.
- Methodology: F. Légaré FB AJ F. Luconi GG AB JS ML ST.
- Project administration: AF F. Légaré FB.
- Resources: F. Légaré FB GG ML ST.
- Supervision: F. Légaré AF FB ML.
- Validation: F. Légaré AF ST FB AJ F. Luconi GG AB JS ML.
- Visualization: F. Légaré AF ST FB AJ F. Luconi GG AB JS ML.
- Writing – original draft: F. Légaré AF ST FB.
- Writing – review & editing: F. Légaré AF ST FB AJ F. Luconi GG AB JS ML.
References
- 1. Campbell EG, Rosenthal M (2009) Reform of continuing medical education: investments in physician human capital. JAMA 302: 1807. pmid:19861675
- 2. Davis NL, Davis DA, Johnson NM, Grichnik KL, Headrick LA, Pingleton SK, et al. (2013) Aligning academic continuing medical education with quality improvement: a model for the 21st century. Acad Med 88: 1437–1441. pmid:23969360
- 3.
Joint Centre for Education in Medicine. Good CPD Guide (1999). Sutton, Surrey: Reed Healthcare 23.
- 4.
Institute of Medicine (US). Committee on Planning a Continuing Health Professional Education in the Health Professions (2010) Redesigning Continuing Education in the Health Professions. Washington (DC): National Academies Press. 296 p.
- 5. Horsley T, Lockyer J, Cogo E, Zeiter J, Bursey F, Campbell C. (2016) National programmes for validating physician competence and fitness for practice: a scoping review. BMJ open 6: e010368. pmid:27084276
- 6. Solé M, Panteli D, Risso-Gill I, Döring N, Busse R, McKee M, et al. (2014) How do medical doctors in the European Union demonstrate that they continue to meet criteria for registration and licencing? Clinical Medicine 14: 633–639. pmid:25468850
- 7. Karg A, Boendermaker PM, Brand PL, Cohen-Schotanus J (2013) Integrating continuing medical education and faculty development into a single course: effects on participants' behaviour. Med Teach 35: e1594–1597. pmid:23796056
- 8. Presseau J, Hawthorne G, Sniehotta FF, Steen N, Francis JJ, Johnston M, et al. (2014) Improving Diabetes care through Examining, Advising, and prescribing (IDEA): protocol for a theory-based cluster randomised controlled trial of a multiple behaviour change intervention aimed at primary healthcare professionals. Implement Sci 9: 61. pmid:24886606
- 9. Marinoupoulos SS, Dorman T, Ratanawongsa N, Wilson LM, Ashar BH, Magaziner JL, et al. (2007) Effectivenness of continuing medical education. Evid Rep Tech Assess 149: 1–149.
- 10. Grimshaw J, Eccles MP, Walker AE, Thomas RE (2002) Changing physicians' behavior: what works and thoughts on getting more things to work. J Contin Educ Health Prof 22: 237–243. pmid:12613059
- 11. Godin G, Belanger-Gravel A, Eccles M, Grimshaw J (2008) Healthcare professionals' intentions and behaviours: a systematic review of studies based on social cognitive theories. Implement Sci 3: 36. pmid:18631386
- 12. Moore DE, Green JS, Gallis HA (2009) Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof 29: 1–15. pmid:19288562
- 13. Miller GE (1990) The assessment of clinical skills/competence/performance. Acad Med 65: S63–67. pmid:2400509
- 14. Davis DA, Thomson MA, Oxman AD, Haynes RB (1995) Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA 274: 700–705. pmid:7650822
- 15. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. (2001) Changing provider behavior: an overview of systematic reviews of interventions. Med Care: II2–II45. pmid:11583120
- 16. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. (2006) Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA 296: 1094–1102. pmid:16954489
- 17. Ratanawongsa N, Thomas PA, Marinopoulos SS, Dorman T, Wilson LM, Ashar BH, et al. (2008) The reported validity and reliability of methods for evaluating continuing medical education: a systematic review. Acad Med 83: 274–283. pmid:18316877
- 18. Légaré F, Freitas A, Thompson-Leduc P, Borduas F, Luconi F, Boucher A, et al. (2015) The majority of accredited continuing professional development activities do not target clinical behavior change. Acad Med 90: 197–202. pmid:25354076
- 19. Légaré F, Borduas F, Freitas A, Jacques A, Godin G, Luconi F, et al. (2014) Development of a simple 12-item theory-based instrument to assess the impact of continuing professional development on clinical behavioral intentions. Plos One 9: e91013. pmid:24643173
- 20. Légaré F, Borduas F, Jacques A, Laprise R, Voyer G, Boucher A, et al. (2011) Developing a theory-based instrument to assess the impact of continuing professional development activities on clinical practice: a study protocol. Implement Sci 6: 17. pmid:21385369
- 21. Guest G, Bunce A, Johnson L (2006) How many interviews are enough? An experiment with data saturation and variaility. Field Methods 18: 59–82.
- 22.
Miles MB, Huberman AM (1994) Qualitative Data Analysis: An Expanded Sourcebook. Newbury Park, CA: Sage Publications.
- 23.
Schostak J, Davis M, Hanson J, Schostak J, Brown T, Driscoll P, et al. (2010) The Effectiveness of Continuing Professional Development—Final report. UK: College of Emergency Medicine, Federation of Royal Colleges of Physicians and Manchester Metropolitan University.
- 24. Forsetlund L, Bjorndal A, Rashidian A, Jamtvedt G, O'Brien MA, Wolf F, et al. (2012) Continuing education meetings and workshops:effects on professional practice and health care outcomes. Cochrane Database of Syst Rev Apr 15;(2):CD003030
- 25. Kerby DS (2014) The simple difference formula: an approach to teaching nonparametric correlation. Innovative Teaching 3 (1).
- 26. Dijkstra RF, Braspenning JC, Uiters E, van Ballegooie E, Grol RT(2000) Perceived barriers to the implementation of diabetes guidelines in hospitals in The Netherlands. Neth J Med 56: 80–85. pmid:10759018
- 27. Grol R, M W (2004) What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust 180: S57. pmid:15012583
- 28. Krathwohl D (2002) A revision of Bloom's taxonomy: an overview. Theory into practice 41: 212–260.
- 29. Slotnick H (1999) How doctors learn: Physicians’ self-directed learning episodes. Acad Med 74: 1106–1117. pmid:10536633
- 30. Godin G, Sheeran P, Conner M, Germain M (2008) Asking questions changes behavior: mere measurement effects on frequency of blood donation. Health Psychol 27: 179–184. pmid:18377136