Decision-makers in organizations providing continuing professional development (CPD) have identified the need for routine assessment of its impact on practice. We sought to develop a theory-based instrument for evaluating the impact of CPD activities on health professionals' clinical behavioral intentions.
Methods and Findings
Our multipronged study had four phases. 1) We systematically reviewed the literature for instruments that used socio-cognitive theories to assess healthcare professionals' clinically-oriented behavioral intentions and/or behaviors; we extracted items relating to the theoretical constructs of an integrated model of healthcare professionals' behaviors and removed duplicates. 2) A committee of researchers and CPD decision-makers selected a pool of items relevant to CPD. 3) An international group of experts (n = 70) reached consensus on the most relevant items using electronic Delphi surveys. 4) We created a preliminary instrument with the items found most relevant and assessed its factorial validity, internal consistency and reliability (weighted kappa) over a two-week period among 138 physicians attending a CPD activity. Out of 72 potentially relevant instruments, 47 were analyzed. Of the 1218 items extracted from these, 16% were discarded as improperly phrased and 70% discarded as duplicates. Mapping the remaining items onto the constructs of the integrated model of healthcare professionals' behaviors yielded a minimum of 18 and a maximum of 275 items per construct. The partnership committee retained 61 items covering all seven constructs. Two iterations of the Delphi process produced consensus on a provisional 40-item questionnaire. Exploratory factorial analysis following test-retest resulted in a 12-item questionnaire. Cronbach's coefficients for the constructs varied from 0.77 to 0.85.
A 12-item theory-based instrument for assessing the impact of CPD activities on health professionals' clinical behavioral intentions showed adequate validity and reliability. Further studies could assess its responsiveness to behavior change following CPD activities and its capacity to predict health professionals' clinical performance.
Citation: Légaré F, Borduas F, Freitas A, Jacques A, Godin G, Luconi F, et al. (2014) Development of a Simple 12-Item Theory-Based Instrument to Assess the Impact of Continuing Professional Development on Clinical Behavioral Intentions. PLoS ONE 9(3): e91013. https://doi.org/10.1371/journal.pone.0091013
Editor: Manuel João Costa, University of Minho, Portugal
Received: December 13, 2013; Accepted: February 5, 2014; Published: March 18, 2014
Copyright: © 2014 Légaré et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study was funded by a Partnership for Health System Improvement grant from the Canadian Institutes of Health Research (CIHR; 2010–2013; grant # 200911PHE-216868-PHE-CFBA-19158) and by the Ministère de la Santé et des Services Sociaux du Québec (MSSS), QC, Canada. FL, GG and JMG hold Canada Research Chairs. The following CME organizations provided funding for this study: Consortium pédagogique - Secteur DPC, Faculty of Medicine, Université Laval, Quebec, QC, Canada; Practice Enhancement Division, Collège des médecins du Québec, Montreal, QC, Canada; Fédération des médecins spécialistes du Québec, Montreal, QC, Canada; Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, QC, Canada; Université de Montréal, QC, Canada; and Centre for Continuing Health Professional Education, McGill University, Montreal, QC, Canada. No funding bodies had any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript' on the online submission form.
Competing interests: The authors have declared that no competing interests exist.
Continuing professional development (CPD), including continuing medical education (CME) , is the method most commonly used by physicians to improve their knowledge and skills , . However, in spite of physicians being regularly exposed to new research findings through attending CPD activities, incorporating new knowledge into professional practice is a slow process and delays patients' access to treatments of proven benefit . In addition, producing and accrediting CPD events requires human resources, technology and other materials. In 2012, there were 1319 CME providers accredited by state medical societies in the US reporting expenses of $140.2 million for such activities . It is vital to ensure that these resources constitute a worthwhile investment.
This study was initiated in 2009, when several decision-makers in CPD organizations in Canada formally met with knowledge translation (KT) researchers and identified the need for a short, user-friendly instrument that could be used to routinely assess the impact of CPD activities on clinical practice . At the present time, most frameworks used to evaluate CPD are derived from Kirkpatrick's model , which assesses training effectiveness by measuring four distinct outcome levels: 1) participants' reactions to an educational activity; 2) participants' knowledge, skills, or attitudes; 3) transfer of learning to practice; and 4) the results of the newly acquired behavior on organizational outcomes such as productivity and quality. However, several CPD decision-makers in our milieu have noted that instruments for assessing the impact of CPD activities mostly focus on outcome levels 1 and 2, while outcome levels 3 and 4 are more important for CPD. They also expressed the need for an instrument that was theory-based, i.e. based on factors that are known to predict clinical behavior in health professionals.
A systematic review of 76 studies that used social-cognitive theories for explaining health professionals' clinical behavior  showed that the Theory of Planned Behavior (TPB)  was the most appropriate one for predicting health professionals' behaviors, while Triandis' theory  was a better model for explaining their intentions to perform the behaviors. A model that integrates these two well-known theories proposes that three categories of variables drive behavior: 1) behavioral intention (an individual's motivation) to adopt a specific behavior or not; 2) beliefs about one's capabilities (i.e. health professionals' perceptions of facilitators and barriers to adopting the behavior); and 3) habit (the frequency of performing the behavior in the past). The same integrated model of healthcare professionals' behaviors suggests that behavioral intention can be explained by: 1) beliefs about one's capabilities (also a determinant of behavior, as mentioned above); 2) beliefs about consequences; 3) moral norm (feeling of personal obligation regarding the adoption of the behavior); 4) social influences (perception of approval or disapproval by persons significant to the individual regarding the adoption of the behavior); 5) role and identity (beliefs about whether such behavior should be adopted by someone of similar age, sex or social position), and 6) characteristics of health professionals based on their socio-demographic data.
In the context of CPD, the constructs of the integrated model correspond to three of the four outcome levels proposed by Kirkpatrick's model. The Kirkpatrick Level 2 learning outcomes, attitude and skill improvements, relate to behavioral intention, whereas the Level 3 outcomes, transfer of learning to practice, relate to clinical behavior. The necessary transition from Level 2 to Level 3 can only occur as a result of changes promoted by the content and format of the CPD activity. This transition will depend on acquisition of skills, but also on other variables captured by the integrated model. The aim of the research project was to develop an instrument based on this integrated model to assess the impact of continuing professional development on clinical behavioral intentions . This paper presents the first steps in the development of this instrument.
A detailed protocol of this study has been published and is available on-line .
Ethics approval for the project was received from the Research Ethics Board Committee of the Centre Hospitalier Universitaire de Québec (CHU de Québec) on 30 June 2010 (project # S10-06-033). Participants provided written informed consent to participate in this study. The ethics board committee approved the consent procedure used.
Phase 1 – Systematic review
We updated an existing systematic review to October 31, 2010 . Briefly, we sought instruments based on social-cognitive theories whose purpose was to predict healthcare professionals' clinically-oriented behavioral intentions and behaviors. Only instruments in English or French were included for item extraction. We extracted all items from eligible instruments and created an inventory of items. Pairs of reviewers independently analyzed all items to verify that they were adequately formulated to assess the constructs of the integrated model. Pairs of reviewers reclassified items when necessary to better match these constructs . We computed the percentage of agreement between the authors' initial classifications and reviewers' findings on item phrasing. Discrepancies in reclassifications were resolved through discussion among team members. We removed duplicates by searching for common phrase structures to reduce the total number of items to be analyzed for each construct.
Phase 2 – Selection of a preliminary set of items
Members of the research team formed a partnership committee of researchers and CPD decision-makers (n = 8) to select a preliminary set of items through a 3-step process. First, we provided members of the committee with the full list of items derived from Phase 1 for each of the constructs of the integrated model. Each member individually reviewed the inventory and ranked each item according to two criteria: a) how representative of the construct it was, and b) its applicability to different types of CPD activities. Second, AF produced a comprehensive list containing the most voted-for items for each of the constructs. Third, two conference calls were conducted to review the most voted-for items and to reach a consensus about which items should be included in a preliminary set of items.
Phase 3 – E-Delphi study
Participants and recruitment strategy.
An international group of experts was recruited; individuals were identified through team members' networks. To be eligible, they had to have expertise in knowledge translation, organization of CPD activities, clinical practice, measurement and evaluation, continuing medical education, or social-cognitive theories. They had to speak English or French (we conducted our e-Delphi in both languages).
Data collection and analysis.
The process was conducted in a quasi-anonymous manner. Registered participants' emails were known only to the research coordinator to allow for sending reminders. Respondents' judgements and opinions remained strictly anonymous to other members of the expert group. For each round, participants were emailed a link to the questionnaire in the language of their choice (English or French), and were allotted two weeks to complete it. Email reminders were sent 48 hours before the deadline for each round.
The international group of experts was then asked to evaluate whether the items would be relevant for a generic tool that could be easily adapted to any CPD activity, using a 5-point Likert scale (1 = item completely irrelevant, 5 = item completely relevant). A clinical vignette illustrating how the proposed items could be used in a CPD activity whose learning objective was to perform a knee evaluation was given as an example, but participants were asked to rate their response to each item formulated in general terms (e.g. I intend to adopt the behavior described in the training activity objectives in my practice). In the first round, participants were asked simply to rate their responses to each item. In the second round, distributions of respondents' answers to each item in the previous round were presented in percentage form. In both rounds, participants were encouraged to comment both on the relevance of particular items and on the relevance of the questionnaire as a whole to evaluating the impact of the CPD activity on adoption of a clinical behavior. As there are no definite criteria for determining consensus in a Delphi study , content validity was set a priori when at least 75% of participants had reached agreement on the relevance of an item. A partial consensus was reached when more than 60% but less than 75% of participants agreed on an item's relevance. Absence of consensus was determined to be when less than 60% of participants agreed on the relevance of an item. Once the experts had completed this task, the partnership committee reviewed the final list of selected items. The committee analyzed the experts' comments on each item and reformulated the original items when judged necessary. Items that did not reach a consensus rate of at least 60% were excluded.
Phase 4 – Assessment of the reliability and validity of the new theory-based instrument
Participants and recruitment strategy.
The provisional instrument resulting from Phase 3 was assessed by participants recruited during a scientific conference at Université Laval, held in May, 2012. The conference included 20 lectures covering various topics related to emergency medicine, family medicine, palliative care, perinatology, and internal medicine. The target population was family physicians and residents. All participants who attended an eligible CPD activity were asked to complete the provisional questionnaire at the end of the activity and then again two weeks later. This resulted in recruitment of participants in 18 out of the 25 activities offered.
Data collection and analysis.
Participants were asked to complete a self-administered paper-based questionnaire that was generated specifically for each CPD activity retained. The questionnaires were adapted by including one learning objective from the specific CPD activity in the wording of all items. Participants completed their questionnaire immediately after the CPD activity and again two weeks later. Statistical analysis was conducted on the healthcare professionals' evaluations of the items in such as a way as to preserve the structure of the integrated model, i.e. as all constructs had to have at least two items, the best scoring items for each construct were retained. Descriptive statistics including means, standard deviations, median and frequency were computed to summarize health professional characteristics and item responses for the test retest. Weighted kappa coefficients were calculated to assess test-retest reliability . Exploratory factor analysis (EFA) was used to explore the factorial structure of the questionnaire. The number of factors was determined by three criteria: a) eigenvalues superior to 1; b) proportion of variance explained by the factor superior to 5%; and c) number of theoretical variables of the integrated model. Items were loaded if the load factor was ≥0.4. Cronbach alpha coefficients were calculated to assess internal consistency of each construct proposed by the EFA and to reduce the number of items in the questionnaire. A Cronbach alpha value ≥0.7 was considered acceptable. Then for each construct, missing values for the mean calculation were imputed if at least two items were available.
Figure 1 shows the four phases in the development of the instrument.
Phase 1 – Systematic review
We retrieved the 52 studies from the original systematic review  reporting good psychometric properties and identified 20 new studies from the updated searches covering October 30, 2007 to October 31, 2010 , , , , , , , , , , , , , , , , , , , . Our review included 72 studies , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , . We were able to get access to published versions or copies of 40 instruments , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , . Authors of the 32 remaining tools , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,  were contacted and copies of their instruments requested. We received seven answers , , , , , , , resulting in 47 eligible instruments , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,  that were included for data extraction. A total of 1218 items were identified. Among these, 194 items (16%) were not appropriately labeled in the published papers for measuring the constructs of the theoretical model used for instrument development. Overall, the percentage of agreement among reviewers on this assessment was 84%.
Phase 2 – Selection of a preliminary set of items
Table 1 shows the results of each step in the generation of a preliminary set of items. First, the percentage of agreement among reviewers regarding mapping of the initial 1218 items onto constructs was 84%. Second, the removal of duplicates reduced the total number of items to 360 (see Table 1 for details). Third, members of the partnership committee individually analyzed each of the 360 generic items, and found only 95 items that met the two pre-established criteria, i.e. a) they were representative of one of the theoretical constructs of the integrated model and b) they could be adapted to any CPD context. The committee then re-analyzed these 95 items over two conference calls and selected 61 items to form a preliminary set that matched all the constructs.
Phase 3 – E-Delphi study
Seventy experts from different disciplines registered in the e-Delphi study (see Table 2). This group was composed equally of men and women. The majority of the experts registered in the study were French speakers (70%) and were from Canada (90%). Experts from the USA (n = 3), Australia (n = 2), Switzerland (n = 1) and The Netherlands (n = 1) also registered to participate. Table 3 summarizes the number of items per construct at each step of the e-Delphi study. Forty-six respondents participated in Round 1 (Figure 1). After synthesis of experts' responses, consensus was achieved among at least 75% of respondents that of the 61 items, 11 items were relevant and 14 items were irrelevant. These 25 items were excluded from Round 2 in order to focus on the items for which there was less clarity.
For the second e-Delphi round, a 36-item questionnaire was sent back to the experts along with the mean scores for relevance obtained for each item at the first round. Forty-one respondents participated in Round 2 (Figure 1). After synthesis of the experts' responses, 15 further items were found relevant and 10 items were found irrelevant by at least 75% of respondents.
Taking both the integrity of the integrated model of healthcare professionals' behaviors and the results of the e-Delphi study into consideration, the partnership committee designed a 40-item provisional questionnaire which included 37 items that had achieved full (26) or partial (11) consensus after two rounds. In addition, there was consensus among the international experts that the three “moral norm” items, judged irrelevant in the e-Delphi process, would be relevant if formulated differently. Following their advice, the partnership committee rephrased these three items positively instead of negatively and reintegrated them into the provisional instrument. This resulted in a 40-item questionnaire with between two and ten items for each construct: beliefs about consequences (n = 9), beliefs about capabilities (n = 9), social influences (n = 5), intention (n = 5), role and identity (n = 4), moral norm (n = 4), habit/past behavior (n = 2), and behavior (n = 2).
Phase 4 – Assessment of the reliability and validity of the new theory-based instrument
Characteristics of healthcare professionals included in the study.
Table 4 shows the characteristics of the 138 participants included in this phase of the study, i.e. healthcare professionals who were engaged in CPD activities at a scientific congress and who filled out questionnaires (test). Of the 138 participants, 76.8% were female. Among the participants, 42% were residents, 49.3% were family physicians, 2.2% were specialists, and 6.5% were healthcare professionals other than physicians. The same questionnaire was returned two weeks later (retest) by 88% of participants (121/138). In contrast with the procedure described in the study protocol , the partnership committee reviewed the preliminary analysis based on the items validated in the e-Delphi study and removed 16 out the 40 items to respect the principles of the theoretical model. Specifically, the items concerning the construct “habit” were removed because they did not reflect the model's definition of habit (poor wording).
Test retest reliability.
The first data collection in this phase and the second data collection two weeks later provided data for assessing the reliability of the instrument over time. Test-retest reliability was moderate (Table 5) with weighted kappa values between 0.4 and 0.6.
Exploratory factorial analysis (EFA).
EFA showed that four items loaded on more than one construct and they were therefore excluded. One item did not load on any construct and five items loaded on the wrong constructs, leaving 14 remaining items. All factors had eigenvalues superior to 1; the proportion of variance explained by the factor was superior to 5%. During this process, the items relating to the construct “role and identity” were removed because no items loaded on any factors as defined by EFA.
Cronbach alpha coefficients.
Cronbach alpha coefficients were considered acceptable (range: 0.79 to 0.89). Two items were removed because of poor consistency (Cronbach alpha <0.70), resulting in a final questionnaire of 12 items (Table 6).
This study established the reliability and validity of a simple 12-item theory-based instrument for evaluating the impact of CPD activities on health professionals' clinical behavioral intentions. Based on an extensive systematic review, we found a total of 1218 items from 47 eligible instruments. Sixteen per cent were not appropriately labeled and most were duplicates, leaving us with 360 original items covering the theoretical constructs of the integrated model of healthcare professionals' behaviors. In a second phase, members of a partnership committee selected 61 of these 360 items based on how well they represented one of the theoretical constructs of interest and on their adaptability to any CPD context. Then, through an e-Delphi process, an international group of experts from a wide range of disciplines achieved consensus on 37 items which were included in a first version of a 40-item questionnaire. In a last phase, 138 physicians helped establish that the final 12-item instrument had validity including adequate factorial structure and reliability. These results lead us to make four main observations.
First, during our initial phase we found that a significant proportion of the items derived from the systematic review, i.e. 16%, were not appropriately formulated to measure the constructs of the theories the authors had used to develop their tools, but applied instead to other constructs. This may have created a bias in the findings of the authors we consulted regarding the relationship between one given variable (e.g. attitude of health professionals) and another (e.g. clinical behavior). While these authors often associated the weakness of their results with the absence of validation procedures for their questionnaires , , , they rarely questioned if their poor results were a consequence of inappropriate wording of items in relation to the theory on which the questionnaires were based . The majority of items, moreover, although they had slight variations in wording, were found to be duplicates. These observations clearly indicate that there is a need for a theory-based instrument that is validated and reliable for assessing the impact of CPD activities on health professional behaviors. This will help standardize the presentation of the many factors that are likely to influence the uptake of new clinical behaviors . In turn this will: 1) facilitate comparisons between similar studies, 2) make it possible to carry out a systematic review in this area, 3) help inform policy makers about how to change clinical behaviors and, most importantly, 4) contribute to the elaboration of a theoretical base for translating evidence into clinical practice.
Second, in subsequent phases of this project, it became clear that theory-based items used for studying healthcare professionals' behaviors were not all relevant or applicable to the CPD context. CPD developers who had initiated this project had made very clear that they wanted a short, simple instrument that could be easily disseminated across the CPD environment. Their difficulties in this regard are consistent with those reported by implementation scientists trying to assess the impact of their interventions with busy clinicians. The proposed instrument is based on well-validated socio-cognitive theories, and yet includes only 12 items. It may therefore contribute not only to a more theory-based approach to CPD assessment but to a more pragmatic one.
Third, the e-Delphi study asked the international experts to evaluate items in generic terms, but to facilitate comprehension we presented an example extracted from a clinical vignette used in a specific CPD activity (knee evaluation). This may explain why the majority of the international experts judged that the items relating to “moral norm” were not relevant when negatively formulated. However, in the last phase of this study we demonstrated that the instrument can be easily adapted to different clinical areas of medicine. When we tested the instrument with participants in a scientific congress, we observed that the items adapted well to a variety of clinical areas such as emergency medicine, family medicine, palliative care, perinatology, and internal medicine. Adapting the tool to each context simply requires rewording each of the 12 items so that they include the observable behavior described in the learning objective of the activity. Indeed, the generic and adaptable nature of the instrument to a wide range of clinical domains is its main advantage over other such instruments .
Lastly, in order to develop this tool, our multidisciplinary partnership committee used a highly systematic multi-phase approach to refine the link between the items and the theoretical constructs they are supposed to measure. Although designing CPD strategies that will impact health professionals' behavior and improve patient outcomes is complex, effective CPD is an essential vehicle for knowledge translation . During our whole project, a strong partnership between researchers and CPD developers ensured that all concerns would be taken into account so that our final instrument would prove useful to both educators and researchers. We believe that the newly developed instrument will not only help CPD developers improve the effectiveness of their training programs by paying more attention to the socio-cognitive factors known to predispose healthcare professionals to change their clinical behavior, but also help researchers who are interested in further exploring these socio-cognitive factors.
This study has some limitations. Studies reporting the development of instruments are generally not well-indexed in electronic databases . Some eligible instruments may not have been captured by the search strategy used. Our analysis was also restricted to instruments of which it was possible to obtain a copy. However, given the large number of duplicates among the items found, we are confident that finding additional instruments would not have had an impact on our study results. In this study, we accepted the premise that behavioral intention is an immediate antecedent of behavior, and that intention provides a proxy measure of physicians' behavior  that corresponds to Kirkpatrick's Level 3 . Nonetheless, further studies are necessary to demonstrate a correlation between behavioral intention and observed behavior of healthcare professionals subsequent to CPD interventions.
In conclusion, we believe that this new 12-item theory-based instrument with robust metric properties is appropriate for the routine assessment of the impact of CPD activities on clinical behavioral intention change. We plan further studies to validate on a larger scale its ability to predict behavior and its sensitivity to change in response to CPD activities. We will also verify the acceptability of the 12-item instrument among CPD providers by examining the barriers and facilitating factors associated with the implementation of the new tool in the province of Quebec, Canada.
The members of the CPD-KT team are: Stephane Turcotte, Public Health and Practice-Changing Research Group, CHUQ Research Centre (CRCHUQ), Quebec, Canada; Renée Drolet, Public Health and Practice-Changing Research Group, CHUQ Research Centre (CRCHUQ), Quebec, Canada; Réjean Laprise, Recherche et développement, Office de développement professionnel, Fédération des médecins spécialistes du Québec, Quebec, Canada; Michel Rousseau, Department of Psychoeducation, Faculty of Social Sciences, Université du Québec à Trois Rivières, Quebec, Canada; Andrée Boucher, Centre de pédagogie appliquée aux sciences de la santé, Faculty of Medicine, Université de Montréal, Quebec, Canada; Joan Sargeant, Division of Medical Education, Faculty of Medicine, Dalhousie University, Nova Scotia, Canada; Gilles Voyer, Centre de formation continue, Faculty of Medicine, Université de Sherbrooke, Quebec, Canada, Michel Labrecque, Public Health and Practice-Changing Research Group, CHUQ Research Centre (CRCHUQ), Quebec, Canada.
We convey our thanks to Louisa Blair who edited this article.
Conceived and designed the experiments: F. Légaré FB AJ GG F. Luconi JG AF CPD-KT team. Performed the experiments: F. Légaré AF FB AJ GG F. Luconi. Analyzed the data: F. Légaré FB AJ GG F. Luconi AF. Wrote the paper: F. Légaré AF. Revised the manuscript critically for important intellectual content and all approved the final version submitted: F. Légaré FB AF AJ GG F. Luconi JG CPD-KT team.
- 1. Frank J (2005) The CanMEDS Physician Competency Framework. Better standards. Better physicians. Better care. Office of Education of The Royal College of Physicians and Surgeons of Canada
- 2. Campbell EG, Rosenthal M (2009) Reform of continuing medical education: investments in physician human capital. JAMA 302: 1807–1808.
- 3. Federation of Medical Regulatory authorities of Canada (2008) Position statement 2008.
- 4. Agency for Health Research and Quality (2001) Translating research into practice (TRIP)-II. Washington, DC: Agency for Health Research and Quality.
- 5. Accreditation Council for Continuing Medical Education (2012) ACCME Annual report data 2012. Available: http://wwwaccmeorg/sites/default/files/630_2012_Annual_Report_20130724_1pdf.
- 6. Centre de développement professionnel continu de la Faculté de médecine de l'Université Laval, Collège des médecins du Québec et Centre de Recherche du CHU de Québec (2011) Forum d'échange sur le transfert des connaissances et le développement professionnel continu. Available: http://w3fmedulavalca/dpctc/?id=1038. Accessed 2013 May 1.
- 7. Kirkpatrick DL (1994) Evaluating training programs: the four levels. Berrett-Koehler Publishers Inc., San Francisco, California.
- 8. Godin G, Belanger-Gravel A, Eccles M, Grimshaw J (2008) Healthcare professionals' intentions and behaviours: a systematic review of studies based on social cognitive theories. Implement Sci 3: 36.
- 9. Ajzen I (1991) The theory of planned behavior. Organ Behav Hum Decis Process. 50: 179–211.
- 10. Triandis HC (1980) Values, attitudes and interpersonal behavior. In: Page MM Lincoln, NE, editors. Nebraska symposium on motivation beliefs, attitudes and values, Vol 1. University of Nebraska Press. pp 195–259.
- 11. Legare F, Borduas F, Jacques A, Laprise R, Voyer G, et al. (2011) Developing a theory-based instrument to assess the impact of continuing professional developement activites on clinical practice: a study protocol. Implement Sci 6: 17.
- 12. Hsu CC, Sandford BA (2007) The Delphi technique: making sense of consensus. Practical Assessment, Research and Evaluation 12: 7.
- 13. Landis JR, Koch GC (1977) The measurement of observer agreement for categorical data. Biometrics 33: 16.
- 14. Askelson NM, Campo S, Lowe JB, Dennis LK, Smith S, et al. (2010) Factors related to physicians' willingness to vaccinate girls against HPV: the importance of subjective norms and perceived behavioral control. Women Health 50: 144–158.
- 15. Bercher D (2008) Attitudes of paramedics to home hazard inspections: Applying the theory of planned behavior. Fayetteville: University of Arkansas. 140 p.
- 16. Bernaix LW, Schmidt CA, Arrizola M, Iovinelli D, Medina-Poelinez C (2008) Success of a lactation education program on NICU nurses' knowledge and attitudes. J Obstet Gynecol Neonatal Nurs 37: 436–445.
- 17. Bonetti D, Johnston M, Clarkson J, Turner S (2009) Applying multiple models to predict clinicians' behavioural intention and objective behaviour when managing children's teeth. Psychol Health 24: 843–860.
- 18. Casper ES (2007) The theory of planned behavior applied to continuing education for mental health professionals. Psychiatr Serv 58: 1324–1329.
- 19. Casper ES (2008) Using implementation intentions to teach practitioners: changing practice behaviors via continuing education. Psychiatr Serv 59: 747–752.
- 20. Eccles MP, Francis J, Foy R, Johnston M, Bamford C, et al. (2009) Improving professional practice in the disclosure of a diagnosis of dementia: a modeling experiment to evaluate a theory-based intervention. Int J Behav Med 16: 377–387.
- 21. Ellery J, McDermott RJ, Ellery PJ (2007) Computers as a formal continuing education tool: moving beyond intention. Am J Health Behav 31: 312–322.
- 22. Fields S (2008) The role of the Theory of Planned Behavior in therapists' involvement of parents in youth treatment. St. Petersburg: University of South Florida. 94 p.
- 23. Green H, Johnston O, Cabrini S, Fornai G, Kendrick T (2008) General practitioner attitudes towards referral of eating-disordered patients: a vignette study based on the theory of planned behaviour. Ment Health Fam Med 5: 213.
- 24. Hanbury A, Wallace L, Clark M (2009) Use of a time series design to test effectiveness of a theory-based intervention targeting adherence of health professionals to a clinical guideline. Br J Health Psychol 14: 505–518.
- 25. Harbour VJ (2009) Using the theory of planned behavior to gain understanding of primary-care providers' behavior in counseling patients to engage in physical activity. Salt Lake City, Utah: University of Utah. 131 p.
- 26. Khanna R, Kavookjian J, Scott VG, Kamal KM, Miller LA, et al. (2009) Using the theory of reasoned action to determine physicians' intention to measure body mass index in children and adolescents. Res Social Adm Pharm 5: 170–181.
- 27. Marrone SR (2008) Factors that influence critical care nurses' intentions to provide culturally congruent care to Arab Muslims. J Transcult Nurs 19: 8–15.
- 28. Natan MB, Beyil V, Neta O (2009) Nurses' perception of the quality of care they provide to hospitalized drug addicts: testing the theory of reasoned action. Int J Nurs Pract 15: 566–573.
- 29. Pielak KL, McIntyre CC, Tu AW, Remple VP, Halperin B, et al. (2010) Identifying attitudes, beliefs and reported practices of nurses and doctors as immunization providers. J Adv Nurs 66: 1602–1611.
- 30. Pradel FG, Obeidat NA, Tsoukleris MG (2007) Factors affecting pharmacists' pediatric asthma counseling. J Am Pharm Assoc (2003) 47: 737–746.
- 31. Ramsay CR, Thomas RE, Croal BL, Grimshaw JM, Eccles MP (2010) Using the theory of planned behaviour as a process evaluation tool in randomised trials of knowledge translation strategies: A case study from UK primary care. Implement Sci 5: 71.
- 32. Shoham S, Gonen A (2008) Intentions of hospital nurses to work with computers: based on the theory of planned behavior. Comput Inform Nurs 26: 106–116.
- 33. Ten Wolde GB, Dijkstra A, Van Empelen P, Knuistingh Neven A, Zitman FG (2008) Psychological determinants of the intention to educate patients about benzodiazepines. Pharm World Sci 30: 336–342.
- 34. Bernaix LW (2000) Nurses' attitudes, subjective norms, and behavioral intentions toward support of breastfeeding mothers. J Hum Lact 16: 201–209.
- 35. Bonetti D, Eccles M, Johnston M, Steen N, Grimshaw J, et al. (2005) Guiding the design and selection of interventions to influence the implementation of evidence-based practice: an experimental simulation of a complex intervention trial. Soc Sci Med 60: 2135–2147.
- 36. Bonetti D, Pitts NB, Eccles M, Grimshaw J, Johnston M, et al. (2006) Applying psychological theory to evidence-based clinical practice: identifying factors predictive of taking intra-oral radiographs. Soc Sci Med 63: 1889–1899.
- 37. Chau PYK, Hu PJH (2001) Information technology acceptance by individual professionals: A model comparison approach. Decision Sciences 32: 699–719.
- 38. Crawford TE (1998) Factors influencing critical care nurses' involvement with families in the intensive care unit. Halifax: Dalhousie University. 254 p.
- 39. Daneault S, Beaudry M, Godin G (2004) Psychosocial determinants of the intention of nurses and dietitians to recommend breastfeeding. Can J Public Health 95: 151–154.
- 40. Dilorio C (1997) Neuroscience nurses' intentions to care for persons with HIV/AIDS. J Neurosurg Nurs 29: 50.
- 41. Eccles MP, Grimshaw JM, Johnston M, Steen N, Pitts NB, et al. (2007) Applying psychological theories to evidence-based clinical practice: identifying factors predictive of managing upper respiratory tract infections without antibiotics. Implement Sci 2: 26.
- 42. Edwards HE, Nash RE, Najman JM, Yates PM, Fentiman BJ, et al. (2001) Determinants of nurses' intention to administer opioids for pain relief. Nurs Health Sci 3: 149–159.
- 43. Farris K, Kirking D (1995) Predicting community pharmacists' intention to try to prevent and correct drug therapy problems. J Soc Adm Pharm 12: 64–64.
- 44. Farris KB, Schopflocher DP (1999) Between intention and behavior: an application of community pharmacists' assessment of pharmaceutical care. Soc Sci Med 49: 55–66.
- 45. Faulkner G, Biddle S (2001) Predicting physical activity promotion in health care settings. Am J Health Promot 16: 98–106.
- 46. Feng JY, Wu YW (2005) Nurses' intention to report child abuse in Taiwan: a test of the theory of planned behavior. Res Nurs Health 28: 337–347.
- 47. Foy R, Bamford C, Francis JJ, Johnston M, Lecouturier J, et al. (2007) Which factors explain variation in intention to disclose a diagnosis of dementia? A theory-based survey of mental health professionals. Implement Sci 2: 31.
- 48. Foy R, Walker A, Ramsay C, Penney G, Grimshaw J, et al. (2005) Theory-based identification of barriers to quality improvement: induced abortion care. Int J Qual Health Care 17: 147–155.
- 49. Gagnon MP, Godin G, Gagne C, Fortin JP, Lamothe L, et al. (2003) An adaptation of the theory of interpersonal behaviour to the study of telemedicine adoption by physicians. Int J Med Inform 71: 103–115.
- 50. Gagnon MP, Sanchez E, Pons JM (2006) From recommendation to action: psychosocial factors influencing physician intention to use Health Technology Assessment (HTA) recommendations. Implement Sci 1: 8.
- 51. Gilomen-Study G (1998) Predicting managed care prescribing using the Theory of Reasoned Action and the Theory of Planned Behavior. Chicago: University of Illinois. 149 p.
- 52. Godin G, Beaulieu D, Touchette JS, Lambert LD, Dodin S (2007) Intention to encourage complementary and alternative medicine among general practitioners and medical students. Behav Med 33: 67–77.
- 53. Godin G, Naccache H, Brodeur JM, Alary M (1999) Understanding the intention of dentists to provide dental care to HIV+ and AIDS patients. Community Dent Oral Epidemiol 27: 221–227.
- 54. Godin G, Naccache H, Morel S, Ebacher MF (2000) Determinants of nurses' adherence to universal precautions for venipunctures. Am J Infect Control 28: 359–364.
- 55. Harrell GD, Bennett PD (1974) An evaluation of the expectancy value model of attitude measurement for physician prescribing behavior. J Mark Res 269–278.
- 56. Herbert KE, Urmie JM, Newland BA, Farris KB (2006) Prediction of pharmacist intention to provide Medicare medication therapy management services using the theory of planned behavior. Res Social Adm Pharm 2: 299–314.
- 57. Hoppe C (1999) Predicting health professionals management of obesity. London, UK: King's College London (University of London).
- 58. Hu PJ, Chau PYK, Sheng ORL, Tam KY (1999) Examining the technology acceptance model using physician acceptance of telemedicine technology. Journal of Management Information Systems 16: 91–112.
- 59. Jenner E, Watson P, Miller L, Jones F, Scott G (2002) Explaining hand hygiene practice: an extended application of the theory of planned behaviour. Psychol Health Med 7: 311–326.
- 60. Jurgens DW (1996) Nurses' intentions to administer morphine for post-operpative pain: an application of Azjen's theory of planned behaviour. Saskatoon: University of Saskatchewan. 256 p.
- 61. Kittisopee T (2001) The influences of personality, social cognition, and environmental factors on pharmaceutical care. Iowa City: The University of Iowa. 224 p.
- 62. Lambert BL, Salmon JW, Stubbings J, Gilomen-Study G, Valuck RJ, et al. (1997) Factors associated with antibiotic prescribing in a managed care setting: an exploratory investigation. Soc Sci Med 45: 1767–1779.
- 63. Lavoie M, Blondeau D, Godin G (1999) Intentions to select a given level of care when confronted with an ethical issue: the impact of a living will. J Appl Soc Psychol 29: 772–785.
- 64. Legare F, Godin G, Ringa V, Dodin S, Turcot L, et al. (2005) Variation in the psychosocial determinants of the intention to prescribe hormone therapy prior to the release of the Women's Health Initiative trial: a survey of general practitioners and gynaecologists in France and Quebec. BMC Med Inform Decis Mak 5: 31.
- 65. Legare F, Graham ID, O'Connor AC, Aubin M, Baillargeon L, et al. (2007) Prediction of health professionals' intention to screen for decisional conflict in clinical practice. Health Expect 10: 364–379.
- 66. Liabsuetrakul T, Chongsuvivatwong V, Lumbiganon P, Lindmark G (2003) Obstetricians' attitudes, subjective norms, perceived controls, and intentions on antibiotic prophylaxis in caesarean section. Soc Sci Med 57: 1665–1674.
- 67. Mason HL (1983) Using attitudes and subjective norms to predict pharmacist counselling behaviors. Patient Couns Health Educ 4: 190–196.
- 68. Maue SK, Segal R, Kimberlin CL, Lipowski EE (2004) Predicting physician guideline compliance: an assessment of motivators and perceived barriers. Am J Manag Care 10: 383–391.
- 69. Millstein SG (1996) Utility of the theories of reasoned action and planned behavior for predicting physician behavior: a prospective analysis. Health Psychol 15: 398–402.
- 70. O'Boyle CA, Henly SJ, Larson E (2001) Understanding adherence to hand hygiene recommendations: the theory of planned behavior. Am J Infect Control 29: 352–360.
- 71. Payant L (2006) Factors related to childbirth nurses' intentions to provide continuous labour support to women during childbirth. Ottawa: University of Ottawa. 120 p.
- 72. Pessoa-Silva CL, Posfay-Barbe K, Pfister R, Touveneau S, Perneger TV, et al. (2005) Attitudes and perceptions toward hand hygiene among healthcare workers caring for critically ill neonates. Infect Control Hosp Epidemiol 26: 305–311.
- 73. Puffer S, Rashidian A (2004) Practice nurses' intentions to use clinical guidelines. J Adv Nurs 47: 500–509.
- 74. Quinn R (1996) Attitude, subjective norm, behavioral intention, and patient teaching among nurses [Doctor]: Philadelphia: Temple University. 226 p.
- 75. Renfroe DH, O'Sullivan PS, McGee GW (1990) The relationship of attitude, subjective norm, and behavioral intent to the documentation behavior of nurses. Sch Inq Nurs Pract 4: 47–60 Discussion pp 61–44.
- 76. Ryu S, Ho SH, Han I (2003) Knowledge sharing behavior of physicians in hospitals. Expert Syst Appl 25: 113–122.
- 77. Sauls DJ (2007) Nurses' attitudes toward provision of care and related health outcomes. Nurs Res 56: 117–123.
- 78. Tessaro I, Highriter M (1994) HIV and the work intentions of public health nurses. Public Health Nurs 11: 273–280.
- 79. Vermette L, Godin G (1996) Nurses' intentions to provide home care: the impact of AIDS and homosexuality. AIDS Care 8: 479–488.
- 80. Walker A, Watson M, Grimshaw J, Bond C (2004) Applying the theory of planned behaviour to pharmacists' beliefs and intentions about the treatment of vaginal candidiasis with non-prescription medicines. Fam Pract 21: 670–676.
- 81. Wallace KG, Koeppel K, Senko A, Stawiaz K, Thomas C, et al. (1997) Effect of attitudes and subjective norms on intention to provide oral care to patients receiving antineoplastic chemotherapy. Cancer Nurs 20: 34–41.
- 82. Walsh AM, Edwards HE, Courtney MD, Wilson JE, Monaghan SJ (2005) Fever management: paediatric nurses' knowledge, attitudes and influencing factors. J Adv Nurs 49: 453–464.
- 83. Watson P, Myers L (2001) Which cognitive factors predict clinical glove use amongst nurses? Psychology, Health and Medicine 6: 399–409.
- 84. Werner P, Mendelsson G (2001) Nursing staff members' intentions to use physical restraints with older people: testing the theory of reasoned action. J Adv Nurs 35: 784–791.
- 85. Wilson C (1998) Applying the Theory of planned behavior to predict and explain school psychologist's use of the consultation model. Lincoln: University of Nebraska. 200 p.
- 86. Saillour-Glenisson F, Michel P (2003) Individual and collective facilitators and barriers to the use of clinical guidelines by physicians: a literature review. Revue Épidémiologique de Santé Publique 51: 65–80.
- 87. Tian J, Atkinson NL, Portnoy B, Lowitt NR (2010) The development of a theory-based instrument to evaluate the effectiveness of continuing medical education. Acad Med 85: 1518–1525.
- 88. Legare F, Borduas F, MacLeod T, Sketris I, Campbell B, et al. (2011) Partnerships for knowledge translation and exchange in the context of continuing professional development. J Contin Educ Health Prof 31: 181–187.
- 89. Elwyn G, Edwards A, Mowle S, Wensing M, Wilkinson C, et al. (2001) Measuring the involvement of patients in shared decision-making: a systematic review of instruments. Patient Educ Couns 43: 5–22.