Tablet computer enhanced training improves internal medicine exam performance

Background Traditional teaching concepts in medical education do not take full advantage of current information technology. We aimed to objectively determine the impact of Tablet PC enhanced training on learning experience and MKSAP® (medical knowledge self-assessment program) exam performance. Methods In this single center, prospective, controlled study final year medical students and medical residents doing an inpatient service rotation were alternatingly assigned to either the active test (Tablet PC with custom multimedia education software package) or traditional education (control) group, respectively. All completed an extensive questionnaire to collect their socio-demographic data, evaluate educational status, computer affinity and skills, problem solving, eLearning knowledge and self-rated medical knowledge. Both groups were MKSAP® tested at the beginning and the end of their rotation. The MKSAP® score at the final exam was the primary endpoint. Results Data of 55 (tablet n = 24, controls n = 31) male 36.4%, median age 28 years, 65.5% students, were evaluable. The mean MKSAP® score improved in the tablet PC (score Δ + 8 SD: 11), but not the control group (score Δ- 7, SD: 11), respectively. After adjustment for baseline score and confounders the Tablet PC group showed on average 11% better MKSAP® test results compared to the control group (p<0.001). The most commonly used resources for medical problem solving were journal articles looked up on PubMed or Google®, and books. Conclusions Our study provides evidence, that tablet computer based integrated training and clinical practice enhances medical education and exam performance. Larger, multicenter trials are required to independently validate our data. Residency and fellowship directors are encouraged to consider adding portable computer devices, multimedia content and introduce blended learning to their respective training programs.


Introduction
Traditional teaching concepts in medical education do not take full advantage of information technology, despite the fact that modern clinical medicine and biomedical science are packed with digital media resources reaching from multidimensional virtual imaging data of the human body to complex video animations of human physiology.
Medical education ideally happens at the bedside [1], not in lecture halls. Although the use of wireless enabled mobile communication devices [2] including (tablet) computers, personal digital assistants [2] and smartphones [3]-that can help incorporate, process and deliver the ever increasing rich media and information content at the point of care in real time-is substantially increasing [4][5][6], scientific data on their efficacy in medical education and clinical training is limited.
Here we present prospective data demonstrating that Tablet PC enabled eLearning significantly impacts on exam performance and prospect for future medical trainees.

Methods
This single center, prospective, controlled study was conducted on an internal medicine ward at Charité Medical Center's Virchow Hospital, Medical School of the Humboldt-University of Berlin. For the purpose of the study the ward was equipped with three wireless access points (Enterasys, Salem, NH, USA) linking it to the hospital's intra-and global internet as well as a Net Education Center (Hewlett Packard, Palo Alto, CA, USA), a cart housing and charging Tablet PCs.

Active participants and controls
Participation was voluntary and in accordance with both institutional policies and all applicable laws including data privacy legislation. The institutional review board of Charité University Hospital in Berlin confirmed the information provided to participants was in line with the local ethical requirements (No.: EA1/386/16). Eligible participants included consenting medical students in their final year of medical school (acting interns) and postgraduate year 1 to 3 residents doing a rotation on the selected internal medicine ward as a mandatory part of their training curriculum. All participants signed a contract consenting to and detailing the conditions of the study. Timing and duration of their rotation were predetermined by medical school, hospital and physician board rules and regulations. Final year medical students (acting interns) interns did four month rotations, while residents did 6 month rotations. The consecutive cohort of all participants was alternatingly assigned to either the active test (tablet) or traditional education (control) group, respectively.
The active test group was profiled and examined (see below), received a Tablet PC to keep for the entire duration of their rotation and use the multimedia training and education package (see below) in-and outside the medical center campus (i.e. at home and commuting to work).
The control group did not receive a Tablet PC and was only profiled and examined (see below) and had access to all conventional education and training resources (i.e. library, books, journals) on campus.

Objectives and outcomes
The primary objective was to test the hypothesis that Tablet PC enhanced education significantly impacts on participants' performance in medical board exams. The final MKSAP 1 exam score was the primary endpoint. Moreover, we aimed to identify participant's characteristics impacting on the final exam score.

Participant profiling
Both control and tablet group participants had to complete an extensive questionnaire to collect their demographic data and evaluate their educational status, computer affinity and skills, problem solving strategy, eLearning knowledge and judge their self-estimated medical knowledge to assess potential confounding factors on the overall outcome, respectively.

Tablet computers
The HP Compaq tc4200 [7] (Hewlett Packard, Palo Alto, CA, USA) and IBM ThinkPad X41 [8] (IBM, Armonk, NY, USA) are ultraportable notebooks that also convert into tablets. They incorporate technology to provide wireless connectivity and improved battery performance. Tablet PCs are fully functional personal computers delivering performance and compatibility in an innovative form factor. They offer wide-viewing angle displays on protective glass featuring a digital eraser pen that writes like an actual pen.
Custom software package and programming Through this central interface all participants could register, complete their profile questionnaire, take the initial and final knowledge assessment exams, access a variety of multimedia training and education resources as well as the medical center's electronic patient care systems.
The American College of Physician℠ (Philadelphia, PA, USA) kindly provided us with a special electronic version (in XML format) of their MKSAP 1 14 software that allowed integration into our database system and parsing with a random generator.
The multimedia package included access to the institutional collaborative online course management systems (Moodle and Blackboard), eBooks (Springer Nature Science and Business Media, New York, NY, USA), eJournals, educational slide kits, podcasts, videos, animations, images from major biomedical and scientific publishers or professional societies as well as twitter feeds and selected hyperlinks to biomedical and scientific web resources.

Initial and final knowledge assessment
To determine the impact of Tablet PC based education we decided to objectively assess and compare all participants' knowledge in internal medicine at two time points. Importantly, none of the participants had access to or were able to practice the exam questions used in this study or underwent any kind of special knowledge exam preparation.
New medical knowledge recognition [9] and concept identification [10] can be computationally evaluated with the American College of Physicians℠ (ACP) Medical Knowledge Self-Assessment Program MKSAP 1 first introduced in the 1970s [11,12]. MKSAP 1 [11] closely resembles the official American Board of Internal Medicine (ABIM) multiple choice question format and style and has been successfully used to evaluate knowledge and analyze currency of ABIM 1 diplomats [13]. Predictive validity for the ABIM 1 exam has been demonstrated in the past [14]. The internal medicine exam performance of both, the control and the tablet group was tested by administration of 215 out of 1400 random generator selected, equally distributed questions from all eleven MKSAP 1 categories (Foundations of Internal Medicine, Cardiovascular Medicine, Gastroenterology and Hepatology, Rheumatology, Neurology, Hematology and Oncology, Infectious Diseases, Pulmonary and Critical Care Medicine, General Internal Medicine, Endocrinology and Metabolism) parsed from the current ACP's MKSAP 1 digital edition pool.
Our rationale for using MKSAP 1 was its proven track record in evaluating internal medicine knowledge. Although primarily designed for resident use, we felt that final year medical students, i.e. acting first year medical residents could be reliably subjected to it as well. In our opinion the benefit of using a vetted, validated questionnaire such as MKSAP 1 would outweigh its potential limitations and was preferable to designing a brand new knowledge assessment tool.

Data processing and statistical analysis
All statistical analyses were performed with SPSS 22 (IBM, Armonk, NY, USA) software. For descriptive statistics, means and standard deviations, medians and inter quartile ranges (IQR) or absolute and relative frequencies were reported where applicable. Data are expressed in box plots. Both, the Mann-Whitney-U-test [15] and Fisher's-exact-test [16] were used to compare participants profile data. The t-test for independent samples or one-way ANOVA was used to test associations of participant's characteristics with their final score. All variables with a p-value < 0.1 were also tested in a multiple regression model for the final score values. For the multiple regression model self-rated knowledge was dichotomized into excellent/good vs. passable/adequate. Additionally t-tests for related samples were employed to check for significant differences between the mean initial and final exam scores. A two-sided significance level of 0.05 was used. The main hypothesis was the existence of group differences in final scores after accounting for baseline scores and possible confounders. All other tests were secondary. No adjustment for multiple testing was applied.

Participant flow and recruitment
We recruited 80 participants for this study between 2008 and 2012. Data of 55 participants (tablet n = 24, 50% male; controls, n = 31, 25.8% male; median age 28 years) were evaluable and analyzed. The remaining participants' data was incomplete and was excluded from the analysis. Fig 1

Participant profiles
Socio-demographics. Most participants were German nationals. There were no statistically significant differences in age, gender or educational background. Table 1 Exposure to US or other foreign medical education. A fifth of participants had received medical education in foreign countries such as Argentine, Chile, France, Iceland, Italy, Malawi, Russia, Spain, Sweden, Switzerland, The United Kingdom and The United States. However, while many participants were familiar with the term US medical licensing exam (USMLE 1 ), only three participants had actually received medical training in the US. None had ever taken the exam. Table 1 Computer affinity and skills. Most participants owned at least one computer which was a notebook or laptop in half of the cases. However, they mostly used it at home or work and only in less in quarter of cases in other campus locations. Table 1 Currently exposure to eLearning and preferred problem medical solving resources. Participants' exposure to eLearning prior to this study was very limited with one year of experience on average. Their favorite source for medical problem solving were still articles that they preferably looked up on PubMed or Google, and books. Table 1 Self-rated internal medicine knowledge. The majority of participants in both the control and Table PC groups rated their internal medicine knowledge as "passable" or "good" at entry into the study. Only one participant (control group) rated its knowledge as excellent. Table 1 Outcomes and estimation Improved exam performance in the tablet group. The final mean MKSAP 1 score was higher in the tablet group (mean (SD): 59 (19)) compared to the control group (mean (SD): 48 (10)) (p<0.001) Table 2   Characteristics associated with improved exam performance. At bivariate level baseline of all variables tested only tablet pc use and self-rated excellent internal medicine knowledge at baseline had a significant impact of the final exam score. Table 2 After adjustment for baseline score, tablet pc knowledge and self-rated excellent internal medicine knowledge the tablet group showed on average 11% higher MKSAP test results compared to the control group (p<0.001, main hypothesis) Table 3 Discussion We demonstrate for the first time that in a prospective cohort of final year medical students and residents doing an internal medicine inpatient service rotation at an academic medical center the use of a wireless tablet computer based integrated education and portable hospital workstation significantly improves board style exam (MKSAP 1 ) performance. This was true even after adjustment for baseline score, Tablet PC knowledge and self-rated excellent internal medicine knowledge.
Unsurprisingly, the overall absolute MKSAP 1 scores at both time points and in both the control and the tablet group were lower compared with the US national average [17]. This is likely owed to the fact that none of the participants in our study were native English speakers and their exposure to US medical education was very limited. Furthermore, unlike US medical students, residents and foreign (international) medical graduates in the US none had ever taken MKSAP 1 or ABIM 1 exams before or participated in regular in-house exams with comparable questions very commonly administered in the US. Moreover, none of the participants practiced MKSAP 1 , USMLE 1 or ABIM 1 style exams before or during this study either.
Being naïve regarding this exam type and in relation to prior US medical education can also be considered a strength of our study. Achieving a maximum score was not the goal here, but rather to investigate if the educational tablet system would improve exam performance, i.e. has a significant impact on internal medicine knowledge, which was shown in our results.  Interestingly, the scores significantly worsened in the control group. Perhaps their motivation was lower due to the lack of the incentive of an otherwise desired technical device, which may have been an additional stimulus beyond the actual education software in the tablet computer group. While commonly employed and well proven according to some [18,19], measurements and metrics may actually also deteriorate individual physician performance [20,21]. Tablet PCs in medicine Our study design was unable to detect any such an effect. The difference between Tablet PC and control group could however not be attributed to other socio-demographic factors or computer affinity surrogates either.
Our study furthermore demonstrates that the improved exam performance was significantly associated with the self-rated internal medicine baseline knowledge of participants. This appears plausible as a technical education tool can obviously not replace prior factual medical knowledge acquisition nor supersede basic pedagogic principles.
Our work has limitations. The number of evaluable cases was small and thus the data of this pilot study needs to be validated in larger series before conclusions can be generalized. We also experienced the problem of participant attrition, well known from major educational research studies [22]. The high drop-out rate may relate to the voluntary nature of study participation and perhaps conceiving the extra exams or device as a burden. At the same time the Hawthorne effect [23] also known as observer [24] effect, i.e. the reactivity in which study participants modify or improve aspects of their behavior (exam performance) in response to their awareness of being observed. We have not controlled our analysis for this effect and sample size was likely to small to address this issue. Moreover, only a computer defined random selection of 215 out of all 1400 MKSAP 1 questions was administered per exam. To avoid skewing Control group (n = 31) mean MKSAP ® score Δ -7 (SD: 11). Tablet group (n = 24) mean MKSAP ® score Δ + 8 (SD: 11). The overall result is also reflected in the MKSAP ® median initial and final score change distribution by grouped by medical subject categories. Fig 3. https://doi.org/10.1371/journal.pone.0172827.g002 Tablet PCs in medicine of the selection their category distribution was maintained. Still, the MKSAP 1 edition we used was designed for residents and could have potentially overwhelmed some of the participating final year medical students.
The impact of computer enhanced education on board style exams has been studied before. One group compared scores on preceptor evaluations with National Board of Medical Examiners (NBME) Subject Exam, and a standardized patient (SP)-based exam to complete assigned web cases versus students not completing the assignment. The authors controlled for prior academic performance and clerkship timing using US Medical Licensing Exam (USMLE) Step 1 scores and rotation order. They reported that students completing the web case assignment scored higher on the NBME subject exam and the SP-based exam [25]. Another study examined the impact of a computer-based program where residents receive a score on a Likert-type  Tablet PCs in medicine scale from an attending for each precept based on their knowledge base. The authors found a significant correlation between the resident's Likert scale scores and their American Board of Family Medicine In-Training Exam scores [26]. Judging from a study in emergency medicine it appears that positive impact of computers is probably independent of the exam style (computerized vs. oral). The authors observed no differences between virtual and traditional groups on critical action scores or scores on eight competency categories [27]. The use of multimedia materials was also studied in dental students, who often have difficulty understanding the importance of basic science classes, such as physiology, for their future careers. The authors reported a significant improvement in unit exam scores [28] Exam performance without a practical clinical skill level assessment does not automatically translate into superior performance in residency or fellowship programs [29]. The utility of educational games (although our system was not programmed as a game) as a teaching strategy for healthcare professionals remains undetermined according to a recent Cochrane analysis [30]. Moreover, different types of physicians have different needs and preferences for evidence-based resources and handheld devices [31]. Another aspect we could not address in our study was demonstrating the link to improved patient outcomes [32][33][34].
In summary, our study provides evidence, that tablet computer based integrated training and clinical practice enhances medical education and exam performance. Larger, multicenter trials are required to independently validate our data. Residency and fellowship directors are encouraged to consider adding computer devices, multimedia content and introduce blended learning to their respective training programs.
Supporting information S1 File. Raw data. This file contains the study raw data, except for any potentially personally identifying information to meet German Federal privacy legislation requirements. (SAV)