Introduction

The omnipresence of the internet has an increasing impact on higher education and on the way students access information for learning. The current generation of undergraduate students grew up in an environment in which the internet and computer-based technologies have become an integral part of life. This cohort, often referred to as “Net Generation” or “Digital Natives” is technological savvy and familiar with the world wide web as a communication platform and source of information [1]. In addition, studies have suggested differences in learning behavior and learning preferences of “Digital Natives” in comparison to preceding student generations. For instance, Oblinger and colleagues proposed that today's students are more comfortable with multimedia learning environments, prefer to be actively engaged rather than being passive consumers of information and favor immediate responses and topic-related discussions with their peers and educators, both in person and online [2].

In parallel, the world wide web has advanced from a network of static web sites to the more user-centered “Web 2.0” that supports active participation, dynamic interaction and collaborative approaches [3]. One key element of the “Web 2.0” are wikis (a term originating from the Hawaiian word for “quick”) which are web applications that allow users to collaboratively create, edit and share content. Wikis have little implicit structure and thus allow content to emerge according to the needs of the users. The most frequented wiki of the internet is the online encyclopedia Wikipedia (www.wikipedia.org). The English language version of Wikipedia (www.en.wikipedia.org) currently consists of 4.54 million articles (yearly change +8.0% or 362,933 articles) written by approx. 40,000 registered editors. Each month, the website has more than 9.5 billion page views and 2.9 million article revisions (http://stats.wikimedia.org; accessed 18. April 2014). The smaller German language edition of Wikipedia (www.de.wikipedia.org) consists of approx. 1.67 million articles (yearly change +8.0% or 133,526 articles) created by approx. 7,900 registered editors. It has approx. 1.27 billion page views and 0.6 million article revisions per month (http://stats.wikimedia.org; accessed 18. April 2014). Since its launch in 2001, Wikipedia has become one of the most visited websites and a frequently accessed source of medical information [4]. Recent studies estimated that Wikipedia is used by up to 70% of junior physicians and practitioners as information source for medical care [5]. Moreover, Wikipedia has been shown to be widely used by undergraduate medical students [6], [7].

In contrast to traditional encyclopedias that are written and edited exclusively by professionals, Wikipedia allows any user to edit content and create articles. This lack of a formal editorial review and the heterogeneous expertise of contributors often results in skepticism by both educators and students whether Wikipedia should be recommended as an information source in undergraduate medical education [8].

While a number of empirical studies has evaluated the medical content of Wikipedia (reviewed in [9]), only few researchers have addressed the quality of pharmacological information. Of these, research has focused on a limited set of drugs or on information for patients and health practitioners [10], [11]. Whether Wikipedia is a suitable source of pharmacological information for students and educators in medical education has not been investigated.

In this study we provide a comprehensive analysis of accuracy and completeness of drug information in Wikipedia in comparison to standard textbooks of basic pharmacology. In addition, references, revision history and readability of the corresponding drug articles in Wikipedia were analyzed to evaluate their potential as a learning resource in undergraduate medical education.

Methods

Study design

One hundred drugs were randomly selected from the list of 300 curricular drugs (www.tum300.de) taught in a basic pharmacology course for undergraduate medical students in Munich (see table S1 for the list of selected drugs). The TUM300 drug list is based on the World Health Organization Model Lists of Essential Medicines and adjusted to include all major pharmacotherapeutic substance classes and drugs that are most commonly prescribed in Germany [12]. For each drug, information on indication (IND), mechanism of action (MA), adverse effects (AE), pharmacokinetics (PK) and contraindications (CI) was excerpted from two German standard textbooks of basic pharmacology [13], [14]. Information on drug-drug interactions was included in the “pharmacokinetics” category. Only information present in both textbooks (data overlap) was included in the study and served as point of reference for the analysis of the respective drug articles in the German language version of Wikipedia (www.de.wikipedia.org; last accessed on March 20th 2013) (Figure 1). An analogous approach was used to compare the data overlap of 50 drugs (see table S1) retrieved from two standard Anglo-American textbooks [15], [16] with the corresponding drug articles in the English language version of Wikipedia (www.en.wikipedia.org; last accessed on June 20th 2014).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Study design.

Pharmacological information of randomly selected curricular drugs was retrieved form two standard textbooks of general pharmacology for five different categories (indication, mechanism of action, adverse effects, pharmacokinetics and contraindications). The textbook data overlap served as point of reference for the corresponding drug articles in Wikipedia. The analysis was performed separately for the German and English language version of Wikipedia and textbooks.

https://doi.org/10.1371/journal.pone.0106930.g001

Results

Accuracy and Completeness

To investigate the accuracy of pharmacological information in Wikipedia, the percentage of correct pharmacological statements in Wikipedia was calculated in comparison to the textbook overlap (Figure 1). The textbook overlap was 55%±12% for the German textbooks and 74%±14% for the English textbooks. Accuracy of pharmacological information was in the range between 99.6 and 100% in the five categories evaluated for the German language version of Wikipedia, with an overall mean score of 99.7%±0.17% (Figure 2A, Table S2). Similar results were obtained with the English language version of Wikipedia (Figure 2A, Table S2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Accuracy and completeness of drug information in Wikipedia.

Analysis of (A) accuracy and (B) completeness of the German (left panel) and English (right panel) language Wikipedia in comparison to textbooks. Accuracy was defined as the degree of closeness to the pharmacological information of the textbooks overlap. Completeness was defined as percentage of pharmacological statements of the textbook overlap present in the respective Wikipedia articles. Box plots showing median, first and third quartile with whiskers depicting the 5% and 95% percentile. In (A), statistical outliers are shown as black dots. Note that some box plots appear as vertical lines due to high accuracy scores and low variability of data. IND, indication; MA, mechanism of action; AE, adverse effects; PK, pharmacokinetics; CI, contraindications.

https://doi.org/10.1371/journal.pone.0106930.g002

Completeness of drug information in Wikipedia was quantified as percentage of the textbook overlap present in the respective Wikipedia drug articles. The overall completeness of drug information in the German language version of Wikipedia was 83.8%±1.5% (p<0.001) (Figure 2B) when compared to the textbook data overlap. Completeness varied in-between categories, and was lowest in the category “pharmacokinetics” (68.0%±4.2%; p<0.001) and highest in the category “indication” (91.3%±2.0%; ns) when compared to the textbook data overlap. Similar results were obtained for the English language version of Wikipedia with an overall completeness score of 93.1%±0.01% (Figure 2B). The most incomplete category was “pharmacokinetics” (87.2%±0.03%) when compared to the English textbook data overlap. It should be noted that the degree of overlap between the reference textbooks may affect the estimate of accuracy and completeness of drug information in Wikipedia. To minimize this effect, we used textbooks of comparable size and scope in the present study.

To assess if the information present in the textbook overlap but missing in Wikipedia (16.2% or a total of 224 pharmacological statements) was didactically relevant for undergraduate teaching of pharmacology, a qualitative re-evaluation was performed by six pharmacology lecturers of five different universities in Germany. Of the 224 statements evaluated, 84 (37.5%) were rated as relevant and 140 (62.5%) as non-relevant for a basic pharmacology course for medical students (Table S3). Thus, the adjusted overall completeness of pharmacological information for teaching in Wikipedia amounts to 93.9%.

Collectively, these results indicate that the drug information in Wikipedia is accurate and presents the majority (>90%) of curricular information essential for undergraduate medical education of pharmacology.

References

The average number of references per article in the German language version of Wikipedia was 14.6 (±1.6), with a minimum of 2 and the maximum of 104 references per article (Table S4). Most references were cited from reliable sources such as peer-reviewed academic publications (24.5%±0.02%), textbooks (22.9%±0.02%) or scientific databases and prescribing information (36.4%±0.02%). Non-peer reviewed or potentially biased information sources such as news media (online or print) or pharmaceutical industry accounted on average for only 3.2%±0.008% and 0.01%±0.005% of article references, respectively. Figure 3 depicts the summative analysis of references of drug articles investigated. Similar results were obtained for references of the English language version of Wikipedia.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Analysis of article references in Wikipedia.

(A) German language version of Wikipedia (n = 100), (B) English language version of Wikipedia (n = 20). Box plots depict the 5–95% percentile.

https://doi.org/10.1371/journal.pone.0106930.g003

Readability

An important aspect for the comprehension of learning materials is their readability, which can be quantified by indices that assess sentence composition, complexity and length. Readability was assessed for Wikipedia and the textbooks using the Amstad readability index [18], a variant of the Flesch Reading Ease adapted for the evaluation of German texts, and the Wiener Sachtextformel 1 [19]. Overall, no significant difference in readability was found between Wikipedia and textbooks (R_Amstad: 7.1±1.7 vs. 7.4±1.8, p = 0.9; R_{1. WSTF}: 15.4±0.5 vs.14.5±0.2, p = 0.07; Figure 4, Table S5). Both media had readability scores indicative of difficult-to-read texts that require tertiary levels of education.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Analysis of readability.

Readability was assessed using the Amstad readability index (R_Amstad) and the Erste Wiener Sachtextformel (R_{1. WSTF}). The scale of R_Amstad is from 0 (very difficult) to 100 (easy to read), for R_{1. WSTF} from 4 (easy to read) to >12 (very difficult). n = 10.

https://doi.org/10.1371/journal.pone.0106930.g004

Frequency of revisions of Wikipedia articles

In contrast to traditional encyclopedias or textbooks that are written and edited by professionals, Wikipedia allows any user to create and edit content. Drug articles in the German language version of Wikipedia had on average 262.8±37.4 edits performed by an average of 142.7±17.6 editors (Table 1 and S6). The majority of editors were registered users of Wikipedia. In addition, Wikipedia articles were maintained by automated or semi-automated scripts (bot editors) that perform simple or structurally repetitive task e.g. removing outdated links or detecting vandalism. Bot edits accounted for approx. 7% of edits of the articles investigated. Of note, articles of general public interest (e.g. on the psychostimulant drug methylphenidate for the treatment of attention-deficit hyperactivity disorder) had significantly more editors and edits (637 and 1418, respectively).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Descriptive statistics of edits and editors of drug articles in Wikipedia.

https://doi.org/10.1371/journal.pone.0106930.t001

Discussion

Since its launch in 2001 Wikipedia has become the largest collaborative online encyclopedia and is widely used by both students and educators in undergraduate medical education [4]. In this article we investigated the accuracy and completeness of drug information in Wikipedia in comparison to standard textbooks of basic pharmacology and evaluated its potential for undergraduate medical education of pharmacology. By analyzing 100 randomly selected curricular drugs in the German language version of Wikipedia, we found that the information in Wikipedia contained few factual errors (<0.3%) and comprised the majority (>93%) of curricular drug information considered essential for undergraduate teaching of basic pharmacology. The drug articles in Wikipedia had an average of 262.8±37.4 edits that were performed by an average of 142.7±17.6 editors. Finally, Wikipedia was generally well referenced with an average of 14.6±1.6 citations per article and had a readability that was on par with standard textbooks of pharmacology. Similar results were obtained for the English language version of Wikipedia. Collectively, our results suggest that Wikipedia is an accurate and informative source of drug information for undergraduate medical students.

As a collaborative encyclopedia with an open editorial policy, the reliability and accuracy of information of Wikipedia has frequently been the focus of intense debate (reviewed in [9]). An example is the single-blinded study by the journal Nature that compared the accuracy of 42 articles on various scientific topics in Wikipedia and Encyclopedia Britannica [20]. The authors identified a total of four severe errors in each of the encyclopedias. Subsequently, a number of empirical studies have assessed the quality of medical information in Wikipedia pertinent to surgical procedures [21], gastrointestinal conditions [22], cancer types [23] and pathology informatics [24]. Despite different methodologies, the main conclusion of these studies was that Wikipedia articles on health topics contain few errors and are well referenced, while the information provided often lacks depth.

Few studies have investigated the quality of pharmacological information in Wikipedia. Clauson et al. compared Wikipedia to the online database Medscape drug reference [10]. By using a set of predetermined questions, the authors found that Medscape provided answers to 82.5%, while Wikipedia could answer only 40%. In particular, Wikipedia articles were missing information on drug dosages, interactions and contraindications. However, it should be noted that the Wikipedia style manual for drug articles discourages detailed dosage information (http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Pharmacology/Style_guide).

Kupferberg and co-worker analyzed the accuracy and completeness of pharmacological information of five different statin drugs [11]. The authors concluded that the drug information is less complete, inconsistent and cannot be recommended as sole information source. It is important to note that the aim of the studies by Clauson et al. and Kupferberg and Protus was to evaluate drug information for patients or health practitioners. In contrast, the present study aimed to analyze drug information in Wikipedia from an educators' perspective and therefore used standard textbooks of pharmacology as point of reference. Pharmacological information on drugs for educational purposes differs in focus and detail of pharmacodynamic and -kinetic content. Moreover, Wikipedia is evolving rapidly. Re-evaluating the drug entries assessed by Kupferberg and Protus in 2010, we found the quality of pharmacological information significantly improved (data not shown). Therefore, the major conclusion of the present study that Wikipedia is an accurate and informative source of drug information for undergraduate medical students, is not incompatible with previously reported findings.

The ease of updating and editing information is a potential strength of Wikipedia, in particular in rapidly evolving disciplines such as pharmacology. However, this may also bear the risk of errors, vandalism or manipulation. Wikipedia has implemented several safety measures and quality surveillance mechanisms to identify and repair spurious information. These include watch lists for recent changes by volunteer editors, page protection for articles that are likely to attract controversy or vandalism, or automated computer scripts (bots) that detect and repair copyright violations, simple vandalism, or spelling and grammatical errors (reviewed in [9].

The effectiveness of these quality surveillance measures has been evaluated in a study by Viegas and co-worker who analyzed the repair dynamics of substantial rewrites of vandalism in Wikipedia [25]. The authors showed that the majority of such acts were repaired within minutes by the Wikipedia community. Another study reported that 42% of damaged articles were repaired within one viewing, while 11% were still present after 100 viewings [26]. These results suggest that while the quality surveillance is remarkably effective, a residual risk remains. A middle way between a closed expert database and an open editorial system may be platforms such as TUM300 (http://www.tum300.de) that allow anonymous user feedback, yet require editorial approval of edits prior to publication.

There are several limitations to this study. In lack of a “golden standard” for the accuracy and comprehensiveness of pharmacological information for undergraduate medical education, we used well established pharmacology textbooks as the point of reference. However, these resources have not been formally evaluated for these criteria. In addition, we analyzed a limited number of drugs in the German and English language version of Wikipedia, and accuracy and completeness of pharmacological information may differ for other drugs. The present study is therefore exploratory in nature and serves as basis for future confirmatory studies with larger sample sizes. It is important to note that pharmacology training of undergraduate medical students is not limited to pharmacodynamic and –kinetic characteristics of drugs. It is not the intention of this study to evaluate the potential of Wikipedia to substitute other learning media, in particular textbooks. Finally, the results of this study cannot be directly extrapolated to other areas of pharmacology (e.g. clinical pharmacology) or to other medical disciplines.

Despite these limitations our results underscore that the collaborative and participatory design of Wikipedia does generate high quality information on pharmacology that is suitable for undergraduate medical education. From a didactic point of view, the possibility to actively participate in editing and contributing to information resources is likely to meet the expectations and preferences of today's student generation.

Supporting Information

Acknowledgments

The authors thank Erik Zachte of Wikimedia foundation for providing statistical data on the number of article editors and revisions in the German Wikipedia. We thank Harald Mückter (Ludwig-Maximilians-University, Munich, Germany) and Christian Mang (Johannes Gutenberg University Mainz, Mainz, Germany) for analyzing of data and critical reading of the manuscript. We are grateful to Alexander Hapfelmeier (Institute for Medical Informatics and Epidemiology, Technische Universität München, Munich, Germany) for expert statistical advice.