https://orcid.org/0000-0001-5142-6122
Figures
Abstract
The use of substandard and counterfeit medicines (SCM) leads to significant health and economic consequences, like treatment failure, rise of antimicrobial resistance, extra expenditures of individuals or households and serious adverse drug reactions including death. Our objective was to systematically search, identify and compare relevant available mobile applications (apps) for smartphones and tablets, which use could potentially affect clinical and public health outcomes. We carried out a systematic review of the literature in January 2020, including major medical databases, and app stores. We used the validated Mobile App Rating Scale (MARS) to assess the quality of apps, (1 worst score, 3 acceptable score, and 5 best score). We planned to evaluate the accuracy of the mobile apps to detect SCM. We retrieved 335 references through medical databases and 42 from Apple, Google stores and Google Scholar. We finally included two studies of the medical database, 25 apps (eight from the App Store, eight from Google Play, eight from both stores, and one from Google Scholar), and 16 websites. We only found one report on the accuracy of a mobile apps detecting SCMs. Most apps use the imprint, color or shape for pill identification, and only a few offer pill detection through photographs or bar code. The MARS mean score for the apps was 3.17 (acceptable), with a maximum of 4.9 and a minimum of 1.1. The ‘functionality’ dimension resulted in the highest mean score (3.4), while the ‘engagement’ and ‘information’ dimensions showed the lowest one (3.0). In conclusion, we found a remarkable evidence gap about the accuracy of mobile apps in detecting SCMs. However, mobile apps could potentially be useful to screen for SCM by assessing the physical characteristics of pills, although this should still be assessed in properly designed research studies.
Citation: Ciapponi A, Donato M, Gülmezoglu AM, Alconada T, Bardach A (2021) Mobile apps for detecting falsified and substandard drugs: A systematic review. PLoS ONE 16(2): e0246061. https://doi.org/10.1371/journal.pone.0246061
Editor: Vijayaprakash Suppiah, University of South Australia, AUSTRALIA
Received: July 22, 2020; Accepted: January 12, 2021; Published: February 4, 2021
Copyright: © 2021 Ciapponi et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This work was funded by an independent grant from HRP (the UNDP/UNFPA/UNICEF/WHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction https://wwww.who.int/reproductivehealth/hrp/en/). The funder’s role was limited to developing the terms of references and reviewing the drafts. The funders had no role in the final content or decision to publish the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The World Health Organization (WHO) defines substandard or “out of specification” medicines as authorized medical products that fail to meet either their quality standards or specifications, or both [1]. Counterfeit or falsified medicines are products that deliberately or fraudulently misrepresent their identity, composition or source. Poor-quality medicines have important adverse health consequences, including the potential for treatment failure, the development of antimicrobial resistance, and serious adverse drug reactions, including death [2]. Apart from safety and effectiveness concerns, substandard and counterfeit medicines (SCMs) can carry economic costs such as the treatment of adverse events by the health system and resources wasted, leading to complications that are borne by consumers, facilities and third-payers. Yet, there are also potential indirect costs to consider, such as loss of productivity due to extra-days of illness, and reduced sales and tax revenues coming from regular medicines [3,4]. SCMs not only leads to public health and economic consequences, but it also weakens efforts, for example, to attain the United Nations Sustainable Development Goal related to achieving universal access to safe and effective care, including essential medicines [5].
Quality assurance of medicines represents a significant challenge for governments, regulators, and pharmaceutical companies at a global level [6,7]. That is why the only way to fight against this problem is a multifaceted approach with all the actors involved participating and targeting various levels of the pharmaceutical supply chain from developers to consumers [8–10]. With this purpose, a variety of technologies from mobile apps and handheld devices to sophisticated analytical chemistry methods, have been developed to detect SCMs [11]. Mobile health (mHealth) is a general term for the use of mobile phones and other wireless technology in medical care. Some software applications (apps) may be able to identify authorized medicines and discriminate against SCMs by detecting differences in several aspects, such as shape, color and others. These apps generally have a database of visual characteristics of currently authorized medicines and use a phone camera to compare it against the sample product to evaluate. Also, some technologies, mostly desktop applications, rely on the same database to detect drug inconsistencies [12,13].
The full extent of the SCM problem is largely unknown, and scientific research is variable and of poor methodological quality [14]. However, an increase in SCM is a growing global concern. If an inexpensive and widely available technology such as an app can help in the screening and possible detection of SCM, it could have significant potential use. Our objective was to systematically search, identify, synthesize and compare relevant mobile apps whose use could beneficially impact on public health decision-making.
Methods
We conducted a systematic review of the literature following the Cochrane methods and the PRISMA guidelines for reporting in S1 File [15,16]. Literature searches designed by a trained librarian were conducted in January 2020 in the Cochrane Database of Systematic Reviews (CDSR), Cochrane Central Register of Controlled Trials (CENTRAL), Database of Abstracts of Reviews of Effects (DARE), MEDLINE, EMBASE, Clinicaltrials.gov, the WHO International Clinical Trials Registry Platform (ICTRP), MedRxiv and SciFinder. The basic search strategy designed for Medline (PubMed) included the following terms: (Mobile Applications[Mesh] OR Mobile App*[tiab] OR Electronic App*[tiab] OR Portable App*[tiab] OR Software App*[tiab] OR Mobile Software[tiab] OR Portable Software[tiab] OR Mobile Based[tiab] OR Medical App*[tiab] OR Mobile Authentic*[tiab] OR Cell Phone[Mesh] OR Mobile Phone*[tiab] OR Cell Phone*[tiab] OR Mobile Telephon*[tiab] OR Cell Telephon*[tiab] OR Cellular Phone*[tiab] OR Cellular Telephon*[tiab] OR Smartphone*[tiab] OR QR[tiab] OR Computers, Handheld[Mesh] OR Handheld Device[tiab] OR iPad[tiab] OR Pill Identificat*[tiab]) AND (Counterfeit Drugs[Mesh] OR Counterfeit[tiab] OR Counterfeit[tiab] OR Fake[tiab] OR Adulterated[tiab] OR Imitation*[tiab] OR Fraudulent[tiab] OR spurious [tiab]). The search terms were modified to suit the requirements of particular databases as was detailed in S2 File. We also searched non-peer reviewed technical reports and other online information, including Apple’s App Store, Google Play Store, and a directed search in Google Scholar. We searched references of included articles and relevant literature reviews.
The strategy for Google Scholar included the following terms: Pill Identifier Tool Medication OR Medicine OR Drug OR Falsified OR Counterfeit OR Substandard OR Surveillance OR Authentication OR Mobile OR Software.
The systematic review protocol was registered in the PROSPERO (CRD42020163075) database in January 2020. Regarding primary eligibility criteria, we included apps claiming to analyze authorized medicines or SCMs identification in their description, aimed to directly detect and recognize solid oral medications, published in English, Spanish or Portuguese language, and available to download from the mentioned app stores. Mobile apps were excluded if they targeted non-human medications, identified illicit drugs, or examined herbals or Chinese medicines. After screening the results for each search database, the selected app names were added. If an app was listed in both stores we report them with a single value after consensus. We downloaded and installed the remaining apps as the first step to assess their eligibility for this review. Apps failing to launch in the test devices were excluded. All Apple test devices ran iPhone operating system (iOS, Apple Inc) 13.3, and all Android test devices ran Android 10.0. We also extracted general information and relevant secondary features that the apps offer, such as information provided about authorized medicines, security and privacy-related features, data sharing and social media and technical support.
All unique articles were independently assessed by two reviewers (MD and TA) based on title and abstract. Those marked for inclusion, or whose title and abstract were not sufficient to determine inclusion, were then reviewed using the full text. Data extraction and risk of bias (quality) assessment were also performed independently by these reviewers, with oversight from two senior reviewers (AC and AB). The risk of bias was planned to be assess by the Cochrane tool for Randomized Controlled Trials (RCTs) designs [17]. For other designs, including cross-sectional or cohorts, we used the NIH Study Quality Assessment Tools [18]. For content analysis, we used the Mobile App Rating Scale (MARS) that is a simple, objective, and reliable tool for classifying and assessing the quality of mobile apps [12]. The MARS app quality is a 19-item, expert-based rating scale to assess the quality of mobile apps. Each question from MARS uses a 5-point scale (1 = inadequate, 2 = poor, 3 = acceptable, 4 = good, and 5 = excellent). This expert scale consists of multiple dimensions that assess different quality aspects of apps, including end-user engagement features, aesthetics and content quality. Comprehensiveness and accuracy of the content and information of the app for the MARS are assessed on questions #15 and #16. We completed all phases of the study selection using COVIDENCE®, a web-based platform designed for the processing of systematic reviews [19]. Authors of articles were contacted to obtain missing or supplementary information when necessary.
A pre-designed general data extraction form was used after pilot testing. We resolved disagreements during all phases by consensus by the two initial reviewers (MD and TA) and, when necessary, a third reviewer (AC or AB) decided on them if a consensus was not reached. We extracted the following: general information about the study (publication type, year of publication, journal, authors’ names, and language), research location (geographical region, country, province, city, and setting) and study population (sample size, age at enrollment, living in rural or urban area under, and dates of initiation and ending of data collection). We performed descriptive analyses of the extracted data, and we structured data in tables, to describe the mobile apps for detecting SCMs which included a set of both general information and relevant secondary features of the apps, and which were available in the app stores.
Results
Our search strategy retrieved 335 references through the literature search in databases and 42 reports of technologies coming from other sources, such as Apple’s App Store, Google Play Store and Google search engines. We finally included 25 mobile apps, 16 websites that compare the given medication with large internal databases, one preprint study evaluating one mobile app and other study comparing four mobile apps (Fig 1), along with a detail of their main characteristics to facilitate comparisons. These reports referred to the usability and the ability of apps to correctly identify pills. However, did not find direct assessment of SF products in the real-world.
Considering the nature of the data, we could not perform a meta-analysis.
Table 1 and Fig 2 summarize the general characteristics of the reviewed apps and the app quality mean MARS scores available on platforms. Of the 25 apps, eight were from Apple’s App Store, eight from Google Play, eight were listed in both stores and one (MedSnap) on none of the platforms, but was identified through Google Scholar and MedRxiv. Twenty-two developers developed the 24 apps available on platforms, and 17 were currently available to detect pills without any cost. Seven were available after purchase or offered more options through a subscription. The last update dates for the Android apps were from February 2010 to March 2020, whereas the iOS apps were generally the most frequently updated, and dates ranged from October 2015 to March 2020. The average rating for the apps was between acceptable and good, with a mean score of 3.78, ranging from a minimum of 1 to a maximum of 5, but sometimes it was not available.
The size of the circles is proportional to the quintiles of MARS mean score, and their color identifies the platform source. The axes of the graph indicate the average user ratings and the number of raters.
Most apps claim to detect pills by evaluating the imprint, color or shape, and only a few offer the possibility of identification through a photograph or bar code. The app quality mean MARS score (Table 1) of the 25 apps was good with 3.17, with a maximum of 4.9 (excellent) for Medscape and IBM Micromedex Drug Reference, and a minimum of 1.1 (inadequate) for Drug Search App. The mean scores of the four dimensions of MARS were examined to investigate the magnitude of the differences in quality in each dimension (Table 1). The functionality dimension resulted in the highest mean score (3.4), whereas the engagement dimension and the information dimension showed the lowest average score (3.0). It should be noted that of the 25 apps, only 4 were verified by evidence in published scientific literature, these being MedSnap, Epocrates, Medscape and IBM Micromedex (Table 2) [20,21]. On the other hand, all of the websites (16) identified out of these platforms are intended for the general population (Table 3). Nevertheless, all the websites and available software use imprint, color or shape to identify the pills.
Relevant secondary features for each app available on platforms were extracted and shown in Fig 3. The overall number of apps with drug database access was 12 (out of 25), and 9 were free to access. Drug history tracking was available in only 6 apps where one has some cost, 16 apps allow sharing while 5 have some cost, and 7 apps require login and provide password protection with only 2 having any cost.
Our search found a cross-sectional study that intended to compare Epocrates, Medscape, IBM Micromedex and Google for clinical management information designed for healthcare professionals for a one-week period [20]. Medical students at an academic hospital in the United States used a score for satisfaction of search and user interface based on a 1 to 5 scale, with 1 representing lowest quality and 5 describing the highest quality. The study concluded that Medscape (Satisfaction 4.92) was the most preferred free mobile app evaluated due to its interactive and educational features, followed by IBM Micromedex and Epocrates (Satisfaction 4.58 and 4.42, respectively). This study was used to complete MARS question #19 (evidence base) about whether the app had been trialed. Further, we found a preprint study whit the objective to evaluate the sensibility and specificity of MedSnap, that is not available on the app stores. In the study [21], MedSnap models were created from trusted and authentic medications and tested against samples of authentic/trusted or falsified artesunate, artemether-lumefantrine, azithromycin, and ciprofloxacin. Results were 100% sensitivity and specificity to detect authentic and counterfeit drugs from 48 samples tested. This study allowed us to complete MARS question #19 and others parts of the questionnaire. It should be noted that this app is the only one where the accuracy to detect counterfeit drugs was formally evaluated through a study.
We used the NIH Study Quality Assessment Tools for the two included studies and in both cases a poor score was obtained, which means a high risk of bias (S3 File).
Discussion
We found 25 apps and 16 websites for pill identification potentially useful for detecting SCM. Despite the variety of available apps, there was only two scientific publications of observational studies considered as high risk of bias, that included three apps and one of them as a preprint report. This highlights the lack of studies evaluating apps and the great need to carry out rigorous studies that evaluate the functioning and usefulness of apps for pill identification and eventually to detect SCMs. MedSnap was the only app where the sensitivity and specificity to detect authentic and counterfeit drugs was assessed [21]. Most mobile apps were developed as pill identifiers, and yet developers state that the apps have the capacity to also detect SCMs (as an example of this would be the apps Medsnap and CheckFake) [22,23]. In real life these apps are mostly used to identify pills by elderly people or people with difficulties to recognize medications but not to identify SCMs. Although it cannot be considered a proxy of accuracy, we rated and compared the quality of the apps globally and by different domains. The global app quality mean MARS score from the 25 apps available on platforms was classified as acceptable (3.17) and it is similar to some review and patients experience studies for some mobile apps for others objectives [13,24–28]. Despite the aim of these apps, a low average score in the ‘information dimension’ (quantity and quality) was observed based on MARS. This is due partly because some of these apps do not seem to use a reliable or verifiable information source for medicines’ attributes. Nevertheless, for the evidence-based question inside the ‘information dimension’, we found only two low-quality studies which evaluated the performance of four highly recognized apps (Epocrates, IBM Micromedex Drug Reference, Medscape and MedSnap) [20,21]. Regarding the MARS questions #15 and #16, which assessed the accuracy and comprehensiveness of the content and information, only 6 of 25 apps had a MARS score higher than 4. This could be a concerning issue since 76% of the surveyed apps were not adequately backed up by any reliable information. As Kim et al, we also noted associations between the average user rating and the information dimension; this means that users may critically evaluate applications and these ratings can potentially be important tools for selecting apps [25].
Most of the identified mobile apps were aimed at consumers or general population, and just one was designed to be used by regulatory authorities, such as customs security personnel. Eight of the 25 apps (32%) were paid apps, and all the websites were open access. This proportion of paid apps was consistent with other studies that systematically reviewed the Google Play Store and Apple’s App Store [13,24,25]. Eight of the 24 apps available in the stores had no average score from the users, but had a very low MARS mean score in our study (from 1.6 to 3.7). Like Kim et al, we found a correlation between the worst scores from the public and low scores in our MARS aesthetics and engagement dimensions [25].
SCMs are a worldwide problem with direct consequences for human health. The problem is much more severe in low- and middle-income countries due to their relatively weak health systems and regulatory processes. In an analysis of 215 misoprostol samples, 55% were within specifications, 85 (40%) were below average in 90% of the labelled content [29]. Of the 85 samples, 14 contained no misoprostol at all. There is a tangible threat to global health security, for example, in antiparasitic drugs (chiefly antimalarials) and antibiotics, increasing transmission, morbidity, mortality, and resistance. This is also the case for medications affecting maternal, neonatal and child health, like misoprostol or mifepristone. In this regard, mobile apps could be a valuable and inexpensive tool for patient empowerment and they have the potential to detect these drugs and save lives at a low cost. However, the fight against falsified and substandard drugs needs a multi-pronged approach with all the stakeholders, and although technology alone cannot solve the problem, it can be an important tool for consumers [11,14]. Those countries with lower regulatory capacity are the most vulnerable to SCM medications. The goal is to include guarantee of good manufacturing practices, and to impose audit control measures on manufacturing companies and distribution of medications [30,31].
While MARS score is helpful to indicate user-friendliness, there is a significant gap in rigorous evaluation of these apps. The most important limitation is that we cannot extrapolate the identification of a pill by a device to the categorization of a product as counterfeit or substandard. To find out whether the product it really of poor quality, additional confirmatory tests will be necessary. Additionally, we do not know how consumers use these applications or what their performance looks like in real life, so this makes conclusions about their accuracy in the detection of substandard-quality drugs difficult. Moreover, the reproducibility of the search on app platforms was not standardized, but iteration using keywords and related apps reduce the probability of missing apps. Finally, mobile apps have frequent updates, and new apps are published on a daily basis.
Our systematic review highlighted an important evidence gap in diagnostic accuracy of mobile apps detecting SCMs, and there is a need for primary studies addressing this issue. A unified global effort to address the important problem of counterfeit and substandard drugs is necessary. Our findings suggest that, although there is no single technology that can meet all the desired requirements for detecting SCMs, mobile apps could constitute a potential valuable real-world tool available to large number of potential users to counter the serious consequences of the SCM problem and help achieve the United Nations’ Sustainable Development Goals.
Supporting information
S3 File. NIH study quality assessment tools for observational studies.
https://doi.org/10.1371/journal.pone.0246061.s003
(XLSX)
Acknowledgments
We would like to express our sincere thanks to our librarian Daniel Comandé, for leading the search strategy. In addition, we would like to thank Antonella Francheska Lavelanet for providing feedback and edits to this paper.
References
- 1.
World Health Organization. Definitions of Substandard and Falsified (SF) Medical Products. 2017 [cited 15 Mar 2020]. Available: https://www.who.int/medicines/regulation/ssffc/definitions/en/.
- 2. Kovacs S, Hawes SE, Maley SN, Mosites E, Wong L et al. Technologies for Detecting Falsified and Substandard Drugs in Low and Middle-Income Countries. PLoS One. 2014;9: e906. pmid:24671033
- 3.
World Health Organization. A Study on the Public Health and Socioeconomic Impact of Substandard and Falsified Medical Products. Geneva, Switzerland: World Health Organization. 2017. Available: https://www.who.int/medicines/regulation/ssffc/publications/se-study-sf/en/.
- 4. Fernandez FM, Hostetler D, Powell K, Kaur H, Green MD, Mildenhall DC, et al. Poor quality drugs: grand challenges in high throughput detection, countrywide sampling, and forensics in developing countries. Analyst. 2011;136: 3073–3082. pmid:21107455
- 5.
World Health Organization. Health in 2015: From MDGs, Millennium Development Goals to SDGs, Sustainable Development Goals. Geneva, Switzerland: World Health Organization. 2015.
- 6. Mackey TK, Nayyar G. A review of existing and emerging digital technologies to combat the global trade in fake medicines. Expert Opin Drug Saf. 2017;16: 587–602. pmid:28349715
- 7. Roth L, Nalim A, Turesson B, Krech L. Global landscape assessment of screening technologies for medicine quality assurance: stakeholder perceptions and practices from ten countries. Global Health. 2018;14: 43. pmid:29695278
- 8. Vickers S, Bernier M, Zambrzycki S, Fernandez FM, Newton PN, Caillet C. Field detection devices for screening the quality of medicines: a systematic review. BMJ Glob Heal. 2018;3: e000725. pmid:30233826
- 9. Cuomo RE, Mackey TK. An exploration of counterfeit medicine surveillance strategies guided by geospatial analysis: lessons learned from counterfeit Avastin detection in the US drug supply chain. BMJ Open. 2014;4: e006657. pmid:25468507
- 10. Rasheed H, Hollein L, Holzgrabe U. Future Information Technology Tools for Fighting Substandard and Falsified Medicines in Low- and Middle-Income Countries. Front Pharmacol. 2018;9: 995. pmid:30233373
- 11. Hamilton WL, Doyle C, Halliwell-Ewen M, Lambert G. Public health interventions to protect against falsified medicines: a systematic review of international, national and local policies. Health Policy Plan. 2016;31: 1448–1466. pmid:27311827
- 12. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps. JMIR mHealth uHealth. 2015;3: e27. pmid:25760773
- 13. Santo K, Richtering SS, Chalmers J, Thiagalingam A, Chow CK, Redfern J. Mobile Phone Apps to Improve Medication Adherence: A Systematic Stepwise Process to Identify High-Quality Apps. JMIR Mhealth Uhealth. 2016;4: e132. pmid:27913373
- 14. Ozawa S, Evans DR, Bessias S, Haynie DG, Yemeke TT, Laing SK, et al. Prevalence and Estimated Economic Burden of Substandard and Falsified Medicines in Low- and Middle-Income Countries: A Systematic Review and Meta-analysis. JAMA Netw open. 2018;1: e181662. pmid:30646106
- 15.
Higgins JPT, Thomas J, Chandler J, MS. C, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated August 2019). Cochrane, 2019. Cochrane, editor. 2019.
- 16. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339. pmid:19622552
- 17.
Higgins JPT, Savović J, Page MJ, Elbers RG SJ. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins J, Thomas J, Chandler J, et al, editor. Cochrane Handbook for Systematic Reviews of Interventions version 60 (updated July 2019). 2019 [cited 15 Mar 2020]. Available: www.training.cochrane.org/handbook.
- 18.
National Institutes of Health (NIH). “Study Quality Assessment Tools.” 2020 [cited 15 Mar 2020]. Available: www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools.
- 19.
Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. 2019.
- 20. Khalifian S, Markman T, Sampognaro P, Mitchell S, Weeks S, Dattilo J. Medical student appraisal: searching on smartphones. Appl Clin Inform. 2013;4: 53–60. pmid:23650487
- 21. Hymel PA, Brossette SE, Wong DY, Zheng N. An Evaluation of the MedSnap Medication Authentication System. medRxiv. 2020.
- 22.
Medsnap Verify Services. 2020 [cited 17 Dec 2020]. Available: https://www.medsnap.com/verify/.
- 23.
Google play. CheckFake (dlt.sg App for drug counterfeit). 2020 [cited 17 Dec 2020]. Available: https://play.google.com/store/apps/details?id=sg.dlt.checkfake&hl=es_419.
- 24. Bardus M, van Beurden SB, Smith JR, Abraham C. A review and content analysis of engagement, functionality, aesthetics, information quality, and change techniques in the most popular commercial apps for weight management. Int J Behav Nutr Phys Act. 2016;13: 35. pmid:26964880
- 25. Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer Mobile Apps for Potential Drug-Drug Interaction Check: Systematic Review and Content Analysis Using the Mobile App Rating Scale (MARS). JMIR mHealth uHealth. 2018;6: e74. pmid:29592848
- 26. Guo Y, Yang F, Hu F, Li W, Ruggiano N, Lee HY. Existing Mobile Phone Apps for Self-Care Management of People With Alzheimer Disease and Related Dementias: Systematic Analysis. JMIR aging. 2020;3: e15290. pmid:32012045
- 27. Woods LS, Duff J, Roehrer E, Walker K, Cummings E. Patients’ Experiences of Using a Consumer mHealth App for Self-Management of Heart Failure: Mixed-Methods Study. JMIR Hum factors. 2019;6: e13009. pmid:31045504
- 28. Adam A, Hellig JC, Perera M, Bolton D, Lawrentschuk N. “Prostate Cancer Risk Calculator” mobile applications (Apps): a systematic review and scoring using the validated user version of the Mobile Application Rating Scale (uMARS). World J Urol. 2018;36: 565–573. pmid:29222595
- 29.
World Health Organization. WHO Drug Information. Quality of medicines. 2016;30.
- 30.
World Health Organization. WHO Global Surveillance and Monitoring System for Substandardand Falsified Medical Products. Geneva, Switzerland: World Health Organization; 2017.
- 31. Fadlallah R, El-Jardali F, Annan F, Azzam H, Akl EA. Strategies and Systems-Level Interventions to Combat or Prevent Drug Counterfeiting: A Systematic Review of Evidence Beyond Effectiveness. Pharmaceut Med. 2016;30: 263–276. pmid:27752221