https://orcid.org/0000-0001-6655-6843
Figures
Abstract
Background
Detection and management of neglected tropical diseases such as cutaneous leishmaniasis present unmet challenges stemming from their prevalence in remote, rural, resource constrained areas having limited access to health services. These challenges are frequently compounded by armed conflict or illicit extractive industries. The use of mobile health technologies has shown promise in such settings, yet data on outcomes in the field remain scarce.
Methods
We adapted a validated prediction rule for the presumptive diagnosis of CL to create a mobile application for use by community health volunteers. We used human-centered design practices and agile development for app iteration. We tested the application in three rural areas where cutaneous leishmaniasis is endemic and an urban setting where patients seek medical attention in the municipality of Tumaco, Colombia. The application was assessed for usability, sensitivity and inter-rater reliability (kappa) when used by community health volunteers (CHV), health workers and a general practitioner, study physician.
Results
The application was readily used and understood. Among 122 screened cases with cutaneous ulcers, sensitivity to detect parasitologically proven CL was >95%. The proportion of participants with parasitologically confirmed CL was high (88%), precluding evaluation of specificity, and driving a high level of crude agreement between the app and parasitological diagnosis. The chance-adjusted agreement (kappa) varied across the components of the risk score. Time to diagnosis was reduced significantly, from 8 to 4 weeks on average when CHV conducted active case detection using the application, compared to passive case detection by health facility-based personnel.
Conclusions
Translating a validated prediction rule to a mHealth technology has shown the potential to improve the capacity of community health workers and healthcare personnel to provide opportune care, and access to health services for underserved populations. These findings support the use of mHealth tools for NTD research and healthcare.
Author summary
Cutaneous leishmaniasis (CL) and other tropical diseases primarily affect people in developing and underserved parts of the world where healthcare infrastructure, resources and trained personnel are scarce. Smartphones, connected to the internet and running innovative health-applications—"apps"—have shown promise to bridge gaps in access to healthcare in even the most remote areas. Using participatory and human centered-design methods, we developed a mobile application for Android smartphones that supports community health workers to screen potential cases of CL in rural settings. We developed this application using a previously validated clinical prediction rule and tested it with patients, community workers and health workers in the municipality of Tumaco on the Pacific Coast of Colombia. Results from this study showed that the mobile app was easy to use, could detect cutaneous leishmaniasis in over 95% of cases, and reduced the time to diagnosis by half compared to passive case detection dependent on consultation at referral facilities. Overall, this work showed the value of designing health interventions with early engagement of all users, and supports the general use of mobile health applications for neglected tropical disease research and care.
Citation: Rubiano L, Alexander NDE, Castillo RM, Martínez ÁJ, García Luna JA, Arango JD, et al. (2021) Adaptation and performance of a mobile application for early detection of cutaneous leishmaniasis. PLoS Negl Trop Dis 15(2): e0008989. https://doi.org/10.1371/journal.pntd.0008989
Editor: Johan van Griensven, Institute of Tropical Medicine, BELGIUM
Received: July 10, 2020; Accepted: November 17, 2020; Published: February 11, 2021
Copyright: © 2021 Rubiano et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data are available at the following URL: https://osf.io/w5zn2/ S1 Software: Guaral/Leishmaniasis app. The most recent version of the software can be downloaded from the Google Play Store by searching “Leishmanisis” or directly at: https://play.google.com/store/apps/details?id=guaralrpctwo.cideim.guaral.guaralrpc.
Funding: This research was supported by the US National Institutes of Health Fogarty International Center (www.fic.nih.gov) grants R21 TW009907 A1, awarded to EAS and NGS, and D43 TW006589 awarded to NGS; COLCIENCIAS contract number 222956934763, awarded to LR; and the Special Programme for Research and Training in Tropical Diseases (TDR), hosted at the World Health Organization, Challenge Grant 2013/386736-0, awarded to NGS. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Cutaneous leishmaniasis (CL) remains a significant public health concern in Colombia due to its high endemicity and prevalence in resource-limited and remote settings, where diagnosis, treatment and monitoring is particularly challenging. In the Americas, Colombia reports the second highest incidence of CL, between 9.99 and 34.17 cases per 100,000 inhabitants over recent years [1]. However, this reported incidence is based on passive surveillance of cases consulting health care institutions, leading to under-ascertainment, -diagnosis and -reporting, which hinder control and the provision of diagnostic and treatment resources in areas of transmission [2].
Diagnosis of CL relies on microscopy or culture of lesion samples, procedures that require highly trained laboratory personnel and/or specialized laboratory equipment, to which there is limited access in areas where transmission occurs [3]. Moreover, many individuals with CL do not seek medical attention because the lesions are painless or because they prefer traditional treatments. Delay or failure to diagnose result in poor outcomes with higher disability, disfigurement and risk of mucosal involvement [4], as well as potentially contributing to ongoing transmission. Access to diagnosis in remote rural areas with limited infrastructure is a priority of control programs yet affordable means of achieving access are not yet available.
Smartphone-based mHealth strategies have recently been developed and evaluated with promising results for surveillance, diagnosis and outcome evaluation of infectious diseases in low-resource settings [5]. Providing mHealth tools to guide community health volunteers (CHV) and health workers (HW) in the presumptive diagnosis of CL in remote areas and their referral for etiologic diagnosis and treatment, could begin to overcome some of the challenges for the control of this disease. Here we describe the user-centered development and field evaluation of a smartphone application that adapted a validated CL prediction rule [3], to provide early, sensitive, case detection and referral for confirmatory diagnosis and treatment in Colombia. In addition to developing the app, this study sought to evaluate its diagnostic performance and usability for detection of CL; and to measure the benefit of its use by CHV, compared to passive case detection, on time to CL diagnosis.
Methods
Ethics statement
This study was reviewed, approved and monitored by the institutional ethical review boards of the International Center for Training and Medical Research (CIDEIM) (reference number 1229) and UC San Diego (reference number 160040). During field data collection, the purpose of the study was explained to potential participants. Written consent of participants was obtained from app users, and adults with suggestive lesions. Legal guardians of minors (aged less than 18 years) provided written informed consent. Those minors who were aged at least 7 years provided written informed assent. Cases were sought only in areas considered safe by the CHVs.
Technology development
We adapted a previously validated clinical prediction rule for CL (Table 1) [3] as a mobile application (app) to support screening for presumptive diagnosis of CL by CHV and HW. The mobile app was created for the Android operating system considering its availability on relatively low-cost smartphones. The app was developed using Android Studio (developer.android.com) and tested using a low-cost, locally available smartphone (Motorola Moto G).
Software development followed principles of human-centered design [6] through an iterative process to achieve an efficient and user friendly app. The user interface was designed collaboratively through a participatory process that included the engineering team, study physicians having expertise in the diagnosis and management of leishmaniasis, and CHV [6]. The design sought to maximize graphic content with an intuitive structure, and to minimize typed data entry and potential inconsistencies. Security features of the app include strong encryption and two-factor authentication.
A second, web-based, application for researchers and physicians was designed to store and process data captured through the mobile application by each end-user (CHV and HW). This web application was developed using a services platform for the collection and management of mobile health data, denoted SND, or System for Neurological Diseases, based on its original purpose (sndweb.azurewebsites.net).
Study design
We performed a prospective study to evaluate the usability and performance of the mobile application for community-based presumptive diagnosis, and referral for confirmatory diagnosis and treatment.
Setting
Field evaluation of the mobile app was conducted in one urban area (town of Tumaco) and three rural areas (La Guayacana, Llorente and El Pinde districts) within the municipality of Tumaco (population of 187,084 inhabitants), located on the southwestern Colombian Pacific Coast. The municipality of Tumaco was selected because it has both urban and remote rural communities, and a high burden of CL, with over 300 cases reported per year from 2016 to 2019 [7–10]. Notably, the municipality has a history of internal armed conflict, and electric power is intermittent. The rural part of the municipality has 365 veredas (townships), of which only a small proportion have road access. For the rest, the main access is by canoe or launch. In particular, the three rural areas of the current study have limited transportation and access to parasitological diagnosis, which is required for authorization of treatment. Maps facilitated by the principal telecommunication providers show no 3G coverage outside of a few of the larger settlements within the municipality [11–13].
Most cases (approximately 90%) in the Pacific coast region of Colombia are caused by Leishmania panamensis with the remainder generally being L. braziliensis. Other species such as L. mexicana are rarely encountered. The proportion of these two species of the Vianna subgenus can vary by focus and sylvatic or peridomestic transmission.
Evaluators
Application users lived and worked within communities in the endemic areas of the municipality of Tumaco. The CHV are unpaid community leaders who voluntarily provide health support in rural areas. They include homemakers, teachers, a radio broadcaster and health promoters of the public health system. They provide information about the diagnosis and treatment of CL to members of the community who are either self-identified and seek assistance of the volunteer, or are identified/referred through community members or during gatherings such as social activities, community centers and services, and marketplace encounters as well as in the conduct of their daily activities.
Four CHV were selected from among the leaders of communities in La Guayacana, Llorente and El Pinde who had experience in community work and CL case surveillance. Other app users included two nursing assistants (auxiliares de enfermería) and one of the study physicians, who are permanent staff of the CIDEIM clinical facility and CL referral center in the urban center of Tumaco, and two nursing assistants employed by the local health department. CIDEIM (Centro Internacional de Entrenamiento e Investigaciones Médicas) is an independent research institution and led the project. Because of their differing experience in attending leishmaniasis patients, data from the CIDEIM nursing assistants are presented separately from those of the health department, henceforth denoted HW and NA, respectively.
Enrollment
CHVs employed the mobile application to evaluate individuals with cutaneous lesions by active case finding in their communities. HW and the attending study physician evaluated patients consulting the referral center using the mobile application. The data from the local health department nursing assistants are presented for usability and sensitivity but not for agreement due to the small number of case assessments (n = 5).
Inclusion criteria for patients
Patients of any gender or race, between 0 and 70 years of age, with at least one cutaneous lesion of less than six months duration, were eligible to participate, and following informed consent, were evaluated with the mobile application.
Implementation
CHVs and NAs received training on the clinical presentation of CL and all users received training on the conceptual basis of the clinical prediction rule, interpretation of the score obtained after evaluating each of the variables, handling of the smartphone, and use of the application. Refresher training, covering the same topics, was conducted six months after initial training, to review and reinforce use of the tool. Individualized training was also provided as needed. A smartphone (Motorola Moto G3) was provided to each user. CHVs and NAs were instructed to apply the prediction rule using the mobile application and refer all evaluated patients, independently of the score, to the CIDEIM clinic for further medical and parasitological evaluation. Although in standard use, only presumptively diagnosed cases would be referred, for the purposes of the current study, all patients were referred in order to fully evaluate the performance and operating characteristics of the mobile application. All patients evaluated by CHVs and NAs were also evaluated at the CIDEIM clinical facility by one or both HW and the study physician who used the mobile app prior to clinical and parasitological assessment.
At the end of the intervention, users were asked to complete an assessment questionnaire consisting of 11 statements on the use and acceptance of the app. The options for each response were displayed on a Likert scale of 1, 2, 3, 4, or 5.
Clinical assessment and management
Clinical history and physical examination were conducted by the study physician who also obtained samples for parasitological evaluation. Treatment and additional procedures (biopsies, laboratory tests) were conducted in accordance with the standard of care [14].
Parasitological assays
Microscopic evaluation for the presence of amastigotes of Leishmania spp in tissue smears, and culture of tissue aspirates in biphasic media, were performed as described elsewhere [15]. Diagnosis of CL was confirmed by visualization of amastigotes in tissue smears or by culture of Leishmania spp from tissue aspirates.
Statistical analysis
Data were extracted from the SND platform and clinical histories, and analyzed using R software version 3.6 [16]. Operating characteristics of the mobile application for presumptive diagnosis of CL were determined considering parasitological diagnosis as the gold standard, and were presented with their corresponding binomial confidence intervals, calculated using the Wilson method [17] (‘binom’ package in R). For concordance analysis of between-rater assessment using the application, Cohen’s kappa was used [18–20] (‘irr’ package in R). Kappa is a chance-adjusted measure of concordance, accounting for higher agreement for a characteristic with high prevalence. For concordance between ordered categories, such as the total score, quadratic weighting of differences was used [21]. Kappa was not calculated if two or more of the four cells were zero. The arbitrary, but widely used, nomenclature of Landis and Koch [20] was used to describe the strength of concordance based on the value of kappa. To designate a presumptive case using the application score, the previously validated threshold point was set at 7 (a score of ≥ 7 or < 7) [22]. The t test was used to compare mean differences in time from onset of symptoms and diagnostic assessment, between the different types of users of the application.
Results
Adaptation of the clinical prediction rule to a mobile application
The clinical prediction rule was iteratively adapted to the form of an application for mobile phones (S1 and S2 Movies and S1 Software (available from: https://play.google.com/store/apps/details?id=guaralrpctwo.cideim.guaral.guaralrpc)). The initial approach was to identify the main components of the application and to design an interaction model that allows the user to focus on the patients rather than on the tool.
After the user logs in with an identifier and password, and enters the basic identification data of the patient (Fig 1A-B), they reach the main screen, which schematically shows six color-coded icons (Fig 1C), each corresponding to one element of the prediction rule (Table 1). The iconography was chosen to facilitate navigation, and guide the question content of the prediction rule app (Fig 1D–1F). Users can tap on icons in any order to enter the corresponding information. Some icons also have multimedia guidance. For instance, the orange lesion icon (Fig 1D) asks about the presence of lesions with raised borders and shows a typical CL ulcer (Fig 1E). Users may indicate the site on the body where each lesion is located simply by touching the screen (Fig 1F). The scoring for this item, lesion site(s) (number 3 in Table 1), is relatively complex and the app obviates the need for manual calculation, which was previously found to be a main source of errors.
(A) Homepage requesting user’s name and ID. (B) Personal information for each patient. (C) Main screen with six icons representing the six items of the prediction rule. (D) Menu for Item 1, lesions. (E) Image for user guidance, showing a typical CL lesion. (F) Interface for item 3, allowing the user to select the exact lesion site(s).
The data flow and storage are shown in Fig 2. As necessary, data captured by the mobile application can be stored off-line (locally on the phone) and transmitted later to the SND platform, when connectivity is re-established.
Users capture the information in the field via the mobile application and synchronize data when a connection is available, via secure cloud storage. The data can be accessed and analyzed by study physicians or researchers via the desktop web application. Sources: the app screen images are created by the authors; other images are in the public domain, taken from publicdomainvectors.org (cloud) and icon-library.net (others).
Usability of the mobile application
Eight users (3 CHV, 2 HW, 1 physician and 2 local health department nursing assistants) completed a questionnaire on the usability of the app (Fig 3). The app scored favorably on all statements, with a median score of 1, 1.5 or 2, on a scale of 1–5, 1 being the most favorable.
The median (range) score per question are: Q1, 1 (range 1–1); Q2, 1 (1–2); Q3, 1 (1–5); Q4, 1.5 (1–3); Q5, 1 (1–5); Q6, 2 (1–5); Q7, 1 (1–2); Q8, 1 (1–2); Q9, 1 (1–2); Q10, 2 (1–4); Q11, 1 (1–2). *Indicates statements that were posed in terms of difficulty of use, rather than ease of use. Here they have been inverted, in order to facilitate comparability across all the statements.
Evaluation of patients with lesions suggestive of CL using the mobile app
Between February 2015 and March 2016, 122 participants with lesion(s) suggestive of CL were evaluated by the physician using the mobile app. Of these, 85 were also evaluated by HW1, 96 by HW2, and 15 by one of the four CHV. Demographic and clinical characteristics of the study participants are summarized in Table 2. Patients were predominantly male (81%), of African descent (50%), and between 15 and 44 years of age (73%). During the study period, a further four people were screened by the CHV with the app but were not included due to: lack of parasitological data (n = 2), reported duration of lesions of more than six months, which was an exclusion criterion (n = 1), and missing evidence of consent (n = 1).
Of the 122 participants, most (n = 107, 88%) had parasitologically confirmed CL (Table 2). Of those positive, all but one were positive on microscopy, with the exception being positive on culture. Microscopy was done for all those parasitologically positive, and culture on 22% (24/107). Among those who were parasitologically negative (n = 14, 11%), other diagnoses were clinically defined by the attending physician and included predominantly bacterial infections and vascular ulcers, as well as one case each of sporotrichosis and squamous cell carcinoma. Most had a single lesion (61% of 122) and a large majority (89%) presented ulcerated lesions, with 44% located on exposed areas like the arms. More than half of the patients reported having a lesion with >1 month duration.
Diagnostic performance
The sensitivity of the mobile app was very high (>95%) for all users (Table 3). The CHV results are aggregated because no potential CL patient was assessed by more than one of them. Specificity was not determined due to the small number of participating patients having a confirmed diagnosis of another etiology.
Inter-observer agreement of the mobile application
Overall, with the threshold score of seven for presumptive diagnosis of CL using the mobile application, high agreement was observed between the physician and the HW and CHV (Table 4). However, the concordance (kappa) was low. For example, comparing the assessment of HW1 with that of the physician, there were 80 agreements on being a presumptive case, but no agreements on being CL-negative, yielding a kappa of -0.02. The agreement for individual components of the score varied from ‘almost perfect’ [20] for the site of the lesions, to ‘slight’ or ‘poor’, for the item grouped lesions.
Impact of active community case finding and referral versus facility-based passive detection using the mobile app
Individuals identified and evaluated within their communities using the mobile app by CHV were diagnosed on average 4.4 weeks (n = 15, range 2–12) after initiation of symptoms, compared to 8.5 weeks (n = 102, range 2–21) for those directly consulting the CIDEIM referral center, where they were evaluated by the health personnel using the app. Hence the difference was 4.1 weeks, 95% confidence interval 1.3–6.8, p = 0.005.
Discussion
This study reports the development and evaluation of a mobile adaptation of a validated clinical prediction rule [3,6] for presumptive diagnosis of CL by CHV and health workers. Our findings join a growing corpus of literature demonstrating that mHealth systems using mobile telephones have the potential to be viable and cost-effective strategies to improve access to healthcare, opportune diagnosis and epidemiological surveillance for under-served populations [23]. The current study was carried out in a resource-constrained region with limited health care infrastructure and access to diagnosis and treatment of CL. Furthermore, ongoing conflict, a circumstance that is often associated with CL [24] is a contributing factor to under-reporting and under-ascertainment of this disease.
In this challenging context the training process of the CHVs was re-enforced by their frequent contact with the population at risk of CL. Their adoption of the mHealth tool was facilitated by their ready recognition of the risk factors evaluated (S1 Movie).
The app performed with high sensitivity (>95%) in detecting true CL cases, whether in the hands of a physician, health worker, or community health volunteers. The results of this study confirmed the discriminatory cutoff score of ≥ 7 for presumptive CL derived from the prior validation study. These findings support the feasibility of screening with the app as a first step to triage patients for the confirmatory parasitological diagnosis required to authorize and administer treatment. The app could economize the use of rapid diagnostic (confirmatory) tests by CHV when such tests become available. Importantly, besides facilitating active case detection, the app encourages and empowers community health volunteers through its interactive and intuitive format, and access to the support of health professionals.
Compared to the previous validation study that was based on a mechanical prediction rule device [3], a higher percentage of agreement was observed with the application for most of the prediction rule variables for HW and CHV when they were compared with the study physician. This improvement is likely due to the instructive information that is provided by the application, which facilitates the understanding and assessment of the variables. This improvement was generally observed both for variables related to the clinical characteristics of the lesion (ulcer with raised borders, and lesion site), and for variables related to clinical history such as the presence of risk activity when the infection was acquired and lesion developed. The exception was the variable “no grouped lesions”, which had a lower level of agreement with the application compared to the mechanical prediction rule. This is not surprising because it has been one of the most difficult variables to evaluate by non-medical personnel, since the development of the prediction rule in 1993 [22]. The adaptation that was made for the application evidently did not overcome the limitation that we have experienced with this variable. However, this contrasts with the results of the usability assessment (Fig 3), in which most of the user participants agreed about the ability of the application to identify grouped lesions. The characteristics of lesions may differ in other regions, e.g. in terms of raised borders and whether nodular or ulcerated, and the app could require adaptation in those terms to be effective elsewhere.
In this study we had limited ability to assess other operating characteristics of the app (in particular, specificity and negative predictive value), because few of the eligible cases were of an etiology other than CL. In this area, common differential diagnoses are vascular ulcers, squamous cell carcinomas and bacterial infections.
Inter-observer agreement—the percentage of assessments with the same result—was high, although this was again driven by the high proportion of true CL cases among the participants. We hope to address this limitation in further studies of the app which are currently underway. In areas of low prevalence, the specificity, and positive and negative predictive values of the app score are critical, and it is important to train users in epidemiological, clinical, diagnostic, dermatologic concepts of leishmaniasis, and differential diagnosis.
The point estimates of the chance-corrected concordance (kappa) for the individual components of the score ranged from very high to zero (for grouped lesions), and were generally lower than found in a previous study using a mechanical version of the prediction rule that included a higher proportion of etiologies different from CL [3], although the confidence intervals were generally wide. Another limitation of our study is that most of the data are from users with a higher level of education, and experience with CL, than the target population of users (i.e. CHV). A major reason for this was security concerns, since those perceived as social activists are often targeted by outlaw groups.
The availability of the app as a tool for active case detection in the hands of CHV in rural areas resulted in a significantly shorter interval between the onset of symptoms and diagnosis as compared to the current practice of passive detection through patient consultation of health facilities and health personnel in urban Tumaco. Active case detection in the community, together with referral, has the potential to promote prompt parasitological diagnosis and treatment as well as improve case ascertainment and reporting. Although the operating characteristics of the app require further evaluation, particularly in relation with dermal lesions of different etiologies, the mHealth app could become an integral part of an active community-based surveillance model in low resource areas, possibly involving schools.
The didactic components of the app contribute to its potential as a training resource for CL detection by health care personnel. Future increase in 3G coverage in endemic areas of Colombia anticipate a scenario where mHealth will be even more effective. In the meantime, portable antennas could improve access, while awareness could be enhanced via broadcast media such as radio. On the other hand, although each individual’s scores are necessary for a research study such as the current one, in routine use it may not be necessary to transmit the scores to a physician once they have been generated by the app.
The participation of community or primary health workers presumptively diagnosing cutaneous leishmaniasis based on clinical diagnostic or clinical-historical prediction models, has been evaluated in other settings [25,26]. Another study conducted in Colombia, in three municipalities of the Department of Santander, also found that CHW achieved good sensitivity when they sought a presumptive clinical diagnosis of CL, using a qualitative diagnostic Likert five level-scale [26].
Demonstration of the sensitivity and feasibility of surveillance for active case detection at the community level using mHealth technology not only supports its potential to promote opportune diagnosis and treatment, but also of determining the true magnitude of this neglected disease at the local and national levels in Colombia, as well as potentially in other South American countries. This knowledge would inform and support the allocation of resources for prevention and control.
Supporting information
S1 Movie. Guaral app overview movie, file name “Guaral app overview movie 270px.mp4”.
Overview movie of the overall study using the mobile application. A higher resolution version can be accessed at: https://www.youtube.com/watch?v=DDJ08yesYOM&feature=youtu.be.
https://doi.org/10.1371/journal.pntd.0008989.s001
(MP4)
S2 Movie. Guaral app tutorial video, file name “VIDEO TUTORIAL GUARAL APP INGLES SD.mp4”.
Tutorial video showing how to use the mobile application. A higher resolution version can be accessed at: https://www.youtube.com/watch?v=OjRMA2G92ik&index=9&list=UUYkYVZC4pGQDYPaYe73rRww.
https://doi.org/10.1371/journal.pntd.0008989.s002
(MP4)
Acknowledgments
We are grateful to Drs. Alejandra María del Castillo, Miguel Prieto and David Rebellón for their assistance in the collation of clinical data.
References
- 1. Herrera G, Teheran A, Pradilla I, Vera M, Ramirez JD. Geospatial-temporal distribution of Tegumentary Leishmaniasis in Colombia (2007–2016). PLoS Negl Trop Dis. 2018;12(4):e0006419. pmid:29624582.
- 2. Alvar J, Velez ID, Bern C, Herrero M, Desjeux P, Cano J, et al. Leishmaniasis Worldwide and Global Estimates of Its Incidence. PloS ONE. 2012;7(5). ARTN e35671 pmid:22693548
- 3. Rojas CA, Weigle KA, Barrera L, Collazos C. Surveillance and screening of American cutaneous leishmaniasis by Colombian primary health care workers using a clinical prediction rule. Trans R Soc Trop Med Hyg. 2002;96(4):405–10. pmid:12497977.
- 4.
Holmes KK, Bertozzi S, Bloom BR, Jha P. Major Infectious Diseases Third Edition ed. Washington, DC: World Bank; 2017.
- 5. Wood CS, Thomas MR, Budd J, Mashamba-Thompson TP, Herbst K, Pillay D, et al. Taking connected mobile-health diagnostics of infectious diseases to the field. Nature. 2019;566(7745):467–74. pmid:30814711.
- 6. Navarro A, Rubiano L, Arango JD, Rojas CA, Alexander N, Saravia NG, et al. Developing mobile health applications for neglected tropical disease research. PLoS Negl Trop Dis. 2018;12(11):e0006791. pmid:30383809.
- 7.
Torres Rodriguez GE. Informe del Evento Leishmaniasis, hasta el Periodo Epidemiológico XIII, Colombia, 2016. Bogotá: 2017.
- 8.
Agudelo Chivatá NJ. Informe del Evento Leishmaniasis, Colombia, 2017. Bogotá: 2018.
- 9.
Agudelo Chivatá NJ. Informe del Evento Leishmaniasis, Colombia, 2018. Bogotá: 2019.
- 10.
Agudelo Chivatá NJ. Informe de Evento Leishmaniasis Cutánea, Periodo Epidemiológico XIII. Colombia, 2019. Bogotá: 2019.
- 11.
Claro (Colombia). Cobertura Soluciones Móviles 2019 [10 February 2020]. https://www.claro.com.co/personas/soporte/mapas-de-cobertura/.
- 12.
Movistar (Colombia). Cobertura y Calidad 2019 [10 February 2020]. https://www.movistar.co/web/portal-col/atencion-cliente/cobertura-tecnologia/.
- 13.
Tigo (Colombia). Áreas de cobertura de los servicios en mapa interactivo en WEB 2019 [10 February 2020]. https://www.tigo.com.co/regimen-de-proteccion-al-usuario/mapa-cobertura.
- 14.
Ministerio de Salud y Protección Social. Guía de Atención Clínica Integral del Paciente con Leishmaniasis. Bogotá: 2010.
- 15. Weigle KA, de Davalos M, Heredia P, Molineros R, Saravia NG, D’Alessandro A. Diagnosis of cutaneous and mucocutaneous leishmaniasis in Colombia: a comparison of seven methods. Am J Trop Med Hyg. 1987;36(3):489–96. pmid:2437815.
- 16.
R Core Team. R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2019.
- 17. Agresti A, Coull BA. Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician. 1998;52(2):119–26.
- 18. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37–46.
- 19. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors of kappa and weighted kappa. Psychological bulletin. 1969;72(5):323.
- 20. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. pmid:843571
- 21. Schuster C. A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales. Educational and Psychological Measurement. 2004;64(2):243–53.
- 22. Weigle KA, Escobar M, Arias AL, Martinez F, Rojas C. A clinical prediction rule for American cutaneous leishmaniasis in Colombia. Int J Epidemiol. 1993;22(3):548–58. pmid:8359974.
- 23. Rajput ZA, Mbugua S, Amadi D, Chepngeno V, Saleem JJ, Anokwa Y, et al. Evaluation of an Android-based mHealth system for population surveillance in developing countries. J Am Med Inform Assoc. 2012;19(4):655–9. Epub 2012/03/01. pmid:22366295.
- 24. Berry I, Berrang-Ford L. Leishmaniasis, conflict, and political terror: A spatio-temporal analysis. Soc Sci Med. 2016;167:140–9. pmid:27194448.
- 25. Siriwardana HV, Senarath U, Chandrawansa PH, Karunaweera ND. Use of a clinical tool for screening and diagnosis of cutaneous leishmaniasis in Sri Lanka. Pathogens and global health. 2015;109(4):174–83. Epub 2015/07/18. pmid:26184581.
- 26. Sánchez J, Orozco L, Buendía J, Muñoz G. Validez del diagnóstico presuntivo de leishmaniasis cutánea realizado por mediadores comunitarios en Colombia. Rev Panam Salud Publica. 2007;21(6):335–44.