http://orcid.org/0000-0002-8295-382X
Figures
Abstract
EstomatoNet was created in the south of Brazil to provides specialist support over a web-based platform to primary care dentists for diagnosis of oral lesions. To evaluate the usability of EstomatoNet and to identify user perceptions regarding their expectations and difficulties with the system; and to compare the perceptions of regular users of the service to those of first-time users. Sixteen dentists were selected for the study: 8 were frequent users of EstomatoNet and 8 were residents who had never used the Platform. To assess usability, participants were required to request telediagnosis support for a fictional case provided by the research team. During the process of uploading the information and sending the request, users were asked to “think out loud,” expressing their perceptions. The session was observed by an examiner with remote access to the user's screen (via Skype). After the simulation, users completed the System Usability Scale (SyUS), a validated tool with scores ranging from 0 to 100. The mean SyUS score assigned by frequent users was 84.7±6.6, vs. 82.2±9.3 for residents (satisfactory usability: score above 68). The difference between the groups was not statistically significant (Student t test, P = .55). The residents group took longer (347.1±101.1s) to complete the task than frequent users (252.8±80.3s); however, the difference between the groups was not statistically significant (Student t test, P = .06). In their subjective evaluation, users suggested the inclusion of a field to add further information on outcomes and resolution of the case and changes in the position of the “Send” button to improve workflow. The present results indicate satisfactory usability of EstomatoNet. The Platform seems to meet the needs of users regardless of how experienced they are; nevertheless, a few minor changes in some steps would improve the tool.
Citation: Roxo-Gonçalves M, Trevizani Martins MA, Martins MD, Aita Schmitz CA, Dal Moro RG, D'Avila OP, et al. (2020) Perceived usability of a store and forward telehealth platform for diagnosis and management of oral mucosal lesions: A cross-sectional study. PLoS ONE 15(6): e0233572. https://doi.org/10.1371/journal.pone.0233572
Editor: Simone Borsci, Universiteit Twente, NETHERLANDS
Received: February 13, 2020; Accepted: May 7, 2020; Published: June 5, 2020
Copyright: © 2020 Roxo-Gonçalves et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data underlying this study are included as Supporting Information and have been uploaded to the Open Science Framework (DOI: 10.17605/OSF.IO/8ANG2).
Funding: This work supported by Fundação Instituto de Pesquisas Econômicas (FIPE) https://www.fipe.org.br/ and Hospital de Clinicas de Porto Alegre https://www.hcpa.edu.br/ with the Publication fees. No direct funding was received for this study. The cited funders had no role in study design, data collection, analysis, interpretation, or writing of the report. The first and last authors had full access to all data and had final responsibility for the decision to submit the manuscript for publication.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: PHC, primary health care; SyUS, System Usability Scale
Introduction
A fundamental principle of the Brazilian Unified Health System (SUS) is the constitutional guarantee of access to comprehensive, continuous health care, coordinated through networks, for all citizens. In the SUS, the primary care level is the entry point to the health care system [1–5] and should be capable of resolving most health conditions. Considering that assumption, oral health was introduced as an integral and inseparable component of the SUS in 2004, when the National Oral Health Policy was launched. The SUS oral health care network encompasses specialized dental care, with referrals coordinated by primary health care (PHC) services [6,7].
Telediagnosis as a tool to enhance access to specialized care
Even though face-to-face encounters between health professionals and patients remain the gold standard for patient evaluation in all medical fields, telediagnosis is a useful alternative when the availability of specialized care is limited [8]. Taking this into account, TelessaúdeRS-UFRGS, a major university-based telehealth program in Brazil [9], has created EstomatoNet, a teledentistry service available free of charge to PHC physicians and dentists in the state of Rio Grande do Sul to enhance care, prevent unnecessary referrals to specialists, and decrease the time between referral and specialty consultations [10]. Established in 2015, EstomatoNet is a web-based platform developed by TelessaúdeRS-UFRGS, and currently handles 3.7 thousand monthly requests for teleconsultations and telediagnosis support.
However, despite the advances and contributions of telemedicine, many health professionals still resist this technology. This might be explained, at least in part, by the difficulties involved in learning how to use these systems [11].
Usability in telehealth
Usability is defined as the ability to use a product for its intended purpose [12]. Assessment of usability allows difficulties to be identified and resolved, making sure that telehealth systems do in fact translate into benefits. Usability assessment encompasses features such as ease of learning, ease of retaining the know-how to repeat a task after some time, how fast tasks can be performed, low error rate, and subjective user satisfaction [13]. Thus, the primary aim of the present study was to evaluate the perceived usability of the EstomatoNet Platform. The secondary aim was to compare the perceptions of regular users of the service to those of health professionals who were using it for the first time.
Methods
Study design and recruitment of participants
This is a cross-sectional, observational study using a convenience sample. Among the 71 registered users of EstomatoNet, those 10 who used it most frequently were invited via e-mail to participate in the study. Those who agreed to participate signed an informed consent form and were included as the experienced user group. All were dentists working in the Brazilian Unified Health System. A control group of first-time users was selected among residents from the Integrated Residency Program in Oral Health at the UFRGS School of Dentistry. Residents were not familiar with the EstomatoNet Platform. The Universidade Federal do Rio Grande do Sul review board (GPPG 16–0440) approved this studie.
Study procedures
Initially, individual Skype videoconferences were scheduled with each participant. During the videoconference, the participant was asked by an examiner to activate the “screen sharing” tool to allow observation and analysis of the interaction with the Platform.
After screen sharing was activated, the participant was guided to access and read the tutorial on how to request telediagnosis support through a link to EstomatoNet made available at the TelessaúdeRS-UFRGS portal [10]. This procedure lasted approximately 10 minutes.
Outcomes—Evaluation of usability
Observation and “think out loud” protocol.
After that, the examiner (a professional familiar with the EstomatoNet workflow) shared a simulated case via Skype, including clinical data and photographs, for the Platform test. The same case was presented to all participants. During this simulation, the participants were instructed to “think out loud”, that is, to express their feelings and difficulties while performing the task. In usability research, this approach has been useful to identify problems in information systems [14,15, 16].
All simulations were recorded using a digital camera (Canon EOS Rebel T3, 12 megapixels, 18-55mm lens) to allow analysis of the participants’ actions and expressions and to record the time taken to conclude the procedure. Each video from each participant was analyzed twice by the examiner, to mitigate the possibility of details being overlooked.
System usability scale (SyUS).
Study outcomes included the duration of the interaction, the perceptions expressed by participants regarding the Platform, and usability of the system according to the System Usability Scale (SyUS) [17,18]. This validated questionnaire was composed of 10 items (Table 1) and is highly reliable [18–20] to evaluate usability in different systems [21,22]. The total SyUS score ranges from 0 to 100, with scores higher than 68 considered to be satisfactory [20]. The original language of the instrument was English; it was translated and cross-culturally validated for Portuguese [23].
Sample size and power calculation
Considering that the present study used a convenience sample, its power was calculated according to method proposed by Borsci et al. [24]. To estimate the amount of problems reached by the sample in question, the following equation was used: D = 1-(1-p)n, where p represents the raw p-value, n indicate the sample size and D, the percentage of problems reached by the sample. The weight of each problem was calculated as the sum of users that have detected it, while the count of the problems identified by each subject was used for calculating the raw p-value, and the means of calculating each individual’s p-value.
Results
Characteristics of the sample
Of the 10 invited regular EstomatoNet users, eight agreed to take part in the study (80% response rate). All invited residents agreed to participate.
The age of participants ranged from 22 to 46 years. Most participants (12 of 16) were female (Table 2). As expected, none of the residents had any previous contact with the Platform.
Evaluation of perceived usability: Observation and “think out loud”
The platform consists of three screens. The first one collects the patient’s personal information, the second has questions about the lesion (Fig 1), and the third screen provides a link to attach the photograph. The main difficulty pointed by participants (four dentist and five residents) were to fill the Individual Taxpayer number (ITN) (Fig 1, left screen). Other important question referred by one dentist and three residents was to fill the patient’s National Health Registry (CNS) number (Fig 1). In addition, analysis of Screen 3 revealed two important difficulties: 1) the need to “save a draft” of the request before attaching a photograph and 2) after attaching a photograph, the need to return to Screens 1 or 2 (Fig 1) to find the “send” button. There were four difficulties pointed out by users. The positive and negative perceptions were reported during the simulated requests (Fig 2), whereas the frequencies of problems are depicted in the Tables 3 and 4.
(A) Positive comments. (B) Negative comments.
Evaluation of usability: System usability scale (SyUS)
Individual SyUS scores are shown in Table 5. Regarding previous use of the Platform, heterogeneous results were obtained for dentists, with the number of previous interactions (i.e., previous requests placed) ranging from 3 to 35 (14.1±10.2. The highest score was assigned by two users: the most experienced participant (35 previous requests) and one resident. Table 5 shows scores above 80 for both groups (dentists and residents).
Tables 6 and 7 summarize the study findings. Residents required more time (347.1 s) to complete a request than dentists (252.8 s), who were familiar with the Platform workflow (Table 6). Data analysis showed an inverse correlation between SyUS score and time required to place the request (R = −0.54, P = .03, Pearson’s correlation).
Discussion
Principal results
The creation of a platform to support PHC in Brazil was a significant effort to improve health care services in a large country with uneven professional training standards and different degrees of technological familiarity. In this scenario, assessing usability is crucial for improvement. The results show that EstomatoNet has satisfactory usability. Even though focal problems were detected, even first-time users of the tool did not experience major challenges.
It is known that 85% of dentists find it difficult to detect, diagnose, and treat oral lesions [25]. The establishment of a teledentistry platform is one way of addressing this need. The high usability score obtained by EstomatoNet shows that the platform has the potential to bridge this gap in an efficient manner [26]. Although the use of information technology in health services is promising, few high-quality studies have assessed it [27].
Usability assessed using the SyUS was fully satisfactory according to dentists and residents. The fact that the difference between the groups was not significant indicates that the Platform is user-friendly, easy to learn and to manipulate, even for inexperienced users. Statements such as “the Platform has helped me in many cases” or “I find the Platform easy to use”, made by dentists who already use the Platform, confirm that it is well accepted. Finally, the statements regarding interest and self-confidence in using the system again support this interpretation.
Comparison with prior work
The high usability score obtained by the Platform becomes even more relevant if compared to the SyUS scores obtained by other tools. In a study by Ahn et al. [28], for example, in which five cardiopulmonary resuscitation training apps were assessed, only one app had a mean usability score above 80 (81.17±19.01). Lacerda et al. [29], who compared two cardiology telediagnosis interfaces, found a score below 80 for both (77.5 and 58.8). The differences among studies are probably related to the usability of the tools assessed therein.
Regarding the time required to complete the task (place a request), residents took longer than dentists. This suggests that the more one uses the system, the shorter the time to complete the task. A mean time below 6 minutes to perform the task (347.2s) seems acceptable to obtain the benefit associated with having support provided by a specialist who will help clarify the diagnosis and guide clinical decision-making. In addition, in many cases, specialist support may prevent referrals, reducing costs for the government and favoring professionals and patients.
The correlation test showed that the lower the time required to place the request, the higher the SyUS score. The fact that two unsatisfactory evaluations were made by the participants who took the longest to finalize the task further support this finding. Another factor influencing the SyUS score was experience with the Platform, since the professional with the highest number of previous requests (dentist 4) was the one who assigned the highest SyUS score.
The fact that some participants typed the name of the patient in the field meant for the National Health Registry number is justified by the name being, in general, the first information requested in web forms. The ability to copy/paste to the field and to view/access another window in the system could be considered improvements to the Platform.
In Screen 3, the steps of attaching a photograph and sending the request broke the workflow. Some simple adjustments would be enough to solve this problem. “Save draft” could also be available after the photograph is attached. The “Send” link could be transferred from screens 1 and 2 to screen 3. To avoid compromising task completion, this command could become automatically available as soon as the mandatory fields regarding the case were filled and the photographs attached.
The suggestion to add a field for further discussion of the case is also interesting. First, cases would be closed with information about outcomes, which would be useful for future reference; second, this adjustment would translate into an opportunity for continuity of care [30]. In other words, the specialist consultant could help the PHC dentist to evaluate the results of treatment in patients treated at the PHC level. with more effective follow-up along time.
Limitations
The present study has limitations that need to be addressed. First, the small number of participants and the fact that all were recruited from the same state precludes extrapolation of the present findings to the country. Also, the number of interactions required to reach optimal usability was not assessed. Finally, the interactions assessed were simulations, and additional difficulties could perhaps arise during the placement of a request referring to a real case. All these issues could be the focus of future studies.
Conclusions
The EstomatoNet Platform has satisfactory usability. Some focal problems associated with information fields should be addressed to improve the tool.
Supporting information
S1 Fig. Screens showing steps 1 and 2 required to request telehealth support.
https://doi.org/10.1371/journal.pone.0233572.s001
(PDF)
Acknowledgments
The authors thanks to Rosely de Andrade Vargas and Claudia Buchweitz manuscript formatting final revision. The authors also thanks Erno Harzheim, Jéssica Rodriguez Strey, and Carlos Pilz for their contributions.
References
- 1.
Brasil. Lei n. 8.808, de 19 de setembro de 1990. Dispõe sobre as condições para a promoção, proteção e recuperação da saúde, a organização e o funcionamento dos serviços correspondentes, e dá outras providências. Diário Oficial da União.1990 Set; Seção 1:18055.http://www2.camara.leg.br/legin/fed/lei/1990/lei-8080-19-setembro-1990-365093-publicacaooriginal-1-pl.html. Archived at:http://www.webcitation.org/6y7CzC02M
- 2.
Campos GW, Bonfim JR, Minayo MC, Akerman M, Drumond M Júnior, Carvalho YM, editors. Tratado de saúde coletiva. 2ndedition. São Paulo: Hucitec; 2012. ISBN: 8564806568
- 3.
Mendes EV. As redes de atenção à saúde. Brasília: Organização Pan-Americana da Saúde; 2011. ISBN: 9788579670756
- 4.
Starfield B. Primary Care: Balancing Health Needs, Services, and Technology. New York: Oxford University Press; 1998. ISBN: 0195125436
- 5.
World Health Organization—WHO. Declaration of Alma Ata. International conference on primary health care, Alma-Ata, USSR, 6–12 September 1978. Geneva: WHO; 1978.http://www.who.int/publications/almaata_declaration_en.pdf. Archived at:http://www.webcitation.org/6y8ghLGGx
- 6. de Mello AL, de Andrade SR, Moysés SJ, Erdmann AL. Saúde bucal na rede deatenção e processo de regionalização. CienSaude Colet 2014 Jan;19(1):205–14. pmid:24473617
- 7.
Brasil. Ministério da Saúde. Diretrizes da política nacional de saúde bucal.Brasília: Ministério da Saúde; 2004. http://bvsms.saude.gov.br/bvs/publicacoes/politica_nacional_brasil_sorridente.pdf. Archived at:http://www.webcitation.org/6y7DXMaiQ
- 8. Torres-Pereira CC, Morosini Ide A, Possebon RS, Giovanini AF, Bortoluzzi MC, Leão JC, et al.Teledentistry: distant diagnosis of oral disease using e-mails. Telemed J E Health 2013 Feb;19(2):117–21. pmid:23356381]
- 9. Harzheim E, Gonçalves MR, Umpierre RN, da Silva Siqueira AC, Katz N, Agostinho MR, et al. Telehealth in Rio Grande do Sul, Brazil: Bridging the Gaps. Telemed J E Health. 2016 Nov;22(11):938–44. pmid:27096384
- 10. Carrard VC, Roxo Gonçalves M, Rodriguez Strey J, Pilz C, Martins M, Martins MD, et al. Telediagnosis of oral lesions in primary care: The EstomatoNet Program. Oral Dis. 2018 Mar 5.
- 11. Gaggioli A, di Carlo S, Mantovani F, Castelnuovo G, Riva G.A telemedicine survey among Milan doctors. J Telemed Telecare 2005;11(1):29–34. pmid:15829041
- 12. Bevan N. Measuring usability as quality of use. Software Quality J 1995;4:115–50. ]
- 13.
Nielsen J. Usability engineering. Cambridge: Academic Press; 1994. ISBN: 9780125184069
- 14. De Bleser L, De Geest S, Vincke B, Ruppar T, Vanhaecke J, Dobbels F. How to test electronic adherence monitoring devices for use in daily life: a conceptual framework. Comput Inform Nurs 2011 Sep;29(9):489–95. pmid:21532468
- 15. Boland MR, Rusanov A, So Y, Lopez-Jimenez C, Busacca L, Steinman RC, et al. From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation framework. J Biomed Inform 2014 Dec;52:141–50. pmid:24333875
- 16. Fleming JB, Hill YN, Burns MN. Usability of a Culturally Informed mHealth Intervention for Symptoms of Anxiety and Depression: Feedback From Young Sexual Minority Men. JMIR Hum Factors. 2017 Aug 25;4(3):e22. pmid:28842389
- 17. Harrati N, Bouchrika I, Tari A, Ladjailia A. Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis. Comput Human Behav 2016 Aug;61:463–71. ]
- 18.
Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester B, McClelland IL, editors. Usability evaluation in industry. London: Taylor & Francis; 1996. p. 189–94. ISBN: 100748404600
- 19. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact 2008 Jul;24(6):574–94. ]
- 20.
Sauro J, Lewis JR. Quantifying the user experience: practical statistics for user research. San Francisco: Morgan Kaufmann; 2012. ISBN: 9780123849694
- 21. Orfanou K, Tselios N, Katsanos C. Perceived usability evaluation of learning management systems: empirical evaluation of the System Usability Scale. IRRODL 2015 Apr;16(2):227–46. ]
- 22.
Renault C. Improving web site usability for a better e-learning experience. In: Mendez-Villas A,Martín AS, Mesa González JA, editors. Current developments in technology-assisted education. Badajoz: Formatex; 2006. p. 891–95. ISBN:9788469024728
- 23. Tenório J. M. Cohrs F. M., Sdepanian V. L., Pisa I. T., de Marin H. F. Desenvolvimento e Avaliação de um Protocolo Eletrônico para Atendimento e Monitoramento do Paciente com Doença Celíaca. 2011. Rev Inform Teór Aplic. 2011;17(2):210–20. https://doi.org/10.22456/2175-2745.12119
- 24. Borsci S, Macredie RD, Barnett J, Martin J, Kuljis J, Young T. Reviewing and Extending the Five-User Assumption: A Grounded Procedure for Interaction Evaluation. ACM Transactions on Computer-Human Interaction 2013; 20(5):1–23. ]
- 25. Ergun S, Ozel S, Koray M, Kürklü E, Ak G, Tanyeri H. Dentists' knowledge and opinions about oral mucosal lesions. Int J Oral Maxillofac Surg 2009 Dec;38(12):1283–8. pmid:19651489
- 26.
Preece J, Rogers Y, Sharp H. Interaction design: beyond human-computer interaction. 4th edition. New York: Jonh Wiley & Sons; 2015. ISBN: 9781119088790
- 27. Ozdalga E., Ozdalga A., Ahuja N. The Smartphone in Medicine: A Review of Current and Potential Use Among Physicians and Students. J Med Internet Res 2012;14(5):e128 pmid:23017375
- 28. Ahn C, Cho Y, Oh J, Song Y, Lim TH, Kang H, et al. Evaluation of Smartphone Applications for Cardiopulmonary Resuscitation Training in South Korea. Biomed Res Int 2016;2016:6418710. pmid:27668257
- 29. Lacerda TC, von Wangenheim CG, von Wangenheim A, Giuliano I. Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context. J Biomed Inform 2014 Dec;52:222–30. pmid:25017250
- 30. da Cunha EM, Giovanella L. Longitudinalidade/continuidade do cuidado: identificando dimensões e variáveis para a avaliação da Atenção Primária no contexto do sistema público de saúde brasileiro. Cien Saude Colet 2011;16(Supl. 1):1029–42. pmid:21503452