https://orcid.org/0000-0001-5804-7651
Figures
Abstract
Accurate and user-friendly rapid point-of-care diagnostic tests (POCT) are needed to optimize treatment of Neisseria gonorrhoeae, especially in low-resource settings where syndromic management is the standard of care for sexually transmitted infections. This study aimed to assess the acceptability and usability of a novel lateral flow assay and portable reader for the point-of-care detection of N. gonorrhoeae infection (NG-LFA). This mixed-methods study was conducted as part of a diagnostic performance and usability evaluation of a prototype NG-LFA for detection of N. gonorrhoeae in symptomatic men and women at primary healthcare facilities in the Buffalo City Metro, South Africa. The Standardized System Usability Scale (SUS) was administered, and in-depth interviews were conducted among healthcare professionals (HCPs) and fieldworkers (FWs) at pre-implementation, initial use and 3- and 6-month study implementation to assess user expectations, practical experience, and future implementation considerations for the NG-LFA. Data collection and analysis was guided by the Health Technology Adoption Framework, including new health technology attributes, learnability, satisfaction, and suitability. The framework was adapted to include perceived durability. A total of 21 HCPs and FWs were trained on the NG-LFA use. SUS scores showed good to excellent acceptability ranging from 78.8–90.6 mean scores between HCPs and FWs across study time points. All transcripts were coded using Dedoose and qualitative findings were organized by learnability, satisfaction, suitability, and durability domains. Usability themes are described for each time point. Initial insecurity dissipated and specimen processing dexterity with novel POCT technology was perfected over time especially amongst FWs through practical learning and easy-to-use instructions (learnability). Participants experienced both positive and negative test results, yielding perceived accuracy and minimal testing challenges overall (satisfaction). By 3- and 6-month use, both HCPs and FWs found the NG-LFA convenient to use in primary health care facilities often faced with space constraints and outlined perceived benefits for patients (suitability and durability). Findings show that the NG-LFA device is acceptable and usable even amongst paraprofessionals. High SUS scores and qualitative findings demonstrate high learnability, ease-of-use and suitability that provide valuable information for first-step scale-up requirements at primary healthcare level. Minor prototype adjustments would enhance robustness and durability aspects.
Citation: de Vos L, Daniels J, Gebengu A, Mazzola L, Gleeson B, Piton J, et al. (2023) Usability of a novel lateral flow assay for the point-of-care detection of Neisseria gonorrhoeae: A qualitative time-series assessment among healthcare workers in South Africa. PLoS ONE 18(6): e0286666. https://doi.org/10.1371/journal.pone.0286666
Editor: Francis Xavier Kasujja, Medical Research Council / Uganda Virus Research Institute & London School of Hygiene and Tropical Medicine Uganda Research Unit, UGANDA
Received: January 20, 2023; Accepted: May 22, 2023; Published: June 2, 2023
Copyright: © 2023 de Vos et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The dataset for the System Usability Scale (SUS) has been uploaded to the Open Science Framework data repository. The identifier for this dataset is DOI 10.17605/OSF.IO/EFVHS. The qualitative data are not publicly available due to privacy or ethical restrictions as ensured by participant informed consent and South Africa’s protection of personal information act (POPIA). The relevant contact for any data inquiries and access upon reasonable request is: Freedom Mukomana <Freedomm@foundation.co.za> Data Manager, Foundation for Professional Development
Funding: The performance evaluation was funded by a subaward from the Global Antimicrobial Resistance Innovation Fund (GAMRIF) via FIND (recipient CF). The Foundation for Professional Development led the field evaluation in the Buffalo City Metropolitan Health District and laboratory work was conducted by the University of Pretoria. Access to the Neisseria gonorrhoeae lateral flow assay (and reader) system and test kits were sponsored by FIND as developed and provided by DCN. FIND supervised data collection procedures and manuscript draft revisions. (Funder website: https://www.gov.uk/government/groups/the-global-amr-innovation-fund). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: BCM-HD, Buffalo City Metropolitan Health District; FIND, The Foundation for Innovative New Diagnostics; FPD, Foundation for Professional Development; FWs, Fieldworkers; HCPs, Healthcare professionals; HCWs, Healthcare workers; IDIs, In-depth interviews; NG-LFA, Neisseria gonorrhoeae lateral flow assay (and reader); POCT, Point-of-Care Test; STIs, Sexually transmitted infections; SUS, System Usability Scale; WHO, World Health Organization; Xpert, GeneXpert®
Introduction
The global burden of sexually transmitted infections (STIs), particularly Neisseria gonorrhea, rests mostly in low- and middle-income countries (LMICs), including African nations, with limited to no change in incidence over time [1, 2]. For N. gonorrhoeae alone, there are an estimated 2 million cases per year in South Africa [3]. Left untreated, N. gonorrhoeae can lead to infertility and pelvic inflammatory diseases in women [4]. Rapid non-molecular, point-of-care tests (POCT) have the potential to address this disparity by optimizing STI treatment, and to reduce the burden of STIs by diagnosing asymptomatic infections [4–6]. Current laboratory diagnostics are expensive, not often available and takes a day or more for return of results, delaying patient diagnosis and treatment [2, 5, 7]. Rapid diagnostics have the potential to change current syndromic management (i,e. reduce over- and undertreatment) and facilitate same-day treatment that can reduce STI prevalence [8–13]. Currently, only rapid, high-performing non-molecular POCT are available for HIV and syphilis but not for other STIs, including N. gonorrhoeae [14]. Developing new POCT for N. gonorrhoeae requires a two-pronged approach that involves device development and usability [5, 14].
Specifically, coinciding with POCT performance evaluations, usability assessments with healthcare workers (HCWs) are essential to bringing such tests into clinical practice [8, 15]. The aims of these usability assessments are to determine if HCWs can learn how to conduct the POCT efficiently and effectively as well as evaluate their perspectives of using the POCT in their clinical setting and workflow. Appropriate training and quality monitoring models for HCWs to conduct POCT need to be developed in conjunction with device development [6, 9]. Such training models can ensure testing quality across different levels of staff from paraprofessionals to medical doctors and nurses. Further, training should be evaluated and modified as a result of pilot testing for quality monitoring and to support scale-up [8, 16, 17].
Assessing POCT usability by HCWs in regards to specimen handling, processing, and interpretation of results during test development can inform training and quality monitoring for roll-out [18, 19]. Specimen collection and preparation have been shown to present the greatest challenges in the testing process [15, 20]. Conversely, specimen processing and result interpretation have been shown to be easily completed especially if the results can be objectively displayed on the device [20–23]. During usability assessments, POCT learning demonstrates how health workers that initially describe challenges with specimen preparation (e.g. pipetting etc.) become more confident with adequate training and practice over time [15, 22, 24]. Models have demonstrated that untrained clinical staff can achieve consistent and effective device use, and clinically acceptable results, with appropriate instructional material and training [17, 25, 26]. Such task-shifting has expanded testing for HIV and has the potential to serve as a model for STI testing, accelerating prevention efforts [27].
In response to the need for N. gonorrhoeae POCT devices, FIND and WHO developed a target product profile (TPP) for the further development of a N. gonorrhoeae POCT [14]. As a result, a Lateral Flow Assay (LFA) was developed through a collaboration between FIND and DCN Diagnostics (Carlsbad, CA, USA) [28]. To support device development and feasibility, end-user experiences were assessed among two different levels of HCWs in South Africa. Using a time-series approach of interviews and surveys, we assessed changes in acceptance and usability in this setting with implications for implementation.
Materials and methods
Study design and setting
This mixed-methods study was conducted to assess the usability of this novel prototype N. gonorrhoeae lateral flow assay (NG-LFA) and reader that was developed to support N. gonorrhoeae testing capacity in low-resource settings [14, 28]. This evaluation was part of a cross-sectional study examining the diagnostic performance of the device [28, 29] and was conducted between November 2021 and September 2022. Usability was assessed amongst HCWs. Participants took part in a time-series assessment to outline changes in usability and acceptability at key time points for skilled learning.
HCWs using the NG-LFA device were located at five facilities in the Buffalo City Metropolitan Health District (BCM-HD) in the Eastern Cape Province, South Africa with designated STI research infrastructure and capacity (Fig 1). Facility types included public primary healthcare facilities and community health centers that serve an estimated population of ~755,000 with a population density of 298 persons/km2 (census 2011) [30].
Description NG-LFA under evaluation
At the time of the study, the NG-LFA was in assay-locked and ready for technology transfer for manufacturing scale. The NG-LFA device consists of a single-use disposable cassette using a lateral flow assay for the detection of N. gonorrhoeae. The lateral flow assay uses a fluorescent europium conjugate [28, 29]. Unlike most lateral flow assays, the NG-LFA test results are not visible to the naked eye. Rather, the test requires a small portable reader for the detection of fluorescence and classification of results (positive, negative, error) with test results available within 25 minutes. As part of the comprehensive field evaluation, the NG-LFA was compared to the Xpert CT/NG Assay® (Cepheid, Sunnyvale, CA) to assess diagnostic performance using fresh male urine and female vaginal swab specimens [29]. For symptomatic patients (N = 400), the NG-LFA showed a sensitivity of 91.7% and specificity of 96.3% amongst women with vaginal discharge syndrome and 96.1% sensitivity and 97.2% specificity amongst men with male urethral discharge. In case of a discordant result between the NG-LFA and Xpert®, the specimen was re-run on a new NG-LFA cassette. In case of an NG-LFA reader error, staff were instructed to check and re-insert the cassette to ensure the correct orientation.
Materials
Specimen collection kits and set-up for NG-LFA testing included: rayon swabs for vaginal specimen collection, urine jars and transfer pipets for male first-void urine, lysis buffer bottle and disposable tubes with dropper cap, as well as the NG-LFA device and NG-LFA reader (Figs 2 and 3). The NG-LFA reader is a compact 3D-printed prototype (7 x 9 x 6.5 cm, 90 g) and includes a small battery pack/power bank (165 g) with a grey USB cable (Fig 3). Pre-made positive and negative control cassettes were available for quality control. The reader presents an on/off button (a green light indicates that the device is on), a Wi-Fi connectivity button (not used here), three result indicator lights (-, !, + ), and a slot for insertion of the NG-LFA cassette. Within 5 seconds of insertion, the cassette test results were indicated by a single white light on the reader, displaying whether the outcome was negative (-); positive (+); or error (!). Errors could include a mis-orientation of the cassette, device failure, or reader failure. The POCT was kept in a clear plastic container for safekeeping.
(A) NG-LFA Reader. (B) NG-LFA cassette. (C) Test tube rack. (D) Transfer pipette. (E) Sample tube. (F) Dropper cap. (G) Extraction buffer with medicine dropper.
Implementing staff followed each step for testing set-up through the guidance of an instruction leaflet (S1 File).
The conceptual framework
The study was guided by the Health Technology Adoption Framework [8, 15] to assess acceptance and usability for new technology uptake in organizational settings. In this framework, acceptance and usability define feasibility. These inter-related domains consider how new devices, like NG-LFA, are perceived and believed by end-users to include trust in test results (acceptability), and then how users handle and apply the device and results (usability) [8, 15]. Taken together, these domains are further divided into six sub-domains; learnability, willingness, satisfaction, suitability, efficacy, and effectiveness [8, 19]. For this study, we adapted the framework (Fig 4) [8]. We included learnability, satisfaction, and suitability as relevant sub-domains but did not include willingness, efficacy, and effectiveness. Omitted sub-domains were not relevant to the current evaluation since we were not pilot testing for clinical protocols, and in turn, full feasibility assessment was not the study objective. However, we added durability as a domain to reflect low-resource settings where certain health system factors or challenges may impede the uptake of new technology [8], such as high patient loads, space constraints, and planned power outages (load-shedding) [31]. Further, acceptance and usability are influenced by factors relating to end-user types [8]. We, therefore, conducted a comparison between HCPs and FWs using the NG-LFA.
Study participants
All HCW participants were either directly or indirectly involved in the implementation of the NG-LFA at any time point. These end-users included medical officers, nurses, and fieldworkers working at a primary healthcare level in Buffalo City Municipality-Health District (BCM-HD). All HCPs had 10+ years of experience in the provision of public healthcare services and patient management. All FWs had a high-school-level degree, and some had experience with prior health programs. Nearly all participants were also trained on other STI POCT as part of previous clinical research studies. Throughout data collection, every newly recruited and trained end-user was invited to participate in IDIs.
Study procedures
Study procedures involved the following:
Training.
All staff received a theoretical overview of the testing procedures and were given information on N. gonorrhoeae infection through several lectures. Staff were trained as a group on the study protocol, and the NG-LFA reader. Group training over 3 hours was conducted as a classroom set-up and was facilitated by study investigators. Staff were given step-by-step instructions through quick cards (information leaflets) and supervised along the way. Staff were given all materials needed for specimen preparation and testing set-up. Staff practiced using the control cassettes, positive and negative contrived samples, sample preparation kits, and handling and interpretation of the device. Following the group training, each study site was visited by study investigators for 2–3 hours for additional training, troubleshooting, and site implementation set-up. Newly recruited staff members received a 1-day group training including a theoretical overview of the study protocol and testing procedures from senior supervisors at the office. They shadowed skilled-personnel at the facility-level for 2 days for practical NG-LFA experience. This training covered the same content as the original staff training.
Implementation phase.
During the field evaluation, the NG-LFA testing was performed in a nurses’ or fieldworkers’ consulting room (either within a healthcare facility or study container, Fig 1) within a public healthcare facility. The test was made available for interested and consenting male and female patients (≥ 18 years of age) presenting with male urethritis syndrome, vaginal discharge syndrome, or notified contacts of individuals with STI-associated symptoms. Participating men were requested to produce a first-void urine sample while vaginal swab specimens for NG-LFA testing were collected by nurses and then the test was either performed by the nurse or fieldworker (Fig 5). NG-LFA results were confirmed by Xpert® (Cepheid, Sunnyvale, CA). Registered nurses provided same-day test result notification, partner treatment counseling and pathogen-directed (i.e., targeted) treatment to patients that were willing to wait for the Xpert® results as the gold-standard (NG-LFA results were not used for diagnosis or treatment). If patients were not willing to wait 1.5 hours for the Xpert® results, they were given standard-of-care syndromic treatment.
Data collection tools
Usability surveys.
The System Usability Scale (SUS) was used to assess the usability of the NG-LFA post-training, 3- and 6-month implementation. The SUS is a validated, structured 10-item questionnaire using a 5-point Likert scale [32]. It measures perceived usability including ease of use, system satisfaction, and learnability. The SUS includes statements around technical support needs, training and perceived complexity of use [32]. Participants either agree or disagree with positively and negatively phrased statements about the device (ranging from strongly agree to strongly disagree).
In-depth interviews.
Interviews were conducted with HCPs and FWs over four-time points and administered by a qualitative researcher (Table 1). Specific in-depth interview (IDI) protocols were developed for each study phase including specific focus areas. Questions in these time series interviews focused on end-user acceptability, experiences and usability using the Health Technology Adoption Framework [8, 15] and preferences to support potential scale-up. Each interview protocol contained a set of probes for examples and elaboration.
Table 1 shows key questions of inquiry per study phase:
Data collection procedures
Using a purposive sampling method, participants were invited either in person, telephonically, or via email to participate in a series of surveys and interviews. Potential participants were consented using an informed consent form, before data collection. They were informed that they would be asked about their expectations, experiences, and handling of this novel NG-LFA. Study participation lasted approximately 6 months. The qualitative researcher has > 5 years of experience working on public health research studies and was trained in qualitative interviewing techniques, good clinical practice, and the study protocol. She did not have an established relationship with most of the participants before the study. SUS surveys were self-administered, either on paper or through an online survey link and data was securely collected using REDCap (Research Electronic Data Capture) [33, 34]. IDIs were either conducted in person in a private location or telephonically, as scheduled with each participant. All IDIs were conducted in English and lasted approximately 40 minutes. IDIs were audio-recorded and transcribed. All transcripts were reviewed for accuracy and completion. Regular team meetings were held to discuss and refine interview guides and data collection processes.
Data analysis
SUS.
SUS scores were calculated and analyzed at pre-implementation, and at 3- and 6-months of implementation. Total scores were calculated using the recommended SUS scoring system [32, 35, 36]. The score contribution of each response ranged between 0–4. Five was subtracted from the sum value of the odd-numbered questions. Whereas the sum value for the even-numbered questions was subtracted from 25. These two values were added and multiplied by 2.5 to get a total score out of 100. Scores ranged between worse and best imaginable and if a SUS score is above 68, it is considered to be acceptable [35]. A mean score between 68 and 80.3 means ‘good’, whereas a score above 80.3 means ‘excellent’. A lower score would indicate concerns with the current device. Scores were compared for HCPs and FWs at each time point.
Qualitative.
Data were analyzed using a constant comparison approach guided by the Health Technology Adoption Framework [8, 15]. A codebook was developed by open coding a sub-set of transcripts. Codes were based on the conceptual framework. The main analytical focus was handling, user experiences, patient-provider interactions, and clinical considerations. The codebook was applied to all transcripts by researchers JD, AG, and LDV using Dedoose [Version 9.0.17] [37]. Usability and acceptable sub-domains were compared across study phase and by user clinical role. Comparisons were drafted into a template analysis and iteratively refined through team discussions. Preliminary results were presented and discussed with the full research team. Results were organized according to the relevant attributes and themes.
Ethics
The study was approved by the Faculty of Health Sciences Research Ethics Committee, at the University of Pretoria, South Africa (Reference No.: 510/2021). Consent was sought from all participants using an informed consent form. Recordings of interviews were voluntary. At every interview, participants received an R100 (~5.7 USD/4.7 GBP) grocery voucher for their time.
Results
Participants
A total of 21 HCPs and FWs (20 female and 1 male HCW) were trained on the NG-LFA and participated in the SUS post-training. The original training included 8 HCPs and 13 FWs. Several HCPs and FWs were staff integrated in antenatal care and STI screening services. Fifteen participants (n = 7 FWs, n = 8 HCPs) identified as prospective end-users participated in a pre-implementation IDI (Eligible participant flow for IDIs and SUS have been included in S2 File). As soon as staff had used the NG-LFA in practice, participants were invited to participate in an initial use IDI. At this time point, we conducted a total of 22 IDIs (n = 13 FWs, n = 9 HCPs). Fourteen participants were still involved in the study after 3 months and available for a mid-assessment IDI (n = 8 FWs, n = 6 HCPs). We interviewed a total of twelve participants after six months (n = 7 FWs, n = 6 HCPs). This resulted in a total of 63 IDIs across all time points for analysis. We calculated and compared the SUS scores for participants who evaluated the NG-LFA over 6 months.
Usability scores
The NG-LFA device was considered acceptable at each key timepoint (Table 2). The device was deemed exceptional by HCPs at post-training and after 3-months of use (mean scores 89.9 and 90.6 respectively). Usability mean scores increased for FWs but decreased for HCPs at 6-months use (82.5 and 78.7, respectively). FWs rated the usability of the NG-LFA test as good at each time point (mean scores 79.7; 79.5; 82.5 respectively). Open comments revealed that the device was often perceived as ‘straight-forward’ and ‘easy’. Device features, result interpretation, and step-by-step guidance defined ease-of-use. The system was further perceived as compact, portable, and yielded quick results. The overall mean SUS score for the NG-LFA device after 6-months of use was 80.6 (‘excellent’).
Qualitative findings
Key qualitative findings and themes for learnability, satisfaction, suitability, and durability are presented below. Themes are described across each phase of data collection: 1) Pre-implementation: ease-of-use, dissipating health technology insecurities and benchwork training considerations; 2) Initial Use: refinement of specimen processing and perceived accuracy; 3) Mid-assessment: comfort with specimen processing and design implications; and 4) Post-evaluation: information retention and perceived NG-LFA device longevity.
Illustrative quotes supporting the key findings across the time points and staff type are described in Table 3. We present illustrative quotes from two levels: healthcare professionals (HCPs, shaded in grey), and field workers (FWs, shaded in white).
Pre-implementation.
Benchwork training considerations and untested design impressions. HCPs and FWs described the training for the NG-LFA device as helpful, with no information missing and clear, easy-to-follow steps. Although the testing process was described as such, more FWs than HCPs emphasized difficulties in using the urine transfer pipette for approximate measurements. In response to this difficulty, participants were able to practice specimen preparation several times to build confidence in specimen handling. However, both HCPS and FWs acknowledged the need for additional post-training support especially when staff are new to pipetting. Participants did not expect the light-90g weight of the NG-LFA reader prototype, which was lighter and smaller when they compared it to similar size devices that they used in the past for clinical diagnosis. Further, the size of the reader implied that it was designed to be easily portable, but some HCPs and FWs expressed concern that it would be fragile especially if used often in a setting where power outages were common. However, the supplied battery power bank and the ability to monitor battery consumption was an added benefit.
Ease-of-use dissipates new health technology insecurities. FWs expressed lower confidence with novel rapid tests than HCPs due to experience and the assumption that it would require more technological know-how. However, once engaged in the practical training, FWs were pleasantly surprised by the ease of use. Some HCPs also expressed anticipating advanced handling given prior experiences, but the step-by-step instructions on quick cards offered end-user guidance and enabled confidence. The learning process to use the device was described as, ‘easy to read even for a layman’, and ‘even the field worker can do this’ by FWs, and ‘understand it from the word go, and ‘written in simple English’, by HCPs. Such that FWs were convinced the NG-LFA was not just to be administered by ‘nurses’. During training, participants practiced using supplied control cassettes to assess the quality and accuracy of the device. Participants felt that the daily use of control cassettes and the availability of an ‘invalid’ notification on the reader also offered guidance and little room for error.
Initial use.
Specimen collection and processing dexterity. Quick cards were developed that outlined steps for processing urine and vaginal swab specimens. FWs expressed initial challenges with the preparation of specimens to include getting the approximate measurements of the buffer solution using the medicine dropper, and the manual process of extracting vaginal swabs in the buffer solution. The extraction process/step of vaginal swab requires rigorous handling and squeezing of the rayon swab tip, whereby urine specimen collection and preparation for male patients was perceived as easier. After their initial testing and use of the device, some participants doubted whether their handling of the vaginal swab specimens was sufficient and questioned whether it would yield concordant results.
Technical support and accuracy considerations. Testing did result in a few discordant results between the NG-LFA and the Xpert device®, and for some FWs, this led to confusion. Explanations given by study investigators why the NG-LFA may yield different results, dissipated initial concerns, but this confusion rested in the limited clinical education that FWs possessed. HCPs seemed to show a deeper understanding of the reasons for discordant results (e.g., how weak positives can result in negative LFAs). For all participants, experience with sample handing procedures helped to increase perception that test results were accurate. Despite this discordancy, participants expressed the importance of testing convenience and perceived benefits for patients that the NG-LFA could yield quick results for diagnosis and treatment.
Mid-assessment (3 months).
Comfort with specimen processing. The simple design and clear instructions continued to ensure end-user satisfaction. By 3 months, HCPs and FWs acknowledged increased confidence in specimen processing as compared to the start of implementation. The reader functioned as expected and staff were easily able to interpret the results using the markings on the device. Staff achieved increasing concordant results of the NG-LFA compared to Xpert, which were largely assumed by end-users to be due to enhanced specimen processing dexterity. The reader and NG results gave minimal challenges and testing experiences were similar for HCPs and FWs.
Implications of device design. The size and structure of the device gave participants the ability to move around and use the device between departments and shared workspaces. Furthermore, the device was easy to store, manage and maintain in a confined workspace when compared to other POCTs. Since the NG-LFA reader is a 3D printed prototype, both HCPs and FWs gave examples of minor issues experienced over the past three months with the device. These included the power on/off button falling off and a reader where it was difficult to insert the cassette. Also, some HCPs expressed that a limitation of the device is that it only tests for one STI, leaving other infections unaddressed. However, given that the feature limitations were few, and the test offered quick results and treatment guidance, staff were keen that the device could be developed to benefit patients (described in Table 3).
Post-evaluation (6 months).
Easy retention of information. After 6 months of implementation, participants were able to perform the testing process without referring to the quick card instruction leaflets. FWs gave examples of being able to describe the testing process and device to new staff members and patients. New staff members were trained and shadowed skilled staff members prior to conducting testing themselves, and this adjusted model of training did not impact use. Participants also expressed confidence and trust in the test due to the concordance observed of the NG-LFA and Xpert® test results. All in all, training and practical experience was perceived as adequate for both originally trained and new HCPs and FWs. Although practical experience to refine testing dexterity and exact measurements for liquid transfer was preferred by new staff from the onset of training at the office, to compliment theoretical presentations. However, outcomes revealed that this was still adequately provided either through study-site set-up or shadowing skilled staff.
Perceived longevity of the device. To most participants the NG-LFA was good, but not yet excellent. The perceived longevity of the current device ranged between 9 months and 5 years due to the light weight of the prototype NG-LFA reader. Recommendations by staff for improved robustness would increase usability in high-burden settings. These included added weight to the reader and fixed features (e.g., markings, buttons and batteries). However, the compact size aided the perceived suitability to the clinical environment and movement within the room or other clinic departments. Even after 6 months of use, the prototype proved to have ‘taken its’ place’ by the prospect of offering routine STI testing to patients. The power bank also accommodated planned and unplanned power outages by having a long battery life as the reader required minimal electricity. However, consistent power disruptions and high patient-load required staff to be organized and charge the power bank several times a week to ensure continued clinical use.
Discussion
We assessed if the NG-LFA test was acceptable and usable, exploring relevant Health Technology Adoption Framework attributes [8, 15] among healthcare workers involved in STI service provision. This was the first evaluation of the FIND developed NG-LFA in a LMIC setting, and a time-series, mixed methods assessment allowed us to understand end-user experience for device development and associated training needs [8, 38].
Overall usability was high with SUS score results for the NG-LFA showing comparable outcomes to other novel antigen rapid tests like for SARS-CoV-2 and optical reader prototype for any rapid POCT LFA [39–41]. By administering the SUS over several time points during the study, changes in end-user perceived skills in test performance could be assessed more accurately. Although acceptability was consistent for both groups, a slightly lower overall SUS score for HCPs during implementation was found, but we attribute this change in their shift to specimen collection and clinical management responsibilities, as well as the training model for this study.
Deploying a training model that combines classroom instruction with observed practice and supervision in the clinic for the NG-LFA proved effective with minimal refresher trainings needed. Studies show that practical training and ongoing mentoring ensure testing is performed effectively while also building sustainability of device use [8, 9, 22, 27, 42, 43]. In particular, specimen preparation is an essential step, and as in other POCT studies, we found this to be the most challenging step [20, 21]. Specifically, most participants found it easier to collect and process urine samples as compared to swabs, but over time and with support, swab adeptness improved [24]. Additionally, this training model can be complemented with resources to guide each step like printed instructions as staff move toward testing independence [15, 17, 21, 25]. Such resources reduced the number of incidents of incorrect handling and specimen transfer in our study [15, 20].
In alignment with other studies, [24, 42, 44, 45] NG-LFA device features that were highly acceptable to participants included those that supported easy interpretation of test results, quality control measures, and durability. Simple markings and lighting reduced cases of result misinterpretation (e.g. basic indicators like +/-) [24, 42, 45, 46]. An alert system and control cassettes aided quality control and test accuracy along with limited built-in software to assist easy execution of the test [24, 42]. Finally, device portability and compactness was perceived to support its durability in clinical settings with high patient volume. However, the current device tests for only one STI, gonorrhoea, and device satisfaction amongst HCWs may be higher if it was able to test for other STIs [44].
Among participants, there was a high commitment to support investment in health-improving technologies in order to improve testing and treatment outcomes in this setting [6, 9, 22, 47]. The prospect that the NG-LFA could eventually increase STI testing accessibility and provide quick results for treatment and preventative care motivated participant engagement in this study [2, 9, 15, 24–26, 47]. Further, the design of the device, particularly its size and power independence, facilitated usability and was perceived as ideal for small clinical work areas common in such settings that have a high STI burden. Such results can support antibiotic stewardship for N. gonorrhoeae and other STIs within health districts and municipalities and in turn slow down the process of antimicrobial resistance development, acquisition and emergence.
Lessons learnt
A time-series, mixed-methods approach was effective in identifying when healthcare workers become adept in the use of the NG-LFA device and testing process. This approach for assessing device usability provides information relevant to its development such as end-user expectations and training interventions needed at different timepoints to improve test performance and accuracy [8]. Also, our findings further defined durability in a low-resource setting, as a critical sub-domain for usability not previously included in the Health Technology Adoption framework [8]. Durability emphasizes the influence of organizational and environmental influences on device development such that power, storage, security and space constraints should be integrated in design and features such as the planned power outages in this setting [9, 38, 43, 45, 48].
Implications
Our study showed that paraprofessionals can easily operate the NG-LFA and conduct effective POC STI testing as in similar studies with non-clinically trained staff [13, 25]. Therefore, task-shifting may be feasible where staff like fieldworkers can operate POC tests, which may in turn allow healthcare professionals to prioritize clinical management and treatment [6, 26]. Further, we show the effectiveness of the training model such that new staff were similarly trained by more experienced colleagues and reported comparable challenges and progress in adeptness in test performance. This suggests the adequacy of a ‘train-the-trainer’ approach using our training model to support feasible device scale-up [8, 17].
Limitations
This study was implemented amongst HCWs who are clinical research staff, but they are familiar with routine practice of STI services and all had extensive nursing experience in public clinics. However, insight was gained into device suitability considerations for constrained resource settings by using the device in primary healthcare facilities. Although this study was not designed to assess full feasibility since willingness, efficacy, and effectiveness were not considered at this stage of device development; user-handling experiences and design recommendations can inform future research to identify factors influential in clinic integration like organizational readiness in using the NG-LFA device in STI care and treatment.
Conclusion
Design of POCT for STIs must include end-user preferred features for quality control and easy readability without compromising durability that is contextually grounded. Findings show that the NG-LFA is a usable and acceptable POCT that may easily be considered for use in other LMIC’s based on its characteristics. These may include provisions for commonly found setting limitations such as disrupted power supply and small workspaces. The NG-LFA features can support scale-up in clinical settings to improve access to diagnosis testing and treatment, especially in settings where significant STI burden remains unaddressed, and when misdiagnosis and antibiotic overuse can lead to antimicrobial resistance [49–51]. And, if implemented appropriately with a mentored training program, then POCT can be administered by paraprofessionals so that nurses and doctors can focus on clinical care. This may improve time around results, appropriate use of antibiotics and reach to asymptomatic patients reducing transmission in turn.
Supporting information
S1 File. FIND NG-LFA testing and reader- set up instructions (QuickCards).
https://doi.org/10.1371/journal.pone.0286666.s001
(PDF)
S2 File. Study participation flow for time series assessment: SUS surveys and IDIs.
https://doi.org/10.1371/journal.pone.0286666.s002
(PDF)
Acknowledgments
DCN Diagnostics is a contract developer and manufacturer of lateral flow devices for diagnostics at the point of care. DCN performed the development and manufacturing for the prototype NG-LFA device and POC reader. This study was implemented with the permission and support of the Eastern Cape provincial and Buffalo City Metro health district departments. We would like to especially thank the Foundation for Professional Development healthcare workers for their insights and contributions towards the Neisseria gonorrhoeae lateral flow assay performance evaluation.
References
- 1. Rowley J, Vander Hoorn S, Korenromp E, Low N, Unemo M, Abu-Raddad LJ, et al. Chlamydia, gonorrhoea, trichomoniasis and syphilis: global prevalence and incidence estimates, 2016. Bull World Health Organ. 2019 Aug 1;97(8):548–562P. pmid:31384073
- 2. Wi TE, Ndowa FJ, Ferreyra C, Kelly‐Cirino C, Taylor MM, Toskin I, et al. Diagnosing sexually transmitted infections in resource‐constrained settings: challenges and ways forward. J Int AIDS Soc. 2019 Aug 30;22(Suppl 6):e25343. pmid:31468679
- 3. Kularatne RS, Niit R, Rowley J, Kufa-Chakezha T, Peters RPH, Taylor MM, et al. Adult gonorrhea, chlamydia and syphilis prevalence, incidence, treatment and syndromic case reporting in South Africa: Estimates using the Spectrum-STI model, 1990–2017. Paz-Bailey G, editor. PLoS ONE. 2018 Oct 15;13(10):e0205863. pmid:30321236
- 4. Jongen VW, Loeff MF Schim van der, MH Botha, Sudenga SL, Abrahamsen ME, Giuliano AR. Incidence and risk factors of C. trachomatis and N. gonorrhoeae among young women from the Western Cape, South Africa: The EVRI study. Peters RP, editor. PLoS ONE. 2021 May 3;16(5):e0250871. pmid:33939747
- 5.
World Health Organization. Global Health Sector Strategy on Sexually Transmitted Infections 2016–2021 [Internet]. World Health Organization; 2016 Jun [cited 2022 Oct 19] p. 64. Available from: https://apps.who.int/iris/bitstream/handle/10665/246296/WHO-RHR-16.09-eng.pdf
- 6. Pacho A, Harding-Esch EM, Heming De-Allie EG, Phillips L, Furegato M, Sadiq ST, et al. Facilitators and barriers for clinical implementation of a 30-minute point-of-care test for Neisseria gonorrhoeae and Chlamydia trachomatis into clinical care: A qualitative study within sexual health services in England. Rios-González CM, editor. PLoS ONE. 2022 Mar 10;17(3):e0265173. pmid:35271658
- 7.
World Health Organization. Global strategy for the prevention and control of sexually transmitted infections: 2006–2015: breaking the chain of transmission [Internet]. Geneva: World Health Organization; 2007. Available from: https://apps.who.int/iris/handle/10665/70186
- 8. Ansbro ÉM, Gill MM, Reynolds J, Shelley KD, Strasser S, Sripipatana T, et al. Introduction of Syphilis Point-of-Care Tests, from Pilot Study to National Programme Implementation in Zambia: A Qualitative Study of Healthcare Workers’ Perspectives on Testing, Training and Quality Assurance. Manabe YC, editor. PLoS ONE. 2015 Jun 1;10(6):e0127728. pmid:26030741
- 9. Rasti R, Nanjebe D, Karlström J, Muchunguzi C, Mwanga-Amumpaire J, Gantelius J, et al. Health care workers’ perceptions of point-of-care testing in a low-income country—A qualitative study in Southwestern Uganda. Schallig HDFH, editor. PLoS ONE. 2017 Jul 27;12(7):e0182005. pmid:28750083
- 10. Peeling RW, Holmes KK, Mabey D, Ronald A. Rapid tests for sexually transmitted infections (STIs): the way forward. Sex Transm Infect. 2006 Dec 6;82(suppl V):v1–6. pmid:17151023
- 11. Unemo M. Accurate, rapid, point-of-care tests for sexually transmitted infections. The Lancet Infectious Diseases. 2021 May;21(5):584–6. pmid:33242474
- 12. Unemo M, Bradshaw CS, Hocking JS, de Vries HJC, Francis SC, Mabey D, et al. Sexually transmitted infections: challenges ahead. Lancet Infect Dis. 2017 Jul 9;17(8):e235–79. pmid:28701272
- 13. Fuller SS, Clarke E, Harding-Esch EM. Molecular chlamydia and gonorrhoea point of care tests implemented into routine practice: Systematic review and value proposition development. Oldenburg CE, editor. PLoS ONE. 2021 Nov 8;16(11):e0259593. pmid:34748579
- 14. Ferreyra C, Osborn J, Moussy F, Alirol E, Lahra M, Whiley D, et al. Developing target product profiles for Neisseria gonorrhoeae diagnostics in the context of antimicrobial resistance: An expert consensus. Yansouni C, editor. PLoS ONE. 2020 Sep 1;15(9):e0237424. pmid:32870912
- 15. Asiimwe C, Kyabayinze DJ, Kyalisiima Z, Nabakooza J, Bajabaite M, Counihan H, et al. Early experiences on the feasibility, acceptability, and use of malaria rapid diagnostic tests at peripheral health centres in Uganda-insights into some barriers and facilitators. Implementation Science. 2012 Jan 23;7(5). pmid:22269037
- 16. Mabey DC, Sollis KA, Kelly HA, Benzaken AS, Bitarakwate E, Changalucha J, et al. Point-of-Care Tests to Strengthen Health Systems and Save Newborn Lives: The Case of Syphilis. PLoS Medicine. 2012 Jun 12;9(6):e1001233. pmid:22719229
- 17. Yao K, Wafula W, Bile EC, Cheignsong R, Howard S, Demby A, et al. Ensuring the Quality of HIV Rapid Testing in Resource-Poor Countries Using a Systematic Approach to Training. Am J Clin Pathol. 2010 Oct;134(4):568–72. pmid:20855637
- 18. Parreira P, Sousa LB, Marques IA, Santos-Costa P, Cortez S, Carneiro F, et al. Study Protocol for Two-Steps Parallel Randomized Controlled Trial: Pre-Clinical Usability Tests for a New Double-Chamber Syringe. Int J Environ Res Public Health. 2020 Nov 12;17(8376). pmid:33561056
- 19. The ProSPeRo Network. Standardised protocol for a prospective cross-sectional multicentre clinical utility evaluation of two dual point-of-care tests in non-clinical settings for the screening of HIV and syphilis in men who have sex with men. BMJ Open. 2022 Jun;12(6):e055275.
- 20. Majam M, Fischer A, Ivanova Reipold E, Rhagnath N, Msolomba V, Lalla-Edward ST. A Lay-User Assessment of Hepatitis C Virus Self-Testing Device Usability and Interpretation in Johannesburg, South Africa. Diagnostics (Basel). 2021 Mar 7;11(3):463. pmid:33800060
- 21. Majam M, Mazzola L, Rhagnath N, Lalla-Edward ST, Mahomed R, Venter WDF, et al. Usability assessment of seven HIV self-test devices conducted with lay-users in Johannesburg, South Africa. PLoS One. 2020 Jan 14;15(1):e0227198. pmid:31935228
- 22. Young N, Achieng F, Desai M, Phillips-Howard P, Hill J, Aol G, et al. Integrated point-of-care testing (POCT) for HIV, syphilis, malaria and anaemia at antenatal facilities in western Kenya: a qualitative study exploring end-users’ perspectives of appropriateness, acceptability and feasibility. BMC Health Serv Res. 2019 Dec;19(1):74. pmid:30691447
- 23. Rondy M, Tamboura M, Sidikou F, Yameogo I, Dinanibe K, Sawadogo G, et al. Field Evaluation of the Performance of Two Rapid Diagnostic Tests for Meningitis in Niger and Burkina Faso. Microorganisms. 2021 Apr 14;9(4):832. pmid:33919828
- 24. Natoli L, Guy RJ, Shephard M, Causer L, Badman SG, Hengel B, et al. “I Do Feel Like a Scientist at Times”: A Qualitative Study of the Acceptability of Molecular Point-Of-Care Testing for Chlamydia and Gonorrhoea to Primary Care Professionals in a Remote High STI Burden Setting. Kumar S, editor. PLoS ONE. 2015 Dec 29;10(12):e0145993. pmid:26713441
- 25. Morris SR, Bristow CC, Wierzbicki MR, Sarno M, Asbel L, French A, et al. Performance of a single-use, rapid, point-of-care PCR device for the detection of Neisseria gonorrhoeae, Chlamydia trachomatis, and Trichomonas vaginalis: a cross-sectional study. Lancet Infect Dis. 2021 May;21(5):668–76. pmid:33242473
- 26. Dawkins M, Bishop L, Walker P, Otmaskin D, Ying J, Schmidt R, et al. Clinical Integration of a Highly Accurate Polymerase Chain Reaction Point-of-Care Test Can Inform Immediate Treatment Decisions for Chlamydia, Gonorrhea, and Trichomonas. Sexual Trans Dis. 2022 Apr;49(4):262–7. pmid:34813579
- 27. Shott JP, Galiwango RM, Reynolds SJ. A Quality Management Approach to Implementing Point-of-Care Technologies for HIV Diagnosis and Monitoring in Sub-Saharan Africa. Journal of Tropical Medicine. 2012;2012:1–8. pmid:22287974
- 28.
Gleeson B. Development of an Antigen Rapid Diagnostic Test to Detect Neisseria gonorrhoeae to support the aetiological management of patient with sexually transmitted infections and ensure antibiotic stewardship. Poster Presentation presented at: 23rd IUSTI World Congress; 2022 Sep 4; Victoria Falls, Zimbabwe.
- 29.
Peters R. A novel lateral flow assay for point-of-care detection of Neisseria gonorrhoeae [Internet]. Oral Presentation presented at: 23rd IUSTI World Congress; 2022 Sep 7; Victoria Falls, Zimbabwe. Available from: https://www.iusti2022zimbabwe.com/wp-content/uploads/2022/10/5-Remco-Peters_A-novel-lateral-flow-assay-for-NG-pre-final.pdf
- 30.
Buffalo City [Internet]. Stats SA Department: Statistics South Africa Republic of South Africa. Available from: https://www.statssa.gov.za/?page_id=993&id=buffalo-city-municipality%20
- 31. Martin K, Dziva Chikwari C, Mackworth-Young CRS, Chisenga M, Bandason T, Dauya E, et al. “It was difficult to offer same day results”: evaluation of community-based point-of-care testing for sexually transmitted infections among youth using the GeneXpert platform in Zimbabwe. BMC Health Serv Res. 2022 Dec;22(1):171. pmid:35144602
- 32.
Brooke J. SUS—A quick and dirty usability scale. In: Usability Evaluation In Industry. 1st Edition. CRC Press; 1995. p. 6.
- 33. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics. 2009 Apr;42(2):377–81. pmid:18929686
- 34. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: Building an international community of software platform partners. Journal of Biomedical Informatics. 2019 Jul 1;95:103208. pmid:31078660
- 35. Bangor A, Kortum , Philip , Miller , James . Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies. 2009;4(3):114–23.
- 36. Brooke J. SUS: a retrospective. Journal of Usability Studies. 2013 Feb;8(2):29–40.
- 37.
Dedoose Version 9.0.17, web application for managing, analyzing, and presenting qualitative and mixed method research data (2021) [Internet]. Los Angeles, CA: SocioCultural Research Consultants, LLC; 2021. Available from: www.dedoose.com
- 38. Pai NP, Vadnais C, Denkinger C, Engel N, Pai M. Point-of-Care Testing for Infectious Diseases: Diversity, Complexity, and Barriers in Low- And Middle-Income Countries. PLoS Med. 2012 Sep 4;9(9):e1001306. pmid:22973183
- 39. Krüger LJ, Gaeddert M, Tobian F, Lainati F, Gottschalk C, Klein JAF, et al. The Abbott PanBio WHO emergency use listed, rapid, antigen-detecting point-of-care diagnostic test for SARS-CoV-2—Evaluation of the accuracy and ease-of-use. Mantis NJ, editor. PLoS ONE. 2021 May 27;16(5):e0247918. pmid:34043631
- 40. Krüger LJ, Tanuri A, Lindner AK, Gaeddert M, Köppel L, Tobian F, et al. Accuracy and ease-of-use of seven point-of-care SARS-CoV-2 antigen-detecting tests: A multi-centre clinical evaluation. eBioMedicine. 2022 Jan;75:103774. pmid:34959134
- 41. Hohenstein J, O’Dell D, Murnane EL, Lu Z, Erickson D, Gay G. Enhancing the Usability of an Optical Reader System to Support Point-of-Care Rapid Diagnostic Testing: An Iterative Design Approach. JMIR Hum Factors. 2017 Nov 21;4(4):e29. pmid:29162559
- 42. Palamountain KM, Baker J, Cowan EP, Essajee S, Mazzola LT, Metzler M, et al. Perspectives on Introduction and Implementation of New Point-of-Care Diagnostic Tests. The Journal of Infectious Diseases. 2012 May 15;205(Suppl 2):S181–90. pmid:22402038
- 43. Garcia PJ, You P, Fridley G, Mabey D, Peeling R. Point-of-care diagnostic tests for low-resource settings. The Lancet Global Health. 2015 May;3(5):e257–8. pmid:25889467
- 44. Ferreyra C, Redard-Jacot M, Wi T, Daily J, Kelly-Cirino C. Barriers to Access to New Gonorrhea Point-of-Care Diagnostic Tests in Low- and Middle-Income Countries and Potential Solutions: A Qualitative Interview-Based Study. Sexual Trans Dis. 2020 Oct;47(10):698–704. pmid:32936604
- 45. Hsieh YH, Hogan MT, Barnes M, Jett-Goheen M, Huppert J, Rompalo AM, et al. Perceptions of an Ideal Point-of-Care Test for Sexually Transmitted Infections–A Qualitative Study of Focus Group Discussions with Medical Providers. Kissinger P, editor. PLoS ONE. 2010 Nov 30;5(11):e14144. pmid:21152386
- 46. Jing M, Bond R, Robertson LJ, Moore J, Kowalczyk A, Price R, et al. User experience analysis of AbC-19 Rapid Test via lateral flow immunoassays for self-administrated SARS-CoV-2 antibody testing. Sci Rep. 2021 Dec;11(1):14026. pmid:34234188
- 47. Natoli L, Guy RJ, Shephard M, Whiley D, Tabrizi SN, Ward J, et al. Public health implications of molecular point-of-care testing for chlamydia and gonorrhoea in remote primary care services in Australia: a qualitative study. BMJ Open. 2015 Apr 28;5(4):e006922–e006922. pmid:25922100
- 48. Bianchi F, Clemens S, Arif Z, Sacks E, Cohn J. Acceptability of Routine Point-of-Care Early Infant Diagnosis in Eight African Countries: Findings From a Qualitative Assessment of Clinical and Laboratory Personnel. J Acquir Immune Defic Syndr. 2020 Jul 1;84(1):S41–8. pmid:32520914
- 49.
World Health Organization. Antibiotic resistance [Internet]. Available from: https://www.who.int/news-room/fact-sheets/detail/antibiotic-resistance
- 50.
Centers for Disease Control and Prevention. About Antibiotic Resistance | CDC [Internet]. Antimicrobial Resistance. 2022. Available from: https://www.cdc.gov/drugresistance/about.html
- 51. Aslam B, Wang W, Arshad MI, Khurshid M, Muzammil S, Rasool MH, et al. Antibiotic resistance: a rundown of a global crisis. IDR. 2018 Oct;1645–58. pmid:30349322