https://orcid.org/0000-0002-0862-0824
Figures
Abstract
Usability is an overlooked aspect of implementing lab-based assays, particularly novel assays in low-resource-settings. Esoteric instructions can lead to irreproducible test results and patient harm. To address these issues, we developed a software application based on “Aquarium”, a laboratory-operating system run on a computer tablet that provides step-by-step digital interactive instructions, protocol management, and sample tracking. Aquarium was paired with a near point-of-care HIV drug resistance test, “OLA-Simple”, that detects mutations associated with virologic failure. In this observational study we evaluated the performance of Aquarium in guiding untrained users through the multi-step laboratory protocol with little supervision. To evaluate the training by Aquarium software we conducted a feasibility study in a laboratory at Coptic Hope Center in Nairobi, Kenya. Twelve volunteers who were unfamiliar with the kit performed the test on blinded samples (2 blood specimens; 5 codons/sample). Steps guided by Aquarium included: CD4+ T-Cell separation, PCR, ligation, detection, and interpretation of test results. Participants filled out a short survey regarding their demographics and experience with the software and kit. None of the laboratory technicians had prior experience performing CD4+ separation and 7/12 had no experience performing laboratory-based molecular assays. 12/12 isolated CD4+ T cells from whole blood with yields comparable to isolations performed by trained personnel. The OLA-Simple workflow was completed by all, with genotyping results interpreted correctly by unaided-eye in 108/120 (90%) and by software in 116/120 (97%) of codons analyzed. In the surveys, participants favorably assessed the use of software guidance. The Aquarium digital instructions enabled first-time users in Kenya to complete the OLA-simple kit workflow with minimal training. Aquarium could increase the accessibility of laboratory assays in low-resource-settings and potentially standardize implementation of clinical laboratory tests.
Citation: Vrana JD, Panpradist N, Higa N, Ko D, Ruth P, Kanthula R, et al. (2022) Implementation of an interactive mobile application to pilot a rapid assay to detect HIV drug resistance mutations in Kenya. PLOS Glob Public Health 2(2): e0000185. https://doi.org/10.1371/journal.pgph.0000185
Editor: Abdullah Yusuf, National Institute of Neuroscience Department of Mental Retardation and Birth Defect Research: Kokuritsu Kenkyu Kaihatsu Hojin Seishin Shinkei Iryo Kenkyu Center Shinkei Kenkyujo Shippei Kenkyu Dainibu, BANGLADESH
Received: September 29, 2021; Accepted: January 9, 2022; Published: February 16, 2022
Copyright: © 2022 Vrana et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All data associated with this study are available in the main text and Supplementary Information.
Funding: This work was supported by R01 AI110375 (LMF); R01 AI145486-01A1 (BRL); T32 HD007233 (RK); and UW Bioengineering Pilcher Faculty Fellowship (BRL). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have no conflicts of interest to declare.
Introduction
In resource-rich communities, automation drastically improves the daily operation of clinical laboratories [1–3]. Patient samples can be quickly shipped to centralized laboratories for batch-processing and testing using highly efficient workflows that generate high-quality results while reducing costs and turnaround time [4, 5]. However, total automation is ill-suited for low-resource settings for many reasons. First, shipping of samples to centralized laboratories can take ≥10 days [6], which undermines the benefits of fast turnaround results from an automated workflow. Second, in small communities, the demand of a clinical assay may be low, requiring a longer waiting period to receive enough samples to complete a full batch. Finally, automation is often used in conjunction with high-throughput robotic equipment that is cost-prohibitive for small laboratories. In low-resource settings, high-quality and fast laboratory results for complex assays will likely require unorthodox approaches to build on low-cost equipment and small batches of samples.
HIV infects nearly 40 million people globally [7] and successful management of the infection relies on multiple laboratory tests. Recent advances include point-of-care HIV diagnosis and viral load quantification [8, 9]. Due to the complexity of HIV drug resistance (HIVDR) tests used to guide treatment regimens, these are performed in centralized, highly equipped laboratories [10–13]. In low-resource countries with high HIV prevalence like Kenya, few laboratories have the capacity to test for HIVDR [14]. For laboratories without access to sequencers, an oligonucleotide ligation assay (OLA) has been implemented [15] but onboarding OLA required extensive training due to its complexity.
We envision the use of software to automate a simplified version of OLA that uses low-cost equipment. To that end, we developed “OLA-Simple” which uses lyophilized reagents to simplify the workflow and lateral flow tests to provide visual results [16, 17]. We also developed a software application based on “Aquarium” [18] that employs human-in-the-loop automation to tightly integrate all the steps in OLA-Simple. Aquarium provides step-by-step interactive digital instructions, protocol management, data collection and sample tracking. In a pilot study at the University of Washington, Aquarium enabled minimally-trained students to accurately perform the OLA-Simple workflow [19]. Here, we demonstrate the use of the Aquarium-enabled HIVDR test in a small laboratory in Nairobi, Kenya.
Methods
Study design
A Seattle team travelled to Nairobi, Kenya to set-up a testing site at the Coptic Hope Center for Infectious Diseases, a large-scale antiretroviral treatment site [20], to evaluate the utility and performance of Aquarium in guiding first-time users to perform the OLA-Simple kits. We recruited 12 laboratory technicians from the Coptic Hospital clinical laboratories to perform OLA-Simple from April 4–13, 2018. We estimated that a sample size of 12 technicians, each testing ten HIVDR codons, was adequate to assess test feasibility and performance and to obtain representative feedback on the participants’ perceptions related to the OLA-Simple kit and Aquarium digital instructions. This study was approved by the Institutional Ethics Review Committee (IERC) of the Aga Khan University in Kenya, and Seattle Children’s Research Institute’s IRB. All participants provided written informed consent.
Laboratory setup
The laboratory’s existing thermal cycler, biological safety hood, bench space, and refrigerator were utilized. The Seattle team brought the OLA-Simple kits, a minicentrifuge, micropipettes, scanner (CanoScan LiDE 300), tablets (Fire HD), foot controller, UPS battery backup and surge protector (APC 1500VA Compact), and server (Intel NuC NUC7i3BNH Mini PC/HTPC) to set up and run Aquarium. Aquarium code is publicly available [21].
Evaluation of OLA-Simple
Two technicians worked with the Seattle team each day, with all 12 completing the project over a period of six days. Each participating technician completed a demographic questionnaire and were given a 30-minute introduction to the principles and procedures used in the kit. Then each processed and tested two blinded blood samples. CD4+ cells were separated from 0.5 mL uninfected blood, lysed, and then spiked with mixtures of plasmids containing known HIVDR mutations. After PCR amplification of a region of HIV pol, mutation-specific probes were annealed, ligated and then the ligated products detected using lateral flow strips. The lateral flow strips were scanned, and the images displayed on the tablets were used by the participants to make visual calls and generate a report using Aquarium. Finally, the participants completed a questionnaire to give feedback on their experience with the kits and software.
Preparation of OLA-Simple kits
The OLA-Simple kit was prepared and assembled as previously described [19]. The EasySep CD4+ T-Cell isolation kit (STEMCELL Technologies, Vancouver, CA) was adapted to small blood volume processing, aliquoted and packaged in foil pouches. Reagents for PCR, ligation for detection of five HIV major NNRTI/NRTI resistance codons (K65R, K103N, Y181C, M184, and G190A) and lateral flow strips to detect ligation products were packaged in foil pouches with desiccant. Each kit component was labeled with a unique identifier, matching the images illustrated in Aquarium instructions.
Post-analysis of samples in Seattle
The DNA yield in lysed cells obtained by Kenyan participants was assessed in Seattle by qPCR of human beta globin [22]. The lateral flow images and Aquarium reports generated by each participant were used to assess assay performance and interpretation of visual results. Test accuracy was determined by comparison to the expected genotype at each codon analyzed. Lateral flow strips images were re-analyzed using an in-house Python script [19] to determine if automated analyses improved test accuracy. 95% confidence intervals (CI) are reported for all proportions.
Results and discussion
This work presents the first use of human-in-the-loop automated tutorial of laboratory technicians to enable a resource-limited laboratory to operate an HIVDR test with minimal training. The custom mobile application based on Aquarium operating system describes procedures and workflows for the OLA-Simple kits. Here we show that Aquarium digital guidance allowed first-time users in a small laboratory in Kenya to accurately perform the OLA-Simple kits and derive HIVDR genotypes. We also present and discuss the technicians’ feedback on their experience.
Participant characteristics
The 12 participating technicians were 83% male, median age 30 years old (range 26–42), had a median of 6 (range 3–10) years of experience as a laboratory technician, and all were conversant in English. Their education level ranged from secondary school with a certificate in Medical Laboratory Technology to a master’s degree in a Laboratory Science, which is representative of most clinical laboratories in Kenya. They reported varied levels of experience working with HIV or molecular techniques (Table 1).
Laboratory setup for Aquarium-assisted OLA-Simple training
The installation cost of OLA-Simple is low compared to automated sequencing platforms (> US$ 100,000) as it uses equipment that exists in most laboratories or is relatively inexpensive to acquire. Setting up the assay and software at the Coptic Hope Center laboratory required ~US$1,000 of additional equipment (scanner, microcentrifuge, vortexer, computer tablets, a server, and an uninterruptible power supply). For a laboratory without access to a thermal cycler, a battery-powered portable unit (~$500) can be used [23].
The laboratory was equipped with a Wi-Fi network to run Aquarium and coordinate assay steps across a pre-PCR and post-PCR room (Fig 1A). Participants worked in pairs to follow the OLA-Simple workflow (Fig 1B): sample preparation and PCR set-up were carried out in the pre-PCR room, while PCR, ligation and detection were conducted in the post-PCR room to minimize potential for amplicon-carryover contamination. Each participant processed two uninfected blood specimens and performed mutation testing on two contrived specimens with known HIV mutations following the interactive digital instructions provided by Aquarium.
(A) Laboratory setup: Tablets were connected to a local server to run Aquarium software. The pre-PCR room had a refrigerator/freezer and a BSC where CD4+ separation and PCR reaction were set-up. Technicians controlled the Aquarium-based software on the tablet with a foot pedal (not shown in the picture) while performing CD4+ preparation. The post-PCR room had two designated bench areas to set up ligation and detection separately. This room also contained a thermal cycler and a scanner. (B) Tasks assigned by Aquarium for Kenyan technicians, and assessment of their performance using Aquarium-assisted OLA-Simple. The sample preparation module was separated from the amplification, ligation, detection, and interpretation module. Associated protocol code, and examples of complete runs are publicly available (https://github.com/OLA-Simple/Papers-Vrana-Panpradist-et-al-2021).
To complete the workflow, it took each pair of participants an average of seven hours. This time included the introductory session, completion of surveys and staggering some assay steps due to space and instrument constraints. The turnaround time would likely decrease to 4.5–5 hours once users became familiar with the software and the kits.
Reported turnaround time for centralized HIVDR testing in resource-limited settings is 18 days [24] from sample collection, shipping to the laboratory, batching and testing samples, and transmission of results to the clinic. In this study, OLA-Simple was performed within the hospital grounds and thus could deliver test results within one day. Moreover, each kit tests two samples, a number suitable for the volume of daily or weekly patient samples submitted for HIVDR testing in small laboratories in Kenya, which eliminates the need to wait for an adequate number of specimens to batch for testing.
Performance of participants using OLA-Simple
All 12 participants successfully isolated CD4+ cells from whole blood collected from four donors with yields within the expected range (mean±SD: 686,450±216,500 CD4+ cells/mL) and completed genotyping of two samples using the OLA-Simple kits. The two blinded DNA samples tested by all participants included wild-type genotype only or mixtures of mutant and wild-type genotypes at each of five HIV reverse transcriptase codons, including K65R, K103N, Y181C, M184V and G190A (total of 10 codons/participant) (Fig 2A). Participants correctly genotyped 70/72 (97.2%, 95% CI: 90–100%) mutant codons and 38/48 (79.2%, 95% CI: 65–90%) wild-type codons. The only two false negatives were due to Participant #6 erroneously testing Sample 1 twice and omitting testing of Sample 2. Of 10 false positive results, 2 were due to testing Sample 1 in place of Sample 2 by Participant #6, 1 appeared to be contamination (light mutant signal at codon G190A likely from reusing a pipette tip contaminated with mutant ligation product) and 7 were due to light mutant background at codons K103N (n = 6) and K65R (n = 1). Analysis of the scanned images using our in-house image analysis software improved test accuracy to 97% (116/120, 95% CI: 92–99%) with 97% (70/72, 95% CI: 90–99%) mutant codons and 96% (46/48 95% CI: 86–99%) wild-type codons correctly genotyped (Fig 2B).
(A) Examples of scanned images of test strips for Samples 1 and 2 and test accuracy across each codon based on visual calls made by each participant and post-processing by image analysis software. Sample1 and Sample 2 have different mutation profiles. (B) Mutant (MUT) signal intensity of each lateral flow strip. Middle lines on the box plot indicate medians. Top and bottom lines on box plot indicate interquartile ranges. Dashed lines indicate the detection threshold for MUT signal. Both Sample 1 and Sample 2 have mutant genotype at codon Y181C. Diamonds correspond to signal from the strips of a Sample 1 that was erroneously tested in place of Sample 2.
The image analysis software improved accuracy over visual interpretation of results by establishing a signal threshold above the mutant background signal and thus eliminating false positives. The sample mix-up described above reduced the overall accuracy; this could be improved with changes in the kit labeling system to prevent this type of errors. Correcting for the sample that was added twice, the performance of the assay chemistry combined with software analysis yielded 100% accuracy (120/120, 95% CI: 97–100%).
Feedback from participants
Overall, participants scored the use of software as helpful in learning to perform the assay and enjoyed the clear instructions and interpretability of the results using the tablets. They strongly agreed that they understood the meaning of the bands on the strips, and that the Aquarium instructions were easy to follow (Table A in S1 File). However, several participants felt the procedure was lengthy and involved too many steps (Table B in S1 File summarizes the survey responses). In response, we have subsequently developed new chemistries to reduce assay time to 3.5 hours by replacing the 1.5-hour blood DNA preparation with a 30-minute plasma RNA extraction and the 2-hour PCR with 1-hour RT-PCR.
Our study shows it is feasible to use Aquarium to train local laboratory personnel with basic experience in lab work and onboard an HIVDR test. However, this pilot study was limited to the use of contrived specimens to establish analytical performance and each participant was able to perform the OLA-Simple kit once. Thus, we were not able to assess the ability of each technician to maintain or improve their skills. A larger demonstration and evaluation study that includes HIV-infected clinical specimens and multiple OLA-Simple runs is ongoing in Kenya to assess the robustness of our HIVDR test.
The Aquarium software has useful features for HIVDR testing such as automatic collection of operator interactions in Aquarium’s virtual laboratory notebook that can be useful for troubleshooting. Each kit item is labeled with a unique identifier that Aquarium instructions use in conjunction with corresponding pictures to avoid ambiguity and reduce the extent of in-person training needed. Uniquely labeled items also allow tracking of stock consumption in real-time which can be useful and timesaving for laboratory management. Importantly, Aquarium could link test results to treatment algorithms to advise clinicians, and algorithms could be updated as clinical recommendations or policies change.
Conclusion
Aquarium-based software facilitated deployment of the OLA-Simple in Kenya with minimal training, enabling local laboratory technicians to perform an HIVDR test. Human-in-the-loop automation could facilitate daily operations of laboratory-based assays and increase the performance and accuracy of diagnostic tests in small laboratories.
Acknowledgments
We are grateful to the Coptic Hospital and the Coptic Hope Center for their support and to the technologists from these institutions who participated in this study. We thank Syamal Raychauduri, Dindo Reyes, Kim Polizzi, and Jessica Price at InBios International, Inc. for producing the lateral flow strips used in the validation study; and Aarthy Vallur and Kathryn Hjerrild of InBios International, Inc. for providing feedback on OLA-Simple prototypes.
References
- 1. Sarkozi L, Simson E, Ramanathan L. The effects of total laboratory automation on the management of a clinical chemistry laboratory. Retrospective analysis of 36 years. Clin Chim Acta. 2003;329(1–2):89–94. Epub 2003/02/19. pmid:12589970.
- 2. Holland LL, Smith LL, Blick KE. Total laboratory automation can help eliminate the laboratory as a factor in emergency department length of stay. Am J Clin Pathol. 2006;125(5):765–70. Epub 2006/05/19. pmid:16707380.
- 3. Da Rin G, Zoppelletto M, Lippi G. Integration of Diagnostic Microbiology in a Model of Total Laboratory Automation. Lab Med. 2016;47(1):73–82. Epub 2015/12/31. pmid:26715609.
- 4. Miler M, Nikolac Gabaj N, Dukic L, Simundic AM. Key Performance Indicators to Measure Improvement After Implementation of Total Laboratory Automation Abbott Accelerator a3600. J Med Syst. 2017;42(2):28. Epub 2017/12/28. pmid:29280007.
- 5. Chung HJ, Song YK, Hwang SH, Lee DH, Sugiura T. Experimental fusion of different versions of the total laboratory automation system and improvement of laboratory turnaround time. J Clin Lab Anal. 2018;32(5):e22400. Epub 2018/02/27. pmid:29479855; PubMed Central PMCID: PMC6817108.
- 6. Bolduc S. Sputum collection and transport in Africa: perspectives from Mozambique—part 1: Challenges DNA Genotek’s Infectious Disease Blog 2016 [cited 2021 January 1st].
- 7.
UNAIDS. Global HIV & AIDS statistics—2020 fact sheet. 2020.
- 8. Kulkarni S, Jadhav S, Khopkar P, Sane S, Londhe R, Chimanpure V, et al. GeneXpert HIV-1 quant assay, a new tool for scale up of viral load monitoring in the success of ART programme in India. BMC Infect Dis. 2017;17(1):506. Epub 2017/07/25. pmid:28732472; PubMed Central PMCID: PMC5521114.
- 9. Manoto SL, Lugongolo M, Govender U, Mthunzi-Kufa P. Point of Care Diagnostics for HIV in Resource Limited Settings: An Overview. Medicina (Kaunas). 2018;54(1). Epub 2018/10/23. pmid:30344234; PubMed Central PMCID: PMC6037236.
- 10.
US. Food and Drug Administration. Evaluation of Automatic Class III Designation for Sentosa® SQ HIV-1 Genotyping Assay and Associated SENTOSA® NGS (Next Generation Sequencing) Workflow System. 2020.
- 11. May S, Adamska E, Tang J. Evaluation of Vela Diagnostics HIV-1 genotyping assay on an automated next generation sequencing platform. J Clin Virol. 2020;127:104376. Epub 2020/04/29. pmid:32344322.
- 12. Raymond S, Nicot F, Abravanel F, Minier L, Carcenac R, Lefebvre C, et al. Performance evaluation of the Vela Dx Sentosa next-generation sequencing system for HIV-1 DNA genotypic resistance. J Clin Virol. 2020;122:104229. Epub 2019/12/07. pmid:31809945.
- 13. Dessilly G, Goeminne L, Vandenbroucke AT, Dufrasne FE, Martin A, Kabamba-Mukadi B. First evaluation of the Next-Generation Sequencing platform for the detection of HIV-1 drug resistance mutations in Belgium. PLoS One. 2018;13(12):e0209561. Epub 2019/01/01. pmid:30596682; PubMed Central PMCID: PMC6312258.
- 14. Kingwara L, Karanja M, Ngugi C, Kangogo G, Bera K, Kimani M, et al. From Sequence Data to Patient Result: A Solution for HIV Drug Resistance Genotyping With Exatype, End to End Software for Pol-HIV-1 Sanger Based Sequence Analysis and Patient HIV Drug Resistance Result Generation. J Int Assoc Provid AIDS Care. 2020;19:2325958220962687. Epub 2020/09/30. pmid:32990139; PubMed Central PMCID: PMC7536479.
- 15. Duarte HA, Beck IA, Levine M, Kiptinness C, Kingoo JM, Chohan B, et al. Implementation of a point mutation assay for HIV drug resistance testing in Kenya. AIDS. 2018;32(16):2301–8. Epub 2018/07/14. pmid:30005020; PubMed Central PMCID: PMC6170675.
- 16. Panpradist N, Beck IA, Ruth PS, Avila-Rios S, Garcia-Morales C, Soto-Nava M, et al. Near point-of-care, point-mutation test to detect drug resistance in HIV-1: a validation study in a Mexican cohort. AIDS. 2020;34(9):1331–8. Epub 2020/03/25. pmid:32205723.
- 17. Panpradist N, Beck IA, Chung MH, Kiarie JN, Frenkel LM, Lutz BR. Simplified Paper Format for Detecting HIV Drug Resistance in Clinical Specimens by Oligonucleotide Ligation. PLoS One. 2016;11(1):e0145962. Epub 2016/01/12. pmid:26751207; PubMed Central PMCID: PMC4713472.
- 18. Keller B, Vrana J, Miller A, Newman G, Klavins E. Aquarium: The Laboratory Operating System (Version v2.5.0). Zenodo http://doiorg/105281/zenodo2535715. 2019.
- 19. Panpradist N, Beck IA, Vrana J, Higa N, McIntyre D, Ruth PS, et al. OLA-Simple: a software-guided HIV-1 drug resistance test for low-resource laboratories. EBiomedicine. 2019; 50:34–44. pmid:31767540
- 20. Chung MH, Drake AL, Richardson BA, Reddy A, Thiga J, Sakr SR, et al. Impact of prior HAART use on clinical outcomes in a large Kenyan HIV treatment program. Curr HIV Res. 2009;7(4):441–6. Epub 2009/07/16. pmid:19601781; PubMed Central PMCID: PMC2735680.
- 21.
Vrana J. OLA Simple HIV Drug Resistance Test: An Aquarium Workflow Github.com2020 [cited 2021 January 2nd]. Available from: klavinslab.github.io/ola-simple/.
- 22. Zunt JR, Montano SM, Beck I, Alarcon JO, Frenkel LM, Bautista CT, et al. Human T-lymphotropic virus type 1-associated myelopathy/tropical spastic paraparesis: viral load and muscle tone are correlated. J Neurovirol. 2006;12(6):466–71. Epub 2006/12/13. pmid:17162662; PubMed Central PMCID: PMC2695973.
- 23. Panpradist N, Wang Q, Ruth PS, Kotnik JH, Oreskovic AK, Miller A, et al. Simpler and faster Covid-19 testing: Strategies to streamline SARS-CoV-2 molecular assays. EBioMedicine. 2021;64:103236. Epub 2021/02/15. pmid:33582488.
- 24. Lessells RJ, Avalos A, de Oliveira T. Implementing HIV-1 genotypic resistance testing in antiretroviral therapy programs in Africa: needs, opportunities, and challenges. AIDS Rev. 2013;15(4):221–9. Epub 2013/12/11. pmid:24322382; PubMed Central PMCID: PMC3951902.