Diagnostic performance and usability of the VISITECT CD4 semi-quantitative test for advanced HIV disease screening

Zibusiso Ndlovu; Lamin Massaquoi; Ndim Eugene Bangwen; John N. Batumba; Rachelle U. Bora; Joelle Mbuaya; Roger Nzadi; Nadine Ntabugi; Patrick Kisaka; Gisele Manciya; Ramzia Moudashirou; Harry Pangani; Patrick Mangochi; Roberta Makoko; David Van Laeken; Claude Kwitonda; Yuster Ronoh; Kuziwa Kuwenyi; Reinaldo Ortuno; Douglas Mangwanya; Edmore Zvidzai; Tapiwa Mupepe; Sekesai Zinyowera; Emmanuel Fajardo; Tom Ellman

doi:10.1371/journal.pone.0230453

Peer Review History

Original SubmissionNovember 3, 2019
24 Dec 2019 Decision Letter - Bharat S. Parekh, Editor PONE-D-19-30572 Diagnostic performance and usability of the VISITECT CD4 semi-quantitative test for advanced HIV disease screening PLOS ONE Dear Mr Ndlovu, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but requires some revision as per reviewers' comments. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. We would appreciate receiving your revised manuscript by Feb 07 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. We look forward to receiving your revised manuscript. Kind regards, Bharat S. Parekh, Ph.D. Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Yes ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: 1) I suggest including your reference method in the abstract. 2) On line 176 of the methods section, the following sentence was cut-off “The Visitect CD4 LFA test kit can be stored” 3) When assessing the performance of Visitect CD4 LFA at different reference test cut-offs, I recommend including the number of samples compared at each cut-off. 4) On line 223, the authors state “In restricting the analysis to samples with CD4 < 100 cellsmm3 cut-off in the reference test, the sensitivity of the Visitect CD4 LFA was 98% [IQR: 93.1 – 99.8] and specificity was 66.6% [IQR: 62.2 – 70.8].” If the analysis was truly restricted to reference values < 100 cells/mm3 then there would be no false-positives from the test assay because there would be no reference values > 200 cells/ml. Was this analysis instead performed by changing the cut-off of the reference assay? I recommend a better explanation of this analysis here or in the methods section. 5) The authors shared the Kappa statistic of the Visitect CD4 LFA compared to the Pima CD4, but not the specificity or sensitivity. I recommend sharing the sensitivity and specificity with the Pima CD4 as the reference assay, or an explanation of why the authors chose to use venous blood tested on the BD FACScount as the reference assay for FP samples tested on the Visitect CD4 LFA. Reviewer #2: A few suggestions to strengthen the manuscript: 1. The authors use sensitivity and specificity to describe the correlation between the different methods at the threshold of 200, which introduces some confusion as compared with test evaluations that compare positives and negatives as is done in most method comparisons. It might be more clear to describe the relationship as a correlation or concordance of methods at >200 or <200. Alternatively, it could be stated as the sensitivity or accuracy to detect <200 and the sensitivity or accuracy to detect >200. 2.Precision studies: Were any samples used that produced an equal result intensity with the 200 line? That is, precision studies should include a sample near the cutoff where changes could be realized. The nature of the samples used is not described in the Methods Section. 3. Reproducibility study: What is described is not a reproducibility study; it describes an experiment where incubation times are varied and the effect on results is observed. The authors should consider to remove this because it is small in scope, not relevant for the user because they are supposed to follow instructions, and it might give the impression that incubation times can be varied without having an effect on test results. If it were to be included, then perhaps there should be other investigations also (e.g., varying the number of drops of buffer, using partial volumes of sample, etc.) to be more complete. A true reproducibility study shows how the test performs with multiple testing of the same samples over hours, days, or longer, and with varying conditions (different users, different temperatures on days, etc.). 4. Line 176 is a partial sentence. 5. Line 253 - the ambient temperature should be stated (in some resource limited countries it could be 35C). Stability could also mean how the test performs after storage at different temperatures over time. Therefore, it should refer to "stability of the observed results". 6. In the early years of 2000, there was a method called TRAxCD4 (T Cell Dxs/Innogenetics) that was an ELISA based method that measured soluble CD4 protein in a sandwich configuration (similar to the principle in this LFA). The authors should review the literature and include any references in their manuscript. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: Yes: Niel Constantine, Ph.D. [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0230453.r001
Revision 1
13 Jan 2020 Author Response We would like to thank the reviewers for their time and concise inputs. This study was initially submitted to PLOS-One with results from two study sites (Malawi and DRC); however, the study in Zimbabwe has also been finished recently and we have herein added the results from Zimbabwe for 118 patients tested in Phase 1. We also added the co-investigator details from Zimbabwe and also added the Ethical Approvals obtained from the Zimbabwe Medical Research Council. Responses to reviewer comments Reviewer #1: Reviewer comment 1: I suggest including your reference method in the abstract. Author response: line 66 and 67 of the abstract do mention the reference test method used in the study (i.e BD FACSCount assay for CD4 testing) Reviewer comment 2: On line 176 of the methods section, the following sentence was cut-off “The Visitect CD4 LFA test kit can be stored” Author response: We have corrected the sentence and it now reads; “The Visitect CD4 LFA test kit can be stored within temperature ranges of 2 - 30oC” (line 195). Reviewer comment 3: When assessing the performance of Visitect CD4 LFA at different reference test cut-offs, I recommend including the number of samples compared at each cut-off. Author response: We have added the number and percentage of samples for each of the cut-off ranges (line 248, table 3) Reviewer comment 4: On line 223, the authors state “In restricting the analysis to samples with CD4 < 100 cellsmm3 cut-off in the reference test, the sensitivity of the Visitect CD4 LFA was 98% [IQR: 93.1 – 99.8] and specificity was 66.6% [IQR: 62.2 – 70.8].” If the analysis was truly restricted to reference values < 100 cells/mm3 then there would be no false-positives from the test assay because there would be no reference values > 200 cells/ml. Was this analysis instead performed by changing the cut-off of the reference assay? I recommend a better explanation of this analysis here or in the methods section. Author response: We have modified the sentence to reflect that the sensitivity/specificity of Visitect CD4 LFA at the different cut-offs is purely based on the reference assay cut-offs, whilst the Visitect CD4 LFA remains with its standard threshold of 200cells/mm3. Line 242-244. Furthermore, we avoid mentioning the words ‘false positive/false negative’ during the interpretation of the results of this restriction analysis. Reviewer comment 5: The authors shared the Kappa statistic of the Visitect CD4 LFA compared to the Pima CD4, but not the specificity or sensitivity. I recommend sharing the sensitivity and specificity with the Pima CD4 as the reference assay, or an explanation of why the authors chose to use venous blood tested on the BD FACScount as the reference assay for FP samples tested on the Visitect CD4 LFA. Author response: we used the Kappa statistic to compare diagnostic performance of Visitect CD4 LFA to PIMA (and not the sensitivity/specificity) because PIMA is not a reference standard method for CD4 cell count testing, as such Kappa statistic is the most correct/recommended way to assess the agreement between two methods (where there is no reference/gold standard) as both are considered imperfect tests. Reviewer #2: Reviewer comment 1: The authors use sensitivity and specificity to describe the correlation between the different methods at the threshold of 200, which introduces some confusion as compared with test evaluations that compare positives and negatives as is done in most method comparisons. It might be more clear to describe the relationship as a correlation or concordance of methods at >200 or <200. Alternatively, it could be stated as the sensitivity or accuracy to detect <200 and the sensitivity or accuracy to detect >200. Author response: Yes, we understand the potential ‘confusion’ that could arise with interpreting results for such a binary-test which doesn’t necessarily provide the traditional ‘positive’ or ‘negative’ results. The consequence is that Visitect CD4 LFA results that are <200 can loosely be called ‘positive’ whilst those >200 can be said to be ‘negative’. In our study and in data analysis packages, as long as diagnostic result outcomes are dichotomous/binary, they are automatically labeled as ‘normal’ and ‘abnormal’, which enables determination of sensitivity/specificity to a reference standard method. This is regardless of whether the result is positive/negative or is <200/>200. Nonetheless, in the manuscript, we do emphasize (in multiple occasions) that the calculated outcomes of performance are sensitivity/specificity to detect CD4 cell counts at the 200cells/mm3 threshold (meaning either >200 or less than 200), line 230-232. Reviewer comment 2: Precision studies: Were any samples used that produced an equal result intensity with the 200 line? That is, precision studies should include a sample near the cutoff where changes could be realized. The nature of the samples used is not described in the Methods Section. Author response: Line 199-201 in Methodology section does partly mention the strategy for assessing the precision of the index test. In line 250-253 (results section), we do mention the outcomes of precision assessment, and the fact that we used 12 samples and 2 had CD4 cell counts close to the 200cells/mm3 threshold. Nonetheless, we certainly do acknowledge the need to describe the samples used for precision; however, since we used a range of samples (in terms of CD4 cell counts) we could not manage to describe all the sample types used within the methodology/results section. Reviewer comment 3: Reproducibility study: What is described is not a reproducibility study; it describes an experiment where incubation times are varied and the effect on results is observed. The authors should consider to remove this because it is small in scope, not relevant for the user because they are supposed to follow instructions, and it might give the impression that incubation times can be varied without having an effect on test results. If it were to be included, then perhaps there should be other investigations also (e.g., varying the number of drops of buffer, using partial volumes of sample, etc.) to be more complete. A true reproducibility study shows how the test performs with multiple testing of the same samples over hours, days, or longer, and with varying conditions (different users, different temperatures on days, etc.). Author response: We acknowledge that our reproducibility part of the study is limited in scope and the idea behind it was purely to explore the effect of varying the incubation times only; as we pragmatically know that this will be one of the biggest independent factors to affect test results in different real user settings. Practically, we know health-care workers sometimes do not follow the instructions for use (IFU) for many POC diagnostics and with this multi-step test; we sought to show the potential consequences of not following the instructions. We do acknowledge that our findings may be taken to mean that the incubation times can be changed, but we specifically sought to provide a clue on this question that these incubation times are un-forgiving and testers must adhere to manufacture times. To simplify the text in the manuscript, we have removed part of the reproducibility data (Table 4) to ensure limited information remains; and the Discussion section (line 339-347) provide a clear emphasis on following manufacture incubation times. Reviewer comment 4: Line 176 is a partial sentence. Author response: We have corrected the sentence and it now reads; “The Visitect CD4 LFA test kit can be stored within temperature ranges of 2 - 30oC” (line 195). Reviewer comment 5: Line 253 - the ambient temperature should be stated (in some resource limited countries it could be 35C). Stability could also mean how the test performs after storage at different temperatures over time. Therefore, it should refer to "stability of the observed results". Author response: we have modified this and clarified that the ambient laboratory temperature range was 24–28oC. Line 269 Furthermore, we have added the word ‘result re-reading’ in the subtitle (line 268) to emphasize that the stability results are just from re-reading the stored test cassettes only. Reviewer comment 6: In the early years of 2000, there was a method called TRAxCD4 (T Cell Dxs/Innogenetics) that was an ELISA based method that measured soluble CD4 protein in a sandwich configuration (similar to the principle in this LFA). The authors should review the literature and include any references in their manuscript. Author response: We have done a literature review of this TRAXCD4 technology and many others during the early 2000 and we have added a reference of one of the review articles on this (reference number 7). We however, did not add the whole technology appraisal from early 2000s in detail in the manuscript as most of these technologies are obsolete. Also we wanted our Introduction Section to focus in the currently commercially available point-of-care (POC) or near-POC CD4 assays in the market (line 110-113). Attachments Attachment Submitted filename: Rebutal letter_PONE-D-19-30572.doc https://doi.org/10.1371/journal.pone.0230453.r002
2 Mar 2020 Decision Letter - Bharat S. Parekh, Editor Diagnostic performance and usability of the VISITECT CD4 semi-quantitative test for advanced HIV disease screening PONE-D-19-30572R1 Dear Dr. Ndlovu, We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements. Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication. Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. With kind regards, Bharat S. Parekh, Ph.D. Academic Editor PLOS ONE Additional Editor Comments (optional): Thank you for addressing reviewers' comments. Manuscript is now acceptable for publication. Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #1: (No Response) ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: No ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: No ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: 1) Again, on line 242, the authors state, "In restricting the analysis to samples with CD4<100cells/mm3 cut-off in the reference test, the sensitivity of the Visitect CD4 LFA (at its standard 200cells/mm3 threshold) in detecting samples with CD4<100cells/mm3 was 98.1% [IQR: 93.5 – 99.8] and specificity was 68.0% [IQR: 64.1 – 71.7]". If you truly restricted the analysis to samples with CD4 < 100 cells/mm3, the specificity would be 0% as there would be 0 samples > 100 cells/mm3 for the reference assay. I suspect the authors analyzed all samples (less than and greater than the cut-offs), however varied the reference test cut-offs. If this is true, changing this sentence to say, "In analyzing samples at varying reference test cut-offs, the sensitivity of the Visitect CD4 LFA..." would render it accurate. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No https://doi.org/10.1371/journal.pone.0230453.r003
Formally Accepted
5 Mar 2020 Acceptance Letter - Bharat S. Parekh, Editor PONE-D-19-30572R1 Diagnostic performance and usability of the VISITECT CD4 semi-quantitative test for advanced HIV disease screening Dear Dr. Ndlovu: I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. For any other questions or concerns, please email plosone@plos.org. Thank you for submitting your work to PLOS ONE. With kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Bharat S. Parekh Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0230453.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .