Equipment

Calibrated micro-pipettes used were Rainin pipet-lite or pipet-lite XLS L2, L20, and/or L200. Calibrated digital timers were VWR® Traceable® Four-Channel Alarm Timer with Clock Model 62344–641. Calibrated thermometers were VWR® Traceable® Econo Refrigerator Thermometer Model 36934–132. Digital camera (single lens reflex) Nikon D70 was fitted with AFS Nikor 18-70mm 1:3.5–4.6G ED lens. A light box (Finnhomy Professional Portable Photo Studio Photo Light Studio Photo Tent Light box Table Top Photography Shooting Tent Box Lighting Kit, 16” x 16” Cube) was used and the light output was measured by Sekonic Spectromaster C-800-U: CCT = 5599k, Ra = 83.9, Lux = 9050lx, x = 0.3302 and y = 0.3464. Specimen Storage Tubes (0.5mL polypropylene screw cap cryovial (Sarstedt)) were used and provided by the Fred Hutchinson Cancer Research Center, where long-term specimen inventories are maintained. White background for reading tests was standard white printer/copier paper.

Determining the number of positives/ negatives specimens for qualification

The number of specimens required for testing HIV RDTs, with appropriate acceptance criteria, was developed through a review of ISO 2859–1:1999 [27]. ISO 2859–1:1999 provides different sampling plans as a function of variables such as lot size, acceptance quality limit, and inspection level. Lot size refers to the number of units manufactured in a production lot. Acceptance quality limit (AQL) represents the percent level of defects in a lot that are tolerated or allowed by the end user. Depending on the product, HIV RDTs can generally be considered to have specificities and sensitivities of ~99%, which could correspond to an AQL of approximately 1%. ISO 2859–1:1999 provides seven inspection levels that are divided into general (III, II, I) and special (S-4, S-3, S-2, and S-1) categories. Level II is considered the default level, where consideration of other options is a function of the level of discrimination needed and on sample size logistics (III is the highest / S-1 is the lowest). Each sampling plan provides accept/reject criteria that correspond to the maximum number of failures that are acceptable and the minimum number of failures that reject the lot, respectively. For each sampling plan, ISO 2859–1:1999 provides information that allows one to determine an operating characteristic curve, which is a plot of the likelihood of lot acceptance/rejection during testing as a function of true percent non-conformity of the product through continual manufacturing. The following discussion is focused on single sampling plans (for normal inspection); ISO 2859–1:1999 provides options (more advanced circumstances) for double sampling plans and reduced or tightened inspection.

Different sampling plans were compared across several variables, namely lot size, acceptance quality limit, and inspection level. For lot size, a range of 35,001–500,000 units is understood to cover relevant sizes for RDTs. Acceptance quality limits of 0.65, 0.40, 0.25, and 0.15 were considered to provide options that are under a 1% defect level (to maintain ~99% sensitivity/specificity). Inspection Levels II, I, S-4, and S-3 were reviewed to encompass the default II level and more practical levels (lower amounts) based on sample availability. Also, lower sampling options were considered due to the difficulty of obtaining and maintaining the appropriate positive/negative specimens. Additionally, this lot testing approach is in the context of being conducted on product that has ostensibly been approved and released by the manufacturer’s standard quality control procedures, thereby allowing this approach to have less rigidity than a strict interpretation of ISO 2859–1:1999 and allowing the same approach to be broadly applied to RDTs, regardless of manufacturer, lot size, or product design (e.g., sensitivity or specificity).

Due to limited specimen availability, plans that utilize 200 samples or less for the entire lot size range were considered as viable options and are listed as follows (AQL, inspection level, number of specimens, accept / reject ratio): (1) 0.65, S-4, 80, 1/2; (2) 0.65, S-3, 20, 0/1; (3) 0.40, S-4, 125, 1/2; (4) 0.40, S-3, 32, 0/1; (5) 0.25, S-4, 50, 0/1; (6) 0.15, S-4, 80, 0/1). From ISO 2859–1:1999, operating characteristic curves for these respective options were plotted (Fig 1) at a 20% (full scale) and 1% (magnified scale) true percent non-conformity level. From these curves, any of the plans would perform well with catastrophic product failure because all would reject at least 80% of the lots at greater than 10% non-conformity. However, HIV RDTs were performing at an overall approximate non-conformity level of ~0.1–0.2% (percent of discordant RDT results with applied positive and negative specimens) based on initial data from the United States Agency for International Development’s Global Health Supply Chain Quality Assurance program (implemented by FHI 360). Therefore, sampling plan 2 (0.65, S-3) is not recommended for HIV RDTs because it is the least stringent at a 2–3% or greater non-conformity level. Sampling plans 4 (0.40, S-3), 5 (0.25, S-4), and 6 (0.15, S-4) are not recommended either because 5% or more of lots are expected to be rejected at a 0.1–0.2% non-conformity level (too stringent). Therefore, plan 1 (0.65, S-4) was chosen as the optimal plan for HIV RDTs. Relative to plan 3 (0.40, S-4), plan 1 (0.65, S-4) would reject fewer lots at the 0.1% - 0.2% non-conformity level while also requiring 45 fewer samples to conduct the evaluation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Operating characteristic curves from ISO 2859–1: 1999 for the lot size range of 35,001–500,000 for various sampling options (AQL, inspection level).

For each AQL / inspection level provided in the figure, the respective number of samples and accept/reject ratio is listed as follows for single sampling plans at normal inspection: (1) 0.65, S-4, 80, 1/2 (green highlighted selected for use); (2) 0.65, S-3, 20, 0/1; (3) 0.40, S-4, 125, 1/2; (4) 0.40, S-3, 32, 0/1; (5) 0.25, S-4, 50, 0/1; (6) 0.15, S-4, 80, 0/1). Operating characteristic curve data in ISO-2859-1: 1999 is provided as expected percentage of lots to be accepted (%A) but presented here as an expected percentage of lots rejected (%R = 100 - %A) to visually correlate increases in percent non-conformity with increases in lot rejection.

https://doi.org/10.1371/journal.pone.0284175.g001

The selection of plan 1 (0.65, S-4) would require 80 positive specimens and 80 negative specimens per lot of HIV RDTs evaluated. The more ideal approach is to test each lot with 80 unique specimens in single (both for positives and negatives), however, an alternate approach could be to test with 20 specimens in quadruplicate (for both positives and negatives) so that 80 total RDTs are tested each with positive and negative specimens. While the former approach is preferred in terms of maximum specimen diversity, the latter may be necessary depending on specimen cost and availability. This sampling and testing scheme (utilizing an AQL of 0.65) could be considered consistent with a product providing an overall sensitivity and specificity each of approximately 99.35% (or higher). From a review of operating characteristic curves (see Fig 1) [27], this evaluation scheme can be expected to screen out over 90% of the lots that exhibit true sensitivity and/or specificity of 95% or lower. Furthermore, there is a high probability (>80%) that lots will be found acceptable when operating at a true sensitivity and/or specificity of 99% or higher. For this evaluation scheme, the lower bounds on specificity / sensitivity when testing one lot are estimated in Table 1 [28–30] for the respective number of defects and confidence levels for testing scenarios involving either 80 specimens with 1 replicate each or 20 specimens with 4 replicates each. The higher level of the lower bounds for 80 unique specimens (1 replicate) indicates a closer correlation for results per lot with generally expected sensitivity/specificity levels over time relative to using 20 specimens (4 replicates), but the latter option is considered viable depending on the circumstances.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Lower bounds on specificity and sensitivity, estimated through the Wilson scores approach [28–30], for the respective number of defects and confidence levels for testing scenarios involving either 80 specimens (with 1 replicate each) or 20 specimens (with 4 replicates each).

https://doi.org/10.1371/journal.pone.0284175.t001

Specimen sources

Human serum specimens were purchased from ProMedDx, LLC with available characterizations for individual positive and negative samples provided in respective certificates (i.e., serialized identifier, gender, age, and available test results). The specimens are collected from US donors and thus the positive specimens are not necessarily representative of the global genetic diversity of HIV and the negative specimens are not necessarily representative of the intended RDT target population. Donor age, gender, and race were not considered for inclusion in the specimen set. It is highly probable that due to the demographics of the positive specimen donor population (previously diagnosed HIV positive) that most or all donors may actively be on HIV antiviral treatment and are not necessarily representative of the intended RDT target population. As a contingency for an unknown level of unexpected results, and to meet the ideal ISO sampling plan of 80 specimens, more than 80 positives and 80 negatives were selected for the qualification. For HIV-1 positive serum evaluated through ProMedDx, 83 unique samples (30 mL of each in 3 separate 10 mL vials) were selected based on the highest volumes (mL) per sample available and on “Reactive” results for qualitative tests (Abbott EIA and PRISM ChLia) or positive indicating numerical results for qualitative tests (Abbott Architect and Siemens Centaur). All positive samples were selected with the above criteria and with also having HIV viral load data provided in the ProMedDx certificate from the Roche PCR test, tested “Reactive” on the Abbott HIV AB HIV-1/HIV-2 (rDNA) EIA, and tested “Non-Reactive” for HBsAg and HCV on Abbott PRISM (ChLIA). For negative serum, 84 unique samples (~175 mL of each in a single bulk container) were purchased with certified information with “Non-Reactive” results with PRISM ChLia (Viral testing for HIV, HBsAg, HCV), “Non-Reactive” results with Roche Cobas TaqScreen (NAT Testing for HIV, HBV, HCV), and “Non-Reactive” results for RPR (syphilis) with the Arlington Scientific Card Test. All samples from ProMedDx were additionally labeled with a simplified code (correlated with the serialized identifier from ProMedDx within internal database). Positives were coded as P001PM through P083PM and negatives were coded as N001PM through N084PM. The WHO International Standard HIV-1 p24 Antigen NIBSC code: 90/636 was used as the antigen positive (Ag) sample and diluted in negative serum.

HIV RDTs for panel qualification

Various products were obtained from the respective manufacturers for use in panel qualification. Approximately 1000 units from each of 3 lots were obtained for use from Determine HIV-1/2 (Alere, Product number: 7D23), HIV Combo (Alere, Product number: 7D2843), Uni-Gold HIV (Trinity Biotech, Product number: 1206502), SD Bioline HIV-1/2 3.0 (Standard Diagnostics, Product number: 03K10, 03FK16), and OraQuick HIV-1/2 (OraSure, Product number: 5X4-0010). Product names will not be provided in correlation with the Ab results but will rather be referenced with a product code to protect manufacturer confidentiality. HIV Combo will be directly referenced when discussing Ag results because it was the only product utilized that could detect Ag.

Panel qualification

Because long-term storage conditions are required at -80°C (approximately -70°C to -90°C), sample specimens were obtained from the respective suppliers in bulk containers that were received frozen (packed in dry ice). Specimens need to be thawed to room temperature for actual testing of RDTs, however frequent freeze-thaw cycles will degrade specimen integrity. Therefore (after receipt from the supplier), a portion of each specimen was thawed at room temperature and aliquoted to smaller volumes for long-term storage at -80°C so that only the volume needed for testing can be removed for use to incorporate a single thaw cycle. Positives already came in 3 separate 10 mL bulk containers and only one container was thawed for further aliquoting. Negative specimens came in ~175 mL units that were thawed and frozen in multiple 10 mL containers, where only one 10 mL container per specimen was aliquoted and used for qualification. For this panel qualification, an optimal aliquot level was determined to yield minimal wastage of specimen volume over time based on manufacturer instructions outlining the volume required for testing and on the number of lots that can be tested. The following volumes of plasma/serum are needed for a single analysis with each of the listed HIV RDTs (Determine—50 μl [31, 32]; Uni-Gold–one drop (~ 50 μl) [33]; SD Bioline—10 μl [34]; OraQuick—~5+ μl [35]). Various aliquot amounts were considered, but the 250 μl aliquoting volume was selected based on input from industry representatives to minimize impact from potential volume loss over time.

For specimen aliquoting, the commercially sourced sample (10 mL container) was thawed at 23°C or cooler (without the use of a heating block). The thawed sample was adequately mixed (i.e., via vortexing or inversion over 10 times) and then aliquoted into separate 250 μl portions contained in specimen storage tubes. Each separate specimen aliquot was given a label that links or correlates the source code provided by the commercial source. All specimen aliquots were returned to appropriate freezer conditions until ready for use. It should be noted that each 250 μl aliquot is only to be used as a single thaw sample but can be used multiple times if stored at 2–8°C for 5 days.

For each lot of RDTs received, 4 replicates of each specimen (positive and negative) were conducted, thus yielding 12 replicates across each RDT type per specimen. The replicates establish performance consistency from test-to-test and lot-to-lot. All testing was conducted according to the section outlining the standard procedure below with adaptations to specific manufacturer’s instructions for use for each product. The implementation of four replicates per lot/specimen combination is conducted to allow flexibility in using the panel for under 80 specimens / single application or 20 specimens / 4 replicate approach. A given specimen and lot are considered qualified provided 100% concordance is obtained from the results, meaning no false positives or false negatives (also screening for invalid tests) are observed. Qualified lots can subsequently be used as reference (control) lots during routine testing.

When the test was read at the required time, a qualitative intensity was noted for the test line with one of the following options: 0 –negative; w–test line is faintly observable; 1 –test line intensity is weaker than the control line; 2 –test line intensity is comparable to the control line; 3—test line intensity is stronger than the control line. Two technicians independently interpreted and recorded the test results. In the case of discrepant quantitative results, a third independent reader interpreted the tests. For information and general documentation purposes, digital images were obtained for test RDTs between the minimum / maximum read time (essentially within several minutes of when the visual judgement is made for the set of samples). Digital images were taken at the time of testing.

Standardized procedure to evaluate HIV RDTs with qualified panels

For each RDT production lot to be evaluated, a data collection form is prepared that indicates the specific panel to be used for the testing, where the specimens identified are thawed (at room temperature without use of heating block) for use that day. The remaining specimen is not refrozen for later use but may be stored at 2–8°C for 5 days for future use. A total of 80 HIV positive specimens (81 if a combination Ab/Ag test is being evaluated) and 80 HIV negative specimens are used to evaluate each production lot. Specimen thawing should be coordinated with the number of lots that could readily be completed in a single day so that the proper set of panels are thawed and used that day. RDTs should be placed on the laboratory bench top that is covered with a white background (i.e., white laboratory paper towels or bench pads) to provide a uniform background for result interpretation. A maximum of 10 RDTs are to have specimens applied at the same time. Each RDT (unit) is labeled to indicate the specimen that will be used for the evaluation of that specific unit, referencing the data collection form for lot under evaluation. RDTs shall be tested according to the package insert/instructions for use (IFU). Specific items of note are the following: the volume of specimen and method for specimen application, volume of kit buffer, and minimum / maximum reading time after specimen application. Deviations from the package insert/IFU must be recorded and explained.

Each specimen should be thoroughly mixed (e.g., vortexed for approximately 2 seconds) and subsequently applied to the test device at the specified volume. If the product instructions allow for specimen addition via a specimen transfer device not included in the kit, specimen addition using a calibrated micropipette is preferred. If the procedure calls for the addition of a buffer, the buffer is applied according to the package insert. A timer is started after the addition of specimen or buffer (if used) to the first RDT (of the group selected for evaluation during the same time period). A single timer can be used for the evaluation of a set of devices (maximum of 10 at a time); the use of multiple timers allows for parallel evaluation of multiple sets of devices. The laboratory environment should use general overhead lighting and not use directed incandescent lighting to prevent inadvertent heating of samples/kits during testing. Ideal lighting would occur from general overhead fluorescent or LED light at a range of approximately 2000–3000 Lux. Results are read during the specified time period indicated in the package insert. The time of reading (relative to the sample application time) for each subset of tests evaluated at the same time should be recorded. All results should be interpreted according to package insert (i.e., positive/+/reactive, negative/-/non-reactive, inconclusive/indeterminate, invalid, etc.) and recorded on the data collection form (with photographs, if feasible). Unexpected results (e.g., weak positives, false positives, false negatives, other anomalies, etc.) should be recorded in the data sheet and photographed for records.

From the data collection forms, results should be compiled noting the test results and agreement (conforming or nonconforming) with the expected results for each specimen. The number of conforming results for the panel shall be tallied and recorded. The number of invalid tests should be recorded as well (less than or equal to 1% invalid tests could be considered acceptable). Based on ISO 2859–1:1999 (Single Sampling, Normal Inspection, S-4 Level, Sample Size Code J, AQL 0.65), no more than one nonconforming RDT result out of 80 RDT results is acceptable (for each set of positive and negative panels). If no more than one nonconforming test result is observed for each of the positive (associated with sensitivity) and the negative panels (associated with specificity), the lot under evaluation is considered acceptable.

If 2 or more nonconforming test results are found for either the positive or negative panels, the results are considered unacceptable, and a laboratory investigation needs to be completed, which reviews and documents the following items to assess whether any source of laboratory error can be found: review inventory forms, kit storage temperature and humidity check, kit testing temperature and humidity check, specimen storage check, kit packaging check, kit expiry date check, kit procedure/ instructions/ insert check, review of calculations, review of equipment calibration status used for testing, review of data collection forms, review of correspondence, review all kit paperwork, photographs of nonconforming results, volume of remaining specimens(s), discordant tests retained, repeat testing of kit with new specimen aliquots, testing of specimen with reference lots. A reference lot is defined as a lot that was previously tested that provided no nonconformities for either positive or negative samples and still within product shelf life.

If a laboratory error is found to be the root cause of the unexpected result, the original data shall be maintained on file for the lot and testing repeated after appropriate corrective / preventative actions are implemented. If no laboratory error is found, a report should be prepared that includes the out of specification investigation summary and the testing details from the investigation.

Conclusions

Due to a lack of publicly available international standards for quality assurance of diagnostic devices, an approach was developed for monitoring consistency of manufacturing for HIV RDTs based on ISO 2859–1:1999 sampling plans and assessment criteria. From a review of ISO 2859–1:1999 and performance characteristics for HIV RDTs, an acceptance quality limit (AQL) of 0.65 was selected for application towards HIV serology/ab detecting RDTs. Based on a 0.65 AQL, the following criteria for each lot were developed: 80 HIV RTKs tested with Ab positive specimens (accept 1 defective result / reject 2 defective results) and 80 HIV RTKs tested Ab negative specimens (accept 1 defective result / reject 2 defective results). To perform this type of evaluation, Ab positive and negative specimens (panels) needed to be qualified to establish a baseline response per product type for future use in monitoring ongoing product consistency. Five prominent HIV RDTs brands were used in the panel qualification. For negative panels, 80 unique negative specimens were qualified for all RDT brands. For positive panels, 80 unique positive specimens were qualified for brands A, B, D, and E. Product C was only able to have 74 positives qualified, resulting in future monitoring with 20 qualified positives with 4 RDTs tested per specimen. Using p24 spiked samples, panel qualification was also conducted for Ag detecting RTKs (HIV Combo), where an approach was provided for ongoing QC monitoring for 4th generation antigen/antibody combination products. This qualification activity has provided a mechanism for monitoring HIV RDTs and could readily be replicated in a similar manner with other types of diagnostics. Furthermore, this set of panels and basic testing approach may be useful for field investigation, small scale stability checks, and proficiency testing amongst multiple lab sites.