The validity of diagnostic cut-offs for commercial and in-house scrub typhus IgM and IgG ELISAs: A review of the evidence

Background Scrub typhus is a neglected tropical disease that causes acute febrile illness. Diagnosis is made based upon serology, or detection of the causative agent–Orientia tsutsugamushi–using PCR or in vitro isolation. The enzyme-linked immunosorbent assay (ELISA) is an objective and reproducible means of detecting IgM or IgG antibodies. However, lack of standardization in ELISA methodology, as well as in the choice of reference test with which the ELISA is compared, calls into question the validity of cut-offs used in diagnostic accuracy studies and observational studies. Methodology/Principal findings A PubMed search and manual screening of reference lists identified 46 studies that used ELISA antibody cut-offs to diagnose scrub typhus patients, 22 of which were diagnostic accuracy studies. Overall, 22 studies (47.8%) provided little to no explanation as to how the ELISA cut-off was derived, and 7 studies (15.2%) did not even state the cut-off used. Variation was seen locally in reference standards used, in terms of both the diagnostic test and cut-off titer. Furthermore, with the exception of studies using ELISAs manufactured by InBios, there was no standardization of the selection of antigenic strains. As a result, no consensus was found for determining a cut-off, ELISA methodology, or for a single value diagnostic cut-off. Conclusions/Significance We have concluded that there is a lack of consensus in the determination of a cut-off. We recommend interpreting the results from these studies with caution. Further studies will need to be performed at each geographic location to determine region-specific cut-offs, taking into consideration background antibody levels to discriminate true disease from healthy individuals.


Introduction
Scrub typhus is a neglected tropical disease caused by the obligate intracellular bacterium Orientia tsutsugamushi [1].Transmission of the bacteria to humans occurs via the bite of larval trombiculid mites, known commonly as chiggers [2].It was formerly thought to be confined to the 'tsutsugamushi triangle', encompassing Pakistan, Northern Australia and parts of Russia.However, cases acquired in Chile [3,4], possibly Africa [5,6], as well as the Middle East [7] (by a proposed novel species O. chuto), have been reported, suggesting that its endemicity may be more widespread than previously thought.
Patients typically present with acute febrile illness, but if left untreated, this may progress to systemic infection and multi-organ failure, contributing to an estimated median mortality rate of 6.0% for untreated and 1.4% for treated scrub typhus [8] highlighting the importance of early and accurate diagnosis.A characteristic necrotic lesion, or eschar, at the inoculation site may serve as a diagnostic clue, however its presence varies, ranging from 9%-97% depending on the population [9,10].Given that other febrile illnesses such as typhoid, dengue and leptospirosis have similar clinical manifestations as scrub typhus, laboratory test is essential to differentiate scrub typhus from other undifferentiated fever [11].
Serological methods are more often used to diagnose scrub typhus due to their simplicity and cost-effectiveness [12].The indirect immunofluorescence assay (IFA) is considered the "gold standard," but the requirement of a fluorescent microscope and the subjective nature of reading slides limits its application in rural areas where this disease is most prevalent [8,[11][12][13][14].The Weil-Felix test is convenient to perform but suffers from poor sensitivity and specificity [13,14].Given the limitations of other serological methods, the enzyme-linked immunosorbent assay (ELISA) is acknowledged as a reasonably simple to perform alternative, providing an objective optical density (OD) result using an automated plate reader, that is reproducible in most clinical laboratory settings [13].
Despite the apparently standardized and objective ELISA platform, the diagnostic accuracy is influenced by methodological and patient factors.Methodological factors may include the composition of antigenic strains and their origin, and the choice of diagnostic cut-off.Patient factors are mainly centered on elevated levels of background immunity in endemic areas that may give rise to false positive results.Therefore, to ensure accuracy of diagnosis, standardized methodologies and locally validated OD cut-off levels for ELISA are urgently needed [8].
This review therefore aims to summarize (1) the differences in ELISA methodologies, (2) the OD cut-offs used for diagnosing scrub typhus in research, and (3) the rationale behind the selection of certain OD cut-offs for scrub typhus diagnosis in previously published diagnostic accuracy studies and observational studies.

Search strategy and eligibility criteria
A scoping review was performed.Searches were performed by one author (MP) on the PubMed electronic database using the following search terms: "scrub typhus," "tsutsugamushi", "immunoassay", and "ELISA".The search was restricted to papers published in English, up to 16 th October 2017.The titles and abstracts were screened for relevance.The full-text of relevant articles were assessed to determine eligibility.Diagnostic accuracy and observational studies using ELISA to diagnose scrub typhus in human were included.We excluded co-infection studies, case studies and studies investigating variations of the conventional ELISA methods (e.g., dot-ELISA).Reference lists of the relevant articles were also screened in order to identify additional studies.The protocol of this review was registered in the International Prospective Register for Systematic Review (PROSPERO) with registration number CRD42017078596.

Data extraction and analysis
Data was extracted by one author (MP), and where the information was unclear a second researcher was consulted (SDB).Details of the sample size, location, study date, reference test, cut-off, method used to calculate the cut-off, and ELISA methodology (antigenic strain and antibody isotype) were compiled into summary tables.The studies were grouped according to study design (diagnostic accuracy study or observational study), type of ELISA (in-house or commercial), and study location.The data was summarized using narrative synthesis.We did not evaluate minutiae of individual ELISA protocols, but instead focusing on the wider issues such as the methodologies used to determined diagnostic cut-offs.

Diagnostic accuracy studies: Cut-offs used and methodology for selecting cut-offs
Diagnostic cut-offs.Eighteen studies (81.8%, 18/22) stated diagnostic cut-offs with considerable variation noted between the cut-offs (Table 1).Diagnostic cut-offs for IgM ranged from 0.064 [22] to 1.474 [23] OD (both 1:400 sample dilution) and IgG cut-offs ranged from 0.11 [22] to 1.305 [24] OD (both 1:100 sample dilution) which were all from Thai studies (Table 1).Variation was also apparent in cut offs selected for Indian studies, despite the exclusive use of the InBios ELISA where the IgM cut-offs ranged from 0.41 [25] to 1.0 [14] OD (Table 1).Ten (43.5%, 10/22) studies investigated both IgG and IgM.Four of them (40.0%,4/10) determined higher cut-off values for IgG than IgM.In two cases, the same cut-off was applied to both isotypes.For example, Kim et al calculated a cut-off of 0.2 OD for both isotypes despite using different reference standard cut-offs (Table 1) [15].
Fifteen (83.3%, 15/18) diagnostic studies that determined ELISAs accuracy used IFA or IIP as the reference test.IFA/micro-IFA were used in 11 studies (61.1%, 1/18), with diagnostic cut-off titers ranging from 1:10 to 1:12,800 for IgM (Table 1).Five of these studies (45.5%, 5/11) also used a 4-fold rise between paired samples as a seropositivity criteria in addition to a defined cut-off titer.There were four (22.2%, 4/18) studies that used the indirect immunoperoxidase (IIP) assay as reference test.Two studies that assessed ELISA accuracy did not clearly mention the reference assay.Other reference modalities used include PCR, presence of eschar, in vitro isolation, and response to antibiotic treatment [12,23,24,26].In one study (4.5%, 1/ 22), a combination of the above was used in the form of scrub typhus infection criteria (STIC) composite [12].
The most commonly used method to determine a diagnostic cut-off was the addition of standard deviations (SD) to the mean OD of negative controls (72.7%, 16/22).There were four studies adding 2 SD, nine diagnostic studies adding 3 SD to the mean OD and one study calculated mean OD + 2, 3, and 4 SD (Table 1).
Variation in derived IgG cut-offs were noted within two studies that had each used both ROC curves and the mean OD to determine a cut-off for the same population [19,24].IgM cut-offs remained roughly the same when applying different methods (ROC curves and mean OD) in three studies [12,24,26].The remaining 16 studies either gave unclear information on the method used, unjustified methods to determine seropositivity (four studies), arbitrarily selected cut-offs (one study), or cut-offs from unpublished data (one study).

Observational studies: Cut-offs used and methodology for selecting cutoffs
Out of a total of 24 observational studies, seven (29.2%, 7/24) stated the method to determine the diagnostic cut-off and four (16.7%, 4/24) studies were unclear about how they derived the cut-offs stating "as used in other studies" or similar wording (Tables 2 and 3).Of the remaining studies, 13 (54.2%,13/24) provided no clear explanation as to how the cut-off was selected, however 0.5 OD was used for IgM and/or IgG diagnosis for 11 (45.8%,11/24) of these studies (Tables 2 and 3).Of the 19 observational studies using the InBios ELISA, seven (36.8%, 7/19) (Table 2) obtained local controls to determine a region-specific cut-off using the mean + 2 or 3 SD method.In the case of the NMRC in-house ELISA studies, the majority of studies (80.0%, 4/5) (Table 3), instead of calculating a single cut-off, patients were diagnosed with scrub typhus if they passed two criteria: 1) IgM OD �0.5 at a 1:100 dilution, and 2) a summed total OD of �1.0 of 4 sequential 4-fold dilutions.i.e., 1:100, 1:400, 1:1,600, 1:6,400) (Table 3).

Discussion
The application of appropriate diagnostic cut-offs is important for timely scrub typhus patient management using appropriate antibiotic therapy and to prevent complications leading to significant detrimental effect.This review has determined that there was a significant lack of consensus regarding methodologies, application and diagnostic cut-offs for ELISAs used for the diagnosis of scrub typhus infections.However, the reasons are complex and require further investigation.Approximately half of the observational studies provided no or insufficient justification for the OD cut-offs, and two studies did not specify the cut-off they used.Although the 0.5 OD cut-off was used commonly in InBios ELISAs studies and used by the Indian Council of Medical Research (ICMR), this is probably an appropriate estimation for certain parts of India with limited application in other geographic locations.This cut-off should be applied only in regions where it has been validated by testing samples from healthy controls to determine the level of background immunity in the normal population.In some cases, it is difficult to select a cut-off as demonstrated by Blacksell et al, where optimal OD cut-offs ranged from 0.2-0.6OD depending on the reference standards used [12].Several studies used the same cut-off for IgG and IgM, despite the differences in immunity dynamics of the different antibody isotypes-this should be taken into consideration when interpreting results of such tests [15,24,27,28], as generally, upon infection a spike in IgM is seen, followed by increased levels of IgG, which also confers long-term protection.
There was a lack of uniformity of approach regarding the diagnostic accuracy studies to determine appropriate ELISA cut-offs for various geographic locations.The reference methodologies varied from Bayesian LCM using composite scrub typhus infection criteria (STIC), IFA, through to mean + SD in healthy controls.In most cases, there was no clear justification for the reference test cut-offs employed, and it is likely that some of these cut-offs were not appropriate for the location in which they were being used.For example, while an IFA cut-off of 1:400 is often set in Thailand, it has been suggested to have a high false-positivity rate [29].Subsequently, the diagnostic accuracy of composites such STIC have also been suggested to overestimate scrub typhus positivity compared with index [30].Bayesian LCM is being increasingly used to determine true diagnostic accuracies, as they do not assume any reference diagnostic test is perfect [30].A recent study calculated-using this method-an admission IgM IFA cut-off of �1:3,200 or at least a 4-fold rise to �1: 3,200 in the convalescent-phase sample to provide the highest accuracy [29].Only one study in this review used Bayesian LCM; combining IFA, PCR, eschar and culture results as reference standards to interpret ELISA results [23].Given that the reference standards all have different accuracies, using a composite in a Bayesian approach helps to eliminate bias.Other studies used different approaches that may compromise accuracy.For example, in one study, a response to an unnamed antibiotic, along with positivity by either PCR or presence of an eschar, was used as the diagnostic criteria [26].Generally, doxycycline is prescribed to treat scrub typhus, however since it is a broad-spectrum antibiotic and also used to treat leptospirosis and murine typhus, a response to treatment may not point specifically to scrub typhus as the cause of illness.
A number of factors may have an influence on the diagnostic accuracy of ELISAs including antigenic composition and sample population.Differences in ELISA methodologies were observed where studies used local antigenic strains or incorporated these into pooled Karp, Kato and Gilliam antigens to supposedly increase the accuracy of the test.In general, higher ODs were obtained when using homologous antigens, therefore variation in cut-offs were likely to be seen depending on the antigen being used and the locally circulating strains.In India, the use of the InBios ELISAs (which used Karp, Kato, Gilliam and TA716 strain antigens) was widely implemented, providing a more standardized means of diagnosing scrub typhus.Jiang et al demonstrated a trivalent r56-kDa protein to be superior to both monovalent r56-kDa Karp and whole-cell Karp, Kato and Gillam ELISAs [16].The antigens used for deployed soldiers or travelers need to be carefully considered, and results need to be interpreted with caution, given their background immunity is likely to differ significantly from those living in endemic areas.Nevertheless, standardized, region-specific antigen preparations should be used in ELISAs, taking into consideration the circulating strains.
Regarding study populations, the use of samples from diseased or normal subjects as well as the geographic origin of the subjects can affect the derived diagnostic cut-off.In one study, serum samples were collected from Australia and Thailand, but it was unclear to which population the cut-off was applied, or whether the cut-off was calculated using results from both the populations despite differences in endemicity [18].
In addition to a lack of ELISA methodology standardization there was also lack of consensus in what is considered as the gold standard reference assay to determine diagnostic cut-offs.The absence of standardized methods and appropriate cut-offs has implications for seroepidemiology and clinical studies, as well as clinical decision making.On one hand, lower cut-off would result in false positives results risking unnecessary treatment and increasing probability of antimicrobial resistance.On the other hand, higher cut-off would result in false negative results risking cases to be missed.
This review has several limitations.First, it only investigated studies published in English, which may limit literature retrieval.Second, only one author performed the article selection and data extraction, however, any unclear data was discussed amongst the authors in order to limit bias.Lastly, the ELISA protocol was not examined as a factor.This needs to be considered when interpreting results, as differences in protocol (e.g. the amount of antigen used in plate coating) can influence the sensitivity and specificity of ELISA tests, that in turn influence the selection of optimal cut-offs.To limit the heterogeneity caused by different ELISA protocol, variations of the conventional ELISA were excluded from the review, and the InBios ELISA studies were grouped together in the analysis.
Further research will need to be conducted to determine local levels of background immunity, as well as to identify circulating strains, in order to make informed decisions for a regionspecific, standardized ELISA methodology and cut-off.

off selection rationale Isotype Cut-off (OD) a Reference test Sample size Study
studies (65.2%, 30/46) with the remainder being in-house assays.Ten studies (21.7%, 10/46) used ELISA methods developed by the US Naval Medical Research Centre (NMRC) (Tables