Ethics Statement

The protocol for KPS1 was approved by the Human Use Review and Regulatory Agency of the Office of the Army Surgeon General, the Institutional Review Board of the University of Massachusetts School of Medicine, and the Thai Ethical Review Board of the Ministry of Public Health, Thailand (protocol #654). The study protocol for KFCS was approved by Thailand Ministry of Public Health Ethical Research Committee, Siriraj Ethics Committee on Research Involving Human Subjects, Institutional Review Board for the Protection of Human Subjects State University of New York Upstate Medical University, and Walter Reed Army Institute of Research Institutional Review Board (protocol #2119). Formal written consent was obtained from all adults and children were enrolled after both written assent and consent from the parent/guardian.

Mixture model

Gaussian mixture models have successfully been applied to individual antibody titers in multiple pathogens including DENV and can provide an unbiased and unsupervised approach to quantifying seropositivity in a population of interest [19–23,28]. These methods implement a hierarchical probabilistic clustering framework where the data generating process is assumed to be produced in some predefined manner while associated parameters are subsequently estimated. We extend previous methodologies that either investigated multiple serological assays at a single time point, or a single assay at multiple time points to allow for the analysis of multiple serological assays measured at an acute and convalescent sample. The objective being to determine whether the patient is or is not acutely infected with the pathogen of interest, in this case DENV. In Fig 1 a set of theoretical antibody trajectories for a single assay are presented for infected (red) and uninfected (blue) individuals. Given an acute and convalescent pair of samples we developed a method, implemented in a Bayesian framework using stan, that can disentangle these two groups of infected and uninfected individuals. Here we will outline the methods used to infer infections from individual samples from a single assay, paired samples from a single assay, and then how to incorporate multiple assays taken at paired samples.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Theoretical relationship between time since symptom onset and titers in one infected (red) and three uninfected (blue) individuals.

These titers are mapped onto a two-dimensional representation of acute and convalescent titers. Ellipsoids represent the expected distribution of acute and convalescent titers for infected and uninfected individuals while the underlying marginal distribution of these acute titers is presented at the top. Opacity of each pair of points represents titers at each acute sample. For the infected individual (red) the lightest point represents how titers compare when sampling close to the date of symptom onset while the darkest red point represents how titers compare when sampling more than a week from symptom onset, often defined as a “recent” infection. The marginal distribution of acute titers colored by infected and uninfected individuals is presented above the acute and convalescent titer plot to demonstrate how these recent infections can present.

https://doi.org/10.1371/journal.pcbi.1013708.g001

Mixture models for serological data from a single acute or convalescent sample

We fit univariate mixture models to acute or convalescent samples independently (acute and convalescent geometric mean HAI, IgG, and IgM). We call these the acute or convalescent (GM HAI, IgG, or IgM) univariate mixture model hereafter. This is done using a single univariate mixture model with total mixtures, each of which has a mean and variance of and respectively for each mixture m. The mixture with the highest average titers contained individuals who had the most recent boost in titers. These models were implemented to demonstrate the performance of each serological assay when only acute or convalescent samples are available. When fitting these models we choose the number of mixtures to be two, one for the infected and one for the uninfected classes.

Mixture models for serological data from paired acute and convalescent samples

For paired serological samples we will first outline how this model is derived under the assumption that acute and convalescent samples were analyzed with a single assay. For each individual, , in the sampled population of size , we have measured acute and convalescent titers ( and respectively). Let us assume that the likelihood function for convalescent titers is dependent on two normal distributions, one representing the distribution of convalescent titers given acute titers if an infection occurs and one when no infection occurs. We write this as follows:

(1)

Where , and represents the probability of not being infected while represents the probability of being infected. Let represent the minimum baseline boost of serological titers and be the relative increase in convalescent titers as a function of acute serological titers. In addition, and represent the variance of each normal distribution around their respective means.

For the multivariate case, where more than one assay’s titers are incorporated, we extend the above to a multivariate normal distribution (MVN). We assume that each individual has acute and convalescent samples for k total assays measured as and , where represents the assay number. Let and be the vectors containing all the observed acute and convalescent serological responses for an individual across all assays. These will contribute to the likelihood via two mixtures that are assumed to be multivariate normally distributed:

(2)

Again, let be the probability of not being infected while is the probability of being infected such that . Let and now be vectors the length of the number of assays sampled () that represent the minimum baseline boosted serological titers and the relative increase in convalescent titers as a function of acute serological titers. Finally, let and be the covariance matrices associated with each mixture.

Combining univariate and paired mixture models to identify recent infections

Since mixtures fit to paired acute/convalescent data may fail to capture infections where the acute titer was already very high (as illustrated in dark red in Fig 1), we combine univariate mixtures from acute samples with the paired mixture described above. This approach utilized the same mixture model approach outlined above to ensure differentiation of the high acute titers. For the identification of these recent infections we used a stepwise approach to combine an acute univariate model with a paired multivariate model. For the univariate approach we identified the number of mixtures that best explained the distribution of acute titers and assumed the mixture with the highest mean titers represented a recent infection. We determined if an infection can be considered recent by identifying if the assigned probability to this mixture with the highest mean titers was greater than 50%. For those that were not recent we found their probability of infection using the multivariate mixture model. We attempted to fit alternative approaches that incorporated the time since symptom onset as a covariate in the mixture but found these did not perform as well as this stepwise approach as multiple recent infections were missed (S1 Fig).

Simulation of serological data

To assess the performance of our proposed model we first used simulated data generated using serosim [38], a flexible R package capable of simulating serological data which we ran in R version 4.3.3 [39]. We simulated the serological response to three serological assays (IgG, IgM, and GM HAI) for 500 individuals (Fig 2). This was done by defining exposure, infection given an exposure, and subsequent antibody kinetics post infection. The population as a whole underwent an extended period of low force of infection to the pathogen during which individuals could be infected, followed by a short period of high force of infection that infected approximately half of the population. After every successful infection an individual’s antibody responses would undergo a boost in titers for all three assays followed by antibody decay. Upper and lower bounds of detection for each assay were pre-defined, and the relationship between an individual’s initial titers and the expected boost in titers was also defined such that individuals with high titers experienced a smaller boost in titers. The chosen parameters for all antibody kinetics can be found in S1 Table. Note that this approach assumes an immediate boost post infection followed by a decay in titers, and does not consider differences in the timing of boosts that may exist between primary and post-primary infections. Simulated individuals could experience up to four sequential infections. Each individual’s antibody profile, reflective of the past infection history and measured through antibody levels, impacted their probability of infection during a subsequent exposure window through a logistic transformation reflecting immune-mediated protection. Individuals with a simulated log2 GMHAI titer greater than 8 had a 50% chance of getting infected as compared to those with no titers. To generate the dataset for analysis, antibody titers consistent with an acute and convalescent time point were extracted from the simulated data. Three different amounts of observational noise were added to the data to quantify how measurement error impacts model performance. The relative observational noise used is presented in S2 Table allowing for direct comparison across assays as well as across the different datasets (simulated and cohort data). The code we utilized for these simulations can be found in the GitHub repository github.com/marcohamins/linking-multiple-sero-assays.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Simulated serological data using serosim and subsequent results using the full mixture model on (a) HAI, (b) IgG and (c) IgM data with additional observational noise.

For each we present the predicted infection status using the mixture model approach along with the true underlying infection status. Parameters chosen for forward simulation are presented in S1 Table and resulting metrics of accuracy are presented in S4 Table.

https://doi.org/10.1371/journal.pcbi.1013708.g002

Cohort data description

Data for this analysis was collected from two cohort studies. Both studies were conducted in Kamphaeng Phet province in Northern Thailand and their characteristics are presented in Table 1. The first cohort study, the Kamphaeng Phet Study (KPS1), recruited 3,647 children (aged seven to 15) from 12 different primary schools and ran from 1998 to 2002 [40,41]. The study involved blood draws four times a year (January, June, August, and November) as well as active surveillance performed during the rainy season (June - November), where any children who experienced a febrile illness were evaluated and had acute and convalescent blood samples taken, with an interval of 14 days between collections. This led to 210 acute blood draws from active surveillance that underwent dengue RT-PCR while both acute and convalescent samples underwent serological testing (HAI, IgM, and IgG) for DENV and Japanese Encephalitis virus (JEV).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics for Kamphaeng Phet cohort studies split by source (KPS1 and KFCS) as well as across the entire dataset. Information on age, hospitalization as well as serological assay results for RT-PCR, haemagglutination inhibition assay (HAI), and enzyme immunoassay (EIA) are presented.

https://doi.org/10.1371/journal.pcbi.1013708.t001

The second cohort study, the Kamphaeng Phet Family Cohort Study (KFCS) began in 2015 and remains active [42]. For this analysis we included all data collected between 2015 and 2022. During this period, 494 multigenerational households were enrolled consisting of 3220 individuals who provided yearly blood samples through routine visits. In addition, if an individual in the cohort reported a febrile event at any point during the year, an acute and two convalescent samples were obtained, with intervals of approximately 14 and 28 days. Only episodes where both the acute and first convalescent sample were collected were utilized in this analysis to ensure consistent sample timing across datasets (n = 508). Similar to the previous cohort, acute samples underwent RT-PCR while acute and convalescent samples underwent HAI, IgM, and IgG testing for DENV and JEV. If this individual’s acute sample was a confirmed DENV infection via RT-PCR, acute and convalescent specimens were collected from all other members of the household and underwent identical serological and molecular testing.

It is important to highlight that the data from these two cohorts were taken from two different subpopulations in Kamphaeng Phet and had different inclusion protocols. Individuals in KPS1 are those who had a confirmed DENV infection via RT-PCR, HAI, or EIA. However, in KFCS if someone in a household had a RT-PCR confirmed infection the rest of the household were also sampled in a cluster investigation. This can be found in the higher percentage of cases that were RT-PCR confirmed in KPS1 compared to KFCS.

Laboratory methods and serological interpretations

A complete outline of the laboratory methods used for each serological assay are outlined in S1 Text and can be found in Hamins-Puertolas et al. (2023) [8]. When using HAI, infection is usually defined by a 4-fold or greater rise in HAI titers for any of the four DENV serotypes and JEV when comparing acute and convalescent sera. A higher seroconversion titer for JEV than for any DENV serotype is interpreted as a JEV infection event. All HAI titers we report are geometric mean HAI titers (GM HAI). For EIA units, an IgM ≥ 40 is usually used as a positive cutoff value. Evidence of DENV infection was classified by a DENV to JEV IgM ratio ≥ 1.0, and JEV infection when the ratio was < 1.0. Further classification into primary and post-primary for each assay is outlined in S1 Text.

Model fitting

All models were implemented in stan [43]. We fit 15 total models: Models 1–9 incorporated data from each combination of the three assays at both acute and convalescent time points (Table 2) while Models 10–15 fit univariate mixture models to one assay and time point (acute or convalescent) at a time (S6 Table). Models 7 and 9 combined multivariate and univariate mixture models to account for high acute titers. All models were estimated using Cmdstanr v0.5.3 with four independent chains, each of length 1000 after 2000 discarded for warm-up and model code can be found in the github repository github.com/marcohamins/linking-multiple-sero-assays [44]. Convergence was defined to be when R-hat < 1.1 and the effective sample size >300 for model parameters, after which posterior draws were pooled for reported parameter estimates. To assess accuracy we performed 4-fold cross-validation with testing set predictions used to assess accuracy. Bayesian priors are outlined in S3 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparison of each accuracy metric (sensitivity, specificity, F1, and area under the curve for reporter operator curve (AUC ROC)) on each model. Models are defined by which assays were incorporated (i.e., HAI, IgG, and/or IgM). Bolded values represent the best performing model for the infection definition and metric of interest.

https://doi.org/10.1371/journal.pcbi.1013708.t002

We removed any individuals whose GM HAI titers had reduced between acute and convalescent samples by more than 25%. These waning titers may reflect a DENV infection in the previous few months, either missed by the active surveillance program or subclinical. However, since no determination can be made, we removed these individuals from classification (n = 41) leading to a total of 677 total samples for which accuracy is assessed [45].

Assessing model performance

In order to directly compare the results from each of the models against both the simulated and the cohort data, we utilized sensitivity, specificity, area under the reporter operator curves (ROC AUC), and F1-score. The F1-score is a measure of predictive performance that weights the number of correctly identified infections (TP) by the number of missed infections (FN) and incorrectly identified non-infection events (FP). The precise formula is TP/(TP + 0.5*(FP + FN)). For the Thai data we presented these evaluation metrics for two definitions of what a true positive infection was for classification purposes. The first evaluation metric only considered RT-PCR positive infections while the second definition considered an individual to be infected if either RT-PCR, HAI, or EIA were positive. The former being a more specific definition since RT-PCR will have few false positives and more false negatives while the latter is more sensitive due to allowing either RT-PCR or serological results to define an infection.

Assessment of model performance to identify infections using simulated data

We first evaluated the performance of models on multiple sets of simulated data. We simulated paired (acute/convalescent) serological data for three independent biomarkers (GM HAIs, IgM and IgG) for 500 individuals of whom approximately 50% had been recently infected with a pathogen we intended to detect.

We first fit models incorporating data from the three simulated assays (GM HAIs, IgM and IgG) to try and infer the infection status of each individual (Fig 2). The model incorporating all three assays had an F1 score increasing from 93.9-98.0% as observational noise was decreased. This model consistently outperformed the models that incorporated any one or two of the assays when comparing sensitivity, F1, and AUC ROCs (S4 Table). Specificity was also highest in this model in all but one observational noise simulation. We then fit models including paired acute and convalescent samples from individual and combined sets of assays, leading to seven total models. Models including any two combined assays outperformed any singular assay, with increases in F1 scores and AUC ROC ranging from 0.1 to 27.3 and 0.1 to 8.0 percentage points respectively across the various observational noise simulations. The largest increases in performance were improvements on models that just had IgM or IgG data incorporated. The incorporation of multiple assays also reduced the decrease in performance that was experienced across all models as observational noise increased. For example, the model incorporating just HAI had F1 scores decrease 12.3% points as observational noise increased while models incorporating HAI and another assay (either IgG or IgM) had an average decrease in F1 score of 6.8%. Two models incorporating just IgM were unable to be fit at higher levels of observational noise, likely due to a loss of signal from titer boosts post infection. The remaining six models all performed well in scenarios ranging from small to large amounts of observational noise.

Application of the model to paired serological data from Thailand

We then assessed the performance of our methods on data from the two longitudinal cohorts based in Kamphaeng Phet, Thailand, where 677 paired sera from 560 individuals were taken that had 232 RT-PCR confirmed infections. We assessed the performance of 15 models under two different definitions of infection, described in the Methods section. This allowed us to compare a more specific gold standard (infections confirmed by RT-PCR) where few false positives and more false negatives are expected, to one that is more sensitive (infections defined by RT-PCR or serological results). We ran a total of 15 models, each model incorporated a different number of samples (acute and/or convalescent) and assays (all combinations of IGM, IgG, and HAI). This includes the seven models tested on simulated data, as well as variations developed to better capture recent infections where serological titers are already high at the acute sample, as well as single time point mixture models for all assays at both acute and convalescent samples.

For the first definition, using RT-PCR results, we found that all models except one (Model 2 with just IgM data) performed as well if not better than the previously implemented classifications based solely on HAI (HAI interpretation), IgM and IgG values (EIA interpretation), or combinations of these interpretations (Table 2). For this definition we found that the best performing approach according to the F1 metric and AUC ROC was the model that incorporated information from just IgM and IgG data (Model 6). However, the mixture that combined all data sources had better sensitivity (96.1% in Model 9 vs 91.8% in Model 6), which may be a more relevant metric of performance for this highly specific definition. With the more sensitive definition of infection that incorporates RT-PCR, HAI, and EIA information we found that using just GM HAI data (Model 3) resulted in the highest F1 and AUC ROC scores. All models with two or more data streams except Model 6 (IgG and IgM) had high F1 and AUC ROC scores (>90%). Note that the comparisons to the HAI- and ELISA-based interpretations for this gold-standard definition of infection are not meaningful since the interpretation of infection is part of the gold-standard definition, however we reported these values for full transparency (i.e., specificity compared to this definition was 100% in all of these clinical interpretations by construction).

Overall, we found that combining multiple data streams using this mixture model approach consistently led to higher performance when compared to currently implemented interpretations. Model 9 (GM HAI, IgG, IgM, and an acute GM HAI univariate mixture model, Fig 3) consistently performed well across both RT-PCR and combined serological definitions of infection (Table 2), where it had high sensitivity (96.1% and 85.2%) and specificity (86.2% and 97.7%). We also found that when a DENV infection was classified by one, two or all three assays (RT-PCR, HAI, and EIA illustrated in S2 Fig), this model identified an infection in 48.7% (19/39), 97.0% (65/67), and 100% (208/208) of individuals respectively (S3 Fig, S5 Table). This model also identified three additional infections that were missed by all three interpretations, two of which corresponded to high acute and convalescent GM HAI titers while one was near the decision boundary for the IgM paired samples and had an inferred probability of infection of 57% (S4 Fig). In addition, Model 7 (IgM, IgG, and a separate acute IgG classifier), consistently performed as well or better than the currently implemented algorithms. In individuals where one, two, or all three serological assays (RT-PCR, HAI, and EIA) classify there to be an infection, this model correctly identified an infection in 30.8% (12/39), 85.1% (57/67), and 97.1% (202/208) of individuals respectively with no additional infections inferred when all three serological assays did not classify there to be an infection.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Acute and convalescent serological data from Thai cohorts used in this analyses, colored by true infections status and prediction of infection during model testing.

a) Geometric mean titers of a haemagglutination inhibition assay (GM HAI) for all four serotypes of dengue virus. b) Immunoglobulin M (IgM) c) Immunoglobulin G (IgG) d) Ratio of IgM to IgG at both acute and convalescent sera samples.

https://doi.org/10.1371/journal.pcbi.1013708.g003

In addition to performing equally or better than a standard algorithm, a useful feature of the proposed mixture model methodology is that beyond classifying individuals, it provides an estimate for the probability that an individual belongs to the infected class, and thus the “certainty” of classification (S5 Fig). In the aforementioned Models 7 and 9 we find that 85.5% (579/677) and 96.9% (656/677) respectively of paired sera have an inferred probability of infection below 5% or above 95%, indicating high certainty in a sizeable majority of cases. For the remaining paired samples with lower certainty (between 5 and 95% inferred probability of infection), Models 7 and 9 had 50% (49/98) and 67% (14/21) from disagreeing interpretations (i.e.,. one or two positive serological interpretations) respectively(S3 Fig).

Finally, we compared the performance of our paired acute/convalescent models to univariate mixture models independently fit on acute or convalescent data for each serological assay (S6 Table). We found that each acute data stream performed poorly against both definitions of infection. Titers had not sufficiently risen to distinguish infected individuals from uninfected individuals. The univariate convalescent mixture models performed better than their acute counterparts with increases in F1 and AUC ROC scores ranging from 19.8 to 56.7 and 39.1 to 48.1 percentage points. Convalescent IgG and IgM both performed at similarly high levels within these univariate models. For example, when using the RT-PCR definition of infection with F1 scores of 66.4% and 67.4% and AUC ROC scores of 92.6% and 91.3% while the convalescent GM HAI had scores of 56.8% and 90.9%. None of these models performed better than models that incorporated both acute and convalescent titers from the same assay demonstrating the importance of acute samples in providing background immunological context for individuals, but that reliance on acute samples alone is not sufficient for accurate diagnosis.

Estimation of parameters related to antibody kinetics

We also investigated whether these methods can infer parameters associated with antibody kinetics. For the simulated data, we find that we are able to reconstruct the simulated relationship between acute titers and expected boosts across all three assays (S6 Fig). The reconstructed relationship remains consistent across varying levels of observational noise. Although estimates do diverge from the ground truth, this is due to the functional form constraint defined by the mixture models. We also estimate this same value for the Thai dataset and find that the expected GM HAI, IgG, and IgM additive titer boost in a fully susceptible individual are 160, 36, and 17 respectively (S7A). The relationship between this expected boost and an individual’s acute titers is also quantified (S7B Fig).