Protection status inferences

Humoral and T cell immunogenicity data previously reported for 25 participants who acquired HIV after week 28 (cases) and 125 participants who did not acquire HIV over all follow-up (controls) from among vaccine recipients in the HVTN505 trial (Fig 1A) were analyzed [3,5,7]. Vaccine recipients who acquired HIV have been definitively exposed to the virus and are also demonstrably unprotected by the vaccine. In contrast, given the assumptions that the vaccine did have some, albeit small, protective effect and that exposure rates are low, the vaccine recipient group who did not acquire HIV is presumed to be a mixture comprising a minority of protected (if exposed, would not have acquired HIV) and a majority of unprotected (if exposed, would have acquired HIV) individuals. The PU learning pipeline employed seeks to infer the protection status of these unlabeled vaccine recipients without HIV (hereafter “unlabeled”) based on defining the degree of similarity in their response profiles to the vaccine recipients with HIV, and then to use these newly inferred protection status class labels to identify candidate CoP (Fig 1B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Positive-Unlabeled Learning-based pipeline to infer protection status and identify candidate correlates of protection in HVTN505 participants.

A. Timeline of HVTN505 Trial. Study participants received placebo or a recombinant DNA plasmid vaccine injection on days 0, 28 and 56 and a recombinant adenoviral serotype vector vaccine injection on day 168. Immune responses to vaccination were assessed on week 28 and the clinical endpoint (HIV acquisition status) monitored between week 28 and month 24. B. Schematic of PU Learning approach to infer protection status. Vaccine recipients that have acquired HIV are definitively unprotected and represent known positives (P). Unlabeled (U), or vaccine recipients that have not acquired HIV are hypothesized to include both protected participants as well as unprotected participants who were simply not exposed. In Step 1, a PU Bagging classifier was used to define class probabilities for actual PU labels and permuted PU class labels. In Step 2, the PU Bagging scores for participants in the unlabeled (U) class were defined. In Step 3, these class scores were used to generate a binary protection status class inference. In Step 4, immunogenicity features that help classify inferred protection status class labels or show univariate associations with predicted class are identified as candidate correlates of protection.

https://doi.org/10.1371/journal.pcbi.1013705.g001

We sought to employ a state-of-the-art PU learning method, termed PU Bootstrapped Aggregation, or Bagging, to predict the control group participants’ level of protection based on the degree of similarity of their immunogenicity profiles as compared to those observed among those who acquired HIV. By aggregating predictions from numerous classifiers built with positive class samples and bootstrapped unlabeled class samples, we computed a protection score for each unlabeled control group participant by averaging across their “out-of-bag” (OOB) predictions. A binary protection status classification boundary was then defined with or without knowledge as to the proportion of “protected” participants expected within the cohort using the rank of this score such as provided by the calculation of vaccine efficacy. Confidence in the robustness of these scores and their associated class inferences was evaluated using permutation tests [18]. Lastly, inferred protection status classes that met robustness testing criteria were then used to identify candidate CoPs. With the goal of CoP discovery in mind, and especially in cases where the same immunogenicity data is used to infer protection status as well as be evaluated as potential CoP, exceeding the performance baseline set by permutation testing is key to combating type I errors and over-optimism in identification of potential CoP expected to result from circular analysis.

Dimensionality reduction methods were used to visualize relationships between immunogenicity data and HIV acquisition status outcomes in an unsupervised fashion (Fig 2A). Visual inspection of serostatus labels following Principal Component Analysis (PCA) and t-distributed Stochastic Neighbor Embedding (T-SNE) showed some evidence of similarity among the positive case samples within the larger space observed for the control samples. This pattern suggested that semi-supervised approaches could have the potential to meaningfully differentiate among unlabeled samples that were more or less similar to the known positive class.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Immunogenicity profiles relate to acquisition status and can be used to infer protection status.

A-B. Principal component analysis (PCA, top) and T-SNE (bottom) to visualize the global structure of the dataset. Samples are labeled according to the HIV acquisition status outcome (A) or inferred protection status score obtained from the PU Bagging algorithm (B). C. Explicit Positive Recall (EPR, left) and Mean Bagging Score (MBS, right) value distributions were calculated for known positive class (Acquired HIV) samples for models trained on actual (yellow) and permuted (orange) class labels. Two sample independent T-test p values comparing score distributions for actual and permuted class labels are reported in inset. Mean and (standard deviation (SD)) of scores for the actual class labels are presented below each distribution. D. Statistical analysis comparing class probability scores of known actual and permuted positive samples. Statistical significance between score distributions for actual and permuted labels was defined by p-value from z-score, and effect size estimated using Cliff’s Delta. E. PC biplots depicting classification of unlabeled samples with knowledge of the ratio between protected and unprotected subjects among the unlabeled class set to 10, 30, and 50% (from top), or without the benefit of knowledge of the proportion of positive and negative classes anticipated among unlabeled samples (bottom). Participants classified as _infProtected are indicated in blue and _infUnprotected in red.

https://doi.org/10.1371/journal.pcbi.1013705.g002

Using a previously described and validated PU Bagging approach [19], each sample was assigned a consensus score reflecting its likelihood of classification as an unprotected (positive) sample across repeated modeling (Fig 2B). Reassuringly, visual inspection of PU Bagging scores appeared concordant with expectations based on HIV acquisition status labels, however, a rigorous means to assess the robustness of these class inferences was needed. To this end, given the absence of ground truth protection status labels to validate PU Bagging score-based predictions, we employed a positive set evaluation method to serve as a confidence test for the robustness of prediction results. We used permutation testing to determine whether the degree of concordance obtained between scores of known positive samples yielded for the participants who acquired HIV exceeded scores that were observed when HIV acquisition status labels were permuted and permuted “known positive” class samples were drawn at random. While such a test cannot establish that class predictions are biologically correct, it can effectively evaluate whether they are statistically different than would be expected at random; that is, it allows us to establish confidence in whether the case group does or does not have a distinct immune response from members of the control group drawn by chance.

We first compared the overall positive set PU Bagging scores under true acquisition outcome labels (actual) and those observed across multiple different permuted acquisition labels following label scrambling (permuted). For both repeated ten-fold and leave-one-out cross-validation runs, both explicit positive recall (EPR) and Mean Bagging Score (MBS) values for the actual positive samples typically exceeded those of permuted positive samples, providing evidence of a biological difference in immunogenicity profiles observed in vaccine-recipients who went on to acquire HIV during the trial’s follow up period as compared to those who did not (Fig 2C). Consistent with previous work [17], for the true positive class, MBS values were more tightly distributed across replicates than were EPR scores under a 30-time repeated k-fold “spy positive” evaluation [20]. Variance in the distribution of these scores after label permutation was also greater. Differences between actual and permuted HIV acquisition status class labels were observed for both positive set prediction evaluation metrics across various choices of K (Fig 2D). Two-sample independent T tests showed the expected inflation of significance associated with an increasing number of permutations; z-score p values were less susceptible to this artifact and sometimes but not always exceeded the conventional threshold of statistical significance. Effect size assessed using Cliff’s Delta indicated a medium to large difference between actual and permuted positive class scores, providing further analytical basis for confidence in the potential biological relevance of protection status inferences based on reliable differentiation of the actual true positives (Fig 2D).

While PU Bagging scores provide a means to compare the relative differences in protection inference confidence, they don’t provide an explicit boundary between classes. For cases in which the proportion of protected participants among all participants who did not acquire HIV are known or may be reasonably estimated, such as by overall vaccine accuracy, a percentile cutoff based on PU bagging score rank can be used to set the class threshold. Visualization of protection status class labels assigned based on different degrees of efficacy, or to a score set to 0.5, demonstrate how binary class assignments relate to the lead principal components in this study (Fig.2E). To characterize the reproducibility of PU scores, class scores for the unlabeled samples was characterized across 30 modeling iterations (S1 Fig in S1 File), demonstrating good robustness for scores that result from different folds.

Relationships with behavioral risk

Additionally, an exploratory analysis was conducted to assess the correlation between clinical traits and acquisition outcome and the potential confounding effect between covariates among vaccinated participants in the studied cohort (S1 Table in S1 File). Consistent with what was observed in the entire cohort, an association was identified between behavioral vulnerability (OR = 4.32, p = 0.0216) and acquisition outcome among cases and controls, adjusted for other covariates (BMI, Race, Age) [3]. Intriguingly, a significant association between behavioral vulnerability and inferred protection status classified by the PU Bagging pipeline was not observed (OR = 1.76, p = 0.217) as either a univariate factor or in a multivariate logistic regression model. This result suggests that immunogenicity measurements possess the potential to forecast participants’ expected protection level regardless of behavioral vulnerability factors that are correlated to the likelihood of HIV acquisition.

Next, we sought to leverage behavioral risk information to explore the validity of protection inferences, as behavioral vulnerability can be considered a proxy for HIV exposure. PU scores for participants with high behavioral risk were compared between those who did and did not acquire HIV (S2 Fig in S1 File). PU scores were significantly (p = 0.047) greater in high behavioral risk participants that acquired HIV as compared to those with high behavioral risk that did not acquire HIV. This result provides support for the potential biological validity of class predictions as it shows that inferred protection scores are associated with HIV acquisition among vaccine recipients with greater likelihood of exposure.

Immunogenicity feature contributions to inferred protection status

To define the features that contribute to inferred protection status classes, we next trained a Random Forest (RF) classifier, a decision tree-based machine learning model, to predict membership in inferred protection (_infProtection) and infection status classes based on immunogenicity data. Stratified five-fold cross-validation was used to fine-tune the hyperparameters of the RF classifier and estimate the generalizability of the established model on unseen data [21]. As a pre-requisite for using these models to assess feature contributions, classification accuracy, as defined by area under the Receiver Operator Characteristic curve (ROC-AUC), was considerably greater for predictions of inferred protection status (Mean AUC: 0.94; SD: 0.03)) as compared to HIV acquisition status labels (Mean AUC: 0.66; SD: 0.04) (Fig 3A). This result suggests that the model has learned patterns and dependencies in the data that contributed to these class inferences, as is expected from its reliance on this data in training. Given that this circularity will drive over-optimism in classification performance, this high degree of accuracy should not be interpreted as validating the protection status predictions, but rather as simply a basis for confidence in the relevance of features used to model inferred protection status.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Cross-validated multivariate modeling and permutation testing support the analytical robustness of inferred protection status.

A. Receiver Operator Characteristic (ROC) curves of a Random Forest classification model under stratified 5-fold cross validation to predict inferred protection (_infProtection, top, in blue) or acquisition (bottom, in purple) status. Gray shaded area represents ±1 SD of the performance in each fold. Dotted line indicates the performance expected at random, and area under the curve (AUC) values are reported in inset. B. Feature importance scores of the top ranked humoral response features, colored by feature type (legend at bottom right), for each classification task. C. Boxplots depicting feature importance scores for Con S Env gp140-specific IgA and ADCP for inferred protection and acquisition class labels in 100-time repeated modeling. Statistical significance was defined by Mann-Whitney U test. D. Boxplots of model performance assessed by permutation test. 30-time repeated cross-validated AUC (left) and Balanced Accuracy (right) under predicted protection level and acquisition label was compared to that under randomly shuffled labels for each (permuted labels). The multiple-time cross-validated performance was annotated with the format in Mean (Standard Deviation). Mann Whitney U test and Cliff’s Delta Estimate (annotated on the top of the panels) were employed to determine the statistical significance between groups (****: p-value < 0.0001). (E-F) Identification of correlates. E. Mann-Whitney U tests were employed to compare each immunogenicity response feature for HIV acquisition status (left) and inferred protection status (right) class labels. Features are ordered on y-axis according to the rank of -Log10 p-adjusted obtained from protection label and colored by feature type. F. Volcano plots depicting confidence and magnitude of differences in immune response features (colored by type) associated with HIV acquisition (left) and inferred protection (right) status. The Benjamini-Hochberg procedure was used to adjust p-values from Mann Whitney U test to reduce the false discovery rate.

https://doi.org/10.1371/journal.pcbi.1013705.g003

We next investigated the immunogenicity response features important to these classifications. As compared to prediction of acquisition status, which relied largely on both CD4 and CD8 T cell response attributes as well as IgG response magnitude, and to a lesser extent on virus-specific IgG3 levels, IgG3 features, supported by CD8 T cell features, and additionally IgA and antibody-dependent phagocytosis (ADCP) measurements contributed to classification of inferred protection status (Fig 3B). In order to evaluate the robustness of these contributions to predicting inferred protection status and to directly compare with contributions to acquisition status, we repeated classification modeling 100 times. As might be expected from their potential contributions to inferred protection status, importance scores for both features were significantly greater under the label of inferred protection as compared to that under acquisition outcome (Fig 3C), demonstrating a reproducible association between Env gp140 specific IgA and ADCP responses and vaccine-elicited immune protection as inferred by PU learning.

The robustness of these models was evaluated across 30 repeated cross-validation runs with area under the receive-operator curve (AUC) and balanced accuracies reported as outcome metrics. Inferred protection status class labels were more accurately predicted when actual rather than permuted labels were used in training models optimizing either overall or balanced accuracy (Fig 3D). In contrast, the accuracy of predictions of acquisition status using actual class labels was not always better than that achieved when models were trained on permuted acquisition status labels (Fig 3D). These results emphasize both the distinction between inferred protection status and acquisition status as well as the difficulty inherent to modeling acquisition status in studies where both vaccine efficacy and pathogen exposure rate is low and there is significant overlap between the immunogenicity profiles of many participants without HIV and those that were exposed and acquired HIV, resulting in natural confusion between them.

As an alternative to understanding contributions of combinations of immunogenicity response features to modeling acquisition and inferred protection status classes, we next evaluated associations between each feature and class assignments by Mann-Whitney U test, a non-parametric bivariate method that determines whether a statistically significant difference is observed between different groups of samples. In general, each of the roughly two dozen individual features that were significantly distinct between acquisition status groups exhibited the same directionality and greater confidence for inferred protection status groups, consistent with their playing a key role in driving protection status inferences (Fig 3E). Beyond Env-specific IgG3 and CD8 + T cellular responses, which showed high importance and significance in both multivariate modeling and bivariate statistical tests for both acquisition and inferred protection classes, we observed increased univariate statistical significance in FcγRIIa- and FcγRIIIa-binding of Env-specific antibodies under the inferred protection as compared to HIV acquisition status class labels (Fig 3E-F). Consistent with the increased importance of ADCP and Env gp140–specific IgA responses observed in multivariate models, bivariate statistical significance was also seen in the Mann-Whitney U test (Fig 3D). Whereas ADCP activity was roughly two-fold higher among participants inferred to be protected, Env-specific IgA was roughly two-fold lower (Fig 3F). In total, roughly twice as many additional features exceeded the nominal significance threshold. Similar results were observed when adjusting for clinical traits (S3 Fig in S1 File).

A subset of features that were either previously reported as humoral CoR or newly identified here as inferred CoP based on their ability to predict or associate with inferred protection status are shown for the ConS gp140 specificity for individual participants based on their acquisition or inferred protection status (Fig 4A). As examples, differences in the relationships between classes for ConS gp140-specific IgG and IgA for were pronounced (Fig 4B, for U statistics see S2 Table in S1 File): whereas a difference was not observed between acquisition status classes (p = 0.1 and 0.4 for IgG and IgA, respectively), there was high confidence in a group difference based on inferred protection status (p = 0.0029 and 0.00033). Consistent with the unrelated prior RV144 HIV-1 vaccine efficacy study [22,23], while IgG responses were elevated among participants inferred to have been protected, the opposite directionality was seen for IgA.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Immunogenicity feature relationships with and contributions to models of acquisition and inferred protection status classes.

A. Boxplots of raw value distribution of notable ConS gp140 Env-specific humoral response measurements according to HIV acquisition and inferred protection (infProtection) status classes. Statistical significance defined by T test. B. Table of p values from Mann Whitney U test and Logistic regression model following adjustments based on clinical traits as covariates. P-unadjusted: p-values directly obtained from bivariate tests (left) or logistic regression models (right). P-adjusted: Benjamini-Hochberg method was used to reduce false discovery rate for multiple test hypothesis. Newly inferred candidate correlates are indicated in bold.

https://doi.org/10.1371/journal.pcbi.1013705.g004

Expanding on this latter observation, given that IgA measurements in this study were bimodal, with a large number of values around zero, they were also treated as a categorical factor including two levels, IgA negative (log-transformed value = 0) and IgA positive (log-transformed value > 0). A logistic regression model was employed to determine whether an association was observed between Env gp140-specific IgA+ and IgA- responders and inferred protection class. Here, after adjusting for clinical traits (including age, BMI, behavioral vulnerability, and race), HIV-1 Env Con S gp140 IgA binding MFI were identified to be inversely correlated to inferred vaccine-mediated protection as both a continuous (OR = 1.17, p = 0.0012) as well as a categorical variable (OR = 2.78, p = 0.0035) (S3 Fig in S1 File). In contrast to inferred protection status, no significant association was observed with acquisition outcome, as previously reported [7]. Overall, this analysis demonstrates that PU learning-based inferences of protection status strengthened confidence in previously reported CoR and identified novel candidate CoPs suggested by their ability to contribute to or associate with predicted protection status.

Ethics statement

Participants in HVTN 505 (NCT00865566) provided written informed consent [3] for both the main study and this additional use of their samples. Analysis of HVTN 505 samples reported here was reviewed and approved through the Institutional Review Board at Dartmouth College.

Datasets

This study utilized case-control immunogenicity and clinical information from 189 participants in the HVTN 505 clinical trial: 150 received a recombinant DNA plasmid vaccine injection followed by a recombinant adenoviral serotype vector vaccine injection, and 39 received placebo injections. HIV acquisition was assessed between Months 7 and 24. Vaccine recipients (N = 150) were labeled as “Positive” if HIV was diagnosed (N = 25) during the follow-up period, and unlabeled (N = 125) if they were not diagnosed (Fig 1A). These cases and controls were matched on a set of criteria as specified in Janes et al., [5].

Immunogenicity features

As described by Neidich et al., Antibody-Dependent Cellular Phagocytosis (ADCP) [47], log-transformed multiplex Binding Antibody Multiplex Assay (BAMA) [48] and Fc Array assay [49,50] fluorescence intensities, and CD4 + & CD8 + T cellular responses [5] were assayed at 7 months after the first dose of vaccine [7]. To decrease false discoveries, a Wilcoxon test with Benjamini-Hochberg Procedure was performed on each immune feature to compare responses in placebo and vaccinated participants. Immunogenicity features for which p > 0.05 were excluded from further analysis based on lack of a vaccine-associated response.

Clinical information

HVTN 505 study participants were restricted to cisgender men and transgender women who had sex with men. As described by Hammer et al. [3], the age, body mass index, ethnic information, and behavioral vulnerability scores were identified based on a weighted average of two variables from the ACASI questionnaires that were collected at baseline for each participant. Baseline characteristics of sample donors based on HIV acquisition status and inferred protection status groups are presented in S1 Table in S1 File.

Positive unlabeled learning

Positive-unlabeled bootstrapped aggregation (Bagging) algorithms were first adapted to score the probability of the unlabeled samples being classified as “unprotected”. We bootstrapped samples from the unlabeled class with matched size to known positive samples and built a classifier to classify the remaining unlabeled samples. By iteratively bootstrapping different sets of unlabeled samples and predicting the rest of the unlabeled samples with established classifiers, the probability score of each unlabeled sample to be classified as “unprotected” (hidden positive) was calculated as the sum of its prediction when “out-of-bag” (OOB) divided by the number of iterations for which the sample was OOB. In this work, the number of total bootstrapped iterations was set to 200. A support vector machines (SVM) classifier with radial basis function (RBF) kernel, which demonstrated memory efficiency and effectiveness in high dimensional datasets, was used for the PU Bagging approach. To reduce the variance from a single-time prediction, the final PU bagging score for each unlabeled sample was averaged across 30 repetitions.

Positive set evaluation

Because samples from definitively protected participants are not available to validate negative class predictions, confidence in predictions can only be assessed using the class of participants who acquired HIV. We therefore implemented testing to score the positive examples. Analogous to cross-validation, the samples that were labeled positive were randomly allocated into 5 folds. Each fold was rotated as “spy positive” samples and mixed with unlabeled samples. On each iteration, PU bagging SVM (RBF) was performed between the (k-1) folds of positive samples and unlabeled samples plus “spy positive” samples. Two scoring methods were utilized to assess the positive set using the PU bagging score of each positive sample as “spy positive”. By treating samples with PU bagging scores ≥0.5 as positive, explicit positive recall (EPR) was adapted to score the positive set according to the proportion of replicates in which positive samples were correctly predicted.

An additional scoring method, termed Mean Bagging Score (MBS), was calculated by averaging the PU bagging score of all positive samples. In this work, 30 different fold splits of the positive set were performed under the actual label of infection.

Permutation comparison

To establish a baseline comparison of the positive class scores (EPR and MBS) obtained, a permutation test was leveraged. In permutation comparisons, the class label of HIV acquisition was randomly shuffled multiple times, and the scores of the resulting permuted infected (P’) set were calculated. In this work, the number of permutations was set to 30 or 100. Three statistical methods were employed to compare scores. Welch’s T-test and Cliff’s Delta statistics were applied to compare the scores for actual HIV serostatus labels as well as for permuted acquisition label. P-values of group z-scores were calculated between the sample mean of scores from actual labels and the distribution of scores from permuted labels. The z score was calculated as:

Where denotes the mean score of positive sets based on 30 repetitions, while μ0 represents the mean score obtained by repeatedly permuting the labels and refers to the standard deviation of the score distribution within the permutation group, calculated using n-1 degrees of freedom. In this analysis, an upper-tailed test was conducted, assuming the hypothesis that .

Inferring label of varied protection level

Class scores were used to assign binary protection status class labels. Unlabeled samples were classified according to multiple cutoffs based on PU bagging score rank and the hypothetical level of protection (10%, 30%, 50% hypothesized protected). In the absence of information regarding the expected level of protection, binary protection status labels were inferred by aggregating the votes of the predictions from three previously described labeling strategies [17]: (1) Inferring binary class labels according to bagging score cutoff at 0.5, (2) Using 3-fold cross-validation to train a fine-tuned SVM classifier with known positive and matched number samples with lowest score rank as “reliable negative (RN)” samples, which was used to classify the rest of the unlabeled samples, (3) Training a biased classifier with known positive and RN (N_RN = 25), then, iteratively expanding the RN set by including the rest of the U that were predicted as “negative”. In practice, the iteration was halted when no new negative samples were included or the number of iterations reached 20. Here, participants without HIV without PU Bagging Scores calculated were classified in the “unprotected” class by default.

Supervised machine learning

Random forest classifiers were trained with all humoral response features with stratified 5-fold cross-validation for hyperparameter tuning. Centering and Scaling were used to process the immune features that presented different ranges of values. Features were ranked according to impurity-based feature importance under the label of inferred protection (i: protection) as candidate Correlates of Protection (CoPs) and under infection outcome as Correlates of Risk (CoRs). In this work, the top-ranked features with an importance score > 0.15 were shown. AUC and F1 scores under 5-fold cross-validation were employed to evaluate the generalizability of the model. In 100-time repeated modeling, the pooled importance score of selected features was compared between that under the acquisition outcome and inferred label of varied protection level. The model was implemented using the scikit-learn library (version 1.1.3) on Python 3.11 [51].

Statistical analysis

Mann Whitney U tests were used to exclude the non-vaccine-elicited humoral responses from the feature set and test for the significance of immune features associated with infection or inferred protection status. Logistic regression models were built to assess the correlation between predicted outcome or clinical endpoint to immune features with covariates adjusted (age, BMI, race, behavioral risk score). The Benjamini-Hochberg procedure was performed to adjust for multiple hypothesis testing when immunogenicity features were compared between either acquisition or _infProtection status groups. Consistent with convention, a p-value of 0.05, set prior to analysis, was used as a cutoff to determine statistical significance. Mann Whitney U tests were performed using SciPy library on Python 3.11 [52]; the Logistic regression model was implemented in R (version 4.2.2).