Benchmarking dataset

To assess the capability of computational design methods for TCR designing, we curated a dataset of TCR:pMHC complexes, involving both MHC-1 and MHC-II, with solved 3D structures that are not included in the ProteinMPNN training dataset (August 31, 2021, date cut-off) and not in AlphaFold2.3 (September 30, 2021, date cut-off). This date cut-off ensures that the complexes under consideration were not part of ProteinMPNN or AlphaFold2.3 training datasets, which could potentially lead to an overestimation of sequence recovery or modeling accuracy results. ESM-IF1 training involves the utilization of protein models derived from UniRef sequences and experimentally determined structures from the CATH 4.3 database. However, it’s important to note that the CDR3 sequences, which are central to this study, are not included in the UniRef dataset as they are generated through gene rearrangement. Additionally, the CATH 4.3 release occurred in 2019, which falls within the data cut-off period for ProteinMPNN and AlphaFold2.3.

The structures were collected from the TCR3d database [30] on December 13, 2023, and selected based on a resolution cut-off of 3.25 Å. We verified whether the chosen TCR:pMHC complexes had a related PDB entry released before the cut-off dates. For this reason, the CDR3β sequences of the TCR from each complex were submitted to a blast search using the blastp server [31]. Additionally, we verified redundancies within the selected dataset of TCR:pMHC complexes (considering the same TCR, peptide and MHC). For this reason, complexes with different peptide or MHC sequences, despite having the same CDR3, were maintained in the dataset (S1 Table). Structures with unnatural amino acids were excluded from the dataset. Following these filtering steps, a total of 32 and 6 TCR:pMHC structures (MHC-I and MHC-II, respectively) were selected, as listed in S1 Table.

ProteinMPNN and ESM-IF1 design protocols

For ProteinMPNN design, we used the v_48_020 model with default backbone noise (0.00) and sampling temperature of 0.1 to generate 10 designs per each test case, unless otherwise specified. The ProteinMPNN code used for running the desing was obtained from https://github.com/dauparas/ProteinMPNN. To evaluate the effects of sampling temperature on sequence generation, we utilized a range of temperatures from T = 0.000001 to 5. Additionally, to assess the impact of the number of generated sequences, we varied the quantity of generated sequences per test case from 5 to 500. A total of five design strategies were evaluated: (1) restricted the design to CDR3 (α and β TCR chains) within a proximity of 5 Å to either the peptide or MHC, (2) CDR1, CDR2, or CDR3 positions (α and β TCR chains) within a 5 Å distance to the peptide or MHC, or (3) the entire CDR3 (α and β TCR chains), (4) all CDR1, CDR2 and CDR3 positions (α and β TCR chains) and (5) all TCR variable domain positions (α and β TCR chains). Positions outside the specified criteria were held constant, and for the designed positions, all amino acid substitutions, including cysteine, were allowed. All tests were performed maintaining the same seed.

For ESM-IF1 design, we employed the esm_if1_gvp4_t16_142M_UR50 (fair-esm v2.0.1) model with sampling temperature of 0.2 to generate 10 designs per each test case, unless otherwise specified. Similarly to ProteinMPNN, effects of sampling temperature on sequence generation were evaluated using a range of temperatures from T = 0.000001 to 5, and the quantity of generated sequences per test case varied from 5 to 500 to assess the impact of the number of generated sequences. Five design strategies were explored: (1) restricting the design to CDR3 (α and β TCR chains) within a proximity of 5 Å to either the peptide or MHC, (2) including CDR1, CDR2, or CDR3 positions (α and β TCR chains) within a 5 Å distance to the peptide or MHC, (3) encompassing the entire CDR3 (α and β TCR chains), (4) all CDR1, CDR2 and CDR3 positions (α and β TCR chains) and (5) all TCR variable domain positions (α and β TCR chains). Positions outside the specified criteria were held constant. The designed positions were changed to <mask> tokens, and all amino acid substitutions, including cysteine, were allowed. Given the multi-chain structure, a padding of 10 <pad> tokens was used to separate the chains. The ESM-IF1 design protocol used in our study is available at https://github.com/LBC-LNBio/ESMIFDesign. The original ESM-IF1 code is available at https://github.com/facebookresearch/esm.

In the design process CDR1, CDR2 and CDR3 positions were delimited based on Aho numbering scheme: CDR1 (27–40), CDR2 (58–70) and CDR3 (106–139). In both methods, ProteinMPNN and ESM-IF1, we utilized the default decoding processes. For ESM-IF1, sequences are sampled autoregressively. In contrast, ProteinMPNN employs an order-agnostic autoregressive approach where the decoding order is randomly determined. Except for the tests where we removed the pMHC, in all design strategies we used the pMHC context (the TCR bound complex) for designing the TCR.

Rosetta design protocol

For Rosetta design, we utilized Rosetta version 3.5.1, and the protocol was executed using RosettaScripts. The target design positions were the same as those in deep learning-based methods, and we utilized a Rosetta resfile to indicate the positions and allowed substitutions (in this case we excluded cysteine) (resfile example in S1 Appendix). Throughout the mutation process, we applied the InterfaceDesign2019 fast relax protocol. A total of 10 designs were generated per test case. Alternatively, we generated a total of 1000 designs per test case and ranked the designs based on the Interface Analyzer dG_separated term or the total_score term, considering the ref2015 score function to identify the top 10 best-scored designs. A representative example of the RosettaScripts xml file is provided in S2 Appendix.

Design quality assessment at sequence level

To evaluate the success of designs, we compared the designed sequence to the native sequence by using the sequence recovery metric. The sequence recovery (Eq 1) measures the percentage of positions in designed sequences that match the amino acids in the native sequence. Given two sets of corresponding positions X = {x₁, x₂, …, x_n} and Y = {y₁, y₂, …, y_n} of equal length n, the sequence recovery is defined as: (1) where δ(x_i, y_i) is defined as: (2)

The percentage sequence recovery of 100% indicates that the designed sequence is identical to the native sequence, and 0% indicates no recovery. When evaluating the sequence recovery, we only considered non-redundant sequences with each test case. The percentage of non-redundant sequences generated by ProteinMPNN and ESM-IF1 is represented by the Uniqueness metric (described below), as shown in S5 Fig, considering the corresponding sampling temperature.

When comparing ProteinMPNN and ESM-IF1 across different sampling temperatures, we employed two additional sequence metrics: Uniqueness and Entropy. The Uniqueness (Eq 3) represents the percentage of unique sequences within the designed set. Given a set of m designed sequences , with u denoting the count of unique sequences, the Uniqueness is calculated as: (3)

The Entropy relies on Shannon’s information-theoretic entropy (Eq 4) and measures the diversity of the designed sequences. For each position j in the designed sequences of length L, the probability distribution of amino acids is estimated from the observed frequency. The Shannon’s information-theoretic entropy of the position j in the designed sequences is defined as: (4) where p_i is the probability of amino acid i at position j in the designed sequences.

The entropy of each position is then averaged across all positions to obtain the Mean Sequence Entropy (MSE) as defined in Eq 5: (5)

Additionally, we explored physicochemical similarity by considering changes from an amino acid in the native sequence to another in the design sequence within the same group. Amino acid groups were created using hierarchical clustering of BLOSUM62 amino acid features into 6 groups ([A, G, S], [C], [D, E, P, T], [Q, N, H, R, K], [I, L, M, V], [F, Y, W]).

Assessment of naturalness of V(D)J recombination

To evaluate the probability of designed CDR3 sequences being generated by the natural V(D)J recombination process, we utilized the Optimized Likelihood estimate of immunoGlobulin Amino-acid sequences (OLGA) [28]. We installed OLGA from https://github.com/statbiophys/OLGA and executed it via the command line using the flags humanTRA or humanTRB to compute the generation probabilities for CDR3α and CDR3β, respectively, with all other parameters set to default.

Assessment of TCR sequence variability and TCR interface contacts

To understand the amino acid variability across the TCR variable domain sequence, we compiled a set of human TRAV, TRAJ, TRBV, TRBJ, and CDR3 sequences. We collected 113 TRAV domain alleles, 147 TRBV domain alleles, 71 TRAJ, and 16 TRBJ alleles from IMGT, including all functional, open reading frame, and pseudogene alleles (https://www.imgt.org/vquest/refseqh.html). The sequences were renumbered according to the Aho numbering scheme using ANARCI [32] for alignment in sequence logos and CDR annotations. TRAJ and TRBJ sequences underwent multiple sequence alignment (gap opening of 10 and gap extension of 4) using the R msa package. We collected 19,948 human CDR3α and 20,662 CDR3β sequences with known antigen specificity from VDJdb [33] in November 2023. To number the CDR3s according to Aho, we first constructed each full TCRα or TCRβ sequence using Stitchr [34], then applied ANARCI to number the complete sequence. Subsequently, only the CDR3 positions, including gaps, were extracted. The sequence logos were generated in R using the ggseqlogo package [35].

To compute the contacts made by the TCR variable domain with the peptide or MHC in the MHC-I and MHC-II test cases, we utilized the Biopython package [36]. A contact was defined based on a distance threshold of 5 Å involving all atoms.

Identification of TCR buried and hotspot positions

To compute sequence recovery over buried positions in the TCR:pMHC complexes of the test cases, we identified buried positions in native structures based on the relative solvent accessibility (RSA). Initially, the solvent-accessible surface area for each position was determined using the FreeSASA Python package. Then, the ratio between the calculated accessible area and the maximum allowed accessibility for each amino acid was estimated to obtain the RSA. We classified a position as buried if its RSA was lower than 0.2. The reference maximum accessibilities were obtained from [37].

The hotspot positions were determined by performing an alanine scanning calculation using Rosetta 2.3 software. In this analysis, a residue was considered a hotspot if the absolute calculated ΔΔG for alanine substitution was greater than 0.5 kcal/mol. The alanine scanning command is provided in S3 Appendix, along with an example of the mutation list file used in this protocol (S4 Appendix).

Design quality at structural level

To evaluate the structural quality of ProteinMPNN or ESM-IF1 TCR designs, we modeled these designs and employed various orthogonal approaches for assessment. This included AlphaFold2-based modeling, Rosetta-based scores and molecular dynamics simulations with MM/PBSA calculations. Further details regarding the latter are provided in the following section.

The ProteinMPNN or ESM-IF1 design sequences were modeled with TCRModel2, a modified version of AlphaFold2 focused on TCR modeling [29]. The standalone version of TCRModel2, available at https://github.com/piercelab/tcrmodel2/, was employed with default parameters, except for the max_template_date flag, which was set to 2021-09-30 (AlphaFold2.3 cut-off date). Relaxation with Amber was allowed. The confidence score, which is a weighted combination of ipTM and pTM (ipTM*0.8 + pTM*0.2) [38], was extracted from the TCRModel2 output statistics file. The RMSD between the CDR3 or all CDRs of the design structure and the native structure was estimated considering the CDR3 or all CDRs backbone atoms, respectively, after superposition of the design structure to the native structure by the MHC. RMSD calculations were performed in R using the Bio3D package [39] and the DockQ calculation was performed using its standalone version obtained from https://github.com/bjornwallner/DockQ [40].

For the assessment of the TCR:pMHC interface and complex quality, we employed Rosetta Interface Analyzer, utilizing Rosetta software (version 3.5.1), with the entire protocol executed through RosettaScripts. Upon mutations of the native structures according to the designed amino acids, the complex was relaxed with the FastRelax protocol using backbone constraints to prevent structure displacement. The REF2015 score function was applied throughout the protocol, with dG_separated and total_score terms extracted from the output file. An example of the RosettaScripts used for this purpose is provided in S5 Appendix.

Random dissimilar TCR sequences were generated by substituting the native amino acids at the designed positions with randomly selected amino acids from different groups. Amino acid groups were defined by clustering the amino acids into six groups based on hierarchical clustering of BLOSUM62 features, the same used for computing sequence similarity.

TCR:pMHC binding affinity dataset and benchmark

To validate the capability of the binding affinity methods employed in this study to accurately predict TCR:pMHC interaction and stability, we created two different datasets of TCR:pMHC complexes. The first set comprised solved 3D structures of wild-type and mutant pairs, along with their corresponding binding affinity data. The second set used the same wild-type solved structures but expanded the number of mutants by modeling them, rather than using solved structures.

To construct the first dataset of TCR:pMHC structures, we selected wild-type and mutant pairs from the ATLAS server [41] and excluded from the dataset samples in which the experimental binding affinity of the wild-type and mutant complex were not obtained from the same study. By doing this, we seek to minimize noise included from different methods or experimental setups. Following these criteria, 12 TCR:pMHC complexes were found, forming 7 wild-type and mutant pairs (S2 Table). Since native and mutant pairs could have different sequence lengths as a consequence of N or C terminal electron density confidence, we trimmed the longest sequence in the pair to match the length of the shorter sequence. This was done in order to avoid bias created by the differing number of residues in MD simulations and MM/PBSA calculations.

To construct the second dataset, we selected mutants from the same wild-type complexes used in the first dataset, sourced from ATLAS, but lacking solved 3D structures. We only included mutants whose binding affinities were determined in the same study as their corresponding wild-types. To ensure a broad range of binding affinities for analysis, we excluded mutants with redundant binding affinities, applying a cutoff difference of 0.1 kcal/mol. This process resulted in a total of 35 mutants across 5 different wild-type complexes, as detailed in S3 Table. Mutations were introduced using PyMOL package following the same procedures as for TCR designs, with specifics provided in the subsequent section.

Molecular dynamics simulations

To estimate the binding affinity of the TCR:pMHC complexes, we first sampled the conformational space of the complexes with molecular dynamics (MD) simulations.

For MD simulations of native and designed TCRs in complex with the pMHC, we obtained the native TCR:pMHC complexes from the Protein Data Bank (PDB). The complexes were composed by the variable and constant TCR domains, including both α and β chains, the peptide, the MHC and the β2-microglobulin. None of the selected complexes had gaps in the structure. The native complex was processed by removing water and small molecules. Protonation state of residues was determined at a pH of 7.0 using Propka 3.0 [42] with pdb2pqr [43]. The designed TCR complexes were built using the native structures as reference and substituting the designed residues with PyMOL Python package. The protonation states of histidine at non-designed positions in designed complexes were maintained the same as the one in the corresponding native complex.

Prior to protein solvation, water molecules were placed on water sites by following a protocol similar to [44]. Specifically, the three-dimensional reference interaction site model (3D-RISM) [45, 46] and the MoFT suite [47] were used to calculate the solvent density around the protein and to find and place discrete water molecules into the most favorable locations, respectively. This was followed by aligning the complexes with its principal axis of inertia to the z axis, in order to reduce the number of waters using cpptraj from AmberTools23 [48]. All structures were solvated in a rectangular water box such that water molecules encompass the box boundary within 14 Å of the protein and were subsequently neutralized by the addition of Na⁺ ions. The Amber ff14SB force field [49] and the TIP3P water model [50] were used for protein and water, respectively. Disulfide bonds were manually bonded using the information derived from the pdb4amber module from AmberTools23 [48]. The topology and initial coordinates for each solvated system were done using TLeap from AmberTools23 [48].

MD simulations were performed using a graphics processing unit (GPU) accelerated Amber from AMBER22 [51, 52]. Each system was subjected to minimization, equilibration, and production following the protocol reported in [44]. Prior to replicas production, a minimization followed by a short MD simulation was conducted. This involved heating the system from 50 K to 298 K and restraining the protein atom positions. Subsequently followed, the entire system is minimized without any restrictions. The process of generating replicas per complex involved seven steps: (1) the system was heated from 25 K to 298 K for 20 ps at NVT with restraints on the CA atoms with a force constant of 5 kcal/mol/Ų, (2–5) four NPT simulations of 10 ps, restraining the CA atoms with a decreasing force constant from 4 kcal/mol/Ų to 1 kcal/mol/Ų (6) a 1 ns NPT without any restraint, and (7) a 3 ns NPT production run at 298 K. A total of 15 replicas were run for each system. The production simulations were used for MM/PBSA calculations, as well as PCA and RMSD analysis. The temperature was controlled with a Langevin thermostat [53] using a collision frequency of 1 ps⁻¹, while the pressure was set to 1 bar using a Berendsen thermostat [54] with a relaxation time of 1 ps.

The MD simulations of the benchmarked of solved complexes, described in S2 Table, followed the same protocol used above for the native complexes. In this case, no mutation needed to be performed since both native and mutant TCR:pMHC complexes were available. For benchmarking the modeled mutants, we employed the same strategy detailed above for the TCR designs.

MD simulations of both designed and native TCRs in complex with pMHC were analyzed by computing the root-mean-square deviation (RMSD) of the CDR3 backbone atoms (α and β chains, considering only a range of positions from the first designed positions to the last designed positions) between trajectory frames from production runs and the corresponding native crystal structure. This involved superposing the trajectory frames by the MHC of the corresponding native crystal structure. Both superposition and RMSD calculations were conducted using the R Bio3D package [39] in RStudio.

To assess the conformational space sampled by the designs compared to the native TCRs, we employed principal component analysis (PCA). The PCA was carried out using the pca.xyz function of the R Bio3D package [39] in RStudio, utilizing the coordinates of the CDR3 positions from the superposed trajectory used in RMSD calculations. All systems, their respective replicas and designs were included in the calculation. The first two principal components were utilized in the evaluation of conformational space sampling.

Free energy calculations with MM/PBSA

The MM/PBSA calculations were performed over the 15 MD simulations replicas of 3 ns saving every 10 ps, giving a total of 300 frames each. For each system, a total of 4500 frames were used for the calculation. The ionic strength and the internal dielectric constant were set to 0.15 M and 6, respectively. All other options were set to default. The MPI version of the MMPBSA.py [55] from AmberTools23 [48] was used for free energy calculations through MM/PBSA approximation. Unlike [44], no explicit waters at the interface nor entropy corrections were included in the MM/PBSA calculations.

Deep learning-based methods outperforms physics-based methods in recovering TCR native sequences

To assess the capability of deep learning and physical-based methods in designing TCR sequences, based on 3D structures, for targeting specific pMHC complexes, we curated a structural dataset consisting of non-redundant TCR:pMHC complexes (32 MHC-I complexes and 6 MHC-II complexes; see Methods section for details). These complexes served as native reference structures for comparison with the generated designs. Descriptive analyses of both set of structures, including the solvent accessibility across the TCR chains in the bound state and the TCR contacts with the peptide and MHC, are presented in S1 and S2 Figs, respectively, and can provide insightful views of these complexes and interfaces in the design context. Most TCR contacts with the peptide originate from the CDR3, especially the CDR3β, which guides the specificity of this interaction (S1 Fig). Due to the canonical binding mode of the TCR, the CDR3 is typically buried at the pMHC interface, except for longer CDR3 loops that expose the central part to solvent, followed by CDR1 and CDR2 (S2 Fig). To provide context for the TCR sequence variability in our TCR design analysis, we included a per position variability analysis of the TCR variable domain, encompassing CDR3 sequences, in S3 Fig.

As a first TCR design scenario, we focused on designing the TCR CDR3 positions (α and β chains) at the interface with the pMHC (Fig 1A). During the design process, the pMHC and TCR non-designed residues were provided as context for the methods analyzed. Considering this scenario, on average, we designed 10 and 11 positions of the CDR3s per each TCR:pMHC-I and MHC-II test case, respectively (S4 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Sequence recovery analysis of interface CDR3s amino acids in designs with ProteinMPNN, ESM-IF1 or Rosetta Design (InterfaceDesign2019 protocol).

(A) Representative structure of a TCR:pMHC complex (PDB ID: 7nme). The TCR variable and MHC chains were trimmed to just include components spatially related to the interface. The interface is indicated and the CDR3s amino acids composing the interface are shown as sticks. (B) Percentage of sequence recovery per method considering all MHC-I test cases. Each point represents a unique design sequence from a test case. For each test case, a total of 10 designs were generated by each method, but redundant designs were removed from the plot. For ProteinMPNN we employed a temperature sampling of 0.1, whereas for ESM-IF1 a temperature sampling of 0.2 was used (see Methods). Statistical two-sample pairwise comparison between methods were performed using Mann-Whitney test with the R ggpubr package. Significance is indicated above each box plot (**** and ** correspond to a p-value below 0.0001 and 0.01, respectively, while ‘ns’ means no significance). (C) same as (B), but for MHC-II. (D) Maximum sequence recovery obtained for each MHC-I test case and (E) for MHC-II test cases. (F) Sequence logo of three MHC-I test cases: 7na5, 7qhr, and 8shi. Each row of the panel corresponds to a specific test case and each column corresponds to the design method applied. The first column presents the native amino acids.

https://doi.org/10.1371/journal.pcbi.1012489.g001

Our initial assessment of design success involved comparing the identity of amino acids generated at the designed positions to the native amino acids, computing the percentage of sequence recovery for each test case. A critical parameter that affects sequence design, especially the diversity of generated sequences, in deep learning-based methods is temperature sampling. For ProteinMPNN, we employed a temperature of 0.1, which was used by its developers and others [20, 23]. In our assessments, this temperature achieved the highest sequence recovery while producing higher sequence entropy and uniqueness than lower temperatures (S5 Fig). To ensure comparability, we chose a temperature (T = 0.2) for ESM-IF1 that resulted in sequences with similar sequence entropy to those from ProteinMPNN at T = 0.1. These temperatures for ProteinMPNN and ESM-IF1 were also consistent in terms of the percentage of uniqueness (S5 Fig).

On average, ProteinMPNN and ESM-IF1 achieved sequence recovery percentages of 43.9% and 50.1%, respectively, for MHC-I, and 32.0% and 35.6% for MHC-II cases, with each test case generating 10 designs (Fig 1B and 1C). Sequence recovery percentages per target are presented in S4 Fig. Notably, both deep learning-based methods, ProteinMPNN and ESM-IF1, surpassed the sequence recovery of designs generated by Rosetta FastDesign protocol [24], with sequence recovery percentages of 31.7% and 19.6% for MHC-I and MHC-II cases, respectively. In 66% and 84% of test cases (including both MHC-I and MHC-II), ProteinMPNN and ESM-IF1, respectively, generated designs with superior sequence recovery compared to Rosetta Design. As an alternative to the default Rosetta FastDesign approach, we also explored protocol variations. This included generating 1000 designs and calculating the sequence recovery over the top 10 designs ranked by Rosetta energy terms. However, we observed consistent trends across these variations (S6 Fig).

When comparing ProteinMPNN and ESM-IF1, the latter generated designs with higher sequence recovery than ProteinMPNN in 55% of cases, while ProteinMPNN generated higher sequence recovery rates in 29% of cases. On average, ProteinMPNN, ESM-IF1, and Rosetta FastDesign achieved maximum sequence recovery of 53.6%, 60.6%, and 41.5%, respectively, for MHC-I, and 38.3%, 44.6%, and 30.8% for MHC-II (Fig 1D and 1E). A Pearson correlation of 0.51 was observed when comparing the maximum sequence recovery of MHC-I test cases obtained by ProteinMPNN and ESM-IF1 (S7 Fig). Most outliers arose from ESM-IF1 showing improved sequence recovery in comparison to ProteinMPNN. However, there were also test cases where higher maximum sequence recovery was achieved by ProteinMPNN. Together with the fact that no single designed sequence was found in common between ProteinMPNN and ESM-IF1 for the corresponding test cases, these findings suggest that combining designs from both methods may offer advantages, particularly in terms of sequence recovery and increasing diversity.

We explored sequence and structural features that might explain the superior performance of either ProteinMPNN or ESM-IF1 in certain test cases. The test cases were divided into two groups: one where ESM-IF1 achieved higher maximum recovery (13 cases) with a significant improvement of at least 10%, and another where ProteinMPNN did so (8 cases). Analyzing the per-position frequency of native amino acids in the CDR3s from these test cases, we observed a slight increase in the preference of certain amino acids such as at positions 110 and 111 of CDR3β from ESM-IF1 group, but no clear pattern emerged that could explain the differences in maximum sequence recovery (S7C Fig). Additionally, the length of the CDR3s in both groups was not significantly different (S7D Fig), nor were the buried surface area and shape complementarity of the TCR interaction interface (S7E Fig).

When analyzing the composition of native amino acids at positions exclusively recovered by ProteinMPNN, we found a bias toward serine and threonine, suggesting these amino acids are more readily recovered by ProteinMPNN (S7F Fig). Conversely, aspartic acid and histidine were more easily recovered by ESM-IF1 compared to ProteinMPNN. However, this may not fully account for the observed differences in maximum recovery. For instance, in the 7n2p case, which showed the largest difference (71% recovery by ESM-IF1 versus 28% by ProteinMPNN), only about half of the native amino acids that presented preferences for ESM-IF1 recovery (phenylalanine, alanine, and aspartic acid) were present in the 7n2p designed positions (native non-contiguous amino acid sequence FATYSTD encompassing CDR3α and CDR3β). Furthermore, the balance between recovery from CDR3α and CDR3β was maintained in both ProteinMPNN and ESM-IF1. For positions recovered exclusively by ProteinMPNN, 40% were from CDR3α and 60% were from CDR3β. These percentages were consistent for positions recovered exclusively by ESM-IF1, indicating no bias toward specific TCR chains that would explain the recovery differences.

Moreover, no substantial difference was observed in the composition of amino acids in the local environment of positions recovered exclusively by ProteinMPNN or ESM-IF1 (S7G Fig). While the difference in maximum sequence recovery may result from factors not analyzed here, it is possible that the observed differences arise from the sequences or structures encountered during model training.

Given the limited number of MHC-II test cases, we tested if the observed reduced sequence recovery of MHC-II test cases might simply reflect a statistical sampling bias. We computed the cumulative probabilities of the MHC-II maximum sequence recovery averages for ProteinMPNN, ESM-IF1, and Rosetta Design under the distributions of MHC-I maximum sequence recovery averages, constructed by sampling averages from 6 random test cases (the same number as MHC-II test cases) 1000 times. We obtained two-tailed p-values of 0.012, 0.008, and 0.023 for ProteinMPNN, ESM-IF1, and Rosetta Design, respectively. These results indicate that the maximum averages observed for MHC-II test cases significantly differ from those observed for MHC-I. It is not clear whether the difference in MHC-I and MHC-II recovery percentage is related to particular structural features or underrepresentation in training dataset. Future analysis with additional MHC-II test cases will likely shed light on the reasons behind the observed differences in sequence recovery. Given the more comprehensive number of available 3D structures, we focused the subsequent analysis on only the MHC-I test cases. Designed amino acids of three MHC-I test cases from ProteinMPNN, ESM-IF1, and Rosetta are represented as sequence logos in Fig 1F.

Increasing the number of generated designs (up to 500) increased the maximum sequence recovery for some MHC-I test cases (S8 Fig), reaching an average maximum sequence recovery of 59.9% (6% higher than 53.6% with 10 sequences sampled, but not significantly) and 69.1% (9% higher than 60.3% with 10 sequences sampled) for ProteinMPNN and ESM-IF1, respectively. Although this increase in sequence recovery was not consistently observed across all test cases (S8 Fig), these findings suggest that, at least for some cases, expanded sampling may contribute to the generation of designs closer to the native sequence.

From a practical standpoint, in silico generated designs usually need to be ranked and prioritized for subsequent robust in silico evaluation or experimental validation. Particularly for ProteinMPNN, which provides a score for each design sequence, we tested to rank their designs in the expanded sampling set by the provided ProteinMPNN score (defined as a negative average log-probability) to obtain the top 10 best-scored designs. Although in some cases, such as 8d5q, we observed designs with lower scores (the lower the score, the higher the confidence on the design prediction), achieving high sequence recovery rates, this trend was not consistently observed across all test cases (S9 Fig). Additionally, when correlating the scores of all designs with the sequence recovery, we obtained only a moderate correlation (-0.44; S9B Fig). When comparing the sequence recovery of the top 10 ranked designs with the previous 10 designs generated by ProteinMPNN, we did not observe significant differences (average of 44.9% in comparison to the default protocol’s 43.9%) (S10 Fig). Together, these findings indicate that, at least for the CDR3 interface design strategy, 10 designs generated by ProteinMPNN would be sufficient to achieve suitable sequence recovery rates.

To investigate the contribution of each designed position to the sequence recovery, we computed the sequence recovery per CDR3 position (Fig 2A). In general, the sequence recovery was uniformly distributed across the CDR3 positions. In CDR3α, position 133, typically located in the middle of the CDR3 loop, exhibits high sequence recovery in both ProteinMPNN and ESM-IF1. More pronounced differences between ProteinMPNN and ESM-IF1 are observed at CDR3α position 134. In CDR3β, ProteinMPNN achieved the highest sequence recovery at position 109, which usually corresponds to the second serine residue of the CASS motif derived from the V gene. This might suggest some bias from model training. In CDR3α, position 114 shows low sequence recovery for ProteinMPNN, contrasting with the recovery observed for ESM-IF1. However, a limitation of this analysis is that CDR3 sequence positions can vary substantially in terms of 3D structure positioning. Therefore, we mapped the per-position sequence recovery of ESM-IF1 onto the TCR structures (Fig 2B). Consistent with the distribution analysis, after TCR superposition, we observed balanced sequence recovery over the TCR interface (Fig 2B, left panel). A possible tendency of higher sequence recovery is noted in a region between CDR3β and CDR3α, whereas CDR3 peripheral regions appear to exhibit lower sequence recovery (Fig 2B, left panel). By superposing the TCR:pMHC structures by the MHC and mapping the sequence recovery onto the CDR3, we also observe evenly distributed recovery, suggesting no bias toward an interacting region of the pMHC (Fig 2B, right panel).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. ProteinMPNN and ESM-IF1 sequence recovery per CDR3 designed positions.

(A) The distribution of the percentage of sequence recovery per designed CDRα (on the left) and CDR3β (on the right) position is depicted as violin plots. Each point represents the average sequence recovery over non-redundant designed amino acids for a given test case at a given position. The position numbering follows the AHO numbering scheme. The upper bar plot shows the average over each sequence recovery distribution. Positions with fewer than three designed cases were removed for clarity. (B) ESM-IF1 sequence recovery is mapped onto the TCR structures. On the left, structures of test cases are superposed by the TCR, and only the TCR α and β chains are presented. The structures are oriented towards the pMHC plane. The CDR3β that stands out among the others is the long CDR3β from the test case 7l1d. On the right, the structures are superposed by the MHC, and only the CDR3s are shown. A representative peptide is presented as yellow spheres to highlight the orientation of the CDR3 segments in relation to the pMHC interface.

https://doi.org/10.1371/journal.pcbi.1012489.g002

Additionally, besides the design of TCR CDR3 interface positions, we explored three other CDR related design scenarios: the design of all CDR3 positions (α and β chains), not limited to those at the pMHC interface, the design of CDR1, CDR2, and CDR3 positions at the interface with the pMHC and the design of all CDR1, CDR2 and CDR3 positions. When comparing the sequence recovery across the different ProteinMPNN design scenarios, we observed higher recovery rates when designing all CDR3 positions in comparison to the other design scenarios (S11 Fig). This could potentially result from a combination of factors: a bias originating from the conserved N and C termini of the CDR3s across different V and J germlines (S3 Fig), which might be recognized during model training, and the topological characteristics of the CDR3, especially the β chain. In typical TCR binding modes, the CDR3β is often buried at the center of the TCR interface (S2 Fig), making specific contacts with the pMHC and, consequently, more restricted in its amino acid variability during the design. A similar trend was noted in the designs generated by ESM-IF1. It is noteworthy that ESM-IF1 utilizes the UniRef sequence dataset for model training [21]. While full CDR3 sequences are absent in TCR sequences from UniRef, as they are generated through gene recombination, other CDRs that constitute part of the TCR V germline could potentially be included in the ESM-IF1 training set. However, upon analyzing the results from all CDR designs, we did not observe any overestimation of sequence recovery for these cases.

To provide a better understanding of the challenges in designing TCRs, which may also impact interface design scenarios, we expanded our analysis to include the design of the full TCR variable domain (α and β chains) in complex with the pMHC. The average sequence recovery was 47% for both ProteinMPNN and ESM-IF1, similar to that observed in interface design strategies. However, per position sequence recovery analysis (S12A and S12B Fig) reveals that this average is a combination of both lower and higher recovery rates across different TCR positions. Highly conserved TCR framework positions such as 43 and 91 (Aho numbering) and the cysteine (106) and phenylalanine (139) flanking the CDR3 segment (S3 Fig), consistently showed high recovery, contrasting with the more variable positions such as those within the CDRs, which exhibited lower recovery. The trend of sequence recovery along the TCR sequence generally shows a correlation between ProteinMPNN and ESM-IF1 (S12C Fig).

Overall, these findings underscore the capacity of ProteinMPNN and ESM-IF1, utilizing fixed backbone positions, in generating TCR amino acids closely resembling native sequences of TCR binders in comparison to Rosetta Design. We confined the subsequent analyses presented in the following sections to deep learning methods. Additionally, to prevent any potential training information leaks to our test cases, we focused the subsequent analyses on the design of TCR CDR3 interface positions. This scenario is also more relevant since CDRs other than CDR3 can establish additional extensive contacts with the MHC (S1 Fig), thereby increasing the risk of cross-reactivity. When appropriate in subsequent sections, we also evaluated the ProteinMPNN and ESM-IF1 designs of all CDR positions, which can provide useful information in a design scenario when one does not aim to optimize the interface through point mutations but instead is designing a novel CDR interface.

Recovery of native interface amino acid composition and assessment of naturalness in V(D)J recombination

The preservation of interface amino acid composition is crucial, particularly for the TCRs, as it plays a key role in maintaining binding specificity and leads folding into a loop structure. Here, we compared the composition of the CDR3 interface in native sequences and design sequences (Fig 3A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Occurrence of amino acids at CDR3 interface positions in native and designed sequences.

(A) Frequency of each amino acid at the CDR3 designed positions in native and ProteinMPNN (left panel) or ESM-IF1 (right panel). Higher frequency indicates that the amino acid was more frequently observed at the CDR3 interface in the analyzed test cases. Only non-redundant generated sequences are considered in the analysis. The x axis is ordered by the BLOSUM62 amino acid grouping: [A, G, S], [C], [D, E, P, T], [Q, N, H, R, K], [I, L, M, V], [F, Y, W]. (B) Heat map of the frequency of substitutions of the amino acid substitutions in the designed sequences. The x axis represents the amino acids at the native sequences and the y axis represents the corresponding substitution in the designed sequences. A hypothetical frequency of 100% alanine in native sequences and leucine in designed sequence, for instance, indicates that we observed a change from alanine to leucine in all design cases. (C) Distribution of the Estimation of Generation Probabilities (Pgen) of CDR3 Sequences Using OLGA [28]. Pgen was estimated for each CDR3 (α or β) generated by ProteinMPNN (in purple) or ESM-IF1 (in red) from different design scenarios (design of only CDR3 interface positions—upper panel—or design of all CDR3 positions—bottom panel). For comparison, Pgen was also estimated for CDR3 sequences from the test case native structures (in green). This analysis considered only designs from human TCR test cases bound to MHC-1, and only Pgen values greater than 0 were presented in the density plot.

https://doi.org/10.1371/journal.pcbi.1012489.g003

Across the CDR3 (α and β chains) interface of all targets, the designed interfaces generated by both ProteinMPNN or ESM-IF1 generally mirrored the amino acid composition of the native interfaces. As expected (S3C Fig), interface CDR3 positions exhibited a strong bias toward glycine, which was recapitulated by both designing methods. Amino acids with low occurrence in native CDR3 interfaces, such as cysteine, histidine, isoleucine, lysine, methionine, and tryptophan, were also scarce in the design sequences. However, notable differences were observed, including an increased occurrence of leucine, proline, and especially serine in the ProteinMPNN-designed sequences, along with a reduction in acidic and amide amino acid groups (glutamic acid, glutamine, and asparagine). In ProteinMPNN, some native alanine, threonine, and aspartic acid were substituted by serine, leading to an increased frequency of the latter at the CDR3 interface (Fig 3B). Conversely, the reduction in the occurrence of acidic and amine amino acid groups was not observed in ESM-IF1 which seemed to maintain more balanced substitutions than ProteinMPNN, as expected due to its higher sequence recovery (diagonal trend at the heatmap). Notably, a bias towards aspartic acid was observed in sequences generated by ESM-IF1.

To evaluate the naturalness of the CDR3 designed sequence in terms of V(D)J recombination, we utilized OLGA [28] and estimated the probability (Pgen) of the designed CDR3 being generated by a human natural V(D)J recombination process. Designs of the full CDR3 (α or β) sequence by ProteinMPNN or ESM-IF1 have a probability of generation (Pgen) lower than 10⁻¹⁰, suggesting that these sequences would be unlikely generated by the human TCR VDJ recombination process (Fig 3C). A few CDR3β sequences from ProteinMPNN and ESM-IF1, achieved a probability of generation similar to that of CDR3 from native structures. In the scenario where only CDR3 interface positions were designed, the generated sequences achieved a higher probability of generation (Fig 3C). Specifically, 10% of the sequences designed by ProteinMPNN and 16% of those designed by ESM-IF1 had a probability of generation higher than 10⁻¹⁰, which particularly for CDR3β closely aligns with the distribution of probability generation observed in native structures. This may reflect the substantial native sequence recovery achieved by these methods, and it may also be a consequence of keeping the terminal residues of CDR3, located at VDJ junctions, fixed (not included in the set of designed positions) during the design process. This finding highlights the advantages of using this design strategy when one aims to generate native-like CDR3 sequences.

Physicochemical related substitutions and sequence recovery at buried or hotspot positions

Since substitutions by physicochemical similar amino acids at protein interfaces can preserve the interface characteristics and, consequently, protein binding, we evaluated the contribution of physicochemical related substitutions to the MHC-I sequence recovery rate by ProteinMPNN and ESM-IF1. Approximately 24% and 18% of the CDR3s interface positions, which underwent amino acids changes in the design process by ProteinMPNN or ESM-IF1, respectively, exhibited substitutions to physicochemical related amino acids (according to BLOSUM62 grouping, see Methods for details). Considering this similarity metric, the averaged sequence recovery increased from 43.9% to 56.9% and from 50.1% to 59.0% for ProteinMPNN and ESM-IF1, respectively (Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Sequence recovery analysis of interface CDR3s amino acids in terms of identity, similarity, identity at buried positions and identity at hotspot positions for ProteinMPNN (left panel) and ESM-IF1 (right panel).

For both panels, each point of the box plot represents the percentage of sequence recovery of a unique design sequence from the MHC-I test cases, without redundancy. While identity considers only substitutions to the same native amino acid as recovered, the similarity considers as recovered substitutions to the same amino acid (AA) physicochemical class (see Methods). The buried AA identity corresponds to the identity only computed over buried positions (estimated by relative solvent accessibility, see Methods), whereas the hotspot AA identity corresponds to the identity only computed over interface CDR3s hotspot positions, predicted by computational alanine scanning experiments with Rosetta (see Methods). Statistical pairwise comparison assessed the significance between the identity (reference) and the other metrics. It was performed using the Mann-Whitney test with the R ggpubr package. Significance is indicated above each box plot (****, ** and * correspond to a p-value below 0.0001, 0.01 and 0.05, respectively, while ‘ns’ means no significance (p- value ≥ 0.05)). A detailed view of the same evaluated metrics per PDB test case is presented in S13 Fig.

https://doi.org/10.1371/journal.pcbi.1012489.g004

Another potential factor influencing sequence recovery is that, even within the interface, certain amino acids at specific positions are not entirely buried, thus allowing less restriction in accommodating different amino acids. By focusing exclusively on designed amino acids located at buried interface positions, with a relative solvent accessibility (RSA) lower than 0.2 (as detailed in the methods), we observed a slight but significant average increase to 48.7% in the sequence recovery for ProteinMPNN. However, for ESM-IF1 we did not observe a significant increase in the sequence recovery at buried positions. To better understand the challenges posed by solvent accessibility for these methods, we analyzed the RSA for each CDR3 interface designed amino acid and determined whether the original amino acid at those positions was successfully recovered. Despite the statistically significant differences observed between the recovery and non-recovery groups (S14 Fig), the disparity in RSA ratios is relatively minimal (median of 0.05 for recovered vs. 0.09 for not recovered with ProteinMPNN, and 0.06 for recovered vs. 0.09 for not recovered with ESM-IF1). This analysis suggests that, in general, there is no apparent difficulty for both ProteinMPNN and ESM-IF1 in recovering amino acids at both buried and more solvent-exposed positions at the CDR3 interface with the pMHC. A similar trend was observed when designing all TCRαβ variable domain positions, where the correlation between sequence recovery and RSA per position was moderate to low (S12D and S12E Fig). This indicates that solvent accessibility is not the sole factor impacting recovery.

Given that even buried positions within the interface may not always be critical for the interaction, we identified potential hotspots for pMHC engagement. To achieve this, we conducted computational alanine scanning using Rosetta and identified energetically important designed CDR3 residues for the interaction with the pMHC, considering a 0.5 kcal/mol cutoff as described in the Methods section. Despite observing a significant change for ESM-IF1, overall, we did not clearly observe an effect on evaluating sequence recovery at hotspots positions.

Taken together, these results suggest that the interface recovery by deep learning methods could potentially be even higher when considering physicochemical related substitutions, without being strongly influenced by the solvent exposure of positions or their significance as hotspots for the interaction.

Designs generated by ProteinMPNN and ESM-IF1 are dependent on the pMHC interface

Despite the notable sequence recovery achieved in deep learning-based designs, there remains a possibility that the model predominantly learns the probability of an amino acid being part of the CDR3 backbone structure or inter/intra TCR chain contacts rather than specifically interacting with the pMHC. To investigate this, we conducted a similar design protocol for designing the CDR3 interface but excluded the pMHC, effectively removing it from the bound TCR:pMHC structure (Fig 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Maximum sequence recovery of interface CDR3s amino acids in the presence or absence of pMHC structures.

(A) Bar plot displaying the maximum sequence recovery for each test case, with designs generated by ProteinMPNN shown on the top and those by ESM-IF1 on the bottom. Colored bars (purple or red) represent the CDR3 interface designs considering the corresponding pMHC complex, while grey bars represent the interface design of unbound TCRs without the pMHC. (B) Same as (A), but grouping together the maximum sequence recovery values for ProteinMPNN with pMHC, ProteinMPNN without pMHC, ESM-IF1 with pMHC, and ESM-IF1 without pMHC. Statistical comparison between groups were performed using Mann-Whitney test with the R ggpubr package. Significance is indicated above each box plot (**** and *** correspond to a p-value below 0.0001 and 0.001, respectively, while ‘ns’ means no significance (p- value ≥ 0.05)). (C) Scatter plot with a linear trend line and a 95% confidence interval (light blue region) illustrating the correlation between the difference in maximum sequence recovery upon pMHC removal (maximum sequence recovery with pMHC minus sequence recovery without pMHC) for ProteinMPNN and ESM-IF1. A dashed diagonal line is included to aid in visual comparison. The correlation coefficients are indicated in the plot.

https://doi.org/10.1371/journal.pcbi.1012489.g005

As expected, the analysis of maximum sequence recovery, without the pMHC context, revealed a substantial reduction in sequence recovery in 25 (for ProteinMPNN) and 27 (for ESM-IF1) out of 32 cases (Fig 5A). This indicates that, at least for the analyzed TCR:pMHC interfaces, the sequence recovery is not solely a consequence of the TCR backbone structure, indicating that the models have learned to design the TCR interface when in complex with the target pMHC. In general, we observed a significant reduction on the maximum sequence recovery upon removal of the pMHC around 20%, on average, for both ProteinMPNN and ESM-IF1 (Fig 5B). When correlating the differences observed with and without pMHC per test case between ProteinMPNN and ESM-IF1, we found a low Pearson correlation of 0.22 (Fig 5C). This indicates that the impact on sequence recovery caused by removing the pMHC context is not consistent between ProteinMPNN and ESM-IF1 when analyzed on a per-test-case basis. For instance, in the 7n2p test case, no reduction in maximum sequence recovery was observed for ProteinMPNN, while a 40% reduction was noted for ESM-IF1. Thus, the correlation analysis suggests that these methods can behave differently upon pMHC removal depending on the specific test case. This suggests that the influence of pMHC removal on sequence recovery for each test case is not solely determined by the structure of the TCR:pMHC complex itself, but that the methods may behave differently. Intriguingly, in certain cases, such as the 7nme for ProteinMPNN and 6zkw for ESM-IF1, the maximum sequence recovery remained unchanged. To determine if the maintenance of maximum sequence recovery with or without pMHC could be due to recovery at non-interface CDR3 positions, we calculated the maximum sequence recovery solely at CDR3 hotspot positions (S15 Fig). Generally, the impact of pMHC removal was more pronounced for both ProteinMPNN and ESM-IF1 when considering only hotspot positions than general interface positions, underscoring the importance of the specific pMHC interface in the design process. In most cases that maintained the same maximum sequence recovery with or without the pMHC when considering general interface positions, the same was true when considering only hotspot positions. This may suggest that for specific cases, additional factors such as the backbone features or TCR inter/intra-chain contacts may play a crucial role in the design process, keeping a high sequence recovery even without the pMHC. Intriguingly, only ESM-IF1 demonstrated higher maximum sequence recovery without the pMHC than with it in three test cases.

AlphaFold2-based modeling supports the ProteinMPNN and ESM-IF1 TCR designs

In addition to the design evaluation at sequence-level, we performed a structural analysis of TCR designs using a deep learning-based approach. We employed TCRModel2, an AlphaFold2 modified version focused on TCR modeling [29]. Design models exhibiting low atomic deviation from the native structure and high modeling confidence score (weighted combination of ipTM and pTM) [29] (S16 Fig) indicate that TCRModel2 modeling supports the interface designed by ProteinMPNN or ESM-IF1, reinforcing our confidence in the generated design. It is important to note that we restricted our analysis to cases where native TCR:pMHC structures from the MHC-I test cases were accurately remodeled with medium or high accuracy (77% of cases) according to CAPRI criteria assessed by DockQ scores. Furthermore, an additional filter was applied based on the TCRModel2 confidence score (≥0.85). When combined, these criteria selected 60% of the cases for use in the structural analysis of designs.

Considering the CDR3 interface design scenario, in 84% and 89% of the analyzed cases, respectively, the ProteinMPNN or ESM-IF1 design with the highest model confidence reached similar or better model confidence than the median of remodeled native sequences. In seven and eight cases, for ProteinMPNN and ESM-IF1, respectively, the designs achieved a model confidence higher than 0.9 (Fig 6A). When randomly selecting dissimilar amino acids (see Methods) at the same positions (excluding cysteine) instead of generating amino acids by ProteinMPNN or ESM-IF1, we can see a clear drop in the model confidence and we observed random designs with the highest model confidence reaching similar or better model confidence than the median of remodeled native sequences in only 48% of the cases (S17A Fig). Additionally, we also observed a higher variability in the model confidence of random designs, with some designs reaching confidence scores below 0.6. It is worth noting that TCR:pMHC interfaces are relatively large and do not involve only CDR3 contacts, thus the occurrence of random designs with high confidence is not unexpected.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Modeling of TCR CDR3 interface designs with TCRmodel2.

(A) Box plot of model confidence of ProteinMPNN (in purple) and ESM-IF1 (in red) TCR designs for each test case. Each point represents a different design. The model confidence of remodeled native sequences is colored in green. In this case we remodeled the native sequences 10 times to obtain a distribution of modeling scores that represent the native structure. (B) Root-mean-square deviation (RMSD) of CDR3 backbone atoms (both alpha and beta TCR chains) of designs in comparison to the corresponding native crystal structure that originated the designs. RMSDs were determined after structure superposition by the MHC.

https://doi.org/10.1371/journal.pcbi.1012489.g006

We also assessed the deviation between the CDR3 backbone in the designs and the native structure, comparing these deviations to those observed in the remodeled native sequence. Even minimal backbone deviation could be expected due to side chain readjustments required to accommodate designed residues. In 79% of cases for ProteinMPN and 89% for ESM-IF1, at least one design achieved a comparable or lower CDR3 backbone RMSD compared to the median of remodeled native sequence (Fig 6B). Deviations below 2 Å were observed in 53% of cases for both deep learning models. When considering both model confidence and RMSD, in most cases, we observed that the designs, alongside the native sequences, exhibited optimized model confidence and RMSD (S17C Fig). Similarly to the model confidence analysis, we also observed that random CDR3 interfaces achieved similar backbone RMSD to the designs (S17B Fig), which may be a consequence of other non-designed TCR interface positions stabilizing the complex or obtaining non-disruptive designs by chance. This analysis underscores the importance of utilizing multiple evaluation metrics, as even random designs with low RMSD can present poor confidence.

In addition to the CDR3 interface design scenario, we modeled and evaluated designs from all CDRs positions scenario, where the design of all CDR1, CDR2, and CDR3 positions was permitted (S18 Fig). The model confidence from these designs maintained the high confidence observed in the CDR3 interface designs, with the design achieving the highest model confidence reaching similar or better confidence than the median of remodeled native sequences in 84% and 68% of test cases, respectively for ProteinMPNN and ESM-IF1 (S18A Fig). Confidence scores higher than 0.9 were also observed for both design methods, while no random design achieved a confidence level comparable to that of the native sequences. The high confidence is also consistent with the low CDR deviation from the native structure (RMSD below 2 Å in 79% and 63% of test cases for ProteinMPNN and ESM-IF1, respectively).

Collectively, these findings demonstrate that in the majority of tested cases, the designs generated by ProteinMPNN or ESM-IF1 were successfully modeled by TCRModel2 with high confidence scores, showing relatively low deviation from the CDR native structure. These results suggest that AlphaFold2-based modeling supports the potential binding of most TCR designs, from both the CDR3 interface and all CDR positions design scenarios, to the pMHCs in the test cases.

Additionally, we leveraged the remodeled TCR native complexes to evaluate the performance in terms of sequence recovery of ProteinMPNN and ESM-IF1 on those models instead of the X-ray crystallographic structures (S19 Fig). Generally, we did not observe a strong decrease in maximum sequence recovery of CDR3 interface positions, with an average reduction from 52.1% to 45.3% for ProteinMPNN and from 57.9% to 46.7% for ESM-IF1. For two cases (8gom and 8gon), we observed a substantial reduction in sequence recovery (around 30% to 50%), which may be explained by the deviation between the 3D positions from X-ray and modeled design positions since these two test cases have the highest CDR3 backbone RMSD. The impact caused by 3D deviation and the dependency on backbone positions in the design process can be at least partially observed in the Pearson correlation (0.48 for ProteinMPNN and 0.6 for ESM-IF1). This highlights the importance of using accurate modeling methods for modeling TCR complexes when designing unsolved complexes. Intriguingly, in two analyzed cases (7n2n and 7q9b), we observed an increase in maximum sequence recovery for both ProteinMPNN and ESM-IF1, which may indicate that in these cases the modeling could be improving the quality and placement of the backbone structures.

Energetic scoring of the TCR designs

As another structure-based criterion to evaluate the TCR designs, we employed the Rosetta energy function to score the designed structures against the corresponding native TCR complexes (the same set used in the AlphaFold modeling assessment). The scoring was based on two widely used Rosetta terms obtained from the Rosetta InterfaceAnalyzer: dG_separated and total_score. While the former relies on predicted energy changes at the binding interface, the latter provides broader information about complex stability. These two terms can be combined in the design evaluation [56].

The dG_separated scoring indicated that both ProteinMPNN and ESM-IF1 CDR3 interface designs followed a similar scoring profile to that determined for the native structures in most test cases (S20A Fig). Designs with worse score, i.e., higher dG_separated scores, and higher score difference than native structures were observed in only two test cases (7pdw and 8dnt) for ProteinMPNN. Conversely, ESM-IF1 showed a design with equal or better dG_separated score, than the native structure in all test cases. This contrasts with the scores obtained from dissimilar random TCR sequences, which presented higher variability and worse score and higher score differences than native structures in five test cases. This behavior is more pronounced when considering complex stability through the Rosetta total_score (S20B Fig). ProteinMPNN and ESM-IF1 designs clearly followed the energetic profile of native structures in all test cases, contrasting with the worse scores, i.e., higher total_score, presented by the dissimilar random TCRs. A combination between the two Rosetta terms can be observed in S20C Fig.

When analyzing the scoring of designs covering all CDR positions (CDR1, CDR2, and CDR3), we noted a more pronounced distinction between designs with improved Rosetta scores and those disrupting random interfaces (S21 Fig). In 84% of test cases for ProteinMPNN and 74% for ESM-IF1, we observed designs with improved Rosetta dG_separated scores, in contrast to the 31% observed for randomly generated CDR positions. No random TCR achieved a Rosetta total_score comparable to that of native or designed TCRs.

Molecular dynamics simulations and computational binding affinity estimation of the TCR designs

To assess the binding affinity of the TCR designs and compare them with native TCRs, we utilized a robust molecular dynamics simulation protocol coupled with widely used free energy estimation based on a previously benchmarked strategy relying on MM/PBSA calculations [44].

While this MM/PBSA protocol has shown promising results for estimating the binding free energy of two specific TCRs (1G4 and A6), its capacity to generalize to other distinct TCRs remains unexplored. Hence, as a first step to ensure a reliable evaluation of the binding affinity of our TCR designs, we initially benchmarked our protocol to predict free energy changes resulting from mutations in two curated sets of TCR complexes. The first set consisted of wild-type and mutated paired complexes, all with solved 3D structures and experimentally determined binding affinities. The second set, more comparable with the design scenario of this study, included the same wild-type complexes along with mutants lacking solved structures, which were then modeled based on the wild-type structure. The first benchmark test set included 7 different mutants, ranging from single-point to 12 mutations across the TCR α and β chains, while the second set comprised 35 different mutants.

By applying this MM/PBSA protocol, we achieved a Pearson correlation coefficient of 0.9 with experimental data for the set of solved mutant structures (S22A and S22B Fig) and a correlation of 0.74 for the set of modeled mutants (S22C Fig). Additionally, when assessing the MM/PBSA’s ability to discriminate improvements in ΔΔG greater than 0.5 kcal/mol or 2 kcal/mol resulting from mutations in the set of modeled mutants, we obtained balanced accuracy scores of 0.79 and 0.85, respectively (S22D Fig). Collectively, these results support the use of MM/PBSA calculations for comparing the binding affinity of native and designed TCRs.

Since this MD-based protocol is time-consuming, we selected CDR interface designs from 7 TCR:pMHC test cases to perform the MM/PBSA calculations over MD simulation trajectories (Fig 7). The trajectories of the designs were stable, maintaining CDR3 backbone deviation medians below 2.5 Å, in most test cases, similarly as the native trajectory (S23 Fig). In general, the Principal Component Analysis (PCA) indicated that the designed CDR3s sampled a similar conformational space as the native ones, with some exceptions like designs from the 7n2o test case (S24 Fig). However, the deviation between these different sampling spaces is low (below 1 Å), as indicated by the RMSD medians (S23 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Molecular dynamics simulation analysis and binding affinity estimation with MM/PBSA of Native and ProteinMPNN and ESM-IF1 TCR designs.

(A) Box plot of calculated ΔG (in kcal/mol) of ProteinMPNN (in purple) and ESM-IF1 (in red) designs in comparison to the calculated ΔG of the native complex (green diamond) for each evaluated MHC-I test case. Each point corresponds to the median of calculated ΔG from 15 replicas. The detailed distribution of ΔG across replicas and statistical tests are presented in S25 Fig. (B) Visualization of the TCR and peptide interface from the PDB 7pdw, highlighting the interaction between R135 from the TCRα and the D4 residue of the peptide.

https://doi.org/10.1371/journal.pcbi.1012489.g007

Interestingly, in most of the test cases (5 out of 7), we observed designs presenting binding affinity (expressed by ΔG) similar to or better than their corresponding native structure (Figs 7A and S25). Most designs, whether from ProteinMPNN or ESM-IF1, recovered the native binding affinity, presenting no significant difference in comparison to the native ΔG. Nonetheless, in one test case (7q9a) for ProteinMPNN and in three test cases for ESM-IF1 (7n2o, 7q9a and 8dnt), we obtained TCR design with significant higher affinity (lower ΔG) than the native TCR (assessed by a Mann-Whitney test, see S25 Fig). Intriguingly, both methods failed to generate designs with compatible affinity to the native TCR for 7pbe and 7pdw test cases.

A free energy decomposition of the native 7pdw interaction revealed that most of the TCR interaction relies on TCR three arginine from the TCRα (S26 Fig). Particularly, the one with most contribution (R135) anchors to the D4 peptide using a rotamer conformation that allows this interaction even with a high Cα distance and Cβ atoms pointing in opposite directions. The proposed amino acid by ProteinMPNN is a methionine which could interact with hydrophobic residues of TCRβ like V59, while the proposed amino acids by ESM-IF1 are the glutamine or glutamic acid, which could create a polar interaction and/or hydrogen bonds with a cluster of tyrosine residues of TCRβ (Fig 7B). Consequently, the positioning of this arginine hotspot together with the chemical environment can pose a strong challenge to these deep learning methods based on fixed backbones.

As a final test, we estimated the binding affinity of the ProteinMPNN and ESM-IF1 designs generated when designing all CDR1, CDR2 and CDR3 positions for the 7pdw and 7pbe test cases (S27 Fig). We expected that the design methods might produce improvements in the interaction beyond the CDR3 interaction region, such as towards the MHC, potentially improving binding affinity compared to the native complex. For the 7pdw test case, similar to the CDR3 interface designs, the binding energies remained lower than those of the native for all designs of both ProteinMPNN and ESM-IF1. This may highlight the critical role of the R135, also non-recovered in this tested design strategy, for the TCR interaction. However, for the 7pbe, we observed a significant improvement in binding affinity for one ESM-IF1 design (design #4) and a similar binding affinity as the native complex for three ProteinMPNN designs.

A free energy decomposition of the native 7pbe and the ESM-IF1 design #4 revealed positions in the native TCR, outside the CDR3 loops, that either contributed negatively (such as R40 from CDR1β) or did not substantially contribute at all (such as N27 and A30 from CDR1α, S69 from CDR2α, and N60 from CDR2β) to binding with the pMHC (S28A Fig). Specifically, in the ESM-IF1 design #4, the energy of all these positions was improved, contributing to the overall increase in binding affinity observed for this design. In particular, R40 was replaced by E40, potentially minimizing repulsive forces with the proximal peptide arginine R5 (S28B Fig). Another example of beneficial modification in this design was the replacement of S69 with D69, enabling anchoring to arginine R131 of the MHC (S28B Fig). In the context of improving affinity, these findings highlight the benefits of expanding the designed positions beyond the CDR3 loops, particularly for the 7pbe test case.