Background

In 2020, 10 million individuals fell ill with tuberculosis and 1.5 million died from the infection [1]. The problem of multidrug-resistant tuberculosis (MDR-TB)–defined as resistance to isoniazid and rifampicin–has been described by the World Health Organization (WHO) as a global health crisis [1]. Despite advances in diagnostics and treatment, MDR-TB remains under-detected and treatment success remains stubbornly below 60% globally [1]. The SARS-CoV-2 pandemic has likely set back the progress that has been made by years [2].

The WHO has called for universal tuberculosis drug susceptibility testing [3], but two major barriers remain for universal susceptibility testing in the face of rapidly evolving resistance. First, culture-based susceptibility testing is too slow, infra-structure dependent (biosafety level 3 laboratories are required), and technically challenging to offer a realistic solution [4]. Novel molecular assays have been developed to rapidly detect resistance to a subset of first- and second-line drugs, but are limited in the number of resistance-conferring mutations they can detect [5], constraining their sensitivity. Some countries now rely on whole-genome sequencing (WGS) to identify susceptibility to first-line drugs [6], but suffer the same limitations in the number of resistance-conferring mutations detected, as well as financial and technical implementation requirements. Second, culture-based susceptibility testing typically generates binary phenotypes, where susceptible isolates are differentiated from resistant ones based on critical concentrations of antibiotic that are not always firmly relatable to clinical outcome. Semi-quantitative measures of drug susceptibility such as minimum inhibitory concentrations (“MICs”) could be more clinically useful as they lend themselves to the future development of personalized dosing regimens, by indicating whether it might be possible to overcome ‘resistance’ through higher drug dosing.

Machine learning algorithms have accurately predicted binary susceptibility from whole genome sequencing data in Mycobacterium tuberculosis [7–9], and predicted MICs in other pathogens like Salmonella [10], N. gonorrhea [11], S. aureus [12], and E. coli [13], using both genomic mutations [7–9,11,13] and k-mers [10,12] as data inputs. The difficulty in generating semi-quantitative phenotypic data for M. tuberculosis has meant that machine learning approaches have yet to be applied to the problem of predicting such phenotypes from WGS data for this major pathogen.

Here, we assess how well a machine learning system can use genome-wide genomic features to predict MICs for 13 anti-tuberculosis drugs, including for bedaquiline and delamanid for which good molecular assays are currently lacking. We leverage a new, recently published dataset from CRyPTIC (a consortium working on genotypic / phenotypic associations for drug resistance in M. tuberculosis) [14], and adapt previously described extreme gradient boosting machine models for use on genome-wide k-mers directly from unassembled sequencing reads. We then assess whether MIC predictions can be used to accurately predict the more commonly generated binary phenotypes from Mycobacterial Growth Indicator Tubes (MGIT) in an independent dataset.

Results

Dataset characteristics

The CRyPTIC dataset (Table 1) included MICs for 10,859 isolates from 22 countries. Lineages 1 to 4, 6 and Mycobacterium bovis were represented, with lineage 4 (50%, 5,436/10,859) and lineage 2 (35%, 3,745/10,859) the most common (Table A in S1 Appendix). About 28% of samples (3,033/10,859) were multi-drug resistant. MICs were determined using a 96-well broth microdilution plate, and were available for three first-line antibiotics (isoniazid, rifampicin, ethambutol), plus rifabutin, two fluoroquinolones (moxifloxacin, levofloxacin), two injectable agents (amikacin, kanamycin), ethionamide, two newer drugs (bedaquiline, delamanid) and two repurposed drug (linezolid, clofazimine). Pyrazinamide was not present on the microdilution plate for technical reasons. We present results for kanamycin and rifabutin exclusively in Table B in S1 Appendix as they are less commonly prescribed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Sample provenance and resistance profiles in the CRyPTIC dataset.

https://doi.org/10.1371/journal.pcbi.1012260.t001

Minimum inhibitory concentration prediction

We developed a k-mer-based, hypothesis-free, genome-wide supervised machine learning algorithm to predict MICs for 13 antibiotics (as described in Methods and presented in Fig 1). To assess the performance of the machine learning system on the widest possible set of antibiotics, it was initially trained on 75% of the CRyPTIC dataset (8,146 randomly selected isolates). Predictions were made for the remaining 25% (2,713 isolates). We generated confidence matrices for each drug (S1 Fig) and calculated essential agreement rates (when the MIC is correctly predicted within one doubling dilution) (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Illustration of the machine learning system workflow to predict MIC and drug susceptibility status for 13 antibiotics using sequencing data.

Drug names: INH: Isoniazid, RIF: Rifampin, EMB: Ethambutol, MXF: Moxifloxacin, LEV: Levofloxacin, AMI: Amikacin, KAN: Kanamycin, ETH: Ethionamid, RFB: Rifabutin, BDQ: Bedaquilin, CFZ: Clofazimine, DLM: Delamanid, LZD: Linezolid. Groups: Guidelines and drug groupings refer to World Health Organization Consolidated Guidelines on Tuberculosis.

https://doi.org/10.1371/journal.pcbi.1012260.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Essential agreement rates (within 1 doubling dilution) for all drugs.

https://doi.org/10.1371/journal.pcbi.1012260.t002

For the first-line drugs, the essential agreement rate was 94.2% for isoniazid (2,242/2,380), 95.0% for ethambutol (1,811/1,907), and 92.9% for rifampicin (2,086/2,246) (Table 2). We compared essential agreement rates between phenotypically resistant and susceptible isolates (after converting MICs to binary phenotypes using previously defined epidemiological cutoffs [15]) to see if the predictions made by the machine learning model were more accurate for one of the two subgroups. There was no statistically significant difference in essential agreement between the two subgroups for ethambutol and rifampicin. There was a 7.3% decrease in essential agreement for phenotypically resistant samples compared to susceptible isolates for isoniazid (97.8% vs 90.5%, p<0.001). Perfect agreement rates (exact MIC predicted correctly) were on average 30.1% lower than essential agreement rates across drugs (p<0.001) and are presented in Table B in S1 Appendix.

Essential agreement was 94.9% for levofloxacin (1,837/1,936), 90.9% for moxifloxacin (1,544/1,699), 97.5% for amikacin (2,191/2,248), and 91.3% for ethionamide (2,017/2,210). Essential agreement was lower for phenotypically resistant than for phenotypically susceptible isolates for all drugs, with differences of 13.1%, 7.3%, 10.9%, and 20.0%, for levofloxacin, moxifloxacin, amikacin, and ethionamide, respectively (p<0.001). We note that there were fewer phenotypically resistant isolates for these four drugs: 310, 261, 158, and 309, respectively, in the CRyPTIC dataset–which could explain the challenge in lower accuracy. EA was above 93% for all new and repurposed drugs: 93.3% for bedaquiline (1,987/2,130), 96.1% for clofazimine (1,842/1,917), 93.6% for linezolid (1,689/1,805), and 96.4% (1,918/1,990) for delamanid. Essential agreement dropped significantly for phenotypically resistant isolates: 39.3% (11/28) for linezolid, 13.0% for clofazimine (3/23), 10.0% for delamanid (2/20), and 6.2% for bedaquiline (1/16), demonstrating the poor predictive performance for these drugs, with the high essential agreement due to the majority of susceptible predictions. We observe the bimodal distribution of MICs for rifampicin, rifabutin, amikacin, and isoniazid, while those for ethambutol, kanamycin, moxifloxacin, and levofloxacin show a more unimodal distribution, correlating with less accurate predictions distributed across a broader range of the confusion matrix (S1 Fig) and consistent with previous findings in the CRyPTIC dataset [15]. Results by lineage are presented in Table C in S1 Appendix.

Binarising MICs

We assessed whether the machine learning model could be used to predict binary susceptibility status, given binary predictions remain the standard by which most patients are treated. We begin by considering the MIC value predicted by the model as a probability of resistance, where an increase in predicted MIC corresponds to increase in probability of resistance. For example, if all MIC predictions range from 1 to 8, then a prediction of 8 is considered to be a 100% likelihood of resistance and prediction of 1 to a 0% likelihood of resistance, for the purpose of generating a receiver operating characteristic (ROC) curve and of calculating an area under the curve (AUC) metric (Table 3). For first-line drugs, AUC was 0.983 for isoniazid (95% CI 0.979–0.987), 0.992 for rifampicin (0.988–0.994), and 0.965 for ethambutol (0.957–0.973). For second-line drugs, AUC was 0.945 for levofloxacin (0.931–0.963), 0.970 for moxifloxacin (0.958–0.982), 0.970 for amikacin (0.953–0.984), 0.960 for kanamycin (0.941–0.975), and 0.939 for ethionamide (0.924–0.952). For new and repurposed drugs, AUCs were lower—0.908 for bedaquiline (0.876–0.952), 0.879 for clofazimine (0.832–0.934), 0.781 for linezolid (0.695–0.866) and 0.785 for delamanid (0.711–0.861). However, we note that these four results are likely significant overestimations due to the very low number of true phenotypic positives in the test set (between 14 and 19).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Binarised performance of the machine learning system on the CRyPTIC dataset and on an independent dataset.

https://doi.org/10.1371/journal.pcbi.1012260.t003

We then converted the MIC predictions to binary predictions and compared them to the binary phenotypes determined using previously defined epidemiological cutoffs [15] (Methods, Table D in S1 Appendix). For first-line drugs, sensitivity of the machine learning model was 95.0% for isoniazid, 97.1% for rifampicin, and 93.1% for ethambutol, with a specificity of 98.1%, 89.4%, and 97.3%, respectively. Sensitivity was 91.6% for levofloxacin, 95.0% for moxifloxacin, 91.8% for amikacin, and 87.4% for ethionamide, with a specificity of 96.1%, 96.1%, 99.1%, and 89.4%, respectively. For new and repurposed drugs, we find a sensitivity of 87.5% for bedaquiline, 82.6% for clofazimine, 57.1% for linezolid, and 75.0% for delamanid, with a specificity of 76.0%, 72.2%, 87.6%, and 72.4%, respectively (Table 3).

Discussion

We assess for the first time the extent to which machine learning can predict minimum inhibitory concentration of tuberculosis isolates based exclusively on whole-genome sequencing data for 13 antituberculosis drugs, including new and repurposed drugs. We use extreme gradient boosting, previously demonstrated to perform well on other bacteria and for binary tuberculosis prediction, and a set of unassembled genome-wide k-mers from sequencing reads instead of the more common approach of using genomic mutations. We followed best practice guidance for studies evaluating the accuracy of rapid tuberculosis drug susceptibility testing [18].

We find that our machine learning model can predict MICs with an essential agreement rate greater than 93% for first-line drugs, and 90% for other drugs. The WHO target product profiles for priority antituberculosis drugs, do not include thresholds and objectives for MIC prediction, focusing instead on binary metrics like sensitivity and specificity. However, essential agreements are similar to those reported for some other bacteria like Salmonella [10], N. gonorrhea [11], S. aureus [12], and E. coli [13] for which semi-quantitative data are more commonly generated.

MIC data could in the future help salvage the use of some drugs for some patients, by increasing in dose to overcome elevations in MIC that are only marginally above WHO-approved critical concentrations [19]. Much work need to be to identify how this might be done safely and effectively in the clinic, but being able to predict MICs from WGS data is a key step in that direction.

Current WHO guidelines for MDR-TB management recommend offering patients regimens including bedaquiline, linezolid, and potentially also clofazimine [19]. Our sensitivity and specificity for these three drugs fall below WHO TPP requirements. Our results reflect the very low prevalence of resistance to these drugs, which also explains the high negative predictive values and low positive predictive values. As more resistant isolates are collected, the sensitivity and specificity of the machine learning system will almost certainly increase, and negative predictive value decrease a little, as seen for other drugs. Nevertheless, a test with a high negative predictive value, lower positive predictive value, and sensitivity of 70% would still be helpful to clinicians [20]. Even imperfect test performance could still play a key role in preventing the amplification and dissemination of resistance.

Our system had higher sensitivity and negative predictive values on the independent test set compared to the expected performance of Xpert MTB/RIF and Xpert XDR for the drugs these probe, albeit at a small cost in specificity and positive predictive value. There are several explanations, including that the assays only look at eight genes and promoter regions, and exclude rare variants therein [5], while the machine learning system is able to explore genome-wide features, leverage interactions between features, and assess lineage and genetic background through genome-wide features. While we did not compare our approach to the new targeted next generation sequencing assays such as Deeplex MycTB (Genoscreen) [21], the ability to harness genome-wide information comprises a theoretical advantage over such assays.

WGS is likely to be more scalable than phenotypic drug susceptibility testing, with results for all drugs being derived from just a single test. Potential benefits of our approach to the analysis of WGS data include the ability to update and train automatically as new resistant samples are added. As resistance to drugs like bedaquiline emerges, this could help avoid the expensive multi-phase multi-year period required for the development of molecular assays [22,23], or the need to update catalogues through expert review [16]. The U.S. Food and Drug Administration recently released a regulatory framework for ‘live’ modifications to artificial intelligence and machine learning-based software as a medical device [24], and has recently provided clearance or approval for several such diagnostic devices [25], paving the way for clinical implementation and dissemination. Further advantages of our approach include that predictions are made for all isolates, avoiding the exclusion of 4–10% of samples with unknown mutations in candidate genes [17]. The use of k-mers from sequencing reads allows for genome-wide analysis whilst avoiding some of the vulnerabilities associated with read mapping or variant calling that can affect resistance prediction [26]. Whereas sites called with low confidence are filtered out of post-pipeline WGS outputs, our machine learning approach uses all k-mers from reads prior to processing by a pipeline as training features. Finally, by using an interpretable supervised machine learning algorithm, we provide a list of features used for prediction, which in turn can be hypothesis-generating in the search for causal mutations.

We note several limitations to our study. First, we were unable to validate essential agreement of MICs outside of the CRyPTIC dataset, due to the unavailability of such data (CRyPTIC is the first study to systematically collect MIC data across so many drugs). While we validate against binary phenotypes derived from MGIT, a priority of future work should be to perform external validation of MICs when such data becomes available. Second, there were a very small number of isolates phenotypically resistant to bedaquiline, linezolid, delamanid, and clofazimine in the CRyPTIC dataset. A third limitation is the use of a previously published literature-derived catalogue, rather than either of the more recently published WHO-endorsed catalogues [16], because the WHO catalogue of mutations associated with drug resistance in M. tuberculosis was developed using samples from both the CRyPTIC and independent datasets. It should be noted here that resolving any contradictions between catalogue-based predictions and ML predictions are intrinsically fraught, as ML predictions are not based on causality. Fourth, we report the performance of Xpert MTB/RIF and Xpert XDR in silico, but clinical performance of the actual tests might differ.

In summary, this study demonstrates that WGS can be combined with simple machine learning algorithms to provide MIC predictions for many drugs recommended for the treatment of susceptible and of MDR-TB. This study shows how a composite machine learning system could be used to help guide therapy, whilst being straightforward to updated as increasing numbers of resistant samples to new and repurposed drugs are collected. Further data and external validation are required before clinical implementation.

Methods

Author contributions

Conceptualization: David A Clifton, Timothy M Walker, Tim EA Peto, A Sarah Walker, Philip W Fowlder, Derrick W Crook, Alexander S Lachapelle

Data curation: Philip W Fowler, Zamin Iqbal, Martin Hunt, Daniel J Wilson, Sarah J Hoosdally, Ana Lúıza Gibertoni Cruz, Jeff Knaggs, Sarah Earle, Timothy M Walker, Alexander S Lachapelle

Formal analysis: Alexander S Lachapelle, Timothy M Walker

Funding acquisition: Camilla Rodrigues, David Moore, Derrick W. Crook, Daniela M. Cirillo, Zamin Iqbal, Nazir A. Ismail, Nerges Mistry, Stefan Niemann, Tim E.A. Peto, Guy Thwaites, A. Sarah Walker, Timothy M Walker, Daniel J. Wilson, David A Clifton

Investigation: The CRyPTIC Consortium

Methodology: Alexander Lachapelle, Timothy M Walker, Samaneh Kouchaki, Sarah G Earle, Yang Yang, David A Clifton

Project administration: The CRyPTIC Consortium

Resources: The CRyPTIC Consortium

Software: Alexander Lachapelle, Samaneh Kouchaki, David A Clifton

Supervision: Alexander S. Lachapelle, Timothy M Walker, David A Clifton

Validation: Alexander Lachapelle, Timothy M Walker

Visualization: Alexander Lachapelle

Writing – original draft: Alexander Lachapelle, Timothy M Walker

Writing – review & editing: The CRyPTIC Consortium

Supporting information

Acknowledgments

We thank Faisal Masood Khanzada and Alamdar Hussain Rizvi (NTRL, Islamabad, Pakistan), Angela Starks and James Posey (Centers for Disease Control and Prevention, Atlanta, USA), and Juan Carlos Toro and Solomon Ghebremichael (Public Health Agency of Sweden, Solna, Sweden).