Ethics approval and consent to participate

The research was performed in accordance with the Declaration of Helsinki. Collection of the clinical strains and the determination of the genotypic drug resistance profile was obtained from the Human Research Ethics Committee of the University of the Witwatersrand in South Africa and the Institutional Review Board of the University of North Carolina at Chapel Hill in the United States for the training dataset, and from the Stellenbosch University Health Research Ethics Committee for the validation dataset. Participants in the training dataset gave verbal informed consent by phone (recorded) as approved by the Human Research Ethics Committee of the University of the Witwatersrand in South Africa and the Institutional Review Board of the University of North Carolina at Chapel Hill in the United States. The Research Ethics Committee or Institutional Review Board of Human Research Ethics Committee of the University of the Witwatersrand in South Africa and the Institutional Review Board of the University of North Carolina at Chapel Hill in the United States waived the need of informed consent for patients who had died or were lost to follow-up from TB care prior to study enrolment and could not be contacted despite multiple attempts. For the validation dataset, the Stellenbosch University Health Research Ethics Committee waived the need for informed consent. The study titled "A personalized recommendation system for Whole Genome Sequencing-based individualized treatment for drug resistant tuberculosis" reported in this manuscript was approved by the Ethics Committee of the University Hospital Antwerp and the University of Antwerp.

Development of the RR-TB treatment recommender CDSS

The development process of the RR-TB treatment recommender CDSS consisted of the assembly of the knowledge base, development of a heuristic model prototype, feedback harvesting from experts, application of machine learning methods to analyze the feedback, assessment of the CDSS performance, and external validation [14]. The stakeholder group included experts in pathogen genomics (genotype-phenotype associations), pharmacology (drug properties, mechanisms of action, drug-drug interactions, synergy and antagonism between drugs), medicine (treatment of RR-TB in high and low burden settings), and computer and data science. In addition, public health practitioners and patients were included to hear their viewpoints. Stakeholder discussions were held in 2019 and focused on the minimum number of effective drugs to be included in a regimen, the role of an intensive treatment phase, drug toxicity, burden of treatment monitoring, drug properties that define effectiveness, and level of drug resistance.

The key features of individual drugs and treatment regimens were quantified by five experts in 2019, through expert panel discussions guided by review of published and unpublished data. The key features identified by the stakeholders were toxicity, early bactericidal activity (EBA), bactericidal activity, sterilizing activity, mode of administration (oral vs injection), mechanism of drug action, propensity to acquire resistance, and cost (S1 Table). After the first round of feedback, QTc prolongation was added as a separate feature, resulting in nine features to characterize individual drugs in the model. Toxicity was classified for each drug as life threatening, permanent (e.g., hearing loss), short-term with possible effect on adherence (e.g., nausea) or minimal without patient impact (e.g., liver function test abnormality grade 2). Based on a combination of frequency of occurrence and severity, the level of toxicity for each drug was graded on a scale of 1 to 3. QTc prolongation was classified into four categories (none, low, moderate, or high). Early bactericidal activity (EBA), bactericidal activity and sterilizing activity were classified as one of five categories (very low, low, moderate, high, very high). EBA classification was based on the drop in log₁₀ colony forming units (CFU) in the first days of treatment observed in experimental EBA studies and bactericidal activity on the drop in log₁₀CFU in the first six months of treatment. Sterilizing activity, or the ability to achieve stable cure was defined based on a drug’s ability to prevent relapse in human or animal model studies. The propensity to acquire resistance was classified into four categories (low, moderate, high, and none) based on expert opinion. Mode of administration was defined as either oral or injection (intravenous or intramuscular). All cost for the drugs at recommended dosing was quantified as the cost for one month of treatment in South Africa [15–18].

In the heuristic model prototype all features received equal weight, possible interaction between features were not considered, and all regimen features were normalized to obtain a regimen score with a higher score representing a better regimen.

A training dataset of 129 unique WGS-derived drug resistance profiles was obtained using WGS data and interpretation of variants based on the 2021 WHO catalogue of mutations in Mtb [19]. Of these 129 unique resistance profiles, 119 (92%) were resistant to rifampicin, 106 (82%) to isoniazid, and 29 (22%) to fluoroquinolones from 303 anonymized South African RR-TB (accessed on 2017.10.10) patients was used to solicit feedback from experts. For each unique drug resistance profile, all available four-drug regimens (defined as a regimen not containing any drug to which the Mtb isolate is resistant on WGS) were generated by the treatment recommender model, using all drugs that can be prescribed for RR-TB treatment. A sampling function was used to increase the likelihood for feedback on highly ranked (i.e., better) compared to lower ranked regimens. Regimens were replaced to ensure that multiple experts could provide feedback on the same regimen. Experts were asked to state if they would prescribe the proposed regimen to a patient with the specified resistance profile. If they rejected the regimen, the reason for rejection and suggestion of an alternative regimen was requested. After the first round of feedback, experts discussed which features were missing, obsolete, or sub-optimally quantified and revisions to the model were made. After each round of expert feedback harvesting, a random forest classifier using 100 trees with a maximum depth of 10 and default parameters (from the scikit-learn library [20]) was trained and evaluated using the feedback from the experts. After each round of machine learning, a patient level ‘leave-one-out’ cross validation strategy was used to assess model performance [14]. A leave-one-out cross validation at the patient level means that the model was trained on the data for all patients except one. The dataset and was then used to make recommendations for the excluded patient. This analysis was then repeated until all patients were excluded once. This analytic approach was chosen due to the limited amount of data and because observations were not independent of each other. The random forest classifier was then used to predict for each possible treatment regimen the probability that a regimen is classified as ‘good’. Finally, the regimens were ranked according to these probabilities to identify the optimal (highest ranked) regimen. The result of the machine learning analysis was then used to develop the next version of the treatment recommender model. The iterative feedback harvesting process was continued until no further significant improvements were made in the model prediction of a ‘good’ treatment for a specified resistance profile. After 3 rounds of feedback and machine learning, the treatment recommended by the model was accepted by the experts in 89%, 95%, and 95% of the cases, demonstrating no further improvement after the third round. [14]. The machine learning model and its parameters were decided upon before the start of the study and kept fixed throughout it, since the training data had to be iteratively generated through the expert feedback harvesting and we did not want to update the methodology mid study.

The model was then validated using a dataset of 64 unique anonymized WGS-derived drug resistance profiles that were not present in the training dataset (accessed on 2020.01.13) [14].

For the final treatment recommender CDSS, the model was retrained on a dataset of 3895 treatment regimen-expert feedback pairs obtained from 268 unique WGS-derived resistance profiles consisting of 129 unique resistance profiles used during training, the 64 unique resistance profiles used during validation, and 75 unique resistance profiles generated from the 129 profiles excluding pretomanid, high-dose rifampicin, and cycloserine as possible drugs. As the model was to be used in a clinical trial, the model was retrained using all available resistance profile-treatment regimen data with expert feedback to make sure the final model included as much training data as possible. Pretomanid, high-dose rifampicin and cycloserine were excluded at this stage as these drugs were not recommended or used for clinical care in South Africa at the time of model development, due to the cost involved (timewise for the experts providing their feedback) as reevaluating all the original resistance profile-treatment regimen pairs would not be worth the time investment. Additionally, this also allows shows that the exclusion of certain drugs is possible should the clinical setting require it. The treatment recommender model with training data and instructions on usage are available from https://github.com/LennertVerboven/treatment_recommender.

Development of interactive online web interface

We developed a user-friendly, interactive web application optimized for mobile phone (available from https://github.com/LennertVerboven/treatment_recommender_webapp). The front end was written in JavaScript using the React library. The back end written in Python3, using the Django web development framework, exposes a representational state transfer (REST) application programming interface (API) to which the clients can make secured hypertext transfer protocol secure (HTTPS) requests. The backend implements the connection between the REDCap [21] server containing the patient data and the treatment recommender machine learning model to make individualized treatment recommendations.

The online web interface allows health care workers to log in securely using username and password authentication and then shows a list of their patients. For each patient, the app lists the patient’s demographic information (name, date of birth, clinic, phone number), current treatment regimen, contra-indications based on clinical information, toxicity, stockouts, the WGS-based resistance profile, and the recommended regimen. The physician can then start an interactive process to enter any contra-indications, toxicity, or drug stockouts for any of the drugs included in the recommended regimen, after which an updated treatment regimen recommendation is proposed in real-time. This iterative process is completed once the physician accepts the treatment recommendation. At that time, a pdf report is generated for inclusion in the patients’ paper medical file.

To ensure data security, all data transfers between the front end and the server hosting the webapp used an encrypted connection. All patient data was stored on a secure server at the University of Stellenbosch [21].

Acceptance assessment of the RR-TB treatment recommender CDSS

The treatment recommender CDSS and its accompanying webapp were implemented as part of the SMARTT pragmatic trial (Clinicaltrials.gov Identifier NCT05017324) [22]. The trial aimed to determine the effectiveness of a WGS-based drug susceptibility testing strategy to guide individualized treatment for patients diagnosed with RR-TB.

Acceptance of the treatment recommender and its accompanying online app by clinicians caring for trial participants was assessed in three different ways. First, we investigated the proportion of trial participants for whom the treatment proposed by the treatment recommender CDSS was prescribed. Because this is an ongoing trial, we performed this analysis on the first 20 patients randomized to the intervention (WGS) arm. Second, we determined the proportion of patients for whom clinicians would prescribe the regimens proposed by the treatment recommender in their routine practice, i.e., outside of the trial setting. This was done by presenting the treatment regimens recommended by the CDSS for 15 unique resistance profiles to five physicians who had experience with the CDSS in the trial setting and agreed to participate in this assessment. Resistance profiles were selected from those that have been documented in the region where these physicians practice. Physicians were asked if they would prescribe the proposed regimen for a patient receiving routine RR-TB care. Third, we performed a survey to assess the factors that may influence health care workers’ use of the treatment recommender CDSS. The questionnaire (S2 Table) was developed based on the modified technology acceptance model (Fig 1) and consisted of 36 questions investigating eight domains within the technological, individual, and organizational context (Table 1) [23]. Participants were asked to rate each question on a 5-point Likert scale. Responses were transformed into a score ranging from 0 (Strongly disagree) to 4 (strongly agree), except for questions 8 and 32 for which the rating was inverted as these questions were posed negatively. For each domain, the median and range of the sum of the scores were calculated.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Modified technology acceptance model as created by Gagnon et al. [23].

https://doi.org/10.1371/journal.pone.0306101.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The eight domains of the technology acceptance model and their definitions [23].

https://doi.org/10.1371/journal.pone.0306101.t001

Selection and quantification of individual drug features

The fluoroquinolones, moxifloxacin and levofloxacin are oral drugs that inhibit DNA gyrase of Mtb [24]. Fluoroquinolone toxicity was classified as low (score 1 for levofloxacin, 1.25 for moxifloxacin) given the low (1.2%) incidence of serious adverse effects (SAE) in RR-TB patients receiving fluoroquinolones [25]. QTc prolongation on moxifloxacin is moderate [26] and experts classified the risk of QTc prolongation for levofloxacin as low. The monthly cost is $3.87 for levofloxacin and $8.85 for moxifloxacin [18]. The EBA of fluoroquinolones is very high based on a fall in log₁₀CFU of 0.45 for moxifloxacin and 0.53 for levofloxacin [27]. Experts classified the bactericidal activity as moderate for levofloxacin and high for moxifloxacin. The sterilizing activity of fluoroquinolones was judged to be moderate [28–30]. Experts determined that fluoroquinolones have a low propensity to acquire resistance.

Rifabutin is an oral drug that kills mycobacteria by inhibiting the Mtb RNA polymerase [31]. Toxicity was classified as low (score 1.5) [32]. Experts stated that rifabutin does not prolong the QTc interval. The cost of 1 month treatment is $57.6 [18]. The EBA of rifabutin is low (fall in log₁₀CFU 0. 0.07 [33]) and sterilizing activity is very high [34]. Experts classified the bactericidal activity as moderate. Rifabutin has a low propensity to acquire resistance.

Bedaquiline is an oral drug that inhibits the Mtb ATP synthesis [35]. Experts classified the toxicity for bedaquiline as low (score 1). The risk of QTc interval prolongation during bedaquiline treatment is moderate. Cost of one month bedaquiline treatment is $65 [18]. The EBA of bedaquiline is low with a fall in log₁₀CFU of 0.04 [36] in the first two days of a loading dose of 400mg/day. Experts classified the bactericidal activity as moderate and sterilizing activity as very high. Bedaquiline has a moderate propensity to acquire resistance.

Clofazimine is an oral drug that inhibits protein synthesis [37]. Experts classified the toxicity of clofazimine as relatively high (score 2.25), it has no effect on the QTc interval [38]. The monthly cost of clofazimine treatment is $29.96 [18]. Clofazimine has a very low EBA as clofazimine treatment does not result in a drop in CFU counts in the first week of treatment [39]. Experts classified the bactericidal activity as moderate and sterilizing activity as low, and it has a low propensity to acquire resistance.

Linezolid is an oral drug that inhibits protein synthesis [40]. The risk of toxicity caused by linezolid is high (score 3) [41]. Expert opinion stated that linezolid has no effect on the QTc interval. The cost is $42.7 per month of linezolid treatment [18], it has a moderate EBA with a 0.17 drop in log₁₀CFU in the first two days of treatment [27]. Experts classified the bactericidal and sterilizing activity of linezolid as high and the propensity to acquire resistance as low.

Ethambutol is an oral first line drug that inhibits the Mtb cell wall synthesis [42]. Toxicity is low (score 1), with a risk of SAE of 0.5% [25]. Expert opinion stated that ethambutol has no effect on the QTc interval. One month of ethambutol treatment cost $2.19 [18]. Ethambutol has a low EBA with a 0.245 drop in log₁₀CFU in the first two days of treatment [43]. Experts classified the bactericidal activity as moderate, and it has almost no sterilizing activity [34] and a moderate propensity to acquire resistance.

Pyrazinamide is an oral drug that disrupts plasma membranes [44]. Toxicity of pyrazinamide is low (score 1) based on a 2.8% risk of SAE on pyrazinamide treatment [25]. Experts judged that pyrazinamide treatment has no effect on the QT interval. One month of treatment cost $2 [18]. Pyrazinamide has very low EBA [43, 45] and very high sterilizing activity [34]. Experts classified the bactericidal activity as moderate. Pyrazinamide has a high propensity to acquire resistance.

Isoniazid is an oral drug that inhibits Mtb mycolic acid synthesis [46]. Experts classified the toxicity as low (score 1) for isoniazid and high-dose isoniazid (score 1.25). Experts judged that isoniazid has no effect on QTc interval. One month of treatment cost $0.61 for isoniazid and $1.22 for high-dose isoniazid [18]. Isoniazid has a very high EBA with a drop in log₁₀CFU of 0.50 [34] and moderate sterilizing activity [34]. Experts classified the EBA of high-dose isoniazid also as high and viewed the bactericidal activity of isoniazid and high-dose isoniazid as high. It has a low propensity to acquire resistance.

The thioamides, ethionamide and prothionamide, are oral drugs that inhibit Mtb cell wall synthesis [47]. Thioamides have a high toxicity (score 3) with a 8.2% SAE risk [25]. Experts judged that thioamides do not influence the QTc interval. The cost of one month of treatment is $11.8 for ethionamide and $18.95 for prothionamide [18]. Experts classified the EBA and bactericidal activity of thioamides as moderate, the sterilizing activity as low. Thioamides have a moderate propensity to acquire resistance.

Carbapenems, imipenem-cilastatin and meropenem, are administered via daily infusion together with oral clavulanate. They inhibit peptidoglycan synthesis [48]. Experts classified carbapenems as moderately toxic (score 1.75). Experts stated that carbapenems do not affect the QTc interval. Carbapenems are expensive drugs with a cost of $439.2 for one month imipenem and $338.55 for one-month meropenem [18]. The EBA is very low with small increase in log₁₀CFU in the first 14 days of imipenem treatment [49]. Experts classified carbapenems as having a low bactericidal activity. In combination with clavulanate, carbapenems have a high sterilizing activity [50]. Carbapenems have a low propensity to acquire resistance.

The aminoglycosides (amikacin, kanamycin, and capreomycin) and streptomycin, drugs which need to be administered by daily intramuscular injection, inhibit protein synthesis [51]. Of the aminoglycosides only amikacin is currently recommended for treatment of RR-TB. Amikacin and streptomycin treatment is highly toxic (score 3) with 7.3% of patients reporting SAE for both injectables [25]. Experts judged that amikacin and streptomycin have no effect on the QTc interval. The EBA of the injectable drugs is low, with a loss of log₁₀CFU of 0.05 for amikacin [27] and 0.04 for streptomycin [27]. The bactericidal activity and sterilizing activity of amikacin and streptomycin are also low [34, 52]. Both amikacin and streptomycin have a low propensity to acquire resistance.

Cycloserine and terizidone are oral drugs that inhibit protein synthesis [53]. These drugs have a high level of toxicity (score 2.75 for cycloserine and 3 for terizidone with a 4.5% and 9.1% SAE risk in patients treated with cycloserine and terizidone, respectively) [25, 54]. Experts stated that these drugs do not influence the QTc interval. The cost of one month treatment is $32.7 for cycloserine and $219.6 for terizidone [18]. Experts classified the EBA, bactericidal and sterilizing activity as low, moderate, and low for both drugs respectively. These drugs have a moderate propensity to acquire resistance.

Para-aminosalicylic acid (PAS) is an oral drug that inhibits DNA precursor synthesis [55]. PAS treatment is highly toxic (score 3) as 12.2% of patients treated with PAS reporting SAE [25]. There is no data to suggest that PAS influences QT interval. One month of PAS treatment costs $122 [18]. PAS has a moderate EBA with a drop in log₁₀CFU of 0.259 in the first two days of treatment [45]. Experts classified the bactericidal activity of PAS as low and its sterilizing activity as moderate. PAS has a moderate propensity to acquire resistance.

The nitroimidazo-oxazoles, delamanid and pretomanid are oral drugs that inhibit mycolic acid synthesis [56]. At time of the development of the treatment recommender, only delamanid was registered for RR-TB treatment in South Africa. Experts classified toxicity of delamanid as low (score 1). QT prolongation is also low [57]. One month of delamanid treatment costs $308.63 [18]. Delamanid has a low EBA (drop in log₁₀CFU of 0.066 [27]), a low bacterial and moderate sterilizing with dose-dependent killing rates. At the highest dose of delamanid, its sterilizing activity is superior to isoniazid and equal to rifampicin [58]. Delamanid has a moderate propensity to acquire resistance.

Selection and quantification of regimen features

Experts agreed that an effective treatment regimen should consist of four effective drugs i.e., drugs to which no resistance has been detected. Fourteen features were defined to characterize the treatment regimens in the model. Features one to six are the sum of features of the four drugs included in the regimen: the sum of EBA, bactericidal activity, sterilizing activity, toxicity, propensity to acquire resistance and cost. Feature seven to nine assess whether the regimen adheres to the principles to construct a TB treatment regimen developed by Van Deun et al. [59] which state that a regimen should contain 1) at least one core drug that has a high bactericidal and sterilizing activity, 2) at least one highly bactericidal companion drug, one moderately bactericidal companion drug, and two moderately sterilizing companion drugs (a single companion drug can satisfy multiple requirements) and 3) a combination of a core and companion drugs. Features 10 and 11 quantify either the number of interactions between drugs in the regimen synergistic (pyrazinamide and bedaquiline, pyrazinamide and clofazimine, pyrazinamide and delamanid [60, 61]) or antagonistic (moxifloxacin and rifampicin, bedaquiline and rifampicin [62, 63]). Feature 12 checks whether all four drugs have a different mechanism of action. Given the current goal to preferentially administer all-oral regimens, feature 13 indicates whether the regimen is an all-oral regimen. Because experts rejected regimens containing two or more drugs with a moderate to high QTc prolongation effect during feedback, feature 14 was added to evaluate whether the sum (with low, moderate, and high equaling 1, 2, 3 respectively) of the QTc prolongation of the drugs is greater than 3.

Binary features are directly used as input for the treatment recommender model, while the continuous features undergo two transformations. Each continuous feature is therefore represented twice in the final treatment regimen feature set. The continuous features are normalized from 0 to 1, where the regimens with the highest score and lowest score obtain a value of 1 and 0 for that feature respectively. The second transformation divides the feature range into four parts (very high, high, low, very low) and classifies each continuous feature withing these classes. Negative features, such as cost, toxicity, antagonism, and propensity to acquire resistance are inverted such that higher values always equal patient benefit. Given that the total range for features varies when regimens are excluded (due to resistance), these transformations are performed after eliminating regimens containing drugs to which resistance was detected.

Table 2 shows the 14 features (only showing the continuous features before transformation) of the 10 highest ranked regimens by the CDSS for a patient infected with a Mtb strain that is resistant to rifampicin, isoniazid, ethambutol, ethionamide, prothionamide, rifabutin, and streptomycin. The regimen ranked number 1 would be recommended, the other nine regimens ranked in the top 10 for this resistance profile are increasingly suboptimal treatment regimens and the 100^th ranked regimen is an indication of a poor regimen for this resistance profile. The recommendation probability is the probability that the treatment recommender would classify this regimen as appropriate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The 10 highest ranked regimens (plus the 100th ranked regimen) for a Mtb strain.

https://doi.org/10.1371/journal.pone.0306101.t002

Acceptance of the RR-TB treatment recommender model in a clinical trial setting

Of the first 20 patients enrolled in the SMARTT trial for which the treatment recommender was used to recommend the individualized treatment regimen, all 20 (100%) patients received the regimen recommended by the treatment recommender CDSS when taking the Mtb resistance profile, clinical data of the patient and drug availability at the clinic into account. For 7 (35%) patients, the physician used the webapp to make changes to the initial recommendation had to be adjusted because of the presence of a clinical contra-indication for one of the drugs (n = 3) or because one of the drugs included in the recommended regimen was not available at the clinic (n = 4), information which was not available when the initial regimen was computed. For these seven patients the treatment recommender CDSS was rerun after the physician entered this information in the webapp, and these patients were prescribed the recomputed treatment regimen (Fig 2). For one patient, the regimen had to be recomputed at week 8 of individualized treatment due to a serious adverse effect to linezolid.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Treatment individualization flow for the first 20 patients in the SMARTT trial WGS arm.

https://doi.org/10.1371/journal.pone.0306101.g002

Acceptance of treatment recommender CDSS in a routine care setting

All physicians agreed to prescribe the recommended regimen for 4 of the 15 profiles, and all physicians disagreed with 2 of the 15 profiles (Table 3). The reason for most rejections were deviations from national RR-TB treatment guidelines. The most common reasons were that the South African guidelines state that bedaquiline should not be combined with moxifloxacin and that group A drugs should always be combined with group B drugs. In addition, a regimen was rejected because the physician believed that resistance to isoniazid always occurs in case of ethionamide resistance, which is not the case.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Acceptance of the CDSS recommendation for individualized RR-TB treatment in routine care setting.

https://doi.org/10.1371/journal.pone.0306101.t003

Acceptance of the treatment recommender CDSS system and its accompanying webapp

When using the modified technology acceptance model as a guide to assess the acceptance of the treatment recommender CDSS and its accompanying webapp (S2 Table), we found that the score was high (≥75%) for most domains except for the subjective norm (67%) and facilitators domain (58%) (Table 4). The facilitators domain scored low because physicians stated that their settings did not have the required infrastructure. Internet connection or access to email is not always available in more rural settings and they often use personal devices that do not have infrastructure support. The full questionnaire with answers is available in S2 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Acceptance of the treatment recommender CDSS by technology acceptance model domain.

https://doi.org/10.1371/journal.pone.0306101.t004