Data source

This prospective observational study was conducted in one tertiary hospital (Donostia University Hospital), three regional hospitals (Deba Behea Hospital, Goi Urola Hospital, Bidasoa Hospital), and two private health clinics (Gipuzkoa Polyclinic and Asuncion Clinic), offering a total of approximately 1,725 beds and all located in the province of Gipuzkoa, Basque Country, Spain, with a population of 720,000.

The mean annual incidence rate of tuberculosis in Gipuzkoa from 2013 to 2020 was 14 cases per 100,000 population (and it remained under 20 cases per 100,000 throughout this period) [12]. Data collection has been highly reliable in Gipuzkoa since 1995, and a Tuberculosis Control Program was implemented across the entire region of the Basque Country in 2003 [13].

For the purpose of this study and following the approach of previous authors since the nineteen-nineties, the term “prevalence” was used to refer to the number of cases of a specific type of pleural effusion divided by the total number of pleural effusions studied in a given clinical setting: in other words, the “pre-test probability” [14, 15]. A prevalence of less than 10% is considered low [16]. Overall, 1,177 cases of pleural effusion were diagnosed in Gipuzkoa throughout the period 2013–2020, and 45 of them were tuberculous; hence, the local prevalence of TPE was 3.8%.

Due to the fact that virtually all cases of TPE become exudative and lymphocytic [17–19], and given our limited funding and the high cost of some tests (e.g., the Xpert MTB/RIF polymerase chain reaction [PCR] test for Mycobacterium tuberculosis—MTB -, which was performed in all cases), we decided to include only lymphocytic exudative pleural fluid samples in this study. During the whole study period, the first pleural fluid sample was only exudative and neutrophilic in 1 out of the 45 patients diagnosed with TPE; his case was included in the prevalence calculation, but his data were not included in the rest of the analysis.

We considered it appropriate to conduct this prospective study following a PIRTO strategy:

• P (Population): patients aged 16 years or older diagnosed with lymphocytic exudative pleural effusion from March 2013 to June 2020 in a low prevalence setting like ours
• I (Index Test): ADA level in pleural fluid, combined with patients´ age and routine pleural fluid parameters obtained from the first pleural fluid sample, and using machine learning algorithms
• R (Reference test): diagnosis according to the composite reference standard for TPE, that is, culture positive for MTB or Xpert MTB/RIF assay positive for MTB in pleural fluid, pleural tissue or sputum; or granulomatous inflammation in pleural tissue
• T (Target condition): diagnostic accuracy of the aforementioned parameters for TPE in a low prevalence setting
• O (Outcome): results obtained, in terms of sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) from two different machine learning approaches: tuberculous/other and tuberculous/malignant/other.

We report the above items by following the STARD guidelines [20]. Inclusion criteria in the study were: 1) age over 16 years, 2) de novo diagnosis of pleural effusion that is both exudative (according to Light´s criteria [21]) and lymphocytic (defined as a lymphocytes accounting for more than 50% of nucleated cells in pleural fluid [2]) in nature, and 3) etiology of the effusion determined according to the composite reference standard. Patients were not included in the study if they had a history of any disease that could explain the development of a pleural effusion (e.g., patients previously diagnosed with lung cancer who developed pleural effusion).

Further, patients in whom no definitive diagnosis was reached were not included in the study. When more than one thoracocentesis was performed, only the results (routine biochemical parameters, cell count and ADA level) from the first pleural fluid sample were included in the statistical analysis, but all samples were taken into account for obtaining the gold standard diagnosis of every patient (e.g., positive culture or malignant cells in a subsequent sample).

Through the whole period of study all the clinical information and reference standard results were available to all the members of the Consortium.

All patients gave written informed consent to their inclusion in the study before diagnostic procedures were performed. Only one patient was a minor. She was seventeen years old and her parents signed the informed consent. The protocol was evaluated and approved by the Clinical Research Ethics Committee of Euskadi (Record number 11/12).

All included patients reported demographic data (age, sex and nationality) and underwent the following types of testing:

• Pleural fluid tests including measurement of pH, glucose, protein, lactate dehydrogenase, and ADA levels, total and differential red and white cell counts, cytology, aerobic and anaerobic cultures, Lowenstein-Jensen medium and BACTEC MGIT (BACTEC Mycobacteria Growth Indicator Tube 960 System; BD Diagnostic Instrument Systems, Sparks, Maryland, USA) cultures, and Xpert MTB/RIF assays for MTB, as well as assessment of serum glucose, protein and LDH levels
• Aerobic, Lowenstein-Jensen medium and BACTEC MGIT cultures, along with cytology, of any spontaneous or induced sputum, when it was possible to obtain such samples
• Chest computed tomography (CT)
• Screening for human immunodeficiency virus (HIV), in line with the Tuberculosis Control Program for all patients diagnosed with tuberculosis implemented in the Basque Country since 2003 [13]
• Fibrobronchoscopy, depending on the CT findings, samples taken being sent for histopathological examination, aerobic and anaerobic cultures, Lowenstein-Jensen and BACTEC MGIT cultures, as well as Xpert MTB/RIF assays for MTB
• Pleural biopsy (either closed or thoracoscopic), when necessary to reach a diagnosis, samples taken being sent for histopathological exam examination, aerobic and anaerobic cultures, and Lowenstein-Jensen and BACTEC MGIT cultures, as well as Xpert MTB/RIF assays for MTB.

Pleural fluid ADA level was measured using an automated ultraviolet kinetic assay (Roche Diagnostics, Mannheim, Germany). White blood cell count was obtained with a automated hematology analyzer (Sysmex XN-1000, Roche Diagnostics), and a differential count was performed manually, after Wright staining. Mycobacterial cultures were performed on Lowenstein-Jensen medium and also in MGIT 960 culture tubes.

Regarding diagnostic criteria, patients were classified as having 1) tuberculous pleural effusion when cultures or Xpert MTB/RIF assays for MTB in pleural fluid, pleural biopsy or sputum were positive, or when granulomatous inflammation was evident in pleural tissue; 2) malignant pleural effusion (MPE) when malignant cells were found in pleural fluid or pleural biopsy tissue, or paramalignant pleural effusion when cancer was diagnosed de novo, no other cause of pleural effusion was identified and the pleural effusion and tumour were concurrent and developed in parallel, but no malignant cells were demonstrated in the pleural fluid or tissue [22, 23]; 3) parapneumonic effusion when there was any exudative pleural effusion associated with clinical and radiological pneumonia and complete clinical recovery and resolution of abnormal findings with antibiotics as the only treatment; and 4) miscellaneous effusion when they had any other type of pleural effusion with another aetiology and defined by well-established clinical criteria.

Calculation of sample size was performed by using a statistical program (Epidat 3.1). For a sensitivity of 95%, specificity of 90%, prevalence (pre-test probability) of 10%, significance level of 5%, power of 80% and precision of 5%, the minimum sample size was 200 patients. Since a 25% loss was estimated, final sample size was set to 250 patients.

Fig 1 shows a flowchart depicting the patient selection process.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Patient selection process.

https://doi.org/10.1371/journal.pone.0259203.g001

Statistical analysis

With the data collected from the patients included, two different analyses were performed. The main analysis used machine learning techniques to classify samples in two categories: tuberculous and non-tuberculous. A secondary analysis used the same techniques to additionally distinguish malignant cases, yielding three categories: tuberculous, malignant and “other”.

Both analyses split the data into a training set (80%) and test set (20%), to maintain the proportion of the three classes in both sets (19.1% tuberculous, 53.9% malignant and 26.9% other). The training and testing processes used the following features: age, ADA level and routine parameters from pleural fluid (pH, glucose, protein, and lactate dehydrogenase levels and total and differential red and white cell counts). S1 Table shows an estimate of the relative importance of these features in the classification process. We carried out a data standardization process before training: for each feature in the training set, we first subtracted the mean value (such that standardized values always had a zero mean), and then divided by the standard deviation (such that the resulting distribution had unit variance). The mean value and standard deviation calculated in the training set were also used in the standardization process of the test set.

As we did not have a priori knowledge about what type of classifier would work best in the proposed problem, six of the most widely used were trained and tested: multi layer perceptron, logistic regression, support vector machine, decision tree, K-nearest neighbors and random forest. We used the Python scikit-learn implementation of these classifiers [24]. A set of different values was considered for each of the classifier parameters [25] in combination with a 5-fold cross validation [26], to determine the combination of parameters that optimized a given metric. In the case of the classification between tuberculous and non-tuberculous, a binary classification problem, we used accuracy, the AUC and the F1 score, which is the harmonic mean of precision and recall [27]. The AUC led to the best parameterizations in this case. In the classification between tuberculous, malignant and other effusions, this being a multiclass classification problem, we used different metrics, namely, the weighted F1 score, balanced accuracy and the one-vs-rest AUC, the last of these yielding the best parameterizations.

In the binary classification problem (tuberculous vs. non-tuberculous), the validation predictions were used to calculate receiver operating characteristic (ROC) [28] and precision-recall (PR) [29] curves in order to obtain decision thresholds that maximized a given metric. The Youden index [30] was used to select a good balance between sensitivity and specificity in the ROC curves. The F1 score was used in the PR curves. It is worth mentioning that these predictions were “clean”, that is, they were obtained from the validation subset using a classifier trained with the rest of subsets within each fold of cross validation. These clean predictions were aggregated to calculate the validation results for each method.

In the three-class classification problem, the goal was to maximize the number of correct classifications of tuberculosis and MPE. Misclassification of samples of the class “Other” was considered a minor problem.

We applied two different approaches to accomplish this task. The first was a classification pipeline that used the previously trained binary classifier in combination with another classifier that distinguished between malignant effusions and the rest (Fig 2). Hence, a sample was analyzed in the first classifier, and if that classifier predicted tuberculosis, that was the final classification for that patient. Otherwise, the second classifier analyzed the sample to predict whether it corresponded to a tumor or some other disease. This second classifier was trained with the same training set as the first one, but not taking into consideration the samples corresponding to tuberculosis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Classification pipeline.

Classification pipeline.

https://doi.org/10.1371/journal.pone.0259203.g002

This first approach had the advantage that the thresholds of the two classifiers could be adjusted seeking to maximize the number of tuberculosis cases that were correctly predicted as such in the first classifier, as well as the number of tumors correctly classified in the second one.

The second approach to solve this problem was a multiclass classifier that distinguished at the same time between tuberculous, malignant and other effusions. Using a single classifier reduced the training effort but complicated the adjustment of the decision thresholds to make it possible to prioritize the prediction of tuberculosis and tumors, over the class “Other”. In addition to all the aforementioned analysis, and considering the dynamics in our previous retrospective study, we also assessed the diagnostic accuracy of ADA level in this “lymphocytic exudative” scenario in terms of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Thereafter, we did the same, incorporating all the variables included in the machine learning algorithm. Our aim was to perform a Bayesian analysis of both sets of diagnostic variables.

Classifying tuberculous vs. non-tuberculous effusions

Figs 3 and 4 show the ROC and PR curves for the binary classifier, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Receiver operating characteristic curves using validation predictions.

The dots correspond to the points that maximize the Youden index.

https://doi.org/10.1371/journal.pone.0259203.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Precision-recall curves using validation predictions.

Precision-recall curve for each method using the validation predictions. The dots correspond to the points that maximize the F1 score.

https://doi.org/10.1371/journal.pone.0259203.g004

Table 2 lists performance scores for each method using three different thresholds: 0.5, the one that maximizes the Youden index on the ROC curve and the one that maximizes the F1 score on the PR curve. KNN was the classifier with the lowest value for the AUC (0.94); Logit and DT achieved an AUC of 0.97; whereas SVC, RF and MLP obtained 0.98.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Accuracy (Acc), sensitivity (SEN), specificity (SPF) and F1 score (F1) of all the classifiers in the validation stage, using three different thresholds (T): 0.5, the one that maximizes the Youden index on the receiver operating characteristic curve and the one that maximizes the F1 score on the precision-recall curve, and their confidence intervals at 95% level.

https://doi.org/10.1371/journal.pone.0259203.t002

The classifiers with the best set of parameters (S4 Table) and decision thresholds found were trained with the complete training set (184 samples). These classifiers were then tested using the 46 samples of the test set. Table 3 shows the scores for each method using the best threshold found in the validation stage.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Threshold (T), area under the curve (AUC), accuracy (Acc), sensitivity (SEN), specificity(SPF) and F1 score (F1) of all the classifiers in the test stage, using the best thresholds found in the validation stage with their confidence intervals at 95% level.

https://doi.org/10.1371/journal.pone.0259203.t003

In the validation stage, as can be seen in Table 2, the SVC achieved the best results: accuracy of 97%, AUC of 0.98, sensitivity of 91% and specificity of 98% (threshold of 0.35). In the testing phase (Table 3), however, the best results were obtained with the logistic regression: accuracy of 96%, AUC of 0.96, sensitivity of 100% and specificity of 95% (threshold of 0.28).

Classifying tuberculous vs. malignant vs. other effusions

The first approach to performing a three-fold classification used the previous binary classifier for its first stage. The results obtained for the second stage of the pipeline are provided in the Supporting information section: S1 and S2 Figs, S5 and S6 Tables. Considering the validation results for each of the binary classifiers of the pipeline, we decided to use the SVC in both stages.

The pipeline achieved an overall accuracy of 79% in the validation predictions and 80% in the test set. To determine how the three-class classifier behaved in terms of sensitivity and specificity in the two-class problem (Tuberculous and Other), we have combined the results of the classes “Malignant” and “Other” into a single class. By doing this, the sensitivity and specificity would remain the same in the Tuberculous/Other classification, but this type of classifier would also allow us to detect 90% of malignant cases.

The second approach to solve the multiclass classification problem was to train, from scratch, a classifier that could distinguish between tuberculous, malignant and other effusions. The scores obtained with these classifiers are reported in the Supporting Information section: S7 and S8 Tables.

None of these classifiers outperforms the combination of classifiers from the previous section in the validation predictions. As can be seen in Table S6 Table, SVC and decision tree classifiers obtain the best results in terms of accuracy (76% and 74%, respectively). The decision tree achieved a sensitivity of 91% and specificity of 94% in the Tuberculous/Other classification.

As a result, in the validation predictions, we were able to classify correctly 32 out of 35 cases of TPE and 89 out of 98 cases of MPE (sensitivity of 91.4% for TPE and 90.8% for MPE); conversely, a total of 3 additional cases were mistakenly classified as TPE out of 149 non-TPE cases and 29 cases were mistakenly classified as MPE out of 86 non-MPE cases (specificity of 98.0% for TPE and 66.3% for MPE). Furthermore, all 3 cases of TPE misdiagnosed were classified as MPE, theoretically leading to additional diagnostic procedures, and only 6 out of the 98 cases of MPE were misdiagnosed as “Other”, that is, 93.9% of cases of MPE (92/98) were classified as MPE or TPE. The relatively low specificity of the classification for MPE was mainly due to overdiagnosis of MPE amongst cases in the other diagnosis group. Indeed, we deliberately implemented the classification into the three groups prioritizing the overdiagnosis of MPE over its underdiagnosis.

In the test set, we were able to correctly classify 8 out of 9 cases of TPE and 24 out of 26 cases of MPE (sensitivity of 88.8% for TPE and 92.3% for MPE); conversely, a total of 2 additional cases were mistakenly classified as TPE out of 37 non-TPE cases and 6 cases were mistakenly classified as MPE out of 20 non-MPE cases (specificity of 94.5% for TPE and 70.0% for MPE). Only 1 of the 26 cases of MPE was misdiagnosed as “Other”, that is, 96.1% of cases of MPE (25/26) were classified as MPE or TPE.

Performance of the classifiers versus ADA alone

We compared the results of our machine-learning-based classifiers with those derived from using ADA as the only diagnostic criteria. We obtained three thresholds for ADA: default threshold (in this case 40 U/l), the threshold that maximized the Youden index in the ROC curve, and the threshold that maximized the f1-score in the PR curve. It is interesting to note that the threshold that maximized the Youden index with our samples was 40 U/l as well. The threshold that maximized the f1 score in the PR curve was 43 U/l. With these thresholds two set of metrics were extracted:

• Threshold = 40.00: Accuracy 0.93, Sensitivity 0.97 and Specificity 0.93
• Threshold = 43.00: Accuracy 0.95, Sensitivity 0.94 and Specificity 0.95

Our classifiers obtained better Sensitivity/Specificity values. For example, Logistic Regression obtained 0,97/0,94 (youden 0.91). SVC and MLP gets 0,94/0,97 (youden 0.91). Also, better Accuracy is obtained with SVC (0,97) and MLP (0,96).

Regarding the Bayesian analysis of the diagnostic accuracy of two sets of diagnostic variables and with a pre-test probability of 3.8%:

• for ADA >40 U/l (plus lymphocyte percentage > 50%), sensitivity, specificity, PPV and NPV were 97%, 93%, 35% and 100% respectively.
• for the whole set of variables included in the machine learning algorithms, sensitivity, specificity, PPV and NPV were 91%, 98%, 64% and 100% respectively.

Pre- and post-test probabilities of both diagnostic options are shown in Fig 5. Additionally, Table 4 lists the aforementioned parameters with respect to the corresponding pre-test probability up to 50%. Notably, for a pre-test probability of 10% (the upper limit of low prevalence) and using the full set of variables included in the machine learning algorithms, the PPV already reaches 83.5% and NPV remains at 99%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Post-test probability of TPE after positive (top) or negative (bottom) results of 1) ADA>40 U/l plus implicit lymphocyte percentage >50% alone or 2) in addition to age and routine pleural fluid parameters included in the machine learning algorithms; for different pre-test probabilities of disease.

https://doi.org/10.1371/journal.pone.0259203.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Comparative table of Bayesian probabilities of test parameters used: ADA > 40 U/l (plus lymphocyte percentage > 50%) versus the whole set of variables included in the machine learning algorithms.

PPV: positive predictive value. NPV: negative predictive value.

https://doi.org/10.1371/journal.pone.0259203.t004