ࡱ > { bjbjzz 4 m + 8 T h , 8 T r w ( " [# ` 7 7 7 7 7 7 7 $ ; = r 8 i # w w # # 8 x8 ( ( ( # 1 8 ( # 7 ( ( : - , - Ke ' -
1 8 0 8 -
6> ( d 6> - - T 6> ). t # # ( # # # # # 8 8 ( # # # 8 # # # # 6> # # # # # # # # # : Text S1
Laboratory Methods
RNA sample extraction and processing
Whole blood (2.5ml) was collected into PAXgene blood RNA tubes (PreAnalytiX), incubated for 2 hours, frozen at -20oC within 3 hours of collection, and then stored at -80oC. RNA was extracted using PAXgene blood RNA kits (PreAnalytiX) according to the manufacturers instructions at one site (Cape Town) to minimize any sample handling bias. The integrity and yield of the total RNA was assessed using an Agilent 2100 Bioanalyser and a NanoDrop 1000 spectrophotometer respectively. Total RNA was then shipped to the Genome Institute of Singapore. After quantification and quality control, biotin-labeled cRNA was prepared using Illumina TotalPrep RNA Amplification kits (Applied Biosystems) from 500ng RNA. Labeled cRNA was hybridized overnight to Human HT-12 V4 Expression BeadChip arrays (Illumina). After washing, blocking and staining, the arrays were scanned using an Illumina BeadArray Reader according to the manufacturers instructions. Using Genome Studio software the microarray images were inspected for artifacts and QC parameters were assessed. No arrays were excluded at this stage.
Statistical Methods
Microarray analysis
Mean raw intensity values for each probe were corrected for local background intensities and a robust spline normalisation ADDIN EN.CITE Schmid20109[1]9917Schmid, R.Baum, P.Ittrich, C.Fundel-Clemens, K.Huber, W.Brors, B.Eils, R.Weith, A.Mennerich, D.Quast, K.Boehringer Ingelheim Pharma GmbH & Co, KG, Birkendorfer Str, 65, 88397 Biberach/Riss, Germany.Comparison of normalization methods for Illumina BeadChip HumanHT-12 v3BMC GenomicsBMC Genomics349112010/06/08Analysis of VarianceCell LineGene Expression ProfilingHumansOligonucleotide Array Sequence Analysis/*statistics & numerical dataRNA, Small Interfering/geneticsReverse Transcriptase Polymerase Chain ReactionTaq Polymerase/metabolismTime FactorsTransforming Growth Factor beta/pharmacology20101471-2164 (Electronic)
1471-2164 (Linking)20525181http://www.ncbi.nlm.nih.gov/pubmed/2052518130916251471-2164-11-349 [pii]
10.1186/1471-2164-11-349eng[ HYPERLINK \l "_ENREF_1" \o "Schmid, 2010 #9" 1] (combining quantile normalisation and spline interpolation) was applied to each array. Expression values were transformed to a logarithmic scale (base 2), and for each transcript. Differential expression between patient groups was identified by fitting a linear model to each transcript using LIMMA ADDIN EN.CITE Smyth200412[2]121217Smyth, G. K.Walter and Eliza Hall Institute. smyth@wehi.edu.auLinear models and empirical bayes methods for assessing differential expression in microarray experimentsStat Appl Genet Mol BiolStat Appl Genet Mol BiolArticle 33(1)2006/05/0220041544-6115 (Electronic)
1544-6115 (Linking)16646809http://www.ncbi.nlm.nih.gov/pubmed/1664680910.2202/1544-6115.1027eng[ HYPERLINK \l "_ENREF_2" \o "Smyth, 2004 #12" 2]. P-values were adjusted using the method of Benjamini and Hochberg ADDIN EN.CITE Benjamini199513[3]131317Benjamini, Y.Hochberg, Y.Benjamini, Y
Tel Aviv Univ,Sackler Fac Exact Sci,Sch Math Sci,Dept Stat,Il-69978 Tel Aviv,Israel
Tel Aviv Univ,Sackler Fac Exact Sci,Sch Math Sci,Dept Stat,Il-69978 Tel Aviv,IsraelControlling the False Discovery Rate - a Practical and Powerful Approach to Multiple TestingJournal of the Royal Statistical Society Series B-MethodologicalJ Roy Stat Soc B MetJournal of the Royal Statistical Society Series B-MethodologicalJ Roy Stat Soc B MetJournal of the Royal Statistical Society Series B-MethodologicalJ Roy Stat Soc B Met289-300571bonferroni-type proceduresfamilywise error ratemultiple-comparison proceduresp-valuesbonferroni procedure19950035-9246ISI:A1995QE45300017<Go to ISI>://A1995QE45300017English[ HYPERLINK \l "_ENREF_3" \o "Benjamini, 1995 #13" 3]. Transcripts with |log2 FC| >0.5 were taken forward to variable selection with elastic net ADDIN EN.CITE Zou200514[4]141417Zou, H.Hastie, T.Hastie, T
Stanford Univ, Dept Stat, Stanford, CA 94305 USA
Stanford Univ, Dept Stat, Stanford, CA 94305 USA
Stanford Univ, Dept Stat, Stanford, CA 94305 USARegularization and variable selection via the elastic netJournal of the Royal Statistical Society Series B-Statistical MethodologyJ Roy Stat Soc BJournal of the Royal Statistical Society Series B-Statistical MethodologyJ Roy Stat Soc BJournal of the Royal Statistical Society Series B-Statistical MethodologyJ Roy Stat Soc B301-32067grouping effectlars algorithmlassopenalizationp >> n problemvariable selectiongene-expressionmicroarray dataregressioncancerclassificationshrinkagelasso20051369-7412ISI:000227498200007<Go to ISI>://000227498200007English[ HYPERLINK \l "_ENREF_4" \o "Zou, 2005 #14" 4]. This threshold was chosen in order to e n s u r e t h a t d i f f e r e n t i a l e x p r e s s i o n f o r s e l e c t e d v a r i a b l e s c o u l d b e d i s t i n g u i s h e d u s i n g t h e r e s o l u t i o n o f q t P C R . T h e a n d p a r a m e t e r s o f e l a s t i c n e t , w h i c h c o n t r o l t h e s i z e o f t h e s e l e c t e d m o d e l , w e r e o p t i m i z e d v i a t e n - f o l d c r o s s - v a l i d a t i o n ( C V ) . T h e w e i ghts assigned by elastic net to the trained model were used within a linear regression model to classify samples in the test set.
Disease risk score
For each individual, we calculated the disease risk score using the minimal transcript selected sets for TB versus LTBI and TB versus OD. The score is based on subtracting the summed intensities of the down-regulated transcripts from the summed intensities of the up-regulated transcripts. The risk score was calculated on normalised intensities. The disease risk score for individual i is:
where: n the number of up-regulated number of transcripts in the signature in disease of interest compared to comparator group(s).
m the number of down-regulated number of transcripts in the signature in disease of interest compared to comparator group(s).
The threshold for the classification was calculated as the weighted average of risk score within each class (group of patients), with weights given as inverse of the standard deviation of the score within each class. The thre s h o l d f o r t h e c l a s s i f i c a t i o n b e t w e e n g r o u p u a n d v i s s h o w n b e l o w :
w h e r e : a v e r a g e o f t h e d i s e a s e r i s k s c o r e i n t h e g r o u p .
s t a n d a r d d e v i a t i o n o f t h e d i s e a s e r i s k s c o r e i n t h e g r o u p .
W e f u r t h e r e x p l o r e d t h e e f f e c t o f a d j u s t i n g t h e t h r e s h o l d f o r t h e disease risk score in assigning individual patients to TB or non-TB groups. By accepting a percentage of patients as not classified by our test and thus requiring further investigation, the majority of patients under investigation are accurately assigned, suggesting that the score might be used as an initial screening rule in/rule out approach to TB diagnosis. To calculate the indeterminate zone, we calculated the lower and upper threshold which were calculated as the weighted average with weights given by QUOTE , QUOTE respectively:
When w=1 the formula is equivalent to the main threshold formula.
Further evaluation of how a test based on our signatures might be used operationally and assessment of its cost effectiveness is needed if our TB versus LTBI and TB versus OD signatures are to be translated into simple cheap assays.
Evaluation of the classification of the disease risk score (DRS) and the signatures
To evaluate the performance of the DRS as a classifier we used different measures (AUC, sensitivity, specificity, PPV, NPV, and likelihood ratios).
The calculation of the confidence intervals for the area under a receiver operating characteristic curve (AUC), the sensitivity and the specificity was based on a non-parametric stratified bootstrap resampling (each replicate contained the same number of cases and controls as the original sample) ADDIN EN.CITE Robin2011225[5]22522517Robin, X.Turck, N.Hainard, A.Tiberti, N.Lisacek, F.Sanchez, J. C.Muller, M.Biomedical Proteomics Research Group, Department of Structural Biology and Bioinformatics, Medical University Centre, Geneva, Switzerland. Xavier.Robin@unige.chpROC: an open-source package for R and S+ to analyze and compare ROC curvesBMC BioinformaticsBMC bioinformaticsBMC BioinformaticsBMC Bioinformatics7712Biological Markers/analysisComputational Biology/*methodsConfidence Intervals*Data Interpretation, StatisticalHumansProgramming Languages*ROC Curve*Software20111471-2105 (Electronic)
1471-2105 (Linking)21414208http://www.ncbi.nlm.nih.gov/pubmed/21414208306897510.1186/1471-2105-12-77[ HYPERLINK \l "_ENREF_5" \o "Robin, 2011 #225" 5], with 2000 bootstraps, as recommended by Carpenter et al. ADDIN EN.CITE Carpenter2000263[6]26326317Carpenter, J.Bithell, J.Medical Statistics Unit, London School of Hygiene and Tropical Medicine, Keppel Street, London WC1E 7HT, U.K.Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticiansStat MedStatistics in medicineStat MedStatistics in medicineStat MedStatistics in medicine1141-64199*AlgorithmsBirth Weight*Computer Simulation*Confidence IntervalsDrug Therapy/statistics & numerical dataHumansInfantInfant, Newborn/growth & developmentLeukemia, Myeloid, Acute/drug therapy*Models, BiologicalProportional Hazards ModelsRandomized Controlled Trials as TopicRecurrence/prevention & controlRemission Induction2000May 150277-6715 (Print)
0277-6715 (Linking)10797513http://www.ncbi.nlm.nih.gov/pubmed/10797513[ HYPERLINK \l "_ENREF_6" \o "Carpenter, 2000 #263" 6]. We also employed the exact binomial ADDIN EN.CITE Clopper193471[7]717117Clopper, C.J.Pearson, E.S.The use of confidence or fiducial limits illustrated in the case of the binomial.BiometrikaBiometrika404-413261934[ HYPERLINK \l "_ENREF_7" \o "Clopper, 1934 #71" 7] to calculate the confidence intervals (Table S6).
We used the estimated sensitivity and specificity to calculate the positive and negative predictive values (PPV and NPV) using the following formulas:
and interpreting the prevalence as the probability before the test is carried out that the subject has the disease as suggested by D. Altman ADDIN EN.CITE Altman1994534[8]53453417Altman, D. G.Bland, J. M.Medical Statistics Laboratory, Imperial Cancer Research Fund, London.Diagnostic tests 2: Predictive valuesBMJBmjBMJBMJ1023096947Diagnostic Tests, Routine/*standardsHumansPredictive Value of Tests1994Jul 90959-8138 (Print)
0959-535X (Linking)8038641http://www.ncbi.nlm.nih.gov/pubmed/80386412540558[ HYPERLINK \l "_ENREF_8" \o "Altman, 1994 #534" 8]. In this case, we assumed a clinical setting, such as the one used to recruit samples in Malawi, in which approximately 58% of patients with suspected TB had culture confirmed TB (254 TB confirmed cases / 437 patients with suspected TB), as well as calculating more conservative values assuming a prevalence of 20% (as a more typical proportion would be 15%-25% in quality controlled laboratories in primary care settings in high-burden countries in sub-Saharan Africa). PPV and NPV can be interpreted as the probability that a sample with a positive test has active TB, and the probability that a sample with a negative test result does not have active TB respectively, and as such represent the diagnostic value of a test (Table S7). We also report positive and negative likelihood ratios along with their confidence intervals employing the method described in ADDIN EN.CITE Simel1991301[9]30130117Simel, D. L.Samsa, G. P.Matchar, D. B.Center for Health Services Research in Primary Care, Durham Veterans Administration Medical Center, North Carolina 27705.Likelihood ratios with confidence: sample size estimation for diagnostic test studiesJ Clin EpidemiolJournal of clinical epidemiologyJ Clin EpidemiolJournal of clinical epidemiologyJ Clin EpidemiolJournal of clinical epidemiology763-70448*EpidemiologyHumansProbabilitySampling Studies*Sensitivity and Specificity19910895-4356 (Print)
0895-4356 (Linking)1941027http://www.ncbi.nlm.nih.gov/pubmed/1941027[ HYPERLINK \l "_ENREF_9" \o "Simel, 1991 #301" 9] (Table 3).
Smaller sets of transcripts
Although the models suggested by elastic net were the smallest ones to provide us with the best classification, we wanted to further explore the performance of even smaller lists of transcripts. Instead of optimizing via ten-fold cross-validation (CV) both t h e a n d p a r a m e t e r s o f e l a s t i c n e t w h i c h c o n t r o l t h e s i z e o f t h e s e l e c t e d m o d e l , w e u s e d = 1 w h i c h i s t h e p e n a l t y f o r l a s s o t h a t g i v e s s m a l l e r m o d e l s . T h e n , w i t h i n t h e c r o s s - v a l i d a t i o n s t e p o f c h o o s i n g , w e f o r c e d t h e p e n a l t y t o b e s u c h t h a t t h e e r r o r w ould remain within one standard deviation of minimum error. This process resulted in 21 transcripts for the TB versus LTBI comparison (12 overlapping with the 27 transcript signature) and 29 transcripts for the TB versus OD comparison (14 overlapping with the 44 transcript signature). Smaller models have reduced sensitivity (6% -10% lower than the original models) while specificity remained the same (Table S8). When the DRS was calculated, sensitivity and specificity were 89%, 95% CI (78-97) and 89%, 95% CI (79-97) respectively for the TB versus LTBI comparison. As for the TB versus OD comparison, when the DRS was calculated, sensitivity and specificity were 83%, 95% CI (69-93) and 88%, 95% CI (75-97) respectively. Smaller models have mainly reduced sensitivity.
Smear-negatives
We have included 31 smear-negative patients with TB (with definite negative smear status) in the analysis of the adult cohort (7 TB HIV-uninfected and 24 TB HIV-infected). The TB/LTBI and the TB/OD DRSs were applied to these patients and as controls we used the LTBI and OD patients from the test set, while maintaining the same threshold. The performance of the TB/LTBI signature was comparable to the performance in the HIV-infected group and the performance of the TB/OD signature was almost the same as in the larger smear-negative and smear-positive group. Confidence intervals for the sensitivity and specificity of smear-negative patients with TB were calculated using both the bootstrapping and the exact binomial method (Table S9). These confidence intervals overlapped the corresponding CIs for the larger smear-positive and smear-negative group.
Analysis of validation datasets
For validation of the performance of the disease risk score based on the TB/LTBI 27 transcript signature and TB/OD 44 transcript signature, we used the whole blood expression dataset of Berry et al. ADDIN EN.CITE ADDIN EN.CITE.DATA [ HYPERLINK \l "_ENREF_10" \o "Berry, 2010 #28" 10] generated using Illumina HT12 V3 Beadarrays comparing TB with LTBI and other infections in an Africa cohort (accession series GSE19491). The cohort comprised HIV-uninfected individuals; no TB HIV-infected adult cohort was publically available. For each testing dataset (SA GSE19442, OD GSE22098), both quantile and robust spline normalisation were applied separately to the arrays and the data was log transformed however the results were the same regardless of normalisation method.
For the evaluation of the performance of our TB/LTBI 27 transcript signature, we used TB and LTBI patients in the SA testing set (TB n=20, LTBI n=31). The probe ILMN_3247506 (FCGR1C) was not on the HT12 V3 beadarray. For the evaluation of the performance of our 44 TB/OD transcript signature, we used TB patients from the SA testing set (TB n=20) and OD patients that did not include patients with systemic lupus erythematosus as they were judged to be a rare disease in an African setting (n=82). The probes ILMN_3287952 (LOC100133800), ILMN_3215715 (LOC389386) and ILMN_3308961 (MIR1974) were not on the HT12 V3 Beadchip.
For testing the performance of the reported 393 TB versus LTBI signature and the 86 TB versus OD signature on our African dataset, the disease risk score was calculated with these signatures as previously described, although 7 probes in the reported signatures were not present on the HT-12 V4 Beadchip (TB vs. LTBI 6 probes, TB vs. OD 1 probe).
In order to compare directly the differences of the performance of our signatures to the signatures presented in the Berry et al. ADDIN EN.CITE ADDIN EN.CITE.DATA [ HYPERLINK \l "_ENREF_10" \o "Berry, 2010 #28" 10], we calculated the differences of the means of the measures of classification (namely the AUC, the sensitivity and the specificity) on our test set along with their 95% confidence intervals, using the following mathematical formulas:
References
ADDIN EN.REFLIST 1. Schmid R, Baum P, Ittrich C, Fundel-Clemens K, Huber W, et al. (2010) Comparison of normalization methods for Illumina BeadChip HumanHT-12 v3. BMC Genomics 11: 349.
2. Smyth GK (2004) Linear models and empirical bayes methods for assessing differential expression in microarray experiments. Stat Appl Genet Mol Biol 3(1): Article 3.
3. Benjamini Y, Hochberg Y (1995) Controlling the False Discovery Rate - a Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B-Methodological 57: 289-300.
4. Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B-Statistical Methodology 67: 301-320.
5. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, et al. (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12: 77.
6. Carpenter J, Bithell J (2000) Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med 19: 1141-1164.
7. Clopper CJ, Pearson ES (1934) The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26: 404-413.
8. Altman DG, Bland JM (1994) Diagnostic tests 2: Predictive values. BMJ 309: 102.
9. Simel DL, Samsa GP, Matchar DB (1991) Likelihood ratios with confidence: sample size estimation for diagnostic test studies. J Clin Epidemiol 44: 763-770.
10. Berry MP, Graham CM, McNab FW, Xu Z, Bloch SA, et al. (2010) An interferon-inducible neutrophil-driven blood transcriptional signature in human tuberculosis. Nature 466: 973-977.
ILULU Consortium members
Institute of Infectious Diseases and Molecular Medicine, University of Cape Town Nonzwakazi Bangani, Lizl Bashe, Melina Carr, Hannah P. Gideon, Rene Goliath, Yekiwe Hlombe, Vanessa January, Bekekile Kwaza, Suzaan Marais, Marc Mendelson, Tolu Oni, Fadheela Patel, Ronnett Seldon, Relebohile Tsekela, Katalin A. Wilkinson, Robert J. Wilkinson, Kathryn Wood; London School of Hygiene and Tropical Medicine/Karonga Prevention Study Lyn Ambrose, Amelia C. Crampin, Hazel M. Dockrell, Neil French, Lumbani Munthali, Bagrey Ngwira, Amos Phiri, Femia Zgambo; A n
) ,
սzbbzJ9 hO hO CJ OJ QJ ^J aJ /hO hO 5CJ OJ PJ QJ ^J aJ mH sH /hO hO CJ H*OJ PJ QJ ^J aJ mH sH &hZ CJ OJ PJ QJ ^J aJ mH sH ,hO hO CJ OJ PJ QJ ^J aJ mH sH /hO hO 6CJ OJ PJ QJ ^J aJ mH sH /hO h 5CJ OJ PJ QJ ^J aJ mH sH )h 5CJ OJ PJ QJ ^J aJ mH sH )hN 5CJ OJ PJ QJ ^J aJ mH sH A , , . . . / 0 0 1 z1 |1 $d `a$gd $d a$gdy $d a$gd}_ d @&