Performance of Multiplex Commercial Kits to Quantify Cytokine and Chemokine Responses in Culture Supernatants from Plasmodium falciparum Stimulations

Background Cytokines and chemokines are relevant biomarkers of pathology and immunity to infectious diseases such as malaria. Several commercially available kits based on quantitative suspension array technologies allow the profiling of multiple cytokines and chemokines in small volumes of sample. However, kits are being continuously improved and information on their performance is lacking. Methodology/Principal Findings Different cytokine/chemokine kits, two flow cytometry-based (eBioscience® FlowCytomix™ and BD™ Cytometric Bead Array Human Enhanced Sensitivity) and four Luminex®-based (Invitrogen™ Human Cytokine 25-Plex Panel, Invitrogen™ Human Cytokine Magnetic 30-Plex Panel, Bio-Rad® Bio-Plex Pro™ Human Cytokine Plex Assay and Millipore™ MILLIPLEX® MAP Plex Kit) were compared. Samples tested were supernatants of peripheral blood mononuclear cells of malaria-exposed children stimulated with Plasmodium falciparum parasite lysates. Number of responses in range that could be detected was determined and reproducibility of duplicates was evaluated by the Bland-Altman test. Luminex® kits performed better than flow cytometry kits in number of responses in range and reproducibility. Luminex® kits were more reproducible when magnetic beads were used. However, within each methodology overall performance depended on the analyte tested in each kit. Within the Luminex® kits, the Invitrogen™ with polystyrene beads had the poorer performance, whereas Invitrogen™ with magnetic beads had the higher percentage of cytokines/chemokines with both readings in range (40%), followed by Bio-Rad® with magnetic beads (35%). Regarding reproducibility, the Millipore™ kit had the highest percentage (60%) of cytokines/chemokines with acceptable limits of agreement (<30%), followed by the Invitrogen™ with magnetic beads (40%) that had tighter limits of agreement. Conclusions/Significance Currently available kits for cytokine and chemokine quantification differ in reproducibility and concentration range of accurate detection. Luminex®-based kits with magnetic beads perform the best. Data highlights the importance of testing different kits before each study to choose the most appropriate, depending on the priority of the cytokines assessed.


Introduction
Complex infectious diseases such as malaria induce intricate immune responses that may result in pathogenesis or protection.It is unlikely that single immune mediators on their own correlate with disease risk or protective immunity.Instead, multiple cytokines and chemokines are involved in pathological and immunological processes [1], reflecting the different capacities and functions of immune cells such as immunoregulation, proliferation, activation or cytotoxicity.Immune assays such as ELIspot, intracellular cytokine staining and flow cytometry or ELISA are limited in the number of variables able to measure at the same time in a single assay, whereas quantitative suspension array technology allows the simultaneous measurement of different cytokines/chemokines.This technology is based on a capturedetection sandwich type assay using fluorescent microspheres, and only requires small volumes of samples.The possibility of measuring multiple variables is of importance for malaria and other major infectious diseases in which no immune correlates of protection have been identified [2,3].Quantitative suspension array technology allows a more in depth characterization of antigen specific responses, increasing the chances to detect cytokine/chemokine responses associated with protective immunity.
The introduction of fluorescent bead-based technology represents a significant advancement for cytokine/chemokine profiling.The assessment of multiple analytes permits to discover patterns and relevant correlations and may be more powerful to capture the emergence of complex responses associated to protection or pathology.The requirement of a reduced sample volume and time saving-advantages make it an attractive method to study immune responses and biomarkers in large-scale pediatric field studies.Multiplexing technologies have proved crucial in deciphering patterns and provide insight to pathology and immunity in a number of studies [4][5][6][7][8][9].However, there are different methods (LuminexH-based or flow cytometry-based) and several available commercial kits that vary in absolute cytokine concentration, sensitivity, reproducibility and cost [10][11][12][13][14][15][16].Khan et al. compared four different LuminexH-based kits [14] and found that despite non-comparable absolute concentrations, cytokines may follow similar patterns between different kits.However, only five analytes were compared in this study.It has been recommended to use the same method if comparisons are to be made between different data sets and to validate them for reproducibility and precision to ensure accurate protein identification [17].
To date, most of the studies comparing different multiplex kits have measured cytokines in serum or plasma samples [10][11][12]14,16].In this study, we evaluate cytokine/chemokine concentrations in supernatants of stimulated peripheral blood mononuclear cells (PBMC).We previously compared two LuminexH methods, one from Bio-RadH and another from Invitrogen TM , and two flow cytometry methods, one from Becton Dickinson TM (BD TM ) and one from Bender MedSystemsH that were commercially available in 2007 [15].However, this technology is continuously evolving and to our knowledge data of newest available commercial kits are missing in the literature, particularly regarding the performance of magnetic beads.Therefore, the aim of this study is to compare the performance of six human cytokine/chemokine multiplex kits currently available using culture supernatants from PBMC stimulated with malaria parasite lysates.The kits from the following manufacturers were tested: eBiosciencesH (formerly Bender MedSystemsH), Bio-RadH, Millipore TM , Invitrogen TM and BDH.

Ethics Statement
The study has been approved by the ethics committees of the Hospital Clinic of Barcelona (Spain) and the National Health Bioethics Committee of Mozambique.Written informed consent was obtained from parents or guardians.

Samples
Culture supernatants had been harvested after fresh stimulation of 200,000 PBMC with a P. falciparum schizont lysate (3D7 strain), corresponding to 0.6 million infected erythrocytes, for 48 h and 72 h, as described [15].Briefly, PBMC obtained from children were isolated using a Lymphoprep gradient and resuspended in RPMI 1640 medium with 10% FCS, 100 IU/ml penicillin, 0.1 mg/ml streptomycin (all SigmaH), and 2 mM L-glutamine (GIBCO/Invitrogen TM ).Blood samples had been collected into EDTA microtainers by finger-prick from children ,2 years old living in a malaria endemic area in the context of a study conducted at the Centro de Investigac ¸ao em Sau ´de da Manhic ¸a, southern Mozambique, between 2005 and 2009 (ClinicalTrials.govNCT00231452) [18].Following the incubation period, the supernatants were stored at -80uC in Manhic ¸a and shipped in dry ice to Barcelona for analysis.Before the tests, supernatants were thawed, aliquoted and frozen again.

Cytokine and Chemokine Measurement by Multiplex Bead Array Kits
Initially we evaluated four different kits: (i) the Human Th1/ Th2 11-Plex FlowCytomix TM kit plus 3 extra cytokines from eBioscienceH (formerly Bender MedSystemsH, flow cytometry, non-magnetic beads), (ii) the Human Cytokine 25-Plex panel from Invitrogen TM (LuminexH, non-magnetic beads), (iii) the Bio-Plex Pro TM Human Cytokine 17-Plex Assay from Bio-RadH (Lumi-nexH, magnetic beads) and (iv) the MILLIPLEXH MAP 13-Plex Kit from Millipore TM (LuminexH, magnetic beads).During the testing, Invitrogen TM made available a new improved multiplex kit with magnetic beads, the Human cytokine Magnetic 30-Plex Panel (LuminexH) that was included in a second round of tests together with a BD TM Cytometric Bead Array (CBA) Human Enhanced Sensitivity 3-Plex kit (flow cytometry, non-magnetic beads).Therefore, two rounds of tests were performed in this study.In the first round, 37 samples from 48 h stimulations were analyzed in duplicates, whereas in the second round 40 samples from 72 h stimulations were analyzed also in duplicates.The second round included all 37 samples from the first round, but at a different stimulation time, plus 3 additional new samples.Nevertheless, only 20 of these samples could be analyzed with the BD TM CBA kit.In addition, a negative control of only media was added in each plate.The cytokines/chemokines tested in each kit are detailed in Table 1 and Table 2.The Bio-Plex assay kit from Bio-RadH, the Invitrogen TM Human cytokine Magnetic 30-plex panel, and the BD TM CBA assays were carried out in the presence of a representative from the manufacturer.All methods were performed by the same operator according to manufacturers' instructions.All kits supplied lyophilized standards that were reconstituted and diluted at 7 serial concentrations following manufacturer's instructions (standard curves).Standards included all recombinant cytokines tested and were considered as positive controls for the procedure.However, for the Bio-RadH, Invitrogen TM and Millipore TM kits that recommend 50 ml, samples were diluted 1:2 in culture media, to be able to compare the performance of the kits using minimum equivalent volumes (25 ml).Bead fluorescence readings were done by LuminexH (Bio-PlexH 100) or by a flow cytometry (BD TM FACSCanto II) apparatus.

Statistical Analysis
Data for each kit was analyzed as recommended by the manufacturers.Concentration of each analyte was obtained by interpolating fluorescence intensity to at least 7-point dilution standard curve supplied by the kit and calculated by the Bio-PlexH software (Bio-PlexH Manager version 4.0), by the FlowCytomix TM Pro 2.2.1 software (eBiosciencesH) or the BD FCAP Array TM Software, depending on the kit.The proportion of samples that had both readings within the accepted recovery range (between 70-130%) was determined.The reproducibility of the different multiplexed methods was evaluated by describing the limits of agreement between duplicates in range for each combination of method and analyte using the Bland-Altman test.Mean difference dot plots with the limits of agreement were constructed [19].Comparison of variability and bias between the two readings was evaluated using the MethComp package [20] in R statistical software (version 2.12.2) [21].P-values lower than 0.05 were considered significant.

Results and Discussion
In this study we compared six different kits (Table 1) using supernatants of antigen-stimulated PBMC from children to measure cytokine and chemokine responses specific to P. falciparum malaria.Quantitative suspension array technology allowing the multiplex detection of up to 39 cytokines (depending on kit) is a unique tool to study immune responses in small sample volumes.It is particularly relevant in malaria research since no immune correlates of protection have yet been identified.The quantitative microsphere-based suspension array system provides a more comprehensive representation of the immune responses than assays assessing individual biomarkers and it is useful to address the complexity of immunity to infectious diseases.
We evaluated the performance of each kit by calculating the number of samples that had both readings in the accurate range of detection (Table 2) and the reproducibility (Table 3) of each analyte tested for each kit.Differences of less than 30% in the limits of agreement were considered acceptable.Tables 1 and 2 show the number of analytes tested as well as the specific cytokines/chemokines analyzed, respectively.There are no gold standards available to evaluate the accuracy and precision of the absolute concentrations measured by the kits.In each kit the cytokines were chosen by us based on the analytes available, the total number of analytes included by the manufacturers and the cytokine priorities based on the malaria scientific literature.As an example of the Bland-Altman plots generated for each cytokine in each tested kit, the graphs for IL-10 and IL-2 are represented in Figure 1, whereas the rest are shown in Supporting Information (Figure S1, FS2, FS3, FS4, FS5, FS6, FS7, FS8, FS9, FS10, FS11, FS12, FS13, FS14, FS15, FS16, FS17, FS18, FS19, FS20, FS21, FS22, FS23, FS24, FS25, FS26, FS27, FS28, FS29, S30).
In general we found that the proportions of both readings in range were very low (Table 2) as in most analytes less than 75% of both duplicates were in range independently of the kit.This was mainly due to low cytokine and chemokine concentrations in supernatant samples that were below the limits of detection (e.g.IL-2), although in some cases the analyte concentration was over the accurate range (e.g.IL-8 and IL-6 for Invitrogen TM ).This was expected as in our previous malaria studies antigen-specific responses in children were rather low [15,22].
The FlowCytomix TM kit had low percentages of both readings in range with the exception of IL-6 and IL-8 that had .75% of readings in range (Table 2), although the reproducibility among duplicates was poorer and none of the cytokines/chemokines had acceptable limits of agreement (Table 3).The Bio-RadH kit had a higher proportion of both readings in range for overall cytokines/ chemokines being .75% for 35% of the cytokines/chemokines tested (G-CSF, GM-CSF, IL-10, IL-1b, IL-4 and IL-7), although only 2 (GM-CSF and IL-8) out of 17 cytokines/chemokines (11.8%) had acceptable limits of agreement.The Invitrogen TM kit with polystyrene beads had low percentages of both readings in range (only RANTES had .75%)and low reproducibility (none of the cytokines/chemokines fulfilled the criteria of ,30%).In contrast, the Invitrogen TM kit with magnetic beads (INV-MAG) detected many more cytokines in range (40% of cytokines/ chemokines had .75% of both readings in range: EGF, HGF, IFN-a, IL-12, IL-13, IL-1Ra, IL-2R, IL-6, IP-10, MIP-1a, RANTES and VEGF), although some other cytokines/chemokines of interest did not perform as well (e.g.IFN-c).The reproducibility of INV-MAG was greater than the other kits (40% of cytokines/chemokines had acceptable and tighter limits of agreement), but data could not be compared directly as the tests were done at different incubation time points.The Millipore TM kit did not show a high proportion of both readings in range with the exception of IL-1b and TNF that had .75%(15.38% of cytokines/chemokines tested), and overall showed a good reproducibility (62% of cytokines/chemokines tested fulfilled our criteria).The Enhanced Sensitivity CBA did not demonstrate to have more sensitivity than the INV-MAG kit and performed poorly.
Overall, we found that each kit performed differently in number of samples with duplicate readings in range and reproducibility depending on the cytokine/chemokine tested.In our hands, the performance of LuminexH methods was better than the flow cytometry methods in terms of proportion of samples in range and reproducibility, as previously observed [10].These data differ from our previous observations that flow cytometry methods were more reproducible [15], but the LuminexH kits used in both studies were different.Within the LuminexH kits, the Invitrogen TM kit with non-magnetic beads had the poorer performance, whereas the Invitrogen TM kit with magnetic beads had the higher percentage of cytokines/chemokines with both readings in range, followed by the Bio-RadH kit with magnetic beads.Regarding reproducibility, the Millipore TM kit had the highest percentage (60%) of cytokines/ chemokines with acceptable limits of agreement, followed by INV-MAG TM kit (40%) that had tighter limits of agreement.However, we could only test 13 cytokines/chemokines with the Millipore TM kit, in contrast to the 30 cytokines/chemokines assessed with the INV-MAG.In general, kits with magnetic beads performed better than kits with polystyrene beads.Magnetic beads may decrease bead loss during washes and avoid plate clogging and leaks, which occasionally happen when performing non-magnetic bead assays.In addition, magnetic beads reduce non-specific binding, increasing reproducibility and robustness of the assay.Several reasons may account for the variation in the performance of multiplex kits from different manufacturers as well as the heterogeneity in analytes within each kit.Probably the most important factor is the different antibody used for capture and detection of individual analytes by each manufacturer.This has also been described in a study comparing ELISA kits [23] in which it was suggested that the nature of the pairs of monoclonal antibodies employed in each ELISA kit did not permit comparable recognition of cytokines in samples.In addition, there may be differences among purified recombinant proteins used to generate the standard curves as well as in the assay buffers supplied by the manufacturers.
In addition to the individual cytokine performance in each kit, other factors may be important in the final selection of a multiplex cytokine profiling method.In Table 1 we compare some of these factors such as the number of cytokines measured within each kit, the volume of sample needed or the time and cost of the assays.Overall, LuminexH-based kits are the fastest, considering the incubation and reading time.Invitrogen TM was the most cost effective at the place and time the study was performed.Unfortunately, other commercially available kits such as ProcartaH

Conclusions
Differences in the number of samples detected in accurate range and reproducibility were observed depending on the method used and even the cytokine detected, although LuminexH-based kits with magnetic beads proved to be better.This suggests that the same kit should be used throughout a given study.When selecting the most suitable multiplex cytokine profiling method, a prioritization of analytes of interest should be done and other factors such as the number of cytokines measured with each kit, the time and cost of the assays may be also considered as they vary widely.Although it would be advisable to test different kits before starting any study and qualify or validate them, our data can help the decision making process to select the most appropriate multiplex suspension array kit in similar studies.

Figure 1 .
Figure 1.Mean difference dot plots of IL-10 and IL-2 cytokines for each kit tested.A) IL-10 and B) IL-2 disagreement plots show the difference between the duplicates against the geometric mean of both values of a sample tested with eBioscienceH FlowCytomix TM (FlowCytomix), Millipore TM MILLIPLEXH MAP Plex Kit (Millipore), Bio-RadH Bio-Plex Pro TM Human Cytokine Plex Assay (Bio-Rad), Invitrogen TM Human Cytokine Magnetic 30-Plex Panel (INV-MAG) and BD TM Cytometric Bead Array Human Enhanced Sensitivity kit (BD CBA).The middle line is the mean difference and the two extreme lines are the limits of agreement (values are truncated to the first decimal) calculated by Bland-Altman test.doi:10.1371/journal.pone.0052587.g001

Figure
Figure S1 Mean difference dot plot of EGF.Disagreement plot shows the difference between the duplicates against the geometric mean of both values of a sample tested with Invitrogen TM Human Cytokine Magnetic 30-Plex Panel (INV-MAG).The middle line is the mean difference and the two extreme lines are the limits of agreement calculated by Bland-Altman test.(PDF) Figure S2 Mean difference dot plot of FGF-Basic.Disagreement plot shows the difference between the duplicates against the geometric mean of both values of a sample tested with Invitrogen TM Human Cytokine Magnetic 30-Plex Panel (INV-MAG).The middle line is the mean difference and the two extreme lines are the limits of agreement calculated by Bland-Altman test.(PDF) Figure S3 Mean difference dot plots of G-CSF for each kit tested.Disagreement plots show the difference between the duplicates against the geometric mean of both values of a sample tested with A) Bio-RadH Bio-Plex Pro TM Human Cytokine Plex Assay (Bio-Rad), and B) Invitrogen TM Human Cytokine Magnetic 30-Plex Panel (INV-MAG).The middle line is the mean difference and the two extreme lines are the limits of agreement calculated by Bland-Altman test.(PDF) Figure S4 Mean difference dot plots of GM-CSF for each kit tested.Disagreement plots show the difference between the duplicates against the geometric mean of both values of a sample tested with A) Bio-RadH Bio-Plex Pro TM Human Cytokine Plex Assay (Bio-Rad), B) Human Cytokine 25-Plex panel from Invitrogen TM (non-magnetic beads) and C) Invitrogen TM Human Cytokine Magnetic 30-Plex Panel (INV-MAG).The middle line is the mean difference and the two extreme lines are the limits of agreement calculated by Bland-Altman test.(PDF) Figure S5 Mean difference dot plot of HGF.Disagreement plot shows the difference between the duplicates against the geometric mean of both values of a sample tested with Invitrogen TM Human Cytokine Magnetic 30-Plex Panel (INV-MAG).The middle line is the mean difference and the two

Table 2 .
Proportion of duplicate readings in range.

Table 3 .
Limits of agreement for each cytokine/chemokine tested.