^{1}

^{*}

^{1}

^{1}

^{1}

^{1}

^{2}

^{2}

^{3}

^{1}

The authors have declared that no competing interests exist.

Conceived and designed the experiments: KM EL MM CAL. Performed the experiments: KM BD GT AD SEM. Analyzed the data: KM EL MM MPF CAL. Wrote the paper: KM EL MM MPF CAL.

Vaccines that interrupt malaria transmission are of increasing interest and a robust functional assay to measure this activity would promote their development by providing a biologically relevant means of evaluating potential vaccine candidates. Therefore, we aimed to qualify the standard membrane-feeding assay (SMFA). The assay measures the transmission-blocking activity of antibodies by feeding cultured

Continuous efforts to reduce malaria burden, such as application of insecticide treated nets, anti-malarial drugs and indoor insecticide spraying, have contributed to a decrease in mortality due to malaria, particularly due to

The standard membrane-feeding assay (SMFA) has been utilized widely to assess the transmission-blocking potential of test antibodies both in preclinical and clinical vaccine development (transmission-blocking refers to reduction in oocyst intensity throughout this manuscript unless specified, while further studies are required to determine the relationship to the prevalence in mosquitoes). In this assay, a mixture of cultured

A robust assay to measure biological activity is essential for vaccine development

Of the seven characteristics listed in the Q2(R1) guidelines for assay validation, we decided to qualify the SMFA with respect to four characteristics. The first one is Precision, focusing specifically on Repeatability and Intermediate Precision. In the case of SMFA, Repeatability was determined by evaluating intra-feed variability, and Intermediate Precision by inter-feed variability. The second characteristic is Linearity: in the context of SMFA, this was determined by evaluating whether (a transformation of) the % inhibition result is directly proportional to (a transformation of) the concentration of transmission-blocking antibody. We also evaluated Range of the SMFA: i.e., the interval between the upper and lower levels of transmission-blocking activity in which the analytical procedure has a suitable level of Precision and Linearity. The fourth characteristic is Specificity: i.e., whether we can detect transmission-blocking activity of test antibody in the presence of unrelated antibodies which may be expected to be present in a test sample.

The ultimate goal is to establish a robust SMFA which can provide a biologically relevant means for making an informed Go/No-go decision (especially SMFA with human antibodies before performing large Phase 2 or 3 studies). While basic methodologies are similar, there are several differences in different laboratories (e.g., different haematocrits, different mosquitoes, etc.), and the impact of such differences on the final readout (i.e., % inhibition) is not clear. As an initial attempt, in this study we decided to evaluate the four characteristics listed above when the assay was performed by our current method. More specifically we focused only on the feeding portion of the assay. We generated a quantity of mouse 4B7 monoclonal antibody (mAb) to perform the qualification, since 4B7 mAb is directed to the Pfs25 antigen and has been well-characterized for its transmission-blocking activity

In the pre-qualification experiments, serial dilutions of 4B7 mAb (ranging from 1 to 375 µg/ml) were tested over 6 independent feeding experiments (Feed 1–6) to determine suitable concentrations to use in the qualification experiments. In this series of experiments, each concentration of 4B7 mAb was tested in a single COM (container of mosquitoes, viz., a group of mosquitoes which were housed in the same container and were fed the same final mixture) in each feed. As shown in a previous study with 4B7 mAb

Various concentrations of 4B7 mAbs (ranging from 1 to 375 µg/ml) were tested over 6 independent feeding experiments (Feed 1–6). Different symbols represent data from different feeding experiments.

We first evaluated two aspects of Precision: Repeatability (intra-feed variation) and Intermediate Precision (inter-feed variation). Since there were three COM for the negative control and three COM of 4B7 mAb at each concentration, 9 different values of PIm can be calculated at each concentration in each feed (

Four concentrations (1, 6, 23 and 94 µg/ml) of 4B7 mAb were tested in triplicate in each feed, and three independent feeds were performed (Feed 7, 8 and 9). Since there were 3 COM of negative control and 3 COM of 4B7 mAb at each concentration, 9 different numbers of PIm were calculated (individual dots) at each concentration in each feed. Bar represents the mean of the 9 calculated data.

4B7 |
Intra-feed variance |
Inter-feed variance |
||||||

Overall | Feed 7 | Feed 8 | Feed 9 | Overall | 7 & 8 | 7 & 9 | 8 & 9 | |

1 | 189.6 | 172.8 | 157.0 | 239.0 | 261.2 | 347.2 | 145.5 | 291.1 |

6 | 164.7 | 77.3 | 197.1 | 219.5 | 150.8 | 103.8 | 129.3 | 219.3 |

23 | 17.1 | 10.7 | 35.4 | 5.1 | 52.1 | 54.2 | 82.2 | 20.0 |

94 | 0.5 | 0.6 | 0.9 | 0.0 | 2.6 | 2.0 | 4.6 | 1.1 |

Concentration of 4B7 mAb in a feeder [µg/ml].

Intra-feed variance estimates the variability between three PIm values (each one using one test COM and one control COM) where the test samples have the same 4B7 concentration and both PIm are measured on the same feed. We use U-statistics to estimate the intra-feed variance for each of the 3 feeds, as well as to estimate the overall estimate that combines the 3 feeds.

Inter-feed variance is similar to the intra-feed variance, except that the variability is between two PIm values from identically concentrated test samples where one value is measured on one feed and the other value is measured on a second feed. Again we use U-statistics. We give the pairwise estimates and an overall estimate of inter-feed variance.

We then assessed Linearity of SMFA (whether a transformation of the test result is directly proportional to a transformation of the concentration of active antibody). When the square root of 4B7 concentration (x-axis) was plotted against the ratios of the mean between control and test on a log-scale (y-axis), the data were approximated by a linear relationship (^{2} = 0.88, slope was significantly different from zero, p<0.0001).

Various concentrations of 4B7 mAb were tested in the qualification experiments (Feed 7–9). For these data the first COM negative control is matched with the first COM of the 4B7 mAb at each concentration, the second with the second, etc. The square root of 4B7 concentration is shown on the x-axis, and the ratio of mean oocyst (mean of oocysts in control divided by mean of oocysts in test) is plotted on a log scale (shown on left side of y-axis, the associated PIm value is shown on the right side of the y-axis). Points with the same symbol use the same control, and points with the same color are from the same feed. Dotted line represents the best-fit line.

While Linearity was demonstrated over the concentrations tested (

Specificity (whether we could detect transmission-blocking activity of test antibody in the presence of unrelated antibody which may be expected to be present in a test sample) was assessed to check whether this assay is useful to test mouse polyclonal antibodies in the future. We decided to test the 4B7 mAb at 23 and 94 µg/ml, with or without normal mouse IgG. The two concentrations of 4B7 were selected, since it is difficult to evaluate the effect of addition of normal mouse IgG if the 4B7 mAb itself shows variable PIm which would be the case at lower concentrations. As part of pre-qualification experiments, we tested normal mouse IgGs at concentrations ranging from 0.2 to 1.5 mg/ml. The normal mouse IgG showed 45% inhibition compared to the negative control (p = 0.019 by a Mann-Whitney test) at 1.5 mg/ml in a feed, while the same IgG showed insignificant inhibitions at the second highest dose tested, i.e., 0.75 mg/ml, in the two independent feeds (−8% inhibition, p = 0.946 in one feed; 10% inhibition, p = 0.217 in another feed). Based on those data, we decided to use a concentration of 0.75 mg/ml of normal mouse IgG to evaluate Specificity in the qualification experiments. When 23 µg/ml of 4B7 was tested in the presence and absence of normal mouse IgG, a mean PIm of the three feeds was 81.3% and 82.4%, respectively (

Two concentrations (23 and 94 µg/ml) of 4B7 mAb were tested with or without 0.75 mg/ml of normal mouse IgG (NMAb). PIm without NMAb were calculated 3 times for each feed (using 3 test COM and 3 separate control COM, 20 mosquitoes in each COM) and 1 time (1 test COM and 1 control COM, 20 mosquitoes each) for PIm with NMAb.

We generated a model using the data from these 9 feeding experiments (pre-qualification and qualification feeds) to estimate the effect of modifications in the assay and/or analytical methods to guide future studies. As shown in

For each COM, mean number of oocysts and standard deviation were calculated. Data from all COM tested in 9 independent feeding experiments are shown. Different symbols represent data from different feeding experiments and the line represents the best-fit curve from the zero-inflated negative binomial model. The R^{2} value for the fit is 0.94. Gray lines represent 95% confidence intervals calculated using the t-distribution (for the means) or chi square distribution (for the standard deviations).

We simulated how much variance could be reduced by dissecting more mosquitoes at each concentration of 4B7 mAb (_{1} and T_{2}) and true PIm of T_{1} was higher than that of T_{2}. In each given condition, the probability of feeds in which T_{1} showed higher PIm (i.e., lower mean oocyst number) than T_{2} was calculated using data from 10,000 simulations (_{1} = 50 or 70% inhibition, and T_{2} = 0, 10, 20, 30, 40 or 50% inhibition were tested. For dissection, three different dissection scenarios were simulated: 1) total of 20 mosquitoes were dissected from a single COM (m = 20); 2) total of 60 from single COM (m = 60), and 3) total of 60, but from three COM (m = 20×3). In addition, we simulated either: 1) T_{1} and T_{2} were tested in the same feeding experiment (SF), or 2) tested in different feeding experiments (DF). For example, if the true PIm of T_{1} = 50% and T_{2} = 30%, 20 mosquitoes were modelled as from a single COM for each sample, and the two samples were tested in the same feeding experiment (m = 20 SF), the probability was calculated as 0.72 (_{1} and T_{2} were tested in the same feeding experiment (SF) than when tested in different feeding experiments (DF). While dissecting more mosquitoes from a single COM (m = 20 vs. m = 60) increased the probability of feeds in which T_{1} showed lower mean oocyst number than T_{2}, the level of increase was less than 0.1. On the other hand, the difference in probability was larger when a condition where a total of 60 mosquitoes were dissected from 3 COM (m = 20×3) as compared to the other condition where 60 mosquitoes were dissected from a single COM (m = 60). We also evaluated the effect of the number of oocysts in the control in this simulation, but changing the mean of oocysts in the control from 4 to 30 had no noticeable effect on the probability (

In this simulation, we assumed there were two test samples (T_{1} and T_{2}), and true PIm of T_{1} (50 or 70% inhibition compared to control) was higher than the true PIm of T_{2} (0, 10, 20, 30, 40 or 50%). Three different dissection conditions were simulated; 1) total of 20 mosquitoes were dissected from a single COM (m = 20), 2) total of 60 from a single COM (m = 60), and 3) total of 60, but from three COM (m = 20×3). In addition, we stimulated either: 1) T_{1} and T_{2} were tested in the same feeding experiment (SF), or 2) tested in different feeding experiments (DF). We assumed the mean number of oocysts in the control was 20. For each test condition, 10,000 data were generated to calculate the probability of feeds in which T_{1} showed higher PIm (i.e., lower mean oocyst number) than that T_{2}.

4B7 |
Condition 1 | Condition 2 | Expected Ratio of variance |
||

Mosq |
COM |
Mosq |
COM |
||

1 | 60 | 1 | 20 | 1 | 0.78 |

6 | 60 | 1 | 20 | 1 | 0.80 |

23 | 60 | 1 | 20 | 1 | 0.76 |

94 | 60 | 1 | 20 | 1 | 0.66 |

1 | 20 | 3 | 60 | 1 | 0.33 |

6 | 20 | 3 | 60 | 1 | 0.32 |

23 | 20 | 3 | 60 | 1 | 0.33 |

94 | 20 | 3 | 60 | 1 | 0.38 |

Concentration of 4B7 mAb in a feeder [µg/ml].

Number of mosquitoes dissected per Container of Mosquitoes (COM).

Number of COM used.

Average of variance (SMFA Condition 1)/variance (SMFA Condition 2) from 100,000 simulations. This many simulations ensures that we have 95% confidence that the estimates of the expected variance ratios are within 0.03 of their true values.

We then simulated 100,000 data sets from the model to estimate whether calculation of PIm using median of oocyst number was better than PIm represented as the arithmetic mean in terms of variance (

4B7 |
Mean |
Median |
Expected ratio of variance |

1 | 2242 | 3039 (16) | 0.74 |

6 | 746 | 1010 (6) | 0.74 |

23 | 134 | 185 (7) | 0.72 |

94 | 4.5 | 6.1 (7) | 0.73 |

Variances were calculated in the condition where 20 mosquitoes from a single COM were dissected.

Concentration of 4B7 mAb in a feeder [µg/ml].

Variance of PIm when calculated using arithmetic mean.

Variance of PIm when calculated using median. The value “n missing” is the number out of 100,000 simulations with median = 0 in the control so that percent inhibition of a test could not be calculated.

Average of variance (SMFA using mean)/variance (SMFA using median) from 100,000 simulations. This many simulations ensures that we have 95% confidence that the estimates of the expected variance ratios are within 0.04 of their true values.

We selected PIm as the main readout of SMFA in this study, however, % inhibition of prevalence (PIp, an increase in the proportion of mosquitoes that have no oocysts) also has been used in many other studies. Therefore, we assessed the relationship between PIm and PIp using the model. The adequacy of the negative binomial model was evaluated by plotting the mean oocyst count with the proportions of mosquitoes with any oocysts. We included the predicted proportion from the model together with a nonparametric smoother of the proportions (

Each point represents one COM. Black line is the fit from the zero-inflated negative binomial model. The blue dotted line is a nonparametric moving window average (specifically, a kernel smoother with a normal kernel with bandwidth 0.5 log_{10} chosen to be slightly overfit).

The % inhibition of prevalence (PIp) is plotted against % inhibition in mean oocyst intensity (PIm) at different mean number of oocysts in the control.

In the present study, we qualified the SMFA using mouse 4B7 mAb and normal mouse IgG, as it is currently performed, in terms of its Precision (more specifically, Repeatability and Intermediate Precision), Linearity, Range, and Specificity. While the word “assay validation” has been used in quite a few publications for many assays, there are limited studies where each individual characteristic of a biological assay is assessed systematically. To the best of our knowledge, this is the first study to qualify multiple characteristics of the SMFA with respect to ICH Q2 (R1). In addition to the qualification of the assay, we generated a model to estimate the impact of modifications in the assay and to evaluate analytical methods for their utility in generating robust data.

According to the ICH Harmonised Tripartite Guideline Q2(R1), up to seven characteristics need to be considered for assay validation depending on the type of assay. The SMFA is one of a few biological assays widely utilized to test functional activity of antibodies both in preclinical and clinical vaccine development. However, the limited number of studies published to date have addressed a single validation parameter, Intermediate Precision (inter-feed variation)

We evaluated Repeatability (intra-feed variability) and Intermediate Precision (inter-feed variability) at four concentrations of 4B7 mAb, which cover a range of % inhibition (from ∼20 to ∼100% inhibition). We have shown that variances in PIm were dependent on the concentration of 4B7 mAb. The data indicate that interpretation of results is difficult if a test sample has weak inhibitory activity, due to relatively large error of the assay, while we could obtain higher % inhibition consistently in any given feed if the test sample has strong activity. For the Linearity, the ICH Q2 (R1) recommends testing a minimum of 5 concentrations. However, since we have a practical limitation on the number of COM that can be comfortably handled in a single feeding experiment, we decided to test 4 different concentrations in order to evaluate Precision and Specificity at the same time. As shown in

The SMFA is a complex assay and many factors are considered to be possible sources of variability in the SMFA, such as the batch of human serum used in the gametocyte culture, temperature control (especially for late stages of gametocytes), and perhaps the size of mosquitoes. In this study we focused only on the feeding portion of the assay, as it is very difficult to evaluate all factors in a single study. We standardized the gametocyte culture as much as possible (e.g., culture volume, haematocrit, starting parasitemia, maintenance of temperature during medium change and feeding experiments) before starting this qualification study. We strictly followed our standard operating procedure to minimize the variation in gametocyte culture, preparation of samples and feeding experiment. In addition, we measured wing size of mosquitoes from four different COM and found that the coefficients of variation (CV) of wing size were 2.3–4.1%, which was much smaller than CV for the oocyst numbers (58.6–106.6%, data not shown). Despite efforts to standardize the assay, as it is well accepted, the mean numbers of oocysts in the control can still be highly variable; the mean in the control groups ranged from 5.6 to 60.7 in the 9 feeding experiments in this study. The qualification undertaken here was performed to improve the understanding of the uncertainty when interpreting data from both a single and multiple feeding experiments.

The SMFA is a labor-intensive assay and it takes about one month from starting a culture to the final oocyst counts. Therefore, generation of empirical data to judge the effect of assay and/or analysis modifications on the final readout requires considerable investment of effort and time. Hence, we tried to answer these questions using a model. In addition to our current study (

Many SMFA studies have used 30 or fewer mosquitoes per sample _{1} has 70% inhibitory activity, and T_{2} has 0%, even 20 mosquitoes from single COM gave more than a 90% probability of seeing a feed in which T_{1} showed higher PIm than that T_{2}. Dissection with more mosquitoes did not dramatically increase this probability. However, showing that a test has larger mean intensity than a control is often not enough for pre-clinical and clinical vaccine development. If we wish to have a high confidence that two test samples have different activities, we will likely need to dissect more than one COM of 20 mosquitoes for the test and control.

We also used the simulation model to determine whether calculating PIm using median number of oocysts is better than using arithmetic mean number of oocysts. There was a strong relationship between the mean and standard deviation of oocyst number in each COM and the relationship fitted the zero-inflated negative binomial model (

In this study, we used % inhibition in mean oocyst intensity (PIm) as the main readout of SMFA. Another readout, % inhibition of prevalence (PIp), an increase in the proportion of mosquitoes that have no oocysts, has also been used in many studies. Therefore, we modeled the relationship between PIm and PIp. The PIp readout is thought to be the best predictor of vaccine efficacy under field conditions, as it has been suggested that a single oocyst can generate a large number of infectious sporozoites

Because antibodies are thought to be the major effectors blocking transmission from human host to mosquito vector, SMFA is one of the few biological assays by which the potential efficacy of transmission-blocking vaccine candidates may be evaluated. The direct membrane-feeding assay (DMFA) is another assay which has been used to measure biological activity of test antibodies. The DMFA utilizes patient blood as a source of gametocytes instead of cultured parasites in the SMFA. Therefore, the SMFA is considered to be a relatively better controlled assay compared to the DMFA. It is still controversial whether the data obtained by SMFA correlates with the data generated in DMFA

4B7 hybridoma cells were obtained from the Malaria Research and Reference Reagent Resource Center (MR4) as part of the Biodefense and Emerging Infections Resources Repository, National Institute of Allergy and Infectious Disease, National Institutes of Health:

Gametocyte culture of _{2}, 5% CO_{2} and 90% N_{2} for 16–18 days with daily medium change. There was no addition of fresh uninfected erythrocytes, except on day 2 when the culture was divided to two cultures and fresh uninfected erythrocytes were added to them (giving a total of four cultures). For each feeding experiment, two or three cultures were selected based on their stage V gametocytemia and exflagellation activities and pooled. The average (standard deviation) of stage V gametocytemia in Feed 1–9 was 2.1 (0.6) %. The culture was centrifuged at 2000 g for 10 minutes, the medium of the mature gametocyte culture was replaced with normal human serum, and normal red blood cells (RBCs) were added to make a gametocyte mixture with 0.15–0.2% stage V gametocytemia at 50% haematocrit (0.5–1.3×10^{5}/µl of stage V gametocytemia in the final feeder). Sixty µl of a test sample (a defined concentration of 4B7 mAb with or without normal mouse IgG in 1×PBS) was mixed with 200 µl of the gametocyte mixture, and the final mixture was immediately fed to ∼50 of 3–6 days old female

Serial dilutions of 4B7 mAb (ranging from 1 to 375 µg/ml) were tested over 6 independent feeding experiments (Feed 1–6). For normal mouse IgG without 4B7 mAb, concentrations ranging from 0.2 to 1.5 mg/ml of IgG were tested over 3 independent feeding experiments.

Based on the data from the pre-qualification feeds, the concentrations of 4B7 mAb and normal mouse IgG to be used for the qualification study were determined. In each feeding experiment, four concentrations of 4B7 mAb (1, 6, 23 and 94 µg/ml) were tested in triplicate (total of 12 COM), and normal mouse IgG (0.75 mg/ml) was tested with or without 4B7 mAb (either 23 or 94 µg/ml) as single samples (total of 3 COM). As a negative control, a feed without any antibody was also tested in triplicate (total of 3 COM). Therefore, we used a total of 18 COM for each feeding experiment, and three independent feeding experiments were performed in the qualification study (Feed 7–9).

Percent inhibition of mean oocyst intensity (PIm) was calculated as: 100×{1−(mean number of oocysts in the test)/(mean number of oocysts in the control)}.

Testing for feed effects on the mean oocysts in control groups was done by analysis of deviance from a quasi-Poisson model. In the qualification experiments, there were 3 COM of test for each concentration of 4B7 mAb and 3 COM of control within a feed. Thus, there were 9 different ways to estimate PIm. Because all 9 PIm values were not independent, we used generalized U statistics to combine them and got an unbiased estimate of Repeatability (intra-assay variability) and Intermediate Precision (inter-assay variability)

We then generated a model to estimate the effect of modifications in this assay (e.g., enumeration of oocysts from more mosquitoes per COM, testing more COM per feeding experiment) and to compare different analytical methods (e.g., use median instead of arithmetic mean). In this modelling, data from both the pre-qualification and qualification feeds (total of 9 feeds) were used, except for the feeding data with normal mouse IgG. We used a generalized linear mixed model (GLMM), specifically, a zero-inflated negative binomial model with random effects for both feed and COM. Let Y_{ijk}_{ijk}, where π is the zero inflation parameter, and μ_{ijk} is the random mean effect from the negative binomial portion of the model. The random mean is modelled as log (μ_{ijk}_{ij}_{i}_{ij}_{ij}_{i} is the normally distributed random effect for the _{ij}

While PIm was the main readout of SMFA in this study, to evaluate the relationship between PIm and % inhibition of prevalence (PIp), the zero-inflated negative binomial model was used with dispersion parameter, θ, and the given mean numbers of oocysts in the control. PIp was calculated as: 100×{1−(proportion of mosquitoes with any oocysts in the test)/(proportion of mosquitoes with any oocysts in the control)}, where the proportions come from the zero-inflated negative binomial model.

All statistical tests were performed by R (version 2.15.2) or Prism 5 (GraphPad Software Inc, La Jolla, CA), with the GLMM models fit using the glmmADMB R package

^{a}Specific measurement in the context of SMFA.

(DOCX)

(PDF)

_{1} and T_{2}), and true PIm of T_{1} (50 or 70% inhibition compared to control) is higher than the true PIm of T_{2} (0, 10, 20, 30, 40 or 50%). Three different control conditions were simulated; 1) mean number of oocysts in the control is 4 (Co = 4), 2) mean of 10 (Co = 10), and 3) mean of 30 (Co = 30). In addition, we stimulated either: 1) T_{1} and T_{2} are tested in the same feeding experiment (SF), or 2) tested in different feeding experiments (DF). We assumed 20 mosquitoes are dissected from a single COM. For each test condition, 10,000 data were generated to calculate the probability of feeds in which T_{1} showed higher PIm (i.e., lower mean oocyst number) than that T_{2}.

(PDF)

We would like to thank Andre Laughinghouse, Kevin Lee, Tovi Lehman and Robert Gwadz for insectary support.