^{1}

^{2}

^{3}

^{4}

The authors have declared that no competing interests exist.

Conceived and designed the experiments: WKW RBC CCH WW. Performed the experiments: CCH. Analyzed the data: WKW RBC CCH WW. Wrote the paper: WKW RBC WW.

Particle Swarm Optimization (PSO) is a meta-heuristic algorithm that has been shown to be successful in solving a wide variety of real and complicated optimization problems in engineering and computer science. This paper introduces a projection based PSO technique, named ProjPSO, to efficiently find different types of optimal designs, or nearly optimal designs, for mixture models with and without constraints on the components, and also for related models, like the log contrast models. We also compare the modified PSO performance with Fedorov's algorithm, a popular algorithm used to generate optimal designs, Cocktail algorithm, and the recent algorithm proposed by [

Mixture experiments are widely used in food processing, chemical, manufacturing, agricultural, cosmetics and pharmaceutical industries. For example, [

The aim of this paper is to introduce a popular optimization technique already widely used in engineering and computer science research and modify it to find various optimal designs for different types of mixture models. Particle Swarm Optimization (PSO) techniques have been around for more than ten years, but interestingly haven’t made much impact in statistical applications to date. Our experience reinforces the widely held findings that PSO is a very simple and powerful optimization tool. It requires no assumption of the objective function and has only a few easy-to-use tuning parameters. PSO is intriguing because even though the theory is not fully developed, its repeated successes and increasing widespread applications in various disciplines have even resulted in at least 3 journals that mainly track PSO development and applications in various fields.

Section 2 provides statistical setup and briefly reviews general theory for optimal experimental designs before discussing mixture models and their optimal designs. In Section 3, we propose a PSO-based method for finding a variety of optimal designs for different types of mixture experiments. In Section 4 we show that this procedure is efficient for finding various types of optimal or nearly optimal designs for different types of mixture models, including several new models for which optimal designs are not known analytically. Section 5 compares PSO performance with the popular Fedorov-exchange algorithms for finding optimal designs and we conclude in Section 6 with a summary of the advantages of the proposed PSO method over current methods for generating optimal designs.

Our interest is in the general linear model given by
_{i}’s) and the proportions (_{i}’s) of observations to be taken at these points. Once the sample size _{i} and _{i}, _{i} observations at _{i}, _{1} + … + _{k} = _{i} is a positive integer.

Following convention, the worth of a design _{ξ}(

The design criterion is formulated as a convex function of the information matrix. For example, ^{−1} and _{R}

Given a statistical model and a convex design criterion, [

a design ^{−1}(

a design ^{−1}(^{−1}(^{−1}(

The worth of a design _{L})^{−1}/^{−1}, where _{L} is the _{L}. To maintain this interpretation, ^{−1}(_{D})|^{1/d} where _{D} is the

It appears much of the recent design work for mixture models focuses on finding designs robust to model mis-specification. For example, [

We assume our mixture experiments have _{1}, _{2}, …, _{q} defined on the regular _{1}, _{2}, …, _{q}) given by
_{1}, _{2}, …, _{q}, _{1} _{2}, _{1} _{3}, …, _{q−1} _{q}), the second degree Scheffé’s polynomial mixture model is
^{th} order has the form:

[

[^{q−1}. Both _{1} = (4^{1/2}/(^{1/2} + 2_{1}/(4^{1/2} equally distributed among points of the form (1/2, 1/2, 0, …, 0), …, (0, …, 0, 1/2, 1/2). When _{1}, _{2}, _{3}) = (0.1418, 0.1873, 0.0128), where _{1}, _{2} are as before and _{3} is now the weight at each of the support point of the form (1/3, 1/3, 1/3, 0 …, 0), …, (0, …, 0, 1/3, 1/3, 1/3).

We next consider two third-degree polynomial models for mixture studies; the first one does not incorporate a 3-way effect and is given by
_{i} = 1, _{j} = 0, _{k} = 0, _{i} = 1, _{j} = 0, _{k} = 0, _{i} = _{j} = _{k} = 1/3, _{l} = 0,

An analytical description of the optimal design is desirable but as the above results show, they can be complicated even for relatively simple models and more frequently because of the mathematical complexity, they are usually not available. A more practical approach to find optimal designs is to use an algorithm. The next section describes a PSO-based algorithm that seems to have great potential for finding many types of optimal designs quickly for a variety of mixture and mixture-related models, including optimal designs for mixture models with constraints or on an irregular simplex or for some sub-models in

Particle swarm optimization (PSO), proposed by [^{th} in the PSO algorithm in its search to find the optimum is as follows. At times ^{th} particle at time _{t} modulates the influence of the former velocity and can be a constant or a decreasing function with values between 0 and 1. For example, [_{i} is the personal best (optimal) position attained by the _{g} is the global best (optimal) position attained among all particles up to time _{i} = Φ(_{i}) and _{g}). The two random vectors in the PSO algorithm are _{1} and _{2} and their components are usually taken to be independent random variables from _{1} is the cognitive learning factor and _{2} is the social learning factor. These two constants determine how each particle moves toward its own personal best position or overall global best position. The default values for these two constants in the PSO codes are _{1} = _{2} = 2 and they really seem to work well in practice for nearly all problems that we have investigated so far. Note that in

For our mixture design problem, the search space is the set of all approximate designs defined on the design space Ω, which is either a regular or irregular _{t} we used for finding optimal designs for mixture experiments is the linear decreasing function that varies from 0.9 to 0.4.

Following convention, PSO can and should always be modified to take advantage the special features of the optimization problem at hand. For our mixture experiments design problems, we found that a more effective way to first optimize over the regular hypercube, and then use a projection function to identify our target optimal design in the search space Ω. To fix ideas, suppose the given mixture model has ^{m} denote the _{i} is a ^{q}, ^{k} and define ^{q−1})^{k} × ^{k−1} by

Our modified PSO algorithm is based on the projection function ^{th} particle, i.e. ^{th} particle has ever visited, and (ii) let ^{gbest} provides the optimal value for the criterion among all the positions that all of the particles have ever visited. The strategy for the ^{th} particle, ^{th} iteration is as follows:

Generate a new velocity ^{th} iteration, _{t} is the inertia weight, _{1}, _{2} are two pre-specified positive constants, and _{1}, _{2} are

The next location for the ^{th} particle is

Project

Update the current best for each particle

Update the inertia weight _{t+1} = _{t}), where

^{gbest}as our “best” design after a pre-specified maximal number of iterations is reached or when the criterion value does not change much according to some user-specified tolerance level.

The key advantage of this modified PSO algorithm is that it operates on the simple hypercube first before it projects any non-feasible point into (^{q−1})^{k} × ^{k−1}. This simplifies and makes the computation more efficient. If we had directly implemented PSO to search for the optimal mixture design, our experience is that some of the sequentially generated particles “flew” outside the simplex and the subsequent work required to ignore them or bring them back to the simplex can complicate and prolong the search for the best design considerably. Because we had expanded the search space from the simplex to the hypercube, multiple solutions can exist but PSO is able to handle problems with multiple solutions [

In this section, we apply our ProjPSO algorithm to generate different types of optimal designs for different types of mixture experiment. We have verified that our algorithm produced the same optimal designs for many common models reported in the literature and for space consideration, we do not include them here. We focus on new optimal designs where analytical formulae for the optimal designs are not available or optimal designs that may be known but not commonly seen in the literature, such as those for Kasatkin’s polynomials. All ProjPSO-generated designs have been verified to be optimal using an equivalence theorem by plotting the directional derivative of the criterion at the ProjPSO-generated optimal designs over the design space. Figs

We implemented the ProjPSO algorithm written in MATLAB codes in a PC with a 2.67GHz Intel(R) Core(TM) i7 CPU. All CPU times reported in this paper were based on this hardware. We always start with a modest size of the particles and a modest number of the iterations and increase them sequentially to expand the dimensionality of the experimental region as the model has more parameters. In almost all cases, ProjPSO was able to generate designs which were optimal or very close the theoretical optimal designs after a couple of minutes of CPU time. For example, to find the

^{st}, 5^{th}, 10^{th}, 20^{th}, 30^{th} and the 40^{th} iterations and show the very rapid convergence of the ProjPSO procedure to the optimum. The figures show that the ^{th} iteration and ProjPSO used 10 more iterations to ensure that the criterion value, each support point and each mass agree with their previous values up to 5 decimal places. After 3 seconds of CPU time, the ProjPSO algorithm generated a design equally supported at the 3 vertices, which is the theoretical optimal design.

Each sub-figure displays the ProjPSO-generated mixture design or the global best design ^{gbest} at a particular iteration. After 40 iterations and only 4 seconds of CPU time, ProjPSO converged to the

In practice, we first searched using a flock where all have design points equal to the number of parameters in the model using the ProjPSO algorithm. When repeated number of times failed to produce the optimal design using different flock size and maximum number of iterations, this may suggest that the optimal design is not minimally supported. We then increase the number of support points of each design in the flock by one and repeat the process. Our guiding principle is larger number of particles or larger number of iterations for more complex models. Our experience is that the time required to generate the optimal design is usually fast and the difference in additional computational time required by either increasing the number of particles or iterations is usually not large. For instance, in the examples below, the number of particles we chose to generate the optimal designs for the linear Scheffé polynomial models were 64, 128 and 256 for

We next applied ProjPSO to find the

We were also able to verify ProjPSO-generated designs for Scheffé’s cubic model with and without 3-way effects are the same as those reported in [

We also used ProjPSO to determine optimal designs for several submodels obtained by deleting a few interaction terms from the full cubic model. These submodels or incomplete (IC) models are less studied even though they have been used in the development and optimization of microemulsion formulations in mixture experiments in the pharmaceutical industry [

model | type | support points | weights |
---|---|---|---|

IC Model A | ∑_{i} _{i} _{i}+_{13} _{1} _{3}+_{23} _{2} _{3}+∑_{i<j} _{ij} _{i} _{j}(_{i}−_{j})+_{123} _{1} _{2} _{3} |
(1, 0, 0), (0, 1, 0) | 0.0833 ×2 |

(0, 0, 1) | 0.1111 ×1 | ||

(0.2764, 0.7236, 0), (0.2764, 0, 0.7236) | 0.1111 ×4 | ||

(0.2113, 0.7887, 0), (0.7887, 0.2113, 0) | 0.0833 ×2 | ||

(0.3333, 0.3333, 0.3333) | 0.1111 ×1 | ||

IC Model B | ∑_{i} _{i} _{i}+_{23} _{2} _{3}+∑_{i<j} _{ij} _{i} _{j}(_{i}−_{j})+_{123} _{1} _{2} _{3} |
(1, 0, 0) | 0.0938 ×3 |

(0, 0.2764, 0.7236), (0, 0.7236, 0.2764) | 0.1250 ×2 | ||

(0.7887, 0.2113, 0), (0.7887, 0, 0.2113) | 0.0937 ×2 | ||

(0.2113, 0.7887, 0), (0.2113, 0, 0.7887) | 0.0937 ×2 | ||

(0.3333, 0.3333, 0.3333) | 0.1250 ×1 | ||

IC Model C | ∑_{i} _{i} _{i}+∑_{i<j} _{ij} _{i} _{j}+_{13} _{1} _{3}(_{1}−_{3})+_{23} _{2} _{3}(_{2}−_{3}) |
(1, 0, 0) | 0.1250 ×3 |

(0, 0.2764, 0.7236), (0.2764, 0, 0.7236) | 0.1250 ×2 | ||

(0.7236, 0, 0.2764), (0, 0.7236, 0.2764) | 0.1250 ×2 | ||

(0.5, 0.5, 0)) | 0.1250 ×1 | ||

IC Model D | ∑_{i} _{i} _{i}+∑_{i<j} _{ij} _{i} _{j}+_{13} _{1} _{3}(_{1}−_{3})+_{23} _{2} _{3}(_{2}−_{3})+_{123} _{1} _{2} _{3} |
(1, 0, 0) | 0.1111 ×3 |

(0, 0.2764, 0.7236), (0.2764, 0, 0.7236) | 0.1111 ×2 | ||

(0.7236, 0, 0.2764), (0, 0.7236, 0.2764) | 0.1111 ×2 | ||

(0.5, 0.5, 0)) | 0.1111 ×1 | ||

(0.3333, 0.3333, 0.3333) | 0.1111 ×1 |

We next apply ProjPSO to generate

model | type | support points | weights |
---|---|---|---|

Becker Model 1 Becker (1968, 1978) | ∑_{i} _{i} _{i}+∑_{i<j} _{ij}(_{i} _{j})^{(1/2)}+_{123}(_{1} _{2} _{3})^{(1/3)} |
(1, 0, 0) | 0.1429 ×3 |

(0.5, 0.5, 0) | 0.1429 ×3 | ||

(0.3333, 0.3333, 0.3333) | 0.1429 ×1 | ||

Becker Model 2 Becker (1968, 1978) | ∑_{i} _{i} _{i}+_{12} _{1} _{2}/(_{1}+_{2})+_{13} _{1} _{3}/(_{1}+_{3})+_{23} _{2} _{3}/(_{2}+_{3})+_{123} _{1} _{2} _{3} |
(1, 0, 0) | 0.1429 ×3 |

(0.5, 0.5, 0) | 0.1429 ×3 | ||

(0.3333, 0.3333, 0.3333) | 0.1429 ×1 | ||

Becker Model 3 Becker (1968, 1978) | ∑_{i} _{i} _{i}+_{12}min{_{1}, _{2}}+_{13}min{_{1}, _{3}}_{23}min{_{2}, _{3}}+_{123}min{_{1}, _{2}, _{3}} |
(1, 0, 0) | 0.1429 ×3 |

(0.5, 0.5, 0) | 0.1429 ×3 | ||

(0.3333, 0.3333, 0.3333) | 0.1429 ×1 | ||

Kasatkin 3rd Order Model Kasatkin (1974) | (1, 0, 0), (0, 1, 0) | 1/4 ×2 | |

(0.2764, 0.7236, 0), (0.7236, 0.2764, 0) | 1/4 ×2 | ||

Kasatkin 4th Order Model Kasatkin (1974) | (1, 0, 0), (0, 1, 0) | 1/5 ×2 | |

(0.1727, 0.8273, 0), (0.8273, 0.1727, 0) | 1/5 ×2 | ||

(0.5, 0.5, 0) | 1/5 | ||

Kasatkin 5th Order Model Kasatkin (1974) | (1, 0, 0), (0, 1, 0) | 1/6 ×2 | |

(0.1175, 0.8825, 0), (0.8825, 0.1175, 0) | 1/6 ×2 | ||

(0.3574, 0.6426, 0), (0.6426, 0.3574, 0) | 1/6 ×2 |

To test the capability of our algorithm ProjPSO, we applied it to mixture models with many factors. For this purpose, we consider Scheffé linear mixture model with

For the linear model with 10 parameters, the number _{i}, _{i} is the 0 vector except its ^{th} component is equal to unity. With _{i},

We next apply ProjPSO to find a

This subsection shows ProjPSO can be directly modified to find optimal designs for the linear log contrast models proposed by [_{i}/_{j} ≤ 1/^{q−1}.

As an illustration, consider the log contrast model with

To find the optimal design using ProjPSO, we redefined the regressors as log(_{i}/_{q}) and also amended the projection operator in ProjPSO so that it projects into the right space that includes the additional constraints, _{i}/_{j} ≤ 1/

We note here that this is one situation where we did not obtain good results using the default values _{1} = _{2} = 2 in the ProjPSO algorithm. This may be due to the smaller design space resulting from the several constraints. Our experience suggests that setting _{1} = _{2} = 0.5 seems to work well for log contrast models.

Mixture experiments sometimes have physical constraints imposed on the components. Because of practical or cost considerations, upper or lower bound constraints are imposed on some of the _{i}’s with user-specified constants _{i} and _{i}, such that _{i} ≤ _{i} ≤ _{i},

The first example concerns a cubic mixture model without the 3-way interaction term and there is a constraint on the percent of the first component in the mixture being not larger than one half, i.e. _{1} ≤ 0.5. The mean function is the cubic model in

The goal of our second example is to further test the ability of ProjPSO to find a _{i} is restricted by 0 ≤ _{i} ≤ _{i} and the _{i}’s are independently drawn from _{i}’s are 0.6133, 0.8572, 0.5478, 0.8094, 0.5075 and 0.6871. We employed ProjPSO with 15,360 particles and 6000 iterations to search for the

There are several algorithms for finding optimal designs for mixture experiments. Early algorithms were quite comprehensively reviewed in [

In this section, we compared results from the ProjPSO techniques with a few popular or recent methods for finding optimal designs for mixture models. We first discuss the various methods in the following order: (a) the OPTEX procedure in the commercial software package SAS, (b) the AlgDesign package in the free software R and (c) two recent modified exchange types of algorithms, one called the Cocktail algorithm proposed by [

(a) Comparison with the OPTEX procedure in SAS:

Commerical statistical software packages like SAS and JMP typically have a few menus for finding optimal designs for multi-factor polynomial models and mixture models, such as Scheffé polynomial models. However, these packages usually are available for searching exact

To compare the _{PSO−D} is

The OPTEX procedure in SAS uses the modified Fedorov’s exchange algorithm to generate _{SAS−D} after 1000 iterations:
_{SAS−D}))/_{PSO−D}))}^{1/8} = 0.9841, implying that the ProjPSO-generated design _{PSO−D} is more efficient. The ProjPSO-generated design continue to outperform the SAS-generated _{PSO−D} as expected.

(b) Comparison with the AlgDesign package in R:

We next compare performance of the algorithm called AlgDesign in the free software R package for generating optimal approximate designs for mixture models. This package uses the Federov exchange algorithm under the name optFederov to calculate approximate designs for the

We implemented the AlgDesign procedure using a grid set with 100 points uniformly spread out for each factor. Results found from AlgDeisgn and our ProjPSO algorithm were basically the same but we observed optimal designs found from the latter are sometimes slightly better in terms of the criterion value. For example, for the full cubic model with 3 factors, the optimal design _{AD−D} found by AlgDesign had 33 design points whereas the one found by ProjPSO _{PSO−D} had 10 points. The relative _{AD−D}))/_{PSO−D}))}^{1/10} = 0.9985. As another example, for the quadratic model with 4 factors, AlgDesign produced a 25-point _{PSO−A})^{−1}/_{AD−A})^{−1} = 0.9668. In either case, the ProjPSO-generated design wins.

(c) Comparison with two new modified exchange type algorithms:

Recently, two state-of-the-art algorithms for finding _{1}, …, _{r}) is the probability vector for all these points as potential support points of the optimal designs. Clearly, _{i} is the

We obtained the MATLAB code for the Cocktail algorithm from Yaming Yu’s web-site and recoded it using the weights exchange algorithm to search for ^{−6} and 127.4690 seconds of CPU time, the algorithm stopped at the 42^{th} iteration. The Cocktail algorithm generated design _{Cocktail−D} has 10 points at
_{PSO−D} is {_{Cocktail−D}))/_{PSO−D}))}^{1/8} = 1.0000 and so both designs are very close in terms of the

[

The stopping criterion for the YBT algorithm is based on the maximal values of the directional derivative of the objective function at all the design points. Following [^{−6}. After 206.225 seconds, the best design found by the YBT algorithm is the same as _{Cocktail−D}.

Grid size | 101 | 501 | 1001 | 2001 |
---|---|---|---|---|

CA | 12 points, 0.9999, 1.15625 secs | 12 points, 1.0000, 43.4063 secs | 10 points, 1.0000, 125.219 secs | 10points, 1.0000, 648.922 secs |

Yang | 12 points, 0.9999, 4.988 secs | 12 points, 1.0000, 141.232 secs | 10 points, 1.0000, 206.225 secs | 12 points, 1.0000, 1250.920 secs |

It is clear from the tables that the OPTEX procedure and the two new exchange algorithms can quickly produce highly efficient designs when we use a small grid size to search for the _{Cocktail−D} has one pair of points,

The above comparisons were carried out using the Scheffe polynomial

Grid size | 101 | 501 | 1001 | 2001 |
---|---|---|---|---|

CA | 12 points, 0.9747, 0.734 secs | 12 points, 0.9949, 19.312 secs | 12 points, 0.9974, 83.062 secs | 10points, 0.9987, 357.750 secs |

Yang | 12 points, 0.9747, 2.665 secs | 12 points, 0.9949, 36.074 secs | 11 points, 0.9974, 207.143 secs | 10points, 0.9987, 700.814 secs |

We note that the grid points (1, 0, 0), (0, 1, 0) and (0, 0, 1) are not feasible for the Cocktail and YBT algorithms and so they can only identify the grid points close to these corner point. In contrast, the ProjPSO algorithm assumed a continuous design space and was able to identify design points very close to the 3 corner points. More specifically, ProjPSO was able to determine, for example, (1.0000, 1.91e-019, 0), (0, 1.0000, 3.48e-017) and (0, 4.56e-018, 1.0000) as design points. As a concrete example, consider the Becker’s Model 1.

Grid size | 101 | 501 | 1001 |
---|---|---|---|

CA | 9 points, 1.0000, 0.937 secs | 9 points, 1.0000, 87.281 secs | 9 points, 1.0000, 1153.880 secs |

Yang | 9 points, 1.0000, 5.374 secs | 9 points, 1.0000, 103.417 secs | 9 points, 1.0000, 395.564 secs |

_{0} with _{0} observations and based on its results, we want to augment the design by another _{1} more observations so that the combined design _{0} + _{0} when we calculate the information matrix in the code. Specifically, we choose _{1} so that as a function of the weighted combination of the information matrices from _{0} and the second stage design _{1}, the

We demonstrate this procedure using Becker’s model 1 with _{0}, is equally supported at 8 points
_{1} = 8 more observations so that the combined design is _{1} to have the following support points:
_{0} is to augment it by _{1} so that the new design is

There are discrepancies in the above results from different packages and algorithms. Some are minor and others are less minor. For instance, only the ProjPSO algorithm was able to determine the

Requiring that a grid set be pre-specified to search for the design points of the optimal design implies that the resulting generated design can only be supported at a subset of the grid points. A fine grid leads to a more accurate search for the optimal design but at the expense of computational time. For example, if there are only 3 factors in the mixture model, it takes only 2 seconds to generate a set of 2001 uniformly spaced points for each factor, but when there are 4 factors in the model, it takes around 19570 seconds to generate 201 equally spaced points for each factor in the simplex. ProjPSO works on the a continuous domain and differentiates itself from many other algorithms by not requiring the user to specify a candidate set of designs points before the search can begin. We view this feature of ProjPSO a distinct advantage over its competitors.

Other advantages of using a PSO-based algorithm for finding optimal designs for mixture models over current methods are (1) the time required to generate the optimal design is generally faster than current methods; (2) it can be used to find optimal designs for models not available in the common statistical software packages; for example, ProjPSO finds the

PSO also compares favorably with other metaheuristic algorithms in one aspect. Many researchers from various fields frequently report that the tuning parameters in PSO seem easy to use and are not as sensitive as those in other metaheuristic algorithms. For example, in genetic algorithms (GA) and simulated annealing (SA), all tuning parameters have to be carefully selected before the algorithms work well, see for example _{1} = _{2} = 2 to successfully search for the various optimal designs for our mixture design problems. The only exception was the case when we wanted to find a _{1} = _{2} = 0.5 seems to work better than the default values with _{1} = _{2} = 2.

Our experience with ProjPSO suggests that for finding an optimal approximate design, only two parameters in the ProjPSO algorithm seem to matter; the flock size and the number of iterations. The rest of the parameters can be set to their default values. A larger size of randomly generated flock of birds covers a broader range of the search space and so is suitable for more complex and high dimensional problems. A larger number of iterations minimizes the chance of early termination and allows ProjPSO additional time to find the optimum, which it usually does not need for solving our design problems. Our typical value for a flock size is 256 or 512 for more complex models and smaller for less complex models. A typical maximum iteration number that we used is 300.

We plan to do follow up work in [

_{s}

_{p}

^{−1}