Optimal shortening of uniform covering arrays

Software test suites based on the concept of interaction testing are very useful for testing software components in an economical way. Test suites of this kind may be created using mathematical objects called covering arrays. A covering array, denoted by CA(N; t, k, v), is an N × k array over Zv={0,…,v-1} with the property that every N × t sub-array covers all t-tuples of Zvt at least once. Covering arrays can be used to test systems in which failures occur as a result of interactions among components or subsystems. They are often used in areas such as hardware Trojan detection, software testing, and network design. Because system testing is expensive, it is critical to reduce the amount of testing required. This paper addresses the Optimal Shortening of Covering ARrays (OSCAR) problem, an optimization problem whose objective is to construct, from an existing covering array matrix of uniform level, an array with dimensions of (N − δ) × (k − Δ) such that the number of missing t-tuples is minimized. Two applications of the OSCAR problem are (a) to produce smaller covering arrays from larger ones and (b) to obtain quasi-covering arrays (covering arrays in which the number of missing t-tuples is small) to be used as input to a meta-heuristic algorithm that produces covering arrays. In addition, it is proven that the OSCAR problem is NP-complete, and twelve different algorithms are proposed to solve it. An experiment was performed on 62 problem instances, and the results demonstrate the effectiveness of solving the OSCAR problem to facilitate the construction of new covering arrays.


Introduction
Functionality tests during software development demand special attention, and they are generally important for preventing malfunctions in software components. During the testing phase, it is desirable to find all errors that could arise in a software component before it is delivered to the user. If a software component has a large number of parameters, then testing it exhaustively might be expensive because of the large number of configurations that can arise from the different parameters' values; e.g., a software component with just 20 parameters of 2 different a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 values each would require 2 20 = 1,048,576 tests. An alternative is to test the system using a small, randomly generated test suite, but in this case, there is no guarantee of the testing coverage; instead, a better choice is to use a combinatorial testing approach that provides a coverage guarantee for small test suites. This combinatorial testing approach (also called interaction testing) guarantees the coverage of all interactions of a certain size among different values of the input parameters of a software component. This approach is based on evidence presented by [1] that many errors are produced by the interactions of only a few parameter values. Specifically, the cited authors showed evidence that test suites with an interaction size of 6 are sufficient to detect all known errors in a collection of different software components.
A uniform covering array (CA), denoted by CA(N;t, k, v), is a commonly used structure in interaction testing. It is an array C with dimensions of N × k constructed over Z v ¼ f0; . . . ; v À 1g with the property that every N × t sub-array covers all members of Z t v at least once. The value of N is the number of rows of C, i.e., the number of test cases; k is the number of columns or parameters; v is the number of values that each parameter can take; and t is the degree of interaction among the parameters. Because there are k t À Á sets of t columns {c 1 ,. . .,c t }, the number of different t-tuples that must be covered at least once in C is v t k t À Á . When a specific t-tuple is missing in a set of t columns (c 1 , . . ., c t ), we refer to it as a missing t-wise combination (or a missing combination, for short). Below, a CA(6; 2, 5, 2) in which all 2 2 5 2 À Á t-wise combinations are covered at least once is shown.
The covering array construction (CAC) problem is the search for the covering array number (CAN), i.e., the minimum value N for which an array CA(N;t, k, v) still exists. Formally, the CAN can be defined as CAN(t, k, v) = min{N|9 CA(N; t, k, v)}.
In some theoretical studies, the following definition is adopted: CAN(t, k, v) = O(v t log k) [2]. This definition is interesting because as the number of columns grows linearly, the number of rows grows only logarithmically. This is an advantage of such combinatorial structures because of the possibility of deriving small test suites. For instance, for a software component with 126 binary parameters, exhaustive testing would require 2 126 tests, whereas interaction testing with strength 2 would require only 10 tests.
A complementary problem to the CAC problem is known as the test suite reduction problem (TSRP), which consists of finding, for a given array, the smallest subset of rows that covers all t-wise combinations [3]. The CAC problem is a special case of the TSRP in which the input is an array that contains all v k distinct test cases.
For some special cases, there are algorithms that can solve the CAC problem in polynomial time: • when k = t + 1 [6].
However, the CAC problem remains highly combinatorial in most cases. Moreover, some variants have been proven to be NP-complete; e.g., the work presented in [2,7] shows the NPcompleteness of the problem of extending a matrix by one row with no fewer than m missing t-wise combinations. The problem defined in the current work is also NP-complete, as proven in this paper.
These approximate algorithms can be used to build non-optimal CAs in a reasonable time; some of these algorithms depend on the quality of their inputs to produce small CAs. Most of the time, these inputs are based on matrices that are nearly CAs. The objective of the present work is to construct matrices with sufficiently few missing combinations to still be considered quasi-CAs. Such arrays are created by solving the problem known as the Optimal Shortening of Covering ARrays (OSCAR); related results were published in [24]. The OSCAR problem is relevant to the construction of CAs because it can produce smaller CAs or excellent initialization matrices for meta-heuristic algorithms for constructing CAs. The main contributions of this work are as follows. It formalizes three of the five algorithms presented in [24]. It also presents seven new approximate strategies for solving the OSCAR problem. In addition, the present work offers a complete analysis of the performance of all of the new and old algorithms, something that has not been done before. Furthermore, it proposes three new benchmarks with more than 800 OSCAR instances, which extend the range of study to matrices with strengths of t = {2, 3, 4, 5}, whereas previous works have studied only t = 2; these benchmarks are used as part of the experiments conducted to analyze the strategies. These experiments not only evaluate how effectively the algorithms solve the OSCAR problem but also compare the best of them against state-of-the-art strategies. These experiments provide evidence that solving the OSCAR problem using the proposed approaches enables the creation of quasi-CAs that are better than other reported initialization functions and even than the fast and versatile IPOG-F, a state-of-the-art algorithm for constructing CAs; the main result is that the arrays produced using the proposed algorithms have 90% fewer missing t-wise combinations than those generated using the other approaches considered for comparison. This paper is organized as follows. In the problem definition section, the OSCAR problem is formally defined; its NP-completeness is proven, and some of its applications are described. In the related work section, some of the work related to initialization functions for meta-heuristics for CA construction is presented. Subsequently, the algorithms proposed in this work for solving the OSCAR problem are presented. In the experimentation section, an experiment performed to test the proposed algorithms for the construction of matrices with few missing combinations is presented. Finally, in the conclusions section, final comments regarding this work are provided.

Problem definition
Let A denote a CA(N; t, k, v) or a quasi-CA(N; t, k, v) (a quasi-CA is a matrix with a relatively small number of missing t-combinations). Then, the OSCAR problem can be defined as minftðB N 0 Âk 0 ÞjB is a submatrix of Ag, where tðBÞ is a function that counts the number of missing t-wise combinations in the given array and N 0 = N − δ and k 0 = k − Δ are defined in terms of two predefined integer values, 0 δ N − v t and 0 Δ k − t, which satisfy δ > 0 _ Δ > 0. Hence, an OSCAR instance is specified by the elements ðA; d; DÞ.
The search space for an OSCAR instance consists of all submatrices B of the given matrix A. Accordingly, the number of feasible solutions that form such a space can be estimated to be and k kÀ D À Á represent the numbers of different ways to choose subsets of rows and columns, respectively, from the original matrix A. Throughout the remainder of this document, for a given submatrix B, we use J R to denote the subset of rows chosen from A and J C to denote the subset of columns.
Alternatively, the matrix A can be represented by another matrix A 0 with dimensions of N Â k t À Á . This matrix has the same number of rows as A and contains one column for each subset of t columns derived from A. Each cell a 0 i;j 2 A 0 contains a value from the set {0, 1, . . ., v t − 1}; this value represents the t-tuple covered by row i in the subset of t columns associated with column j.
The OSCAR instance ðA; d; DÞ ¼ ðCAð4; 2; 3; 2Þ; 1; 0Þ is shown in Tables 3, 4 and 5. The initial matrix A is shown in Table 3, the t-tuples and sets of columns are shown in Table 4, and the new matrix representation A 0 is presented in Table 5 (t-wise combinations covered).
https://doi.org/10.1371/journal.pone.0189283.t001 Table 2. Solution to the OSCAR problem, B ¼ CAð4; 2; 3; 2Þ, when δ = 2 and Δ = 2.  Finally, the tuple ðA 0 ; d; DÞ is used to define an instance of the OSCAR problem, and the NP-completeness of the problem can be proven based on this new representation. The remainder of this section is devoted to this proof.

The proof that the OSCAR problem is NP-complete
To demonstrate the NP-completeness of the OSCAR problem, it is necessary to show it to be equivalent to a problem that is already known to be NP-complete. For this purpose, this work presents the transformation of the maximum cover (or MAXCOVER) problem (cf. [25] for a review of this problem) into the OSCAR problem. For the proof, the previously defined notation ðA 0 ; d; DÞ for an OSCAR instance is extended to ðA 0 ; d; D; hÞ, where the value h denotes an integer that supports the following question: is there a sub-array B of A 0 with dimensions of (N − δ) × (k − Δ) such that tðBÞ h? This question transforms the OSCAR problem into its decision form, which is required for this demonstration.
First, it is proven that the OSCAR problem is NP in nature. Let us begin with the case in which Δ = 0, meaning that the matrix B is a subset of only the rows of A 0 . Clearly, the size of the search space is reduced to N NÀ d À Á . The claim that the problem is NP in nature holds because computing the value of tðBÞ would require time proportional to O N Â k t À Á À Á to examine all possible t-wise combinations, which are equal in number to the number of columns of B. In other words, the question of whether tðBÞ h for the OSCAR problem can be answered in polynomial time in the dimensions of B. Now that it has been shown that the OSCAR problem is NP in nature, let us proceed with the transformation of the NP-complete MAXCOVER problem. The objective of the MAX-COVER problem is to cover a given set Q ¼ fq 1 ; q 2 ; . . . ; q l g, regarded as the universe. To achieve this goal, we must use a subset of Y ¼ fY 1 ; Y 2 ; . . . ; Y m g, where each subset Y i Q, for all 1 i m, is given in advance and has a size of at most C. This problem can be characterized by the tuple ðQ; Y; CÞ and can be transformed into an OSCAR instance ðA 0 ; d; D; lÞ as follows: a) The matrix A 0 is constructed, with m + 1 rows and l + max{|Y i |} + 1 columns. b) For 1 i m and 1 j l, the value a 0 i;j of each cell is 1 if subset Y i covers element q j or 0 otherwise. c) For 1 i m and j > l, the value a 0 i;j of each cell is 0. d) For i = m + 1, the value a 0 i;j of each cell is 0 if 1 j l or 1 otherwise. e) The values of δ, Δ, and h are set to m − C, 0, and 0, respectively. The matrix A 0 can be constructed in a time of O(lm), and the derived OSCAR instance is denoted by ðA 0 ; d; 0; 0Þ. Table 6 shows an example of the transformation of the MAXCOVER problem into the OSCAR 1 problem. The following elements are used in this case: The array A 0 has dimensions of 6 × 9, and the δ is equal to 5 − 3 = 2. Finally, to complete the proof that the OSCAR problem is NP-complete, we demonstrate that the OSCAR instance ðA 0 ; d; 0; 0Þ built from the MAXCOVER instance ðQ; Y; CÞ has a solution if and only if the latter has a solution. For this purpose, we start by showing that an optimal solution for ðA 0 ; d; 0; 0Þ must include row Y 6 of A 0 . This fact can be easily proven since all t-tuples must be covered in ðA 0 ; d; 0; 0Þ and those with the value 1 in any column j > l can only be covered by row Y m+1 . The next step is to show that there is a solution with C subsets for the MAXCOVER instance iff there is a matrix with C + 1 rows that solves ðA 0 ; d; 0; 0Þ. This condition can also be easily proven. We first note that the t-tuple with value 0 is covered for any column j l by the row Y m+1 . The same tuple is also covered for any column j > l by any row from {Y 1 ,. . .,Y m }. With this information, the only t-tuples that remain uncovered are those with value 1 in any column j l. Given that during the construction of the OSCAR instance, a t-tuple with value 1 is assigned only to those rows in columns j l that are associated with a subset of Y, the following claim is valid: any subset of Y that is formed of C elements and represents a solution for the MAXCOVER instance can also be transformed into a solution for the OSCAR instance. This claim is justified since the associated rows with the chosen C elements cover all t-tuples for any column but those with value 0 in columns j > l. Then, it is necessary only to add row m + 1 to cover the missing t-tuples. It is also true that a solution with C + 1 rows for the OSCAR instance is a valid solution for the equivalent MAXCOVER instance, since it is necessary only to choose those subsets of Y associated with the rows selected in the solution for the OSCAR instance. Finally, if one of these instances has no solution, then neither does the other; this claim holds because of the equivalence between such solutions, which has already been shown. Hence, it is demonstrated that a solution to the MAXCOVER problem implies a solution to the OSCAR problem. Table 6. The MAXCOVER instance specified by Q ¼ fq 1 ; q 2 ; q 3 ; q 4 ; q 5 g, Y ¼ fY 1 ¼ fq 1 ; q 2 ; q 5 g, Y 2 = {q 2 , q 4 , q 5 }, Y 3 = {q 1 , q 4 , q 5 }, Y 4 = {q 1 , q 2 , q 3 }, Y 5 = {q 2 , q 3 , q 4 }}, and C = 3 represented as an OSCAR instance.
Finally, any instance of the OSCAR problem for the case of Δ > 0 is equivalent to k kÀ D

À Á
instances of the problem with Δ = 0. Since it has been proven that instances of this special case are NP-complete, then the general case of the OSCAR problem is at least as complex.

Applications of the OSCAR problem
Methods of solving the OSCAR problem have the following applications: a) they can reduce the search space in the CAC problem; b) they can directly construct CAs, when there are no twise combinations missing in the matrices they generate; c) they can be used as initializing functions for meta-heuristics for CA construction; d) they can aid in the identification of better upper bounds for CA matrices; and e) they can be used for fine-tuning in experimental design. Each of these applications is detailed in the remainder of this section. The OSCAR problem successfully yields a quasi-CA that has zero a small number of missing t-wise combinations. Such a situation is convenient since instead of searching for a , corresponding to the original domain, it may be possible to construct such a CA from a relaxed region of a smaller size, The second and third applications of the OSCAR problem are related to the construction of CAs. The OSCAR problem enables the direct construction of CAs when tðBÞ ¼ 0, i.e., when B is a CA. Additionally, whenever the matrix constructed as a solution to an OSCAR instance is not a CA (i.e., the number of missing t-tuples is greater than zero), this solution can still be used indirectly for CA construction because it can serve as the initial solution for meta-heuristic algorithms. Note that the performance of a meta-heuristic for constructing CAs depends on the quality of the initial matrix. Hence, the sub-array obtained as a solution to the OSCAR problem is adequate for this purpose because it has only a few missing t-wise combinations; this is in contrast to arrays of the same size constructed using random initialization functions, which are likely to be missing a large number of the possible t-wise combinations due to their random nature. Some of the existing meta-heuristic algorithms designed for CA construction, which show dependence on the initial matrix, are reported in [21,23,26,27]. It is in algorithms of this type that the OSCAR problem finds its main area of application, namely, the generation of initial matrices with few missing t-wise combinations.
The fourth application of the OSCAR problem is the identification of new upper bounds for CA matrices. Many such upper bounds have been reported in the literature. For example, the best upper bounds for some CAs can be found in the repositories of [28,29]. In addition, some bounds on CAN(t, k, v) can be found in [29]; however, the corresponding CAs have values of N that are far from optimal.
Because of the hardness of the CAC problem, the value of CAN(t, k, v) for any arbitrary set of values of t, k, and v is generally unknown. However, suitable new upper bounds can be obtained from existing matrices; e.g., between CA(174; 2,110,9) and CA(177; 2,117,9), the upper bounds on the required numbers of columns for the cases of N = 175 and N = 176 are unknown, but it can be inferred that they should be between 111, . . ., 116. Because most of these upper bounds have not been shown to be optimal, the question arises as to whether other upper bounds can be found. We conclude that inputs derived by solving the OSCAR problem can be used to test potential upper bounds in order to find new bounds for CA(N; t, k, v); this can be achieved through the proper selection of the values δ and Δ used to reduce the matrix size.
Some specific cases of the values of δ and Δ are as follows: 1. When δ > 0 and Δ = 0, i.e., only the number of rows is to be reduced, the rows that are selected to be discarded are those whose elimination results in the minimum number of missing combinations in the final array.
2. When δ = 0 and Δ > 0, i.e., only the columns of columns is to be reduced, the columns that are selected to be discarded are similarly those whose elimination results in the minimum number of missing t-wise combinations. However, this case makes sense only when the array A is not a CA.
Finally, another application of solutions to the OSCAR problem is their direct use in testing scenarios. Through the careful selection of the OSCAR problem parameters δ and Δ, it is possible to ensure that resulting sub-array has the desired numbers of rows (i.e., test cases) and columns (i.e., parameters) to produce a quasi-CA (with 90-100% coverage of the t-tuples) that provides the required level of assurance.
The proposed methodology for the construction of CAs consists of generating an initial solution for a meta-heuristic algorithm by solving an instance of the OSCAR problem; i.e., the OSCAR problem is solved to obtain the solution B, which is then used as the initial array in a meta-heuristic algorithm.

Related work
The construction of CAs is a highly combinatorial problem that can benefit from the use of approximate algorithms to construct CAs of a desired size within a reasonable amount of time. Many researchers, instead of directing their efforts toward finding CAs with the minimum number of rows using an exact approach, have designed approximate algorithms to improve the best known upper bound for CAs and then reduce the gap between that bound and the CAN. These CA construction algorithms can be classified, in accordance with their characteristics, into the following types: (a) algebraic approaches, (b) exact approaches, (c) greedy approaches, (d) transformations, and (e) meta-heuristic approaches.
Algebraic methods have the characteristic that the CA construction process involves formulas or operations using mathematical objects such as vectors, finite fields, groups, and CAs with small values of t, k, and v. Some algebraic methods yield optimal constructions, including the CA(N; 2, k, 2) methods of [30] and [31]; Bush's construction method for CA(N; t, q + 1, q), where q is a prime or a prime power and q t (cf. [5]); and the zero-sum method of [6], which yields an optimal CA(t, t + 1, v) for any t ! 2. The main feature of these approaches is that most of them require small CAs or quasi-CAs from which to construct larger CAs.
Exact methods are exhaustive approaches for the construction of optimal CAs. Although some approaches include techniques for accelerating the search process, they generally require exponential time to complete their task, making them practical only for the construction of small optimal CAs. This category includes branch-and-bound (B&B) strategies, such as the work proposed by [10], which incorporates symmetry-breaking techniques, partial t-wise verification and fixed blocks in the bounding process, and the work of [8], which, for the generation of a non-isomorphic CA(N; 2, k, 2), uses a pruning strategy based on bounds defined by the minimum ranks established in terms of the CA size.
Greedy strategies are commonly used for combinations of the parameters N, t, k, and v for which exact methods are impractical, with the basic purpose of producing a good solution in a short time. The majority of commercial and open-source tools for generating test data (including AETG [32], TCG [17], ACTS [33], IPOG-F [19], and DDA [34]) use greedy algorithms for CA construction.
Transformations generally exploit the structure of existing CAs either to make them smaller or to support other approaches, e.g., algebraic approaches, in creating smaller CAs. This task is usually performed in one of two ways: a) through the identification of redundancy or b) through the construction of submatrices. Redundancy in a CA can be identified through the permutation of rows or columns or through the changing of symbols (cf. [35], [36] and [37]). However, approaches based on the construction of submatrices provide a better basis for new CAs, and the present work can be considered to be of this type.
Finally, similar to greedy methods, meta-heuristic approaches are strategies that are not guaranteed to find a CA with the minimum number of rows. In practice, meta-heuristic methods yield very good results, but they consume more CPU time than greedy algorithms. Some meta-heuristics that have been used to solve the CAC problem include simulated annealing (SA) [22], tabu search (TS) [38], memetic algorithms (MAs) [27], and genetic algorithms (GAs) [21].
For all of the strategies described above, the main goal is the construction of CAs, i.e., matrices with zero missing t-wise combinations. However, the CA construction performance of algebraic and meta-heuristic approaches is improved when the initial matrices are quasi-CAs, i.e., when they are missing only a small number of the possible t-wise combinations. This situation raises the question of how an initial matrix should be constructed for these approaches. The answer is to use initialization functions. Hence, these initialization functions are a key element of the development of meta-heuristics for CA construction.
The main initialization functions used in state-of-the-art methods are as follows: a) random matrix initialization [21,23,26,27], b) initialization with a balanced number of symbols per column [27], c) initialization through row augmentation [39], d) initialization based on submatrices [40], and e) initialization based on greedy strategies [41,42]. The four first strategies do not consider the number of missing t-wise combinations in the construction of the initial matrix. Strategies of the last type can be used to build CAs, but they are typically larger than the required matrix size; this situation results in random discarding of rows and/or columns that is also not optimized in terms of the number of missing t-wise combinations. Hence, an alternative is to use an existing matrix of greater size and optimize the row/column reduction process until a matrix of the required size is obtained. This optimization is exactly equivalent to solving the OSCAR problem, and this work proposes a wide variety of new meta-heuristic and hybrid strategies for this purpose.
In summary, whereas CA construction approaches (e.g., exact, greedy, algebraic, meta-heuristic and transformation methods) produce matrices with no missing t-wise combinations, the strategies presented in this work solve the OSCAR problem to generate quasi-CAs. Quasi-CAs are important because they can be used as initial matrices for CA construction strategies based on algebraic and meta-heuristic methods and can thus improve the performance of these methods in the construction of new CAs.
The remainder of this section provides a more detailed introduction to some of the relevant initialization functions found in the scientific literature related to this topic. These four initialization functions will be denoted by I 1 , I 2 , I 3 , and I 4 in this paper; Fig 1 shows an example of each one initialization function.
Each of the four initialization functions creates an array C with N rows and k columns, in which each cell is initialized with a symbol of the given alphabet {0, 1, . . ., v − 1} of v symbols. The function I 1 is presented in [21,23,26,27]; this function initializes each cell c ij of C NÂk with a symbol drawn at random from the set {0, 1, . . .,v − 1}. Fig 1 (random) shows an example of the use of I 1 to initialize a matrix C 10Â4 .
The function I 2 initializes C NÂk with a balanced number of randomly generated symbols per column. Each column k i , where 1 i k, will contain an almost uniform distribution of the symbols {0, 1, . . ., v − 1}. To achieve such uniformity, a symbol is generated at random for each of the N rows of column k i , but during the random generation process, it is ensured that the first For example, in a 10 × 4 matrix C with an alphabet size of v = 3, each of the four columns contains R 1 ¼ 3 À 10 À b 10 3 c3 À Á ¼ 2 symbols that appear b 10 3 c ¼ 3 times and R 2 ¼ 10 À b 10 3 c3 ¼ 1 symbol that appears d 10 3 e ¼ 4 times; this situation is exemplified in Fig 1 (balanced). The use of I 2 guarantees that each column has a balance in the cardinalities of each symbol, something that cannot be guaranteed when using I 1 .
The function I 2 is a generalization of the initialization function presented in [27] for solving the binary CAC problem using an SA approach.
The function I 3 initializes C NÂk one row at a time. This function generates the first row r 1 at random; i.e., each of its cells will contain a symbol randomly chosen from {0, 1, . . .,v − 1}. Subsequently, each new row is selected from a set of two random candidate rows d 1 and d 2 and is added to C. The chosen candidate row is the one that maximizes the Hamming distance with respect to all rows r s that already exist in C. The Hamming distance between two rows is equal to the number of positions at which the corresponding symbols are different; correspondingly, the Hamming distance between a candidate row d j and all rows already in C is equal to the number of positions l in each row r s that differ from the corresponding positions in d j , summed over all existing rows r s . Formally, this latter definition can be expressed as P kÀ 1 l¼0 hðr s;l ; d j;l Þ, where i is the number of rows already added to C and h(r s,l , d j,l ) = 1 if r s,l 6 ¼ d j,l or 0 otherwise. This process is repeated until all N rows have been created. This initialization function has been used previously in [39].
An example of the selection of a row as defined in I 3 is shown in Fig 2; the matrix C already contains 2 rows, and the third row will be the candidate d 1 because it maximizes the value of gðd j ; CÞ. Finally, the function I 4 initializes C NÂk based on groups of t columns. This function is based on the sub-array C 0 ¼ CAðv t ; t; t; vÞ, which is constructed using the v t combinations of symbols derived from an alphabet of size v and a strength value of t; e.g., C 0 ¼ CAð3 2 ; 2; 2; 3Þ will be formed of the elements in the set {00, 01, 02, 10, 11, 12, 20, 21, 22}, where each element represents a row in C 0 . The function I 4 is performed in two steps. In the first step, C 0 is used to define the symbols in the first t columns of the matrix C. During this process, juxtaposition of C 0 is applied to complete the N rows of C; specifically, C 0 is juxtaposed b N v t cv t times, and the remaining N À b N v t cv t rows of C are filled with the first rows of C 0 . In the second step, the first t columns of C are copied into the next subset of t columns whose symbols have not yet been defined, and the values are changed in some pairs of rows; these changes are executed by randomly choosing d N 2 e pairs of rows and, for each pair, exchanging the values of those columns in each row. This step is repeated until all k columns of C have been defined. If the number of columns in the last subset (t 0 ) is smaller than t, then only the first t 0 columns of C 0 are used. https://doi.org/10.1371/journal.pone.0189283.g001 Optimal shortening of uniform covering arrays This function is a generalization of the last initialization function presented in [40]. An example of this initialization method is shown in Fig 1 (

Algorithms for solving the OSCAR problem
This paper has formally defined the OSCAR problem and has proven that it is NP-complete. Now, various strategies are proposed for solving this problem. This section is devoted to this purpose; throughout the remainder of the section, each proposed approach is described in detail.
Given that a solution to a specific instance of the OSCAR problem is defined by two sets, J R and J C , and considering that each of these two sets can be selected using one of three different approaches (exact (E), greedy (G), or meta-heuristic (M)), it is possible to define 9 basic algorithms, as shown in Table 7. The superindices for the EE and GG options indicate the number of variants that have been defined. For the EE approach, the two corresponding algorithms are denoted by EE CR (first the number of columns is reduced, then the number of rows) and EE RC (first the number of rows is reduced, then the number of columns). Three variants have been defined for the GG approach; these variants are denoted by GG CR (first the number of columns is reduced, then the number of rows), GG RC (first the number of rows is reduced, then the number of columns), and GGR C (the numbers of columns and rows are reduced in an alternating fashion). Thus, we ultimately present a total of 12 possible algorithms for solving the OSCAR problem.
We first describe the reduction of the numbers of rows and columns of the initial matrix A using the greedy approach. Afterward, the three greedy algorithms GG RC , GG CR and GGR

À Á
; these algorithms are based on a B&B approach [43]. Next, the meta-heuristic algorithm for solving the OSCAR problem is presented; this algorithm is based on the SA approach. Finally, the six hybrid algorithms GE, EG, GM, MG, ME and EM for solving the OSCAR problem are defined.
Greedy algorithms GG CR , GG RC , and GGR C for solving the OSCAR problem The proposed greedy algorithms are based on two functions (F R and F C ) that reduce the number of rows or columns one element (row or column) at a time, starting from the array A, while considering the number of missing t-wise combinations after the reduction process. Examples of all of the greedy strategies proposed in this section are presented based on the OSCAR instance shown in Fig 3. It presents the problem instance specified by the matrix A ¼ CAð6; 2; 5; 2Þ and the values Δ = 1 and δ = 2. It shows each combination of columns, or each t-tuple, that is derived from A and all of the possible t-wise combinations of symbols that could be found in each of them; it also shows the sets A R and A C of rows and columns, respectively. Besides, it shows the auxiliary structure P, which is used to store the number of times that each t-wise combination is covered in each t-tuple; this structure P is a matrix of v t = 2 2 rows and k t À Á ¼ 5 2 À Á ¼ 10 columns, in which each cell p i,j contains the number of times that the i th twise combination of symbols appears in A in the subset of columns defined by the j th t-tuple.
Greedy approach for reducing the number of rows. The greedy function that reduces the number of rows is denoted by F R , and it is defined below. Let A R ¼ fr 1 ; r 2 ; :::; r N g be the set of rows of A, and let O ¼ fo 1 ; o 2 ; . . . ; o N g be a vector in which each element is associated with a row r i and has a value equal to the number of t-wise combinations that are exclusively covered by the associated row. The function F R selects the row {r i |i = min j {o j }} to be discarded; ties are broken randomly.
The function F R uses the vector O that describes the initial array A to choose a row r i to be discarded such that the value o i is minimized. Discarding that row from A results in an array Optimal shortening of uniform covering arrays A 0 such that tðA 0 Þ ¼ tðAÞ þ o i , since once row r i is discarded, the t-wise combinations that were covered exclusively by row r i are no longer covered in A 0 . Therefore, the resulting array A 0 without row r i will be missing the minimum possible number of t-wise combinations because o i has the minimum value among the elements of O. When o i = 0, row r i is clearly superfluous, since it does not cover any t-wise combinations exclusively.
Every time that a row r i is discarded in the reduction process, the number of rows that cover each of the t-wise combinations covered by the discarded row r i must be decreased by one. Whenever a t-wise combination is then covered by only a single remaining row j, the value of o j must be increased by one. We update the vector O in this way.
The time required to initially populate O for the function (in all algorithms with a greedy component, this process is called getN()), since it is necessary to explore all rows per set of t columns, to determine the number of times that each ttuple is covered, and to confirm the t-wise combinations that are covered by only one row. The time required to discard a row and update O is O N þ ðN À 1Þ k t À Á À Á , since it is necessary to first explore the vector O and then, for each set of t columns, verify the number of times that each t-tuple is covered in at most N − 1 rows. Tables 8, 9 and 10 illustrate the application of getNðAÞ and F R ðOÞ to the matrix A defined in Fig 3. The vector O shown in Table 8 is the result of the call to getNðAÞ. Each element of this vector has a value equal to the number of unique t-wise combinations covered by the corresponding row; e.g., the value o 1 = 2 implies that row r 1 contains two t-wise combinations that are exclusively covered by this row (these are the symbol combinations 00 and 10 corresponding to the t-tuples (c 2 , c 4 ) and (c 3 , c 5 ), respectively). Now, a call to F R ðOÞ will result in an arbitrary selection from among the rows {r 1 , r 2 , r 5 , r 6 }; let us assume that r 2 is chosen. The elimination of this row will produce the new vector O shown in Table 10. To illustrate the update operation of F R , Table 9 shows how the auxiliary structure P is modified in accordance with the t-wise combinations that are eliminated with the deletion of row r 2 ; note that there are 8 new t-wise combinations that are now uniquely covered in the remaining rows. In the new  Table 9. Greedy approach for reducing the number of rows: Examples of the getN() and F R ðOÞ functions. P matrix.

t-tuple
https://doi.org/10.1371/journal.pone.0189283.t009 vector O, the value of the element corresponding to each of these rows is incremented by the number of t-wise combinations in that row for which the corresponding value in P has been changed to 1 after the elimination of row r 2 . For example, the t-wise combinations 01, 00 and 10 associated with t-tuples (c 1 , c 2 ), (c 1 , c 3 ), and (c 2 , c 3 ) are newly exclusively covered by row r 5 after the removal of row r 2 ; consequently, o 5 is increased from 2 to 5 in the new vector O. Greedy approach for reducing the number of columns. The function that reduces the number of columns using the greedy approach is denoted by F C and is defined below. Let A C be the set of columns of A; let K, with dimensions of k × k, be an array in which each element k i,j stores the number of times that columns i and j together are involved in a missing t-wise combination; and let U ¼ fu 1 ; u 2 ; . . . ; u k g be a vector in which u i ¼ P k j¼1 k i;j . The function F C selects the column {c i |i = max j {u j }} to be discarded; ties are broken randomly. Whenever a column i is discarded, the vector U is updated by subtracting the value k i,j from u j for all j 6 ¼ i. Each element in U is associated with a column, and its value is equal to the number of times that column is involved in a missing t-wise combination.
In summary, the function F C chooses a column i associated with the maximum value u i in the vector U. When discarding column i, we obtain an array A 0 such that tðA 0 Þ ¼ P kÀ 1 j¼0 u j , since once column i has been discarded, the associated missing combinations involving column i are deleted. Therefore, the resulting array A 0 will have the minimum number of missing t-wise combinations, since u i has the greatest value among the elements of U.
When we discard a column, the values of the elements of U must be updated. To do so, a value of −1 is assigned to u i , and the value of each element u j such that j 6 ¼ i is updated based on its interaction with the recently discarded column; i.e., u j = u j − k i,j . This process is intuitively illustrated as follows. Suppose that we have a set of n criminals who are accused of having committed m crimes together, and suppose that the authorities have found that a certain criminal s is the only one who committed l of these crimes, where l m; then, the number of crimes of which each of the remaining criminals is accused must be decreased in accordance with his initially suspected degree of participation in committing crimes with criminal s.
The time required to initially populate U and K for the function (in all algorithms with a greedy component, this process is called getK()), since for each set of t columns, all N rows must be explored, the vector G must then be updated based on the missing t-wise combinations in these columns, and K must be updated for all possible pairs in this set of t columns. The time required to discard a column and update U and K accordingly is O(2k), since the vector U must be explored to obtain the column i with the greatest value, and column i of K must then be explored to update U. Tables 11 and 12 illustrate the application of getkðAÞ and F C ðK; k; UÞ to the matrix A defined in Fig 3. The matrix K and the vector U shown in Table 11 are the results of the call to getkðAÞ; given that the initial matrix A is a CA, all values in K and U are zero because there Table 11. Greedy approach for reducing the number of columns: Examples of the getkðAÞ and F C ðK; k; UÞ functions. are no missing t-wise combinations. Now, a call to F C ðK; k; UÞ will result in the arbitrary selection of a column from among {c 1 , c 2 , c 3 , c 4 , c 5 }; let us assume that c 2 is chosen. The elimination of this column will produce the new vector U shown in Table 12, which also has zero missing t-wise combinations because the values c j , for j 6 ¼ 2, are all zero. Now that the greedy functions F R and F C have been defined, the three greedy algorithms are introduced below.
Greedy algorithm GG CR . GG CR is a greedy algorithm that first reduces A to a matrix with k − Δ columns using the function F C . The newly formed array is denoted by A 0 and has N rows and k − Δ columns. Then, A 0 is further reduced to a matrix with N − δ rows using the function F R , yielding the solution B. Algorithm 1 describes the GG CR approach. Algorithm 1 The time required to execute GG CR can be calculated from the times required for populating and updating the necessary structures, as follows: During the execution of this algorithm, Δ columns are first discarded from A following the defined reduction process, resulting in an array A 0 with k − Δ columns; then, the necessary structures for eliminating rows from A 0 are populated, and finally, δ rows are discarded following the defined reduction process, yielding the solution B.  Fig 4(a) shows the changes made to A due to the elimination of columns (see the loop in lines 5 to 8); for each iteration i of this loop, the table presents the initial vector U, the set J C of columns chosen so far, the vector U new obtained by updating U after the elimination of the column c selected in that iteration, and the resulting matrix A 0 after that iteration. This part of the algorithm is performed only once because Δ = 1. Subsequently, Fig 4(b) presents the changes made to the last matrix A 0 obtained in the previous process due to the elimination of rows (see the loop in lines 11 to 14); for each iteration i of this loop, the table presents the vector O derived from the previous matrix A 0 , the set J R of rows chosen so far, the updated vector O new obtained after the elimination of the row r selected in that iteration, and the resulting matrix A 0 after that iteration. This second loop is repeated twice because δ = 2. The last matrix A 0 obtained in the second part of GG CR is returned as the final matrix B. Greedy algorithm GG RC . The greedy algorithm GG RC first removes δ rows from A using the function F R to obtain an array A 0 with N − δ rows and k columns. Then, A 0 is reduced to a matrix with k − Δ columns using the function F C to obtain the final solution B. Algorithm 2 describes the GG RC approach.  The time required to execute GG RC can be calculated from the times required for populating and updating the necessary structures. The result is , since N − δ rows are first discarded from the input array A, generating an array A 0 with N − δ rows and k columns, and this array is then reduced to one with k − Δ columns to obtain the solution B. Optimal shortening of uniform covering arrays filled with a value of one. Once the Δ columns have been distributed among the δ rows, one row is discarded, and then, the number of columns is reduced to k − d i . Hence, with the exploration of each element i of the vector D, the numbers of rows and columns of the array A 0 will be decreased to N i = N − i and k i ¼ k À P iÀ 1 j¼1 d j , respectively. Algorithm 3 describes the GGR C approach.
The time required to execute GGR C can be obtained by considering how the numbers of columns and rows of A 0 will be reduced while exploring the vector D; the result is As each element i of D is explored, first, the necessary structures are populated to discard rows from the array A 0 , which has N − i + 1 rows and k À P iÀ 1 j¼1 d j columns, and the number of rows is reduced to N − 1. Next, it is necessary to populate the structures needed to discard columns from the new array A 0 with the reduced number of rows, and then, the number of columns is reduced to k − d i . Because Δ is not greater than δ in this instance, the number of columns that must be eliminated with the elimination of each row is given by the vector D ¼ fd 1 ¼ 1; d 2 ¼ 0g; i.e., after the deletion of the first row, one column must be deleted, and then the algorithm proceeds to the deletion of the second row to satisfy the value δ = 2. The example shown in Fig 6 illustrates the reduction process for the instance given in Fig 3. The first column lists the main structures that are changed during the execution of the algorithm. Each of the remaining columns in Fig  6 represents a different iteration of the main loop of the algorithm. if Δ > δ then 6: for

Meta-heuristic algorithm MM
The approximate algorithm MM for searching for a solution to the OSCAR problem is based on the SA approach and is described in Algorithm 4. This approach is a general-purpose stochastic optimization strategy that has been proven to be an efficient means of approximating global optimal solutions to many NP-complete combinatorial optimization problems. In this strategy, a solution W u is first constructed using the Initialize(. . .) method, and this solution is designated as the first global best solution W Ã ; then, the algorithm enters an iterative improvement process, controlled by the length of the Markov chain, until a certain termination criterion is achieved. In each iteration of this improvement process, a new solution W v is generated using the GenerateNeighbor(.. .) method, and this new solution is substituted for W u whenever its quality is superior to that of the current solution or the probability condition is satisfied. The probability condition is based on the Boltzmann distribution, and it is defined with respect to the values of an initial temperature T i , a final temperature T f , and a quality function τ(. . .) of W u and W v . The global best W Ã is also updated every time a solution W v improves upon it. The details of these procedures are presented in the remainder of this subsection. A solution is represented by a vector W with N + k elements, which identify the subsets of the rows and columns of the matrix A that are used to construct a new submatrix B. All of the elements are binary; each of the first N elements is associated with a particular row in A, and each of the last k elements is associated with one of its columns. The subset J R & A R of rows to be excluded from the new submatrix B consists of the rows of A R that are associated with the corresponding elements in W that have a value of 1. Similarly, the subset J C & A C of columns to be excluded from the new submatrix B consists of the columns of A C that are associated with the corresponding elements in W that have a value of 1.
The Initialize(N,k, δ, Δ) method is used to construct the first solution W u in the proposed strategy. This method uses the best greedy algorithm among those proposed in this paper. The greedy strategy is chosen based on preliminary experiments for SA OSCAR .
Once the initial solution W u has been created, it is modified using the neighborhood function GenerateNeighborðS; W u ; PÞ. This method randomly chooses from among three predefined strategies, S 1 , S 2 and S 3 , to create a new solution W v . In the strategy S 1 , a row of W u is exchanged; one existing row in the solution is randomly removed and replaced with a different row not previously included. The strategy S 2 follows the same approach as that of S 1 but for columns. Finally, the strategy S 3 is the combination of the previous two. The use of the neighborhood function is controlled by the parameter L, which is called the Markov chain length.
The quality function, or evaluation function, that is used to measure the fitness of a solution is derived from the definition of the OSCAR problem. This function, denoted by tðWÞ, counts the number of missing t-wise combinations. We note that one missing t-wise combination represents a t-tuple for a particular combination of columns that the matrix does not contain but would be required to cover in order for the matrix to be a CA(N − δ; t, k − Δ, v).
Finally, the cooling schedule is controlled by a cooling factor α, which is used to gradually decrease an initial temperature T i until it reaches a given final temperature T f , marking the end of the algorithm. Note that the algorithm also includes an alternative termination criterion, which is defined as a maximum number of iterations of the main loop.
The time required to execute MM is O(iL), where i ¼ log a T f À log a T i log a a is the number of temperature decrements necessary to reach T f .

Exact algorithms EE RC and EE CR
The exact algorithms EE RC and EE CR were previously reported in [24]. They follow the B&B strategy (cf. [43]) and avoid the need to explore the entire feasible region to find the optimal solution. The general idea behind these algorithms is described in Algorithm 5 and Algorithm 6. These previously presented algorithms first construct an initial solution B using a greedy strategy; then, they remove from this solution all possible combinations of rows and columns such that the resulting matrix B 0 has N − δ rows and k − Δ columns. Because the order in which the rows and columns are removed matters, the strategies differ in their selection of which elements are removed first. Whereas EE RC first removes a subset of columns, EE CR first removes a subset of rows. Both algorithms, after the first selected elements have been removed, perform a B&B search over the columns and/or rows, testing each element one by one, in order to find the submatrix B Ã with the minimum number of missing t-wise combinations. We note that the elimination of the second set of elements obeys a lexicographical order given by the columns or rows that are being deleted. Moreover, during the search process, both algorithms avoid the elimination of certain columns or rows that would exert undesirable effects on the final submatrix (i.e., selections for which the number of missing t-wise combinations would increase over a certain upper bound); for a more detailed description of the algorithms, refer to [24]. The time required to execute either since, in the worst case, it is not possible to discard solutions.

Hybrid algorithms for solving the OSCAR problem
Hybrid algorithm GE. The algorithm GE combines the greedy and exact approaches to solve the OSCAR problem. The algorithm proceeds in two phases. First, it chooses a set of Δ columns and removes them from the initial matrix A; the resulting matrix is denoted by A 0 and has dimensions of N × (k − Δ). Afterward, the algorithm discards δ rows from A 0 in a greedy manner to construct a possible solution B 0 , i.e., a matrix with dimensions of (N − δ) × (k − Δ), for the OSCAR instance at hand. To obtain the best solution B, the algorithm explores all possible combinations of Δ columns and identifies the best matrix B from among all matrices B 0 constructed during the process described above. Algorithm 7 for j 0 to j < δ do 8: r F R ðOÞ 9: The algorithm GE for solving the OSCAR problem is described in Algorithm 7. Each combination of k − Δ columns is represented by the vector J C . Each new combination of k − Δ columns is computed by the function GREATERTHANPOLYNOMIAL() [44]. An array A 0 , with dimensions of N × (k − Δ), is constructed using the columns indicated by J C . Then, the algorithm populates the necessary structures to reduce A 0 to a matrix with N − δ rows, and this reduction process yields an array B 0 . The best solution that has been found so far during the exploration process is represented by B. Whenever tðB 0 Þ < tðBÞ for a newly constructed matrix B 0 , the matrix B is replaced with B 0 .
The time required to execute GE can be derived from the times required for populating the necessary structures, reducing the number of rows, and updating the necessary values. This . Hybrid algorithm EG. The algorithm EG also combines the exact and greedy approaches to find a solution to the OSCAR problem; compared with GE, the difference is that it explores all possible combinations of δ rows that can be eliminated from the original matrix. The algorithm proceeds in two phases. First, it chooses a set of δ rows and removes them from the initial matrix A; the resulting matrix is denoted by A 0 and has dimensions of (N − δ) × k. Subsequently, the algorithm greedily discards Δ columns from A 0 to construct a possible solution B 0 , i.e., a matrix with dimensions of (N − δ) × (k − Δ), for the OSCAR instance at hand. To obtain the best solution B, the algorithm explores all possible combinations of δ rows and identifies the best matrix B from among all matrices B 0 constructed during the process described above.
The algorithm EG for solving the OSCAR problem is described in Algorithm 8. Each combination of N − δ rows is represented by the vector J R . Each new combination of N − δ rows is computed by the function GREATERTHANPOLYNOMIAL(). An array A 0 , with dimensions of (N − δ) × k, is constructed using the rows indicated by J R . Then, the algorithm populates the necessary structures to greedily reduce A 0 to a matrix with k − Δ columns, and this reduction process yields an array B 0 . The best solution that has been found so far during the exploration process is represented by B. Whenever tðB 0 Þ < tðBÞ for a newly constructed matrix B 0 , the matrix B is replaced with B 0 . Algorithm 8 Hybrid algorithm GM. The algorithm GM uses a hybrid strategy that combines the SA meta-heuristic [45] with the greedy approach to construct a solution to the OSCAR problem. In each iteration of GM, a local search is performed over the possible set of columns that can be eliminated to obtain a matrix A 0 with dimensions of N × (k − Δ). Afterward, the matrix A 0 is subjected to a greedy process to reduce its size by δ rows and thus to construct a solution B with dimensions of (N − δ) × (k − Δ) for the OSCAR instance at hand. Once the matrix B has been built, the Boltzmann criterion is used as usual in SA. The details of the strategy are presented in the remainder of this subsection.
Algorithm 9 describes the proposed GM approach for solving the OSCAR problem. The algorithm GM uses a vector W u of size k to represent the state of each column in the solution B. The elements of the vector take values of w i 2 {0, 1} for 1 i k, where a value of 0 indicates that the corresponding column is not present in the solution and a value of 1 indicates otherwise. In addition, two constraints are imposed to obtain a proper OSCAR solution: there must be k − Δ elements with a value of w i = 1, and there must be Δ elements with a value of w i = 0. Algorithm 9 1: function GMðA; d; D; T i ; T f ; a; EÞ 2: for i 0 to i < L do 9: W v GENERATENEIGHBORðW u Þ

12:
for i 0 to i < δ do 13: The algorithm GM uses a set of perturbations to the vector W u as its neighborhood function. For this purpose, it chooses two elements w i and w j , where w i 6 ¼ w j , and interchanges their values. The new solution formed via this perturbation, which is a neighbor of W u , is denoted by W v .
Finally, the evaluation function used in GM is τ, the number of missing combinations in a created matrix. This function is also used to evaluate the matrices created during the local search.
The time required to execute GM is proportional to the number of temperature decrements necessary to reach T f and F R is the time cost of the greedy approach for eliminating rows. Hybrid algorithm MG. The algorithm MG uses another hybrid strategy that combines the SA meta-heuristic [45] with the greedy approach to construct a solution to the OSCAR problem. In each iteration of MG, a local search is performed over the possible set of rows that can be eliminated to obtain a matrix A 0 with dimensions of (N − δ) × k. Afterward, the matrix A 0 is subjected to a greedy process to reduce its size by Δ columns and thus to construct a solution B with dimensions of (N − δ) × (k − Δ) for the OSCAR instance at hand. Once the matrix B has been built, the Boltzmann criterion is used as usual in SA. The details of the strategy are presented in the remainder of this subsection.
Algorithm 10 describes the proposed MG approach for solving the OSCAR problem. The algorithm MG uses a vector W u of size N to represent the state of each row in the solution B.
The elements of the vector take values of w i 2 {0, 1} for 1 i k, where a value of 0 indicates that the corresponding row is not present in the solution and a value of 1 indicates otherwise. The constraints imposed to ensure a proper OSCAR solution are as follows: there must be N − δ elements with a value of w i = 1 and δ elements with a value of w i = 0. Algorithm 10 for i 0 to i < L do 9: W v The algorithm MG uses a set of perturbations to the vector W u as its neighborhood function. For this purpose, it chooses two elements w i and w j , where w i 6 ¼ w j , and interchanges their values. The new solution formed via this perturbation, which is a neighbor of W u , is denoted by W v .
Finally, the evaluation function used in GM is τ, the number of missing combinations in a created matrix. This function is also used to evaluate the matrices created during the local search.
is the number of temperature decrements necessary to reach T f and F C is the time cost of the greedy approach for eliminating columns. Hybrid algorithm ME. The algorithm ME combines the meta-heuristic and exact approaches to solve the OSCAR problem, using a strategy based on the exploration of all possible combinations of Δ columns that can be eliminated from the original matrix. The algorithm proceeds in two phases. First, it chooses a set of Δ columns and removes them from the initial matrix A; the resulting matrix is denoted by A 0 and has dimensions of N × (k − Δ). Then, the algorithm uses the SA approach to discard δ rows from A 0 to construct a possible solution B 0 , i.e., a matrix with dimensions of (N − δ) × (k − Δ), for the OSCAR instance at hand. To obtain the best solution B, the algorithm explores all possible combinations of Δ columns and identifies the best matrix B from among all matrices B 0 constructed during the process described above.
Algorithm 11 describes our ME approach. For each possible combination of columns J C & A C , the algorithm performs a meta-heuristic search to define the set of rows J R & A R . To determine each element in J C , the function GREATERTHANPOLYNOMIAL() [44] is used to systematically generate each different combination of columns. Algorithm 11 while n < E and T i < T f do 10: for j 0 to j < L do 11: W v The time required to execute ME is O k is the number of temperature decrements necessary to reach T f . Hybrid algorithm EM. The algorithm EM also combines the meta-heuristic and exact approaches to solve the OSCAR problem; compared with ME, the difference is that it explores all possible combinations of δ rows that can be eliminated from the original matrix. The algorithm proceeds in two phases. First, it chooses a set of δ rows and removes them from the initial matrix A; the resulting matrix is denoted by A 0 and has dimensions of (N − δ) × k. Then, the algorithm uses the SA approach to discard Δ columns from A 0 to construct a possible solution B 0 , i.e., a matrix with dimensions of (N − δ) × (k − Δ), for the OSCAR instance at hand. To obtain the best solution B, the algorithm explores all possible combinations of δ rows and identifies the best matrix B from among all matrices B 0 constructed during the process described above.
Algorithm 12 describes our EM approach. For each possible combination of rows J R & A R , the algorithm performs a meta-heuristic search to define the set of columns To determine each element in J R , the function GRATERTHANPOLYNOMIAL() [44] is used to systematically generate each different combination of rows. Algorithm 12 1: function EMðA; d; D; T i ; T f ; a; EÞ 2: while n < E and T i < T f do 10: for i 0 to i < L do 11: W v is the number of temperature decrements necessary to reach T f . In the next section, we demonstrate the performance of our 12 algorithms.

Experimentation
This section presents the experimental design used to test the performance of the proposed algorithms for solving the OSCAR problem. The methodology consisted of the following steps: 1) A set of benchmark instances was defined.
2) The parameters of the SA algorithm were subjected to a fine-tuning process.
3) The performances of the algorithms were evaluated by using them to solve the benchmark problem instances. 4) A performance comparison against stateof-the-art initialization functions was conducted. 5) The results derived from the algorithms were used to define new upper bounds for existing CAs. The proposed algorithms were implemented in the C language and compiled using gcc with the optimization option -O3. We used a computer with 72 Intel Xeon 1.6 GHz CPU cores and RAM of 64 GB. The remainder of this section describes the experimental methodology in detail.

Definition of the benchmarks
This subsection introduces the three benchmarks used to properly test the proposed set of OSCAR algorithms. The benchmark L 1 (S1 dataset) consists of 12 small CAs, which are described in Table 13, and it is used to analyze the performance of all algorithms presented in this document; then, the algorithms that achieve the best experimental results on this benchmark in terms of both time and solution quality are further tested on the following benchmark. The benchmark L 2 (S2 dataset), presented in Table 14, consists of 62 CAs; it is an extension of the benchmark presented in [24] such that the adjusted values of δ and Δ provide support for the discovery of a greater number of new upper bounds for the related CAs. This benchmark aids in the identification of the OSCAR solver with the best overall experimental performance, and it is also used to compare the results of the proposed OSCAR solvers against other stateof-the art initialization functions. Finally, the benchmark L 3 (S3 dataset) consists of 820 instances (see Table 15); this benchmark is used to evaluate the quasi-CA construction performance of IPOG-F, a classical and versatile (in the sense that it can rapidly construct any type of CA) greedy algorithm that is widely used in the literature, against the best OSCAR strategies identified in the experiments on the previous benchmarks in terms of both the time required for matrix construction and the quality of the constructed matrices. Table 15 presents the instances included in benchmark L 3 , organized into 20 sets. In each set, one OSCAR instance is defined per value of k considered (from 10 to 50), as shown in column 1; the remaining columns show the values for v, t, δ, and Δ, which correspond to the alphabet size, the strength, and the numbers of rows and columns to be eliminated, respectively. We note that the benchmark L 3 is also characterized by its wide variety of values of the strength t and the alphabet size v.

Fine-tuning of the parameters of MM
The MM approach is the basis for several of our other approaches. Because this approach uses the SA algorithm, a fine-tuning process is necessary to adjust the values of its parameters to improve its performance. During the tuning process performed in this study, the Markov chain length L, the final temperature T f , and the initialization function G were fixed; all remaining parameters (i.e., the initial temperature T i , the decrement factor α, and the maximum number of evaluations E) were subjected to adjustment. Because different neighborhood Table 13. Benchmark L 1 , which is composed of 12 small instances of the OSCAR problem. The column 2 shows the CA uses as initial array, while the columns 3 and 4 show the number of rows δ and columns Δ to be shortened, respectively.

Instance
A δ Δ Optimal shortening of uniform covering arrays functions are used in our approach, each with a certain probability of being applied, a fourth parameter was also considered during the tuning process: the application probability of each neighbor function, denoted by P. The goal of this fine-tuning process was to test the performance of MM using different configurations of the parameter values to identify the configuration that yielded the best performance. The sets of values considered for the parameters T i , α, and E were {1, 4}, {0.90, 0.99}, and {100L, 500L}, respectively. In the fine-tuning approach presented in [46, 47], a CA is used as a means of systematically sampling the entire set of parameter value combinations; the method starts at an initial level of interaction t, which is used to construct a CA (N; t, k, v), and t is then increased until the generated sample is suitable for the purposes of the experiment. The present study required the smallest possible sample in order to reduce the experimental time; this sample was constructed using an interaction level of t = 2. A summary of the final combinations of values tested, derived from the constructed CA(4; 2, 3, 2), is shown in Table 16. Table 14. Benchmark L 2 , which is composed of 62 instances of the OSCAR problem. Each instance shows the initial array A, and the number of rows δ and columns Δ to be shortened. Optimal shortening of uniform covering arrays Meanwhile, the vector of probabilities P used for the initialization functions S i was defined based on solutions to the Diophantine equation a 1 x + a 2 x + a 3 x = 10, following the approach presented in [44]. During this process, each of the 66 solutions to the Diophantine equation was used to generate a possible vector P, in which the probability value for each initialization function i was estimated as x i 10 . Because we considered 4 different configurations of the values of the parameters T i , α, and E and 66 configurations of the probability vector P, the experiment to fine-tune MM Table 15. Groups of instances' sets that form the benchmark L 3 . The column 1 is the identifier of the groups. The columng 2 shows the ranges of k, the number of columns. The remaining columns are the alphabet v, strength t, and rows δ and columns Δ to be shortened.

Instance set
A  The results obtained from the fine-tuning process indicated that the optimal parameter values for MM are T i ¼ 4, α = .99, and E ¼ 100L and that the desired solution to the Diophantine equation is a 1 = 4, a 2 = 3, and a 3 = 3. This configuration was also used in the algorithms GM, MG, ME, and EM, which also use the meta-heuristic MM approach.

Evaluation of the 12 proposed algorithms
This section presents the evaluation of the 12 proposed algorithms for solving the OSCAR problem. All algorithms were tested on the smaller set of 12 instances, L 1 , to identify the three best algorithms. Then, the larger set L 2 was solved using only those three algorithms to further evaluate the general performance of these approaches.
The algorithms were first tested using the benchmark consisting of 12 OSCAR instances derived from 10 CAs taken from the literature. The values δ and Δ for these instances were fixed such that the size of the resulting array B would represent a possible new upper bound. In addition, these instances were created such that the size of the search space would permit us to solve them using all 12 algorithms; this was a concern because exact algorithms must explore all possible combinations of rows and columns, and therefore, if the search space is too large, they may require an excessive amount of time. Table 17 presents the results obtained using each algorithm based solely on the greedy approach (i.e., the algorithms GG CR , GG RC , and GGR C ) when solving the benchmark L 1 . Note that GG RC and GGR C show better performance than GG CR ; when all instances are considered, the former algorithms result in equal or fewer missing t-wise combinations compared with the latter. Therefore, the findings show that it is beneficial to eliminate rows before columns (as in GG RC and GGR C ) when working with initial matrices that are already CAs. This is because when rows are removed from the matrix, those that contribute the least to the CA are chosen for deletion, and the t-wise combinations that are lost as a result can subsequently be compensated for by eliminating the columns that produce them. Meanwhile, although the time performance Optimal shortening of uniform covering arrays of GGR C is superior to that of the others for this particular set of instances, it will worsen rapidly with increasing values of δ and Δ. In general, the time performance of GGR C will be the worst among the three greedy approaches, as indicated by the theoretical complexities presented alongside the definitions of these algorithms, mainly because of the greater number of calls to the greedy strategies for eliminating rows and columns. Table 18 shows the results obtained using the hybrid approaches that combine the greedy strategy with either the exact approach or the meta-heuristic approach (i.e., the algorithms GE, EG, and GM) when solving the benchmark L 1 . An increase in running time is observed for these approaches, mainly due to the use of the more elaborate strategies of the exact and metaheuristic algorithms. However, the results achieved also improve upon some of the results obtained by the solely greedy algorithms. All of these hybrid algorithms achieve the same results; however, the average time increase for GM is much greater than that for the algorithms that include exact strategies. Let's point out that the small amounts of times appearing in the exact approach are indeed a result from its expected theoretical behavior. Table 19 shows the results for another set of hybrid approaches, all involving the meta-heuristic strategy in combination with either the exact approach or the greedy approach (i.e., the algorithms MG, ME, and EM), when solving the benchmark L 1 . From these results and the previous ones shown in Table 18 for GM, it can be seen that the algorithms GM, MG, ME, and EM all find solutions with a comparable number of missing t-wise combinations to those in the solutions created by the other (exact or greedy) approaches. However, it should be noted that the initial matrices used in these algorithms were the best solutions obtained by a greedy algorithm, and in most cases, the differences in the number of missing t-wise combinations between these initial matrices and the results reported by the hybrid algorithms are nearly zero. These findings indicate that the contribution of these hybrid approaches is minimal. Table 20 shows the results of solving the benchmark L 1 using the algorithms MM, EE CR , and EE RC . Note that EE CR exhibits better performance than EE RC because there are more possible ways to select rows than columns. However, exhaustive search algorithms are impractical for finding a solution to an OSCAR instance except when the values of δ and Δ are both quite small. The exact algorithm EE CR is more suitable when N − δ > k − Δ since a greater portion of the search space is defined by N − δ; otherwise, EE RC behaves better. However, the execution time of the proposed exact algorithms grows with the desired degree of reduction for a given instance, and they can become infeasible.
Finally, some additional important observations are noted in the following. First, for every instance, the solutions obtained by the algorithms GE and EG have the same number of missing t-wise combinations. When N − δ > k − Δ, EG requires more time than GE; similarly, when N − δ < k − Δ, GE requires more time than EG. These findings suggest that GE is appropriate when N − δ > k − Δ and that EG is appropriate when N − δ < k − Δ.
Second, as δ and Δ increase for a given array A, the number of missing t-wise combinations produced by the pure greedy algorithms increases in comparison with the results of the hybrid algorithms that include exact strategies, i.e., EG and GE. For example, for the instances with the input array CA(255; 2, 18, 15), we note that the solution obtained by GGR C when δ = 3 and Δ = 1 has 267 missing t-wise combinations, whereas the solution obtained by GE has 257 missing t-wise combinations; similarly, when δ = 9 and Δ = 11, the solution obtained by GGR C has 65 missing t-wise combinations, whereas the solution obtained by GE has 61 missing t-wise combinations. It can be inferred that the inclusion of an exact strategy contributes to reducing Optimal shortening of uniform covering arrays the number of missing t-wise combinations, at the cost of an increase in the time required to build the matrix. Third, and most importantly, the algorithms that showed the best performance in the experiment were GGR C , GE and EG; all of them obtained comparable solutions, with only small differences in both quality and time cost. The algorithms that include meta-heuristic strategies consumed considerably more time but showed little difference in the quality of their solutions, whereas the exact approaches are too expensive for large values of δ and Δ.
To further evaluate the proposed approaches, the algorithms GGR C , GE and EG were used to solve the benchmark L 2 , which includes larger CAs. Table 21 summarizes the results obtained when solving L 2 . In addition to this experiment, an instance specified by the array A ¼ CAð136; 5; 68; 2Þ and values of δ = 2 and Δ = 33 was also solved using the meta-heuristic algorithm MM; this algorithm produced a solution B with zero missing t-wise combinations, meaning that the approach constructed a new CA of the form CA(134; 5,35,2). This last result serves as evidence that an approach based on seeking a solution to the OSCAR problem can also be used to construct CAs.

Performance comparison with state-of-the-art initialization algorithms
This subsection evaluates the performance of the proposed OSCAR approaches against the performance of several state-of-the-art initialization functions. For this purpose, the initialization functions described in the related work section are considered, and their results are compared with the best solutions obtained using the approaches proposed in this work. The performance comparison was performed as follows. The benchmark L 2 was chosen as the set of instances to be used in this evaluation. First, the OSCAR algorithms proposed in this work were used to solve the benchmark, and the best matrix B among all of the results was obtained for each instance. Then, the initialization functions, denoted by I i , were used to construct arrays S i of the same dimensions as the matrices derived by solving the OSCAR instances; i.e., for each instance, we constructed an array S with N − δ rows and k − Δ columns. Once all of the solutions generated by the OSCAR algorithms and the state-of-the-art initialization functions had been obtained, they were evaluated with regard to the function τ, i.e., the number of missing t-wise combinations in each newly constructed matrix. Table 22 summarizes the results of this experiment. Column one shows the identifier of each instance in L 2 , column two shows the number of missing t-wise combinations in the best solution obtained using the OSCAR approaches, and columns three to six present the numbers of missing t-wise combinations in the solutions derived using the state-of-the-art initialization functions I i . Note that for the last problem instance, one of the proposed OSCAR approaches (the metaheuristic algorithm MM) was able to construct a CA, as seen from the fact that the new matrix has zero missing t-wise combinations.
Performance comparison with IPOG-F, a state-of-the-art CA construction approach The experiment presented here involves the comparison of the GG RC and EG strategies against the state-of-the-art IPOG-F algorithm for CA construction. The goal in this experiment was to evaluate the performance when constructing CAs and/or quasi-CAs using IPOG-F, a fast greedy algorithm for CA construction that is widely used in the literature and is versatile in the sense that it can rapidly construct any type of CA. The GG RC and EG strategies are among the best of the proposed OSCAR solvers, as indicated by the experiments on the previous benchmarks. Both of these strategies were compared against IPOG-F in terms of the matrix construction time and the matrix quality (i.e., the number of missing t-combinations). Table 23 summarizes the results of this comparison on L 3 . Column 1 lists each set of instances in the benchmark. Columns 2 to 4 present the accumulated solution quality (i.e., the accumulated number of missing t-wise combinations) per set for each strategy. Columns 5 to 7 report the accumulated time per set and strategy. The experiment reported in this section was conducted to test IPOG-F as an approach for constructing CAs and/or quasi-CAs. The results shown in Table 23 reveal that the matrices constructed using EG RC have up to 90% fewer missing t-wise combinations than those constructed using IPOG-F. In addition, EG RC could generate CAs in 40 of the 820 OSCAR instances by reducing the number of missing t-wise combinations to 0, whereas IPOG-F failed to obtained any CA with the desired numbers of rows and columns. Finally, EG RC achieved better running times than IPOG-F for small values of v and t; however, the time performance of EG RC rapidly worsened with increasing values of the alphabet size and strength. By contrast, the GG RC strategy achieved time consumption results similar to those of IPOG-F while also improving the solution quality, making it a better choice than IPOG-F for the construction of quasi-CAs.

Applications of the proposed approaches for solving the OSCAR problem
In this subsection, we demonstrate that the matrices constructed by solving the OSCAR problem can be used as initial matrices for meta-heuristics for CA construction to assist in the construction of better matrices. For this purpose, the outputs of the initialization functions described in the related work section and the best solutions obtained using the proposed OSCAR approaches were used as the initial matrices for a meta-heuristic reported in [22]. Table 24 shows the new upper bounds for CAN(t, k, v) obtained using our proposed methodology. The second column shows the new CA bounds obtained when using the best matrices generated by the proposed OSCAR algorithms as the initial matrices for the meta-heuristic algorithm, and the third column shows the previous upper bounds for those CAs. The best produced solution for each specific instance of the OSCAR problem was used as the initial matrix for the meta-heuristic CA construction algorithm reported in [22], which is also based on the SA algorithm. Because of the small number of missing t-wise combinations in all of the produced initial matrices, the performance of the meta-heuristic algorithm was improved. The results define new upper bounds on CAN(t, k, v) for several CAs.

Conclusions
The present work has indirect implications for the interaction testing of software by aiding in the construction of tests of economical size (a feasible number of test cases). In particular, this paper presents and analyzes strategies for the construction of arrays with sufficiently few Optimal shortening of uniform covering arrays missing combinations to be considered quasi-CAs. Such arrays are constructed by solving the problem known as the Optimal Shortening of Covering ARrays (OSCAR) problem. The development of these strategies is motivated by the fact that the arrays thus produced can be used as excellent initialization matrices for algebraic or meta-heuristic approaches for the construction of CAs, which are mathematical objects that have broad applications in the testing of software components. This work presents an analysis of twelve different strategies for solving the OSCAR problem. Five of them correspond to greedy and exact approaches previously described in the literature, whereas the remaining seven algorithms are newly proposed here. The new approaches involve the use of simulated annealing and hybridization in their design. We note that this work also provides pseudocodes for the design of all presented algorithms, including, for the first time, the designs for the greedy approaches, which have been only briefly described in previous works. In addition, to test these strategies, three new OSCAR benchmarks with more than 1,000 instances have been designed, representing a considerable improvement over the previously reported 20-instance benchmark in terms of both size and variety in the values of the strength and alphabet size parameters, t and v, respectively.
The experimental design developed for the comparative analysis involved all three proposed benchmarks. The first benchmark, which consists of small instances, was solved using all twelve strategies: three greedy algorithms fGG CR ; GG RC ; GGR C g, two exact algorithms fEE CR ; EE RC g, one meta-heuristic algorithm fMMg, and six hybrid approaches fGE; EG; GM; MG; ME; EMg. Using this benchmark, the algorithms were compared in terms of running time and solution quality (measured as the number of missing t-wise combinations in each constructed array). As expected, the results showed that the greedy algorithms were the fastest, the exact algorithms yielded the best solutions, and the meta-heuristic provided a balance between quality and time. It was also observed that the solution quality of the pure greedy algorithms worsened with increasing instance size, but this situation could be addressed through the use of hybrid algorithms. The hybrid algorithms involving a mixture of greedy and exact approaches had higher running times but also higher solution quality. The first experiment indicated that hybrid algorithms involving a mixture of the meta-heuristic and greedy strategies are a viable alternative. Such strategies had somewhat higher running times for array construction but resulted in fewer missing t-wise combinations than the hybrid greedy approaches, mainly when the numbers of rows and columns to be deleted were high. In terms of solution quality, the experimental results indicated that the best algorithms were GE and EG because they yielded solutions with as few missing t-wise combinations as the exact approaches EE RC and EE CR but in less time. In terms of running time, the experiment indicated that the best algorithms were GGR C and GG RC because they were faster than any other algorithms while maintaining an acceptable solution quality; however, we note that for larger instances, the time performance of GGR C will be worse than that of GG RC because it is more strongly affected by the instance size and the numbers of rows and columns to be removed. The second benchmark was used to perform an in-depth analysis of some of the best strategies, namely, GGR C , GE and EG. This experiment tested the performance of these algorithms on larger OSCAR instances to yield a better understanding of their behavior. The best solutions were still produced by the hybrid greedy-exact approaches GE and EG, but in the latter approach the time increased exponentially. By contrast, the pure greedy algorithm GGR C   Table 23. Summary of the results of evaluating the performance of algorithms E 1 = IPOG-F, E 2 ¼ GG RC , and E 3 ¼ EG 3 , over the benchmark L 3 . The performance is measured in the missing t-wise combinations, and in the time (in seconds) spent to find it.

Instance set
Missings Time Optimal shortening of uniform covering arrays continued to be fast, and its solutions only slightly deviated from those of the hybrid algorithms. After this analysis, the same benchmark was used to compared the best results from these approaches against the initialization functions generated using state-of-the-art methods.
The experimental results showed that in all instances, the number of missing t-wise combinations was reduced by approximately 90% in the matrices constructed using the proposed approach in comparison with those taken from the literature. Finally, an experiment was conducted using the third benchmark to test IPOG-F as an approach for constructing CAs and/or quasi-CAs. The results revealed that with EG RC , the number of missing t-wise combinations was reduced by up to 90% compared with IPOG-F. Moreover, it was found that EG RC could obtain CAs in 40 of the 820 OSCAR instances by reducing the number of missing t-wise combinations to 0, whereas IPOG-F failed to obtain any CA with the desired numbers of rows and columns. Finally, it was observed that the running time of EG RC was better than that of IPOG-F for small values of v and t but worsened rapidly with increasing values of the alphabet size and strength. By contrast, the GG RC strategy achieved running times similar to those of IPOG-F while also improving the solution quality, making it a better choice than IPOG-F for the construction of quasi-CAs. A major drawback of some of the proposed approaches (with the exception of the greedy ones) is the time consumed to solve the problem, which increases with the numbers of rows and columns to be eliminated. Moreover, the experimental design could be improved to test a wider range of possible values to adjust the meta-heuristic and investigate a wider number of strategies. The ranges of values of the alphabet size and strength parameters should be extended to further probe the resulting changes in performance of the different strategies. Future work should also address the lack of an in-depth analysis of the use of the meta-heuristic approach to properly characterize its region of importance. In general, a more extensive characterization study could provide better insight into the behavior of these strategies, and this remains as future work.