2.1 Previous work on RALSP

Research on the RALSP focuses on flexibility improvement, multi-objective optimization, and adaptation to dynamic environments. Among pioneering works, Koren, Heisel [2] defined the RMS paradigm using modular machine tools and open-architecture controllers. After that, researchers have expanded the research on RMS in various dimensions. Specifically, Hasan, Jain [4] proposed a “Service Level” index to quantify reconfiguration effort. Colledani, Gyulai [10] developed an integrated method for system design and reconfiguration planning under uncertain demand. Dou, Li [11] adopted the NSGA-II for bi-objective optimization of cost and tardiness in reconfigurable flow lines.

For multi-objective optimization of RALSP, Goyal and Jain [12] combined MOPSO with maximum deviation theory. Yuan, Deng [13] considered cloud manufacturing and used distance-sorting PSO. And Prasad and Jayswal [14] incorporated reconfiguration effort, profit-to-cost ratio, and due dates using Shannon entropy. For modular products and dynamic environments, Pattanaik and Jena [15] used a Pareto-based heuristic. Yuan, Yu [16] designed a memetic algorithm. And Yang, Liu [17] used hybrid PSO for assembly sequence and equipment selection. Meanwhile, some attention is paid to reconfiguration efficiency and sustainability. Specifically, Yelles-Chaouche, Gurevsky [18] formulated MILP for task reassignment cost. Tremblet, Yelles-Chaouche [19] considered uncertain product arrival. Delorme and Gianessi [20] minimized power peaks. In addition, Gholami, Delorme [21] used fuzzy logic for sustainable supply chains.

As the application of hyper-heuristics to RALSP is limited, this section reviews the integration of hyper-heuristics and reinforcement learning in the related scheduling problems. Cano-Belmán, Ríos-Mercado [22] developed a scatter search-based hyper-heuristic for mixed-model sequencing. Mosadegh, Fatemi Ghomi [23] developed Q‑learning‑based simulated annealing. Afterwards, Özbakır and Seçme [24] proposed a hyper-heuristic for stochastic parallel lines. And Zhou and Zhao [25] developed a hyper-heuristic to optimize the material feeding. Guo, Liu [26] proposed the hyper-heuristic for integrated process planning and scheduling. Lu, Gao [27] solved PCB assembly line scheduling problems with a hyper-heuristic optimizer. Reviews on reinforcement learning-based hyper-heuristics and scheduling applications can be found in Li, Wei [28] and Vela, Valencia-Rivera [29].

Apart from the integration of hyper-heuristics and reinforcement learning, the integration of reinforcement learning (especially Q-learning) with metaheuristic algorithms has produced promising performance through adaptive scheduling. For instance, Yan and Wang [30] proposed a double-layer Q-learning algorithm in aircraft assembly scheduling, and Zhang, Tang [31] developed a Q-learning-based multi-objective evolutionary algorithm in assembly line balancing. In recent years, Meng, Li [32] proposed a Q-learning-inspired differential evolution algorithm for mixed-model assembly line balancing and sequencing. Rauf, Mumtaz [33] developed a multi-objective intelligent hybrid genetic algorithm integrated with Q-learning-based parametric tuning, and Wen, Liu [34] developed PSO with Q-learning for aircraft pulsating assembly line scheduling. For broader reviews, see Bortolini, Galizia [5] on RMS, Gianassi, Leoni [1] on resource management in mixed-model and multi-model assembly lines, and Battaïa, Delorme [35] on line balancing and model sequencing.

2.2 Research gap and contributions

Through the literature review, several research gaps can be identified. Firstly, the existing mathematical models for RALSP might have the following drawbacks: nonlinearities and logical inconsistencies. They are usually developed for describing the problem, and they cannot be solved by the standard optimization tools. Secondly, while some metaheuristics have been developed, there lacks the studies on the intelligent and adaptive hyper-heuristic frameworks, which could dynamically leverage the strengths of multiple metaheuristics for a special multi-objective RALSP. Thirdly, the integration of machine learning (e.g., Q-learning) into metaheuristic or hyper-heuristics for adaptive operator and parameter selection still needs more attention. To address these gaps, the main contributions of this paper are as follows.

1. 1) Formulation of a novel and tractable multi-objective mathematical model: This study develops three fully linearized MILP sub‑models—one for each objective: reconfiguration cost, workload equalization, and logistics leveling. This formulation corrects flaws in earlier models and is directly solvable by commercial solvers, providing exact benchmarks for small to medium instances.
2. 2) Development of one new and effective hyper-heuristic: The proposed QMOHH selects one promising metaheuristic operator from eight metaheuristic operators with the Q-learning. Meanwhile, it utilizes a density-aware leader selection strategy with a survival-time decay factor to select the global best solution for population evolution. The density-aware leader selection strategy provides more computation effort to the isolated Pareto solutions and ensures that each solution will be explored in iterations, hence ensuring an effective balance between exploration and exploitation.
3. 3) Comprehensive experimental study and validation: This study conducts a comprehensive evaluation of the QMOHH utilizing 120 generated benchmark instances. And QMOHH is compared with nine other popular multi‑objective algorithms. Computational results along with statistical analysis demonstrate the superiority of the proposed QMOHH in convergence, diversity, and overall solution quality.

The proposed QMOHH differs from existing reinforcement-learning-assisted metaheuristics in three key aspects. 1) Most studies embed Q-learning into a single metaheuristic (e.g., Mosadegh, Fatemi Ghomi [23]; Zhang, Tang [31]). QMOHH instead acts as a hyper-heuristic that selects among eight complete operators from four metaheuristics (PSO, TLBO, WOA, GWO), leveraging diverse search behaviors within a unified framework. 2) Existing approaches often use algorithm-specific states (e.g., diversity, temperature) and scalar fitness rewards. QMOHH defines the state purely by search progress and the reward as the number of newly added non-dominated solutions, directly aligning adaptation with multi-objective optimization goals. 3) Unlike crowding-distance-based or random selection, QMOHH introduces a survival-time decay factor that reduces the attractiveness of long-surviving solutions, preventing stagnation and promoting exploration in sparse regions.

3.1 Problem description

In practice, assembly lines are primarily categorized into three types, as illustrated in Fig 2. Single-model assembly lines are dedicated to the high-volume production of a single product type, where no sequencing problem arises due to fixed product variety, and the primary optimization objective is line balancing—assigning assembly tasks to stations based on one or several criteria. Mixed-model assembly lines assemble a family of products that are similar in structure, function, and assembly methods to meet multi-variety, small-batch demands, requiring simultaneous consideration of both line balancing and product sequencing. Reconfigurable assembly lines, designed for customized and small-batch production, sequence a set of structurally similar but distinct products by rapidly switching between them via hardware and software adjustments. In most studies, line balancing in a reconfigurable assembly line is assumed to be predetermined, so optimization focuses on product sequencing with objectives of minimizing reconfiguration cost, equalizing production workload, and leveling material logistics.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Production scheduling characteristics of the three assembly line types.

https://doi.org/10.1371/journal.pone.0348884.g002

The RALSP addressed in this study involves sequencing product types from the same family. The demand for the th product type is , and the total demand is . The number of stations and the processing time for each product at each station are predetermined and fixed. To simplify the problem, a minimum production cycle is often used in the literature [13]. Here, the original demand is replaced by , where is the greatest common divisor of the demands for all product types. Consequently, the number of products to be sequenced is reduced to . For example, if the demands for four products (A, B, C, D) are 200, 200, 300, and 100, respectively, and their greatest common divisor is 100, the minimum production cycle is calculated as [1–3].

Before presenting the mathematical formulation, the following basic assumptions are presented: 1) the number of product types and stations is fixed and known; 2) the reconfiguration cost for switching between any two product types is known in advance; 3) the part requirements for each product type and the associated part-frequency constraints are known; 4) the processing time for each product type at each station and the constant interval time between successive product startups are known; 5) all stations are utilized for assembling every product, with the assembly process for each starting at the first station and finishing at the last; 6) and the transportation time for moving products between stations is negligible.

3.2 Mathematical model

This section presents the complete and tractable mathematical formulation for the RALSP. A key contribution of this study is the development of three fully linearized MILP models, each dedicated to one of the three conflicting objectives: reconfiguration cost minimization, production workload equalization, and logistics leveling. Unlike previous studies that often present conceptual or nonlinear formulations that are challenging to solve directly, the developed models are designed for solvability by standard MILP solvers (e.g., CPLEX, Gurobi), providing exact benchmarks and enabling practical deployment for small-to-medium instances. The parameters and variables are first described below.

Download:

• PNG
  larger image
• TIFF
  original image

https://doi.org/10.1371/journal.pone.0348884.t008

The first model (Model 1) addresses the objective of minimizing the total reconfiguration cost incurred when switching between different product types on the assembly line. In multi-model assembly lines, reconfiguration costs are high and cannot be neglected. Reconfiguration costs consist of expenses caused by changing hardware components, changing tools, reprogramming software, and downtime during changeovers. The minimization of reconfiguration cost is crucial to maintain the production efficiency and the profit of the assembly line. The decision variables for Model 1 are first introduced as follows.

Download:

• PNG
  larger image
• TIFF
  original image

https://doi.org/10.1371/journal.pone.0348884.t009

Model 1 minimizes total reconfiguration cost. Equation (1) sums the costs of transitions between consecutive products, including from the last to the first. Constraints (2) and (3) enforce that each position has exactly one product and that demand is satisfied. Constraints (4)–(6) linearize the product-of-binary terms for consecutive positions using auxiliary variables ; constraints (7)–(9) handle the cyclic transition. The complete linearization is further discussed in Section 3.3.

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

In assembly lines, a steady supply of parts to stations is important to prevent bottlenecks, avoid shortages, and reduce in-process inventories. The second mode (Model 2) takes the supply of parts to stations into account by optimizing workload equalization, which is achieved by minimizing violations of part-frequency constraints. Here, part-frequency constraints denote that, for one sliding window of consecutive product positions, there are at most products requiring part within the consecutive positions. The violations of part-frequency constraints could disrupt material flow, increase logistics complexity, and result in uneven station loads. Hence, the minimization of violations of part-frequency constraints is important in real applications to promote smoother production and better resource utilization. The decision variables utilized in Model 2 are defined below.

Download:

• PNG
  larger image
• TIFF
  original image

https://doi.org/10.1371/journal.pone.0348884.t010

Model 2 minimizes violations of part-frequency constraints. Equation (11) sums all violations. Constraints (12)–(13) are the same assignment and demand constraints as in Model 1. Equation (14) computes part usage within sliding windows using modulo arithmetic. Constraints (15)–(16) use the big-M method to detect violations: if usage exceeds , then . The linearization is detailed in Section 3.3.

(11)

(12)

(13)

(14)

(15)

(16)

(17)

In mixed-model and reconfigurable lines, it is often desirable to have a constant or nearly constant rate of output for each product variant (a concept known as “rate smoothing” or “level scheduling”). This constant rate aims at stabilizing downstream processes, including part supply, material handling, and delivery schedules. The third mode (Model 3) considers this situation to optimize logistics leveling, which is achieved by minimizing the deviation of actual cumulative production from an ideal and evenly-spaced production rate for each product type. Model 3 quantifies and minimizes these deviations, and the decision variables for Model 3 are introduced as follows.

Download:

• PNG
  larger image
• TIFF
  original image

https://doi.org/10.1371/journal.pone.0348884.t011

Model 3 minimizes logistics leveling by reducing deviation from ideal production rates. Equation (18) linearizes the absolute deviation. Constraints (19)–(20) are the assignment and demand constraints. Equations (21)–(24) compute station completion times with inter-station intervals. Constraints (25)–(26) calculate cumulative counts. Equation (27) defines makespan . Constraint (28) contains a bilinear term , which is handled by a two-stage ε-constraint (see Section 3.3). Equation (29) linearizes the deviation using positive and negative slack variables. Notice that constraint (28), , contains a bilinear term (), rendering the model a bilinear program. To ensure exact solvability via standard MILP solvers and to establish a reliable benchmark, this study implements an -constraint method in two sequential linear stages. Stage 1 minimizes the total completion time subject to constraints (19)–(27) and the original assignment constraints, obtaining its optimal value . Afterwards, fixing in constraint (28) (which then becomes linear), Stage 2 solves the resulting pure MILP to minimize the total absolute deviation (objective [18]). This decomposition guarantees that both sub-models are linear and can be solved to optimality. The two-stage procedure ensures that the obtained solution is Pareto-optimal with respect to the third objective, while maintaining linearity in each stage.

(18)

(19)

(20)

(21)

(22)

(23)

(24)

(25)

(26)

(27)

(28)

(29)

(30)

3.3 Analysis of the formulated models

The three MILP sub-models presented in Section 3.2 are designed to be directly solvable by standard commercial solvers, thereby providing exact benchmarks for small-to-medium instances. To highlight the advancements achieved, this section compares the proposed formulations with prior work, particularly the models introduced by Yuan, Deng [13].

The formulations in Yuan, Deng [13] serve as a conceptual framework for the RALSP but suffer from fundamental limitations that prevent direct solution by MILP solvers. 1) Their workload equalization model defines the objective via a conditional rule if , which cannot be expressed as linear constraints without auxiliary binary variables and big-M transformations. 2) Their delayed workload model involves max() functions in constraints (8) and (9), which are inherently nonlinear and non-convex and cannot be handled by MILP solvers without omitted linearization steps. 3) Moreover, the three sub-models use inconsistent variable definitions and constraint structures, making it impossible to combine them into a single multi-objective MILP model that can be solved simultaneously. In contrast, the models in this study share a common variable set and can be jointly addressed through -constraint or weighted-sum methods.

Consequently, while the earlier work provides valuable insights into the problem structure, it does not yield a solvable MILP formulation. The models proposed in this study overcome these limitations through complete linearization strategies: Model 1 linearizes product-of-binary terms using auxiliary variables and linear constraints (4)–(9); Model 2 encodes part-frequency constraint violations using the big-M method, transforming the original logical condition into linear inequalities (15)–(16); and Model 3 handles the bilinear term in constraint (28) via a two-stage -constraint approach, while linearizing absolute deviations using positive and negative slack variables. These linearizations ensure that all three models are directly solvable by commercial solvers such as CPLEX and Gurobi, providing exact optimal solutions for small-to-medium instances and reliable benchmarks for evaluating heuristic algorithms. For practical multi-objective optimization, the linear models can be combined using weighted-sum or -constraint methods, maintaining computational tractability.

4.1 Main procedure of QMOHH

The framework of the proposed QMOHH is outlined in Algorithm 1. QMOHH begins by initializing a population, along with a Pareto archive for maintaining non-dominated solutions. A Q-learning agent is also initialized to guide operator selection.

During each iteration, the algorithm determines its current state based on evaluation progress and uses an ε-greedy policy to select one of the eight heuristic operators. If required, a global best solution is chosen from the archive using the new density-aware leader selection strategy. The selected operator is employed to generate new solutions, and subsequently, the new solutions are evaluated. Afterwards, the personal best solutions and Pareto archive are updated when necessary. A reward—based on the number of newly added non-dominated solutions—is computed and used to update the Q-table. This main loop is repeated until the termination criterion (a maximum number of evaluations in this study). After completing the main loop, the Pareto archive is returned as the achieved approximate Pareto-optimal set by QMOHH. With the Q-learning and new density-aware leader selection strategy, QMOHH achieves the proper balance between exploration and exploitation throughout the search process.

Algorithm 1. Proposed Q-learning-based multi-objective hyper-heuristic

Input: Instance data, parameters (population size N and maximum evaluations)

Output: Achieved an approximate Pareto archive

1: Initialize population P randomly under the demand constraints;

2: Evaluate the initial population, where the objective values for each solution are calculated;

3: Update Pareto archive with the initial population;

4: Update personal best solutions (P_best) with P_best ← P;

5: Create states, actions, and a Q-table to initialize the Q-learning agent;

6: Achieve current state based on the current progress (evaluations/max_evaluations);

7: While evaluations < max_evaluations do

8:  Achieve the current state based on evaluations/max_evaluations;

9:  Achieve the selected operator for population evolution with Q-learning;

10:  If the selected operator requires a global leader then

11:   Achieve global best with the density-aware leader selection strategy from the Pareto archive;

12:  End-if

13:  Obtain a new population (P_new) with the selected operator;

14:  Evaluate the new population and update the evaluation count;

15:  Update personal best solutions (P_best) if the new solution dominates the current solution;

16:  Update Pareto archive with non-dominated solutions from P_new;

17:  reward ←the number of newly added non-dominated solutions;

18:  next_state ← determine next state based on the evaluation progress;

19:  update Q_table;

20:  P ← select the N solutions for P and P_new;

21:  current_state ← next_state;

22: End-while

23: Return Pareto archive// Final Pareto-optimal solution set

4.2 Solution presentation

A product sequence of length is necessary for RALSP to achieve objective values. In product sequence, each position corresponds to a product. Here, demand constraints for each product type must be satisfied in this product sequence. Suppose that there are four product types (A, B, C, D) with demands [1–3]. One feasible sequence is [A, B, C, A, C, D, B, C].

The proposed QMOHH utilizes a string of floating-point numbers with the size of for encoding. Based on the floating-point numbers, the random-key method is applied to achieve a feasible product sequence. Fig 3 provides an example solution presentation, where there are four product types (A, B, C, D) with a minimum production cycle of [1–3]. Eight floating-point numbers are randomly generated within the range [0.00, 1.00] for encoding, and the original task sequence is [A, A, B, B, C, C, C, D]. After sorting the floating-point numbers in ascending order, the final product sequence could be achieved, namely [B, C, C, B, A, A, D, C].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Solution presentation.

https://doi.org/10.1371/journal.pone.0348884.g003

Once the product sequence is determined by sorting the floating-point numbers in encoding, three objectives could be calculated directly on the basis of the three linearized MILP models in Section 3.2, respectively. Specifically, reconfiguration cost is achieved by summing the cost incurred between consecutive product pairs in the task sequence. Production workload equalization is calculated by summing the violations of the part-frequency constraints along the product sequence. Logistics leveling is calculated by summing the deviation of the actual cumulative production from the ideal, evenly-spaced production rate for each product type. Clearly, the demand constraints for each product type are satisfied for any string of floating-point numbers with the size of This floating-point encoding scheme ensures the generation of feasible sequences throughout the search process and facilitates the application of the various metaheuristic operators described in Section 4.3.

4.3 Metaheuristic operators

The proposed QMOHH employs eight heuristic operators derived from four well-established metaheuristics. Each heuristic operator is adapted to modify the floating-point numbers in encoding, aiming at obtaining diverse and feasible product sequences.

PSO-based operators: Three operators are adapted from PSO [36]. These include the first operator with basic position update, the second operator combining position update and uniform mutation to emphasize exploration, and the third operator combining position update and non-uniform mutation. The third operator utilizes the non-uniform mutation with time-decreasing strength to achieve phased refinement.

TLBO-based operators: Three operators are derived from TLBO [37]. These include the teacher-phase operator simulating a teacher guiding the students, the learning-phase operator simulating peer-to-peer learning among students, and the mixed TLBO phase combining the teacher phase and learning phase to integrate global and local search.

WOA-based operator: This operator implements the three characteristic behaviors of WOA—encircling prey, bubble-net attacking, and random search—probabilistically [38]. Adaptive parameters shift the emphasis from exploration to exploitation throughout the optimization process.

GWO-based operator: Based on the GWO [39], this operator employs a hierarchy of three leading solutions (alpha, beta, delta) selected from the archive to guide population updates. The implementation of three leaders emphasizes the search around the space between Pareto solutions.

Notice that the global best solution is necessary for all the heuristic operators except for the learning-phase operator in multi-objective optimization, and the new density-aware leader selection strategy could be utilized to obtain the global best solution in iterations. All operators are utilized to modify the continuous encoding scheme. And any new continuous encoding scheme could be transferred into a feasible product sequence utilizing the decoding in Section 4.2, and hence, the demand constraints are satisfied.

Among the eight heuristic operators, some operators emphasize global exploration while others emphasize local refinement. They provide the Q-learning mechanism with a rich set of strategies to dynamically select from, and hence enable effective adaptation to the evolving solution space.

4.4 Q-Learning mechanism for operator selection

Q-learning is a model-free, value-based reinforcement learning algorithm that has been widely utilized in the literature. Q-learning operates by estimating the expected cumulative reward (Q-value) for taking a specific action in a given state [31]. The algorithm has a Q-table where each entry contains state and action indicating the expected utility of choosing action in state . The Q-values are updated using the expression [31] in iterations, where is the learning rate, is the discount factor, is the immediate reward received after taking action in state , and is the next state.

(31)

In the proposed QMOHH algorithm, Q-learning is responsible for selecting one of eight heuristic operators at each iteration adaptively. Q-learning starts with determining state representation, action set, and reward function, and in the main loop, Q-learning conducts the action selection and policy update iteratively.

State representation: The state is defined based on the progress of the optimization process. The algorithm’s evaluation progress, calculated as the ratio of completed evaluations to the maximum allowed evaluations , is discretized into ten intervals (0.0 to 0.9). This discretization creates a state space that effectively captures the different stages of the search, allowing the Q-learning policy to be phase-dependent. Unlike approaches that rely on problem-specific metrics (e.g., population diversity, temperature), this state definition is problem-independent and requires no instance-specific tuning, facilitating generalization to other scheduling problems.

Action set: The action set contains eight elements corresponding to eight heuristic operators in Section 4.3, including three PSO-based operators, three TLBO-based operators, one WOA-based operator, and one GWO-based operator.

Reward function: After applying the selected operator and evaluating the new populations, the reward is calculated as the number of new non-dominated solutions added to the external Pareto archive from the newly generated population. This design directly aligns the Q-learning objective with the goal of multi-objective optimization—expanding the Pareto front—rather than relying on scalar fitness improvements commonly used in single-objective or parameter-tuning contexts.

Action selection and policy update: At each iteration, the current state is determined, and later one heuristic operator is selected based on the Q-table utilizing ε-greedy policy. After generating the new population with the selected heuristic operator, the new population is evaluated, and the reward is computed. Later on, the Q-table is updated using the standard Q-learning update rule based on the reward.

4.5 New density-aware leader selection strategy

A global best solution (gbest) is necessary for population evolution in iterations, and the method of selecting gbest from the Pareto archive is crucial for guiding the search direction. Different from the methods in Coello, Pulido [36], the proposed QMOHH employs a new density-aware leader selection strategy that incorporates a survival-time decay factor. The strategy calculates a modified crowding distance for each archive solution as , where is the original crowding distance in Deb, Pratap [40], is the solution’s selection time in iterations, and is a decay rate (e.g., 0.5). This formula reduces the attractiveness of long-surviving solutions, even if they reside in sparse regions. The survival-time decay factor can be intuitively understood as a “freshness” penalty: a solution that has been selected as the global leader many times becomes progressively less likely to be chosen again, giving newer or less frequently selected solutions an opportunity to guide the population. This mechanism prevents the algorithm from getting trapped around a few dominant individuals and promotes continuous exploration of sparser regions of the Pareto front. Hence, this new density-aware leader selection strategy encourages exploration of new areas and prevents search stagnation. The leader selection process is illustrated in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Flowchart of the enhanced global best (gbest) selection process.

https://doi.org/10.1371/journal.pone.0348884.g004

The survival-time decay factor reduces the selection probability of solutions that have been repeatedly chosen as leaders, ensuring a balanced exploration of the Pareto front. This mechanism ensures diversity-driven exploration by consistently favoring leaders from less crowded regions of the Pareto front, while the decay factor automatically balances this with the need to exploit known high-quality areas over time.

6.1 Experimental design

To evaluate the proposed QMOHH, 120 test instances are generated based on realistic production scenarios. Each instance consists of four product types. The minimum production cycle (product demands) is defined using 20 pre-determined patterns, with total demand per cycle ranging from 10 to 29. Each pattern satisfies that the greatest common divisor of the four demands is 1, ensuring that no further reduction is needed. The number of stations takes six fixed values: 5, 10, 15, 20, 25, and 30, each producing 20 instances. The processing time for each product at a given station is uniformly generated between 6 and 25 time-units. Reconfiguration costs are generated as an matrix with zero diagonal and off-diagonal entries uniformly drawn from 0 to 12 (in thousand CNY). Part requirements are generated by first creating a random integer between 0 and 8 for each product-part pair; the binary requirement is set to 1 if this integer > 0, otherwise 0. The number of part types per instance is randomly set between 12 and 16. For part-frequency constraints, the window length is randomly selected from and the maximum allowable count is randomly chosen between 1 and . The interval time between successive product startups is fixed to 5 time-units. Instances are classified by scale according to the number of stations: instances with 5 or 10 stations are categorized as small-scale; those with 15 or 20 stations as medium-scale; and those with 25 or 30 stations as large-scale. This instance set comprises 40 small-scale, 40 medium-scale, and 40 large-scale instances, for a total of 120 instances. All instances, along with the source code of all algorithms, are publicly available at https://github.com/zixiangliwust/Instances_RALSP under the MIT license.

Three indicators are used to evaluate the Pareto solution sets: GD (generational distance) measures convergence to the reference front; (ε-indicator) quantifies the smallest factor needed for the obtained set to dominate the reference; and 1-NHV (complement of normalized hypervolume) assesses both convergence and diversity. Smaller values indicate better performance for all indicators. These indicators have been widely applied in the literature [41,42] and detailed calculation procedures are available in the source code on GitHub (see Data Availability).

To evaluate the performance of the proposed QMOHH, it is compared with nine other multi-objective algorithms, including multi-objective simulated annealing algorithm (MOSA) [43], multi-objective restarted simulated annealing (MRSA) [41], multi-objective artificial bee colony algorithm (MOABC) [44], improved multi-objective artificial bee colony (IMOABC) [42], GDE3 [45], MOEAD [46], NSGA-II [40], multi-objective PSO algorithm (MOPSO) [36], a new PSO-based metaheuristic (SMPSO) [47].

The termination criterion for all tested algorithms is a total of 100,000 evaluations. The proposed linearized MILP models are solved using the IBM ILOG CPLEX Optimization Studio (version 22.1.1) solver. And all tested algorithms are implemented in Python 3.10 on a personal computer equipped with an Intel(R) Core (TM) Ultra 9 185H, with parallel computation employed to accelerate the experiments. For each independent run, the random seed is set to the current system time to ensure statistical independence across the 10 runs per instance.

All algorithms are calibrated using the Taguchi experimental design method to ensure fair comparison. A set of 30 representative instances (covering small, medium, and large scales) is used for calibration. For each algorithm, an orthogonal array is employed to evaluate five key parameters at four levels. The response metric is the average 1-NHV over 10 independent runs. The optimal parameter combination for each algorithm is selected based on the signal-to-noise ratio. The final calibrated parameters for QMOHH are: population size , learning rate , discount factor , initial ε-greedy exploration probability (decaying linearly to 0.1), survival-time decay factor , uniform and non-uniform mutation probabilities with perturbation 0.5 and termination at evaluations. These values are explicitly defined in the source code, which is publicly available at https://github.com/zixiangliwust/Instances_RALSP. The same calibration procedure is applied to the nine compared algorithms; their final parameter settings can also be found in the code.

6.2 Evaluating the MILP model with ε-constraint method

The results in Table 5 illustrate the computational behavior of the MILP models under the ε-constraint method with a time limit of 300s per subproblem. Here, total time(s) reports the total computation time. From this table, it is observed that, as the number of stations () or the total number of products () increases, the total runtime rises substantially. For small instances (e.g., , ), all 147 ε-combinations are solved to optimality within 20s, yielding 40 valid solutions and 3 Pareto points. In contrast, for the largest instance (, ), the total runtime exceeds 7888s, and only 5 out of 147 subproblems reach optimality. Instances with larger (i.e., longer product sequences) consistently require longer solution times, even when the number of stations remains moderate. For instance, with , increasing from 10 to 29 raises the total runtime from 23s to nearly 600s, while the number of optimal subproblems drops from 36 to 10.

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Computational results of the ε-constraint method on selected instances.

https://doi.org/10.1371/journal.pone.0348884.t005

The computational cost becomes prohibitive when both and exceed moderate values. Consequently, the MILP models are best suited for small-scale instances (e.g., , ) where exact Pareto frontiers can be obtained within acceptable time. For larger problems, the use of metaheuristics such as QMOHH is essential. Moreover, finer ε-grids (e.g., smaller interval steps) would further increase the number of subproblems, making the approach even more computationally demanding.

These findings confirm that the developed MILP formulations provide exact benchmarks for small-to-medium instances, while larger-scale problems require efficient meta-heuristic methods.

6.3 Evaluating the improvements of QMOHH

To validate the contribution of each proposed component, the complete QMOHH is compared against four variants, as defined below. All four variant algorithms, along with the complete QMOHH, are evaluated on the 120 benchmark instances described in Section 6.1, following the same experimental protocol.

1. MOHH-v1: Only three PSO-based operators are utilized. It is utilized to establish a PSO-centric baseline to assess the value of the utilization of eight heuristic operators.
2. MOHH-v2: Q-learning mechanism is disabled, and heuristic operators are randomly selected. It is utilized to assess the value of adaptive operator selection with the Q-learning mechanism.
3. MOHH-v3: Q-learning controller is replaced with a Deep Q-Network (DQN). It is utilized to examine the impact of a different reinforcement learning architecture.
4. MOHH-v4: The standard method to select the global best solution from MOPSO is utilized to replace the new density-aware leader selection strategy. It is utilized to test the advantage of the new density-aware leader selection strategy.

In this section, the 1-NHV indicator is utilized to measure the performance of these five algorithms, where the smaller value of 1-NHV denotes the better overall solution quality. The average 1-NHV values over the 120 instances are presented in Table 6. The Friedman test is conducted on the mean ranks of the algorithms for each performance indicator. Fig 6 depicts the means plot of the average ranks of MOHH methods in terms of the 1-NHV indicator.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Average 1-NHV values of QMOHH and its variants.

https://doi.org/10.1371/journal.pone.0348884.t006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Means plot of the average ranks and 95% confidence intervals of MOHH methods in terms of 1-NHV.

https://doi.org/10.1371/journal.pone.0348884.g006

The results in Table 6 verify the clear superiority of QMOHH, which achieves the lowest average 1-NHV of 0.232. MOHH-v1 achieves worse performance (0.290) compared to QMOHH, underscoring the benefit of integrating a diverse set of metaheuristic operators. Again, MOHH-v2 achieves worse performance (0.248) compared to QMOHH, indicating that disabling the Q-learning mechanism leads to a noticeable performance degradation. This finding highlights the critical role of adaptive operator selection with the Q-learning mechanism. Notably, MOHH-v3 exhibits the poorest performance among the compared methods, suggesting that the Q-learning framework is more suitable and efficient than a more complex deep reinforcement learning approach for online heuristic operator selection. MOHH-v4 obtains worse performance (0.242) compared to QMOHH, confirming the effectiveness of the new density-aware leader selection strategy.

This ablation study conclusively validates that each core component of QMOHH—the Q-learning-based adaptive operator selection, the ensemble of diverse heuristics, and the new density-aware leader selection strategy—contributes uniquely and essentially to its superior search capability and final solution quality for the multi-objective RALSP.

6.4 Comparative study

To comprehensively evaluate the performance of the proposed algorithm, QMOHH is compared with nine other state-of-the-art multi-objective algorithms in Section 6.1. All algorithms are evaluated using the same 120 benchmark instances described in Section 6.1, under the identical termination criterion. Each algorithm is executed 10 times on each instance. The parameter settings for each algorithm are determined through the Taguchi calibration widely utilized in published papers.

Table 7 provides the average 1-NHV values from 10 independent runs across 120 instances. For space reasons, only the results of the best four algorithms are presented here. Detailed results for all evaluation indicators are omitted due to space constraints, but are available upon request. As the smaller 1-NHV value indicates the better algorithm performance, the algorithms can be ranked in increasing order of the overall average 1-NHV values presented in Table 7. Namely, QMOHH ranks first as it achieves the best overall average value (0.198), demonstrating the superiority of QMOHH over all other compared algorithms. MOPSO ranks second with 0.222, SMPSO ranks third with 0.236, and MOEAD ranks fourth with 0.287. Among the methods, MOSA and MRSA exhibit the weakest performance.

Download:

• PNG
  larger image
• TIFF
  original image

Table 7. Results of the average 1-NHV by the four best-performing algorithms.

https://doi.org/10.1371/journal.pone.0348884.t007

This ranking clearly shows that QMOHH holds a clear advantage over the compared methods in terms of 1-NHV. In fact, the superior performance of QMOHH is attributed to adaptive operator selection with the Q-learning mechanism and the new density-aware leader selection strategy. These improvements together are in favor of QMOHH achieving a proper balance between exploration and exploitation, and hence QMOHH produces outstanding performance for the multi-objective RALSP.

To validate the performance differences statistically, this section conducts the non-parametric statistical test due to the violation of normality assumptions required by ANOVA. The Friedman test is conducted separately on the mean ranks of the algorithms for each performance indicator (GD, , and 1-NHV). The results of the Friedman test reject the null hypothesis (p-value < 0.01) for all three indicators, confirming that there exist statistically significant differences among the algorithms’ performances.

Fig 7 illustrates the means plot for the average ranks of the algorithms in terms of GD. Clearly, QMOHH achieves the lowest average rank, indicating its superior performance in convergence. Except for MOPSO, confidence intervals of QMOHH and the other eight algorithms do not overlap, and hence it is safe to say that QMOHH outperforms the other eight algorithms statistically in terms of GD. In short, QMOHH obtains the lowest average rank in terms of GD in this statistical analysis.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Means plot of the average ranks and 95% confidence intervals of the algorithms in terms of GD.

https://doi.org/10.1371/journal.pone.0348884.g007

Similarly, Fig 8 presents the results for the indicator. QMOHH again attains the best (lowest) average rank. In this case, the confidence intervals of MOPSO, SMPSO and QMOHH overlap, which is common when multiple high-performing algorithms are compared. As the confidence intervals of QMOHH and the other seven algorithms do not overlap, QMOHH outperforms the other seven algorithms statistically in terms of . In short, QMOHH consistently achieves the lowest mean rank and achieves competing performance again.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Means plot of the average ranks and 95% confidence intervals of the algorithms in terms of .

https://doi.org/10.1371/journal.pone.0348884.g008

Finally, Fig 9 illustrates the ranking results for the 1-NHV indicator. Consistent with the other evaluation indicators, QMOHH secures the top rank. The confidence intervals of QMOHH and MOPSO overlap, yet QMOHH’s lower average rank signifies its superior ability to balance convergence and diversity in the obtained Pareto front.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Means plot of the average ranks and 95% confidence intervals of the algorithms in terms of 1-NHV.

https://doi.org/10.1371/journal.pone.0348884.g009

To further validate pairwise differences, Dunn’s multiple comparisons test is conducted as a post-hoc analysis following the Friedman test. The results show that QMOHH significantly outperforms all compared algorithms except MOPSO and SMPSO in terms of three evaluation indicators. These findings are consistent with the overlapping confidence intervals observed in Figs 7–9 and confirm that QMOHH achieves superior or competitive performance while being statistically superior to the majority of the state-of-the-art algorithms. The comparative study, along with the non-parametric statistical analysis, confirms the effectiveness and robustness of the proposed QMOHH in solving the multi-objective RALSP under consideration. Despite some overlap in confidence intervals, QMOHH achieves the top ranking in terms of all three performance indicators; it could be concluded that the proposed methodology produces superior performance in solving the multi-objective RALSP.