Dynamic performance–Energy tradeoff consolidation with contention-aware resource provisioning in containerized clouds

Containers have emerged as a more portable and efficient solution than virtual machines for cloud infrastructure providing both a flexible way to build and deploy applications. The quality of service, security, performance, energy consumption, among others, are essential aspects of their deployment, management, and orchestration. Inappropriate resource allocation can lead to resource contention, entailing reduced performance, poor energy efficiency, and other potentially damaging effects. In this paper, we present a set of online job allocation strategies to optimize quality of service, energy savings, and completion time, considering contention for shared on-chip resources. We consider the job allocation as the multilevel dynamic bin-packing problem that provides a lightweight runtime solution that minimizes contention and energy consumption while maximizing utilization. The proposed strategies are based on two and three levels of scheduling policies with container selection, capacity distribution, and contention-aware allocation. The energy model considers joint execution of applications of different types on shared resources generalized by the job concentration paradigm. We provide an experimental analysis of eighty-six scheduling heuristics with scientific workloads of memory and CPU-intensive jobs. The proposed techniques outperform classical solutions in terms of quality of service, energy savings, and completion time by 21.73–43.44%, 44.06–92.11%, and 16.38–24.17%, respectively, leading to a cost-efficient resource allocation for cloud infrastructures.


I. Introduction
Nowadays, data centers are growing exponentially due to cloud services' popularization [1]. Cloud Service Providers (CSPs) use this kind of infrastructure to offer different tools and resources. They are mostly grouped into several types of services: Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), Storage as a Service (STaaS), Communications as a Service (CaaS), Network as a Service (NaaS), Monitoring as a Service (MaaS), a rapidly grooving Serverless computing, etc. [2]. Virtual Machines (VMs) and Containers (CTs) are the backbones of the virtualized services provided on-demand.
The efficient use of the data center infrastructures is fundamental for users and CSPs. For example, the low utilization of the servers is a critical factor for energy consumption inefficiency. Traditionally, researchers have focused on CPU utilization, where the VM placement problem is usually solved by NP-hard bin-packing. Few studies consider additional factors, such as reducing power consumption and avoiding SLA violations due to resource contention.
From a business perspective, reducing energy consumption leads to a significant reduction of environmental concerns and costs, hence reducing the final price for the user and increasing earns for CSPs [3].
The optimization of energy consumption is a challenge despite the diversity of existing energy management strategies. The energy efficiency of clouds has become a crucial research issue [2], mainly after introducing the green computing paradigm. An underlying technology involved in resource utilization and energy consumption of data centers is virtualization.
Resource virtualization refers to the abstraction of the hardware in a computer. Efficient resource usage, portability, scalability, and fast deployment are advantages of virtualization in the cloud infrastructure. The Virtual Machine Monitor (VMM), also called the hypervisor, is the main component of the hardware virtualization systems. This software allows the simultaneous execution of multiple guest VMs on a single server [4]. Some responsibilities of the VMM are strengthening the isolation between VMs and the management of the hardware resource.
The Operating System-level (OS-level) virtualization can be used either alternatively or in addition to hardware virtualization. Virtual CTs can be created on each OS [5,6]. Nowadays, the containerization technique is a buzzword for the cloud industry, especially in data centers [7]. The CTs are beneficial for CSPs since they can be more densely packed than VMs.
Each CT provides an isolated space for the user by encapsulating a group of processes separated from others in the system. The CTs can share the host kernel, libraries, and binaries, making them adequate technology to be used in scientific workflow platforms [8]. Examples of container implementations include Linux Containers (LXD), Docker, Kubernetes, OpenVZ, Singularity, etc.
In general, efficient job scheduling and load balancing over computational nodes remain challenging in massive, dynamic, elastic, diverse, and heterogeneous computational environments such as clouds.
The introduction of CTs technology brought several advantages to the cloud domain, but relevant topics in its field must be improved. Scheduling, load balancing, security, energy consumption, among others, are current topics of high importance to make cloud environments more efficient and accessible for the users.
Several researchers address the problem of energy consumption and/or makespan in data centers. The speed scaling method is highly used to aboard problems where the main objective is to save the energy subjected to the execution time constraint [8][9][10][11].
Unfortunately, these strategies affect the Quality of Service (QoS), which can be expressed using priorities, and Service Level Agreements (SLA), among others. Moreover, priority-based job scheduling introduces additional challenges because jobs with higher priorities must be served before those with lower priorities. Furthermore, priorities might influence the resource assignment, e.g., high-priority jobs might receive higher computing power than lower-priority jobs.
Several contention-aware resource allocation strategies to reduce energy consumption and increase performance are proposed [12][13][14][15][16]. Resource contention emphasizes avoiding cohosted applications that contend for shared resources. These works study the power consumption in environments with bare metal and VMs infrastructure. However, environments with CTs can increase the number of applications competing for shared resources, increasing energy consumption.
This paper focuses on the online jobs' allocation that contends for resources in a containerbased cloud environment. Our contribution is multifold: • We present job allocation as the multilevel dynamic bin-packing problem that provides a lightweight runtime solution that minimizes contention and energy consumption while maximizing utilization.
• The energy model considers types of applications and their execution on shared resources generalized by the job concentration paradigm.
• Two and three-level hybrid heuristic algorithms with container selection, capacity distribution, and contention-aware allocation are designed to consolidate resources to maximize utilization and reduce resource contention.
• Extensive experiments with eighty-six scheduling heuristics considering scientific workloads with memory and CPU-intensive jobs demonstrate that proposed techniques outperform classical solutions in terms of energy savings and completion time.
The proposed heuristics use different amounts of information in the allocation process. They improve the performance of well-known heuristics and provide a good compromise between QoS, makespan, and energy. We demonstrate that they are suitable for containers in cloud environments as more efficient and valuable solutions.
The rest of the paper is structured as follows. Section II discusses related work. Section III defines the infrastructure, scheduling, and energy model for allocating jobs into a containerbased cloud. Section IV describes the processing speed models of the jobs. Section V discusses job allocation strategies. Section VI introduces the experimental setup. Section VII presents the experimental analysis. Finally, we conclude and discuss future work in Section VIII.

II. Related work
The interest and usage of container-based technologies in cloud environments have been growing significantly. Several works pointed out that CTs are the future of clouds [17].
Several container technologies were developed, where each of them provides different deployment, management, orchestration, and communication. In this section, we review some of these container-based technologies. Then, we present the latest advances in the area of our interest.
An LXD container [18] is an OS-level virtualization technique that runs multiple isolated Linux systems on a single Linux control host. The LXD resides directly on top of the host kernel. Hence, it does not need any extra abstraction layer or another guest OS. A namespace provides an interface for the Linux kernel features and isolations of resources for each CT. The control group manages the allocation of resources (metering and limiting). Docker [19] is an open-source container project that simplifies the deployment of services with the methodology of one process per container. The Docker Container Engine, an additional abstraction layer, runs a single application in each virtualized instance. This methodology of execution attaches the CT lifetime with the finish time of the application. Docker daemon constructs a writable layer at the top of the CT read-only image to execute the processes.
Kubernetes [20] is an open-source system for managing the lifecycle of heterogeneous containerized applications (deployment, scaling, orchestration, etc.). The smallest deployable computing unit (POD), created and managed by Kubernetes, encapsulates a set of containers tightly coupled with some shared resources. It groups containers with shared storage/network resources for easy management of logical units of an application. A POD can be replicated along with several machines for scalability and fault tolerance purposes. Still, two services that listen on the same port cannot be deployed inside a POD.
OpenVZ [21] is a container-based virtualization technology to manage multiple secure and isolated Linux containers on a single physical server. Virtual Private Servers (VPSs) and CTs are basic units that share hardware and run on the same OS kernel as the host system. Kernel namespaces allow each CT to have an independent set of resources.
Singularity [22] is an open-source scientific container solution supporting an application in existing and traditional High-Performance Computing (HPC) resources. It offers computing mobility, reproducibility, user freedom, and integration with any scientific computational workflow. A Singularity CT image encapsulates the OS-system environment and all application dependencies necessary to run a defined workflow. Table 1 shows the relevant characteristics of the related works. It highlights a research gap in the domain of CTs and resource contention. Our approach focuses on the completion time and energy consumption of allocation strategies in container-based cloud environments under resource contention.
Xu et al. [23] discuss the brownout model to reduce data center energy consumption. This approach can reduce energy consumption by selectively and dynamically deactivating optional  [24] increase the functionality of the brownout to reduce data center energy consumption. They proposed a brownout-based approximate Markov Decision Process approach to improve tradeoffs between energy-saving and user discounts.
Xu et al. [25] propose several scheduling algorithms for managing microservices and CTs to reduce power consumption with QoS constraints. They achieve better performance in both objectives than the baseline algorithms.
Xu and Buyya [26] develop BrownoutCon to deal with overloads and reduce power consumption on cloud systems. It is integrated with Docker Swarm [36] to demonstrate its efficiency in managing containers under several policies.
Gholipoura et al. [27] propose a multi-criteria decision-making method for cloud environments with VMs and CTs. The consolidation defines the migration policy based on the virtual resource, CTs, and VMs.
Piraghaj et al. [28] present a set of allocation policies for energy saving on the Containers as a Service (CaaS) cloud paradigm. The authors propose an architecture and several algorithms to minimize energy consumption while maintaining the required SLA and QoS.
Khan et al. [29] study consolidation algorithms for effective migration of VMs, CTs, and applications to save energy without negatively impacting service performance. The results show the impact between the migration of applications and the migration of VMs, the resource consolidation considering various workloads, resources, and datacenter set-ups.
The cloud-based solutions community has also generated Internet of Things (IoT) research using VMs and CTs. The IoT workloads contain workflows with different types of jobs: CPU Intensive (CI), Memory Intensive (MI), and Bandwidth Intensive (BI) [33].
Celesti et al. [30] study the overhead costs of CT virtualization on an IoT device with a Raspberry Pi and Docker engine. The authors highlight the overhead introduced by container virtualization in a real scenario.
Dambreville et al. [31] discuss energy consumption reduction by introducing a prediction process in the scheduling task. The authors modify the available servers to fit the prediction and schedule all of the jobs on the available servers.
Cui and Xiaoqing [32] propose a scheduling solution in cloud computing for workflow tasks based on genetic algorithms. The fitness function defined the weighted sum of the userdefined deadline and the total cost of the workflow application execution. A top-down leveling is defined as the longest path from the task to the leaf task in the workflow for each task. This level is used as a task priority. According to a descending order of these priorities, the tasks are scheduled for the available hosts.
Tchernykh et al. [33] analyze several solutions for a digital twin workflow allocation in the virtual resource in a cloud infrastructure. The goal is to minimize the rent cost and satisfy the computational resources demand; the workloads include CI, MI, and BI jobs in order to model applications with different requirements. The authors propose allocation algorithms based on heuristics, metaheuristics, and mixed-integer programming to find low-cost solutions.
Several approaches focus on resource contention issues, where co-hosted applications contend for shared resources increasing energy consumption and reducing performance.
Armenta-Cano et al. [12,13] propose a resource allocation model for online energy-aware scheduling with job concentration. The authors characterize the energy consumption of applications and their combinations. It is used for heterogeneous job allocation to avoid resource contention.
Sheikhalishahi et al. [14] propose a multilevel resource contention-aware scheduling for energy-efficient in distributed systems. The authors define a resource contention metric for high-performance computing workloads. The approach models the interaction between system and scheduler information concerning jobs and resources.
Muraña et al. [15] study the power consumption for scientific computing applications in multicore systems. The authors evaluate the power consumption of applications in single and combined executions on Intel and AMD servers. The results indicate a strong dependency between the type of applications and power consumption.
Van Beek et al. [16] develop a CPU-contention predictor for Business-Critical Workloads (BCW) in data centers. It estimates performance degradation for VMs, hence, the risks of SLA violations.
Lovász et al. [34] present an energy-performance aware model for VMs allocation in heterogeneous servers as a variant of the multidimensional vector packing problem. The authors propose a prediction model to estimate performance degradation when different services are consolidated.
Blagodurov et al. [35] propose scheduling strategies to mitigate different resource contention sources on multicore processors. The authors define a classification scheme for contention in a cache space, memory controller, memory bus, and prefetching hardware.
In general, the different types of containers allow running a higher number and variety of applications in a single resource. Moreover, the energy consumption of the system is increased if the applications contend for resources. The following section defines the model to characterize job allocation in a container cloud environment aware of resource contention.

III. Model
In this section, we formulate the infrastructure, job, and energy consumption models, and define the optimization criteria of the job allocation problem. The main notations are summarized in Table A1 of Appendix.

A. Scheduling model
The IaaS environment is represented by a set of M servers (processors), P = {p 1 , p 2 ,. . .,p M }. Each server p k , for all k = 1,. . .,M, has a maximum processing capacity Q k expressed in Millions of Instructions Per Second (MIPS), and runs a set of m k containers, C k ¼ fc k 1 ; c k 2 ; . . . ; c k m k g. Each container c k i , for all i = 1,. . .,m k , has a processing capacity (CPU quota) q k i , expressed in MIPS, such that P m k i¼1 q k i � Q k . The total number of containers running in the infrastructure is denoted by m ¼ P M k¼1 m k : Jðc k i Þ defines the subset of jobs running in the container c k i and aðc k i Þ is the number of SLA violations.
We consider a set of n independent jobs, J = {j 1 , j 2 ,. . .,j n } that must be scheduled on the set of containers C ¼ S M k¼1 C k . Each job j j is described by a tuple (r j , s j , w j , ρ j , o j ) that consists of its release time r j �0, the minimum required processing speed s j in MIPS, the total amount of work w j in millions of instructions, the job type ρ j (CI and MI), and the priority expressed by an integer value o j . r j is not available before the job is submitted, and ρ j is only used for the system to compute the energy consumption. At any given time t, a processing speed s 0 j ðtÞ might be assigned to job j j , when s 0 j ðtÞ < s j then the resource incurs in SLA violations aðc k i Þ ¼ aðc k i Þ þ 1 for j j in c i . Additionally, w j can be used as an estimator of the finish time of j j , concerning s 0 j ðtÞ. Several techniques can be used to estimate an accuracy value of w j .
In this paper, we limit its use to identify CTs with a major amount of pendant processing. The idea is to study dynamic performance degradation and energy consumption increase due to shared resource contention and present consolidation heuristics based on contention-aware resource provisioning.
We consider this problem as a special case of multilevel dynamic bin-packing (online and non-clairvoyant). Bins represent CTs, and the jobs define the contribution to CT utilization. An additional element to the traditional dynamic bin-packing focuses on the contentionaware distribution of available processing capacity.
The processing speed of a job can be changed during its execution. It can be increased and reduced but should not be lower than the minimum required processing speed s j to satisfy SLA. When a job finishes its execution, the available processing capacity of the container is distributed between running jobs according to the strategies described in Section IV.
C max is the maximum finishing time (completion time or makespan) of all jobs. Let f i be the finishing time of j j job. C max is defined as: The total number of SLA violations is calculated as follows: In this paper, we consider three conflicting objectives for optimization: SLAv, E, and C max : The measure of the energy consumption E depends on the model. Traditional energy models do not take into account the types of applications and heterogeneity of workloads.
The next section describes an energy model that considers system performance degradation due to jobs contend for resources [12].

B. Energy model
Let us describe the energy consumption model used in our experimental evaluation. The power consumption of the system is defined by where E op (t) is the energy consumption of the infrastructure at time t calculated as a sum of the energy consumption e proc k ðtÞ of all individual processors k at time t: Two types of consumption are distinguished in the processor: static and dynamic. The static one is the power consumed by the component without performing useful work due to the leakage current, also known as idle power or base power e idle . The dynamic energy consumption is produced when an application utilizes components during its execution e used .
The energy consumption of the processor k at time t, e proc k ðtÞ of Eq (5), is computed as: where o(t) = 1, if the processor is on at time t, and o(t) = 0, otherwise, and the constant value e idle k does not depend on the time t.
The energy consumption of the processor k at time t depends on its utilization level and concentration of different types of jobs: where e max k defines a constant for the maximum energy consumption of the processor at full capacity, 0�F k (t)�1 is the portion of the power consumed by different types of jobs, and g k is a fraction of the total energy consumption introduced by job type combinations.
Known energy models are based on the observation that CPU utilization is highly correlated with overall energy consumption. Linear and nonlinear models are frequently used. In this work, we follow a nonlinear hybrid energy consumption model proposed in [12,13] as a function of CPU utilization and concentration of jobs of different types.
We calculate F k (t) as a sum of the portion of the power consumed by each job as follows: where f k d defines the fraction of energy consumption of a given job type d when its processor utilization is U k d ðtÞ at time t. Fig 1 shows f k d of CI and MI applications versus CPU utilization. We see that each job type contributes differently to the total energy consumption with the same CPU utilization.
Results are obtained by the well-known performance tool suite LIKWID for the GNU/ Linux operating system and SysBench benchmark.
The server used in the experiments is an Express x3650 M4, with two Xeon Ivy Bridge processors E5-2650v2 95W, default clock speed of 2.6 GHz. Each processor has eight cores and two threads per core (16 with hyperthreading), level 1 memory of 32 kB, level 2 of 256 kB, level 3 of 20 MB. Two Non-Uniform Memory Access (NUMA) domains of 32 GB each, with a total memory of 64 GB are used. The server OS is a CentOS Linux release 7.1.1503.
The combination of applications is considered in the following manner. Let us assume two types of jobs. The total CPU utilization is calculated as: where U k CI ðtÞ and U k MI ðtÞ are the utilization of all jobs of type CI and MI, respectively, executed on the processor at time t. They are calculated as: where s 0 j ðtÞ is a processing speed assigned to job j j at time t. g k ðφ k CI ðtÞÞ is a fraction of the total energy consumption introduced by the job types' combination.
φ k CI ðtÞ is the concentration (proportion) of the job CI (quantity of job utilization in a processor utilization) at time t. It is a measure of the total fraction of CPU capacity consumption of all jobs of a given type in the total capacity.  Fig 2, we see a reduction in energy consumption when jobs of both types are allocated to the processor.
The coefficient with the lowest energy consumption for CI and MI concentration is between 0.125 and 0.88. For instance, the concentration 0.125 is obtained for one MI and seven CI applications per eight-core processor. The concentration 0.88 is obtained for seven MI and one CI applications. If both concentrations are about 0.5, g k = 0.84.
The concentration φ k CI ðtÞ ¼ 0 implies that all applications in the processor are MI type, φ k MI ðtÞ ¼ 1, and vice versa. In both cases, there is no reduction in energy consumption because g k ðφ k CI ðtÞÞ ¼ 1.
A Lagrange interpolating polynomial represents the energy profile of each job type [15]. We use the obtained equations to estimate the energy consumption of jobs in the energy model.

IV. Processing capacity model
In this section, we focus on three models of the processing capacity distribution between jobs during allocation and execution: required capacity, full proportional capacity, and full priority capacity.

A. Required capacity
In the Required Capacity model (RC), the assigned job speed s 0 j ðtÞ cannot be higher than the minimum required processing speed s j to satisfy SLA.
The main idea of the RC model is to limit the utilization of CTs to avoid that jobs conflict for shared physical resources. We limit the utilization of the CTs to avoid the saturation of the resource due to several CTs can be hosted in it.
The threshold of the job allocation and available capacity redistribution is the minimum processing speed of each job. Formally, The two additional models keep resource utilization in full. The only difference is the distribution of the exceeding capacity, additional capacity after satisfying the basic requirement of jobs running in the CT.

B. Full proportional capacity
In the Full Proportional capacity model (Prop), unallocated available capacity in the container is distributed proportional among jobs. Formally, we assume that j i is assigned to the container c k j .

C. Full priority capacity
In the Full Priority capacity model (Prio), unallocated or available capacity in the container is distributed between jobs based on their priorities. Formally, it assumes that j j is assigned to the i container, where factor c k  The scheduler assigns the total CPU capacity to job j 1 at time 0, s 0 1 (0) = 1,000. Job j 2 arrives at time t = 50, then the capacity of c k i is distributed between both jobs according to Eq (16)

V. Job allocation
This section describes the set of scheduling heuristics for job allocation. They are similar to the well-known dynamic bin-packing problem, a variation of the classical NP-hard optimization problem.
Several heuristics were proposed to affront bin-packing problems producing solutions in a reasonable time frame. Both time and performance are fundamental in online resource allocation. Therefore, we propose heuristic-based scheduling strategies for the job allocation to CTs.
The scheduler performs two main tasks: CT selection and CPU assignation. In the selection stage, the scheduler decides whether the job is placed into one of the available CTs or a new CT should be invoked.
In the CPU assignation stage, the scheduler establishes the capacity inside the CT to execute the job. We use different factors in the system to allocate the jobs in the CT selection stage.

A. Container selection strategies
As an example, we assume that the scheduler follows the RC model. According to this model, the processing speed s 0 j ðtÞ assigned to a job j j at time t must never be higher than the minimum required processing speed s j , i.e., s 0 j ðtÞ � s j . The first two baseline strategies are: • Random (Rand)-the scheduler allocates a job to a random container c k i 2 C; 1 � k � M; 1 � i � m k . • Round Robin (RR)-the scheduler allocates a job using a Round Robin strategy, distributing to containers in rotation.
These strategies are "blind" because they do not use specific information about job allocations. Other strategies take into account information about the job to be assigned and CTs.
We consider three scenarios in the selection of a CT: 1. At least one container has sufficient capacity to execute the arriving job without SLA violation.
2. At least one container is not running at its full capacity.
3. All containers are using their maximum capacity.
Several policies can be used to face each scenario. A specialized policy per scenario can provide strategies with better results.
Whenever a job arrives, the scheduler uses the first policy to select a CT with enough capacity to execute it. If the job cannot be allocated without SLA violation, the scheduler employs a second policy to choose between CTs with available capacity. Finally, if all containers are running at full capacity, the scheduler uses the third policy to choose a CT. Table 2 presents the selection policies of CTs that have sufficient capacity to execute the arriving job. Table 3 shows the policies to select one CT among sufficient capacity containers. The preferred candidates are those that are not running at their full capacity.
Finally, we use two policies for CT selection when all CTs are working with full capacity: MinTasks and MinSLAv. Both strategies are described in Table 3.

B. Resources allocation strategies
Once a CT is selected, the scheduler assigns the CPU capacity to a job inside the selected CT. If the CT is selected with enough capacity, there are no SLA violations; otherwise, SLA violations can occur.
The capacity assignment is provided by the three models defined in section IV.

Strategy Description
First Fit (FFit) Allocates job j j to the first container capable of executing it, the first CT that satisfies Allocates job j j to the container with the smallest utilization left, Min Worst Fit (WFit) Allocates job j j to the container with the largest utilization left, Max

Minimum Amount of Work (MinAW)
Allocates job j j to the container with the minimum total amount of pending work for jobs running in the container, Minð P 8 d 2Jðc i Þ w 0 d Þ where w 0 d defines the amount of work processed of j d from r d to t in c i .

MaxSLAv
Allocates job j j to the container with more SLA violations, Max(α(c i )) for c i .

MinSLAv
Allocates job j j to the container with less SLA violations, Min(α(c i )) for c i .

Rand
Allocates job j j randomly to an active CT that satisfies q i À ð RC strategies combine three policies to allocate jobs to CTs. For instance, FFit-MinTask-MinSLAv strategy selects between CTs with enough capacity using the FFit policy. MinTask policy is used when some CTs have free capacity but not enough to satisfy the capacity requested by the job, and MinSLAv policy selects CT at maximum capacities. The RC strategy reassigns capacity when it is overpassed by the requested capacity of the jobs.
As an example, Algorithm 1 presents the FFit-MinTask-MinSLAv strategy. The worst-case occurs when all CTs are full and active. Then the scheduler searches in the list of active CTs to allocate the job (line 2) with complexity O(k) for k containers. It cannot assign the job at this stage, so it uses the second strategy (line 8) with the same complexity O (k). Finally, the strategy runs the procedure in line 10 with O(k) complexity. The procedure is performed each time a job arrives, so the algorithm has an asymptotical complexity O(nk). In Prop and Prio strategies, the CT selection is a combination of two policies. CT always has full capacity, but the strategy can estimate if an allocation generates SLA violation. For instance, with the FFit-MinTask-Prio strategy, the scheduler selects a CT using FFit strategy as a first policy and MinTask as the second policy. Once a container has been selected, Prio model reassigns the processing capacity using Eq (16). Algorithm 2 presents the FFit-MinTask strategy.
Similar to Algorithm 1, the complexity of Algorithm 2 is bounded by O(k) in lines 2 and 8, so the complexity of the procedure is defined by O(nk). Note, if we can create an unlimited Table 3. Selection policies of containers with available capacity.

MinTask
Allocates job j j to the container with the minimum number of assigned jobs, Min(J(c i )) for c i .

MaxCap
Allocates job j j to the container with maximum capacity available, Allocates job j j to the container with minimum capacity available, Allocates job j j to the container with less SLA violations at time t, Min(α(c i )) for c i . https://doi.org/10.1371/journal.pone.0261856.t003 number of CTs in the infrastructure then both procedures have an asymptotical factor of (n 2 ) in the worst-case.

VI. Experimental setup
All experiments are performed on the standard trace-based simulator CloudSim v3.0.3 [37]. It supports the modeling and simulation of large-scale cloud computing environments. Cloud-Sim includes classes and interfaces such as data centers, single computing nodes, an autonomous platform for modeling clouds, service intermediaries, provisioning policies, allocation strategies, among others. We extended its functionality with our scheduling algorithms, energy model, dynamic jobs arrival, container deployment, statistical analysis, and workload processing. The simulator represents an excellent tool to develop our experiments [38].
Algorithms are implemented using jdk 1.8.0_221 64-bit. The execution was performed by a computer with Windows 10 Pro OS, an Intel (R) Core (TM) i5-8400HQ CPU 2.8 GHz, 8 GB of memory, and 500 GB of HDD.

A. Scenario
We use a two-tier topology with M processors at the first level and m containers at the second level. We perform an experimental analysis with a limited but a representative number of physical resources. In the experimental environment based on CTs on bare metal, two CTs can be executed in one physical resource. Under this scenario, the jobs of both CTs contend for the underlay resources.
The setup defines a basic cloud environment that simplifies the analysis and evaluation of the proposed strategies and maintains the relevant elements of real environments. This configuration can be generalized considering the characteristics and size of resources, CTs capabilities, and workloads. Moreover, the energy model of specific resources and job energy profiles (i.e., BI) and their combination (concentration). e idle and e max values are defined according to [12,13]. The collected data from a Power Distributor Unit (PDU) showed that the e idle of the server is 86 Watts (W) on average. A similar procedure under a fully busy processor was performed to find e max . Table 4 presents the experimental setup.

B. Workload
The workload is based on HPC traces of parallel workloads [39] and grids from Grid Workload Archive [40]. We use the Standard Workload Format (SWF) with two additional fields to describe the requested work speed and the type of work. Several filters were applied to workloads to eliminate jobs with inconsistent information.
The performance of our strategies was evaluated in a homogeneous resource environment using 30 workloads of 24 hours to obtain valid statistical values. Variability of the arriving time, size, and types in the workload is fundamental to evaluate our strategies' efficiency under different scenarios properly. Fig 6 shows the number of jobs of two types during 30 days. We observe that there is no predominance of one type of job in logs. Some weeks have more jobs of type CI, and others more jobs of type MI. The total number of jobs in the workload is 109,345. Day twenty-two has the biggest workload among all, with more than 20,000 jobs.
CI jobs request high computational power in their processing, e.g., calculate prime numbers, sorting, search, graph traversal, etc. In MI jobs, the job processing is limited by the speed of memory to feed data to the processor, e.g., work with datasets much larger than the available cache on the system.

C. Analysis methodology
The optimization problem is to minimize three conflict objectives: SLAv, C max , and E, see Eq (3). In multi-objective optimization, one solution can represent the best solution concerning C max , while another solution could be the best one concerning E or SLAv. The goal is to obtain a set of compromise solutions that represents a good approximation to the Pareto front. If a solution cannot be improved in terms of all objective functions, then the outcome is Pareto optimal. Pareto fronts can be compared via visual observation of the solution space and formal approaches.
Set Coverage (SC) metric is a formal and statistical approach to calculate the proportion of solutions dominated between two sets [41]. Larger values of SC represent a better approximation of the Pareto front. The dominance operator is not symmetric, so it is necessary to compute the dominance of one set over the other and vice versa.
A multi-objective optimization problem can be simplified to a single objective through different methods. The prevalent approach is the aggregation method, which defines weights for each objective to model preferences and performs a single weighted sum. That is, the preferences can explicitly specify the importance of every criterion or relative importance between criteria.
The degradation in performance (relative error) indicates the ratio of a metric generated by the algorithms to the best-found solution [42]. The analysis is conducted as follows. First, the degradation in performance of each strategy is computed as follows: strategy criterion value best found criterion value À 1 Then, the strategies are ranked based on the average of their computed values considering all the scenarios. The best strategy with the lowest average performance degradation has a rank of 1. Low ranks identify strategies that perform reliably well over different scenarios, they represent a good compromise for all test cases.
One disadvantage of the degradation approach is to use the mean values to analyze the results. Hence, they can be influenced by a small portion of data with a large deviation. For deeper analysis, we present the strategies' performance profiles to help with the interpretation of the data.
The Performance Profile (PP) defines a non-decreasing, piecewise constant function δ(τ) that presents the probability that a ratio τ is within a factor of the best ratio. The function δ(τ) is the cumulative distribution function. Strategies with a large probability for small τ are to be preferred [43].
To choose an adequate strategy, we compare the performance of all strategies by the three analysis methodologies: degradation, PP, and SC.

VII. Experimental evaluation
In this section, we present results of evaluation of 86 strategies. Their names are combinations of strategies on the first, second, third levels, and capacity models. Table 5 shows 56 strategies for RC model. Table 6 shows 14 strategies that can be combined with Prop and Prio model (28 in total), Random (Rand), and Round Robin (RR).
The performance of our strategies is evaluated based on the simulation of 30 days of 24 hours each. Runtime of one strategy simulation of the 24 hours workload takes about 1-2 second, on average.
Let us consider the best and worst performance strategies for each objective independently. The results of other strategies were omitted for simplicity and clarity. Fig 8 presents SLAv for the best and worst strategies during 30 days on 24 hours average. WF_MinTask_Prio has the best performance. Its SLAv varies from 27 to 5,294 per day. SLAv  Table 7 presents the average degradation of SLAv, E, C max , and their means, see Eq (18). The last five columns contain the ranking of SLAv, E, C max , Rank mean, Rank, their means, and the final ranking. Rank SLAv is based on the number of SLA violations. Rank E depends on the For energy consumption, the best strategy is BF_MinTask_Prop, with an average E degradation 0.027. It allocates the arriving job to the container with the smallest available capacity. If any container has enough space to provide the required speed to the job, the scheduler selects the container with the least number of tasks. MaxSLAv_MinTask_Prio and MaxSLAv_Min-Task_Prop follow the best strategy with degradation 0.037 and 0.038, respectively.

A. Degradation in performance analysis
For C max degradation, WF_MinTask_Prop is the strategy with the best performance with a degradation of 0.087. The arriving job is assigned to the container with the biggest available capacity. At the second level of assignation, it selects the container with the least number of tasks. The second and third best strategies with degradations of 0.87 and 0.96 are FF_Min-Task_Prio and MaxSLAv_MaxCap_MinTask.  According to the final Rank, strategies with a good compromise between SLAv, E and C max , are BF_MinTask_Prop, MinAW_MinTask_Prop, and MaxSLAv_MinTask_Prop. The final Rank of RR is 85. It means that at least 84 strategies performed better than RR. Rand and RR provide 129,551% of worse solutions with respect to the best solutions found in the simulation. While, BF_MinTask_Prop is away from the best solutions between 78.1% for SLA, 2.7% for E, and 10.3% for C max .   The first strategy provides 70% of its solutions within a factor of 1.05. The second strategy has 95% of solutions below a factor of 1.08. Fig 14 presents the PP of eight strategies according to mean E and C max in the interval τ = [1.0, 1.3]. If we choose a factor of 1.08 from the best result as the scope of our interest, Max-SLAv_MinTask_Prio has a 72% probability of being the winner. FF_MinTask_Prio is the strategy with 92% of solutions within a factor of 1.18. MaxSLAv_MaxCap_MinTask is the strategy with all its solution within a factor of 1. 28.

B. Performance profile analysis
We see that the best strategies include MinTask allocation. Minimizing the number of the running of jobs on the resource reduces possible contention and SLAv increasing the assigned speed to each job. Hence, it reduces the energy consumption and job completion time.

C. Set coverage analysis
Let us analyze Pareto fronts of strategies considering three optimization criteria: SLA, E and C max .
Two sets of non-dominated solutions are compared using the SC metric. The rows of the Table 8 show the values SC(A, B) for the dominance of strategy A over strategy B. The columns indicate SC(B, A), that is, the dominance of B over A.
The column "Average best" shows SC of row A over each of the best strategies. The column "Average all" shows SC of row A over all studied strategies. The rankings "Rank best" and "Rank all" are based on the average dominance over best and all studied strategies, respectively. Similarly, the last four rows show average dominance B over A, average dominance B, and two ranks in each column, respectively. The higher ranking implies that the front is better. The most significant values are highlighted in bold and red.
According to the SC metric, the strategies with the best compromise between SLAv, E and C max are WF_MinTask_Prop, BF_MinTask_Prop, and MaxSLAv_MinTask_Prop considering all strategies, and WF_MinTask_Prop, BF_MinTask_Prop, and MinAW_MinTask_Prop according to the best strategies.
WF is trying to allocate jobs with more capacity left to avoid resource contention. WF_Min-Task_Prop improves the non-dominated fronts of other strategies in the range of 10-36.7%, with 33.3% for the best strategies and 51.6% for all strategies, on average, occupying the first and the second ranks, respectively. We see that WF_MinTask_Prop can provide solutions with better quality, on three objectives, with respect to other strategies.
SC(A, WF_MinTask_Prop) displays that strategy is dominated by other strategies with a maximum of 16.7%. These ranges show that other strategies have a better performance than WF_MinTask_Prop. In general, a desired strategy exhibits the behavior of WF_MinTask_-Prop. SC(A, RR) and SC(A, Rand) are equal to 1, hence, all the strategies dominate RR and Rand. Moreover, SC(RR, B) and SC(Rand, B) are zero, so any solution of Rand and RR is dominated by other strategies. We see that similar to the performance profile analysis, best strategies include MinTask allocation. Moreover, the MinTask and Prop provide the four best strategies in both ranks.

VIII. Conclusion
Consolidation of services is a key technology to reduce resource overprovisioning and energy consumption. However, when multiple jobs and processes run on a multicore CPU, they can compete for shared resources such as caches, memory controllers, memory buses, prefetching hardware, disks, networking, etc. This resource contention can invoke performance degradation defeating the benefits of the consolidation, leading to QoS violation and increasing energy consumption.
In this paper, we consider potentially damaging effects of job consolidation. We propose a novel method of consolidation based on the job concentration paradigm that avoids allocation of jobs of the same type on shared resources. It reduces resource contention and makes job placement more efficient with the energy-utilization tradeoff.
We present online job allocation heuristics for heterogeneous infrastructures as a multilevel variant of the dynamic bin-packing problem considering container selection, capacity distribution, and contention-aware allocation. We study eighty-six scheduling heuristics. We distinguish them depending on the type and amount of information they use for allocation. We provide their comprehensive analysis on a real workload.
The results show that proposed allocation strategies carefully guided by the energy, performance, and QoS provide a reduction of 21.73-43.44%, 44.06-92.11%, and 16.38-24.17% of these criteria, respectively, with respect to known strategies used in the literature.
However, further study is required to assess allocation strategies on bigger variety of infrastructures, processors, and containers. It is important to analyze network-intensive, diskintensive, etc. job types, combining their execution over different scenarios.