1.1 Overview

A highly challenging aspect of High Performance Computing (HPC) is the need to reformulate and restructure scientific algorithms to perform well on different types of parallel architectures. In the current hardware landscape, a special focus is devoted to many-core platforms, which include homogeneous systems like the AMD Zen processors investigated in this article, or heterogeneous systems which use a many-core device as an accelerator, including GPUs or Intel’s now discontinued Xeon Phi platform. Optimized simulation software can be developed for given platforms using device specific programming languages or programming paradigms, including the use of multi-threading on many-core CPUs, or the use of dedicated languages like OpenCL or Cuda on GPUs. In these cases, the obtained code can be very different from a desirable code for scientific research, as a strong emphasis is put on meeting hardware constraints rather than express the actual numerical model or simulation algorithm. Furthermore, this approach requires different code bases to be developed for different architectures, which strongly limits its portability and long-term maintainability.

For this reason, multiple language extensions (such as OpenMP and OpenACC) or completely new programming languages (such as Futhark) have been introduced, which propose to achieve notable performance improvements on parallel hardware in a platform-independent manner, through the application of targeted directives or through the adoption of a specific implementation model. In the recent literature, special attention is devoted to functional paradigms, the principles of which are summarized by the so-called map-reduce formalism. In this case, an algorithm to be executed on a collection of data elements is expressed in terms of two element access functions, the “map” operation that applies a transformation to a data element, and the “reduce” operation that expresses a global reduction over the transformed elements. The actual, repeated execution of these operations is expressed by the framework instead of the user, allowing for automatic adaptation and optimization of a code on different devices.

This philosophy of automatic acceleration of functional code is embraced by the C++17 standard. It introduces execution policies for standard algorithms, which we will refer to as Parallel Algorithms, that allow parallel and SIMD optimizations. Examples of algorithms that are accounted for by execution policies are transform-reduce (the C++ equivalent of map-reduce), for_each, which only addresses the “map” part of map-reduce, or problem-specific functions like the sorting algorithm sort. Parallel Algorithms do not necessarily exhibit better performance than traditional language extensions, as a given implementation of the C++ standard library may for example decide to execute Parallel Algorithms with the help of OpenMP directives. They do however offer an elegant formalism, as all parallel constructs are expressed as an inherent part of the C++ language. They furthermore offer a strong guarantee of cross-platform compatibility and long-term maintainability. Indeed, the C++ language is standardized by the International Organization for Standardization ISO and widely embraced by the scientific community. In addition, the maintainability of a code written in C++ is guaranteed, because the language is known for its success in maintaining backward compatibility over several decades.

While the formalism of Parallel Algorithms is quite recent, it has been tested quite thoroughly on many-core CPU systems thanks to the available implementation through Intel’s Threading Building Blocks, or its implementation in Visual C++. Only recently though, less than three months before the initial submission of this manuscript, an implementation has been made available that ports Parallel Algorithms to GPUs in form of the NVIDIA HPC SDK C++ compiler, allowing to take profit of a truly cross-platform experience of this formalism.

In this manuscript, we propose a model for the implementation of lattice Boltzmann (LB) codes within the framework of C++ standard algorithms, and show the potential for accelerating such codes with the help of execution policies. One of the challenges consists in the implementation of the streaming step, which is non-local and does therefore not fit naturally into the framework of a repeated application of the “map” element access function to single elements. Instead, we take profit from the liberties allowed by the C++ standard to deduce the global index within a data collection for certain types of iterators (including raw pointers), and manipulate these indices further to achieve non-local data accesses. While this strategy incurs the cost of added arithmetic operations, the numerical tests show that these do not impact the overall performance in a way that would be prohibitive for the use of this method in a HPC environment.

The proposed modeling environment relies on the usual LB collision-streaming cycles but expresses the local collision through a cell-independent scheme. This approach opposes the frequently adopted strategy of adjusting the local collision scheme to implement boundary conditions (see e.g. the Palabos software [1]). Instead, the streaming model is locally adjusted to react to encounters with a wall and implement general Dirichlet boundary conditions, following the common, link-based bounce-back approach [2, 3]. This approach allows to build a framework with the ability to treat both bulk and boundaries coherently across a large variety of collision models, in a manner that is compatible with efficient many-core parallelism. The presented validation case of a lid-driven 3D cavity shows that the selected boundary treatment does not need to hide behind more sophisticated approaches, as the present framework shows to produce stable and accurate results even with the notoriously unstable BGK collision model. It is pointed out that, although this path is not further explored in this work, the link-based bounce-back collision model can be extended to the treatment of general curved boundary conditions while remaining entirely local (see e.g. [4]).

The result of our efforts is a framework that allows to freely combine six different memory access schemes with nine different LB collision models and bounce-back based Dirichlet walls. In all cases, efficient, platform agnostic many-core parallelism is available. The framework is disseminated in form of the open-source C++ library stlbm, which offers an extensive collection of reusable codes and patterns for the acceleration of LB software. To enhance the educational value of the stlbm library, selected self-contained single-file codes are also provided, which reduce the full software framework to a single memory access scheme and LB collision model. They illustrate that it is possible to write a cross-platform, efficiently parallelized LB application in a readable manner and with less than 500 lines of code, including all necessary data analysis and post-processing functionalities.

The remainder of this introductory section provides a short overview of programming models used to implement parallel software on many-core platforms, and specifically on strategies used to accelerate LB programs on such platforms. Section 2 summarizes the method used in this article to implement the LB algorithm and explains in some detail the six schemes (with different data structures and memory access patterns) and nine collision models available in stlbm. It then explains the programming model of C++ Parallel Algorithms, and explains the methodology used to fit the LB algorithm in this model. It finally lists the hardware platforms used to test the performance of the stlbm library. Section 3 presents a validation case for the stlbm library (a flow in a lid-driven cavity) and performance measurements for the various implementation schemes, collision models, and hardware platforms considered in this article. Finally, conclusions are drawn in Section 4. Section 5 provides a link to the repository containing the open-source stlbm library and to the detailed performance measurements obtained with stlbm.

1.2 Programming models for many-core platforms

The present article investigates two types of many-core platforms, namely homogeneous systems, in the form of many-core CPUs, and heterogeneous systems, in the form of systems accelerated with GPUs. In the latter case, the term “heterogenous” refers to the fact that the program is initially executed on a host (the CPU) which handles the program setup and data initialization, after which the computationally intensive tasks are typically offloaded to a large extent to a device (the GPU). Other types of devices, which are not explicitly reviewed here, include FPGAs and Neural Processing Units (NPUs).

Applications on many-core platforms can be programmed using low-level programming models, which include threading mechanisms such as POSIX Threads [5] for homogeneous systems. On heterogeneous systems, a strong standard is available in the form of OpenCL, which offers a vendor-independent programming model. In practice however, the vendor specific Cuda programming language [6] has established itself as a de facto standard for heterogeneous GPU platforms, a fact which is sometimes attributed to the difficulty of achieving optimal cross-vendor performances in view of the diversity of many-core architectures [7].

Higher-level programming models, which are the focus of this article, relieve the programmers from hardware specific details and allow an increased focus on the implemented algorithm, and potentially a greater independence from the selected hardware platform. The two reviewed approaches are

• Directive based programming models. This approach, based on extra-language directive annotations, includes industry standards like OpenMP [8], which is mostly targeted at homogeneous many-core systems, and OpenACC [9] for heterogeneous platforms. They typically allow a code to run both on conventional platforms (on which the directives are simply ignored) and on accelerator-supplied systems.
• Programming models based on C++ and on the C++ STL. C++-based programming models include DPC++ that is part of Intel’s OneAPI [10], C++ AMP which is based on DirectX11, AMD’s GPU-oriented C++ API [11], and the unified PACXX programming model [12] that allows to write both host and device code within the C++14 standard. Recently, an increased interest is observed for programming models that are based on the C++ Standard Template Library (STL) or on extensions thereof. For example, Thrust [13] is a parallel template library in C++ inspired by the STL, which allows a hardware independent programming style for both many-core CPUs and Nvidia GPUs. However, hardware independent program development for many-core platforms is also possible within the strict limits of the C++17 standard, which introduces the concept of Parallel Algorithms. This concept is exploited by Intel’s Threading Building Blocks (TBB) [14], as well as by the STL implementation of Visual C++, which provide a backend to C++ Parallel Algorithms for homogeneous many-core systems. Finally, NVidia introduced very recently a version of the NVidia HPC SDK which offers an NVidia GPU backend to C++ Parallel Algorithms. The TBB and the NVidia HPC SDK backends are used in the present work to test the performances of the proposed programming model for LB applications.

A more thorough review of programming models for homogeneous and heterogeneous many-core platforms is found in [7].

1.3 Acceleration of lattice Boltzmann codes on many-core systems

The LB method has always been considered an excellent candidate for efficient parallelism, most notably thanks to its convenient separation into a local collision step and a streaming step with non-local memory accesses of limited extent. Without investigating the numerous implementations and methodologies in detail, we point out the importance which has been devoted, across various systems and platforms, to the reduction of the memory footprint, the improvement of memory access patterns, and the definition of adequate and efficient data structures. Pohl et al. [15] propose an implementation strategy which requires the allocation of a single set of populations only (referred to as a compressed grid approach) and executes a collision and streaming step for all cells within a single memory traversal (referred to as a fused collision-streaming step). This approach, which is at the heart of the open-source WaLBerla library [16], stands at the beginning of a long list of subsequent, similar LB implementation models. We mention in particular the swap algorithm [17], which is used in the open-source Palabos library [1] and the AA-pattern [18], which was described specifically in the context of GPU implementations of LB models, but has proved highly successful on CPU systems as well [19].

With the advent of general-purpose GPU (GPGPU) computing, GPUs have quickly become a target for LB implementations as well. An original proposition published in [20] proved promising in spite of its fairly poor performance, and was further improved in form of the 2D LB code proposed in [21]. Further performance improvements were exhibited over the years, often by fine-tuning memory access patterns, as published for example in [22–25].

To a lesser extent, LB implementations using the cross-platform formalism OpenCL are found in the literature, which document attempts to target GPUs of different vendors, but also the now discontinued Xeon Phi architecture of Intel (see for example [26, 27]). We also point out the attempts performed with the open-source Sailfish project [28] to use meta-programming capabilities of Python to target multiple platforms through both a Cuda and an OpenCL backend.

2.1 LB algorithm

The Lattice Boltzmann method is a very specific (physical and numerical) discretization of the Boltzmann equation that splits explicit time iteration into a collision and a streaming step. Space is subdivided into N_tot cells that are in principle equally spaced with a distance δx along the three principal Euclidean directions. Inhomogeneous cell arrangements can be achieved through mesh refinement strategies [1, 29, 30], which are however not taken into account in this work. The state of the system is defined on every cell by a certain amount of scalar values f_k(k = 0, …, Q − 1), hereafter called populations. Their number Q depends on the stencil chosen to discretize the phase space (i.e., physical discretization of the Boltzmann equation). The present work is based on the commonly used 19-velocity stencil D3Q19 [3, 31, 32], a choice that limits the current scope of the implementation to isothermal flows. A time iteration takes the populations from their state at time t to the next state t + δt, where δt is a constant discrete time step: (1) (2) The 19 discrete velocities c_k, which connect lattice nodes with near neighbors in lattice units (the components of the c_k have integer values), are defined for example in [3]. The temporary values , sometimes called “outgoing populations”, may be stored either in a separate cell array, or in the same array as the f_k, using an in-place value replacement scheme (see Section 2.3).

With the link-wise bounce-back scheme used presently to implement Dirichlet boundaries, populations that encounter a physical wall on the way from the original cell to the neighboring one, revert direction and are assigned to the original cell, in opposite direction [3, 33]. The adapted streaming step reads: (3) Here, stands for the opposite direction of k (i.e., ). In addition, u_w is the velocity imposed by the Dirichlet boundary condition, and ρ_w is the “wall density”. In the present work, ρ_w = 1 because the flow runs in a quasi-incompressible regime and exhibits density fluctuations around 1. This boundary condition scheme places the wall half-way between a fluid node and the next solid node, although the exact wall location may be viscosity-depend for low Reynolds number flows. This problem is circumvented through the use of the TRT collision term [34] (one of the nine models implemented in this article), which should be the preferred choice in pore-scale porous media flows and other high-viscosity flows that are sensitive to the wall location.

The link-wise bounce-back (3) scheme only involves populations that originate from, or move toward, fluid cells. Therefore, no storage of populations is required on solid cells. The strategy chosen in stlbm to implement boundary conditions consists in allocating nevertheless a layer of wall cells in the solid domain, with 19 storage locations like common fluid cells. These wall cells do not store populations, but are rather assigned pre-computed values of the momentum exchange term appearing in Eq (3). Thus, the 19 storage locations pop_k of a cell are utilized as follows: (4) (5) As explained below, the situation is slightly more complex with one of the three implemented memory access schemes, the AA-pattern, which obeys Eq (4) at even time steps only, but retrieves pre-collision populations from neighboring fluid cells at odd steps instead. With the compact storage scheme expressed in Eqs (4) and (5), the streaming step is expressed in a uniform manner as an access of neighboring cells to achieve two goals, namely, (1) to copy populations and execute conventional streaming, and (2) to read the pre-computed momentum exchange term and sum it up with a reverted population. This latter possibility is expressed in the rightmost column of Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The algorithms executed for the three implemented schemes.

https://doi.org/10.1371/journal.pone.0250306.t001

Implementations of the LB algorithm can be found in the literature using both single-precision and double-precision floating point values. Single-precision implementations are particularly popular on GPUs, which are usually substantially more efficient in single precision, except on few high-end GPUs targeted at HPC platforms (which are not tested in the present work). Single-precision computations in LB simulations carry however the risk of round-off errors that are difficult to anticipate, especially in research codes, although mitigation strategies exist (see e.g. [35]). To emphasize its goal, which is to be reused in the context of general-purpose research and production codes, the stlbm library has been developed and tested with double precision variables only (although an adaptation to single precision would be easy).

2.2 LB collision models

In the present work, nine collision models are considered to evaluate the impact of the collision step (1) on the overall performance of the LB code. Every model is written based on the unified framework proposed by Coreixas et al. [36, 37], which is particularly useful to derive efficient formulations that do not rely on a matrix form.

More precisely, these collision models include standard operators such as the BGK operator with a weighted second-order equilibrium (BGK-W2 [38]), a non-weighted fourth-order equilibrium (BGK-NW4 [36, 39]), and the two-relaxation-time formulation (TRT [34, 40]). In addition, multi-relaxation-time collision models based on raw (RM [41–43]), Hermite (HM [44–47]), central (CM [39, 48–50]), central Hermite (CHM [51–53]) moment spaces, as well as cumulants (K [54–57]), are further considered. Finally, a multi-relaxation-time formulation of the recursive regularized (RR) collision model is included due to its interesting stability property [58–62]. It is worth noting that most of these multi-relaxation time collision models have been proposed for the D2Q9 and D3Q27 lattices, but in this work, we used non-weighted D3Q19 formulations that are summarized in Appendix G of Ref. [36]. It was also made possible for the user to freely adjust the bulk viscosity, through an extra parameter, in order to further increase the stability of the code when needed.

More details regarding the implementation of these collision models is provided in Section 2.6, whereas their impact on the performance is presented in Section 3.3.2.

2.3 Data structure and ordering of populations

To implement an LB scheme, the populations f_k must be stored at a provided memory location. This memory location can vary in time, and it may or may not be identical for pre- and post-collision variables (f_k and respectively). A naive attempt to implement an in-place algorithm, which is to rely on the same storage space for populations at time t and t + δt, would lead to a conflict with the non-local nature of the streaming step. Indeed, as post-collision populations are streamed to neighboring cells, they potentially overwrite populations that have not yet completed their collision-streaming cycle, hence leading to erroneous results. In search of a solution, additional constraints to consider are:

• Reduction of memory requirements. A low memory footprint is desirable because (1) memory may be a critical resource on a given system, and (2) the total amount of memory processed by a system may impact its performance.
• Reduction of memory accesses. Although the actual picture is complicated by the presence of cache mechanisms, performance is generally improved by limiting the number of read- and write-accesses to the main memory.
• Thread safety. This feature is required to allow the multi-threaded execution models used in this article. Thread safety is achieved without synchronization primitives or locks, by applying schemes that naturally avoid race conditions. A global synchronization is applied only once per time iteration (or twice, in case of the swap algorithm). Within this context, achieving thread safety amounts to making sure that the collision-streaming algorithm is independent of the order of traversal of the cells.

This article relies on three popular thread-safe schemes (double-population scheme, swap algorithm, and AA-pattern), which are shortly described below. Their corresponding algorithms are summarized in Table 1. All three schemes rely on an array of N_tot × 19 floating-point variables pop. This amounts to 19 variables for each node location x, which are accessed through the syntax pop_k(x). The actual memory layout of the poparray is discussed at the end of this section.

2.4 STL and Parallel Algorithms

The C++ Standard Template Library (STL) supports a functional programming style, in which algorithms are supplemented with element access functions which customize their behavior. The algorithms apply these functions repeatedly to the elements of a data container, either through function pointers or function objects. At the application programmer level, an explicit loop is then replaced by a single call to an STL algorithm, hence leading to a programming style that is sometimes considered to be more expressive. The machine codes generated by an algorithm invocation or an explicit loop do not necessarily differ, as a compiler can take advantage of the mechanisms of C++ templates to translate the algorithm to an identical or similar loop as the hand-written one. This is especially true when the implementation of the element access function is known by the compiler at the time of the algorithm instantiation, in which case the function body can be efficiently inlined.

The following C++ code extract illustrates this concept with the STL algorithm for_each, used to compute macroscopic variables in the manner of a typical C++ program:

using namespace std;

struct Cell {

   array <double, 19> f;

   double rho;

   array<double, 3> u;

};

vector <Cell> cells(nCells);

array <int, 3>* c = …; // Lattice velocities

// … Initialize data, start principal time loop.

for_each(execution::par_unseq, begin(cells), end(cells),

        [c](Cell const& cell)

{

   cell.rho = 0.;

   cell.u = {0., 0., 0.};

   for(int k = 0; k < 19; ++k)

{

        cell.rho += cell.f[k];

        cell.u[0] += cell.f[k] * c[k][0];

        cell.u[1] += cell.f[k] * c[k][1];

        cell.u[2] += cell.f[k] * c[k][2];

}

});

In this example, the data layout follows the principles of an array-of-structure (aos), as all variables related to a given cell are located at consecutive memory addresses. The element access function provided to the for_each algorithm takes in this case the form of a lambda expression (a lambda for short), which defines an anonymous function object. A lambda can obtain access to variables from the surrounding scope, in a manner specified in the capture clause. In this example, the clause [c] indicates that the variable c, which refers to the discrete velocities, is captured by value. For reasons linked to limitations of the GPU backend of Parallel Algorithms, as explained below, the stlbm library defines [c] as a raw pointer to a heap array.

Finally, the code extract makes use of a parallel version of the for_each algorithm. Parallel algorithms are available starting with the C++17 standard. They receive an execution policy as an additional argument to indicate a parallelization strategy they may use to accelerate the execution of the algorithm. The four possible strategies are

2.5 Parallel LB algorithm

The code extract of the previous section uses an anonymous lambda to express the element access function of an STL algorithm, following standard practices in simple STL usage scenarios. This strategy however forces the access function to be implemented at its point of usage, which strongly limits the possibility to properly organize a larger code base. In the following, function objects are therefore instantiated from named classes, both to encourage the reuse of the proposed code in other projects, and to avoid misunderstandings linked to the implicit syntax of lambda capture clauses.

The encapsulation of element access functions in named classes opens the possibility for their instantiation (the function object) to be named and obtain an extended lifetime. It is even possible for the data such as the LB populations to be owned and managed by the same object, in which case the LB data and algorithms are unified under the same abstraction. In the present manuscript, we prefer however to treat these objects as stateless algorithms, and to manage data in a different scope. The purpose of this approach is again the ease of reuse of the proposed codes which, in a considered frequent use scenario, should be applied to accelerate selected parts of an existing code with a pre-existing memory management strategy. This LB algorithm class adopts the following canonical shape:

struct LBM {                      // (1)

   using CellData = array <double, 19>; // (2)

   CellData*lattice;                 // (3)

   array <int, 3>* c; // lattice velocities  // (4)

   double* t; // lattice weights

   double omega; // relaxation parameter

   Dim dim; // nx x ny x nz dimensions

   void operator () (CellData& cell) {       // (5)

       // Implement collision-streaming for a cell.

   }

};

The enumerated lines require the following clarifications:

1. (1). The class is declared with the struct keyword to provide public access to all data and methods, as data encapsulation or any other elements of object-oriented programming are outside the scope of the presented work.
2. (2). CellData names the type of a single data element provided to the for_each algorithm, which must be chosen according to the data layout. Here, the example of an aos layout with consequent cell data is chosen.
3. (3). The class variables of the five following lines are the articulated counterpart of a lambda capture. A pointer to the full lattice is captured to allow access to the neighboring cells. In the case of the soa data alignment, the full lattice is also required to gather all populations of the current cell.
4. (4). We chose to capture the lattice constants (velocities and weights) by reference and the relaxation time and lattice dimensions by value.
5. (5). To turn class instances into a function object, the function call operator is overloaded to implement a collision-streaming cycle for a single mesh grid cell.

A canonical code for the application of an LBM instance to an LB lattice is expressed as follows:

vector <LBM::CellData> lattice_vect(num_elements); //  (1)

LBM::CellData* lattice = &lattice_container[0];    //  (2)

LBM lbm{lattice, &c[0], &t[0], omega, {nx, ny, nz}]; //  (3)

for_each (execution::par_unseq,

      lattice, lattice + num_elements, lbm);

This extract is again commented at the enumerated lines:

1. (1). The number of elements equals the number of cells (aos) or 19 times the number of cells (soa).
2. (2). The vector is simply used here for automatic management of heap data. The data can also be allocated with the new operator and deleted manually.
3. (3). Just as in a lambda capture, any variable visible in the present scope can be provided to initialize a LBM object.

Independently of the memory layout strategy (aos or soa), the data container lattice_vect is required to allocate a number of floating point variables equal to 19 times the number of cells. Differences only appear in their usage pattern. In the algorithmically simpler case of aos, these variables are regrouped by cells, as the type CellData refers to sequences of 19 floating point variables. The for_each algorithm is then simply applied to the full data space, cell by cell. In the soa layout on the other hand, the cell data is non-contiguous in memory, and the CellData type is chosen to refer to a single floating-point variable. The for_each algorithm can consequently not be applied to the full data space –as this would lead to too frequent invocations of a cell-level collision-streaming cycle– but spans only over the elements of the first population f₀. It is the responsibility of the methods of LBM to compute the indices of the other populations inside the array lattice and access them manually. Similarly, indices are computed to access neighboring cells for the streaming step.

The index of the currently processed cell is not explicitly provided by the for_each algorithm to its element access function. While the index could be reconstructed in a non-parallel version of for_each by giving up the stateless nature of LBM and incrementing an internal index at each function call, such an approach would be in obvious violation with a multi-threaded or unsequenced execution of the algorithm. Instead, we choose to rely explicitly on the fact that the iterators in this invocation of for_each are raw pointers, and deduce the index from the difference of memory address between the current and first element of the lattice. While it is clear that such an approach could not be extended to other types of C++ containers and iterators, it should also be mentioned that this limitation to the scope of raw data arrays is rather expected in the context of massive HPC. It is further pointed out that the proposed code preserves a consistent high-level structure. Violations of the safety of data types and memory bounds are limited to a small group of data access function which encapsulate the cases of pointer-based arithmetic operations. Consequently, the data types and data access functions are defined as follows in a array-of-structure respectively a structure-of-array layout:

// Implementation for array-of-structure layout

struct LBM {

   using CellData = array<double, 19>;

   double & f (int i, int k) {

      return lattice[i][k];

   }

   // … implementation of collision models

};

// Implementation for structure-of-array layout

struct LBM {

   using CellData =double;

   double & f (int i, int k) {

     return lattice[k * dim.nelem + i];

   }

   // … implementation of collision models

};

Based on these access functions, the collision models can then be defined in an identical manner for both data layouts. Furthermore, the strategy for evaluating the index value and manipulating 3D indices is layout independent:

struct LBM {

    // … data definitions

    // Function i_to_xyz transforms a linear index

    // into a Cartesian coordinate triplet

    auto i_to_xyz (int i) {

      int iX = i / (dim.ny * dim.nz);

      int i_yz = i % (dim.ny * dim.nz);

      int iY = i_yz / dim.nz;

      int iZ = i_yz % dim.nz;

      return std::make_tuple(iX, iY, iZ);

    };

    void operator() (CellData& cell)

      {size_t i = &cell—lattice;

      // … implementation of collision-streaming

    }

}

The complete source code of the stlbm can be accessed under the terms of an open-source license under S1 Source Code.

2.6 Minimization of number of instructions

Two major strategies for improving the efficiency of an HPC code consist in (1) optimizing memory accesses (which we do by testing six different data access strategies), and (2) reducing the number of instructions executed by the code. In LB applications, the latter is typically achieved by unrolling the loops over populations, and regrouping and eliminating terms in the resulting arithmetic expressions. In the present work, two versions of the second-order BGK (BGK-W2) and TRT collision models were implemented. The first, referred to as the educational version, uses only few superficial optimizations and relies on the capabilities of the compiler for further in-depth improvements, in order to encourage readability and re-use of the code. The second, referred to as the unrolled version uses aggressive loop unrolling and regrouping of terms to achieve manual performance gains. Interestingly, while the unrolled code versions systematically yield substantially better performance than the educational ones on GPUs, the flagship AMD CPU used in the tests obtained better results with the educational code for the BGK model.

In the educational code versions, loop unrolling was applied to the computation of the macroscopic variable density and velocity, to eliminate the frequent multiplications with zero-valued components of the discrete velocities c_i. Furthermore, partial sums that were shared by the expressions for density and velocity were regrouped. While the loops for the computation of equilibrium and for executing the collision models were not unrolled, the similarity of the equilibrium term for opposite populations was explicitly exploited. For example, as the second-order weighted equilibrium (BGK-W2) for a direction k is written as (6) the equilibrium value in the opposite direction , defined by the relation , can be computed from E_k through the equation (7) In the unrolled code, all loops in the collision term were additionally unrolled, and redundant terms eliminated.

Regarding more complex collision models, their implementation relies on the computation of post-collision moments of interest, which are converted back to raw moments (RMs) in order to compute post-collision populations . This methodology allows for a drastic reduction of the number of instructions since it naturally (1) discards multiplication by zero-valued terms, and (2) highlights existing symmetries between opposite post-collision populations. By taking advantage of the latter property, one can reduce the number of instructions required to compute by first computing –as already done for in Eq (7). For instance, (8) (9) with c₀ = (−1, 0, 0) = −c₁₀ based on the velocity ordering convention used in the accompanying codes. Hence, (10) where () are post-collision RMs defined as (11) ω_pqr are adjustable relaxation frequencies, and only depend on ρ and u (their full expression can be found for all types of moment, e.g., in Ref. [36]). In addition, the computation of RMs (12) as well as other types of moments, is of paramount importance since it drastically impacts the performance of the code. As a matter of fact, the performance can be divided by a factor two or three if their general formulation is used. Consequently, aggressive optimizations (loop unrolling and regrouping of terms) were also applied to increase the performance of these collision models. For all models other than BGK-W2 and TRT, no “educational” version was produced.

As a sidenote, the multi-relaxation-time formulations considered in this study could be further optimized by limiting their generality in part, e.g. by imposing ω_pqr = 1 in Eq (11). The latter value is commonly used in the LB community for high-order (p + q + r ≥ 3) post-collision moments, as well as those related to bulk viscosity, as it leads to an increased stability for high-Reynolds number flow simulations and/or in under-resolved conditions. Such a choice would allow us to discard the computation of the corresponding moment (12) which remains time consuming despite our optimizations. More quantitative investigations regarding performance gains induced by the equilibration of high-order moments (and those related to the bulk viscosity) will be discussed in a future study. The interested reader may also refer to Refs. [39, 63] for a brief discussion of this aspect in the context of multiphysics flow simulations based on CM-LBMs.

Finally, an extra parameter has been added to all multi-relaxation-time formulations, so that the bulk viscosity can be freely adjusted. This is done by modifying the relaxation of second-order moments as follows [36, 64] (13) (14) (15) where and ω₊ = ω₋ + ω_ν. The kinematic viscosity ν, and its bulk counterpart ν_b, are related to these relaxation parameters through and . The lattice constant c_s is defined as [3].

2.7 Hardware platforms and compilers

The computers used to carry out the benchmark simulations are listed below, along with the compiler and compiler options that were used.

The compiler and compiler options were selected in informal comparative tests designed to identify the most efficient candidates. Except for the GPU versions compiled with nvc++, all binaries were linked with the -ltbb option to link with Intel’s Threading Building Blocks.

3.1 Benchmark case and code validation

The code of this article is applied to the benchmark case of a 3D flow in a lid-driven, cubic cavity, under the conditions of a steady, a laminar unsteady, and a turbulent regime. This test is used both to demonstrate the correctness of the code and to explore its efficiency on different hardware platforms depending on the implementation strategy and the LB collision model. This benchmark case, which can be found in numerous articles (see e.g. [67]), is defined as follows. An incompressible, non-thermal, viscous, Newtonian, and homogeneous fluid, the dynamics of which is described by the Navier-Stokes equations, is enclosed in a cubic domain. The 3D fluid-domain is defined by the open interval (] − L/2, + L/2[)³, and the boundaries consist of the closure of this domain. The velocity vector has three components u = (u_x, u_y, u_z), and the space positions are given by x = x, y, z. A Dirichlet boundary condition for the velocity is enforced on the boundaries, consisting of a no-slip condition on all walls except on the moving lid at x = + L/2. The velocity on the lid is constant with a value u(x = + L/2, y, z) = (0, U, 0). It should be pointed that this problem is mathematically not strictly well defined given the velocity discontinuities in the edges and corners of the lid. For this reason, a smoothly increasing profile is sometimes defined in these regions. This issue is however ignored in the present study because, as the subsequent results show, the LBM copes with this discontinuity without problem. The Reynolds number is defined as Re = UL/ν. For illustrative purposes, the flow field obtained by our solver in a steady and in a turbulent regime is shown in Fig 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Visualization of simulation results for the lid-driven cavity.

Left image: velocity norm and streamlines of the steady-state results at Re = 1000, on the plane defined by z = 0. Right image: vorticity norm on a selected surface at an instantaneous time step for the fully developed flow at Re = 10000.

https://doi.org/10.1371/journal.pone.0250306.g001

The problem is numerically solved with a homogeneous mesh of N × N × N nodes, which includes a wall node on all extremities of the domain. Thus, the cell spacing is δx = L/(N − 2). The time discretization was fixed by setting the reference lattice velocity u_LB ≡ U(δt/δx) to a constant value of 0.06. The Mach number, which in the present context of incompressible flow has a purely numerical significance, is therefore of . The discrete time step is consequently defined as δt = (u_LB/U)δx. The dimensionless time elapsed after N_iter iterations is t = N_iter δt(U/L) = N_iter t_c, with the dimensionless characteristic time being t_c = δt(U/L) = u_LB/(N − 2).

In spite of the simple setup, this problem poses some challenges due to the velocity discontinuity between the top lid and lateral walls, and because of the complexity of the flow field and boundary layers that develop at higher Reynolds numbers. This case is sometimes argued to pose substantial issues of accuracy and numerical stability for the LB method, which are for example overcome in [65] with help of a sophisticated boundary condition scheme. The results in this section show however that the LB framework of the stlbm project is sufficient to achieve satisfying results in all regimes, using either the standard BGK-W2 or the RR collision model.

Fig 2 plots the y-component of the velocity u_y along the center line y = z = 0 against the non-dimensionalized coordinate 2x/L, and the x-component of the velocity u_x along the center line x = z = 0 against the coordinate 2y/L. On the left image, at Re = 1000, the flow is steady. The flow is measured at an instantaneous time step after reaching stationary state and compared against simulation data of a spectral code. On the middle image, at Re = 3200 the flow is laminar and unsteady. The profile is averaged between dimensionless time values of t = 50 and t = 250 and is compared against experimental data published in [66]. On the right image, at Re = 10000 the flow is turbulent. The profile is averaged between dimensionless time values of t = 150 and t = 500 and is again compared with [66]. Excellent agreements are achieved in all three cases.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Profile of cubic cavity flow for two Reynolds numbers.

Simulation results (solid lines) in a cubic lid-driven cavity along two center lines. Left image: Re = 1000, resolution N = 256, collision model BGK-W2, comparison of the results with reference values (circles) from spectral simulations provided in [67]. Middle image: non-stationary laminar flow at Re = 3200 simulated with a resolution N = 256 and collision model BGK-W2, averaged between a dimensionless time t = 50 and t = 250. Right image: turbulent flow at Re = 10000 simulated with a resolution N = 400 and RR collision model, averaged between a dimensionless time t = 150 and t = 500. The curves of the middle and right image are compared with experimental data extracted from figures in [66].

https://doi.org/10.1371/journal.pone.0250306.g002

3.2 Parallel performance on different platforms

The case of a lid-driven cavity has been executed on different platforms to compare the performances. The for_each loops over the cells were executed with the par_unseq execution policy. Furthermore, reported values correspond to a median computed over 30 measurements, the latter being distributed other three days to reduce the impact on the benchmark of potential special events in the tested hardware or operating system. Each measurement consisted of 1000 warm-up iterations without time measurement, followed by 1000 benchmarked time steps. Finally, all test cases were first executed with a BGK-W2 collision model at a resolution of N = 128. A comparative study of the impact of the collision model on performances is also proposed in Section 3.3.2. As conventional in the field of LB, the performance is measured in terms of million lattice node updates per second (MLups), a value that identifies the number of lattice nodes, divided by one million, that are driven through a collision-streaming cycle in a second. To transpose this measurement in terms of time required to update one grid point per iteration (which is more commonly used in the Computational Fluid Dynamics community [68]), one simply needs to take the inverse of the number of lattice node update per second. As an example, a performance of 10 MLups corresponds to 0.1 μs/pt/ite.

The benchmark test was first executed on conventional CPUs with a modest core count. Fig 3 shows the performance obtained using the six different data structures on a 8-core AMD and a 8-core Intel CPU. The results should not be used to set up a competitive comparison between the involved CPU models, which differ in the year of market release, the clock rate, and other parameters. All in all, the results show overall comparable performance across models. The Intel CPU yields consistently best performances with an aos layout, while best performances on the AMD CPU are most frequently obtained with the soa approach, in agreement with the many-core AMD and the GPU architectures presented further down. This difference is mostly explained by the fact that the multi-core scaling performance of the code depends on the CPU and on the data structure. As shown below, the aos implementation strategy has better single-core performance than soa across architectures, but loses its edge over soa at a larger core count. While the AMD CPU reaches its best results with the AA-pattern, it is interesting to note that the swap-aos approach is the optimal choice on the Intel CPU, in spite of its apparent disadvantage due to the separated collision and streaming steps.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Performance of CPUs with a modest core count.

All simulations are executed with the BGK-W2 collision model and N = 128. On both CPUs, best performance was quite unpredictably obtained sometimes with and sometimes without loop unrolling. The plotted value shows therefore always the best of the two.

https://doi.org/10.1371/journal.pone.0250306.g003

In Fig 4, the performances obtained on five many-core platforms are compared, namely a 64-core AMD EPYC, a 48-core Intel Xeon, two consumer-class GPUs and a data-center GPU. It is not typical to include GPUs and CPUs on a same comparison plot, as CPUs are often shown to perform one or two orders of magnitude worse than GPUs. In this case, the common comparison is motivated by the fact that they are measured with the exact identical code, but also by the fact that in our comparison, and if the best performing implementation scheme is selected for each platform, the faster of the two consumer-class GPUs outpaces the CPUs by only roughly a factor four. To achieve best performance, the loops were unrolled on the GPUs and on the Intel Xeon, but were used without unrolling on the AMD CPU. Best performances are systematically obtained with the soa data layout, and the AA-pattern is the clear winner across all CPU and GPU platforms. For this reason, and also because of its lower memory needs, the AA-pattern (combined with the soa data layout) can be considered the most generally well performing choice for many-core platforms.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Performance of many-core CPU and GPU architectures.

All simulations are executed with BGK-W2 collision model and N = 128. For best overall performance, the simulations on the 64-core Epyc CPU were executed without loop unrolling, while the 48-core Intel CPU and the two GPUs used loop unrolling.

https://doi.org/10.1371/journal.pone.0250306.g004

The real life implications of the performance metrics of the code are illustrated for the case of the turbulent cavity flow, which is run with the RR collision model at N = 400 up to a dimensionless time of t = 500 (to gather sufficient statistics), thus requiring a total of 3.3 million iterations. The simulation does not make use of any subgrid-scale turbulence model, but it is stabilized by adjusting the relaxation parameter associated to the bulk viscosity to a value of 1. On the 48-core Intel Xeon CPU, the execution time without outputs is of 7 days and 15 hours, while the RTX 2080 Ti completes the same task in 2 day and 15 hours, and the V100 PCIE in just 1 day and 3 hours.

Table 2 shows the numerical values of the performances obtained on the two CPUs with the aos and the soa implementation. The performances obtained if the program uses only a single core are also shown. It can be seen that on both CPUs, the aos implementation is more performant on a single core, but the soa approach is preferable when all cores are exploited. It can be hypothesized that although in general, the aos approach makes better use of memory caching mechanisms, this advantage is lost when multiple cores compete for access to shared memory space, and cache coherence must be guaranteed. It is further remarked that both CPUs exhibit a comparable parallel efficiency, and that the Intel CPU compensates the lower core count with a stronger single-core performance.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance and speedup of AMD and Intel CPU with large core counts.

Detailed performance figures (in MLups) are provided for the cavity benchmark case with N = 128, BGK collision, and AA-pattern implementation, for the 64-core AMD EPYC CPU and the 48-core Intel Xeon CPU, for program executions using 1 core or all cores. The speedup is the ratio between these two values, and the efficiency is a measure of the scaling performance (100% efficiency stands for an ideal speedup).

https://doi.org/10.1371/journal.pone.0250306.t002

Table 3 provides the numerical values of the performances obtained on the two GPUs with the double-population implementation and the AA-pattern and compares them with a reference performance obtained on the same GPUs with a code extracted from the CuBoltz code, an open-source library that evolved from the work described in [69]. Written in the Cuda language, this code is written by one of the authors of the article and is carefully optimized for performance. The extracted code has been reshaped to solve the exact same problem as the present stlbm tests, as verified through comparative regression tests. The comparison shows that the generic stlbm code ends up only 21% (GTX 1080 Ti) respectively 17% (RTX 2080 Ti) below the performance of a hardware-specific, carefully optimized code. It can be concluded that the formalism of C++ Parallel Algorithms allows to achieve performances adequate for high performance computing, while at the same time offering an elegant and generic programming style.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance values of stlbm and the reference Cuda code CuBoltz for the lid-driven cavity at N = 128, with the BGK-W2 collision model.

Performance is indicated in MLups, followed in parenthesis by the ratio between this performance and the theoretical peak performance, computed for a memory bandwidth of 480 GB/s (GTX 1080 Ti), 616 GB/s (RTX 2080 Ti), and 900 GB/s (V100 PCIE).

https://doi.org/10.1371/journal.pone.0250306.t003

On GPUs, it is useful to compare these performance values against a theoretical peak performance obtained under the assumption that computations are negligible and the overall performance is limited solely by optimally coalesced accesses to central memory. With the two-population scheme and the AA-pattern, all populations need to be read and written once per iteration, which in double-precision representation requires two memory transactions of an 8-byte floating-point number per population and iteration (with the Swap algorithm, the number of memory transactions is doubled). For these two implementation schemes, the peak performance p (measured in LUPS) is therefore established by the formula (16) where bw is the memory bandwidth (in bytes per second). On Table 3, the ratio of measured performance over the theoretical peak performance is indicated in parentheses. It can be seen that the stlbm code reaches performances close to, and in some cases even distinctly superior to, 50% of the theoretical peak performance that can be expected on the respective GPUs. It is worth noting that these performances are obtained for lightly loaded GPUs, i.e., N = 128. By further increasing the load to N = 400, e.g., for the data-center GPU V100 PCIE, the ratio of measured performance is increased to 84%, 63% and 72% for CuBoltz, the two-population scheme and the AA pattern respectively. In any case, this study demonstrates both the quality of the stlbm algorithm, and the suitability of LB codes for GPU platforms.

3.3 Analysis of performance-impacting features

3.3.1 Loop unrolling.

Fig 5 compares the performance of the educational and the unrolled codes on the five tested many-core platforms for the BGK-W2 collision model. The results show that the GPUs experience substantial speed improvements from loop unrolling (the improvement is of 72%, 66%, and 34% on the GTX 1080 Ti, the RTX 2080 Ti, and the V100 PCIE respectively), while the AMD EPYC performs better with the educational code, which presumably allows more targeted compiler optimizations. Interestingly, when the educational code is used, and thus, readability is emphasized over hand-tuned optimizations, the consumer-class GPUs seem to lose their clear-cut performance edge over CPUs, as they are only 39% (GTX 1080 Ti) respectively 110% (RTX 2080 Ti) faster than the AMD CPU. The data-center GPU V100 PCIE on the other hand remains substantially faster even for this version of the code, with a speed up of 298% over the AMD.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Influence of loop unrolling.

All simulations are executed with the BGK-W2 collision model, with the AA-soa scheme, and with N = 128.

https://doi.org/10.1371/journal.pone.0250306.g005

3.3.2 Collision model.

Hereafter, the performance obtained with different LB collision models are investigated on the fastest consumer-level platform, the RTX 2080 Ti GPU. In all cases, the optimized codes with loop unrolling were used, and multi-relaxation-time formulations of collision models were considered.

Generally speaking, Fig 6 shows that the speed difference between the slowest and the fastest collision model remains within a factor two. This means that adopting modern collision models, e.g. to simulate high-Reynolds number flows in a stable and accurate manner, does not come at the expense of a drastic performance loss. Hence, LBMs based on these collision models remain particularly competitive as compared to Navier-Stokes solvers. Quantitatively speaking, single-relaxation-time formulations (BGK-W2 and BGK-NW4) are the fastest. For the latter model, more redundant terms can be pre-computed, hence its better performance. Regarding collision models based on multiple relaxation times, it seems clear that increasing the number of moment space transitions automatically increases the number of floating point operations, hence a decrease of performance. As an example, the CHM-LBM requires two intermediate moment transformations before computing post-collision populations from their RMs, i.e. transformations from CHMs to HMs and from HMs to RMs. The same goes for the K-LBM, whereas the HM-LBM only requires a single moment transformation (from HMs to RMs). In addition, even if the RR-LBM is also based on a collision applied in the HM space, it is faster than the HM-LBM. This is explained by the fact that it only requires the computation of second-order HMs, and recursive computations of high-order HMs only add a few floating point operations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Performance of different LB models.

All simulations are executed with the AA-soa data structure at N = 128 on the RTX 2080 Ti GPU.

https://doi.org/10.1371/journal.pone.0250306.g006

In the end, the present study confirms that the good performances obtained with the stlbm implementation strategy are consistent across all collision models.

3.3.3 Domain size.

Fig 7 shows the performance achieved by stlbm on the fastest consumer-level hardware platform, the RTX 2080 Ti GPU, using different domain sizes, with increments of 1 for the number of cells N in every direction. The performance figures are shown for the AA-soa and double-population-soa implementation strategies, and for the BGK-W2 and RR collision models, to test both a simple and a complex model. In general, the performance increases up to a resolution of roughly 100 × 100 × 100, and stays roughly constant afterwards. The double-population implementation further shows a drop of performance close to N = 90, which is not further explained at this point. This investigation finally leads to the conclusion that the performances of the double-population scheme and the AA-pattern are very similar, and that the slight superiority of the double-population approach observed in Sec 3.2 would be less clear-cut or even inexistent at a different domain size or with a different collision model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Dependence of domain size on performance on GPU.

Left: BGK-W2; Right: RR. Simulations are run on an RTX 2080 Ti GPU.

https://doi.org/10.1371/journal.pone.0250306.g007