1 Introduction

Millimeter-wave (mmWave) communication in the 30–300 GHz band is widely recognized as a cornerstone physical-layer technology for sixth-generation (6G) wireless systems because the availability of large contiguous bandwidth fundamentally enables ultra-high data rates and supports spectrum-intensive services envisioned for future networks [1]. At the network architecture level, 6G is further expected to accommodate massive device connectivity and heterogeneous traffic demands across highly diverse deployment scenarios that extend beyond conventional terrestrial cellular infrastructures [2].

Reconfigurable intelligent surfaces (RISs) have emerged as a key enabler of this paradigm shift by transforming the wireless environment into a programmable entity [3]. Comprehensive surveys have established RIS as a fundamental building block of smart radio environments, capable of reshaping wireless propagation to support emerging 6G services [4,5]. Early RIS research primarily focused on enhancing signal strength and spectral efficiency. More recent studies reposition RISs as multi-functional and reliability-critical platforms that integrate communication, sensing, and wireless power transfer [6,7]. In parallel, advanced RIS architectures, including multi-layer, refracting, and self-powered designs, have been investigated to support satellite-terrestrial and space-air-ground integrated networks [8–10]. For instance, multi-layer stacking RIS receivers have been studied for high-altitude platform (HAP)-enabled simultaneous wireless information and power transfer (SWIPT) networks [8], while self-powered adaptive RIS designs have been considered for secure satellite-terrestrial integration [10]. Refracting RIS has also been investigated for hybrid satellite-terrestrial relay systems [9]. More recently, air-driven seamless and massive access has been discussed for space-air-ground integrated networks, where aerial platforms (e.g., unmanned aerial vehicle (UAV) and HAP tiers) play a coordinating role to enable wide-area continuity of access and large-scale user connectivity across heterogeneous layers [11]. These developments are consistent with broader 6G visions that emphasize scalable and intelligent connectivity across heterogeneous network layers [11].

Despite their promise, RIS-assisted mmWave systems face severe physical-layer challenges. MmWave propagation suffers from high path loss, limited diffraction, and strong sensitivity to blockage, which significantly degrades non-line-of-sight (NLoS) coverage. Recent path-loss measurements and channel modeling studies reveal that RIS-assisted mmWave links exhibit pronounced geometry-dependent attenuation and user-specific disparities, particularly in indoor and outdoor deployments [12,13]. Furthermore, statistical characterizations of RIS-assisted channels demonstrate non-trivial SNR distributions across near- and far-field regimes, which further intensify rate imbalance among users [14]. These model- and measurement-driven observations indicate that channel disparity in RIS-assisted mmWave systems is often structural rather than incidental, making fairness and service reliability first-order design challenges rather than secondary performance metrics.

Existing RIS-assisted multi-user MIMO designs predominantly adopt sum-rate or weighted sum-rate maximization frameworks, often solved via WMMSE, semidefinite relaxation (SDR), or alternating optimization methods [15–17]. Although these throughput-centric approaches improve aggregate spectral efficiency, they systematically favor users with strong composite channels and may severely degrade the service quality of cell-edge or blocked users. This limitation is particularly pronounced in RIS-assisted mmWave systems, where blockage and path-loss disparities are inherently amplified by passive reflection mechanisms. Furthermore, many existing works rely on fully-digital precoding assumptions, which are impractical at mmWave frequencies due to the prohibitive cost and power consumption associated with large numbers of RF chains.

Hybrid beamforming (HBF) architectures mitigate these hardware limitations by reducing RF-chain requirements. Fully-connected (FC) architectures provide high beamforming flexibility at significant hardware cost, whereas non-overlapped sub-array (NOSA) architectures reduce complexity at the expense of spatial degrees of freedom [18–21]. The overlapped sub-array (OSA) architecture generalizes both designs by allowing antenna sharing across RF chains, thereby improving spatial flexibility without incurring the full hardware burden of FC architectures [22,23]. However, when RIS-assisted transmission is combined with OSA-based HBF and discrete phase constraints, conventional sum-rate-driven optimization becomes insufficient to guarantee user fairness or service reliability.

Geometric mean (GM) rate maximization provides a principled mechanism to address this fairness and reliability gap by explicitly balancing individual user rates [24]. Unlike sum-rate objectives, which are insensitive to individual rate collapse, GM-rate penalizes vanishing user rates and enforces balanced performance across all users within a single transmission interval. Moreover, the GM-rate achieves a more favorable trade-off between fairness guarantees and overall performance maximization compared to conventional fairness paradigms, such as the max-min fairness objective or proportional rate constraints. Unlike the max-min objective, which heavily favors fairness at the expense of efficiency, or proportional rate constraints, which incur additional computational complexity, the GM objective offers a tractable and scalable solution that implicitly improves fairness without introducing any associated complex constraints. This property is particularly important in quasi-static RIS-assisted mmWave systems, where limited scheduling diversity and reliance on reflected paths prevent fairness from being achieved through long-term averaging. Under OSA-based hybrid beamforming, GM-rate aligns with reduced spatial degrees of freedom by prioritizing robustness and service continuity over peak throughput.

The above design choices are not independent but are closely connected through both fairness and implementation considerations. While GM-rate maximization addresses the fairness limitations of conventional throughput-oriented designs by preventing rate collapse among users with weak composite channels [24], its practical realization in RIS-assisted mmWave systems requires a hardware-efficient beamforming architecture. The OSA-based hybrid beamforming structure provides an attractive compromise between the beamforming flexibility of fully-connected architectures and the reduced hardware complexity of non-overlapped sub-array designs [22]. However, the resulting system introduces strongly coupled digital precoding, analog beamforming, and RIS phase variables under unit-modulus and discrete phase constraints. These characteristics naturally motivate the use of Riemannian manifold optimization, which enables direct optimization over the feasible set without relying on high-dimensional relaxation methods [25]. Furthermore, because the resulting optimization problem remains highly non-convex and involves multiple coupled variable blocks, a BCD framework is adopted to decompose the original problem into tractable subproblems while preserving monotonic objective improvement during the iterative optimization process [26]. Together, these design choices establish a fairness-oriented, hardware-consistent, and computationally tractable optimization framework for RIS-assisted mmWave communications.

In parallel, artificial intelligence (AI) and deep reinforcement learning (DRL) techniques have been actively explored for RIS and hybrid beamforming optimization [17,27]. Although these data-driven approaches exhibit strong adaptability in complex environments, they typically require large training datasets, lack interpretability, and do not explicitly optimize fairness objectives. Model-based optimization frameworks therefore remain attractive for fairness-critical systems, as they provide transparent performance control, interpretable optimization updates, and direct incorporation of hardware constraints. Accordingly, this work adopts a model-based design philosophy that complements, rather than replaces, AI-based RIS optimization. Table 1 highlights how the proposed framework addresses fairness objectives under practical hardware constraints when compared with classical and AI/DRL-based methods.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Comparative analysis of HBF optimization frameworks for RIS systems.

https://doi.org/10.1371/journal.pone.0354965.t001

As summarized in Table 1, the proposed framework occupies an intermediate position between conventional optimization-based and learning-based beamforming designs. Compared with SDR- and WMMSE-based approaches, it directly accommodates the OSA hybrid architecture and discrete RIS phase constraints without relying on semidefinite relaxation or matrix lifting, while directly handling the structured constraints imposed by the hardware architecture. Compared with AI- and DRL-based methods, the proposed approach does not require offline training and explicitly incorporates user fairness through GM-rate maximization. Additionally, the proposed framework inherits several limitations common to model-based optimization, including its dependence on CSI availability and convergence to a stationary point of a non-convex problem. Moreover, its iterative nature makes it more suitable for scenarios where channel variations occur on a slower time scale than the optimization process. Therefore, the proposed framework should be viewed as a fairness-oriented and hardware-consistent model-based solution that complements, rather than replaces, existing analytical and learning-based approaches.

Meanwhile, OSA-based RIS hybrid beamforming is particularly suitable for practical mmWave deployments where blockage is a major concern, such as indoor hotspots in shopping malls, airports, and large office buildings. In these scenarios, severe penetration loss and frequent human blockage degrade mmWave links. Compared to fully connected architectures, the OSA structure reduces the number of phase shifters and RF interconnections, thereby lowering hardware complexity and power consumption while maintaining satisfactory data rates. Moreover, the control and calibration overhead remains manageable, as RIS phase updates are performed over a low-rate control link at the channel coherence time scale, and with discrete phase shifts of limited resolution, the signaling payload scales linearly with the number of elements and quantization bits.

Motivated by these observations, this paper investigates a downlink multi-user mmWave system assisted by an RIS, where the base station employs OSA-based hybrid beamforming and the RIS operates with discrete phase shifts, with a particular focus on fairness-oriented joint precoder and phase optimization under practical hardware constraints. The proposed framework directly maximizes the GM-rate to ensure fairness while explicitly accounting for hardware limitations imposed by hybrid architectures and discrete phase control. Unlike fully-digital GM-rate designs [24], this work develops a realizable OSA-consistent hybrid beamforming solution.

The main contributions of this paper are summarized as follows:

1. Conceptual novelty: We formulate a fairness-driven GM-rate maximization problem for RIS-assisted mmWave systems with OSA-based hybrid beamforming and discrete RIS phases, thereby extending the applicable configuration concept and explicitly addressing the reliability gap inherent in sum-rate-centric designs prevalent in the existing literature.
2. Methodological differentiation: We propose an OSA-consistent Riemannian manifold factorization that maps the fully-digital GM-optimal precoder benchmark into a realizable hybrid analog-digital structure without resorting to matrix lifting or semidefinite relaxation. We apply a dynamic GM-based weighting approximation in BCD that differs from fixed weights in sum-rate formulations. Furthermore, instead of directly applying the Riemannian manifold conjugate descent, we iteratively search for the optimal digital beamforming matrix via a closed-form expression while reformulating the optimal analog beamforming matrix as a manifold factorization problem.
3. Practical advantages: We develop an interference-aware warm-start strategy that stabilizes the power-constrained precoder update and accelerates convergence by up to 18.18% compared with random initialization. Extensive simulations demonstrate that OSA-based architectures with moderate RF-chain counts and low-resolution (3-bit) phase shifters at both the BS and RIS can closely approach continuous-phase performance, validating the practicality of the proposed framework.

Novelty Statement. To the best of our knowledge, this work represents the first principled attempt to systematically bridge three previously disconnected aspects in RIS-assisted mmWave systems, namely fairness-oriented performance optimization by leveraging a GM objective, practical OSA-based hybrid beamforming architectures, and discrete phase control within a unified Riemannian manifold optimization framework. In terms of the conceptual system model, unlike the prevailing literature that prioritizes sum-rate maximization or relies on idealized fully-digital precoding assumptions [15–17], the proposed framework directly addresses the structural rate imbalance intrinsic to RIS-aided mmWave propagation with the influence of hybrid beamforming, where blockage sensitivity and geometry-dependent attenuation fundamentally limit user-level reliability [12,13]. By integrating GM-rate maximization with an OSA-consistent hybrid realization under discrete phase constraints, this work fills a critical research gap and paves a realizable methodology toward fairness-critical RIS-assisted deployments under stringent hardware limitations [24].

This journal article substantially extends the authors’ preliminary [28] through several conceptual and methodological advancements rather than incremental refinements. Specifically, the journal version introduces: (i) a refined fairness-oriented optimization formulation that explicitly targets user-rate imbalance instead of aggregate throughput maximization, which was not considered in the conference version [24]; (ii) a novel OSA-consistent Riemannian manifold factorization mechanism that rigorously maps the fully-digital optimum into a realizable hybrid analog-digital structure with discrete phase constraints [22,23]; and (iii) an explicit theoretical and practical positioning with respect to emerging AI- and DRL-based RIS optimization approaches [17,27]. Collectively, these extensions establish a more comprehensive and tightly grounded optimization framework with enhanced system-level relevance beyond the scope of the original conference contribution.

2 System model and problem formulation

2.1 System mModel

We consider an RIS-assisted downlink millimeter-wave (mmWave) communication system, where a single base station (BS) equipped with M antennas serves K single-antenna users with the aid of one RIS comprising F passive reflecting elements, as illustrated in Fig 1. The BS adopts a subconnected hybrid analog/digital (A/D) beamforming architecture with N RF chains, where each RF chain is connected to a subset of consecutive antennas with . This subarray-based structure effectively reduces hardware complexity while preserving sufficient beamforming flexibility. Due to the severe blockage and scattering characteristics of mmWave propagation, the direct BS–user links are assumed to be weak and are modeled as scattered paths, while the RIS establishes a dominant reflected link to enhance the received signal quality at the users. The Table 2 summarizes all key symbols, including variables, vectors, matrices, and sets, together with their dimensions and concise descriptions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. List of symbols and notations.

https://doi.org/10.1371/journal.pone.0354965.t002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. System model of the RIS-assisted hybrid beamforming (HBF) mmWave communication system.

https://doi.org/10.1371/journal.pone.0354965.g001

Let be the complex-valued information symbol designated for user k, where . We assume that the signals are mutually independent and normalized to unit power, such that for all . Initially, each signal is weighted by a digital beamforming vector to map the signal onto N RF chains. These digitally precoded signals are then summed across all users. This process results in a composite baseband signal vector of dimension . The composite baseband signal, formed by digital precoding across K users, is subsequently processed by an analog beamforming matrix , where column represents the analog precoder associated with RF chain n, for . This analog precoding stage maps the RF-chain outputs to the physical antenna elements. Under the subconnected architecture, each RF chain is connected to a subset of consecutive antennas. Accordingly, the analog beamforming vector contains nonzero entries, denoted by for , corresponding to the antennas assigned to RF chain n. Each is implemented via a phase shifter that applies a fixed-magnitude complex weight to the signal, satisfying .

In this work, we assume discrete phase control for all analog coefficients. The set of allowable phase shifts is defined as

(1)

where Q₁ denotes the number of control bits per phase shifter. In the special case where , the phase shifts become continuous over .

The structure of depends on the array architecture. In the subconnected case, each column contains exactly non-zero entries corresponding to the antennas connected to its RF chain, while all other entries are zero. The structure of is illustrated as

(2)

where recalls the complex-valued phase shift applied to antenna m by RF chain n, for . All non-zero entries satisfy the unit-modulus constraint, meaning , and are chosen from a discrete phase set .

For the general OSA design, features vectors that contain shifted zeros of size . This design naturally encompasses two special cases: the FC case, where the constraint is , , and the NOSA case, where . As a result, has no zeros or becomes a block diagonal, respectively. Consequently, the total transmit power of the BS is given by

(3)

The RIS is connected to the BS through an RIS control link for transmission and information exchange. Denote the set of all RIS unit cells. The response matrix at the RIS is formulated by

(4)

where and are the amplitude reflection coefficient and the phase shift of the f-th unit cell, respectively. Since we adopt the passive RIS configuration, we have , . Denote as the set of all possible phase shifts for the RIS reflection coefficients, given by:

(5)

where Q₂ is the number of control bits for each RIS element. The special case of corresponds to the continuous phase shifts. We consider the scenario that the BS-user direct link is blocked, and thus the direct path can be ignored. The signal power reflected two or more times is much lower than that reflected just once due to the high free-space path loss. Thus, we ignore the power of the signals that are reflected by the RIS more than once. In addition, we consider that full channel state information (CSI) is available at the BS, and all channels experience quasi-static flat-fading. It is important to highlight that the assumption of CSI at the base station is widely used in HBF for mmWave communication studies [20,24,29], as it enables tractable optimization and serves as a benchmark for performance evaluation. In this work, CSI refers to either perfect or reliably estimated values obtained through standard pilot-based training methods [30], which are practical in slow-fading or quasi-static conditions. Although this assumption simplifies the analysis, it provides meaningful upper bounds for system design. Suppose that is the channel matrix from BS to RIS, is the channel vector from RIS to user k. Then the signal received by the user k can be represented as

(6)

where is the additive white Gaussian noise at the receiver of user k with zero mean and variance . The achievable rate of user k can be expressed as:

(7)

where is the signal-to-interference-plus-noise ratio (SINR) at the k-th user, which can be calculated by

(8)

2.2 mmWave channel model

We adopt the widely used narrowband clustered channel model for mmWave communications. Specifically, the channel matrix between the BS and the RIS can be written as [20]:

(9)

Here, denotes the number of scattering clusters, denotes the number of rays in each cluster, and denotes the channel coefficient of the l -th ray in the i -th propagation cluster. Moreover, and represent the receive array response vectors of the RIS and the transmit array response vectors of the BS, respectively, with and representing azimuth and elevation angles of arrival at the RIS (or departing from the BS). The channel vector between the RIS and the k -th user can be represented as

(10)

Here, , , , , and are defined similarly as above. We assume that the uniform planar array (UPA) structure is adopted at both BS and RIS. Consequently, the array response vector can be denoted as:

(11)

where , is the signal wavelength, d is the antenna or unit cell spacing which is assumed to be half a wavelength, and . A₁ and A₂ represent the number of rows and columns of the UPA in the 2D plane, respectively.

2.3 Problem formulation

Existing works in the literature often use the sum-rate method and the max-min method to quantify the quality of service (QoS) of the network, each of the formulation in favor of the overall rates or resource fairness across users, respectively. To balance this trade-off between total QoS and resource fairness, a proportional fairness constraint is introduced as an alternative method, at the cost of additional complexity of non-convex constraints [31]. Thus, the recent GM rate maximization method overcomes this challenge by considering a different objective function [24]. Our target is to tackle the following problem by jointly optimizing the digital beamforming matrix and the analog beamforming matrix at the BS, as well as the overall response matrix at the RIS, to maximize the geometric mean of users’ rates:

(12a)(12b)(12c)(12d)

From problem (12), we can observe that the GM rate maximization problem is highly non-convex due to the nature of the geometric mean objective function (12a) and the coupling between the three variable matrices of the digital beamformer, RIS reflection coefficient, and the discrete phase shifters of the analog beamformer (12c). To circumvent the difficulty of finding a global optimal solution, we instead adopt a sub-optimal approach using the BCD method. This involves iteratively solving a relaxed version of (12) w.r.t. one variable matrix at a time, while keeping all others fixed.

3 RIS-assisted hybrid beamforming solution for GM rate optimization

In this section, we introduce an efficient methodology to handle the GM rate maximization problem (12) leveraging two-block iterative algorithms. Specifically, we first handle the discrete phase shifts by adopting the projection method that relaxes the discrete phase shifts of analog beamforming and RIS coefficients to continuous ones, then address the relaxed problem with the proposed algorithms, and project the obtained continuous solution back to the discrete set. We introduce an auxiliary variable with and solve (12) iteratively w.r.t. and , respectively while keeping the other fixed. Thus, we consider a relaxation version of (12) as

(13a)

(13b)

(13c)

We note that the maximize the objective function of (13) is equivalent to minimize subject to (13b) and (13c). Notice that the function is the composition of the convex function and the non-convex functions . We then consider the linearization of around , a feasible point that is found from the iteration. Thus, we apply a first-order Taylor approximation at each BCD iteration , transforming problem (13) into the form where denotes the rate of user k evaluated at . Since we have , we can generate the next feasible point from the following equivalent problem:

(14a)

(14b)

where . Here, and are the objective function of (13) and the achievable rate of user k, evaluated at – the feasible point of (13) generated by iteration , respectively. This first-order Taylor approximation reveals a key distinction between the two optimization approaches: in the maximizing sum rate problem, the weight coefficients remain fixed within each iteration, whereas in the geometric mean approach, the coefficients are dynamically modified after every variable update.

Moreover, we note that the linearized problem (14) is still not a convex optimization problem since the matrices and are still coupled in the objective function (14a) while the feasible domain remains non-convex due to the unit-modulus constraint (13c) of passive RIS. This motivates us to employ the BCD technique to decouple the optimization variables. Specifically, the overall optimization process, , is broken down into two sequential sub-iterations consisting of the FD beamforming iteration and the RIS phase update iteration.

3.1 FD beamforming iteration

At each FD Beamforming iteration , we generate a matrix such that the objective function improves: . Following [32], the below inequality holds for all and

(15)

with , , and are determined by , , , , and .

The lower bound of function given by (15) is now concave quadratic. Thus, we can replace each user’s rate in (14a) with its lower-bound counterpart. By using the Lagrangian multiplier method, we can derive the following closed-form solution of the FD Beamforming iteration as follows:

(16)

where , and is a constant selected to satisfy:

To compute , we apply the bisection method to solve the above nonlinear equation.

3.2 Programmable reflecting elements’ descent iteration

Similar to section 3.1, the goal is to determine the next iterative point, , such that it satisfies the improvement condition: . Since the diagonal matrix contains unit-modulus elements, maximizing (13a) w.r.t. is equivalent to maximizing w.r.t. the individual phase shift . Following the approach in [24], the original objective function (14a) can be transformed into the following lower bound

(17)

where and are constants that are independent of , with can be calculated as . We call is the “phase matrix decomposition” coefficient of (14a) at the point , which is given as:

where is the matrix with 1 on the main diagonal at entry (f,f), and are coefficients at user k to compute accordingly. Note that we can represent as , hence introducing the term “phase matrix decomposition.” Similarly, we call the “convex coefficient matrix,” as it results from coefficients of a convex approximation, which is calculated by:

Lastly, we introduce the “convex approximation matrix” , which is the convex lower bound of (14a) at the point :

Noting that for any affine combination of functions of the form , each function is maximized at since . The closed-form solution of (17) is obtained by

(18)

In summary, by apply iteratively (16) and (18) until the objective function (13a) converges, we can find an sub-optimal point for problem (13). The summary of the iterative procedure is given in Alg. 1.

Algorithm 1 Proposed Algorithm for problem (13)

Require: , , , ,

1: while do

2:   Update based on (16)

3:   Update based on (18)

4:   

5: end while

Ensure: Optimized variables ,

3.3 Hybrid beamforming derivation by matrix factorization

In this section, we revisit (12) to derive its optimal solution from . Notice that by relaxing the discrete constraints (12c) and (12d), is the optimal solution for (12). To find the optimal analog and digital beamforming matrices, , from the optimal fully-digital beam , a process known as splitting the fully-digital beam. We consider the following matrix factorization problem [33]

(19a)(19b)(19c)

Since the variables and are coupled in (19a), we again employ the BCD framework. This framework leverages an alternating procedure that, in each iteration , solves for a sub-optimal solution to (19) by combining the Least Squares (LS) solution and Riemannian manifold optimization [34]. Specifically, is determined based on using the following normalized Least Square (LS) solution

(20)

which satisfies the power constraint (19b). Consequently, the remaining task is to solve while keeping fixed. Define

(21a)(21b)(21c)(21d)

where contains non-zero elements of column n of and is the row vector of a matrix with zeros everywhere except digital beamforming element on its diagonal for all and . We note that the shifted zeros of each column vector in are exactly of size . Then, is the solution to the following problem:

(22)

where is a manifold representing the feasible set of . denotes the m-th element of . Here, is a complex circle manifold with the corresponding tangent space , where represents the vector of zeros. The problem (22) is known as a Riemannian manifold minimization problem which can be efficiently solved by the manifold optimization technique. Specifically, we adopt the Riemannian Conjugate Gradient (RCG) algorithm. In general, the procedure of the RCG algorithm is similar to the infamous Gradient Descent Algorithm [20,34], as we first need to calculate the gradient of the objective function:

(23)

Remark on discrete phase activation: At each iteration of Algorithm 2, the continuous-phase solutions of both the RIS and the analog beamformer are immediately projected onto their corresponding discrete phase sets. Specifically, each RIS phase coefficient is mapped to the nearest element in , and each non-zero entry of the analog beamforming matrix is projected onto the discrete phase set . As a result, the discrete phase constraints are enforced per iteration, ensuring that all intermediate solutions remain hardware-feasible throughout the alternating optimization.

It is worth noting that the above discrete phase projection is applied after each continuous update obtained via Riemannian optimization. Each iteration of the RCG algorithm itself consists of four principal operations, namely, computing the Riemannian gradient, performing vector transport to map the previous search direction onto the tangent space at the new point, updating the conjugate search direction using the transported vector and current gradient, and projecting the updated point back to the manifold. With the key steps introduced above, the computation process for solving the problem (22) in each iteration i is expressed as follows:

(24)(25)(26)(27)

It is important to note that the proposed algorithm operates in an iterative manner, and without an appropriate initialization strategy, it may suffer from slow convergence and significant computational complexity. To overcome this, we initialize the analog beamforming by

(28)

where is taken from the first N eigenvectors of , and represents the phase of the (i, j)-th entry of . This initialization approach is based on the analog precoding scheme in [34]. Toward this end, the procedure for solving problem (12) is summarized in Alg. 2 and the summarizing flow-diagram is given in Fig 2. From the flow diagram, the overall optimization can be interpreted as an approximate decomposition of a highly coupled non-convex problem into a convex subproblem iteratively. In the fully digital design stage, the update for is based on Lagrangian duality and the optimal phase of an affine sum. In the hybrid beam decomposition stage, we try to minimize the Euclidean distance between the optimal FD beam and the actual HBF beam, leading to the LS solution for and RCG solution for . Together, these steps progressively enforce unit-modulus constraints while preserving the performance-oriented objective, which results in a tractable optimal solution.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Flow diagram of the proposed algorithm for solving problem (12).

https://doi.org/10.1371/journal.pone.0354965.g002

Remark on the local optimum of proposed method: We note that the solution obtained by the BCD technique always converges to a local optimum as the sequence created by iteratively optimizing the objective is a monotonic sequence. However, no guarantee of global optimality can be proved, as the original problem is strongly nonconvex and mixes integer variables. Hence, our proposed algorithm only guarantees obtaining a local optimum.

Algorithm 2 Proposed Algorithm for problem (12)

Require: , ,

1: Initialize and by (29)

2: Acquire and based on Alg. 1

3: while do

4:   Update by applying (20)

5:   Update based on vectorization (22) and apply RCG algorithm (24) until convergence

6:   

7: end while

8: Remap each RIS phase shifter to the nearest element in

9: Remap each non-zero element of to the nearest element in

Ensure: Optimized variables and

Complexity Analysis of Alg. 2: To analyze the computational complexity of the proposed joint hybrid beamforming and RIS phase shifts design, it is essential to compute the complexity of Alg. 1 along with the complexity of the RCG algorithm. We note that for any dense square matrix , the computational complexity to calculate its inverse is [35]. We also observe that the matrix product between two matrices and is . Thus, the complexity of (16) is for a fixed value of . However, to select the constant , we need an additional iteration complexity of , where is the target accuracy of the bisection method, since this method is [36] and at each iteration we need to calculate the inverse and matrix multiplication of . Next, we consider the complexity of updating . We recall the definition of as (18), which is . The complexity of a two-block BCD algorithm has a complexity of [37]. Thus, the complexity of Alg. 1 is . Similarly, we show the complexity to compute to be while the RCG algorithm for update is , where is the target accuracy of RCG. Since our beam decomposition leverages two-block BCD, its complexity is . Hence, the total complexity of Alg. 2 is .

3.4 Effective initialization for GM rate optimization problem

Given (16), we assume that the parameter can be found by applying the bisection method, which requires prior knowledge of the interval of opposite sign, i.e., the interval that the function exhibits a sign change. As such, we wish to find an appropriate initial value so that we could ensure is within acceptable limits. We first initialize , which corresponds to configuring the RIS as a passive reflector, changing only the direction of the incoming signals without conducting any phase shift. Consequently, the initialization of is given in (29) such that

(29)

Here, we denote . is a matrix with each element independently drawn from the circularly symmetric complex Gaussian distribution, i.e., , and is a parameter to guarantee constraint (13b). The primary motivation for this formulation is to achieve uniform power allocation across users while simultaneously suppressing multi-user interference. In particular, considering the linear independence of the user channels , the matrix operation simplifies such that . This condition effectively nullifies all inter-user interference (IUI), achieving perfect isolation between the K users.

4 Performance evaluation

In this section, we present the performance evaluation of our proposed algorithms through system-level simulations. We model an RIS-aided multiuser millimeter-wave (mmWave) communication system operating at 28 GHz [20]. The BS is located at (0,0) in the two-dimensional (2D) Cartesian coordinate system and is equipped with a UPA structure, totaling M = 100 antennas and N = 10 RF chains. The thermal noise power is set to dBm. The overall system layout is illustrated in Fig 3. The system incorporates an RIS, positioned at (d_RIS, 15). This surface is equipped with a uniform rectangular array, totaling 100 reflective unit cells. The K users are modeled as being uniformly and randomly distributed in a circular area with a 5-meter radius, centered at (50,0). For the mmWave channel model, we adopt a clustered channel approach with clusters and rays per cluster [20]. The azimuth and elevation angles of arrival and departure are modeled using the Laplacian distribution with an angle spread of 10 degrees. The complex gains and follow a complex Gaussian distribution , where the path loss factor PL(d) is defined by [20]

(30)

where , , , and dB. Unless specified otherwise, the remaining system parameters are set as follows: K = 3, , , dB, . We perform simulations based on averaging over 500 independent channel realizations.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Simulated RIS-assisted hybrid beamforming (HBF) scenario.

https://doi.org/10.1371/journal.pone.0354965.g003

Fig 4 illustrates the convergence behavior of the proposed Alg. 1. Specifically, Fig 4(A) compares the convergence rate achieved using our proposed initialization against a random scheme. For the random baseline, we initialized the matrix using complex normal distribution for each element, then normalized it to set the starting point as . As shown in Fig 4(A), the random scheme starts with a very poor geometric mean value and takes more than 2000 iterations to converge to a stationary point. At the same time, the proposed initialization outperforms the initial value from the beginning of the algorithm, while achieving a faster convergence rate within the first 800 iterations and improving the objective value by 18.18%. This significant difference arises because the limit point of the random scheme is merely a random stationary point with no guarantee of global optimality, whereas our proposed approach is designed to effectively cancel interference degradation, thus achieving a superior solution. Fig 4(B) illustrates the convergence speed of the different power usage parameters . The convergence speed is directly proportional to , which is expected since the power resource is utilized from the initial step of the iterative procedure. Critically, we observe that the iterative procedure converges to the same limit point irrespective of the initial value of .

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Convergence behavior of Algorithm 1.

(A) Random initialization versus proposed initialization. (B) Effect of the parameter .

https://doi.org/10.1371/journal.pone.0354965.g004

Fig 5 illustrates the overall performance across different beamforming configurations, where we assume . Specifically, Fig 5(A) shows that the FD scheme achieves the highest performance gains. This result is expected because the FD configuration dedicates one RF chain to each BS antenna, thereby fully exploiting the spatial degrees of freedom and maximizing the system capacity. Furthermore, the FC hybrid architecture consistently outperforms the NOSA approach even when the number of quantization bits is limited. This advantage arises from the higher density of non-zero elements in the FC connection matrix, which provides greater flexibility for the hybrid beamforming decomposition to approximate the optimal FD beamformer. The results indicate that approximately 8 bits of phase quantization are sufficient to achieve a favorable trade-off between hardware complexity and performance for the FC architecture.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. GMSE versus transmit power .

(A) Fully connected (FC) hybrid beamforming. (B) Non-overlapped subarray (NOSA) hybrid beamforming.

https://doi.org/10.1371/journal.pone.0354965.g005

Fig 5(B) presents the corresponding results for the NOSA architecture. In this case, the performance is noticeably lower due to the reduced spatial flexibility caused by the subarray structure. When the number of RF chains , each RF chain simultaneously controls the phases of multiple antennas, which restricts the beamforming resolution and limits the achievable beam decomposition gain. Consequently, the NOSA architecture experiences a larger performance gap compared with the FD benchmark. Nevertheless, as the number of quantization bits increases, the system performance gradually saturates and approaches the continuous phase-shift case . These results suggest that using approximately 3 bits of phase quantization at both the BS and RIS provides a practical trade-off between hardware implementation complexity and achievable performance for the NOSA architecture.

Fig 6 illustrates the relationship between the surrogate objective and user fairness. Specifically, we evaluate the worst-user rate and the median-user rate under the same simulation settings to verify that the GM design effectively improves the performance of the weakest user relative to FC and NOSA. As shown in Fig 6(A), the performance curve of each target user aligns with the geometric mean curve for both FD and FC architectures, confirming the fairness traits of the GM design. However, Fig 6(B) shows a considerable separation between the three benchmarks. This phenomenon arises from the trade-off between architectural complexity and QoS performance in the NOSA scheme, which provides a limited degree of freedom for analog beamforming to suppress inter-user interference. Nevertheless, the GM objective significantly improves the worst-user rate compared to median and dominant users for the NOSA case in the low SNR region while striking a balance between overall performance and maintaining an acceptable performance for the worst-user rate in the high SNR region, demonstrating the ability to reduce rate disparity across users. In detail, the worst-user rate maintains a stable performance around 0.8 bps/Hz for increasing levels of P_max, while the median-user rate and GM rate increase from 0.85 to 0.9 bps/Hz. Nevertheless, the GM objective significantly improves the worst-user rate compared to the median and dominant users for the NOSA case in the low SNR region, while striking a balance between overall performance and maintaining an acceptable worst-user rate in the high SNR region, thereby demonstrating its ability to reduce rate disparity across users. Concretely, the worst-user rate remains stable at approximately 0.8 bps/Hz as P_max increases, whereas the median-user rate and the GM rate increase from 0.85 to 0.9 bps/Hz. Thus, even in a low spatial degree of freedom structure such as NOSA, maximizing the GM rate prevents user-rate collapse due to unfair resource distribution. Consequently, we note that a restricted spatial OSA configuration intensifies the trade-off between overall rate and user fairness, leading to a misalignment between the GM rate and the individual user rates compared to a more flexible configuration.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Rate distribution comparison between geometric mean (GM) and max–min designs.

(A) Fully connected (FC) architecture. (B) Non-overlapped subarray (NOSA) architecture.

https://doi.org/10.1371/journal.pone.0354965.g006

Fig 7(A) shows the inverse relationship between noise power and spectrum efficiency. As the noise power increases, the spectrum efficiency for all users declines exponentially. Specifically, efficiency drops sharply from approximately 35 bps/Hz at a very low noise level, i.e., dBm to just under 5 bps/Hz at a high noise level, i.e., dBm. This decline confirms that, in the optimized beamforming scheme, Gaussian noise is the dominant factor limiting the achievable SINR and thus the spectrum efficiency. Considering the absence of LoS, the channel condition can significantly attenuate the signal power by a factor of or dB. This loss necessitates that the noise power be kept below a threshold, typically dBm, to ensure every user maintains a target achievable data rate.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Performance evaluation under different system parameters.

(A) Spectral efficiency of individual users versus noise power. (B) Spectral efficiency under different numbers of RF chains.

https://doi.org/10.1371/journal.pone.0354965.g007

The impact of the number of RF chains on the spectral efficiency of the hybrid beamforming system is illustrated in Fig 7(B). To isolate this effect, the noise power is fixed at dBm. As the number of RF chains increases, the spectral efficiency of the NOSA configuration improves monotonically due to the additional degrees of freedom available in the digital precoding stage. In the extreme case where N = M = 100, the NOSA architecture reduces to the FD configuration with . Moreover, the results indicate that when the number of RF chains becomes sufficiently large, the NOSA architecture approaches the performance of the FC architecture. Our simulations show that this convergence typically occurs around N = M/2 = 50.

Fig 8 presents a robustness comparison of different beamforming schemes under imperfect channel state information (CSI), where the system parameters are set to N = 50 and . The considered hybrid beamforming configurations are consistent with those evaluated in Fig 5. Imperfect CSI may arise from several practical factors, including channel estimation errors, quantization during CSI feedback, and outdated CSI caused by transmission delay. In this work, the CSI imperfection is modeled as an additive perturbation noise. Specifically, the estimated channel of user k is expressed as

where denotes the actual channel matrix and represents the channel perturbation. The perturbation matrix is assumed to follow a complex Gaussian distribution . Accordingly, the normalized channel perturbation error (NCPE) is defined as [29]

(31)

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Geometric mean (GM) rate performance under imperfect channel estimation conditions.

https://doi.org/10.1371/journal.pone.0354965.g008

For simplicity, we assume identical perturbation levels for all users, i.e., . As illustrated in Fig 8, the performance of all beamforming schemes degrades as the channel estimation error increases. In particular, the GM-rate performance of the FD and FC configurations decreases rapidly with increasing NCPE and eventually falls below that of the NOSA architecture. When the NCPE level reaches 0 dB, meaning that the perturbation power is comparable to the channel power, the achievable GM rate saturates at approximately 2 bps/Hz. These results highlight the strong sensitivity of hybrid beamforming systems to CSI mismatch and indicate that accurate CSI estimation is essential for maintaining the desired quality-of-service (QoS).

5 Conclusions

In this paper, we investigated the spectral efficiency of an RIS-aided scalable hybrid beamforming system operating over the mmWave channel. Given the adopted OSA hybrid architecture at BS, we proposed a joint optimization algorithm to maximize the GM of users’ achievable rates. Based on existing works, we divide the hybrid beamforming problem into the FD beamforming sub-problem and the matrix decomposition problem. We then adopt the iterative BCD method and RCG to solve the resulting optimization problems. An effective initialization method is proposed to enhance the convergence rate of the algorithm. Numerical results demonstrate the effectiveness of the proposed framework in improving fairness-oriented spectral efficiency, convergence behavior, and architectural trade-offs under practical hardware constraints. Specifically, we observed an 18.18% objective value improvement compared to existing methods and identified suitable hardware phase resolutions for RIS-assisted hybrid beamforming systems.

The proposed framework is specifically designed to optimize fairness-oriented spectral efficiency under hardware constraints. Accordingly, the conclusions of this work are restricted to the evaluated aspects, including GM-rate performance, convergence behavior, fairness characteristics, architectural trade-offs, and sensitivity to CSI uncertainty under the considered system model. While the impact of CSI uncertainty has been partially examined through CSI perturbation analysis, energy-efficiency optimization, latency-aware design, and robust formulations for highly dynamic environments remain topics for further investigation.

Acknowledgments

This research is funded by the Hanoi University of Science and Technology (HUST) under project number T2023-PC-030. The authors would like to thank the HUST for the financial support.