Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Spectral efficiency and BER analysis of RNN based hybrid precoding for cell free massive MIMO under terahertz communication

  • Tadele A. Abose ,

    Roles Conceptualization, Methodology, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

    tadenegn@gmail.com

    Affiliation Department of Electrical and Computer Engineering, Mattu University, Mattu, Ethiopia

  • Binyam G. Assefa,

    Roles Conceptualization, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

    Affiliation School of Electrical and Computer Engineering, Dire Dawa University, Dire Dawa, Ethiopia

  • Yitbarek A. Mekonen,

    Roles Conceptualization, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

    Affiliation School of Electrical and Computer Engineering, Dire Dawa University, Dire Dawa, Ethiopia

  • Naol W. Gudeta

    Roles Conceptualization, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

    Affiliation School of Electrical and Computer Engineering, Dire Dawa University, Dire Dawa, Ethiopia

Abstract

In future wireless networks, integrating Terahertz (THz) communication with cell-free massive multiple-input multiple-output (CFMM) systems presents a promising approach to achieving high data rates and low latency. This paper investigates the use of recurrent neural network (RNN)-based hybrid precoding in CFMM systems operating in the THz band. The proposed method jointly designs analog and digital precoders to adapt to dynamic channel conditions and user mobility. However, THz communication is challenged by high path loss and sparse scattering, which complicate accurate channel estimation. To address this, the RNN is trained to predict optimal precoding weights by learning spatial and temporal channel patterns, thereby improving channel estimation and mitigating pilot contamination. Simulation results show that the proposed method achieves higher spectral efficiency and lower bit error rate (BER) than conventional techniques. Specifically, the RNN-based approach attains a spectral efficiency of 10 bps/Hz at a signal-to-noise ratio (SNR) of 30 dB, compared to 8.2 bps/Hz for minimum mean square error (MMSE) precoding. For 16-QAM, the RNN-based method achieves a BER of 10⁻⁶ at an SNR of 11 dB, while MMSE requires 12.5 dB to reach the same BER. Overall, the RNN-based hybrid precoding consistently outperforms traditional methods across various SNR levels, antenna configurations, and user densities, underscoring its potential in next-generation THz wireless systems.

1. Introduction

Massive multiple input multiple output (mMIMO) is considered a revolutionary milestone in wireless communication systems, promising drastic improvements in spectral efficiency, network capacity, and signal quality. The main idea behind mMIMO is to deploy a vast number of antennas at both the base stations and user devices, unlike conventional MIMO systems, which use a limited number of antennas [1]. This significant increase in the number of antennas introduces a transformational dimension to wireless communication, enabling the simultaneous transmission of numerous data streams to many users [2]. This extraordinary level of spatial diversity is set to create game-changing capabilities by allowing better signal separation and boosting overall network performance [3].

With the exponential rise in data-driven applications and the emergence of technologies like the internet of things (IoT) and augmented reality (AR), the demand for high data rates and low latency in wireless communication networks has skyrocketed. To meet these needs, terahertz (THz) communication, operating in the frequency range of 0.1–10 THz, is viewed as a key technology for future wireless networks [4]. THz communication offers ultra-wide bandwidth, supporting data rates of several terabits per second, making it a strong contender for next-generation wireless systems [5].

THz communication presents unique challenges compared to mmWave due to its shorter wavelengths, which result in higher path loss and limited scattering capabilities. The high-frequency nature of THz waves leads to severe signal attenuation, requiring more precise beamforming techniques to maintain signal strength over longer distances. Additionally, hardware imperfections become more pronounced at THz frequencies, where components such as antennas and amplifiers must operate at extreme precision. These physical and technical challenges demand innovative solutions in system architecture, including high-dimensional signal processing, adaptive precoding, and advanced beamforming algorithms to ensure reliable and efficient communication [6].

mMIMO systems signify a paradigm shift from traditional cellular systems with centralized base stations. In contrast, CFMM systems deploy a large number of distributed access points (APs) that jointly serve users without predefined cell boundaries. This setup mitigates inter-cell interference and ensures consistent service quality, especially in high-density and mobile user environments [7].

To fully exploit CFMM in THz communication, hybrid precoding is essential. This approach combines digital and analog precoding to efficiently manage the vast number of antennas in mMIMO systems, balancing hardware complexity, power consumption, and performance. However, conventional hybrid precoding methods struggle with the rapidly changing channel conditions typical of THz frequencies [8].

Advancements in machine learning, particularly RNNs, provide promising solutions to these challenges. RNNs excel at handling sequential data and capturing temporal dependencies, making them ideal for predicting and adapting to varying channel conditions in real time. By integrating RNNs into the hybrid precoding process, we can dynamically adjust the precoding weights to optimize spectral efficiency and minimize the bit error rate (BER), even in the challenging THz band [9].

Numerous studies have highlighted the use of RNNs and other deep learning techniques to enhance mMIMO system performance [10]. RNN-based approaches have proven effective in tasks like channel estimation, beamforming, and interference mitigation, showing significant improvements over traditional methods. These advances demonstrate the potential of RNNs to revolutionize hybrid precoding in CFMM systems, enabling effective utilization of THz communication capabilities [11].

This paper introduces a hybrid precoding scheme based on RNNs, specifically designed to address the dynamic channel conditions prevalent in THz communication. Our proposed scheme leverages the adaptive capabilities of RNNs to effectively manage precoding weights, optimizing performance in the rapidly fluctuating THz environment. Through a detailed analysis, we demonstrate that our RNN based hybrid precoding method significantly enhances spectral efficiency and reduces BER compared to traditional techniques. Moreover, our approach integrates advanced RNN components, such as long short-term memory (LSTM) networks, to further boost system performance in complex and dynamic wireless scenarios. To validate the efficacy of our scheme, we conduct extensive simulations within a CFMM framework, showcasing its superior performance over state-of-the-art systems. These contributions collectively underscore the potential of our method to advance THz communication capabilities, offering substantial improvements in both efficiency and reliability.

The main contributions of the paper are summarized as follows:

  • Introduction of a Hybrid Precoding Scheme Based on RNNs: The paper presents a novel hybrid precoding scheme utilizing RNNs, specifically designed to address the dynamic channel conditions in THz communication. This scheme leverages the adaptive capabilities of RNNs to manage precoding weights effectively, optimizing performance in rapidly fluctuating THz environments.
  • Enhanced Spectral Efficiency and Reduced BER: Detailed analysis demonstrates that the RNN-based hybrid precoding method significantly improves spectral efficiency and reduces BER compared to traditional techniques. The integration of advanced RNN components, such as LSTM networks, further boosts system performance in complex and dynamic wireless scenarios.
  • Extensive Simulations and Validation: Extensive simulations within a CFMM framework validate the efficacy of the proposed scheme, showcasing its superior performance over state-of-the-art systems. These results underscore the potential of the method to advance THz communication capabilities, offering substantial improvements in both efficiency and reliability.

The structure of the paper is outlined as follows: Section 2 provides a comprehensive review of existing literature. Section 3 delves into the system model and problem formulation. Section 4 details the design of MMSE-based hybrid precoding for downlink multiuser massive MIMO (mMIMO) systems. Sections 5, 6, and 7 explore the pilot assignment algorithm aimed at mitigating pilot contamination in cell-free networks, the application of machine learning, and the presentation of results and discussions, including simulation outcomes, respectively. Finally, Section 8 wraps up with the paper’s conclusions.

2. Literature review

The integration of RNNs into hybrid precoding schemes for cell-free massive MIMO systems operating under THz communication is an emerging research area that targets enhanced spectral efficiency and optimized bit error rate BER. Cell-free massive MIMO networks are characterized by their distributed nature, where multiple antennas serve users without the constraints of traditional cell boundaries. This allows for better connectivity and reduced interference, which is crucial in environments with high data demands such as those facilitated by THz frequencies [12].

The pursuit of enhancing spectral efficiency and reducing BER in wireless communication has driven extensive research, particularly in the development of CFMM systems integrated with THz communication [13]. As the demand for higher data rates and more efficient spectrum usage grows, CFMM systems have gained significant attention due to their ability to serve multiple users with minimal interference by utilizing distributed antenna arrays. These systems promise substantial gains in spectral efficiency and robustness, especially in densely populated environments. The integration of THz communication, which operates at extremely high frequencies, further enhances the potential of CFMM systems by offering ultra-wide bandwidths capable of supporting the increasing demands of next-generation wireless networks, such as 6G and beyond. Additionally, researchers are exploring the use of machine learning and artificial intelligence to optimize system parameters in real-time, enhancing the adaptability of CFMM systems to dynamic wireless environments. Collectively, these cutting-edge approaches are paving the way for more reliable and efficient wireless communication systems, with CFMM and THz technologies at the forefront of this evolution [14].

Numerous studies have focused on evaluating the effectiveness of RNN-based hybrid precoding by comparing it to traditional methods and analyzing its performance across various channel conditions and system parameters. RNN-based hybrid precoding has shown promise in optimizing both spectral efficiency and energy efficiency in complex wireless communication environments, particularly in millimeter-wave (mmWave) and massive MIMO systems. Traditional precoding techniques often struggle to adapt to the rapidly changing channel conditions in these systems, leading to suboptimal performance. In contrast, RNN-based approaches can dynamically adjust the precoding process, leveraging their ability to model temporal dependencies and predict future channel states. Researchers have examined the performance of RNN-based precoding in scenarios with different levels of interference, mobility, and noise, demonstrating that it can significantly improve signal quality and reduce BER compared to conventional methods [15].

Hybrid precoding, which combines both analog and digital beamforming techniques, has been instrumental in improving the energy and spectral efficiencies of these systems. RNNs come into play by optimizing the beamforming process. Due to their capability of learning temporal dependencies and adapting to dynamic communication channels, RNNs help predict and adjust the beamforming parameters, significantly reducing errors in signal transmission and enhancing the system’s overall performance, especially when deployed in high-frequency THz bands [16].

Recently, RNNs have emerged as effective tools for tackling the complexities of hybrid precoding in CFMM systems, especially under the demanding conditions of THz communication. The use of RNNs in this area is motivated by the need for low-complexity, yet efficient, precoding solutions that can handle the high-dimensional signal processing required at THz frequencies. Research indicates that RNN-based methods can dynamically adapt to varying channel conditions and maintain robust performance. Various studies have evaluated these techniques, focusing on their ability to optimize spectral efficiency and reduce BER across different scenarios [17].

Furthermore, spectral efficiency in cell-free massive MIMO systems can be substantially improved by employing hybrid precoding strategies that leverage RNNs for dynamic optimization. The distributed antenna systems in these networks contribute to a better signal-to-noise ratio (SNR) across users, which is crucial in THz communications. RNNs provide a predictive mechanism for adjusting precoding strategies in real-time, allowing for more efficient use of the available spectrum and enhancing the data throughput for multiple users in complex scenarios [18].

In their comprehensive survey, Zhang et al. [19] investigate the rapidly evolving domain of CFMM systems, underscoring their transformative potential in wireless communication by delivering reliable service and minimizing interference through the use of distributed antenna arrays. The paper delves into essential components of CFMM, including its fundamental principles, system architecture, and performance metrics. Notably, Zhang et al. point out the significant challenges and opportunities associated with the integration of CFMM and emerging technologies, such as THz communication. They emphasize the necessity for advanced signal processing methods, including hybrid precoding, to effectively manage the high-dimensional and intricate channel characteristics at THz frequencies. However, while the survey provides a valuable overview, it may benefit from a more detailed analysis of specific implementation strategies and the real-world implications of deploying CFMM in conjunction with THz technologies.

The authors explored the application of RNNs for adaptive hybrid precoding mmWave MIMO systems, a precursor technology to the more demanding THz communication. Their study emphasizes using RNNs to dynamically adjust hybrid precoding matrices, effectively balancing digital and analog components to enhance signal transmission efficiency. This adaptive method is particularly advantageous in mmWave systems, where rapid channel fluctuations and high-dimensional signal processing demands are prevalent [20]. However, while their research provides a critical foundation for employing RNNs in high-frequency communication systems, it mainly focuses on mmWave scenarios. To fully extend these advancements to CFMM systems under THz communication, further research is necessary to address the unique challenges posed by THz frequencies.

The authors [21] investigate the effectiveness of RNN-based hybrid precoding in mMIMO systems, particularly in situations where hardware imperfections can significantly affect system performance. This research is essential for comprehending the practical implementation of CFMM systems in THz communication, where similar issues are anticipated. RNNs contribute to maintaining high spectral efficiency and low Bit Error Rate (BER). However, the study predominantly focuses on mmWave frequencies, leaving a gap in understanding how these findings apply to the THz band, where the impact of hardware imperfections is likely to be more severe. Moreover, although the RNN-based approach exhibits resilience to hardware flaws, the challenges related to its computational complexity and real-time processing capabilities in large-scale CFMM systems require further optimization and exploration to ensure practical applicability.

Sun, Yu, Hu, and Peng [22] investigate the application of RNN-based precoding for mMIMO systems, focusing on enhancing spectral efficiency and reducing BER. Their work leverages the dynamic learning capabilities of RNNs to optimize precoding matrices in response to real-time channel variations. The authors [20] demonstrate that RNN-based precoding can significantly outperform traditional linear and nonlinear precoding methods by efficiently managing the high-dimensional signal processing requirements inherent in mMIMO systems. However, their study is primarily focused on conventional mMIMO setups and does not fully address the unique challenges posed by CFMM systems under THz communication.

The use of RNNs in hybrid precoding also aids in reducing the BER by effectively mitigating the effects of interference and signal distortion, which are prominent at THz frequencies. Traditional linear precoding techniques often struggle in such high-bandwidth systems due to the nonlinear nature of the channels, especially in dynamic or dense user environments. By applying machine learning techniques, including RNNs, the precoding process can dynamically adapt to the channel variations, improving both the reliability and the efficiency of data transmission [23].

In summary, while extensive research has demonstrated the potential of RNN-based hybrid precoding to enhance spectral efficiency and reduce BER in mmWave MIMO systems, significant gaps remain in the context of CFMM systems operating under THz communication. Existing studies primarily focus on mmWave frequencies and conventional MIMO setups, often overlooking the unique challenges posed by THz frequencies. Moreover, while RNNs have shown resilience against hardware imperfections and the ability to dynamically adapt to varying channel conditions, their computational complexity and real-time processing capabilities in large-scale CFMM systems require further optimization. Our work addresses these gaps by proposing a novel RNN-based hybrid precoding method specifically tailored for CFMM systems in the THz band. This approach aims to dynamically optimize precoding weights to improve spectral efficiency and minimize BER, overcoming the limitations of traditional methods and paving the way for the practical deployment of next-generation wireless networks.

Expanding on insights from the literature review on RNN-based hybrid precoding methods in wireless communications, we now transition to Section 3. In this section, we outline the system model and problem formulation for our proposed hybrid precoding scheme, which is tailored for CFMM systems operating at THz frequencies.

3. System model and problem formulation

This paper examines a multiuser CFMM system that employs a fully connected hybrid precoding structure. Unlike conventional cellular networks, this system operates within a cell-free architecture, where traditional cell boundaries are removed. By eliminating these boundaries, the network distributes numerous access points (APs) across the entire coverage area. This decentralized deployment of APs improves connectivity, allowing multiple users to be served simultaneously, without the limitations imposed by predefined cell regions. The distributed nature of the APs optimizes resource allocation and enhances the overall system performance, as depicted in Fig 1.

Each AP has antennas and chains to serve user equipment (UE). The statement “Each AP serves K UEs” refers to a system model where the total number of UEs in the simulation is fixed to K per AP. That is, we modeled the system with a limited number of K UEs per AP for simplicity and scalability in simulation. AP selection or dynamic user association was not applied On the UE side, each user is equipped with antennas and an RF chain. Both sides of Fig 1 represent different system configurations with the same set of UEs and APs. The labels are intentionally kept identical to emphasize that the comparison is based on network architecture, not user or access point identity. In CFMM [24], where the traditional cell boundaries are dissolved, each access point (AP) applies a digital baseband precoder of size , represented by , followed by an analog precoder . The analog precoding matrix has a dimension of and is defined as follows

(1)

where, are normalized to satisfy the constant modulus constraints

Let denote the transmitted signal at the AP for the UE and , such that the transmitted signal by the APs can be given by

(2)

where, , is digital precoding vector for the user and are complete hybrid precoding metrics. Then the received signal at the user terminal is

(3)
  • The channel matrix between the transmitter and the user terminal.
  • Transmitted signal for the user.
  • Additive Gaussian noise at the user terminal.

In this study, frequency offset was not explicitly considered in Equation (3) as the focus is on evaluating the performance of the proposed hybrid precoding method under idealized conditions. The omission of frequency offset allows for isolating the effects of the precoding scheme and channel characteristics without introducing additional complexity.

The received signal can be formed after combining at each user terminal is given as

(4)

where, is analog combiner at the user terminal. denotes the Hermitian transpose (conjugate transpose) of a complex-valued matrix or vector and denotes the transpose of a matrix or vector. of dimension represent the THz channel from APs to the user and is the noise at the receiver which is assumed to fellow Gaussian distributed with zero mean and variance . The effective channel (antenna array gain) of the user is

(5)

In the context of CFMM, the signal-to-interference-plus-noise ratio (SINR) at the kth user terminal is a crucial metric for assessing system performance [25].

(6)

Achievable rate of the kth user terminal (UE) represents the achievable data rate and is computed using the SINR. It is given

The spectral efficiency in a cell-free mMIMO system can be calculated based on the Shannon capacity formula, which takes into account the achievable data rate and the bandwidth ()

(8)

where B denotes the system bandwidth used to normalize the achievable rate into spectral efficiency units of bits/sec/Hz.

3.1 Channel model

Due to the short wavelength of THz signals, their transmission suffers from significant attenuation and minimal scattering effects, leading to predominantly line-of-sight (LoS) propagation. Challenges in THz communication include propagation losses, limited scattering, hardware constraints, thermal noise, and atmospheric absorption. Advanced machine learning methods perform channel estimation in THz communication. The channel is a fading channel model tailored for THz communications. It incorporates scattering clusters and propagation paths ( and ) with random gains, azimuth, and elevation angles, capturing spatial and angular diversity. The model accounts for antenna array response at the transmitter and receiver while considering atmospheric effects typical of THz environments. Atmospheric turbulence can significantly impact THz communication channels, as variations in temperature and pressure cause fluctuations in the refractive index, leading to phase and amplitude distortions in the transmitted signals. These fluctuations introduce random fading effects in the channel, thereby degrading signal quality and increasing bit error rates. In particular, the high sensitivity of THz frequencies to turbulence amplifies issues like scintillation, which further complicates channel estimation and correction methods in THz CFMM systems. [26,27].

In this paper, a geometric channel model with scatterers is employed to characterize the THz channel for each user. This approach is fundamentally different from a Rayleigh fading model. While Rayleigh fading assumes a rich scattering environment without a line-of-sight (LoS) component and models channel gains as complex Gaussian random variables with zero mean (i.e., (0,1). our geometric channel model explicitly incorporates a finite number of propagation paths (Np), each with specific gains, angles of arrival and departure (azimuth and elevation), and scatterer positions. This allows us to capture the LoS and sparse multipath nature of THz propagation more realistically. The mention of (0,1). in Eq. (20) and Section 7 was intended only for simplifying the small-scale fading component in some analytical contexts and does not reflect the full model used in the simulations. normalized antenna response vectors. The mean cluster angles are uniformly distributed over [0,2π] reflecting the spatial randomness typical of high-frequency THz channels. It is well-suited for THz CFMM systems, where the propagation environment is dominated by a limited number of significant scatterers, and the impact of beam misalignment, phase front warping, and atmospheric effects is pronounced [28]. Thus, the channel matrix for the user on the subcarrier in the THz CFMM systems can be represented as follows:

(9)

where, is the number of scattering clusters and each cluster contributes paths. is the gain of the ray in the scattering cluster. and ,) represent the normalized antenna array response vectors at the receiver and transmitter, respectively. ( and () are the azimuth (elevation) angles of the ray in the scattering cluster on the receiving and transmitting sides, respectively. The mean cluster angles ( and () are subject to uniform random distribution in the range of [0,2π].

The channel model in this paper utilizes a geometric fading model to better represent the propagation characteristics of THz communication. This model accounts for scattering and LoS propagation typical in THz environments, incorporating scattering clusters and individual propagation paths (5 paths and 10 clusters ), each with random gains and azimuth/elevation angles. As explained in Section 7, The channel matrix is derived by considering the normalized antenna array responses at both the transmitter and receiver, as well as the impact of atmospheric effects such as turbulence, which can lead to phase and amplitude distortions. By leveraging advanced machine learning methods for precoding weight estimation, this approach effectively addresses the challenges posed by atmospheric absorption, thermal noise, and the minimal scattering of THz signals. The dataset used for training includes real and imaginary channel gains, channel phases, spatial correlation between antennas, noise power, transmitted signal power, path loss, shadowing, and fading effects, which are captured in signal strength, phase shift, spatial correlation between antenna elements, noise, signal attenuation, and environmental fluctuations. This enables more accurate modeling of the THz communication channel for CFMM systems.

The primary challenge in using Low-Density Parity-Check (LDPC) codes for THz communication systems, particularly in a CFMM context, is that while LDPC improves error correction, its contribution to overall system performance might be limited compared to the dynamic channel adaptation capabilities of advanced precoding techniques like the RNN-based hybrid precoding we propose. THz communication channels suffer from high path loss, limited scattering, and severe attenuation, which are more effectively addressed by optimizing the channel estimation and precoding strategies rather than solely relying on error correction techniques. although LDPC and other error correction methods are crucial for ensuring data integrity, their performance gains tend to saturate as channel impairments become more severe at higher frequencies, such as in the THz band. Studies have shown that under such conditions, improving channel estimation and beamforming can result in greater spectral efficiency and lower bit error rates compared to relying primarily on error correction techniques [29]

3.2 Problem formulation

In multiuser mMIMO systems, hybrid precoding is primarily formulated as an optimization problem, with the main goal of maximizing the overall sum-rate () for all users, as referenced in [30]. The challenge lies in determining the most effective balance between digital and analog precoding techniques, ensuring optimal signal transmission and improving data throughput across the network. By optimizing this balance, the system seeks to enhance the efficiency of resource utilization, resulting in better performance for each user while maintaining high overall network capacity and minimizing interference. This approach is crucial for maximizing the spectral efficiency and achieving robust communication in multiuser scenarios.

(10)

Deriving the nonconvex optimization directly does not give rise to a straightforward solution [31,32]. However, this can be decomposed into analog and digital parts. For the maximization of the antenna array gain, the effective channel provides a solution in the multiuser mMIMO system. It should be noted that this formulation does not include a mechanism for per-user quality-of-service (QoS) equalization. As a result, the observed system behavior may favor users in better channel conditions, potentially overlooking fairness among UEs. This design choice was intentional to focus on overall spectral efficiency improvements, and future work will incorporate fairness-aware formulations.

(11)

The digital optimization problem can be restated

(12)

In Section 3, we established the system model for our CFMM framework and formulated the problem of optimizing hybrid precoding for THz communication. Building on this foundation, Section 4 introduces a Minimum Mean Square Error (MMSE)-based hybrid precoding design for the downlink multi-user mMIMO system. This approach aims to improve our precoding scheme’s performance by addressing identified challenges.

4. The MMSE based hybrid precoding design for downlink MU mMIMO system

4.1 Analog precoder and combiner design

MMSE criterion is employed to minimize the mean square error (MSE) of the data streams sent to the UE. The expected value of the squared norm of the difference between the transmitted and received signals, , is minimized. This approach helps to derive the closed-form MMSE-based analog precoder for the kth UE under the assumption of full channel state information (CSI) availability [33].

(13)

where, is the number of users, is the transmit power, the noise and is the is the identity matrix variance the MMSE based analog precoder relies entirely on the channel knowledge, denoted as . the MMSE based approach can be used to determine the analog precoder effectively.

(14)

where, is the angle of the element of matrix . Given the analog precoder and the channel , let

(15)

The RF combiner can also be designed based on the MMSE criterion as follows

(16)

The analog combiner for the fully-connected one can be stated as follows

(17)

where, is the phase angle of the element of

4.2 Digital precoder design

The design of the analog precoder in multiuser mMIMO systems is aimed at leveraging the passive gain provided by the antenna array. By utilizing the analog precoder, denoted as , in combination with the channel matrix and the RF combiner formed from the vectors [, …,], an effective digital precoding scheme can be developed to manage and mitigate user interference. The analog precoder works to maximize signal strength by aligning the signals with the array’s physical structure, while the digital precoder further refines the transmission to ensure that interference between users is minimized. In this context, a Minimum Mean Square Error (MMSE)-based digital precoder is applied, offering a precise way to control and reduce inter-user interference, thus optimizing overall system performance and improving data transmission accuracy.

(18)

The power constraints can be satisfied by normalizing the digital precoder for the kth user terminal as follows

(19)

Algorithm 1. MMSE based hybrid precoding

Require: and

    

  Find the Analog Precoder

    

  Calculate effective channel

Digital Part

  Find the Digital Precoder

  Normalize the Digital Precoder

In Section 4, we presented an MMSE-based hybrid precoding design for the downlink multi-user CFMM system. With this framework in place, we now move to Section 5 to address pilot contamination in cell-free networks.

5. Pilot assignment algorithm to mitigate pilot contamination for cell free networks

In this section, we explore the CFMM system, which comprises M access points (APs) each with N antennas, and user equipment’s (UEs) with multiple antennas, randomly distributed across a coverage area. Though typically , CFMM systems perform efficiently regardless of these values due to their scalable architecture. Each AP is connected to a central processing unit (CPU) via a backhaul network. the communication channel between the k-th UE and the m-th AP is modeled as [34,35]

(20)

where, is the spatial correlation matrix incorporating shadowing and path loss features, with dimensions . The large scale fading coefficient, is derived from the trace of , given by , CFMM systems operate in TDD mode, with the total time in a coherence block split into pilot transmission , uplink (UL) data transmission , and downlink (DL) data transmission .

5.1 Pilot contamination (PC) mitigation

Pilot contamination occurs when different UEs use the same pilot sequences, leading to interference [36]. To combat this, an efficient pilot assignment strategy is utilized. Each UE is allocated a pilot based on its distance to the nearest AP or, if multiple APs are equidistant, based on the large-scale fading coefficient [37].

Assignment Algorithm

  • UE Selection: Prioritize UEs with poor channel quality for pilot assignment. Calculate and sort the large-scale fading coefficients for all UEs in descending order.
  • AP Selection: UEs use synchronization signals to communicate with nearby APs. The closest AP serves the UE. If multiple APs are equidistant, the AP with the highest is chosen.
  • Pilot Allocation: The serving AP assigns an unused pilot to the UE and informs neighboring APs to avoid using the same pilot, thereby reducing PC and enhancing system performance.

5.2 Received pilot signal

When UEs transmit pilot signals, the signal received by the m-th AP is:

(21)

where, is the noise power, is the UE’s UL pilot power, and is the received noise matrix, or

5.3 Pilot contamination (PC) and system performance

In scenarios where , pilot sequences are shared among UEs, leading to PC. This interference reduces system performance and degrades the quality of channel estimates. The relationship between PC and the number of AP antennas is critical; higher can mitigate the negative impact of PC, but careful pilot allocation is essential to maintain system performance.

5.4 Optimal pilot allocation algorithm

The algorithm for optimal pilot allocation aims to minimize PC and optimize the quality of channel estimates:

  • Initialization: Start with the set of UEs
  • Beta Calculation: Compute for all UEs and sort them.
  • Pilot Assignment:
    • if to the user
    • if find the best serving AP for each UE and allocate the pilot that minimizes interference.

The paper utilizes a pilot allocation algorithm that provides robust solutions for enhancing the quality of channel estimation and overall system performance. This algorithm intelligently assigns pilots based on large-scale fading coefficients and the spatial distribution of UEs relative to APs, thus minimizing interference and optimizing resource utilization. Additionally, the systematic UE selection process prioritizes those with poor channel quality, ensuring efficient use of resources. These innovative strategies make this approach highly relevant and effective for improving communication in CFMM systems.

In Section 5, we tackled pilot contamination in cell-free networks with an innovative pilot assignment algorithm. With pilot contamination mitigated, Section 6 focuses on integrating machine learning techniques to boost CFMM system performance.

6 Machine learning

Machine Learning (ML) and Deep Learning (DL) have transformed wireless communication by enabling adaptive, data-driven system improvements [38]. In THz communication, where estimating channel parameters is complex, neural networks like Convolutional Neural Networks (CNNs) and RNNs excel in predicting Channel State Information (CSI). CNNs can be effective for spatial feature extraction in channel estimation, while reinforcement learning could enhance adaptive beamforming by optimizing decisions based on real-time feedback, further improving performance in THz communication. [39]. RNNs, particularly LSTM networks, handle sequential data effectively, modeling dynamic behaviors crucial for THz frequencies. These advanced architectures enhance spectral efficiency and reduce BER in next generation wireless networks [40].

6.1 LSTM network architecture for hybrid precoding

LSTM networks are well suited for tasks requiring sequential and temporal data dependencies. In hybrid precoding for CFMM systems under THz communication, LSTMs play a crucial role by dynamically learning and adapting precoding weights based on evolving channel conditions. The LSTM network architecture involves multiple layers to capture complex patterns in CSI. CSI refers to the information that describes the properties of the communication channel, such as signal attenuation and phase shift, which is crucial for adapting the transmission strategy to optimize performance. On the other hand, coding diversity involves using various coding techniques to mitigate the adverse effects of fading and interference, ensuring more reliable data transmission. Integrating these concepts enhances the system’s ability to adapt its beamforming strategies, ultimately improving spectral efficiency and communication quality. The input layer processes CSI data, and subsequent LSTM layers maintain and update the network’s internal state, capturing short-term fluctuations and long-term trends in channel conditions [41]. This temporal processing capability is vital for THz communication, where channel conditions change rapidly and require real-time adaptation. The output layer generates optimized weights for analog and digital precoding stages, enhancing spectral efficiency and minimizing BER [42].

6.2 RNN based hybrid precoding

This adaptation makes the hybrid precoder system capable of optimizing beamforming strategies based on the information learned from RNN training outputs. RNNs dynamically adjust both analog and digital precoders based on changing channel conditions. In hybrid precoding, the goal is to optimize beamforming by selecting the best weights for the analog and digital precoders to maximize spectral efficiency and minimize BER. This integration allows the system to adjust its beamforming parameters over time according to changes in channel conditions and user requirements, thereby enhancing overall performance, spectral efficiency, and communication quality. In contrast to other types of neural networks, such as Convolutional Neural Networks (CNNs), which are particularly effective at extracting spatial features, Recurrent Neural Networks (RNNs) offer a unique advantage in handling temporal dynamics. RNNs are specifically designed to capture sequential dependencies, making them highly adaptable to time-varying data. This temporal adaptability is especially valuable in wireless communication systems, as it allows RNNs to dynamically update precoding parameters in response to short-term fluctuations in the communication channel. This capability becomes critical in high-mobility scenarios, such as THz communication, where rapid changes in the channel conditions must be continuously tracked and managed to ensure reliable and efficient data transmission, as highlighted in [43].

Fig 2 represents the hybrid precoding structure enabled by RNN in CFMM under THz communication. An RNN is used to handle beamforming and input from the channel matrix, spatial correlation, and power fading to optimize the beamforming process.

Workflow

  • Initialization: The dynamic channel data and precoding information are supplied to the RNN.
  • Precoder Generation: The RNN generates analog and digital precoder and analog combiner parameters to enhance system performance.
  • Hybrid Precoder Update: The RNN outputs update on hybrid digital and analog precoders.
  • Transmission and Reception: The transmitted signal will be processed by the hybrid precoder parameters, which at the receiver side should be processed with the RNN predicted analog combiner parameters.

Main data generation parameters for the RNN-based hybrid precoding in CFMM under THz include real and imaginary channel gains, channel phases, spatial correlation between antennas, noise power, transmitted signal power, path loss, shadowing, and fading effects. These are captured in signal strength, phase shift, correlation, noise, signal attenuation, and environmental fluctuations [44]. Besides, the precoding weights of digital and analog parts, as well as the combining matrix of the analog part, directly control the hybrid precoding process and make the data set under the hybrid precoding link extremely diverse, which is crucial for training and testing the RNN model in a robust THz communication environment. This integration contributes to enabling the system to adapt its beamforming strategies toward achieving the goal of better performance in terms of spectral efficiency and communication quality.

The proposed RNN-based hybrid precoding method presents a significant advancement over traditional techniques such as Zero Forcing (ZF), Minimum Mean Square Error (MMSE), MRT (Maximal Ratio Transmission) and Alamouti schemes, particularly in the context of THz communication. Unlike ZF and MMSE, which are predominantly linear methods, the RNN-based approach leverages the adaptive and non-linear processing capabilities of recurrent neural networks, specifically Long Short-Term Memory (LSTM) networks, to optimize precoding weights dynamically. This dynamic adaptation is crucial for THz frequencies, which are characterized by rapid fluctuations in channel conditions due to high attenuation, limited scattering, and sensitivity to blockages. While ZF and MMSE work by solving linear equations to minimize interference and signal distortion, they are less effective in these high-frequency environments, as they fail to capture the temporal and spatial dependencies of the channel. The RNN-based method, on the other hand, continuously learns and updates its parameters based on past channel behavior, leading to more accurate channel estimation and better management of precoding weights, thus achieving superior spectral efficiency and BER performance [45].

Moreover, the comparison to Alamouti’s space-time block coding highlights the differences between space diversity and the hybrid precoding approach. While Alamouti’s method is a simple and effective scheme for achieving diversity in systems with two transmit antennas, it is limited in its ability to scale efficiently in massive MIMO configurations and does not fully exploit the benefits of hybrid analog-digital beamforming. The RNN-based hybrid precoding method not only scales with the increasing number of antennas in cell-free massive MIMO systems but also outperforms Alamouti and other conventional techniques by effectively combining both digital and analog precoding. This hybrid approach optimizes the balance between complexity and performance, reducing hardware constraints and power consumption while maintaining high throughput and low error rates in complex THz communication scenarios. The simulation results, showing the RNN-based method consistently achieving lower BER and higher spectral efficiency across a range of SNR values, further demonstrate its superiority over traditional methods like ZF, MMSE, and Alamouti, especially in next-generation wireless systems [46].

In this study, we implement and evaluate a RNN model for predicting hybrid precoding weights in dynamic wireless communication systems. Our approach leverages 70% of the dataset for training, followed by rigorous testing and validation phases to ensure robust performance. The goal is to optimize spectral efficiency and BER in CFMM systems under THz communication conditions.

  • Training Phase: The RNN is trained using 70% of the dataset composed of dynamic channel data and precoding information, with parameters such as real and imaginary channel gains, channel phases, and noise power. The ADAM (Adaptive Moment Estimation) optimizer is employed to minimize the MSE loss function, with a learning rate of 0.001 and a batch size of 128. The training process iteratively updates the RNN’s weights to enhance its prediction accuracy.
  • Testing Phase: After training, the RNN is tested on 15% of the separate dataset to evaluate its performance. This phase assesses the model’s ability to generalize and accurately predict hybrid precoding weights under varying channel conditions, focusing on metrics like spectral efficiency and BER.
  • Validation Phase: During training, 15% of the validation set is used to fine-tune the model and prevent overfitting. The RNN’s performance on the validation set is monitored to adjust hyper parameters and ensure robust learning. The validation phase is crucial for optimizing the RNN’s ability to adapt to real-time channel variations and maintain high spectral efficiency and low BER in CFMM systems under THz communication. This structured approach ensures that the RNN model is well-trained, accurately tested, and validated for reliable performance in dynamic wireless communication environments.

To tackle the intricate demands of CFMM systems in THz communication, a specialized RNN configuration is crafted. Key components include the ADAM optimizer, chosen for its adaptability to complex optimization landscapes, the TanH activation function for capturing nonlinear relationships in hybrid precoding, and the Mean Squared Error (MSE) loss function, prioritizing precise channel estimation crucial in THz communication. This tailored setup aims to enhance performance and reliability by efficiently handling large datasets, ensuring compatibility with real-world system requirements, and minimizing errors in predicted channel characteristics, thus enabling accurate precoding decisions in dynamic communication scenarios [47,48].

Adaptive coding and modulation play a pivotal role in enhancing the performance and resilience of MIMO systems, particularly in challenging communication environments like those encountered at higher frequencies such as the THz band. These techniques dynamically adjust the coding rate and modulation scheme based on real-time channel conditions, ensuring optimal data transmission while maintaining reliability. By adapting to fluctuating channel quality, including variations in SNR and interference levels, adaptive coding and modulation help to maximize spectral efficiency and minimize BER. This flexibility is crucial for MIMO systems operating in harsh conditions, where fixed modulation or coding schemes often fail to maintain reliable communication. The integration of these adaptive techniques encourages the design of more robust communication systems capable of withstanding the inherent challenges of high-frequency transmission, such as attenuation and fading, while ensuring efficient data throughput and low latency in next-generation wireless networks [49].

After Section 6, which integrates machine learning techniques to enhance CFMM system performance, Section 7 discusses the results and implications of our proposed approaches.

7 Result and discussion

The hybrid precoder simulation parameters for the study include several key elements. The simulation parameters were carefully chosen to reflect realistic conditions for THz communication and CFMM systems, ensuring a comprehensive evaluation of the proposed hybrid precoding scheme. In our study, we use large-scale antenna arrays to support the high demands of these systems. The number of transmit antennas is set to 256, while the number of receive antennas is 64. The simulation involves 5 paths and 10 clusters . The operating frequency is 1 THz. For the angle of arrival (AoA) for clusters, represented as , and the angle of departure (AoD) for paths, represented as ,), values are randomly generated between 0 and 2π. Channel gains for clusters and paths, denoted as , follow a complex normal distribution with a mean of 0 and variance of 1, (0,1). The channel matrix is calculated based on the number of transmit and receive antennas, as well as the channel gains. The noise variance is set at 10 dB. The simulation considers 10 users () and a transmit power () of 30 dB.

To ensure the reliability of the simulation results, an extensive dataset was used for the machine learning training. For each setting, metrics such as spectral efficiency and BER were averaged across all users and all evaluations. This methodology ensures statistically meaningful results and minimizes the impact of randomness or outlier behaviors. To validate the effectiveness of the proposed approach, its performance was benchmarked against a conventional precoding method without machine learning assistance. Results demonstrate that the RNN-based scheme consistently outperforms traditional methods in both sum rate and BER metrics, especially at higher user densities, confirming the model’s adaptability to multi-user environments.

Fig 3, depicting spectral efficiency versus signal to noise ratio (SNR), aligns well with theoretical expectations and the hybrid precoder parameters discussed earlier in this paper. As SNR increases, the spectral efficiency improves across all methods, with the RNN-based approach consistently outperforming traditional methods such as MMSE, ZF, MRT, and Alamouti. Notably, the RNN-based method achieves the highest spectral efficiency across the entire SNR range from -20 dB to 30 dB, showcasing its superiority, particularly at higher SNR values where traditional methods converge towards similar efficiency levels. At 30 dB, the RNN-based method achieves 10 bps/Hz, while MMSE achieves 8.2 bps/Hz, ZF achieves 7.9 bps/Hz, MRT achieves 7.8 bps/Hz, and Alamouti has the least performance at 7.6 bps/Hz. This outcome validates the anticipated performance benefits of the RNN-based approach, especially when considering the hybrid precoding parameters designed to optimize spectral efficiency in a THz-wave CFMM system. Moreover, these results incorporate the contributions of an RNN-trained model, which is trained based on the parameters listed in Table 1. This model, integrated with the hybrid precoding parameters detailed in Algorithm 1, further enhances the system’s performance. The integration of the RNN model with the hybrid precoder parameters exemplifies a robust approach to optimizing spectral efficiency, demonstrating that the RNN-based method not only meets but exceeds theoretical predictions. This confirms its effectiveness in practical applications and highlights its potential for advancing THz wave communication systems.

thumbnail
Table 1. Simulation specifications for RNNs model training.

https://doi.org/10.1371/journal.pone.0328499.t001

Fig 4 illustrates the relationship between spectral efficiency and the number of transmitter antennas (N_TR), with values extending up to 256 antennas. The RNN-based method consistently outperforms other techniques, achieving the highest spectral efficiency across all antenna configurations. Notably, as the number of transmitting antennas increases, the performance advantage of the RNN-based approach becomes even more evident, emphasizing its effectiveness in leveraging large antenna arrays for improved spectral efficiency. At 256 antennas, the RNN-based method reaches a spectral efficiency of 37 bps/Hz, followed by MMSE at 34 bps/Hz, ZF at 32 bps/Hz, MRT at 27 bps/Hz, and the Alamouti scheme at the lowest with 24 bps/Hz. Moreover, the integration of the RNN model with hybrid precoder parameters confirms that the RNN-based method aligns with theoretical expectations. Specifically, equations (14) and (17), which depend on the number of transmitting antennas, are used to calculate spectral efficiency according to equation (8). This relationship underscores the RNN-based method’s capacity to meet theoretical predictions while optimizing performance in THz-wave communication systems. The results demonstrate that as the number of transmitter antennas increases, so does the RNN-based method’s ability to enhance spectral efficiency, making it a strong candidate for mMIMO systems with up to 256 antennas. Furthermore, the improved spectral efficiency is directly related to both the hybrid precoding parameters and the parameters involved in training the RNN model. The model exemplifies a powerful approach to optimizing spectral efficiency in large-scale antenna systems. However, the results would benefit from a clearer presentation to better highlight the relationship between these variables and the practical implications of the findings. The analysis leverages an extensive machine learning dataset to ensure statistical reliability, with results closely aligning with theoretical models and highlighting practical scalability for large mMIMO and THz systems.

thumbnail
Fig 4. The average spectral efficiency versus number of transmitter antennas ().

https://doi.org/10.1371/journal.pone.0328499.g004

Fig 5 illustrates the relationship between spectral efficiency and the number of users for various transmission schemes with up to 50 users. The RNN-based method consistently achieves the best performance across all user counts. As the number of users increases, the performance advantage of the RNN-based method becomes more pronounced, demonstrating its superior capability to efficiently serve multiple users. At 50 users, the RNN-based method achieves a spectral efficiency of 5.6 bps/Hz, while the MMSE method follows closely behind at 5 bps/Hz. The ZF and MRT schemes show similar performances, achieving 4.7 bps/Hz and 4.6 bps/Hz, respectively, with the Alamouti scheme having the lowest performance at 3.9 bps/Hz. These results validate the model’s design and its ability to perform as expected, aligning with theoretical predictions. The increase in spectral efficiency with the number of users from 0 to 50 aligns with Equation 6, where the SINR depends on the number of users, directly affecting spectral efficiency. However, there is a specific point where the spectral efficiency does not significantly increase with the number of users. This plateau effect arises from interference and limited channel resources, which become more prominent as user density increases, thereby capping the achievable spectral efficiency typical of advanced transmission techniques. It is important to clarify that Fig 5 reports the total spectral efficiency (SE) across all users, computed as the sum of individual user SEs,. While it is true that increasing the number of users generally leads to reduced SINR and thus lower individual SEs, the total SE may still increase because it accumulates contributions from a larger user base. This trend reflects the enhanced multi-user capability of CFMM systems, where the aggregate system throughput benefits from user multiplexing, despite per-user trade-offs.

thumbnail
Fig 5. The average spectral efficiency versus number of users.

https://doi.org/10.1371/journal.pone.0328499.g005

Fig 6 illustrates the relationship between spectral efficiency and the number of RF chains for various transmission schemes in a massive MIMO (mMIMO) system, with RF chains ranging from 4 to 8. The RNN-based technique consistently demonstrates the highest spectral efficiency across all RF chain configurations. At the upper end, with 8 RF chains, the RNN-based method achieves a spectral efficiency of 35.5 bps/Hz, outperforming other methods such as MMSE (33 bps/Hz), ZF (31 bps/Hz), MRT (30 bps/Hz), and Alamouti (25 bps/Hz). These results highlight the superior performance of the RNN-based method, particularly as the number of RF chains increases, showcasing its effectiveness in optimizing performance for mMIMO systems with up to 8 RF chains. The RNN-based approach benefits significantly from an increasing number of RF chains, as it allows the system to process more data streams through improved analog and digital precoding weights. This is explained in Section 3, where the fully connected nature of the system and the inclusion of additional RF chains enable enhanced data processing capabilities.

thumbnail
Fig 6. The average spectral efficiency versus number of RF chains.

https://doi.org/10.1371/journal.pone.0328499.g006

The precoding weights, as defined in Equations 14 and 18, play a crucial role in this process by optimizing how signals are transmitted, leading to higher spectral efficiency. The integration of the RNN model with hybrid precoding, as depicted in Fig 2, further refines this process, dynamically adjusting the precoding to optimize system performance. However, it is essential to consider the tradeoff between the number of RF chains and the associated costs and complexity. While increasing the number of RF chains can significantly improve spectral efficiency, this comes with a higher hardware cost and greater computational demands due to the need for more complex processing. Additionally, the improvements in spectral efficiency do not continue indefinitely as the number of RF chains increases. Beyond a certain threshold, the system may experience diminishing returns, where adding more RF chains yields minimal gains due to factors such as interference and the limitations of spatial multiplexing gains.

While SU-MMIMO can achieve high spectral efficiency for a single user by dedicating all resources, it is inherently limited in multiuser scenarios. This study focuses on CFMM, which is specifically designed to serve multiple users simultaneously through distributed APs and hybrid precoding techniques. The proposed RNN-based hybrid precoding method dynamically optimizes both analog and digital precoders, effectively mitigating inter-user interference and enhancing spectral efficiency across multiple users. As shown in Figs 36, CFMM consistently outperforms traditional methods in multiuser setups, making it more practical for dense and dynamic environments where SU-MMIMO would struggle to equitably allocate resources and maintain performance. Thus, while SU-MMIMO may excel for isolated single-user cases, CFMM is better suited for scalable, next-generation wireless networks such as those operating in the THz band [50].

Fig 7 illustrates the BER performance as a function of SNR for 16-QAM under various detection schemes in a system with 256 transmitting antennas. The RNN-based approach demonstrates the best performance, achieving a BER of at 20.5 dB. The MMSE method closely follows, attaining the same BER at 21 dB, while ZF requires 21.5 dB to achieve the same performance. On the other hand, MRT and Alamouti techniques exhibit the least efficiency, reaching at 22 dB and 23 dB, respectively. These results validate the high feasibility of the RNN algorithm in suppressing detection errors within CFMM systems. The superior performance of the RNN-based system stems from the integration of an RNN-trained model with appropriately selected hybrid precoding parameters. This combination significantly contributes to BER reduction by dynamically adjusting precoding weights, as outlined in Algorithm 1, optimizing spectral efficiency, and minimizing detection errors, as demonstrated in Equation 6. Moreover, the figure highlights that the RNN-based approach consistently outperforms alternative methods, especially in the SNR range of 0–15 dB. The observed linear decay in the BER plot as SNR increases is attributed to the progressive enhancement of signal quality relative to noise power. As SNR grows, the error probability diminishes because higher SNR levels lead to more distinct constellation points in 16-QAM, reducing symbol misclassification. Additionally, the RNN’s ability to dynamically adapt the precoding weights further ensures efficient interference mitigation, allowing BER to decrease steadily with rising SNR.

thumbnail
Fig 7. The BER versus SNR for 16-QAM with 256 transmitting antennas.

https://doi.org/10.1371/journal.pone.0328499.g007

Fig 8 showcases the BER performance as a function of SNR for 64-QAM modulation under various detection schemes in a system with 256 transmitting antennas. The RNN-based approach achieves a BER of at 24 dB, marginally outperforming MMSE, which reaches the same BER at 25 dB. Other detection schemes, including ZF, MRT, and Alamouti, follow sequentially, requiring progressively higher SNRs to achieve equivalent BER performance. A critical observation is that the BER curves for 64-QAM decline at a noticeably slower rate compared to those for 16-QAM, emphasizing the greater sensitivity of 64-QAM to noise and signal distortions. This heightened sensitivity underscores the trade-off inherent in higher-order modulation schemes: while 64-QAM enables increased data rates due to its ability to transmit more bits per symbol, it is substantially less robust against noise. This trade-off poses a significant challenge in practical scenarios, especially in environments with fluctuating or suboptimal SNR conditions. The BER curves shown are the average BER across all UEs, computed under varying transmit power conditions while maintaining the same scenario configuration. the integration of the RNN model with the hybrid precoder demonstrates substantial performance gains, enabling dynamic adjustment of precoding weights and enhanced BER reduction. Additionally, while the BER decay signifies rapid improvements in detection performance with increasing SNR, it also reflects the fragility of 64-QAM at lower SNR levels, where error rates are significantly higher. This analysis indicates that while the RNN-based approach holds promise for advancing CFMM systems, further exploration of its adaptability and resilience in diverse operating conditions is necessary. The complexity of 64-QAM, coupled with the marginal gains over traditional methods like MMSE, calls for a deeper evaluation of the computational trade-offs and real-world applicability of the RNN-based model in large-scale deployments.

thumbnail
Fig 8. The BER versus SNR for 64-QAM with 256 transmitting antennas.

https://doi.org/10.1371/journal.pone.0328499.g008

Fig 9 illustrates the BER performance as a function of SNR for 64-QAM modulation in a system with 64 transmitting antennas across various detection schemes. The RNN-based approach achieves a BER of at 25 dB, outperforming other methods such as MMSE, ZF, MRT, and Alamouti, which require progressively higher SNRs to reach the same BER level. This result raises critical questions regarding the effectiveness of the 64-antenna configuration in ensuring robust communication performance. The reduced number of antennas limits the diversity gain and spatial resolution achievable by the system, which directly impacts the BER performance of the employed precoding techniques. Comparatively, in systems with higher antenna counts, such as the previously analyzed 256-antenna configuration, all detection methods demonstrate significantly lower BERs for the same modulation scheme. This improvement underscores the advantages of increased antenna diversity, which enhances signal processing capabilities and resilience against noise and interference. The integration of the RNN model with the hybrid precoder remains pivotal in achieving these BER reductions. By dynamically adjusting the precoding weights and optimizing the system parameters, as outlined in Algorithm 1, the RNN-based approach continues to validate its effectiveness in addressing the challenges posed by complex modulation schemes like 64-QAM, even in lower-antenna configurations. However, the performance gap between 64 and 256 antennas emphasizes the need for further refinement and optimization of the system to fully exploit its potential in practical scenarios.

thumbnail
Fig 9. The BER versus SNR for 64-QAM for 64 transmitting antennas.

https://doi.org/10.1371/journal.pone.0328499.g009

Turbo codes are well-known for their ability to improve BER through iterative error correction, making them highly effective in mitigating noise and interference in wireless communication. However, in this study, the focus is on evaluating the performance of the proposed RNN-based hybrid precoding method, particularly in optimizing precoding weights to adapt to the unique challenges of THz communication. While turbo codes could further enhance BER by improving error resilience, their integration was beyond the scope of this work, as it would require additional computational complexity and changes to the system architecture. Future research will explore the combination of advanced channel coding schemes like turbo codes with the RNN-based precoding approach to achieve even greater BER improvements in THz communication systems [51].

The proposed RNN-based precoder adapts effectively to the dynamic channel conditions, enabling a significant improvement in average spectral efficiency across different transmit power levels. The recurrent nature of the model allows it to track slow variations in THz channels, which are typically dominated by line-of-sight and a few scatterers, leading to more accurate beamforming and reduced inter-user interference.

The scalability and efficiency of the proposed method make it a strong candidate for practical deployment in future THz-based indoor access points and small-cell networks. The RNN structure can be efficiently implemented with modern edge computing resources, minimizing the latency introduced by precoding updates in real-time communications

The proposed work, while innovative and promising, has some limitations, which need to be addressed in future research.

  • Computational Complexity: The method introduces significant computational demands, especially with large-scale antenna arrays and high-dimensional data in THz communication. Real-time processing may be difficult, requiring optimization strategies to make the approach more practical for deployment.
  • Channel Estimation Accuracy: The effectiveness of the RNN-based hybrid precoding heavily relies on accurate channel state information (CSI). Inaccurate CSI can degrade system performance, leading to suboptimal spectral efficiency and increased BER. Enhancing channel estimation techniques is crucial to mitigating these issues.
  • Energy Efficiency: The power consumption of implementing RNNs and hybrid precoding at THz frequencies has not been thoroughly explored. Ensuring energy-efficient operation is crucial for sustainable deployment in large-scale wireless networks.

8 Conclusions

In CFMM systems operating within the THz band, the adoption of a hybrid precoding strategy based on RNNs has emerged as highly effective under challenging conditions. Simulation results demonstrate clear advantages over traditional precoding techniques. specifically, the RNN-based approach achieves superior spectral efficiency and reduced BER. Under varying SNR, the RNN configuration achieves 10 bps/Hz compared to 8.2 bps/Hz with MMSE, showcasing its ability to dynamically adjust analog and digital precoders for optimal performance. Moreover, in scenarios with multiple users, the RNN approach achieves 5.6 bps/Hz, surpassing the MMSE’s 5 bps/Hz, indicating robust handling of dense user environments and high antenna counts crucial for future networks. For BER performance with 16-QAM modulation, the RNN-based method achieves a BER of at 10 dB SNR, ensuring reliable communication amidst the challenges of THz frequencies such as high attenuation and limited scattering. Moving forward, future research should explore scalability in terms of increasing antenna and user numbers, adaptive strategies for diverse wireless environments, and the transition from simulation to practical implementation to validate these promising results in real-world settings and also Future work will focus on extending the RNN architecture to support user-centric cell-free mMIMO scenarios with dynamic AP selection, aiming to further enhance fairness among UEs and address quality-of-service (QoS) differentiation. Additionally, hardware implementation feasibility studies for RNN-based precoding at THz bands will be conducted. Overall, the integration of RNN-based hybrid precoding marks a significant step toward meeting the stringent demands of next-generation wireless networks in THz bands.

Supporting information

S1 Data. Dataset file.

Excel file containing the dataset used for model training in the study.

https://doi.org/10.1371/journal.pone.0328499.s001

(XLSX)

S2 Code. Python code.

Script used for generating the results presented in the manuscript.

https://doi.org/10.1371/journal.pone.0328499.s002

(DOCX)

S3 Text. README file.

Provides instructions and information on the purpose and usage of the accompanying code.

https://doi.org/10.1371/journal.pone.0328499.s003

(TXT)

References

  1. 1. Shi E, Zhang J, Du H, Ai B, Yuen C, Niyato D, et al. RIS-aided cell-free massive MIMO systems for 6G: fundamentals, system design, and applications. Proc IEEE. 2024;112(4):331–64.
  2. 2. Apiyo A, Izydorczyk J. A survey of NOMA-aided cell-free massive MIMO systems. Electronics. 2024;13(1):231.
  3. 3. Saeed MK, Khokhar A, Ahmed S. Pilot contamination in massive MIMO systems: challenges and future prospects. arXiv. 2024.
  4. 4. Wang J, Zhou Q, He J, Habibi MA, Melnyk S, El-Absi M, et al. Terahertz communications and sensing for 6G and beyond: a comprehensive review. IEEE Commun Mag. 2024;62(2):186–95.
  5. 5. Cai X, Cheng X, Tufvesson F. Toward 6G with terahertz communications: understanding the propagation channels. IEEE Commun Mag. 2024;62(2):32–8.
  6. 6. Ruan J, Mao J, Liu T, Svyatoslav S, Sun Z, Wang B, et al. Knowledge bases, cooperation networks, and influential contributions of channel modeling for terahertz communications. IEEE Commun Mag. 2024;62(2):118–24.
  7. 7. Li G, Wu S, You C, Zhang W, Shang G. Asynchronous cell-free massive MIMO-OFDM: mixed coherent and non-coherent transmissions. IEEE Communications Letters. 2024.
  8. 8. Liu Y, Wang J. Low-complexity OFDM-based hybrid precoding for multiuser massive MIMO systems. IEEE Wireless Commun Lett. 2020;9(3):263–6.
  9. 9. Ye H, Liang L, Li GY, Juang B-H. Deep learning-based end-to-end wireless communication systems with conditional GANs as unknown channels. IEEE Trans Wireless Commun. 2020;19(5):3133–43.
  10. 10. Peng Y, Dong X. Machine learning-based hybrid precoding for millimeter-wave massive MIMO systems. IEEE Trans Vehicular Tech. 2020;69(1):1663–7.
  11. 11. Sagir B, Aydin E, Ilhan H. Deep-learning based reconfigurable intelligent surfaces for intervehicular communication. IEEE Trans Vehicular Tech. 2024.
  12. 12. Xia X, Zhu P, Li J, Wu H, Wang D, Xin Y. Joint optimization of spectral efficiency for cell-free massive MIMO with network-assisted full duplexing. Sci China Inf Sci. 2021;64(8).
  13. 13. Lyu S, Wang Z, Gao Z, He H, Hanzo L. Lattice-based mmWave hybrid beamforming. IEEE Trans Commun. 2021;69(7):4907–20.
  14. 14. Wang X, Zhang Z, Li Y. Performance analysis of RNN-based hybrid precoding in massive MIMO systems. IEEE Trans Commun. 2021;69(5):3322–33.
  15. 15. Tan J, Dai L. Wideband channel estimation for THz massive MIMO. China Commun. 2021;18(5):66–80.
  16. 16. Feng C, Shen W, An J, Hanzo L. Weighted sum rate maximization of the mmWave cell-free MIMO downlink relying on hybrid precoding. IEEE Trans Wireless Commun. 2022;21(4):2547–60.
  17. 17. Shalaby M, Hussein HM, Shokair M, Benaya AM. The cell-free mMIMO networks: mathematical analysis and performance evaluation. Telecommun Syst. 2021;77(4):625–41.
  18. 18. Zhang X, Liu Y, Shi J, Zhao J. Cell-free massive MIMO: a survey of current status and future directions. IEEE Commun Surv Tutorials. 2020;22(4):1961–88.
  19. 19. Wang Q, Mao YW, Ren L, Li Z, Liu H. Automatic classification of ECG data quality for each channel. IEEE Access. 2020;8:196094–101.
  20. 20. Naeem M, De Pietro G, Coronato A. Application of reinforcement learning and deep learning in multiple-input and multiple-output (MIMO) systems. Sensors (Basel). 2021;22(1):309. pmid:35009848
  21. 21. Zhang H, Wang X, Chen Y. Robust RNN-based hybrid precoding for massive MIMO systems under hardware impairments. IEEE Trans Wireless Commun. 2021;20(5):3456–68.
  22. 22. Liu X, Qin Z, Zhang Y, Na S, Liu Y, Jin M. A deep learning-based hybrid precoding with true-time-delay for THz massive MIMO systems. In: 2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN). 2024. 317–22. doi: https://doi.org/10.1109/icmlcn59089.2024.10624974
  23. 23. Jeyakumar P, Vishwa S, Prakash V, Dinesh P, Rajesh Ram P, Muthuchidambaranathan P. Long-short term memory based wideband beam tracking scheme for massive mimo systems. Wireless Netw. 2023;30(1):67–76.
  24. 24. Zappone A, Al-Abiad M, Shakir MZ, Debbah M. Robust precoding in massive MIMO: A deep learning approach. arXiv preprint. 2020.
  25. 25. Abose TA, Mekonen YA, Assefa BG, Gudeta NW. Energy efficiency and system complexity analysis of CNN based hybrid precoding for cell-free massive MIMO under terahertz communication. Front Comms Net. 2024;5.
  26. 26. Yousif BB, Elsayed EE. Performance enhancement of an orbital-angular-momentum-multiplexed free-space optical link under atmospheric turbulence effects using spatial-mode multiplexing and hybrid diversity based on adaptive MIMO equalization. IEEE Access. 2019;7:84401–12.
  27. 27. Elsayed EE. Investigations on OFDM UAV-based free-space optical transmission system with scintillation mitigation for optical wireless communication-to-ground links in atmospheric turbulence. Opt Quant Electron. 2024;56(5).
  28. 28. Alkhateeb A, Leus G, Heath RW. Limited feedback hybrid precoding for multi-user millimeter wave systems. IEEE Trans Wireless Commun. 2015;14(11):6481–94.
  29. 29. Sajid WA, Hassan NB, Abdulrazak LF, Khlaponin Y, Ahmed AI, Yashchuk A. Performance comparison of low-density parity-check (LDPC) codes for reliable communication in noisy channels. In: 2024 35th Conference of Open Innovations Association (FRUCT), 2024. 614–24. doi: https://doi.org/10.23919/fruct61870.2024.10516373
  30. 30. Saeed MK, Khokhar A, Ahmed S. Pilot contamination in massive MIMO systems: challenges and future prospects. In: 2024 International Wireless Communications and Mobile Computing (IWCMC). 2024. 1504–9. doi: https://doi.org/10.1109/iwcmc61514.2024.10592426
  31. 31. Ngo HQ, Ashikhmin A, Yang H, Larsson EG, Marzetta TL. Correction to “Cell-Free Massive MIMO Versus Small Cells” [Mar 17 1834-1850]. IEEE Trans Wireless Commun. 2020;19(5):3623–4.
  32. 32. Abose TA, Olwal TO, Hassen MR. Hybrid beamforming for millimeter wave massive MIMO under hardware impairments and imperfect channel state information. In: 2021 IEEE International Conference on Mobile Networks and Wireless Communications (ICMNWC). 2021. 1–6. doi: https://doi.org/10.1109/icmnwc52512.2021.9688468
  33. 33. Du J, Wang Z, Zhang Y, Guan Y, Jin L. Multi-user hybrid precoding for mmWave massive MIMO systems with sub-connected structure. J Wireless Com Network. 2021;2021(1).
  34. 34. Abose TA, Olwal TO, Hassen MR, Bekele ES. Performance analysis and comparisons of hybrid precoding scheme for multi-user mmWave massive MIMO system. In: 2022 3rd International Conference for Emerging Technology (INCET). 2022. 1–6. doi: https://doi.org/10.1109/incet54531.2022.9824401
  35. 35. Abose TA, Olwal TO, Hassen MR. Hybrid beamforming for millimeter wave massive MIMO under multicell multiuser environment. Indian J Sci Technol. 2022;15(20):1001–11.
  36. 36. Saeed MK, Khokhar A, Ahmed S. Pilot contamination in massive MIMO systems: challenges and future prospects. arXiv preprint. 2024.
  37. 37. Deshpande S, Aggarwal M, Sabherwal P, Ahuja S. Deep learning-aided channel estimation combined with advanced pilot assignment algorithm to mitigate pilot contamination for cell-free networks. Indian J Sci Tech. 2024;17(5):465–77.
  38. 38. Mai TC, Ngo HQ, Egan M, Duong TQ. Pilot power control for cell-free massive MIMO. IEEE Trans Veh Technol. 2018;67(11):11264–8.
  39. 39. Peng Y, Dong X. Machine learning-based hybrid precoding for millimeter-wave massive MIMO systems. IEEE Trans Vehicular Technol. 2020;69(1):1663–7.
  40. 40. Yang Y, Xiao Z, Xia P, Xia X. Machine learning for terahertz communications: applications, opportunities, and challenges. IEEE Commun Magazine. 2020;58(3):31–7.
  41. 41. Iliadis LA, Zaharis ZD, Sotiroudis S, Sarigiannidis P, Karagiannidis GK, Goudos SK. The road to 6G: a comprehensive survey of deep learning applications in cell-free massive MIMO communications systems. J Wireless Com Network. 2022;2022(1).
  42. 42. Liu F, Hu Z, Wang S, Zhang X. LSTM-based hybrid precoding for cell-free massive MIMO systems in THz communications. IEEE Trans Wireless Commun. 2020;19(11):7437–47.
  43. 43. Chen X, Zhang Y, Liu J, Chen J. Adaptive hybrid precoding in massive MIMO systems using recurrent neural networks. IEEE Access. 2021;9:76145–55.
  44. 44. Wang H, Liu J, Zhang X, Wang H. Data-driven hybrid precoding for cell-free massive MIMO in THz communications. IEEE Trans Commun. 2021;69(9):5817–28.
  45. 45. Cao M, Yao R, Xia J, Jia K, Wang H. LSTM attention neural-network-based signal detection for hybrid modulated faster-than-nyquist optical wireless communications. Sensors (Basel). 2022;22(22):8992. pmid:36433588
  46. 46. Santumon SD. Space-time block coding (STBC) for wireless networks. Int J Dis Parall Syst. 2012;3(4):183–95.
  47. 47. Shahabodini S, Mansoori M, Abouei J, Plataniotis KN. Recurrent neural network and federated learning-based channel estimation approach in mmWave massive MIMO systems. Trans Emerg Telecommun Technol. 2024;35(1):e4926.
  48. 48. Yu H. Recurrent neural network-based precoding for massive MIMO systems in dynamic environments. IEEE Access. 2024;10(2022):10345–56.
  49. 49. Downey J, Mortensen D, Evans M, Briones J, Tollis N. Adaptive coding and modulation experiment using NASA’s space communication and navigation testbed. In: 34th AIAA International Communications Satellite Systems Conference. 2016. 5736.
  50. 50. Vasudevan K, Singh S, Reddy APK. Coherent receiver for turbo coded single-user massive MIMO-OFDM with retransmissions. In: Multiplexing. IntechOpen; 2019.
  51. 51. Vasudevan K. Near capacity signaling over fading channels using coherent turbo coded OFDM and massive MIMO. arXiv preprint. 2017.