https://orcid.org/0000-0002-1798-2296
Figures
Abstract
This paper investigates the performance of a satellite-terrestrial communication system employing rate-splitting multiple access (RSMA) to accommodate multiple users. The study focuses on assessing the system’s ergodic sum rate and symbol error rate (SER) under shadowed-Rician fading conditions. By deriving closed-form expressions, we analyze the influence of various power allocation strategies on overall system performance. The results, substantiated by numerical simulations, demonstrate that RSMA yields significant improvements over conventional non-orthogonal multiple access (NOMA) with respect to spectral efficiency and interference management. These findings offer valuable insights for the development of more reliable and efficient satellite-terrestrial networks.
Citation: Nguyen H-N, Nguyen QS, Nguyen TN (2026) RSMA-enabled satellite-terrestrial communication networks: Performance analysis. PLoS One 21(8): e0357013. https://doi.org/10.1371/journal.pone.0357013
Editor: Hashmat Fida, Lincoln University College, MALAYSIA
Received: January 19, 2026; Accepted: August 11, 2026; Published: August 26, 2026
Copyright: © 2026 Nguyen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Hybrid Satellite–Terrestrial Networks (HSTNs) are increasingly vital in contemporary wireless systems, providing extensive coverage and dependable connectivity in remote and under-served regions. These networks represent a pivotal technology in the progression of next-generation communication systems, including 6G. HSTNs provide a comprehensive framework for addressing both technological advancements and practical challenges inherent, thereby establishing themselves as a foundational component for the seamless integration of terrestrial and non-terrestrial networks in the 6G era and beyond [1]. With the swift rise in data traffic and the demand for enhanced spectral efficiency, traditional multiple-access methods encounter considerable obstacles in accommodating the growing user base and effectively managing interference [2,3]. To improve spectral efficiency, orthogonal multiple access (OMA) is commonly utilized in satellite communication systems, where distinct frequency, time, or code resources are allocated to different users to prevent interference [4]. Nonetheless, OMA is limited in its spectral utilization, especially when faced with a high number of users [5]. To address this drawback, non-orthogonal multiple access (NOMA) has been developed, enabling multiple users to utilize the same resources through power-domain multiplexing [6,7]. Although NOMA enhances spectral efficiency by facilitating successive interference cancellation (SIC) at the receiver, it also presents challenges like error propagation, issues of fairness, and heightened decoding complexity [8,9]. Recently, Rate-Splitting Multiple Access (RSMA) has emerged as a favorable alternative, encompassing both OMA and NOMA [10]. Unlike NOMA, which depends on SIC for separating signals, RSMA divides transmitted messages into common and private segments, enabling more adaptable interference management [11,12]. The common message is available for all users to decode, while private messages are decoded separately, striking a balance between interference mitigation and spectral efficiency [13]. Research indicates that RSMA surpasses both OMA and NOMA regarding sum rate, user fairness, and resilience to imperfect channel state information (CSI) [14–16]. However, although RSMA has been thoroughly studied in terrestrial networks, its application in satellite-terrestrial communication is still largely uncharted, particularly under realistic fading conditions like shadowed-Rician fading [17].
1.1 Literature review
Since its introduction, RSMA has gained significant research attention, with numerous studies focusing on resource allocation optimization and precoder design to enhance user rate fairness, maximize sum rate, and improve bit error rate (BER) and symbol error rate (SER) performance. For instance, the work in [18] explored a joint optimization framework for rate-splitting precoding, subcarrier assignment, and power allocation to ensure fair rate distribution among users in multi-cast transmissions. To further analyze BER performance under finite constellation and specific coding rates, a transceiver architecture was proposed, incorporating low-density parity-check (LDPC) encoding, modulation, and RS-based signal splitting at the transmitter, along with maximum a posteriori (MAP) detection, LDPC decoding, and SIC at the receiver. Additionally, in [19], researchers examined half-duplex cooperative communication (HCC) for RSMA users, demonstrating that even with partial CSI, jointly optimizing the precoder, time allocation, and relay node selection could still achieve a higher weighted sum rate (WSR) compared to non-cooperative communication (NCC). Existing studies can therefore be broadly categorized into two main methodological directions. Regarding the first methodology, which avoids the use of a precoder, the majority of research concentrates on utilizing the inherent advantages of rate-splitting techniques to enhance capacity. For example, the study in [20] introduced fixed and cognitive rate splitting strategies employing fixed and cyclic single-carrier transmissions to improve the fairness of pairing nearby users with distant users while also reducing their overall outage probability in delay-constrained transmission scenarios. In [21], a novel reconfigurable intelligent surface (RIS) rate-splitting framework was created utilizing an offline control mechanism to minimize outage occurrences for user communication. In [22], the application of rate-splitting for Low Earth Orbit (LEO) satellite communication (SatCom) was investigated with the objective of maximizing the sum rate of LEO SatCom systems. This was achieved through the simultaneous optimization of the power budget across multiple beams, RSMA power allocation for users within each beam, and subcarrier user assignment. Additionally, measures were implemented to ensure that interference temperature constraints to GEO SatCom were strictly maintained. Specifically, the research presented in [23] examined the outage probability and ergodic capacity approximations for MIMO systems while accounting for ULA and imperfect CSI, alongside a simulated assessment with various types of phase-shift keying (PSK) and quadrature amplitude modulation. In the context of downlink multiple-input single-output (MISO) networks featuring cell-edge users supported by reconfigurable intelligent surfaces and LRW precoders, the outage probability was extensively analyzed with a two-layer hierarchical resource sharing approach employing an on-off control method in [24] and a singular RS layer integrated with discrete phase shifters in [25].
1.2 Motivations and contributions
While RSMA has been thoroughly examined within multi-antenna and massive MIMO frameworks, its application to multi-user satellite-terrestrial communication systems remains insufficiently explored [26,27]. Additionally, existing research on RSMA often assumes the availability of perfect CSI, which is difficult to obtain in practical satellite networks due to significant mobility and beam alignment challenges [28]. To tackle this concern, this paper offers an in-depth analysis of RSMA-based satellite-terrestrial communication systems, focusing on ergodic sum rate and SER results under realistic fading conditions. The main contributions of this study are summarized as follows:
- Ergodic Sum Rate Analysis: We establish a closed-form integral expression for the ergodic sum rate in RSMA-driven satellite-terrestrial networks. A high-SNR approximation is included to provide a further understanding of system behavior at elevated transmit power levels.
- SER Evaluation: We evaluate the SER performance of both common and private messages during RSMA transmission. The influence of power allocation and modulation techniques (e.g., BPSK, QPSK) on SER is explored.
- Impact of Power Allocation and Fading Conditions: We investigate how the power coefficient for the common stream affects system performance, particularly under shadowed-Rician fading.
- Comparison with NOMA: We perform numerical simulations to juxtapose RSMA and traditional NOMA, emphasizing RSMA’s benefits in spectral efficiency and interference handling.
From an implementation standpoint, RSMA generally entails lower and more predictable receiver complexity than power-domain NOMA in satellite downlinks. Each user decodes the common stream once, cancels it via a single SIC step, and then decodes its private stream, so the SIC depth per terminal remains essentially constant. In contrast, NOMA typically requires successive decoding of multiple users’ signals (especially at strong users), which increases decoding latency and can amplify error propagation. While RSMA requires optimization of splitting and power allocation, such processing can be centralised (e.g., at the gateway), whereas NOMA additionally relies on user ordering/pairing decisions and related signalling overhead.
The outcomes of this research provide critical insights that are necessary for improving RSMA-based satellite-terrestrial networks. These findings directly contribute to the development of more efficient and robust next-generation satellite communication systems.
1.3 Organization & notations
The remainder of this paper is organized as follows: Section 2 introduces the system model, explaining the satellite-terrestrial channel characteristics and the RSMA transmission framework. Section 3 focuses on the derivation of the ergodic sum rate and its implications for system performance. Section 4 analyzes the SER, considering different power allocation strategies. Section 5 presents numerical results, comparing RSMA with conventional multiple access schemes and discussing key observations. Finally, Section 6 concludes the paper and suggests directions for future research.
For clarity, the key notations used throughout this paper are summarized in Table 1.
2 System model
2.1 System description
Fig 1 illustrates the considered downlink RSMA-enabled satellite–terrestrial system. A GEO satellite S equipped with K antennas simultaneously serves Q single-antenna terrestrial users . The satellite superposes a common stream
and private stream
with power fractions
and
, respectively, under the total power constraint
. The effective channel to user
is
, where
follows a shadowed-Rician model,
is selected according to MRT, and
and
capture the free-space loss and satellite beam gain, respectively. Each user decodes
first while treating all private streams as interference, performs SIC to remove
and then decodes its own private stream in equation (4), leading to the SINRs in equations (5) and (6). Unless otherwise stated, we adopt ideal (benchmark) CSI at the transmitter for MRT beamforming and analysis tractability, as discussed in this section.
Modern satellite systems often use multi-beam technology to enhance coverage and capacity. For geosynchronous Earth orbit (GEO) satellites, these beams are usually generated by array-fed reflectors, which offer better efficiency compared to direct radiating arrays. The beam pattern remains fixed, reducing the need for complex onboard processing.
The signal received by each user depends on the satellite’s beamforming strategy, propagation conditions, and power allocation. The channel gain between the satellite and the qth user is given by
where represents the satellite antenna gain, and
denotes the
Shadowed-Rician channel vector linking K antennas at S. The notation
indicates the conjugate transpose, and
is the
vector of transmit weights. The transmit beamforming vector
is chosen based on the maximum ratio transmission (MRT) principle, such that
, where
indicates the Frobenius norm. Additionally,
represents the instantaneous free space loss [29], where
serves as a scaling factor, with
designated as the Boltzmann constant, T as the noise temperature of the receiver, W as the bandwidth of the carrier, c as the speed of light,
as the frequency of the carrier, and
as the distance separating S and
. Furthermore, the beam gain
of the satellite can be expressed as
where is the first-kind Bessel function with order i,
is the antenna gain at
,
is the angular separation and
in which
represents the 3 dB beam-width. Furthermore, it is assumed that the ideal CSI
is accessible to the satellite user. This can be achieved by using training symbols for the satellite connection, together with an existing feedback link or spectrum manager (which acts as an intermediary between the two systems) for the terrestrial interference link (Perfect CSI is assumed to obtain a tractable analytical benchmark and to highlight the intrinsic RSMA gains under shadowed-Rician satellite fading. In practice, CSI acquisition in satellite–terrestrial links is affected by estimation errors, feedback quantization, and latency. For GEO multi-beam downlinks, the relatively slow variation of the effective channel and the availability of pilot-assisted estimation with a return link (or a spectrum manager) can make the CSI quality sufficiently high to approximate the perfect-CSI benchmark. For more dynamic scenarios (e.g., LEO or rapid beam tracking), imperfect CSI models are required; robust RSMA designs under CSI uncertainty constitute an important extension and will be considered in future work.) [30].
2.2 The signal processing at transceivers
In this system, the satellite uses RSMA to transmit signals to multiple users at the same time. The transmitted signal consists of two parts: a common message that is decoded by all users and private messages intended for individual users. To balance performance, the satellite allocates power to these messages using a power coefficient for the common message, while the remaining power is distributed among the private messages.
The transmitted signal can be expressed as follows:
Here is the total transmit power at S, which
is the common message, allocated with power
and
is the private message for user q, allocated with power
. It is important to note that
. The signal obtained by the qth user can be articulated as
Here represents the additive white Gaussian noise (AWGN) characterized by a mean of zero and a variance denoted by
. To decode the received signal, each user follows a two-step process.
2.3 Decoding the common message
Each user first decodes the common message while treating all private messages as interference. The signal-to-interference-plus-noise ratio (SINR) for decoding the common message at the user q is:
where is the transmit signal-to-noise ratio (SNR), while
and
.
2.4 Decoding the private message
Once the common message is successfully decoded and removed from the received signal, each user proceeds to decode its own private message. The remaining interference comes from private messages intended for other users. The SINR for decoding the private message is given by
The efficiency of this process depends on the power allocation and channel conditions. By carefully choosing and
, RSMA ensures a balance between spectral efficiency and interference management, improving overall system performance.
Having established the signal model and the RSMA decoding procedure, we briefly discuss key practical implementation considerations before introducing the channel model used for performance analysis. Practical deployment of RSMA in satellite-terrestrial networks entails several implementation challenges. First, multi-antenna transmission relies on sufficiently accurate and timely CSI to support beamforming; in operational systems, CSI is typically acquired via pilot-assisted estimation and delivered through a feedback/coordination mechanism, and it may be affected by latency, quantization, and beam pointing errors. In this work, we adopt ideal CSI as a benchmark to enable MRT beamforming and to keep the analysis tractable, while acknowledging that imperfect CSI modeling and robust design are important extensions. Second, RSMA requires adaptive configuration of the common/private power split and, in general, the allocation of the common rate across users, which can impose additional control-plane signaling and scheduling overhead, particularly when users experience heterogeneous channel conditions. Third, although the receiver-side processing in RSMA has relatively bounded complexity—each user decodes the common stream, removes it via SIC, and then decodes its private stream—non-ideal SIC and practical receiver impairments can still lead to residual interference and performance loss. Finally, satellite system constraints (e.g., payload processing capability, feeder-link limitations, and beam management) may restrict the update rate and granularity of precoding and power control, making low-complexity or gateway-centric implementations especially relevant. Motivated by these practical considerations, the following subsection formalizes the satellite channel statistics used in our performance analysis.
2.5 Terrestrial channel model
To facilitate the computation of system performance metrics, we assume that the channel coefficients are independent and identically distributed (i.i.d.). Under this assumption, the probability density function (PDF) of the channel coefficient , for all
, representing the link from the satellite’s kth antenna to the
user, can be expressed as follows:
Where: ,
,
in which
, 2bq, and mq represent the average power of the line-of-sight (LOS) component, the average power of the multipath components, and the fading severity parameter, respectively.
denotes a confluent hypergeometric function of the first kind, as referenced in [31, Eq. (9.210.1)]. For the purposes of this paper, we assume integer values for the Shadowed-Rician fading severity parameter
. This assumption facilitates a more straightforward analysis of the channel’s statistical characteristics and their influence on system performance metrics. Accordingly, equation (7) can be reformulated as follows:
where and
denotes the Pochhammer symbol. Drawing upon the findings presented in [32], the PDF of
under independent and identically distributed (i.i.d.) Shadowed-Rician fading can be expressed as follows:
where
In this context,
The Beta function is referenced from [31, Eq. (8.384.1)]. To obtain the CDF of
, we directly utilize the result from [31, Eq. (3.351.1)]:
where is the lower incomplete Gamma function. Using [31, Eq. (8.352.6)], (12) can be simplified as
Remark 1: This formulation captures the statistical properties of the channel and facilitates a comprehensive understanding of the system’s performance metrics under the designated fading conditions.
3 Integral expression of ergodic sum rate
In terms of the instantaneous SINR, the ergodic rate of the common/private stream for the qth user can be given as [33,34]
where is normalized in the unit of bps/Hz. a = c denotes the common stream, whereas a = p denotes the private stream.
The common stream carries information intended to be decoded by all users at the receiver side. To guarantee that the shared stream can be interpreted by every user, the ergodic rate of the shared stream for each user must meet
In the satellite-terrestrial communication system that employs RSMA, the ergodic sum rate for the qth user is the total of the common rate and the private rate, which can therefore be expressed as
Proposition 1: The ergodic sum rate may be accurately expressed as follows:
,
,
and
.
Proof: First, substituting (5) into (14) we have the ergodic rate for decoding common messages is given by
By changing the variable and with the help of [35],
can be given by
where and
. Plugging (13) into (20),
is rewritten as
To solve the integrals in (21), we use the following transformation using the Meijer G-function [36]
By substituting (22) into (21), can be expressed as
Building upon [31, Eq. (7.811.5)] the ergodic rate of the common stream can be accurately derived as
Similarly, by using (6), the ergodic rate of the private message can be expressed as . Solving
using steps similar to (23) yields
where and
. Using (25) and (24) into (17), we get (18). This completes the proof of Proposition 1.
High SNR Region: When , the law of large numbers may be applied to estimate
and
. The respective ergodic rate boundaries of
and
at
can be derived as
By substituting (26b) and (26a) into (17), the ergodic sum rate under high SNR circumstances may be represented as
Remark 2: The high-SNR approximation reveals that the ergodic sum rate grows logarithmically with transmit power, while the choice of critically affects the balance between common and private message rates. This emphasizes the need for optimal power allocation in RSMA-based satellite-terrestrial networks.
4 SER analysis
The SER for a given modulation scheme can be expressed as [37, Eq. (37)]
where represents either the common message (c) or the private message (p),
is the Gaussian Q-function, r and v are modulation-specific constants (e.g., for BPSK, r = 1, v = 2; for QPSK, r = 2, v = 1) in [38] and
is the SINR for decoding either the common or private message at user q.
To analyze the SER, the derived SINR expressions are substituted into above equation and apply statistical methods to evaluate the expected value over the fading distribution.
4.1 SER of the common message
Proposition 2: Since all users need to successfully decode the common message, its SER is determined by the worst-performing user in the network. Using probability theory and integral approximations, The closed-form approximate expression of SER for the common message is approximated as
where W is a trade-off parameter controlling complexity and accuracy, is the Error function,
,
and
.
Proof: Submitting (5) into (28), we have is given by
From (13), let’s rewrite the result in (30) as
For the first integration of (31), using [31, Eq. (3.361.1)], is given by
where is the Error function. For the second integral, setting
results in
. Hence,
is given as
where and
.
Using the Gaussian-Chebyshev quadrature method in [39,40], we can obtain as
where .
Plugging (34) and (32) into (31), we will acquire (29). The proof is completed.
4.2 SER of private messages
For private messages, the SER depends on the ability of each user to successfully decode its intended signal while treating interference from other users as noise. In a like manner, employing (6), the SER for private messages can be represented as . By following a process similar to that in (29) to deduce
, we arrive at
where and
.
4.3 High SNR region
At high transmit power levels, the SER expressions simplify significantly. In the high-SNR regime (), the SER can be approximated as
Remark 3: This shows that increasing improves the reliability of the common message, but excessive allocation can degrade private message performance.
5 Results and discussions
In this section, numerical simulations are presented to validate the derived formulas. The parameters for Shadowed-Rician fading can be found in Table 2, while the details for the numerical outcomes are consolidated in Table 3. We consider a baseline scenario involving two users (Q = 2). Furthermore, the equivalent noise power at was determined as
[dBm] as referenced and the respective power allocation coefficients are
and
.
Fig 2 presents the ergodic sum rate as a function of transmit power for different values of the power coefficient
under both AS and HS conditions. The results show that the simulation ergodic sum rate increases with
, aligning with the analytical expression in (18). The high SNR region in (27) closely matches the exact analysis at high power levels, validating its accuracy. Under AS conditions, the system achieves a significantly higher sum rate than in HS, demonstrating the impact of severe fading on performance. The choice of
also plays a crucial role, as a higher
initially improves the sum rate but may lead to diminishing gains at higher
. This suggests that optimizing
is essential for maximizing system performance in satellite-terrestrial networks.
Fig 3 clearly demonstrates that the ergodic sum rate increases substantially with the number of satellite antennas (K). The results confirm that increasing K provides improvements of sum rate, driven by superior beamforming gains and more effective interference mitigation. Moreover, it is evident that ergodic sum rates under AS conditions consistently outperform those under HS, owing to reduced fading effects. These findings underscore the decisive influence of channel conditions on system performance and make a compelling case for deploying additional antennas in RSMA-based satellite-terrestrial communication systems.
Fig 4 compares the ergodic sum rate of RSMA, NOMA and OMA as the transmit power increases under AS and HS scenarios. The results show that RSMA consistently achieves a higher sum rate than NOMA at all power levels. This is because RSMA effectively manages interference by splitting messages into common and private parts, allowing for more flexible decoding at the receiver. The performance gap is more noticeable at lower
, where interference plays a bigger role. As
increases, all schemes improve, with OMA being lower at low
(from −30–0 dBm) but becoming the highest from 0 dBm on-wards due to the absence of interference saturation. It is also worth noting that OMA outperforms RSMA in the high-transmit-power region in Fig 4. This behaviour can be explained by the different interference characteristics of the considered schemes. At low and moderate transmit powers, the system performance is strongly affected by noise, and RSMA benefits from its ability to partially decode and manage interference through the common stream. Hence, RSMA achieves a higher ergodic sum rate than NOMA and OMA in this operating region. However, as the transmit power increases, the noise term becomes less significant, and the performance of RSMA becomes increasingly limited by the residual inter-user interference among the private streams. This leads to an interference-saturation effect in the high-SNR regime. By contrast, OMA allocates orthogonal time/frequency resources to different users, thereby avoiding intra-resource inter-user interference. Although this orthogonalization causes a resource-partitioning loss, the absence of inter-user interference enables OMA to achieve a higher sum rate than RSMA at sufficiently high transmit powers under the considered power allocation and channel settings. This observation does not contradict the advantages of RSMA; rather, it highlights that the relative performance of RSMA, NOMA, and OMA depends on the operating SNR regime, interference level, and resource-allocation strategy.
Fig 5 illustrates the variation of the ergodic sum rate changes with the power coefficient for the common stream. The results indicate that increasing
initially improves the sum rate, but beyond a certain point, the performance starts to decline. This happens because allocating too much power to the common stream reduces the power available for private streams, which limits overall system efficiency. The optimal value of
depends on system conditions, such as the number of users and channel fading. The results also show that the sum rate is higher under AS than HS, as stronger fading in HS reduces signal quality. These findings highlight the importance of properly adjusting
to balance common and private stream power for the best system performance.
Fig 6 illustrates the ergodic sum rate of the proposed RSMA scheme versus for Q = 2, 3, and 4 users under both HS and AS conditions. It can be observed that the ergodic sum rate decreases slightly as the number of users Q increases. This trend is theoretically justified since the system operates in an overloaded regime. In such scenarios, the spatial degrees of freedom are limited, which prevents the MRT beamforming from completely eliminating inter-user interference among the private streams. Furthermore, because the common stream’s rate is strictly bounded by the user experiencing the worst channel conditions, adding more users increases the probability of encountering a deeper fade, thereby compressing the common rate floor. Nevertheless, the system avoids severe outage and maintains stable connectivity across all users, confirming the scalability of the RSMA transmission framework under massive access setups.
Fig 7 presents the SER versus the transmit power under different fading conditions. The results show that as
increases, the SER decreases for all cases, which is expected because higher transmit power improves signal strength and reduces detection errors. However, the rate of improvement varies depending on the shadowing conditions. The SER is consistently lower under AS compared to HS since stronger fading in HS degrades the signal quality. The results also confirm that the analytical expression closely follows the simulation data, validating the accuracy of the derived model. These findings emphasize the importance of considering fading effects in system design, especially in low-power scenarios where SER is more sensitive to channel variations.
Finally, Fig 8 illustrates the SER versus the power coefficient for the common stream under BPSK and QPSK modulation schemes. The results show that as
increases, the SER consistently decreases. This trend indicates that allocating more power to the common stream improves the overall detection accuracy, reducing symbol errors. Additionally, the SER for BPSK is lower than for QPSK at the same
, which aligns with the fact that BPSK is more robust to noise and fading compared to QPSK. The SER under AS is also lower than under HS, showing that severe fading conditions significantly degrade performance. These findings highlight the importance of adjusting
appropriately and considering modulation type to enhance reliability in satellite-terrestrial communication systems.
6 Conclusion
This paper investigated the performance of an RSMA-enabled satellite-terrestrial communication system under shadowed-Rician fading. Closed-form analytical expressions were derived for the ergodic sum rate and symbol error rate, providing useful insight into the effects of power allocation, fading severity, and the number of satellite antennas on system performance. The numerical results verified the accuracy of the proposed analysis and showed that RSMA can achieve notable performance gains over conventional NOMA in terms of spectral efficiency and interference management, particularly under challenging fading conditions. Nevertheless, it is important to emphasize that the analytical framework developed in this work relies on several idealized benchmark assumptions. In particular, perfect CSI is assumed to enable tractable MRT beamforming and closed-form performance analysis. In practical satellite-terrestrial systems, CSI may be imperfect due to channel estimation errors, limited or quantized feedback, feedback delay, beam pointing mismatch, and CSI aging. Such imperfections may reduce the accuracy of beamforming and power allocation, thereby degrading the achievable ergodic sum rate and SER performance. In addition, the present analysis assumes ideal SIC at the receivers. In practical implementations, SIC may be affected by decoding errors, receiver impairments, and residual interference. Residual SIC can limit the effectiveness of common-stream cancellation and may increase the interference observed during private-stream detection, leading to performance degradation, especially in interference-limited operating regimes. Moreover, the considered analytical model does not explicitly capture dynamic satellite mobility. While the GEO-based benchmark provides a useful reference case, more dynamic satellite scenarios, such as LEO systems, may experience time-varying propagation distances, Doppler shifts, rapid beam movement, and frequent channel variations. These effects can make CSI acquisition, beam tracking, and adaptive resource allocation significantly more challenging. Therefore, the results presented in this paper should be interpreted as a theoretical benchmark for RSMA-enabled satellite-terrestrial networks under idealized operating conditions.
Future work will extend the proposed framework by incorporating imperfect CSI, residual SIC, hardware impairments, and dynamic satellite mobility. Robust RSMA transmission strategies, adaptive beam tracking, and low-complexity power allocation algorithms will also be investigated to improve the applicability of the proposed framework to practical satellite-terrestrial deployments.
References
- 1. Asgharzadeh-Bonab A, Kalbkhani H, Azimi Y, Ahmadi F. Overview of Hybrid Satellite–Terrestrial Networks (HSTNs): Key Technologies and Upcoming Challenges. J Comput Netw Commun. 2025;2025(1).
- 2. Zhang Y, Zhang H, Zhou H, Long K, Karagiannidis GK. Resource Allocation in Terrestrial-Satellite-Based Next Generation Multiple Access Networks With Interference Cooperation. IEEE J Select Areas Commun. 2022;40(4):1210–21.
- 3. Heydarishahreza N, Han T, Ansari N. Spectrum Sharing and Interference Management for 6G LEO Satellite-Terrestrial Network Integration. IEEE Commun Surv Tutorials. 2024;27(5):2794–825.
- 4. De Gaudenzi R, Garde T, Giannetti F, Luise M. A performance comparison of orthogonal code division multiple-access techniques for mobile satellite communications. IEEE J Select Areas Commun. 2002;13(2):325–32.
- 5. Ghosh J, Ra IH, Singh S, Haci H, Al-Utaibi KA, Sait SM. On the comparison of optimal NOMA and OMA in a paradigm shift of emerging technologies. IEEE Access. 2022;10:11616–32.
- 6. Pham T-HT, Nguyen N-TT, Nguyen Q-S, Nguyen TH, Minh BV, Nguyen Q-S, et al. Performance Analysis in D2D Partial NOMA-Assisted Backscatter Communication. AEEE. 2025;23(3).
- 7. Le V-D, Nguyen H-N, Nguyen SQ, Bui T-T, Hien DC, Kim BS. Enabling D2D Transmission Mode of Reconfigurable Intelligent Surfaces Aided in Wireless NOMA System. AEEE. 2025;23(1).
- 8. Mohsan SAH, Li Y, Shvetsov AV, Varela-Aldás J, Mostafa SM, Elfikky A. A Survey of Deep Learning Based NOMA: State of the Art, Key Aspects, Open Challenges and Future Trends. Sensors. 2023;23(6):2946.
- 9. Kaneko M, Randrianantenaina I, Dahrouj H, Elsawy H, Alouini M-S. On the Opportunities and Challenges of NOMA-Based Fog Radio Access Networks: An Overview. IEEE Access. 2020;8:205467–76.
- 10. Mao Y, Dizdar O, Clerckx B, Schober R, Popovski P, Poor HV. Rate-Splitting Multiple Access: Fundamentals, Survey, and Future Research Trends. IEEE Commun Surv Tutorials. 2022;24(4):2073–126.
- 11. Tegos SA, Diamantoulakis PD, Karagiannidis GK. On the Performance of Uplink Rate-Splitting Multiple Access. IEEE Commun Lett. 2022;26(3):523–7.
- 12. Zhou G, Mao Y, Clerckx B. Rate-Splitting Multiple Access for Multi-Antenna Downlink Communication Systems: Spectral and Energy Efficiency Tradeoff. IEEE Trans Wireless Commun. 2022;21(7):4816–28.
- 13. Katwe M, Singh K, Clerckx B, Li C-P. Improved Spectral Efficiency in STAR-RIS Aided Uplink Communication Using Rate Splitting Multiple Access. IEEE Trans Wireless Commun. 2023;22(8):5365–82.
- 14. Krishnan A, Sabapathy S, Maruthu S. STAR-IRS Assisted Rate Splitting Multiple Access with Perfect and Imperfect CSI for 6G Communication. IEEE Latin Am Trans. 2025;23(1):17–24.
- 15. Chrysologou AP, Tegos SA, Diamantoulakis PD, Chatzidiamantis ND, Sofotasios PC, Karagiannidis GK. On the Coexistence of Heterogeneous Services in 6G Networks: An Imperfection-Aware RSMA Framework. IEEE Trans Commun. 2024;72(11):7152–67.
- 16. Lee B, Shin W. Max-Min Fairness Precoder Design for Rate-Splitting Multiple Access: Impact of Imperfect Channel Knowledge. IEEE Trans Veh Technol. 2022;72(1):1355–9.
- 17. Han S, Li Z, Xue Q, Meng W, Li C. Joint Broadcast and Unicast Transmission Based on RSMA and Spectrum Sharing for Integrated Satellite–Terrestrial Network. IEEE Trans Cogn Commun Netw. 2024;10(3):1090–103.
- 18. Chen H, Mi D, Wang T, Chu Z, Xu Y, He D, et al. Rate-Splitting for Multicarrier Multigroup Multicast: Precoder Design and Error Performance. IEEE Trans Broadcast. 2021;67(3):619–30.
- 19. Zhang J, Ge J. Cooperative Rate-Splitting for Downlink Multiuser MISO Systems With Partial CSIT. IEEE Trans Commun. 2021;69(11):7504–19.
- 20. Liu H, Tsiftsis TA, Kim KJ, Kwak KS, Poor HV. Rate Splitting for Uplink NOMA With Enhanced Fairness and Outage Performance. IEEE Trans Wireless Commun. 2020;19(7):4657–70.
- 21. Bansal A, Singh K, Clerckx B, Li C-P, Alouini M-S. Rate-Splitting Multiple Access for Intelligent Reflecting Surface Aided Multi-User Communications. IEEE Trans Veh Technol. 2021;70(9):9217–29.
- 22. Khan WU, Ali Z, Lagunas E, Mahmood A, Asif M, Ihsan A, et al. Rate Splitting Multiple Access for Next Generation Cognitive Radio Enabled LEO Satellite Networks. IEEE Trans Wireless Commun. 2023;22(11):8423–35.
- 23. Su X, Yuan Y, Wang Q. Performance analysis of rate splitting in K-user interference channel under imperfect CSIT: average sum rate, outage probability and SER. IEEE Access. 2020;8:136930–46.
- 24. Bansal A, Singh K, Li C-P. Analysis of Hierarchical Rate Splitting for Intelligent Reflecting Surfaces-Aided Downlink Multiuser MISO Communications. IEEE Open J Commun Soc. 2021;2:785–98.
- 25. Soleymani M, Santamaria I, Jorswieck EA, Di Renzo M, Schober R, Hanzo L. Rate Splitting Multiple Access for RIS-Aided URLLC MIMO Broadcast Channels. IEEE Trans Wireless Commun. 2026;25:1529–46.
- 26. Li X, Fan Y, Yao R, Wang P, Qi N, Miridakis NI, et al. Rate-Splitting Multiple Access-Enabled Security Analysis in Cognitive Satellite Terrestrial Networks. IEEE Trans Veh Technol. 2022;71(11):11756–71.
- 27. Lee J, Lee J, Yin L, Shin W, Clerckx B. Coordinated Rate-Splitting Multiple Access for Integrated Satellite-Terrestrial Networks With Super-Common Message. IEEE Trans Veh Technol. 2024;73(2):2989–94.
- 28. Wu Y, Zhou F, Wu W, Wu Q, Ng DWK, Quek TQS. Robust Resource Allocation for RSMA Spectrum Sharing Networks. IEEE Trans Wireless Commun. 2024;23(11):16375–89.
- 29. Samy R, Yang H-C, Rakia T, Alouini M-S. Ergodic Capacity Analysis of Satellite Communication Systems With SAG-FSO/SH-FSO/RF Transmission. IEEE Photonics J. 2022;14(5):1–9.
- 30. Shi S, Li G, An K, Li Z, Zheng G. Optimal Power Control for Real-time Applications in Cognitive Satellite Terrestrial Networks. IEEE Commun Lett. 2017;21(8):1–1.
- 31.
Gradshteyn IS, Ryzhik IM. Table of integrals, series, and products. Academic Press; 2014.
- 32. Bankey V, Upadhyay PK, Da Costa DB, Bithas PS, Kanatas AG, Dias US. Performance Analysis of Multi-Antenna Multiuser Hybrid Satellite-Terrestrial Relay Systems for Mobile Services Delivery. IEEE Access. 2018;6:24729–45.
- 33. Tin PT, Nguyen TN, Tran M, Trang TT, Sevcik L. Exploiting Direct Link in Two-Way Half-Duplex Sensor Network over Block Rayleigh Fading Channel: Upper Bound Ergodic Capacity and Exact SER Analysis. Sensors (Basel). 2020;20(4):1165. pmid:32093275
- 34. Vu T-H, Nguyen T-T, Nguyen TN, Tu L-T, Voznak M. Symbiotic Communication Systems in the Internet of Things: A Framework for Double Adaptive Performance Analysis. IEEE Wireless Commun Lett. 2026;15:905–9.
- 35. Nguyen T, Le C, Do D. Performance analysis of multi‐user NOMA over shadowed fading. Electron Lett. 2020;56(15):771–3.
- 36.
Prudnikov AP, Brychkov YA, Marichev OI. Integrals and series, volume 3: more special functions. Gordon and Breech Science; 1990.
- 37. Vu T-H, Benevides da Costa D, Kim S, Pham Q-V. Outage, Capacity, and Error Performance of Downlink RSMA-Based Systems: Analysis and Resource Optimization. IEEE Trans Commun. 2025;73(8):6868–83.
- 38. Tam DT, Nguyen BC, Lam SC, Vinh NV, Nguyen TN. SER performance of millimeter-wave communications with multiple reconfigurable intelligent surfaces and transmit antenna selection. AEU - Int J Electron Commun. 2023;160:154517.
- 39.
Abramowitz M, Stegun IA. Handbook of mathematical functions with formulas, graphs, and mathematical tables. U.S. Department of Commerce, National Bureau of Standards; 1964.
- 40. Nguyen TN, Kim H-Y, Tran PT, Vu Minh B, Kim B-S, Tu L-T, et al. On the Performance of Secured Ambient Backscatter Communications to Protect Digital Content and Copyrights. IEEE Access. 2025;13:195385–400.
- 41. Miridakis NI, Vergados DD, Michalas A. Dual-Hop Communication Over a Satellite Relay and Shadowed Rician Channels. IEEE Trans Veh Technol. 2014;64(9):4031–40.
- 42. Nguyen TN, Van Chien T, Tran D-H, Minh BV, Ngon NC, Voznak M, et al. Cooperative Satellite-Terrestrial Networks With Imperfect CSI and Multiple Jammers: Performance Analysis and Deep Learning Evaluation. IEEE Syst J. 2024;18(4):2062–73.
- 43. Nguyen N-T, Nguyen H-N, Nguyen N-L, Le A-T, Nguyen TN, Voznak M. Performance Analysis of NOMA-Based Hybrid Satellite-Terrestrial Relay System Using mmWave Technology. IEEE Access. 2023;11:10696–707.