Abstract
Unmanned aerial vehicles (UAVs) have been widely applied as aerial relays for satellite-to-terrestrial transmission so that data packets can be delivered either directly from satellites to user equipments (UEs) or relayed via UAVs. To exploit the advantages of such space-aerial cooperative transmission, data transmission, and resource allocation should be jointly optimized under constrained resources in dynamic space-air-ground integrated networks (SAGINs). However, the tight coupling between discrete transmission strategies and continuous resource allocation among users leads to a challenging long-term, large-scale, and non-convex optimization problem. While existing multi-agent reinforcement learning (MARL) techniques have exhibited potential in addressing such problems, their limitations in handling discrete-continuous decisions and ensuring robust multi-agent cooperation often result in significant performance deterioration. In this paper, we propose a new MARL-based Cooperative Transmission Strategy (MCTS) to overcome these limitations. In MCTS, each agent employs a hybrid Actor-Critic (H-AC) algorithm, where the optimal discrete transmission strategy with its associated resource allocation is modeled as parameterized actions and determined by an improved AC structure. To enhance cooperation among agents, a multi-agent cooperation framework is designed based on QMIX, employing a centralized Critic to efficiently guide convergence toward optimal solutions while reducing computational complexity. Simulation results demonstrate the superiority of MCTS over a number of benchmark algorithms and efficiency under various environments, in terms of overall transmitted data and resource utilization.