Abstract

Federated Reinforcement Learning (FRL) offers a powerful framework for distributed autonomous agents to collaboratively learn decision-making policies while preserving data privacy. This is particularly valuable in domains such as connected autonomous vehicles, where data sensitivity and environmental variability present major challenges. However, practical FRL systems face several obstacles, including environmental heterogeneity, high participation costs, and limited budget availability at the server side for incentivizing agents. To address these issues, we propose a clustered FRL framework that organizes agents into groups based on their operational environments, enabling more efficient training and localized coordination. Each cluster is managed by a local server that aggregates agent updates and forwards refined models to a main server for global aggregation. To encourage sustained participation, we design a hierarchical incentive mechanism: at the lower layer, contract theory is employed due to information asymmetry; at the upper layer, a Stackelberg game is formulated to enable the main server to allocate its limited budget strategically across clusters based on the accuracy of the local servers’ trained models.

Keywords: Federated Learning, Reinforcement Learning, Autonomous Driving, Incentive Design,
Stackelberg Game, Contract Theory, Budget-Constrained Optimization