Abstract

The Fifth Generation (5G) networks power diverse applications, from autonomous vehicles to smart cities, by enabling ultra-reliable low-latency communications, massive IoT connectivity, and enhanced mobile broadband. At the core of this advancement is the 5G Service-Based Architecture (SBA), which ensures scalability and flexibility through cloud-native deployment and virtualized Network Functions (NFs). The adoption of the Hypertext Transfer Protocol version 2 (HTTP/2) in the 5G SBA has become essential for enabling efficient communication between NFs. However, the adoption of HTTP/2 for NF communication introduces security risks, including stream multiplexing, slow-rate, and rapid-reset attacks, which can lead to Denial of Service (DoS) and disrupt critical services. Addressing these vulnerabilities is essential to maintaining the stability and security of 5G networks.
This thesis explores the impact of HTTP/2 vulnerabilities on the 5G SBA, identifying attack vectors that compromise the Quality of Service (QoS) of critical services. While prior studies largely assessed these threats theoretically, this research demonstrates the practical vulnerabilities of 5G networks to HTTP/2 attacks, such as Stream Multiplexing Attacks (SMAs). To address these challenges, the thesis introduces 5GShield, an application layer anomaly detection solution using autoencoder-based Machine Learning (ML). By profiling normal NF behavior with application-layer features, 5GShield effectively detects deviations indicative of SMAs. Building on this, 5GGuardian is proposed as a more advanced solution to detect nuanced variations of SMAs. Leveraging 5G-Stream features and a time-series transformer, 5GGuardian captures fine-grained NF behaviors and complex patterns in HTTP/2 streams, achieving superior accuracy for both stealthy and non-stealthy anomalies. Recognizing the limitations of single detection approaches, the research introduces an ensemble learning-based solution that leverages and combines the strengths of multiple ML models trained on different feature sets in order to provide superior detection performance of HTTP/2 attacks, including slow-rate and rapid-reset attacks. By providing scalable and advanced anomaly detection, this thesis strengthens 5G SBA security, ensuring reliable service delivery and supporting the secure growth of future communication networks.