Abstract

In recent years, data center networks (DCNs) have faced growing pressure from AI and ML workloads with intensive communication patterns and stringent latency requirements. Traditional hierarchical architectures like Clos (Fat-Tree) increasingly struggle with scalability bottlenecks, operational complexity, and congestion under bursty traffic. To address these challenges, this work explores the Structured Re-Arranged Topology (STRAT), which combines expander-graph-inspired path diversity with deterministic structure to enable efficient, scalable, and fault-tolerant designs. Unlike rigid designs, STRAT supports incremental growth, reduced cabling complexity, and better load distribution. This thesis evaluates STRAT not only in simulation but also on real programmable-switch hardware, demonstrating its practical viability. A key contribution is DEALER, a congestion-aware, data-plane-friendly forwarding algorithm leveraging programmable switches. DEALER uses a distributed distance-vector protocol and local queue occupancy to balance load among equal-cost and slightly longer paths, achieving significant improvements over ECMP in high-load scenarios while running at line rate on commercial ASICs. To further enhance STRAT, this work integrates a hybrid electrical-optical fabric with Optical Circuit Switching (OCS) links, guided by proactive, ML-based flow classification. An XGBoost model predicts elephant flows early, enabling their diversion to pre-configured optical paths with minimal control overhead. Simulations show reductions in tail latency and improved throughput. Together, these contributions offer a practical, holistic redesign for DCNs, uniting scalable graph-based topologies, programmable forwarding logic, and ML-guided optical hybridization to meet the performance, efficiency, and scalability demands of modern AI and cloud workloads.