Abstract

The proliferation of binary executables and IoT/IIoT firmware in modern computing environments has created an urgent need for robust security analysis methods that can withstand the challenges of code transformation, scalability, and false positives. This dissertation advances the state-of-the-art in binary and firmware security through three complementary research directions. First, we address the challenge of binary function similarity under compiler optimizations, obfuscations, and multi-architecture deployments. We design BinFinder, a neural embedding–based system that significantly improves clone detection accuracy, achieving resilience to complex code transformations and delivering substantial gains in recall over existing approaches. Second, we investigate the detection of taint-based vulnerabilities in IoT/IIoT firmware, where static taint analysis often leads to over-tainting and dynamic symbolic execution results in prohibitive computational costs. To overcome these limitations, we develop OctopusTaint, a static taint analysis framework that integrates advanced data flow analysis, backtracking, sanitization inspection, and post-processing filters. OctopusTaint reduces false positives, accelerates analysis, and uncovers both known and potential 0-day vulnerabilities across real-world firmware datasets. Finally, we focus on the crucial task of validating alerts from static analyzers, mitigating the long-standing problem of excessive false positives. We propose TaintPolygraph, a hybrid validation framework that couples static taint analysis with semantic-aware symbolic execution. By incorporating context-specific constraints and exploitability checks, TaintPolygraph effectively distinguishes genuine vulnerabilities from safe cases, reducing false positives by up to 83% across multiple architectures and operating systems. Collectively, these contributions establish a comprehensive methodology for analyzing binaries and IoT/IIoT firmware, encompassing clone detection, scalable vulnerability discovery, and precise alert validation. This research improves the accuracy of function clone search, enhances the reliability of firmware security analysis, alleviates the burden on analysts, and lays the foundation for scalable and trustworthy security solutions in embedded and IoT ecosystems.