Abstract

This thesis presents a dynamic security vulnerability detection framework that sets up an infrastructure for automatic security testing of Free and Open Source Software (FOSS) projects. It makes three contributions to the design and implementation of a dynamic vulnerability detection system. Firstly, a mathematical model called Team Edit Automata is defined and implemented for security property specification. Secondly, an automatic code instrumentation tool is designed and implemented by extending the GNU Compiler Collection (GCC). The extension facilitates seamless integration of code instrumentation into FOSS projects' existing build system. Thirdly, a dynamic vulnerability detection system is prototyped to integrate the aforementioned two techniques. Experiments with the system are elaborated to automatically build, execute, and detect vulnerabilities of FOSS projects. Overall, this research demonstrates that monitoring program with Team Edit Automata can effectively detect security property violation.