CAREER: Binary-Level Security via ABI-Centric Semantic Inference

Understanding the inner workings of software is essential to protect desktop and mobile computers. Lack of source code for most commercial software makes it necessary to analyze and defend software binaries. However, unlike source code, binaries are devoid of rich semantic information that is crucial for security.

Traditional binary analysis and reverse engineering approaches are limited by factors such as obfuscation, choice of compiler and compilation flags, availability of debug information, and underlying instruction set architecture. This project bridges the semantic gap in binary analysis by leveraging the interface between a binary and its environment.

Such interactions are mandated by the Application Binary Interface (ABI) specification. The project is based on the insight that ABI adherence confers certain properties to a binary that form a strong basis for reverse engineering. Because ABI adherence is necessary for interoperability, relying on ABI cues for reverse engineering offers an unprecedented level of robustness that is impervious to obfuscation and compilation environment (e.g., optimization).

This project utilizes two independent yet complementary mechanisms that leverage language ABIs to vastly improve the state of the art in binary analysis and code-reuse attack detection. It employs a combination of static and dynamic binary analysis approaches in order to derive high-level design information (e.g., object-oriented language class diagrams) from binaries.

Such information is central to solving problems in decompilation, software specialization, software similarity detection, etc. While the project evaluates binaries that adhere to Itanium and Microsoft’s MSVC ABIs, the discovered techniques will be applicable to more modern languages such as Rust. Additionally, this project leverages System V ABI, the most popular C language ABI to derive integrity policies for binaries that run on UN*X (unix like) operating systems, and addresses modern code-reuse attacks that operate within the confines of a statically recovered control-flow graph.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.