Abstract

Protocol implementations are the keystone for services on the Internet, and they are more likely to suffer from Denial-of-Service (DoS) attacks if containing vulnerabilities. Among DoS-related vulnerabilities in protocol implementations, resource exhaustion vulnerability poses an equal level of security risks as the vulnerability that causes the target service to crash. This is because the vulnerability can deplete the resources of system, leading to unavailability of the service. Despite the significance of this type of vulnerability, there has been limited research in this area. Therefore, we first bridge this gap by conducting an empirical study on resource exhaustion vulnerabilities in protocol implementation and find that exhaustion vulnerabilities related to memory are the major issue. 

However, detecting the memory exhaustion vulnerabilities in protocol implementations is challenging due to the various implementation of protocols, lack of proper detection methods, and unaware of memory consumption along with protocol states. To address these challenges, we propose a dynamic framework called Medusa to detect this type of vulnerability, which consist of exploration and verification phases. The exploration phase constructs protocol property graph (PPG) to embeds the states with relevant properties including memory consumption information. The verification phase utilizes PPG to validate viable DoS attack under simulated environments. We evaluate the effectiveness of Medusa through extensive experiments on 21 implementations of 5 protocols, proving its better performance in exploring the memory consumption along with states compared to baselines, and the capability in verification of DoS attacks. Furthermore, Medusa has discovered six 0-day vulnerabilities, and one has been assigned with CVE number.