Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We?

🎉 The research paper "Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We? " was published at 

the ACM International Conference on the Foundations of Software Engineering (FSE 2024).  

✅ You can also access the preprint of this study at arxiv.org.

Abstract

In recent years, the importance of smart contract security has been heightened by the increasing number of attacks against them. To address this issue, a multitude of static application security testing (SAST) tools have been proposed for detecting vulnerabilities in smart contracts.  However, objectively comparing these tools to determine their effectiveness remains challenging. Existing studies have limitations due to the absence of a complete taxonomy and a comprehensive benchmark, which leads to evaluations that are not entirely comprehensive and may display bias. 

In this paper, we first propose a more complete vulnerability taxonomy including 45 unique types for smart contracts. Taking it as a baseline, we develop a comprehensive benchmark that covers almost all types and includes a diverse range of code characteristics, vulnerability patterns, and application scenarios. Based on them, we evaluated 8 SAST tools using this benchmark, which comprises 788 smart contract files and 10,394 vulnerabilities. Our results reveal that the existing SAST tools fail to detect around 50% of vulnerabilities in our benchmark and suffer from high false positives, with precision not surpassing 10%. We also discover that by combining the results of multiple tools, the false negative rate can be reduced effectively, at the expense of flagging 36.77 percentage more functions. Nevertheless, many vulnerabilities, especially those beyond Access Control and Reentrancy vulnerabilities, remain undetected. We finally highlight the valuable insights from our study, hoping to provide guidance on tool development, enhancement, evaluation, and selection for developers, researchers, and practitioners.