Tool Selection

As we mentioned in our paper, to collect representative SAST tool candidates in the recent literature, we followed a three-step process for tool selection. 

1. Systematic Literature Review:

1) We first use several keywords such as “smart contract” and “security tool” to search papers published in top-tier Software Engineering, Security, and Programming Language venues in the last three years, e.g., ASE, ICSE, S&P, PLDI. We obtained 59 research papers related to security for smart contracts as an initial paper list as follows. 

2) Following this, we excluded papers that did not pertain to SAST tools, resulting in 14 relevant papers (details in Google Sheet 2 below.)

2. Tool selection

We initially gathered a list of 44 SAST tools from these 14 recent scientific literature references. Our selection process was structured to ensure the relevance and practicality of the tools for our analysis.

1) Availability: We first excluded 16 tools due to unavailability (commercial or closed-source tools). Notable exclusions in this step included tools like Zeus, Sereum, and MythX.

2) Security Related: Since we would compare and evaluate the effectiveness of vulnerability detection,  we filtered out 2 liters including Solhint and Ethlint.

3) Generalized SAST Tools: We then narrowed our focus to "generalized" SAST tools since we aim to compare and evaluate tools across various (50 unique) vulnerability types in our taxonomy. It required the tools to support at least six types from our taxonomy rather than specializing in a few vulnerability types, such as only aiming at re-entrancy, access control, DoS, or state-insistency, etc. After this filtering, we retained 16 tools but excluded some like SailFish, which only detects SI issues (supporting three types in our taxonomy), and eTainter only for Gas-related issues (supporting three types only).

4) No Additional Input Need: Next, we focused on tools that accept source code directly. We removed 4 tools from our list that required additional inputs like specifications, transactions, or test cases, which were not suitable for our large-scale analysis. 

5) Popularity and Relevance: In the final step, we evaluated the tools based on their # Baselines (over the recent two years), #Citations, and #Stars. We thereby got the seven representative tools that are both popular and relevant in the field. 

For the full list and details on each selection step, please refer to the Google Sheet named "FSE24_SAST_Tool_Candidates" below.

This Google Sheet contains the 14 research papers we used to select popular tools in recent two years (2021-2023).