Testing CodeQL with SAST-MT

From our study towards issues and PR, we observe the challenge and demand for doing testing on any Q-SAST tools. Hence, we developed a proof-of-concept metamorphic testing framework, SAST-MT, to detect FPs, FNs ,and AFs in any Q-SAST framework. Given the popularity of CodeQL, we only support testing of CodeQL in our initial release. Nevertheless, SAST-MT is easily extensible to support any other SAST framework. Adoption/extension of our mutators to the C/C++ code generator (e.g. Csmith, yarpgen ) of any existing test pipeline should be straightforward as well.

Our framework tries to detect analysis inconsistency by first applying semantics-preserving mutations towards the seed (with known vulnerabilities identified by honggfuzz), and recording the mutant and its corresponding parent if any inconsistency is detected. Each mutation transforms programs at different hierarchies, including types, structures ,and data flow, and we will outline the mutation strategies we implemented in a later section. These methods are inspired by code diversification schemes, they are designed to perform incremental mutation to ease debugging. From our testing, we detected 17 false positives and 228 false negatives of CodeQL’s static analysis within 100 hours of testing. We will publicly maintain our framework to benefit follow-up research. Currently, a snapshot of our codebase is released here.