Introduction
From our study towards issues and PR, we observe the challenge and demand for doing testing on any Q-SAST tools. Hence, we developed a proof-of-concept metamorphic testing framework, SAST-MT, to detect FPs, FNs ,and AFs in any Q-SAST framework. Given the popularity of CodeQL, we only support testing of CodeQL in our initial release. Nevertheless, SAST-MT is easily extensible to support any other SAST framework. Adoption/extension of our mutators to the C/C++ code generator (e.g. Csmith, yarpgen ) of any existing test pipeline should be straightforward as well.
Our framework tries to detect analysis inconsistency by first applying semantics-preserving mutations towards the seed (with known vulnerabilities identified by honggfuzz), and recording the mutant and its corresponding parent if any inconsistency is detected. Each mutation transforms programs at different hierarchies, including types, structures ,and data flow, and we will outline the mutation strategies we implemented in a later section. These methods are inspired by code diversification schemes, they are designed to perform incremental mutation to ease debugging. From our testing, we detected 17 false positives and 228 false negatives of CodeQL’s static analysis within 100 hours of testing. We will publicly maintain our framework to benefit follow-up research. Currently, a snapshot of our codebase is released here.
High level overview of mutation passes
In total, we have implemented 15 transformation passes for our testing framework. An intuitive and simplified example can be found on the below table to illustrate how the transformation works(in a high level overview). Nevertheless, we encourage readers to explore our framework.
Issue we reported
Within 100 hours of testing, our framework detected 17 false positive and 228 false negatives of CodeQL. Given a large number of founding from testing CodeQL, we only listed and reported two types of FNs and two types of FPs we were able to generalize and reproduce manually in most of the related seed-QL pairs. We expect more findings to be updated after we received responses from developers regarding root cause, patch commit, etc.
Version affected: v2.8.1 :
CLI: https://github.com/github/codeql-cli-binaries/releases/download/v2.8.1/codeql-linux64.zip
QL Pack: https://github.com/github/codeql/archive/refs/tags/codeql-cli/v2.8.1.zip