Impact of Hyperparameters

Fig. 1 presents the precision, recall, and F1 scores of source code embedding with different 𝜃1 ranging from 0.01 to 1.0, and two settings of 𝛽 (i.e., 40 and 100). When 𝜃1 is 1.0, all source functions are selected, i.e., SafeSCA is degenerated into Centris (DPCNN). Given a fixed 𝛽, the precision decreases and the recall rises with the increase of 𝜃1. When 𝛽 is relatively small, the precision will decrease notably with the increase of 𝜃1 (i.e., the blue dotted curve). However, when 𝛽 is relatively large, the precision is not sensitive, but the recall is not satisfactory when 𝜃1 is small (i.e., the solid orange curve). Compared with precision and recall curves, the F1 score is not sensitive to 𝜃1. Table 3 shows the performance of the source embedding model when 𝜃1 = 2% and 𝛽 = 40. Compared with SafeSCA, it fails to identify functions whose source code are unavailable in the repository, resulting in a low recall. Its cost is tripled since it also embeds functions with representative names.

Fig. 2 presents the precision, recall, and F1 scores of binary code embedding with different 𝜃2. Fig. 10 shows that the precision decreases significantly with the increase of 𝜃2. To avoid high false positive rates, we use a small 𝜃2 = 0.02. Table 3 shows the performance when only binary embedding is used. Besides, we also notice that recent works proposed advanced binary code similarity analysis techniques (e.g., jTrans [69 ]). Since the framework proposed in this paper does not hold any assumption to the underlying binary analysis technique, we envision that SafeSCA can be further improved by replacing the underlying binary embedding model with the latest research outputs.