To preliminarily investigate whether there are shared vulnerabilities between models or features, we assessed the transferability of adversarial samples across 5 classifiers with a fixed feature type and 8 feature types with a fixed classifier. The adversarial samples used across 40 target models are derived from RQ1 (Section Ⅶ B). Therefore, we have two types of transferability studies: different models with fixed features and different features with fixed models.
The following six figures display the transferability of adversarial samples among different models with a fixed feature type. The x-axis represents the target classifiers used for generating adversarial samples, while the y-axis shows the transfer classifiers where these samples are tested.
Notably, single models (i.e., MLP and KNNs) demonstrate almost zero transferability to ensemble models (i.e., AB and RF), emphasizing the robustness of tree-based classifiers and their unique feature distribution, aligning with the finding from robustness analysis in Section Ⅶ B. This lack of transferability even extends from AB to RF, confirming the distinctiveness of each ensemble model's approach to classification. Conversely, when RF is employed as a surrogate model in AMD methods, it effectively facilitates the transfer of adversarial examples, particularly in single models where ASR often exceeds 74%, especially with the MalScan (Katz, Harmonic, Closeness, Average, and Concentrate) features.
This demonstrates that the ensemble voting mechanism of RF, which utilizes multiple decision trees, enhances its ability to generalize across diverse unseen adversarial examples, suggesting it could be a better surrogate model.
MalScan (Degree)
MalScan (Katz)
MalScan (Harmonic)
MalScan (Closeness)
MalScan (Average)
MalScan (Concentrate)
The following five figures illustrates the transferability across various features with a fixed classifier. The x-axis represents the target features used for generating adversarial samples, while the y-axis shows the transfer features where these samples are tested. For convenience, we abbreviate the various feature types of MalScan as follows: Degree as MalScan_D, Katz as MalScan_K, Harmonic as MalScan_H, Closeness as MalScan_C, Average as MalScan_AVG, and Concentrate as MalScan_CON.
These figures showcases the variability in adaptability among different features, aiding in the identification of the most effective features for use as substitutes in black-box attacks. Most features can easily attack MalScan_H, MalScan_AVG, and MalScan_CON, indicating they are the most vulnerable. Surprisingly, these three features successfully achieve mutual attacks in KNN models, suggesting they share similar patterns.
Conversely, it is challenging to attack MaMadroid, APIGraph, and MalScan_K features, which is consistent with the finding from robustness analysis in Section Ⅶ B. However, within the RF model, the MalScan_K exhibits a relatively lower robustness. The observations suggest a low overlap between features of varying abstraction levels, with more abstract features (e.g., APIGraph) proving to be robust against attacks. APIGraph feature performS effectively as attack vectors across most classifiers, especially which nearly manages to compromise all other features when used on RF. This indicates that more abstract features could potentially encapsulate other features.
MLP
KNN-1
KNN-3
Random Forest
AdaBoost