RQ2: Effectiveness of LAD Methods under In Shift Scenarios

Research Question: How effective are LAD methods under normality shift?

Answer to Research Question:

All LAD methods are impacted by normality shifts, which lead to decreased detection performance by up to 34% in F1-Score. Specifically, prediction-based LAD methods are more sensitive to distribution shifts than reconstruction-based methods. We observe that the semantically enhanced traditional method, SemPCA, also performs well in the attack detection tasks, with an average F1-Score of 0.93. Its capability to learn the statistical significance of log distribution allows it to exhibit promising robust detection ability in scenarios with cloud architecture shifts. Its average performance (F1-Score of 0.78) under various shift scenarios outperforms most of the evaluated baselines. On the other hand, even though the semi-supervised PLELog model and the advanced LogAD model demonstrate superior performance in in-distribution data, they tend to experience a more significant performance degradation under shift scenarios compared to traditional deep learning models, with an average decline of 20.9% and 21.3%, respectively.