Abstract:  The exponential growth of scale and complexity in distributed systems necessitates significant maintenance efforts. Logs play an indispensable role in system operation and maintenance since they record crucial runtime information. However, recent studies on log anomaly detection have primarily focused on deep learning methods, which entail high computational complexity for learning temporal and semantic features from logs. Moreover, most deep learning-based approaches for log anomaly detection require supervised training, which is labor-intensive. To address these challenges, this article proposes a framework called GAE-Log. GAE-Log leverages event graphs and knowledge graphs to model logs comprehensively. By integrating temporal dynamics through event graphs and incorporating contextual information from knowledge graphs, GAE-Log enhances the understanding of the system's status. Moreover, GAE-Log employs adversarial training of autoencoders for anomaly detection on logs. The effectiveness of GAE-Log is evaluated through an ablation study and comprehensive comparisons using both public and synthetic log datasets. The results demonstrate that GAE-Log outperforms state-of-the-art methods in log anomaly detection, achieving significant performance improvements.

Date: 2023-08-25

Recommended citation: Y. Xie and K. Yang, "Log Anomaly Detection by Adversarial Autoencoders With Graph Feature Fusion," in IEEE Transactions on Reliability, doi: 10.1109/TR.2023.3305376.

[Download paper here] (https://ieeexplore.ieee.org/abstract/document/10231001)

Abstract:  Hadoop provides a powerful platform that allows reliable, scalable, and distributed processing of massive data sets across a cluster of computers. Log data record events taking place in the Hadoop system that helps to understand system activities and diagnose problems. However, system upgrades and updates often change the syntax and patterns of logs, rendering the machine-learning models that were designed for the legacy system ineffective. Retraining the machine-learning models with new data sets from scratch might improve the accuracy of the machine-learning model. Nevertheless, annotating new data sets is often time-consuming and labor-intensive. In this article, we propose a domain adaptive log anomaly prediction framework called LogAT to effectively transfer learned knowledge from the existing labeled data set (source domain) to the new unlabeled data set (target domain) by adopting an unsupervised domain adaption method. Furthermore, a hierarchical anomaly knowledge graph has been constructed to represent the domain knowledge that facilitates the subsequent detection and diagnosis of system faults. Extensive experiments have been conducted on public and real-world data sets to validate the effectiveness of the proposed framework as well as each module. Our results show that LogAT achieves superior performance over the state-of-the-art methods for predicting log anomalies and acquiring considerable performance improvement in terms of AUC-ROC score on different Hadoop application data sets.

Date: 2022-06-06

Recommended citation: Y. Xie and K. Yang, "Domain Adaptive Log Anomaly Prediction for Hadoop System," in IEEE Internet of Things Journal, vol. 9, no. 20, pp. 20778-20787, 15 Oct.15, 2022, doi: 10.1109/JIOT.2022.3178873.

[Download paper here] (https://ieeexplore.ieee.org/abstract/document/9789316)

Abstract:  The Hadoop platform provides a powerful software framework for distributed storage and processing of massive amounts of data. It is at the heart of big data processing and has found numerous applications in diverse areas, ranging from environmental monitoring to security analysis. To facilitate the storage and processing of big data, a Hadoop platform typically runs on a cluster of servers and may scale up to process big data over thousands of hardware nodes. However, the growing scale and complexity of the Hadoop platform also make it increasingly challenging to manage and operate. In this paper, we present a framework called LogM that leverages not only the deep learning model but also the knowledge graph technology for failure prediction and analysis of the Hadoop cluster. In particular, we first develop a CAB net (Convolutional Neural Network (CNN) with an attention-based Bi-directional Long Short Term Memory (Bi-LSTM)) architecture to effectively learn the temporal dynamics from the sequential log data, which allows us to predict system failures. We then adopt a knowledge graph approach for failure analysis and diagnosis. Extensive experiments have been carried out to assess the performance of the proposed approach. It is seen that LogM is highly effective in predicting and diagnosing system failures.

Date: 2021-04-30

Recommended citation: Y. Xie, K. Yang, and P. Luo, "LogM: Log Analysis for Multiple Components of Hadoop Platform," in IEEE Access, vol. 9, pp. 73522-73532, 2021, doi: 10.1109/ACCESS.2021.3076897.

[Download paper here] (https://ieeexplore.ieee.org/abstract/document/9420108)

Abstract:  With the recent advances in the field of computer vision, Convolutional Neural Networks (CNNs) are widely used in organ segmentation of computed tomography (CT) images. Based on the Dense V-net model, this paper proposes a simplified version with postprocessing methods to help reduce the fragments in organ segmentation results. Compared with the baseline method that uses a sharpmask model with conditional random field (SM+ CRF), our model improves the Dice ratio of the Esophagus, Heart, Trachea, and Aorta by 10%, 4%, 7%, and 6%, respectively.

Date: 2019-04-08

Recommended citation: M. Feng, W. Huang, Y. Wang, Y. Xie. Multi-organ Segmentation using Simplified Dense V-net with Post-processing. In SegTHOR@ISBI 2019.

[Download paper here] ( https://pagesperso.litislab.fr/wp-content/uploads/sites/19/2019/04/SegTHOR2019_paper_11.pdf )

Abstract:  The particle degradation problem of the particle filter (PF) algorithm caused by the reduction of particle weights significantly influences the positioning accuracy of target nodes in wireless sensor networks. This study presents a predictor to obtain a particle swarm of high quality by calculating non-linear variations ranging between particles and flags and modifying the reference distribution function. To this end, probability variations of distances between particles and star flags are calculated and the maximum inclusive distance using the maximum probability of a high-quality particle swarm is obtained. The quality of particles is valued by the Euclidean distance between the predicted and real observations, and hereafter particles of high quality are contained in a spherical coordinate system using the distance as diameter. The simulation results show that the proposed algorithm is robust and the computational complexity is low. The method can effectively improve the positioning accuracy and reduce the positioning error of target nodes.

Date: 2019-03-01

Recommended citation: C. Zhang, T. Xie, K. Yang, H. Ma, Y. Xie, Y. Xu, and P. Luo. (2019), Positioning optimization based on particle quality prediction in wireless sensor networks. IET Netw., 8: 107-113.

[Download paper here] ( https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049/iet-net.2018.5072)