Research Projects
In terms of data analysis and software development, I have played crucial roles in multiple projects, including:
Project 1
Log anomaly detection and failure prediction using unsupervised machine learning methods
Objectives
The project is centered on detecting anomalies and predicting failures in the big data platform. It is a project that collaborated with a top communications carrier.
Results
Employ the regular expressions to remove variables from the log data and then perform log parsing.
Use the bag-of-word model to vectorize logs to obtain structured features.
Devise a clustering algorithm to automatically extract log templates from nearly 50 million logs.
Mine relations among anomalies through association rule mining and store them in the anomaly knowledge base.
Build an anomaly prediction model based on a neural network to locate failures, and feedback to operators in time.
Project 2
Industrial Internet operation and maintenance and optimization methods using big data analysis
Objectives
The project concentrates on anomaly detection within the industrial Internet scenario, particularly for IoT device data. It is a basic scientific research project.
Results
Perform data collection and cleaning for the downstream tasks.
Build a knowledge graph from the system anomalies and relevant solutions.
Improve the anomaly detection performance through knowledge graph embedding to involve features.
Project 3
Automatic operation and maintenance tool for cloud-native architecture
Objectives
The project is focused on creating an automatic tool for the analysis of log data, time series data, and trace data. It is the industry-university research innovation fund project.
Results
Analyze the log data with the open-source platform to obtain structured log data.
Devise an end-to-end neural network model for feature extraction and anomaly detection of sequential logs.
Integrate the log processing and anomaly detection model to form software products.