Optimization of DeepGLEAM on Flu Forecasting Time Series Data

Xiangyi(Tony) Kong, Justin Phen, Rhee Kang

Abstract

The current COVID-19 pandemic and common flu highlight the importance of time- sensitive information in biomedical institutions, politics, and economics. The application of data science in creating real-time predictive models is crucial to help researchers and world leaders better understand disease spread and take preventative measures. Our project aims to optimize the DeepGLEAM model, a deep learning and stochastic process- based predictive model, by better processing raw data and simulating it for improved flu case predictions in the United States. We close the gap in missing data in the Deep- GLEAM model by researching interpolation and imputation techniques to enhance the model’s predictions. The performance of our adjusted model will be compared against a control model for effectiveness.

What are we trying to predict?

Given the baseline of the DeepGLEAM paper, contributed by Professor Yian-Ma and Professor Rose, we aim to predict future flu cases using time-series data with the hybridized model from the paper. Being able to do so would allow us to confirm that the model does in fact perform well beyond just the default COVID case data, and we hopefully are able to optimize the outcome as well.

Data

Our flu forecasting data spans across the year of 2022, with weekly documentation of the cases of flu across 51 states, including the District of Columbia. We have already spent Quarter 1 cleaning the data and spent the first half of Quarter 2 attempting simple interpolation for missing portions of the data (off-season for flu). Due to our default understanding of what the data looks like, we have constructed a main dataframe to test, train, and validate per state. We have also attempted to outsource other parts of CDC released data to create a more comprehensive dataset but we have settled to use the data provided by the mentors.

Methodologies

For flu forecasting, we used DSMLP (UC San Diego’s Data Science/Machine Learning Platform) for training and testing models. In the case of forecasting without interpolation, the limited availability of GLEAM simulation data resulted in a 0.67:0.33 split between the training, validation, and testing sets. Meanwhile, in the case of forecasting with interpolation, we utilized a 50:50 split between the training, validation, and testing sets. Initially, we attempted to interpolated by randomly selecting a portion of the real data and fill in the gaps in the dataset. We utilized 1000 epochs and a batch size of 6 for forecasting without interpolation and 1000 epochs and a batch-size of 11 for forecasting with interpolation. The ADAM optimizer was utilized with a learning rate of 1e−2 for both cases, and early stopping was implemented to prevent overtraining. The input sequential length for the DeepGLEAM Flu model was 1. We also plan on running ground-truth data on DeepGLEAM directly.

For both the Exponential Smoothing (ETS) algorithm and the Auto Regressive Integrated Moving Average (ARIMA) model, we utilized CDC flu ground truth data after 2020-10-10. The parameters for each state were optimized based on grid search results to improve performance.

For Multi-Layer Perceptron (MLP) model, we trained on normalized data for each state individually. In this experiment, a multi-layer perceptron model with 3 hidden layers is developed to solve a regression task. The model is initialized with the input length of 6, output length of 4, and hidden layer length of 256. The model is trained using the Adam optimizer with a learning rate of 0.001. A stepwise learning rate decay is also implemented using the StepLR scheduler. The mean squared loss function is used as the loss function. The model is trained for 300 epochs and the training loss and validation loss are printed every 10 epochs. The best model is saved based on the lowest validation loss.

Method 1 - Basic Interpolation w/ Missing Data

Pandas built-in interpolation package
Failed to make accurate predictions with sudden spike of cases

(Figure Description: Before and After interpolation on US Flu cases)

Method 2 - ARIMA

Replicated as baseline from DeepGLEAM paper
Relatively accurate but negative predictions, which does not make sense because negative flu cases do not exist in real life

(Figure Description: Performance of ARIMA in Comparison to Ground Truth and Prediction Data)

Method 3 - ETS

Roughly accurate, but still underperforms compare to ARIMA

(Figure Description: Performance of ETS in Comparison to Ground Truth and Prediction Data)

Method 4 - Combine MLP & DeepGLEAM

Our best performing model
Outperforms DeepGLEAM and GLEAM

(Figure Description: MLP has historically the lowest MAE, and combined with GLEAM provides the best performing model)

Experiment Results

For evaluating our results, we decide to use MAE (Mean Absolute Error) as our metric and we find that combining MLP & DeepGLEAM performs the best with the lowest MAE across 4 weeks of prediction.

Summary Findings

Upon transitioning from multiple approaches of interpolation, ranging from simple pandas libraries to more advanced neural networks, we have come to realize that the Multi-Layer Perceptron (MLP) merged with GLEAM provides the best model performance, even outperforming the default model provided in the DeepGLEAM research paper.

References

Xinyue Xiong Matteo Chinazzi Alessandro Vespignani Yi-An Ma Rose Yu Dongxia (Allen) Wu, Liyao Gao. Deepgleam: A hybrid mechanistic and deep learning model for covid-19 forecasting, Mar 2021.
Lilian Weng. Domain randomization for sim2real transfer, May 2019.
Yuexin Wu, Yiming Yang, Hiroshi Nishiura, and Masaya Saitoh. Deep learning for epidemiological predictions, Jun 2018.

Page updated

Report abuse