Time Series Analysis using Recurrent Neural Networks — LSTM

Machine learning and Data Science is going through an exciting time that state of the art Deep Learning techniques can be implemented so quickly. I have tried my hands on in the Keras Deep Learning api and found it very convenient to play with Theano and Tensorflow.

Time series analysis is still one of the difficult problems in Data Science and is an active research area of interest. There are so many examples of Time Series data around us. Predicting the energy price, sales forecasting or be it predicting the stock price of Tesla. The stochastic nature of these events makes it a very difficult problem.

Traditional Time Series analysis involves decomposing the data into its components such as trend component, seasonal component and noise.Techniques such as ARIMA(p,d,q), moving average, auto regression were used to analyze time series. Stateful RNN’s such as LSTM is found to be very effective in Time Series analysis in the recent past.

There is an excellent blog by Christopher Olah for an intuitive understanding of the LSTM networks Understanding LSTM .

What is RNN or Recurrent Neural Networks?

RNN is a special case of neural network similar to convolutional neural networks, the difference being that RNN’s can retain its state of information.

What is LSTM?

LSTM is a variant of the RNN architecture.It was difficult to train models using traditional RNN architectures. Recent advancements demonstrate state of the art results using LSTM(Long Short Term Memory) and BRNN(Bidirectional RNN). The LSTM architecture was able to take care of the vanishing gradient problem in the traditional RNN.

Keras — An excellent api for Deep Learning .

Keras is a high-level API for neural networks and can be run on top of Theano and Tensorflow. Working directly on Tensorflow involves a longer learning curve. Keras also helpes to quickly experiment with your deep learning architecture.

Refer to Keras Documentation at https://keras.io/ for detailed information. There are excellent tutorial as well to get you started with Keras quickly.Google Tensorflow just recently announced its support for Keras which is a reminder of its strong base in the community.

A quick starter script if you want to try this yourself.

Python 3.6, Keras, Tensorflow 1.0

The dataset used is from Time Series Data Library, Monthly Boston armed robberies. Link below.

Data Source

model = Sequential()
model.add(LSTM(input_dim=1, output_dim=16, return_sequences=True))
model.add(Dropout(0.2))

model.add(LSTM(32,return_sequences=False))
model.add(Dropout(0.2))

model.add(Dense(output_dim=1))
model.add(Activation("linear"))

model.compile(loss="mse", optimizer="rmsprop")

model.fit(X_train, y_train, batch_size=512, nb_epoch=epochs, validation_split=0.40)

The model need to be tuned for several hyper parameters for good performance, however for the above code below is the prediction.

References:

Understanding LSTM Networks — http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Deep Learning Classes from Jeremy Howard — http://course.fast.ai/

Recurrent Neural Networks — https://www.tensorflow.org/tutorials/recurrent

Keras Documentation — https://keras.io/

A Critical Review of Recurrent Neural Networks for Sequence Learning — http://arxiv.org/abs/1506.00019

Comments