Time-series data analysis

Adaptive Weighting Scheme for Automatic Data Augmentation

Data augmentation methods have been shown to be a fundamental technique to improve generalization in tasks such as image, text and audio classification. Recently, automated augmentation methods have led to further improvements on image classification and object detection leading to state-of-the-art performances. Nevertheless, little work has been done on time-series data, an area that could greatly benefit from automated data augmentation given the usually limited size of the datasets. We present two sample-adaptive automatic weighting schemes for data augmentation: the first learns to weight the contribution of the augmented samples to the loss, and the second method selects a subset of transformations based on the ranking of the predicted training loss. We validate our proposed methods on a large, noisy financial dataset and on time-series datasets from the UCR archive. On the financial dataset, we show that the methods in combination with a trading strategy lead to improvements in annualized returns of over 50%, and on the time-series data we outperform state-of-the-art models on over half of the datasets, and achieve similar performance in accuracy on the others.

Visualising Deep Network's Time-Series Representations

Despite the popularisation of the machine learning models, more often than not they still operate as black boxes with no insight into what is happening inside the model. There exist a few methods that allow to visualise and explain why the model has made a certain prediction. Those methods, however, allow viewing the causal link between the input and output of the model without presenting how the model learns to represent the data. We propose a method that addresses that issue, with a focus on visualising multi-dimensional time-series data. Experiments on a high-frequency stock market dataset show that the method provides fast and discernible visualisations. Large datasets can be visualised quickly and on one plot, which makes it easy for a user to compare the learned representations of the data. The developed method successfully combines known and proven techniques to provide novel insight into the inner workings of time-series classifier models.

Forecasting Financial Time Series Using Robust Deep Adaptive Input Normalization

Deep Learning provided powerful tools for forecasting financial time series data. However, despite the success of these approaches on many challenging financial forecasting tasks, it is not always straightforward to employ DL-based approaches for highly volatile and non-stationary time financial series. To this end, we propose an adaptive input normalization layer that can learn to identify the distribution from which the input data were generated and then apply the most appropriate normalization scheme. This allows for promptly adapting the input to the subsequent DL model, which can be especially important, given recent findings that hint at the existence of critical learning periods in neural networks. Furthermore, the proposed method operates on a sliding window over the time series allowing for overcoming non-stationary issues that often arise. It is worth noting that the main difference with existing approaches is that the proposed method does not just learn to perform static normalization, e.g., using a fixed set of parameters, but instead it adaptively calculates the most appropriate normalization parameters, significantly improving the robustness of the proposed approach when distribution shifts occur. The effectiveness of the proposed formulation is verified using extensive experiments on three challenging financial time-series datasets.

Attention-based Neural Bag-of-Features Learning for Sequence Data

We propose 2D-Attention (2DA), a generic attention formulation for sequence data, which acts as a complementary computation block that can detect and focus on relevant sources of information for the given learning objective. The proposed attention module is incorporated into the recently proposed Neural Bag of Feature (NBoF) model to enhance its learning capacity. Since 2DA acts as a plug-in layer, injecting it into different computation stages of the NBoF model results in different 2DA-NBoF architectures, each of which possesses a unique interpretation. We conducted extensive experiments in financial forecasting, audio analysis as well as medical diagnosis problems to benchmark the proposed formulations in comparison with existing methods, including the widely used Gated Recurrent Units. Our empirical analysis shows that the proposed attention formulations can not only improve performances of NBoF models but also make them resilient to noisy data.

Recurrent Bag-of-Features for Visual Information Analysis

Deep Learning has provided powerful tools for visual information analysis. For example, Convolutional Neural Networks (CNNs) are excelling in complex and challenging image analysis tasks by extracting meaningful feature vectors with high discriminative power. However, these powerful feature vectors are crushed through the pooling layers of the network, that usually implement the pooling operation in a less sophisticated manner. This can lead to significant information loss, especially in cases where the informative content of the data is sequentially distributed over the spatial or temporal dimension, e.g., videos, which often require extracting fine-grained temporal information. A novel stateful recurrent pooling approach, that can overcome the aforementioned limitations, is proposed inspired by the well-known Bag-of-Features (BoF) model, but employs a stateful trainable recurrent quantizer, instead of plain static quantization, allowing for efficiently processing sequential data and encoding both their temporal, as well as their spatial aspects.

Data normalization for Bilinear Structures for time-series

Financial time-series analysis and forecasting have been extensively studied over the past decades, yet still remain as a very challenging research topic. Since the financial market is inherently noisy and stochastic, a majority of financial time-series of interests are non-stationary, and often obtained from different modalities. This property presents great challenges and can significantly affect the performance of the subsequent analysis/forecasting steps. Recently, we proposed the Temporal Attention augmented Bilinear Layer (TABL) has shown great performances in tackling financial forecasting problems. By taking into account the nature of bilinear projections in TABL networks, we propose Bilinear Normalization (BiN), a simple, yet efficient normalization layer to be incorporated into TABL networks to tackle potential problems posed by non-stationarity and multimodalities in the input series. Our experiments using a large scale Limit Order Book (LOB) consisting of more than 4 million order events show that BiN-TABL outperforms TABL networks using other state-of-the-arts normalization schemes by a large margin.

Deep Adaptive Input Normalization

Deep Learning (DL) models can be used to tackle time series analysis tasks with great success. However, the performance of DL models can degenerate rapidly if the data are not appropriately normalized. DAIN is a simple, yet effective, neural layer, that is capable of adaptively normalizing the input time series, while taking into account the distribution of the data. It is trained in an end-to-end fashion using back-propagation and leads to significant performance improvements compared to other evaluated normalization schemes. DAIN differs from traditional normalization methods since it learns how to perform normalization for a given task instead of using a fixed normalization scheme. At the same time, it can be directly applied to any new time series without requiring re-training.

Temporal Logistic Bag-of-Features

Temporal Logistic Neural Bag-of-Features is a lightweight deep learning for time-series analysis. Combining existing BoF formulations with deep feature extractors pose significant challenges: the distribution of the input features is not stationary, tuning the hyper-parameters of the model can be especially difficult and the normalizations involved in the BoF model can cause significant instabilities during the training process. TLBoFs is capable of overcoming these limitations by a employing a novel adaptive scaling mechanism and replacing the classical Gaussian-based density estimation involved in the regular BoF model with a logistic kernel.

Temporal Attention-augmented Bilinear Network

TABL is a neural network layer that incorporates the idea of bilinear projection as well as an attention mechanism that enables the layer to detect and focus on crucial temporal information. The resulting network is highly interpretable, given its ability to highlight the importance and contribution of each temporal instance, thus allowing further analysis on the time instances of interest. TABL can outperform by a large margin other state-of-the-art methods forming much deeper architectures, while requiring far fewer computations.

Attention-based ConvLSTMs for LOB data analysis

The existing literature provides evidence that limit order book data can be used to predict short-term price movements in stock markets. This paper proposes a new neural network architecture for predicting return jump arrivals one minute ahead in equity markets with high-frequency limit order book data. This new architecture, based on Convolutional Long Short-Term Memory with Attention, is introduced to apply time series representation learning with memory and to focus the prediction attention on the most important features to improve performance. The use of the attention mechanism makes it possible to analyze the importance of the inclusion limit order book data and other input variables. Our architecture with this mechanism is used and compared to existing deep learning architectures with the data set that consists of order book data on five liquid U.S. stocks over 18 months. We provide evidence that (i) the new architecture with attention model outperforms existing architectures and (ii) the use of limit order book data was found to improve the performance of the proposed model in jump prediction, either clearly or marginally, depending on the underlying stock. This suggests that path-dependence in limit order book markets is a stock specific feature. Moreover, we find that the proposed approach with an attention mechanism outperforms the multi-layer perceptron network as well as the convolutional neural network and Long Short-Term memory model

Discriminant Bag-of-Words

Discriminant Bag of Words (BoWs) unifies discriminative codebook generation and discriminant subspace learning. It is a framework that is able to, naturally, incorporate several (linear or non-linear) discrimination criteria for discriminant BoWs-based time-series representation. An iterative optimization scheme is used for sequential discriminant BoWs-based time-series representation and codebook adaptation based on classes discrimination in a reduced dimensionality feature space where classes are better discriminated.

References

The list provided in the following may be incomplete. The complete list of papers related to this topic can be found in the lists of journal papers and conference papers.

B. Leporowski and A. Iosifidis, "Visualising Deep Network's Time-Series Representations", arXiv:2103.01716
E. Fons, P. Dawson, X.J. Zeng, J. Keane and A. Iosifidis, “Adaptive Weighting Scheme for Automatic Time-Series Data Augmentation”, arXiv:2102.08310
E. Fons, P. Dawson, X.J. Zeng, J. Keane and A. Iosifidis, “Augmenting transferred representations for stock classification”, IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, Ontario Canada (Online), 2021
N. Passalis, J. Kanniainen, M. Gabbouj, A. Iosifidis and A. Tefas, "Forecasting Financial Time Series using Robust Deep Adaptive Input Normalization", Journal of Signal Processing Systems, accepted December 2020
D.T. Tran, N. Passalis, A. Tefas, M. Gabbouj and A. Iosifidis, “Attention-based Neural Bag-of-Features Learning for Sequence Data”, arXiv:2005.12250
N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Temporal Logistic Bag-of-Features for Forecasting Leveraging High Frequency Limit Order Book Time Series”, Pattern Recognition Letters, accepted 2020
M Krestenitis, N. Passalis, A. Iosifidis, M. Gabbouj and A. Tefas, “Recurrent Bag-of-Features for Visual Information Analysis”, Pattern Recognition, accepted March 2020
D.T. Tran, M. Gabbouj and A. Iosifidis, "Data Normalization for Bilinear Structures in High-Frequency Financial Time-series", International Conference on Pattern Recognition, Milan, Italy (online), 2020
N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Adaptive Normalization for Forecasting Limit Order Book Data using Convolutional Neural Networks”, IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain, 2020
N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Deep Adaptive Input Normalization for Time Series Forecasting”, IEEE Transactions on Neural Networks and Learning Systems, vol. 3, no. 9, pp. 3760-3765, 2020
N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Temporal Bag-of-Features Learning for Predicting Mid Price Movements using High Frequency Limit Order Book Data”, IEEE Transactions on Emerging Topics in Computational Intelligence, (Early Access) DOI: 10.1109/TETCI.2018.2872598, 2019
D.T. Tran, A. Iosifidis, J. Kanniainen and M. Gabbouj, “Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis”, IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 5, pp. 1407-1418, 2019
M. Makinen, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Forecasting jump arrivals in stock prices: new attention-based network architecture using limit order book data”, Quantitative Finance, accepted 2019
N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Deep Temporal Logistic Bag-of-Features for Forecasting High Frequency Limit Order Book Time Series”, IEEE International Conference on Acoustics, Speech, and Signal Processing, Brighton, U.K., 2019
D.T. Tran, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Data-driven Neural Architecture Learning for Financial Time-series Forecasting”, Digital Image and Signal Processing, Oxford, U.K. 2019
A. Tsantekidis, N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, "Using Deep Learning for price prediction by exploiting stationary limit order book features", Applied Soft Computing, 2020
D.T. Tran, M. Magris, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Tensor Representation in High-Frequency Financial Data for Price Change Prediction”, IEEE Symposium Series on Computational Intelligence, Hawaii, USA, 2017
A. Tsantekidis, N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Using Deep Learning to Detect Price Change Indications in Financial Markets”, European Signal Processing Conference, Kos, Greece, 2017
N. Passalis, A. Tsantekidis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Time-series Classification using Neural Bag-of-Features”, European Signal Processing Conference, Kos, Greece, 2017
A. Tsentekidis, N. Passalis, A. Tefas, J. Kanniainen, M. Gabbouj and A. Iosifidis, “Forecasting Stock Prices from the Limit Order Book using Convolutional Neural Networks”, IEEE Conference on Business Informatics, Thessaloniki, Greece, 2017
A. Iosifidis, A. Tefas and I. Pitas, “Discriminant Bag of Words based Representation for Human Action Recognition”, Pattern Recognition Letters, vol. 49, pp. 185-192, 2014
A. Iosifidis, A. Tefas and I. Pitas, “Multidimensional Sequence Classification based on Fuzzy Distances and Discriminant Analysis”, IEEE Transactions on Knowledge and Data Engineering, vol. 93, no. 6, pp. 1445-1457, 2013
A. Iosifidis, A. Tefas and I. Pitas, “Merging Linear Discriminant Analysis with Bag of Words model for Human Action Recognition”, IEEE International Conference on Image Processing, Quebec, Canada, 2015

Google Sites

Report abuse