In the past one year, we've witnessed a significant surge in research papers focusing on time series analysis, particularly emphasizing the application of foundational models for forecasting as well as downstrea tasks such as anomaly detection and prediction. This rapid proliferation of literature raises a critical question about the current state and efficacy of these models in real-world applications. This project critically evaluates the efficacy of Time Series Foundational Models (TSFM) in anomaly detection and prediction tasks. TSFMs are being analysed across multiple datasets, with varying patterns, trends and seasonality (some of them with the absence of discernible trend, seasonality and cyclicity). Additionally, TSFMs require high computational resources while failing to capture sequential dependencies effectively or improve performance in few-shot or zero-shot scenarios.

     Why bother to explore foundational models for time series ?

• • Quick deployment in the absence of enough data.

  • Having one-size fits all setup for time series data (avoiding the need for training/fine-tuning multiple regression models).

  • Exploring data efficient learning methods on time series sensory data.
    
    Implementation

  • Used datasets such as MSL,  Future Factory (FF) dataset, SMD and Pulp and Paper dataset, performing appropriate exploratory data analysis to identify trend and seasonality pattern if any.

  • Evaluated anomaly detection and prediction on various time series foundational models such as Chronos, TimeGPT, Frozen Pretrained transformer (FPT), Moirai and TimeMOE to evaluate their performances in both the zero shot an d fine tuned setting, while comparing their performances to Machine learning model such as weighted XGBoost and deep learning model of autoencoder. 

  • Also, evaluated the time required for fine tuning or training the model for getting the appropriate results.

Results

• • Statistical Deep Learning models consistently outperform TSFM in anomaly detection and prediction both in terms of performance as well as computational efficiency.

  • Statistical models such as weighted XGBoost overall excel in anomaly detection and prediction and are preferred due to its interpretability, using feature importances and ensemble methods to identify variables contributing to anomalies.

  • Deep Learning models, such as autoencoders, learn compressed representation of normal data, making it easy for them to predict anomalies.

  • TSFM designed for general purpose forecasting often struggle to capture task specific nuances needed for anomaly detection and prediction.
    
    Observed limitations of TSFMs

  • TSFMs struggle with datasets that lack clear patterns, trends, or seasonality, especially with datasets exhibiting irregular behavior (e.g., FF dataset). This limitation affects their ability to detect rare and unexpected anomalies effectively, such as in anomaly detection tasks.

  • Only few models such as TimeGPT and FPT are trained for anomaly detection and none for prediction for anomaly prediction using these models we repurposed forecasting to enable us to perform anomaly prediction.

  •  Models like TimeGPT and FPT face difficulties because they aren't sufficiently optimized for detecting rare anomalies. FMs often depend on large-scale data, but time series applications lack abundant anomaly data, leading to issues like missed subtle anomalies or increased false positives.

  • Limited exploration on fine-tuning techniques (only Timegpt, Moirai)

  • Not enough data - Unlike NLP there is no cohesive dataset representation specifically designed and set aside for evaluation, also with the issue of over-representation of domain in the publicly available domain.

Future Scope for TSFMs

• Utilizing data from multiple modalities for training the time series foundational models might help enhance their performance.

• Enhance interpretability, especially for critical applications in healthcare and manufacturing is crucial and this can be done with the help of neuro-symbolic reasoning techniques and domain specific knowledge integration.

• Blending TSFMs with lightweight statistical models (e.g., ARIMA residuals fed into TimeGPT) may offer a combination of interpretability with scalability. For eg. In energy forecasting, ensemble layers could dynamically switch between TSFMs and physics-based models during grid instability events.

• Further research can be focussed on developing novel tokenization, data augmentation, and prompting methods to enhance foundational model learning.

• Incorporate additional variables might help TSFMs to capture complex temporal dynamics (covariate modelling).