Embedded Files
Tutorial of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD 2024
Tutorial of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD 2024
🗓️ September 9th, 2024 - 🕘 h9 am - 🏛️ Room Omikron
🗓️ September 9th, 2024 - 🕘 h9 am - 🏛️ Room Omikron
This tutorial is targeted to both researchers and practitioners who are interested in setting up fastly-trained recurrent neural networks for structured data. It will open their mind to a computing paradigm different from the most well-known ones, which offers state-of-the-art performances on several applications. Such a paradigm should be considered as its own and implement mixtures of approaches to tackle seriously the environmental challenges. It is also recognized in neuroscience as an important principle to interpret and model brain computations.
This tutorial is targeted to both researchers and practitioners who are interested in setting up fastly-trained recurrent neural networks for structured data. It will open their mind to a computing paradigm different from the most well-known ones, which offers state-of-the-art performances on several applications. Such a paradigm should be considered as its own and implement mixtures of approaches to tackle seriously the environmental challenges. It is also recognized in neuroscience as an important principle to interpret and model brain computations.
Basic concepts of Machine Learning and Deep Neural Networks are suggested prerequisites.
Basic concepts of Machine Learning and Deep Neural Networks are suggested prerequisites.
Abstract
Abstract
Reservoir Computing (RC) is a promising approach for time-series forecasting that offers efficient and sustainable deep learning alternatives to traditional neural networks. This tutorial provides an introduction to sustainable deep learning with RC, including its theoretical foundations, practical implementation, and advantages over other approaches. The tutorial starts with an overview of the RC framework, including the Echo State Network (ESN) and Liquid State Machine (LSM) models. The fundamental principles of these models are explored, including their unique network structure, the learning mechanisms, and the fundamentals of RC mathematical background. The tutorial then proceeds to explore the strong links with computational neuroscience, and the exciting recent advances in neuromorphic hardware implementations of reservoirs. Finally, the tutorial covers practical implementation aspects, existing software libraries, as well as successful application examples in several domains including automotive, avionics, health, and robotics.
In addition to providing practical guidance, this tutorial also emphasizes the importance of sustainability in deep learning. The environmental impact of training deep neural networks is substantial, and RC offers a more energy-efficient approach that is particularly well-suited for time-series forecasting. By using RC, researchers can reduce the carbon footprint of their research while achieving state-of-the-art results.
Overall, this tutorial provides a comprehensive introduction to sustainable deep learning with RC for time-series forecasting, and it is suitable for both beginners and advanced researchers looking to explore this exciting area of machine learning.
Chapter 1: Introduction (9.00 - 9.10)
Chapter 1: Introduction (9.00 - 9.10)
Speaker: Claudio Gallicchio
Speaker: Claudio Gallicchio
Preliminaries on Deep Learning for sequential data and Recurrent Neural Networks (RNNs); green AI and the problem of designing sustainable Deep Learning algorithms.
Preliminaries on Deep Learning for sequential data and Recurrent Neural Networks (RNNs); green AI and the problem of designing sustainable Deep Learning algorithms.
Slides: [here]
Chapter 2: Basics (9.10 - 9.40)
Chapter 2: Basics (9.10 - 9.40)
Speaker: Claudio Gallicchio
Speaker: Claudio Gallicchio
Randomization in Deep Neural Networks; Echo State Networks, Liquid State Machines, and deep architectures; quality of dynamics and network topologies; training algorithms; Reservoir Computing for graphs.
Randomization in Deep Neural Networks; Echo State Networks, Liquid State Machines, and deep architectures; quality of dynamics and network topologies; training algorithms; Reservoir Computing for graphs.
Slides: [here]
Chapter 3: Software libraries and applications (9.40 - 10.20)
Chapter 3: Software libraries and applications (9.40 - 10.20)
Speakers: Xavier Hinaut, Paul Bernard
Speakers: Xavier Hinaut, Paul Bernard
Python frameworks for Reservoir Computing; hyperparameter exploration; building complex architectures; example applications (e.g., sound classification with implicit segmentation) and tricks of the trade. Short hands-on lab demonstration (ReservoirPy): Implementing Echo State Networks in Python for time-series prediction and classification.
Python frameworks for Reservoir Computing; hyperparameter exploration; building complex architectures; example applications (e.g., sound classification with implicit segmentation) and tricks of the trade. Short hands-on lab demonstration (ReservoirPy): Implementing Echo State Networks in Python for time-series prediction and classification.
Slides: [here]
Chapter 4: Mathematical foundations (10.20 - 11.00)
Chapter 4: Mathematical foundations (10.20 - 11.00)
Speaker: Andrea Ceni
Speaker: Andrea Ceni
Recurrent Neural Networks as input-driven dynamical systems. Stability of recurrent neural networks; Memory-nonlinearity trade-off; Edge-of-chaos dynamics.
Recurrent Neural Networks as input-driven dynamical systems. Stability of recurrent neural networks; Memory-nonlinearity trade-off; Edge-of-chaos dynamics.
Slides: [here]
Coffee break (11.00 - 11.20)
Coffee break (11.00 - 11.20)
Chapter 5: Computational Neuroscience of Reservoir Computing (11.20 - 12.00)
Chapter 5: Computational Neuroscience of Reservoir Computing (11.20 - 12.00)
Speaker: Peter Ford Dominey
Speaker: Peter Ford Dominey
Temporal processing hierarchy in the brain; relations between reservoir connectivity and temporal hierarchical processing; simulation of human narrative processing using word embeddings and reservoirs; effects of network topology on narrative processing temporal hierarchy.
Temporal processing hierarchy in the brain; relations between reservoir connectivity and temporal hierarchical processing; simulation of human narrative processing using word embeddings and reservoirs; effects of network topology on narrative processing temporal hierarchy.
Slides: [here]
Chapter 6: Neuromorphic hardware implementations (12.00 - 12.40)
Chapter 6: Neuromorphic hardware implementations (12.00 - 12.40)
Speaker: Gianluca Milano
Speaker: Gianluca Milano
Principles of neural networks in unconventional and neuromorphic hardware; Physical substrates for neuromorphic reservoirs: photonics, memristors, spintronics, and mechanical systems.
Principles of neural networks in unconventional and neuromorphic hardware; Physical substrates for neuromorphic reservoirs: photonics, memristors, spintronics, and mechanical systems.
Slides: [here]
Chapter 7: Quo vadis? (12.40 - 12.50)
Chapter 7: Quo vadis? (12.40 - 12.50)
Speaker: Claudio Gallicchio
Speaker: Claudio Gallicchio
Conclusions and outlook.
Conclusions and outlook.
Slides: [here]
Speakers
Speakers
Claudio Gallicchio is an Assistant Professor of Machine Learning at the Department of Computer Science of the University of Pisa, Italy. He received his PhD from the University of Pisa, where he focused on Reservoir Computing models and theory for structured data.
His research is based on the fusion of concepts from Deep Learning, Recurrent Neural Networks, and Randomized Neural Systems. He is the founder and former chair of the IEEE CIS Task Force on Reservoir Computing.
Andrea Ceni received the M.Sc. degree in Mathematics cum laude from Università degli Studi di Firenze, Italy, in 2017, and the Ph.D. degree from the Department of Computer Science of the University of Exeter, UK, in 2021. He has been a Postdoctoral Research Associate at CEMPS, University of Exeter. Currently, he is a Research Fellow at the Department of Computer Science of the University of Pisa, Italy.
His research interests include recurrent neural networks, deep learning, computational neuroscience, chaos theory, and complex systems.
Peter Ford Dominey
Peter Ford Dominey
Centre National de la Recherche Scientifique (CNRS), Paris, France; and INSERM U1093 Cognition, Action et Plasticité Sensomotrice Dijon, France.
Centre National de la Recherche Scientifique (CNRS), Paris, France; and INSERM U1093 Cognition, Action et Plasticité Sensomotrice Dijon, France.
Peter Ford Dominey is a CNRS Research Director at INSERM U1093 in Dijon France. In 1989 and 1993 respectively he obtained the M.Sc. and Ph.D. in computer science from the University of Southern California, developing neural network models of sensorimotor sequence learning, including the first instances of reservoir computing. In 1997 he joined the CNRS at the Institut des Sciences Cognitives in Lyon France and initiated the Robot Cognition Laboratory.
His research interests include understanding and simulating the neurophysiology of cognitive sequence processing, action and language, and their application to robot cognition and language processing, and human-robot cooperation.
Gianluca Milano is currently a permanent researcher at the Italian National Institute of Metrological Research (INRiM). He received a Ph.D. in Physics cum laude from Politecnico di Torino, Italy, in collaboration with the Italian Institute of Technology (IIT).
His main research interests and activities focus on: i) the investigation of electronic and ionic transport properties and physicochemical phenomena in nanodevices and low dimensional systems; and ii) memristive devices and architectures for memory and neuromorphic applications, from material synthesis to device characterization, modeling, and implementation of unconventional and brain-inspired computing paradigms in neuromorphic architectures.
Xavier Hinaut is Research Scientist in Computational Neuroscience at Inria, Bordeaux, France.
His work is at the frontier of machine learning, neurosciences, linguistics and robotics: from the modelling of human sentence processing to the analysis of birdsongs. He leads ReservoirPy library development. His current focus is on creating complex reservoir architectures (Deep Reservoirs, Reservoir Transformers, ...) to imitate brain processing.
Paul Bernard is research engineer in the Mnemosye team, at Inria, Bordeaux, France. He is the actual main developper of ReservoirPy library. He obtained a Engineering Master diploma of ENSC in 2023.
Google Sites
Report abuse