In this deep learning workshop series, we aim to teach both basic concepts and hands-on skills in deep learning. The content will be broad, thus, we structure the topics in six modules of half a day plus two half-day modules as a technical bridge course (called Day-0). This way we offer the courses as a modular system where everyone can choose a subset of courses freely.

All modules consist of two blocks that are each divided into parts of theoretical and computational introductions and computational tutorials, thus we provide all necessary concepts first and practice them in class based on prepared code examples and tasks. Additional literature and online material will be provided for self-study as well.

To allow all participants to practice our tutorial parts, we will have a dedicated capable computation server hosted within IRCN, thus you do not need to have special hardware at hand (but can translate your experiences easily to other machine learning setups).

(updated on 05.12.2020)

Location: via Zoom (the Zoom link has been sent to all registered participants. Email us if you have not received it.)

Dates:

• Tue 08.12.2020 - Day 0 (Basic + advanced computing and programming - technical bridge)

• Fri 11.12.2020 - Day 1 (Machine Learning introduction + Deep Learning basics )

• Tue 15.12.2020 - Day 2 (Advanced Deep Learning A + B)

• Fri 18.12.2020 - Day 3 (Advanced Deep Learning C + State-of-the-art models)

Timeslots:

• morning: 9:15h - 10:45h and 11:15h - 12:45h

• afternoon: 14:15h - 15:45h and 16:15h - 17:45h

Day-0: Technical bridge, Day-1: Basic Deep Learning introduction, Day-2 + Day-3: Advanced Deep Learning concepts and methods

Day-0 morning - Computing and programming basics: Here we introduce all necessary technical details in order to start. The goal is to be able to use the IRCN computing servers and get into basic programming. Specific topics as follows

• Block 0-1 - Computing basics: IRCN server infrastructure, using ssh keys, using git repositories

• Block 0-2 - Programming basics: Python introduction, numpy-package introduction, Python IDEs

Day-0 afternoon - Advanced programming & computing: We add necessary backgrounds data processing and data visualisation with python as well as python environments and collaborative programming. Topics:

• Block 0-4 - Collaborative development: Jupyter notebooks, jupyterhub, Server-based development environment like Colab

• Block 0-3 - Data basics and visualisation: pandas, matplotlib, and scipy packages and git repositories

Day-1 morning - Machine Learning introduction: We introduce basic machine learning concepts and necessary steps to apply deep learning to a problem or a dataset. For this we look into fundamental preprocessing steps and algorithms. Topics:

• Block 1-1 - Deep Learning introduction: learning types, deep learning representations & metrics

• Block 1-2 - Preprocessing & Linear models: data processing, dimensionality reduction, regression

Day-1 afternoon - Deep Learning basics. We introduce basic concepts for supervised and unsupervised machine learning towards deep learning. We apply simple neural architecture to classify data or find structure in data. Topics:

• Block 1-3 - Supervised MLP Learning: perceptron, multilayer perceptron, activation functions, gradient descent

• Block 1-4 - Unsupervised clustering: conventional iterative methods, neural self-organisation

Day-2 morning - Advanced Deep Learning A: We explain and practice important approaches to understand and control neural networks and introduce specialised architecture characteristics for supervised learning.

• Block 2-1 - Training in Deep Learning: data partitioning, regularisation, hyperparameter optimisation, best practices

• Block 2-2 - Advanced Supervised Architectures: filter, Convolutional Neural Networks

Day-2 afternoon - Advanced Deep Learning B: We introduce specialised architecture characteristics for unsupervised learning and explore analyses options for neural network models. Topics:

• Block 2-3 - Advanced Unsupervised Architectures - Towards unsupervised Learning of Images: autoencoder, generative models

• Block 2-4 - Monitoring & Analysing Deep Networks: analysis tools for representations and dynamics, monitoring tools for training

Day-3 morning - Advanced Deep Learning C: We introduce specialised architectures for sequence learning that include the concept of recurrence in neural networks and explore complex dynamics. Topics:

• Block 3-1 - Sequence learning introduction: sequence classification & prediction, recurrent neural architectures, backpropagation through time

• Block 3-2 - Pattern generation and control via RNNs: reservoir computing, FORCE learning, latent variable analysis

Day-3 afternoon - State-of-the-art models: Finally we review the recent state of the art approaches both for solving problems in machine learning and modelling cognition in computational neuroscience and interdisciplinary research. We focus on key improvements that are available as source code and frameworks and explore them in class.

• Block 3-3 - Towards cognitive modelling: feedforward and recurrent network model examples, including models for predictive coding theories, etc.

• Block 3-4 - Towards problem-solving: current deep learning architectures (conceptual), attention, differentiable memory, etc.

All material is shared via a GitHub:

https://github.com/IDLW/IDLW2020

For all blocks, the material contains slides, specific list of references, and appropriate exercises in python jupyter notebooks.