24 May 2019                                         
Scope of the Workshop

Recently, Deep Learning (DL) has received tremendous attention in the research community because of the impressive results obtained for a large number of machine learning problems. The success of state-of-the-art deep learning systems relies on training deep neural networks over a massive amount of training data, which typically requires a large-scale distributed computing infrastructure to run. In order to run these jobs in a scalable and efficient manner, on cloud infrastructure or dedicated HPC systems, several interesting research topics have emerged which are specific to DL. The sheer size and complexity of deep learning models when trained over a large amount of data makes them harder to converge in a reasonable amount of time. It demands advancement along multiple research directions such as, model/data parallelism, model/data compression, distributed optimization algorithms for DL convergence, synchronization strategies, efficient communication and specific hardware acceleration.
In order to provide a few concrete examples, we seek to advance the following pertinent research directions:

  • Asynchronous and Communication-Efficient SGD: Stochastic gradient descent is at the core of large-scale machine learning. Parallelizing SGD gradient computation across multiple nodes increases the data processed per iteration, but exposes the SGD to communication and synchronization delays and unpredictable node failures in the system. Thus, there is a critical need to design robust and scalable distributed SGD methods to achieve fast error-convergence in spite of such system variabilities.
  • High performance computing aspects: Deep learning is highly compute intensive. Algorithms for kernel computations on commonly used accelerators (e.g. GPUs), efficient techniques for communicating gradients and loading data from storage are critical for training performance.
  • Model and Gradient Compression Techniques: Techniques such as reducing weights and the size of weight tensors help in reducing the compute complexity. Using lower-bit representations allow for more optimal use of memory and communication bandwidth.
This intersection of distributed/parallel computing and deep learning is becoming critical and demands specific attention to address the above topics which some of the broader forums may not be able to provide. The aim of this workshop is to foster collaboration among researchers from distributed/parallel computing and deep learning communities to share the relevant topics as well as results of the current approaches lying at the intersection of these areas.

Call for Papers
In this workshop we solicit research papers focused on distributed deep learning aiming to achieve efficiency and scalability for deep learning jobs over distributed and parallel systems. Papers focusing both on algorithms as well as systems are welcome. We invite authors to submit papers on topics including but not limited to:
  • Deep learning on HPC systems
  • Deep learning for edge devices
  • Model-parallel and data-parallel techniques
  • Asynchronous SGD for Training DNNs
  • Communication-Efficient Training of DNNs
  • Model/data/gradient compression
  • Learning in Resource constrained environments
  • Coding Techniques for Straggler Mitigation
  • Elasticity for deep learning jobs/spot market enablement
  • Hyper-parameter tuning for deep learning jobs
  • Hardware Acceleration for Deep Learning
  • Scalability of deep learning jobs on large number of nodes
  • Deep learning on heterogeneous infrastructure
  • Efficient and Scalable Inference
  • Data storage/access in shared networks for deep learning jobs

Author Instructions
Submitted manuscripts may not exceed ten (10) single-spaced double-column pages using 10-point size font on 8.5x11 inch pages (IEEE conference style), including figures, tables, and references. The submitted manuscripts should include author names and affiliations.

The IEEE conference style templates for MS Word and LaTeX provided by IEEE eXpress Conference Publishing are available for download. See the latest versions here.

Use the following link for submissions: https://easychair.org/conferences/?conf=scadl2019

Proceedings of the workshops are distributed at the conference and are submitted for inclusion in IEEE Xplore after the conference.

Organizing Committee
General Chairs
Gauri Joshi, Carnegie Mellon University (gaurij@andrew.cmu.edu)
Ashish Verma, IBM Research AI (ashish.verma1@us.ibm.com)

Program Chairs
Yogish Sabharwal, IBM Research AI
Parijat Dube, IBM Research AI

Local Chair
Eduardo Rodrigues, IBM Research

Steering Committee

Vijay K. Garg, University of Texas at Austin
Vinod Muthuswamy, IBM Research AI

Technical Program Committee
Alvaro Coutinho - Federal University of Rio de Janeiro
Dimitris Papailiopoulos, University of of Wisconsin-Madison
Esteban Meneses, Costa Rica Institute of Technology
Kangwook Lee, KAIST
Li Zhang, IBM Research
Lydia Chen, TU Delft
Philippe Navaux, University of Rio Grande do Sul
Rahul Garg, Indian Institute of Technology Delhi
Vikas Sindhwani, Google Brain
Wei Zhang, IBM Research
Xiangru Lian, University of Rochester

Key Dates
Paper Submission             February  18, 2019
Acceptance Notification     March        5, 2019
Camera-ready due            March       15, 2019


 8:30-8:40: Welcome remarks
 8:40-9:30: Keynote Talk
Scaling up the training of Convolutional Neural Networks
Marc Snir (University of Illinois at Urbana-Champaign, USA)

 9:30-10:00: Coffee break
 10:00-10:25: Random Walk Gradient Descent for Decentralized Learning on Graphs
Ghadir Ayache (Rutgers University, USA) and Salim El Rouayheb (Rutgers University, USA)
 10:25-11:10: Invited Talk
Scalability and efficiency in data mining and machine learning
Wagner Miera Jr. (Federal University of Minas Gerais, Brazil)
 11:10-11:35: ClPy: A NumPy-compatible Library Accelerated with OpenCL
Tomokazu Higuchi (The University of Tokyo Taura Laboratory, Japan), Naoki Yoshifuji (Fixstars Corporation, Japan), Tomoya Sakai (Fixstars Corporation, Japan), Yoriyuki Kitta (Fixstars Corporation, Japan), Ryousei Takano (National Institute of Advanced Industrial Science and Technology, Japan), Tsutomu Ikegami (National Institute of Advanced Industrial Science and Technology, Japan), and Kenjiro Taura (The University of Tokyo Taura Laboratory, Japan)
 11:35-12:00: Towards Native Execution of Deep Learning on a Leadership-Class HPC System
Srikanth Yoginath (Oak Ridge National Laboratory, USA), Maksudul Alam (Oak Ridge National Laboratory, USA), Arvind Ramanathan (Oak Ridge National Laboratory, USA), Debsindhu Bhowmik (Oak Ridge National Laboratory, USA), Nouamane Laanait (Oak Ridge National Laboratory, USA) and Kalyan Perumalla (Oak Ridge National Laboratory, USA)
 1:30-2:20: Invited Talk
Scaling Deep Learning to Exascale – ACM Gordon Bell Prize 2018
Pedro Silva (NVIDIA)

 2:20-2:45: Compression of Deep Neural Networks by combining pruning and low rank decomposition
Saurabh Goyal (IBM Research, India), Anamitra Roy Choudhury (IBM Research, India), and Vivek Sharma (IBM Research, India)
 2:45-3:30: Invited Talk
Research Opportunities in the Intersection of HPC and AI
Marco Aurelio Stelmar Netto (IBM Research, Brazil)
 3:30-4:00: Coffee break
 4:00-4:25: Panel Discussion: Future directions of research on scalable deep learning on the cloud
 4:25-4:30: Concluding remarks and closing

For any queries, please contact scadl.pdi@gmail.com