A. Reuther (MIT Lincoln Lab) – Anatomy of a Supercomputer

10:00 to 11:30, Monday August 26

Laptops are everywhere, and almost all of us are familiar with how to use laptops. And with that familiarity we have picked up some knowledge of how laptops, and the networks to which they connect, work. But most of us have little experience with HPC systems. In this session, we will begin by comparing and contrasting our familiar laptops to HPC systems, particularly with regard to executing scientific computing programs. We will cover processors, interconnects, networks, storage, and job scheduling to build our understanding of the anatomy of a supercomputer. After establishing this system basis, we will continue by introducing and explaining the various HPC workflows under which all parallel and distributed scientific applications fall. We will explain how these workflows are executed on HPC systems. Finally we will discuss how to predict, measure, and optimize how these workflows perform on HPC systems.

See the session materials

Speaker bio: A. Reuther

Dr. Albert Reuther is the manager of the MIT Lincoln Laboratory Supercomputing Center (LLSC). He brought supercomputing to Lincoln Laboratory through the establishment of LLGrid, founded the LLSC, and leads the LLSC Computational Science and Engineering team. He developed the gridMatlab high-performance computing (HPC) cluster toolbox for pMatlab and is the computer system architect of the MIT Supercloud and numerous interactive supercomputing clusters based on Supercloud, including those in the LLSC.

As a computational engineer, he has worked with many teams within the Laboratory and beyond to develop efficient parallel and distributed algorithms to solve a wide array of computational problems. The Supercloud architecture earned him an Eaton Award for Design Excellence and his computational engineering work earned him a 2017 R&D 100 Award. He is the technical chair of the IEEE High Performance Extreme Computing Conference and has organized numerous workshops on interactive HPC, cloud HPC, economics of HPC, and HPC security. His areas of research include interactive HPC; computer architectures for machine learning, graph analytics, and parallel signal processing; and computational engineering.

Dr. Reuther earned a dual BS degree in computer and electrical engineering in 1994, an MS degree in electrical engineering in 1996, and a PhD degree in electrical and computer engineering in 2000, all from Purdue University. In 2001, he earned an MBA degree from the Collège des Ingénieurs in Paris, France, and Stuttgart, Germany.

J.C. Wright (MIT) – HPC programming practices

13:30 to 15:30, Monday afternoon August 26

On the first afternoon of this week long lecture series, we dive in to some practical aspects of using HPC systems with a focus on the Engaging system which is available to the class for demos and practice during the program. This session is divided into three parts: a) Practical ‘nix: connecting and managing your environment, b) the batch system and managing your jobs, 3) building software and the art of debugging. The goal is to give you the tools and knowledge to avoid some common pitfalls in modern computing environments and the ability to figure out why something is not working.

See the session materials 

Speaker bio: J.C. Wright

John Wright is a plasma physicist working on the theory and simulation of radio frequency heating and current drive in fusion plasmas. He has also dabbled in high temperature superconductors. In the course of developing massive parallel versions of the popular radio frequency codes, TORIC and TORLH, he has played a role in maintaining and operating several computer clusters, including Engaging, the computer used for this class. He is interested in parallel programming, scientific data management, and fusion. He has been awarded the Infinite Mile Award at MIT for supporting parallel computer infrastructure and the Landau-Spitzer award for outstanding technical contributions to heating scenarios in collaboration with European colleagues. He received his Doctorate in Plasma Physics from Princeton University in 1998.

J. Candy (General Atomics) - Numerical methods and solvers in the context of modern multicore/accelerated systems

4 PM to 5:30 PM, Monday August 26

In this presentation we will describe a handful of well-established and powerful numerical discretization techniques including (1) upwind schemes for advection, (2) Legendre and Fourier pseudospectral methods, and (3) dealised fast-Fourier transforms (FFTs). The implementation of these techniques will also be discussed in the context of CGYRO and modern parallel computing (MPI plus OpenMP/OpenACC), and the connection to packages like FFTW/cuFFT, LAPACK, and so on. Lessons learned and some guidelines for writing modern HPC code on heterogeneous systems (multicore and accelerated) will be described. Rules of thumb and key results will be favoured over technical details.

See the session slides and code

Speaker bio: J. Candy

Dr. Candy's current research focus is broadly centered on plasma transport, including gyrokinetic and neoclassical theory, simulation, and applications to predictive modeling. He also maintains an active interest in numerical methods, theoretical acoustics and parallel computing. In the past he has worked on both linear and nonlinear kinetic theory as well as turbulent, neoclassical and energetic particle transport. He developed the GYRO (with R. Waltz and E, Belli), CGYRO (with E. Belli) and TGYRO codes, and managed the development of NEO (with E. Belli). These computational tools are in use at fusion laboratories worldwide. Before starting his Ph.D. research, he independently discovered the 4th-order explicit symplectic integrator. Dr. Candy received his Ph.D. in 1994 under the supervision of M.N. Rosenbluth. He has received various Canadian awards, including 4 NSERC postgraduate fellowships, 2 NSERC postdoctoral scholarships, and the Sir James Lougheed Award of Distinction from the Alberta Government. In 2003, he was the inaugural recipient of the Rosenbluth Award for fusion theory. He was 2008 Jubileum Professor at Chalmers University, and was elected Fellow of the American Physical Society in 2009 for discoveries in multiscale turbulence.

T. Mattson (Intel) – an Introduction to Parallel Programming using OpenMP

9 AM to 5:30 PM, Tuesday August 27

Parallel computers require parallel software. Maybe someday we’ll figure out how to generate parallel software from serial code, but until that day arrives, parallel software requires a human to write parallel programs.

In this hands-on series, we will explore parallel programming. We will cover the key concepts and the most common design patterns used in scientific applications. The focus will be on writing multithreaded software for shared memory computers.

We will use the C programming language with OpenMP. It would be great if you could load the latest gcc or icc compilers on your laptops before CPS-FR 2019. If that is not possible, we will find remote servers you can use instead, but it’s best if you use your own laptops so you can continue working with OpenMP on your own after CPS-FR 2019. Don’t worry if you are not a C programmer. We will cover enough C for you to do the exercises associated with these lectures.

By the end of this full day session, you will know enough OpenMP to do serious parallel programming on real applications. The fact you can do so much after just one day of work is why we use OpenMP for this lecture series. It’s that simple to learn.

T. Mattson (Intel) - Parallel programming beyond threads

5:30 PM to 6:30 PM, Thursday August 29.

Now that you have mastered multithreaded programming for shared memory, we will move on to the two other major classes of parallel hardware used in HPC. We will talk about distributed memory systems (such as clusters) and the Single Instruction Multiple Threads or SIMT model (GPUs).

Programming these machines is quite different than using threads with OpenMP. We will show, however, that most of what you learned with OpenMP directly translates onto these other architectures. Given the time we have for this lecture, we will not use a hands-on format and won’t achieve the level of mastery you were able to achieve with OpenMP. We will, however, cover enough to you can deeply understand how to use MPI to program clusters and a SIMT programming model (such as OpenCL, CUDA or OpenMP) for GPUs.

See the session materials and code

Speaker bio: T. Mattson

Tim Mattson is a parallel programmer obsessed with every variety of science (Ph.D. Chemistry, UCSC, 1985). He is a senior principal engineer in Intel’s parallel computing lab at Intel.

Tim has been with Intel since 1993 where he has worked with brilliant people on great projects including: (1) the first TFLOP computer (ASCI Red), (2) the OpenMP and OpenCL programming languages, (3) two different research processors (Intel's first TFLOP chip and the 48 core SCC), (4) Data management systems (Polystore systems and Array-based storage engines), and (5) the GraphBLAS API for expressing graph algorithms as sparse linear algebra. Currently he leads a programming systems research group at Intel and collaborates with researchers at MIT on the intersection of AI and data systems (dsail.csail.mit.edu).

Tim is passionate about teaching. He has been teaching OpenMP longer than anyone on the planet with OpenMP tutorials at numerous venues including every SC conference but one since 1998. He has published four books on different aspects of parallel computing with a new one due late in 2019 titled “The OpenMP Common Core: making OpenMP Simple Again”.

A. Edelman (MIT) - What makes the Julia programming language fast, fun, and flexible?

9 AM to 10:30 AM and 11 AM to 12 PM, Wednesday August 28

Julia is raising the level of abstraction injecting computer science principles into scientific and engineering computations. We will discuss the key research area of performance computing which is not "How do I sacrifice my life to the performance deities for a little wall-clock time?" but rather "How can we all benefit from a high level language that allows for composible, maintainable code, which can work on heterogeneous computers?"

See the session material

More notebooks on Julia

Speaker bio: A. Edelman

Professor Alan Edelman (Math,CSAIL,CCE) loves to prove pure and applied theorems, program computers and everything in between. He has received many prizes including a Gordon Bell Prize, a Householder Prize, and a Charles Babbage Prize, is a fellow of IEEE, AMS, and SIAM, and is a creator of the Julia programming language. He passionately believes in more interactions between classical computer science and computational science.

Edelman's research interests include Julia, high-performance computing, numerical computation, linear algebra, random matrix theory and geometry. He has consulted for IBM, Pixar, Akamai, Intel, and Microsoft among other corporations and has cofounded Interactive Supercomputing and Julia Computing.

K. Fischer (Julia Computing) - Introduction to Julia

12 PM to 12:30 PM and 1:30 PM to 14:30 PM, Wednesday August 28

Invented at MIT, the Julia programming language is a modern language designed to solve the toughest computational science problems with ease. Julia solves the two language problem, combining usability and expressiveness known from languages like Python or MATLAB, with the performance of systems languages like Fortran, C or C++. Julia is the only high-level dynamic language in the "Petaflop club", having demonstrated sustained performance at petascale. Julia has found wide adoption in industry and academia, being used for such diverse tasks as medical image analysis, astronomy, macroeconomic modeling, condensed matter theory, climate science, machine learning and many others.

This talk will cover the basic operating principles of Julia, how it manages to solve the two language problem, as well as highlighting some of the key usage of julia across the various fields of scientific exploration.

See the session slides

Speaker bio: K. Fischer

Keno Fischer is one of the principal developers of the Julia programming language and the Co-Founder and CTO at Julia Computing, the company founded by the creators of Julia to advance Julia in industry. In 2018, Keno was named one of Forbes' "30 under 30" for his work on Julia. Keno holds a Master's degree in Physics from Harvard University.

S. Smith (General Atomics) - Version Control and the OMFIT Framework

2:30 PM to 3:30 PM, Wednesday August 28

Version control of code is essential for coordinating development amongst many developers or even for checkpointing one's own code. The git version control system (VCS) will be introduced, including the concepts of branches, commits, merges, and tags. Websites like GitHub.com, bitbucket.com, or local GitLab installations exist to facilitate distributed development of code with the git VCS. The concepts of repositories, forks, issues, and pull requests will be covered. The concepts of ssh keys will be covered as necessary. One essential part of code development is unit testing. The joining of git sharing websites with continuous testing, such as travis.ci, eases the burden of making sure to run tests for each commit. Finally, for code to be reliably shared, it must be occasionally tagged with a version number to freeze the application programmer interface (API) at that version. Semantic versioning provides the guide for how to label your version number, and how to increment it. The packaging of the released code with a specification of the dependencies can be automated with a package manager, and the conda program's features will be explained as a cross platform package manager, with conda-forge as the recipe distributor.

The OMFIT framework exists to streamline experimental analysis, ease use of remote computing resources where legacy codes are compiled, and facilitate the sharing of code between users. It is written in python, and makes use of the standard stack of python scientific libraries. Another highlight of the framework is the ease with which graphical user interfaces can be developed, which provides convenient interfaces for non-python users. Users of OMFIT enjoy the various existing classes for reading and writing legacy file formats, such as EFIT g-files or MDS+ trees, as well as future formats, such as the ITER Integrated Modeling and Analysis Suite (IMAS) format.

See the session materials

Speaker bio: S. Smith

Sterling Smith grew up in Utah. After high school he spent two years in the Dominican Republic then went on to graduate from Utah State University with a Bachelor's in Physics and a Minor in Mathematics. For graduate school, he attended Princeton University, earning a Master's and Ph.D. in Astrophysical Sciences (Program in Plasma Physics); his thesis on MHD stability was awarded the Ray Grimm Memorial Prize in Computational Physics. He is currently a scientist at General Atomics, employed in the experimental sciences (DIII-D) division. His main physics research involves the validation of transport models for tokamak plasmas. In pursuit of streamlining experimental analysis, model evaluation, and code sharing, he has co-developed the OMFIT framework [http://gafusion.github.io/OMFIT-source/index.html], which is quickly becoming the framework of choice for the analysis of both domestic and international tokamak experiments with over 50 contributors to its codebase. As the main reviewer of OMFIT code (via GitHub pull requests), he earned the award "He sees you when you're coding; he knows which scripts you break" from DIII-D management as nominated by other OMFIT developers. Outside of work, Sterling is active with his wife and 5 children in church and sports activities.

W. Bethel (Lawrence Berkeley National Lab) - Viewing computational and experimental sciences through the lens of Data Science

4 PM to 5:30 PM, Wednesday August 28

Drawing upon case studies, challenges, and opportunities articulated in several reports from the DOE's Office of Science, we will examine data's critical role in science. From these case studies, several common themes emerge, many of which can be viewed through the lens of Data Science, which is a term that includes diverse topics like data modeling, high performance computing, understanding, prediction, and many more. After an introduction to these concepts, we will engage in interactive discussion to identify specific data science challenges and opportunities in computational and experimental fusion science.

See the lecture material

Speaker bio: W. Bethel

In the pursuit of cultivating technologies that can have a positive impact on science and society, Bethel is an innovator, a visionary, an entrepreneur, a mentor, a data and computer scientist, a technologist, and a leader. His technical background in computer and data science spans several fields, which lie at the nexus of computing and data: high performance computing, scientific/information visualization, graphics, scientific data/image analysis, and machine learning. Over the course of his 30+ years in the field, he has been instrumental in advancing the field of high performance scientific visualization through the R&D of new methods and software tools, and applying them with success to many fields of science. He is passionate about enabling scientific knowledge discovery through the convergence of data and computing, a pursuit that requires ongoing evolution and adaptation to a changing technological and scientific landscape. He is a Senior Computer Scientist at the Lawrence Berkeley National Laboratory, an ACM Distinguished Scientist, a Senior Fellow at the Berkeley Institute for Data Science (UC Berkeley), an Adjunct Associate Professor of Computer Science at Old Dominion University, and is a member of ACM (SIGGRAPH, SIGHPC) and IEEE. He earned his PhD in Computer Science from the University of California, Davis.

Y. Marzouk (MIT) - Bayesian modeling and computation for inverse problems

9 AM to 10:30 AM and 11 AM to 12:30 PM, Thursday August 29

Inverse problems formalize the process of learning about a system through indirect, noisy, and often incomplete observations. Casting inverse problems in the Bayesian statistical framework provides a natural framework for quantifying uncertainty in parameter values and model predictions, for fusing heterogeneous sources of information, and for optimally selecting experiments or observations.

These lectures will present fundamentals of the Bayesian approach to inverse problems, covering both modeling issues---e.g., prior distributions, likelihoods, hierarchical models---and computational challenges---e.g., computing posterior expectations via Markov chain Monte Carlo sampling or principled approximations. We will discuss methods that expose low-dimensional structure in inverse problems, that attempt to mitigate the computational cost of repeated forward model evaluations, and that exhibit discretization-invariant performance in large-scale problems. We will also present Bayesian approaches to optimal experimental design, which attempt to answer ubiquitous questions of what or where to measure, what experimental conditions to employ, etc. Here we will introduce a decision theoretic Bayesian design formulation and link it to more classical alphabetic optimality criteria, then focus on computational issues, e.g., how to estimate and maximize expected information gain in various quantities of interest.

See the session slides

Speaker bio: Y. Marzouk

Youssef Marzouk is an associate professor in the Department of Aeronautics and Astronautics at MIT, and co-director of the MIT Center for Computational Engineering. He is also director of MIT’s Aerospace Computational Design Laboratory and a core member of MIT's Statistics and Data Science Center. His research interests lie at the intersection of physical modeling with statistical inference and computation. In particular, he develops methodologies for uncertainty quantification, inverse problems, large-scale Bayesian computation, and optimal experimental design in complex physical systems. His methodological work is motivated by a wide variety of engineering, environmental, and geophysics applications. He received his SB, SM, and PhD degrees from MIT and spent several years at Sandia National Laboratories before joining the MIT faculty in 2009. He is a recipient of the Hertz Foundation Doctoral Thesis Prize (2004), the Sandia Laboratories Truman Fellowship (2004-2007), the US Department of Energy Early Career Research Award (2010), and the Junior Bose Award for Teaching Excellence from the MIT School of Engineering (2012). He is an Associate Fellow of the AIAA and currently serves on the editorial boards of the SIAM Journal on Scientific Computing, Advances in Computational Mathematics, and the SIAM/ASA Journal on Uncertainty Quantification, among other journals. He is also an avid coffee drinker and occasional classical pianist.

R. Moser (UT Austin) - Uncertainty, Validation and Prediction with Computational Models

4 PM to 5:30 PM, Thursday August 29

We are interested in uncertainties in computational models because of their importance to interpreting the results of such models. In this talk we consider the impact of uncertainty on the two such interpretation situations: the validation of models and the use of models for prediction. The formulation of a comprehensive process to validate computational models depends on a number of factors, including how the models are to be used, the nature of the models and the characteristics of the available data. However, in all cases determining whether the model and the available data are consistent in light of the uncertainties in both is fundamental. We will thus discuss techniques for assessing consistency between uncertain models and uncertain data. Included will be discussion of probabilistic validation criteria, the consequences of epistemic and aleatoric uncertainties, the design and use of statistical test quantities, and the selection of data for validation.

One of the most challenging and important applications of computational models of physical systems is to make predictions when no observations of the quantities being predicted are available. This is the usual situation when model results are to be used to support decisions (e.g. design or operations decisions) where predictions are needed precisely because observational data are not available when the decision must be made. Predictions, then, are essentially extrapolations of available information to the quantities and scenarios of interest. We will discuss how such extrapolative predictions can be made reliably, and how uncertainties enter the assessment of prediction reliability. Of particular interest will be the nature and structure of the models involved, uncertainties that may arise from model error and the sufficiency of the validation process.

See the session slides

Speaker bio: R. Moser

Robert D. Moser holds the W. A. "Tex" Moncrief Jr. Chair in Computational Engineering and Sciences and is Professor of Mechanical Engineering in thermal fluid systems. He serves as the Director of the Center for Predictive Engineering and Computational Sciences (PECOS) and Deputy Director of the Institute for Computational Engineering and Sciences (ICES). Moser received his PhD in mechanical engineering from Stanford University. Before coming to the University of Texas, he was a research scientist at the NASA-Ames Research Center and then a

Professor of Theoretical and Applied Mechanics at the University of Illinois. Moser conducts research on the modeling and numerical simulation of turbulence and other complex fluid flow phenomena. He also uses direct numerical simulation to investigate and model turbulent flows, particularly the development and evaluation of large eddy simulation models. Moser has also been working to develop new approaches for the validation of and quantification of uncertainty in computational models and to assess their reliability. He has pursued applications to such diverse systems as reentry vehicles, solid propellant rockets, micro-air vehicles, turbulent combustion, tokamak fusion and energy harvesting. He is a Fellow of the American Physical

Society, and was awarded the NASA Medal for Exceptional Scientific Achievement.

S. Sra (MIT) - Modern Optimization for Machine Learning

9 AM to 10:30 AM, Friday August 30

The lecture will talk about the role of optimization in machine learning. We will begin by recalling the special nature of machine learning tasks, which guides the types of optimization models used as well as the algorithms. We will focus more on some optimization aspects of importance to training deep neural networks, covering some of their optimization properties as well as noting some avenues worth future investigation.

See the lecture slides

Speaker bio: S. Sra

Suvrit Sra is an Associate Professor in the EECS department at MIT. He is also a core faculty member of the Institute for Data Systems and Society (IDSS) and PI in the Laboratory for Information and Decision Systems (LIDS). He is a member of the MIT ML-group (ml.mit.edu) and is also affialited with the Statistics and Data Science Center. His research spans multiple topics in mathematical optimization, in particular for machine learning (he has published an edited book "Optimization for Machine Learning", MIT Press, 2011). More broadly, his research covers theory of deep learning, discrete probability theory, as well as topics in differential geometry and combinatorics. His applied interests range across synthetic biology, signal processing, healthcare AI, and "data-driven science."

A. Canziani (NYU) - Autoencoders and adversarial networks

11 AM to 12:30 PM and 1:30PM to 3:30 PM, Friday August 30

«The brain has about 10^14 synapses and we only live for about 10^9 seconds. Even just considering binary synapses, a learning algorithm would still require 10^5 bits of information per second to learn all the connections in the brain. This motivates the idea that humans must do a lot of unsupervised learning since the perceptual input (including proprioception) is the only place we can get 10^5 dimensions of constraint per second.» Geoffrey Hinton.

In these two lectures we'll explore two common network architectures (variational autoencoders and adversarial generative nets) that learn to capture the structure and properties of high dimensional data without supervision, which is usually very expensive to get. We'll code up an instantiation, use it as generative model, and compare interpolation in the latent and input spaces, paying attention to when we end up outside the data manifold.

See the lecture slides

Speaker bio: A. Canziani

Alfredo Canziani is a Postdoctoral Deep Learning Research Scientist and Lecturer at NYU Courant Institute of Mathematical Sciences, under the supervision of professors KyungHyun Cho and Yann LeCun. His research mainly focusses on Machine Learning for Autonomous Driving. He has been exploring deep policy networks actions uncertainty estimation and failure detection, and long term planning based on latent forward models, which nicely deal with the stochasticity and multimodality of the surrounding environment. In his spare time, Alfredo is a professional musician, dancer, and cook, and keeps expanding his free online video course on Deep Learning and PyTorch.

N. Fulton (MIT-IBM Watson AI Lab) - Introduction to Safe Reinforcement Learning

4 PM to 5:30 PM, Friday August 30

Reinforcement Learning algorithms construct effective control policies by learning what actions are most likely to maximize an observed reward function. Reinforcement learning is at the heart of several recent breakthroughs in computer science, including better-than-human performance on board games and video games. These successes inspire optimism that reinforcement learning can tackle previously intractable real-world control problems but also raise concern about the safety of using reinforcement learning in safety-critical settings. This lecture will introduce reinforcement learning, survey the state-of-the-art in reinforcement learning, and introduce several techniques for ensuring that reinforcement learning algorithms do not take catastrophically unsafe actions during the learning process. Students will learn how to derive fundamental theorems about reinforcement learning algorithms, will implement deep reinforcement learning algorithms in PyTorch, and will apply these algorithms to several safety-critical control tasks.

See the lecture slides

Speaker bio: N. Fulton

Nathan Fulton is a computer scientist in the MIT-IBM Watson AI Lab, where he works on the foundations of safe artificial intelligence. Nathan develops theory, algorithms, and tools for verifying the safety of autonomous physical systems. Nathan is a core developer of the KeYmaera X verification tool, which he developed together with other members of the Logical Systems Lab while earning his Ph.D. in Carnegie Mellon University's Computer Science Department.