Machine Learning Seminar Series

Many emerging machine learning applications are formulated as either minimax or bi-level optimization problems. This includes, for example, problems in adversarial learning and invariant representation learning that impose robustness and invariance on the model via minimax game, and problems in few-shot learning that train bi-level models for accomplishing different tasks. In this talk, we will provide a unified analysis of the popular optimization algorithms used for solving nonconvex minimax and bi-level optimization problems, based on a novel perspective of identifying the intrinsic potential function of these algorithms. In particular, our analysis establishes the model parameter convergence of these algorithms and characterizes the impact of local function geometry on their convergence rates. Our study reveals that, under some conditions, the dynamic of minimax and bi-level optimization algorithms are similar to that of the gradient descent for nonconvex minimization.

Many challenging image processing tasks can be described by an ill-posed linear inverse problem: deblurring, deconvolution, inpainting, compressed sensing, and superresolution all lie in this framework. Recent advances in machine learning and image processing have illustrated that it is often possible to learn inverse problem solvers from training data that can outperform more traditional approaches by large margins. These promising initial results lead to a myriad of mathematical and computational challenges and opportunities at the intersection of optimization theory, signal processing, and inverse problem theory.

In this talk, we will explore several of these challenges and the foundational tradeoffs that underlie them. First, we will examine how knowledge of the forward model can be incorporated into learned solvers and its impact on the amount of training data necessary for accurate solutions. Second, we will see how the convergence properties of many common approaches can be improved, leading to substantial empirical improvements in reconstruction accuracy. Finally, we will consider mechanisms that leverage learned solvers for one inverse problem to develop improved solvers for related inverse problems.

This is joint work with Davis Gilton and Greg Ongie.

Recent literature has witnessed much progress on the algorithmic and theoretical foundations of Reinforcement Learning (RL), particularly for single-agent problems with small state/action spaces. Our understanding and algorithm toolbox for RL under complex environments, however, remain relatively limited. In this talk, I will discuss some of my work on scalable and probably efficient RL for the challenging settings with large spaces and multiple strategic agents.

First, I will focus on simulation-based methods, as exemplified by Monte-Carlo Tree Search (MCTS). MCTS is a powerful paradigm for online planning that enjoys remarkable empirical success, but lacks theoretical understanding. We provide a complete and rigorous non-asymptotic analysis of MCTS. Our analysis develops a general framework based on a hierarchy of bandits, and highlights the importance of using a non-standard confidence bound (also used by AlphaGo) for convergence. I will further discuss combining MCTS with supervised learning and its generalization to continuous action space.

In the second part of the talk, I will discuss on-policy RL for zero-sum Markov games, which generalizes Markov decision processes to multi-agent settings. We consider function approximation to deal with continuous and unbounded state spaces. Based on a fruitful marriage with algorithmic game theory, we develop the first computational efficient algorithm for this setting, with a provable regret bound that is independent of the cardinality and ambient dimension of the state space.

The Compact Muon Solenoid (CMS) experiment is one of the two large, general purpose detectors at the European Organization of Nuclear Research (CERN), Large Hadron Collider (LHC). The CMS Collaboration is a multinational scientific collaboration, supported by 44 funding agencies from around the world. The LHC began operation in 2010 and will continue to run with significant upgrades into the next decade. The CMS experiment has been operational and collecting data for seven of the past ten years.

The data are collected from the approximately 130 million detector channels at a rate of 2 MByte at a rate of 40 MHz. After two stages of real-time online selection, where only those events that are of direct interest to the Physics program are selected, one thousand events, or 2 GBytes of data, are stored per second for future offline processing. Using this data, the properties of the recently discovered Higgs boson and many other physics processes at the very highest particle energies are studied. Managing and analyzing the tens of PetaBytes of data and metadata is a major challenge for the CMS Collaboration, which is only expected to become more challenging later in the decade, when the High Luminosity-LHC becomes operational with a collision rate that is a factor of five larger.

In order to cope with these challenges, the CMS collaboration has relied on an extensive and diverse set of supervised and unsupervised machine learning methods at every stage of the experiment, from the online selection of events to the offline analysis of the recorded data. This has resulted both in more efficient detector operations, as well as a significantly improved the quality of the physics results delivered. Some examples are the use of Boosted Decision Trees (BDTs) implemented into Field Programmable Gate Arrays (FPGAs) to select only those events that have the characteristics of the physics processes under study, with the requisite decision made within hundreds of nanoseconds. Autoencoders have been used to monitor the quality of the data being recorded and flagging anomalous detector channels. In order to optimize the identification and reconstruction of physics objects like electrons, photons, jets of particles, a variety of BDT and Deep Neural Network based algorithms have been developed. Many of the flagship physics results reported by the CMS Collaboration, like the measurement of the properties of recently discovered Higgs bosons, have been made possible by the use of dedicated machine learning algorithms.

In this seminar the authors will review the machine learning paradigm employed by the CMS collaboration through specific illustrative examples and also indicate directions for the future. As the LHC inches towards the close of the two-year shutdown and prepares for the Run 3, the collaboration has a golden opportunity to further explore novel machine learning methods that will continue to revolutionize the physics output of the CMS experiment.

Causality has recently attracted much interest across research communities in machine learning, computer science, and statistics. One of the fundamental problems in causality is how to find the underlying causal structure or causal model. One focus of this talk, accordingly, is how to find causal relationships from observational data, known as causal discovery. Specifically, I will show recent methodological developments in causal discovery in the presence of distribution shifts, together with their theoretical guarantees and other related issues in causal discovery in complex environments. Besides learning causality, another problem of interest is how causality is able to help understand and advance machine learning. I will show how a causal perspective benefits domain adaptation and forecasting in nonstationary environments. With causal representations, one can naturally make predictions under active interventions and achieve the goal by changing the system properly. Even without interventions involved, they help characterize how the data distribution changes from passively observed data, so that knowledge can be transferred in an interpretable, principled, and efficient way.

The empirical success of deep learning in many fields, especially Convolutional Neural Networks (CNNs) for computer vision tasks, is often accompanied with significant computational cost for training, inference and hyperparameter optimization (HPO). Meanwhile, the mystery of why deep neural networks can be effectively trained with good generalization remains not fully unveiled. With the pervasive application of deep neural networks for critical applications on both cluster and edge devices, there is a surge in demand for efficient, automatic and interpretable model inference, optimization, and adaptation.

In this talk, I will present techniques we developed for reducing neural nets’ inference and training cost, with better understanding about their training dynamics and generalization ability. I will begin by introducing the filter pruning approach for accelerating the inference of CNNs and exploring the possibility of training quantized networks on devices with hardware constraints. Then, I will present how the loss surface of neural networks can be properly visualized with filter-normalized directions, which enables meaningful side-by-side comparisons of generalization ability of neural nets trained in different architectures or hyperparameters. Finally, I will revisit the common practices of HPO for transfer learning tasks. By identifying the correlation among hyperparameters and the connection between task similarity and optimal hyperparameters, the black-box hyperparameter search process can be whitened and expedited.

Natural language is a fundamental form of information and communication and is becoming the next frontier in computer interfaces. Natural Language Interfaces (NLIs) connect the data and the user, significantly promoting the possibility and efficiency of information access for many users besides data experts. All consumer-facing software will one day have a dialogue interface, the next vital leap in the evolution of search engines. Such intelligent dialogue systems should be able to understand the meaning of language grounded in various contexts and generate effective language responses in different formats for information requests and human-computer communication.

In this talk, I will cover three key developments that present opportunities and challenges for the development of deep learning technologies for conversational natural language interfaces. First, I will discuss the design and curation of large datasets to drive advancements towards neural-based conversational NLIs. Second, I will describe the development of scalable algorithms to parse complex and sequential questions to formal programs (e.g. mapping questions to SQL queries that can execute against databases). Third, I will discuss the general advances of language model pre-training methods to understand the meaning of language grounded in various contexts (e.g. databases and knowledge graphs). Finally, I will conclude my talk by proposing future directions towards human-centered, universal, and trustworthy conversational NLIs.

The healthcare industry is arriving at a new era where the medical communities increasingly employ computational medicine and machine learning. Despite significant progress in the modern machine learning literature, adopting the new approaches has been slow in the biomedical and clinical research communities due to the lack of explainability and limited data. Such challenges present new opportunities to develop novel methods that address AI's unique challenges in medicine.

In this talk, we show examples of incorporating medical insight to improve the statistical power of association between various data modalities, design a novel self-supervised learning algorithm, and develop a context-specific model explainer. This general strategy can be employed to integrate other biomedical data, an exciting future research direction discussed briefly.

Modern data applications such as distributed machine learning are revolutionizing all aspects of computing based scientific discovery. As new applications, algorithms, and techniques are invented, the underlying distributed system platforms supporting these uses face fundamentally new challenges. One of such challenges is the workload dynamicity that renders static and design-time system decisions impractical in supporting ever-changing application needs. Studying the workload characteristics of these applications and making informed design decisions can significantly improve the efficiency of the underlying distributed system or platform that enables such applications. Similarly, the resource and data heterogeneity also play an important role in defining the overall performance of these applications.

This talk covers two of my projects where performing workload and resource usage analysis enabled us to design better systems. First, I will show how studying the workload characteristics of Docker - the de facto standard for data center containers management, at enterprise scale using IBM production systems enabled us to better deal with workload dynamicity, and create a number of optimizations to improve application performance. Second, I will present how we enhanced the powerful Federated Learning approach in distributed machine learning by making it aware of the underlying platform characteristics, such as resource and data heterogeneity, and show how the heterogeneity can affect the robustness of trained models under adversarial attacks. I will conclude with a discussion of plans for my future research.

The importance of spatial and spatio-temporal data science is growing with the increasing incidence and importance of large datasets such as trajectories, maps, remote-sensing images, census and geo-social media. Applications include Public Health (e.g., monitoring spread of disease, spatial disparity, food deserts), Public Safety (e.g., crime hot spots), Public Security (e.g., common operational picture), Environment and Climate (change detection, land-cover classification), M(obile)-commerce (e.g., location-based services), etc.

Classical data science and machine learning techniques often perform poorly when applied to spatial and spatio-temporal data sets because of the many reasons. First, these dataset are embedded in continuous space with implicit relationships (e.g., distance), which are important. Second, the cost of spurious patterns is often high in many spatial application domains, which ask for guardrails (e.g., statistical significance tests) to reduce false positives and chance patterns. In addition, one of the common assumptions in classical statistical analysis and machine learning is that data samples are independently generated from identical distributions. However, this assumption is generally false due to spatio-temporal auto-correlation and variability. Ignoring autocorrelation and variability when analyzing data with spatial and spatio-temporal characteristics may produce hypotheses or models that are inaccurate or inconsistent with the data.

Thus, new methods are needed to analyze spatial and spatio-temporal data. This talk surveys common and emerging methods for spatial classification and prediction (e.g., spatial autoregression, GWR), as well as techniques for discovering interesting, useful and non-trivial patterns such as hotspots (e.g., circular, linear, arbitrary shapes ), spatiotemporal interactions (e.g., co-locations , cascade , tele-connections ), spatial outliers, and their spatio-temporal counterparts.

Representing structure in data is at the heart of computer vision and machine learning, i.e., the act of converting raw data into a useful mathematical form. In this talk, I will discuss solutions that are broadly characterized into three themes: the black-box, the manual, and the discovered. First, I will discuss how to use deep generative models to learn structures for face images and its application to image inpainting. Going beyond black-box models, I will explain how to manually impose structures in deep-nets for human pose-regression. Specifically, I will introduce chirality nets, a family of deep-nets that respects left/right symmetry of human poses. Lastly, I will illustrate how to discover pairwise word-to-object structures in the context of textual-grounding and discuss current efforts towards discovering general structures.

Today, even after more than a decade of the cloud computing revolution, users still do not have predictable performance for their applications, and the providers continue to suffer loss of revenue due to poorly utilized resources. Moreover, the environmental implications of these inefficiencies are dire: Cloud-hosted data centers consume as much power as a city of a million people and emit roughly as much CO2 as the airline industry. Fighting these implications, especially in the post Moore's law era, is crucial.

My work points out that the root of these inefficiencies is the gap between the users and the providers. To overcome this divide, my research brings out two key insights for building systems that render the cloud smart, a cloud that is easy-to-use, adaptive, and efficient. First, we must design interfaces to these systems that are intuitive and expressive for users. Such interfaces should open a dialog between users and providers, allowing users to specify high-level application goals, and transfer the responsibility of making low-level resource management decisions to the providers. This opens an opportunity for providers to optimize the use of their resources while still best aligning with user goals. Second, to make the resource management decisions in an adaptive manner in increasingly complex cloud systems, we must leverage Data-Driven or Machine Learning (ML) models. In doing so, my work uses and develops ML algorithms and studies the challenges that such data-driven models raise in the context of systems: modeling uncertainty, cost of training, and generalizability. In this talk, I will present two systems, INFaaS and PARIS, designed to demonstrate the efficacy of these two key insights. These systems represent key steps towards building a smart cloud: they significantly simplify the use of cloud, improve resource efficiency while meeting user goals.

Although we have witnessed growing interests from the continuous optimization community in solving the nonconvex and nonsmooth optimization problems, most of the existing work focus on the Clarke regular objective or constraints so that many friendly properties hold. In this talk, we will discuss the pervasiveness of the non-(Clarke) regularity in the modern operations research and statistical estimation problems due to the complicated composition of nonconvex and nonsmooth functions. Emphasis will be put on the difficulties brought by the non-regularity both in terms of the computation and the statistical inference, and our initial attempts to overcome them.

In modern biomedicine, the role of computation becomes more crucial in light of the ever-increasing growth of biological data, which requires effective computational methods to integrate them in a meaningful way and unveil previously undiscovered biological insights. In this talk, I will discuss my research on machine learning for large- and small-data biomedical discovery. First, I will describe a representation learning algorithm for the integration of large-scale heterogeneous data to disentangle out non-redundant information from noises and to represent them in a way amenable to comprehensive analyses; this algorithm has enabled several successful applications in drug repurposing. Next, I will present a deep learning model that utilizes evolutionary data and unlabeled data to guide protein engineering in a small-data scenario; the model has been integrated into lab workflows and enabled the engineering of new protein variants with enhanced properties. I will conclude my talk with future directions of using data science methods to assist biological design and to support decision making in biomedicine.

Natural language generation (NLG) is a key component of many language technology applications such as dialogue systems, question-answering systems, automatic email replies, and story generation. Despite the recent advances of massive language models like GPT3, texts predicted by such systems are far from any human-like language. In fact, they most often produce either nonfactual text, incoherent text, or pragmatically inappropriate text. Also, the lack of interaction with real users makes the system less controllable and nonpractical. My research is focused on developing linguistically informed computational models in a wide range of generation tasks and building real-world NLG systems which can interact with humans. In this talk, I propose three steps to develop human-centric language generation systems: (i) Studying linguistic theories, (ii) Developing theory-informed models, and (iii) Building human-machine cooperative systems. My research lies at the intersection of three fields: computational linguistics as a theoretical basis, modern machine learning as a powerful technical tool, and human-computer interaction as a robust, reliable interactive testbed.

With its touted ultra-high bandwidth and low latency, 5th generation (5G) wireless technology is envisaged to usher in a new smart and connected world. The goals of our research on 5G are two fold. First, we want to get empirical insights of network and application performance of commercial 5G under several realistic settings, and compare them with its predecessor (4G/LTE). Second, we want to identify novel challenges that are 5G-specific and propose mechanisms to overcome them.

My talk consists of three parts. In the first part, I will describe our measurement study of commercial 5G networks with special focus on millimeter wave (mmWave) 5G. It is to our knowledge a first comprehensive characterization of 5G network performance on smartphones by closely examining 5G service of three carriers (two mmWave carriers, one mid-band carrier) in three U.S. cities. This study finds that commercial mmWave 5G can achieve an impressive throughput of 2 Gbps. However, due to the known poor signal propagation characteristics of mmWave, 5G throughput perceived by the user equipment (UE) is highly sensitive to user mobility and obstructions resulting in a high number of 4G-5G handoffs. Such characteristics of mmWave 5G can make the throughput fluctuate frequently and wildly (between a range of 0 and 2 Gbps) which may confuse applications (e.g., the video bitrate adaptation) and bring highly inconsistent user experiences. Motivated by such insights, the second part of my talk will go beyond the basic measurement and describe Lumos5G - a novel and composable ML-based 5G throughput prediction framework that judiciously considers features and their combinations to make context-aware 5G throughput predictions. Through extensive on-field experiments and statistical analysis, we identify key UE-side factors affecting mmWave 5G performance. Besides geolocation, we quantitatively reveal several other UE-side contextual factors (such as geometric features between UE and 5G panel, mobility speed/mode, etc.) impact 5G throughput -- far more sophisticated than those impacting 4G/LTE. Instead of independently affecting the performance, we find these factors may cause complex interplay that is difficult to model analytically. We demonstrate that compared to existing approaches, Lumos5G is able to achieve 1.37x to 4.84x reduction in prediction error. This work can be viewed as a feasibility study for building what we envisage as a dynamic 5G performance map (akin to Google traffic map). In the third part, I will use our 18-months of experience conducting field experiments of commercial 5G to give my thoughts on the current 5G landscape and highlight both the research opportunities and challenges offered by the 5G ecosystem.

For more information, visit us @ https://5gophers.umn.edu

Randomness is a key resource in designing efficient algorithms, and it is also a fundamental modeling framework in statistics and machine learning. Methods that lie at the intersection of algorithmic and statistical randomness are at the forefront of modern data science. In this talk, I will discuss how statistical assumptions affect the bias-variance trade-offs and performance characteristics of randomized algorithms for, among others, linear regression, stochastic optimization, and dimensionality reduction. I will also present an efficient algorithmic framework, called joint sampling, which is used to both predict and improve the statistical performance of machine learning methods, by injecting carefully chosen correlations into randomized algorithms.

In the first part of the talk, I will focus on the phenomenon of inversion bias, which is a systematic bias caused by inverting random matrices. Inversion bias is a significant bottleneck in parallel and distributed approaches to linear regression, second order optimization, and a range of statistical estimation tasks. Here, I will introduce a joint sampling technique called Volume Sampling, which is the first method to eliminate inversion bias in model averaging. In the second part, I will demonstrate how the spectral properties of data distributions determine the statistical performance of machine learning algorithms, going beyond worst-case analysis and revealing new phase transitions in statistical learning. Along the way, I will highlight a class of joint sampling methods called Determinantal Point Processes (DPPs), popularized in machine learning over the past fifteen years as a tractable model of diversity. In particular, I will present a new algorithmic technique called Distortion-Free Intermediate Sampling, which drastically reduced the computational cost of DPPs, turning them into a practical tool for large-scale data science.

Machine learning techniques have been successful for processing complex information, and thus they have the potential to play an important role in data-driven decision-making and control. However, ensuring the reliability of these methods in feedback systems remains a challenge, since classic statistical and algorithmic guarantees do not always hold.

In this talk, I will provide rigorous guarantees of safety and discovery in dynamical settings relevant to robotics and recommendation systems. I take a perspective based on reachability, to specify which parts of the state space the system avoids (safety) or can be driven to (discovery). For data-driven control, we show finite-sample performance and safety guarantees which highlight relevant properties of the system to be controlled. For recommendation systems, we introduce a novel metric of discovery and show that it can be efficiently computed. In closing, I discuss how the reachability perspective can be used to design social-digital systems with a variety of important values in mind.

Machine learning (ML) can surpass humans on certain complicated yet specific tasks. However, most ML methods treat samples/tasks equally, e.g., by taking a random batch per step and repeating many epochs' training on all data, which may work promisingly on well-processed data using sufficient computation but is extraordinarily suboptimal and inefficient from human perspectives, since we would never teach children or students in such a way. On the contrary, human learning is more strategic and smarter in selecting/generating the training contents for different learning stages via experienced teachers, collaboration of learners, curiosity and diversity in exploration, tracking of learned knowledge and progress, distributing a task into sub-tasks, etc., which have been underexplored in ML. The selection and scheduling of data/tasks is another type of intelligence as important as the optimization of model parameters on given data/tasks. My recent work aims to bridge this gap between human and machine intelligence. As we entering a new era of hybrid intelligence between humans and machines, it is important to make AI not only perform like humans in outcome presentations but also benefit from human-like strategies in its training.

In this talk, I will present several curriculum learning techniques we developed for improving supervised/semi-supervised/self-supervised learning, robust learning with noisy data, reinforcement learning, ensemble learning, etc., especially when the data are imperfect and thus a curriculum can make a big difference. Firstly, I will show how to translate human strategies in curriculum generation to discrete-continuous hybrid optimizations, which are challenging to solve in general but we can develop efficient and provable algorithms using techniques from submodular and convex/non-convex optimization. Curiosity and diversity play important roles in these formulations. Secondly, we build both empirical and theoretical connections between curriculum learning and the training dynamics of ML models on individual samples. Empirically, we find that deep neural networks are fast in memorizing some data but also fast in forgetting some others, so we can accurately allocate those easily forgotten data using training dynamics in very early stages and make the future training only focus on them. Moreover, we find that the consistency of model output overtime for an unlabeled sample is a reliable indicator of its prediction correctness and delineates the forgetting effects on previously learned data. In addition, the learning speed on samples/tasks provides critical information for future exploration. These discoveries are consistent with human learning strategies and lead to more efficient curricula for a rich class of ML problems. Theoretically, we derive a data selection criterion solely from the optimization of learning dynamics in continuous time. Interestingly, the resulted curriculum matches the previous empirical observations and has a natural connection to the neural tangent kernel in recent deep learning theories.

Deep neural networks have seen dramatic improvements in performance, with much of this improvement being driven by new architectures and training algorithms with better inductive biases. At the same time, the future of AI is systems which run in an open-ended way which run on data unlike what was seen during training and which can be drawn from a changing or adversarial distribution. These problems also require a greater scale and time horizon for reasoning as well as consideration of a complex world system with many reused structures and subsystems. This talk will survey some areas where deep networks can improve their biases as well as my research in this direction. These algorithms dramatically change the behavior of deep networks, yet they are highly practical and easy to use, conforming to simple interfaces that allow them to easily be dropped into existing codebases.

Despite the phenomenal empirical successes of deep learning in many application domains, its underlying mathematical mechanisms remain poorly understood. Mysteriously, deep neural networks in practice can often fit training data perfectly and generalize remarkably well to unseen test data, despite highly non-convex optimization landscapes and significant over-parameterization. Moreover, deep neural networks show extraordinary ability to perform representation learning: feature representation extracted from a neural network can be useful for other related tasks.

In this talk, I will present our recent progress on the theoretical foundations of deep learning. First, I will show that gradient descent on deep linear neural networks induces an implicit regularization effect towards low rank, which explains the surprising generalization behavior of deep linear networks for the low-rank matrix completion problem. Next, turning to nonlinear deep neural networks, I will talk about a line of studies on wide neural networks, where by drawing a connection to the neural tangent kernels, we can answer various questions such as how training loss is minimized, why trained network can generalize, and why certain component in the network architecture is useful; we also use theoretical insights to design a new simple and effective method for training on noisily labeled datasets. Finally, I will analyze the statistical aspect of representation learning, and identify conditions that enable efficient use of training data, bypassing a known hurdle in the i.i.d. tasks setting.

Deep learning models are often vulnerable to adversarial examples. In this talk, we will focus on robustness and security of machine learning against adversarial examples. There are two types of defenses against such attacks: 1) empirical and 2) certified adversarial robustness.

In the first part of the talk, we will see the foundation of our winning system, TRADES, in the NeurIPS’18 Adversarial Vision Challenge in which we won 1st place out of 400 teams and 3,000 submissions. Our study is motivated by an intrinsic trade-off between robustness and accuracy: we provide a differentiable and tight surrogate loss for the trade-off using the theory of classification-calibrated loss. TRADES has record-breaking performance in various standard benchmarks and challenges, including the adversarial benchmark RobustBench, the NLP benchmark GLUE, the Unrestricted Adversarial Examples Challenge hosted by Google, and has motivated many new attacking methods powered by our TRADES benchmark.

In the second part of the talk, to equip empirical robustness with certification, we study certified adversarial robustness by random smoothing in the L_infty threat model. On one hand, we show that random smoothing on the TRADES-trained classifier achieves SOTA certified robustness when the L_infty perturbation radius is small. On the other hand, when the perturbation is large, i.e., independent of inverse of input dimension, we show that random smoothing is provably unable to certify L_infty robustness for arbitrary random noise distribution. The intuition behind our theory reveals an intrinsic difficulty of achieving certified robustness by “random noise based methods”, and inspires new directions as potential future work.

We are at an inflection point where software engineering meets the data-centric world of big data, machine learning, and artificial intelligence. As software development gradually shifts to the development of data analytics with AI and ML technologies, existing software engineering techniques must be re-imagined to provide the productivity gains that developers desire. We conducted a large scale study of almost 800 professional data scientists in the software industry to investigate what a data scientist is, what data scientists do, and what challenges they face. This study has found that ensuring correctness is a huge problem in data analytics.

We argue for re-targeting software engineering research to address new challenges in the era of data-centric software development. We showcase a few examples of my group's research on debugging and testing of data-intensive applications: e.g., data provenance, symbolic-execution based test generation, and fuzz testing in Apache Spark. We then conclude with open problems in software engineering to meet the needs of AI and ML workforce.

In a number of emerging AI tasks, collaborations among different organizations or agents (e.g., human and robots, mobile units, and smart devices) are often essential to resolving challenging problems that are otherwise impossible to be dealt with by a single agent. However, to avoid leaking useful and possibly proprietary information, agents typically enforce stringent security measures, which significantly limits such kinds of collaboration. This talk will introduce new research directions in privacy-preserving learning beyond state of the art. A particular focus is on a new learning paradigm named Assisted Learning to enable agents to assist each other in a decentralized, personalized, and private manner. The talk will also introduce a new data privacy framework and a vista of future privacy-preserving machine learning.