Over the past two decades, our research group has applied information-theoretic principles, concepts, and techniques to diverse areas of machine learning and data science. This expository presentation follows our journey from compression, to estimation, structure exploitation, functional approximation, classification, and (time permitting) robust and reinforcement learning, with a few life lessons learned along the way.

Dr. Alon Orlitsky received B.Sc. degrees in Mathematics and Electrical Engineering from Ben Gurion University, and M.Sc. and Ph.D. degrees in Electrical Engineering from Stanford University. After a decade with the Communications Analysis Research Department at Bell Laboratories and a year at D.E. Shaw and Company, he joined the University of California San Diego, where he is currently a professor of Electrical and Computer Engineering and of Computer Science and Engineering and holds the Qualcomm Chair for Information Theory and its Applications. His research concerns information theory, statistical modeling, and machine learning, and focuses on fundamental limits and practical algorithms for extracting knowledge from data. Among other distinctions, Alon is a recipient of the 2021 Information Theory Society Claude E. Shannon Award and a co-recipient of the 2017 ICML Best Paper Honorable Mention Award, the 2015 NeurIPS Best Paper Award, the 2006 Information Theory Society Paper Award, and the 1992 IEEE W.R.G. Baker Award. 

In most communication systems, adapting transmission strategies to the (unpredictable) realization of channel output at the receiver requires an unrealistic assumption about the availability of a reliable “feedback” channel. This unfortunate fact, combined by the historical linkage between teaching information theory and digital communication curriculum has kept “feedback information theory” less taught, discussed, appreciated and understood compared to other topics in our field.

This talk, in contrast, highlights important and challenging problems in machine learning, optimization, statistics, and control theory, where the problem of acquiring information in an adaptive manner arises very naturally. Thus, I will argue that an increased emphasis on (teaching) feedback information theory can provide vast and exciting research opportunities at the intersection of information theory and these fields. I will also highlight the successful application of these sequential techniques in a variety of problem instances such as black-box optimization, distribution estimation, and federated learning and control.

Dr. Tara Javidi received her BS in electrical engineering at Sharif University of Technology, Tehran, Iran. She received her MS degrees in electrical engineering (systems) and in applied mathematics (stochastic analysis) from the University of Michigan, Ann Arbor as well as her Ph.D. in electrical engineering and computer science in 2002. She is currently a professor at the University of California, San Diego, where she has appointments in Electrical and Computer Engineering Department as well as Halicioglu Data Science Institute. She is also a founding co-director of the UCSD Center for Machine-Intelligence, Computing and Security. 

Dr. Tara Javidi is a Fellow of IEEE. She and her Phd students are recipients of the 2021 IEEE Communications Society & Information Theory Society Joint Paper Award. She also received the 2018 and 2019 Qualcomm Faculty Award for her contributions to wireless technology. Tara Javidi was a recipient of the 2018 and 2019 Qualcomm Faculty Awards, 2004 National Science Foundation early career award (CAREER), 1999 Barbour Graduate Scholarship, University of Michigan, and the Presidential and Ministerial Recognitions for Excellence in the National Entrance Exam, Iran, in 1992. At UCSD, she has also received awards for her exceptional University service/leadership and contributions to diversity.

Point cloud registration is one of the enabling technologies for 3D image reconstruction. The success of point cloud registration lies in how to identify the accurate matching pairs of points belonging to two overlapping point clouds.  In this work, we propose a novel deep-learning architecture, called ARM-Net, aiming at providing accurate and robust point matching to facilitate the point cloud registration. ARM-Net adopts two dramatically different ideas of network architectures. On one hand, ARM-Net utilizes the dynamic graph convolution to extract the features from the evolving topologies of points in which the neighboring information of a point keeps changing. On the other hand, ARM-Net adopts the multi-layer perceptron (MLP) architecture to obtain the ``global'' encoding of each local topology around each point in which the neighboring information of that point is kept the same. Combining these two architectures, ARM-Net is able to provide more differentiating features to make the matching more accurate and robust. Once features are extracted, the back-to-back transformers ensue to seek the information of self-attention and cross-attention of two point clouds to be registered, which is followed by the matching subnetwork to find the right corresponding points of these two point clouds.

Furthermore, to obtain more local information and improve the distinctness of extracted features, we propose a simple local aggregation module based on K-Nearest Neighbor (KNN) algorithm before the feature extraction. Assuming K is sufficiently large, the local aggregation can provide noise-independent local information and improve the registration performance. The simulation results show the overwhelming performance of ARM-Net in terms of the MSE, RMSE, and MAE of rotation and translation. For example, the Root MSEs (RMSEs) of the rotation error of ARM-Net are 0.000639 and 2.208962 in the full registration and partial registration, respectively while its RMSEs of translation are  and 0.025574 in the full registration and partial registration, respectively. 

Dr. Min-Kuan Chang (SM’23) received the B.S. degree from the National Tsing Hua University, Hsinchu City, Taiwan, in 1996, and the M.S. and PhD. degrees from the University of Southern California, Los Angeles, CA, in 1998 and 2003, respectively, all in electrical engineering. From 2000 to 2003, Dr. Chang was a research assistant in the Integrated Media Systems Center at the University of Southern California, where he worked on QoS provision for wireless multimedia transmission. In 2004, he joined the Department of Electrical Engineering (EE), National Chung Hsing University (NCHU), Taichung, Taiwan. He was the director of Graduate Institute of Communication Engineering (GICE) from 2018 to 2021 and is currently a professor in both Dept. of EE and GICE, NCHU. His current research interests focus on resource allocation and transmission design in wireless federated-learning communication systems, performance analysis and system design in relaying systems, signal processing in point clouds.

Dr. Chang served the local organizing committee of conferences, including the International Symposium on Information Theory and Its Applications, International Symposium on Spread Spectrum Techniques and Applications, IEEE Conference on Industrial Electronics and Applications, all in 2010, and the IEEE International Conference on Control & Automation in 2014. Dr. Chang also served as the PC member of IEEE International Conference on Multimedia Big Data in 2019. From 2019 to 2020, Dr. Chang becomes the officer of student activities of IEEE Taipei Section. From 2021 to 2022, Dr. Chang is the vice chair of IEEE ComSoc Taipei Chapter. Since 2023, Dr. Chang is the chair of IEEE ComSoc Taipei Chapter.

In this talk, we will talk about recent trends and developments on Image Restorations. First, some background knowledge for image restoration will be introduced. Then we will only focus on two categories of image restorations: Image Denoising and Image Deraining. For denoising, we proposed a blind denoising network (SRMNet) by employing a hierarchical architecture improved from U-Net, which is able to deal with both synthetic noise (e.g., Additive White Gaussian Noise (AWGN)) and real-world noise. Specifically, we use a selective kernel with residual block on the hierarchical structure called M-Net to enrich the multi-scale semantic information. Furthermore, our SRMNet has competitive performance results on two synthetic and two real-world noisy datasets in terms of quantitative metrics and visual quality. And the issue of computational complexity will also be discussed. Lastly, we proposed a restoration model CMFNet inspired by the Retinal Ganglion Cells (RGCs) which can handle the image restoration task of deraindrop. For all the proposed models, we will provide a demo for the image restoration results.

Dr. Tsung-Jung Liu received the B.S. degree in Electrical Engineering from National Tsing Hua University, Hsinchu, Taiwan, in 1998, and the M.S. degree in Communication Engineering from National Taiwan University, Taipei, Taiwan, in 2001, and the Ph.D. degree in Electrical Engineering from University of Southern California, Los Angeles, CA, USA, in 2014. He is currently a Distinguished Associate Professor with Department of Electrical Engineering, and Graduate Institute of Communication Engineering, National Chung Hsing University, Taichung, Taiwan. His research interests include computer vision, perceptual image/video processing, visual quality assessment, machine learning/deep learning, and artificial intelligence. He is a member of IEEE, ACM, SPIE, IET, and AAAS. Also, he has published over 80 referred international journal and conference papers. Recently, he received the Best Paper Award at the IEEE 2023 International Conference on Consumer Electronics – Taiwan, for the paper “Image Inpainting Using MSCSWin Transformer and Color Correction”.