HAC-M-DNN: Hardware Aware Compression of Multimodal Deep Neural Networks for Efficient Real-time Edge Deployment (PhD Thesis)
With the rise of Artificial Intelligence (AI), there has been a renaissance in interest in how to apply AI algorithms on low-power embedded systems to expand possible Internet of Things (IoT) use-cases. Multimodal deep neural networks (M-DNN) have recently gained a lot of attention with the classification challenge due to their outstanding performance for computer vision and audio processing tasks. Their goal is to replicate multimodal human perception. This research focuses on a small, low-power software and hardware architecture for multimodal deep neural networks in edge devices with limited resources. Cyclic sparsification and hybrid quantization (4-bit weights and 8-bit activations) approaches are used to compress the models for implementation on small devices. We are the first to show the effectiveness of model compression strategies for multimodal deep neural networks employing cyclically sparsification and hybrid quantization of weights/activations, despite the fact that this is an active study topic. We provide three different case-studies running on various resource-constrained edge devices validating their energy-efficiency upon deployment.
Publications:
“HAC-POCD: Hardware-Aware Compressed Activity Monitoring and Fall Detector Edge POC Devices”. IEEE Biomedical Circuits and Systems Conference (BioCAS), 2023.
“TinyM^2Net-V2: A Compact Low Power Software Hardware Architecture for Multimodal Deep Neural Networks”. ACM Transactions on Embedded Computing Systems, 2023.
“TinyM^2Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices”. TinyML Research Symposium, 2022.
COVID-Matter: A Scalable Multimodal Sensory Machine Learning Framework for Severity Detection of Respiratory Diseases and Pandemic Prevention
The objective of this ongoing research is to utilize deep learning models running on general computing processors to replace what doctors do at triage and telemedicine with the help of passively recorded audio/video and self-declared information. We proposed reconfigurable FPGA hardware for cough and dyspnea detection as well as respiratory sounds classification.
Publications:
Conference: "TinyM^2Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices", tinyML Research Symposium 2022.
Conference: "CoughNet-V2: A Scalable Multimodal DNN Framework for Point-of-Care Edge Devices to Detect Symptomatic COVID-19 Cough", accepted at 7th Annual IEEE EMB Strategic Conference focusing on point-of-care technologies, HI-POCT 2022.
Book Chapter: "A Re-configurable Software-Hardware CNN Framework for Automatic Detection of Respiratory Symptoms", Healthcare Technology Solutions for Pandemics – A Roadmap, Springer Nature.
Journal: "Automatic Detection of Respiratory Symptoms Using a Low Power Multimodal CNN Processor”, IEEE Transactions of Design and Test.
Conference: "CoughNet: A Flexible Low Power CNN-LSTM Processor for Cough Sound Detection", IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS), 2021.
Conference: “Neural Networks for Pulmonary Disease Diagnosis using Auditory and Demographic Information”, epiDAMIK 2020: 3rd epiDAMIK ACM SIGKDD International Workshop on Epidemiology meets Data Mining and Knowledge Discovery,August 2020, USA.
Conference: “End-to-end Scalable and Low Power Multi-modal CNN for Respiratory-related Symptoms Detection ”, 2020 IEEE33rd International System-on-Chip Conference (SOCC 2020), September 2020, USA.
An Energy Efficient and Flexible Multichannel Electroencephalogram (EEG) Artifact Identification
The project aims at an energy efficient and flexible multichannel Electroencephalogram (EEG) artifact detection and identification networks and their reconfigurable hardware implementations. Experiments were done with different deep learning models (i.e. CNN, Depthwise Separable CNN, LSTM, Conv-LSTM) with the goal of maximizing the detection/identification accuracy while minimizing the weight parameters and required number of operations. Low bit-width quantized FPGA and ASIC hardware architectures were also proposed.
Publications:
Book Chapter: "A Flexible Software-Hardware Framework for Brain EEG Multiple Artifact Identification”, Springer Handbook of Biochips.
Journal: "A Flexible Multichannel EEG Artifact Identification Processor using Depthwise-Separable Convolutional Neural Networks ", ACM Journal on Emerging Technologies in Computing Systems (JETC), September 2020.
Conference: "CNN LSTM Combined Network for Artifact Identification in Multi-channel EEG data ", 11th International Conference on Applied Human Factors and Ergonomics (AHFE 2020), July 2020, San Diego, CA, USA.
Conference: "A Low-Power LSTM Processor for Multi-Channel Brain EEG Artifact Detection ", In the proceedings of the 21th International Symposium on Quality Electronic Design (ISQED), March 2020, Santa Clara, CA, USA.
Low Power Deep Neural Network Architectures for Physical Activity Monitoring
CNN and LSTM models were deployed into low bitwidth quantized FPGA and ASIC hardware architectures for monitoring muliple physical activities which enables this project to be implemented on low powered wearable biomedical devices.
Publications:
Book Chapter: "SensorNet: A Scalable and Low Power Deep Convolutional Neural Network for Multimodal Data Classification in Embedded Real-Time Systems ", Springer Textbook on Machine Learning
Conference: “An Energy-Efficient Low Power LSTM Processor for Human Activity Monitoring ", IEEE 33rd International System-on-Chip Conference (SOCC 2020), September 2020, USA
Performance Analysis of Non-Recursive Convolutional Code for Multi-Carrier System
This was my undergraduate final year research project supervised by Dr. Muhammad Ahsan Ullah at Chittagong University of Engineering and Technology (CUET).
Publications:
Conference: “Performance of Multi-Stage Threshold Decoding in MC-CDMA” The 2nd IEEE International Conference on Telecommunications and Photonics (ICTP), 26-28 December, 2017, BUET, Bangladesh.
Conference: "Soft Decision Multi-Stage Threshold Decoding with Sum-Product Algorithm," The Eight Int. Conf. on Computing, Communication and Networking Technologies (ICCCNT), pp. 819-823, July 3-5, 2017, IIT, Delhi, India.