I am mainly interested in low complexity, resource constrained Machine Learning and Signal Processing. In particular, my work is centered around limited precision. The main question I ask is: How much can Machine Learning algorithms tolerate quantization (at the input, weights, and internal representations)? My work is dedicated to answering this question in a fully analytical and rigorous way.
Below is a list of project I have worked on and I am currently working on.
Fixed-point Machine Learning
This research seeks to bring rigor to the design of fixed-point learning systems which is currently being done using trial and error. Specifically, we characterized the precision to accuracy trade-off of support vector machines (SVM). We came up with several bounds that analytically predict the precision requirements of fixed-point SVM for both classification and training using the Stochastic Gradient Descent (SGD). A paper about this topic appeared in ICASSP 2017 - check it out here. The extended version is posted on the arXiv.
Nowadays, I shifted my attention to fixed-point neural networks. Indeed, it is possible to leverage the theory developed in my work on fixed-point SVMs. Neural Networks can essentially be thought of as lots of dot products, non-linearities in between, and hard decisions at the end. It is hence possible to aggregate our prior characterization on the precision to accuracy trade-off of linear combiners across the width and depth of neural networks.
This project proposes a simple but highly efficient architectural idea to reduce the computational cost of Convolutional Neural Networks (CNN). The idea, pitched by my collaborator Yingyan Lin, is to decompose the computation at each convolutional layer into MSB and LSB parts. If the result of the MSB part of some output is negative, it is highly likely that the overall output is negative itself. In such a case, the residual processing is by-passed (clock-gated). My contribution in this project was an analytical validation of the technique. One paper about this project was accepted in ISCAS 2017 and can be found here.
This is the work of my previous collaborator, Dr. Sai Zhang. The idea is to bring computation to the bitlines and cross-bitlines of a sensory array using mixed-signal techniques. My contribution was setting up the algorithm and validation dataset as well as post layout verification. Check out our arXiv paper on the topic.
I greatly appreciate the SONIC center for funding. In fact:
These works were supported in part by Systems on Nanoscale Information fabriCs (SONIC), one of the six SRC STARnet Centers, sponsored by MARCO and DARPA.