In this chapter you will finally start looking after the networks you have build so far. When we build a network for any task, what we worry about most is the accuracy. However when we want to run these models for a realtime task we also need to think about resources. We are now talking about computation time (inference time), number of operations (FLOPS), memory requirement (depends on the number of parameters in the model). In order to reduce complexity we want a model with fewer parameters, fewer operations etc. We talk about 3 aspects prunning, quantization and checkpointing.
Prunning:- Remove some branches from the network. Choose the ones which have low weights and make them 0. This will reduce the computation and make the network faster. You have different types of prunning which guides what do we want to remove.
Quantization:- Your weights are generally FP-32, quantization changes them to FP-16, FP-8 or INT-8. So we can understand multiplying with 32bits will take more time as well as require more space in the memory
Checkpointing:- This is a way in which you balance recomputing the forward pass and saving them
Practice 9: You will find a streamlit based chapter at https://github.com/jhilikb/UCS668
Prerequisite: Make sure that you have the Pruning folder in your course folder UCS668.
Start your python environment (say ucs668) that you set up (in Lab0) for the course using source ucs668/bin/activate (on your terminal if you use Linux/Mac) or using .\ucs668\Scripts\Activate.ps1 (on your powershell if using windows). In both cases you should be in the folder where you created the environment
cd Pruning
streamlit run prune.py
You will see the chapter start on your browser. In your browser, you can now explore the example tasks and understand how to solve them.
Lab:
Start with Resnet-101, Pick 500 images and run classification:-
a. Use the original network
b. Use FP-16
c. Use INT8
In each case record time and accuracy. Note that for time, you need to do 50 warmup runs before recording it.
Take your original Resnet 101 again and apply prunning
a. Use random, structured,unstructured and global with 0.10,0.25,0.50,0.80.
Record all the results
Now write your own classification network (may be the one you used for flower classification). Retrain it again for the flower classification task but with FP16. Note this is not post training quantization. Its Quantization aware training.