In this chapter you will learn about Optimizers. This is the part which takes care of the weight update in the neural network. Going over very quickly...You created a network with 2 convolution layers and a linear layer, the filter weights in the convolution and the matrix you multiply with your features in the linear layer are your network parameters/weights (w1,w2,w3). They are initially assigned randomly. When you compute the output with these weights you get your predicted output. You find loss/error (L) between your predicted output and actual output. Using this loss you compute your gradients. Finally you update the weights based on your gradients. The basic update formula is new_w1= old_w1 - learning_rate* dL/dw1. Different optimizers vary as they use a different variation of this update formula. We are going to learn some common ones. You can refer to https://github.com/jhilikb/UCS668/blob/main/optimization/derivative_chainrule_sgd.pdf to get a basic idea of the process.
Practice 6: You will find a streamlit based chapter at https://github.com/jhilikb/UCS668
Prerequisite: Make sure that you have the optimization folder in your course folder UCS668.
Start your python environment (say ucs668) that you set up (in Lab0) for the course using source ucs668/bin/activate (on your terminal if you use Linux/Mac) or using .\ucs668\Scripts\Activate.ps1 (on your powershell if using windows). In both cases you should be in the folder where you created the environment
cd optimization
streamlit run optimi.py
You will see the chapter start on your browser. In your browser, you can now explore the example tasks and understand how to solve them.
Exercise:
a. Explore the visualizer (https://github.com/jhilikb/UCS668/blob/main/optimization/sgd.html ) to get an idea about the weight update
b. You can dive in further to look for more optimizers in https://docs.pytorch.org/docs/2.14/optim.html#algorithms