CamCNN

It is pretty common to train CNN's on existing datasets (ie CIFAR, MNIST).
However, wrt deep learning, this data is what's most important.

Wouldn't it be much more fun (and pain : ) if you 'made' the data as well!
This project was a fun experiement in my early attempts to train and peak into my own CNN's.

The motive - I want to use data I know, data I can see.
The solution - Just use live laptop webcam footage.

The motive2 - I wanna know what/how a CNN works.
The solution2 - Just visualize the internal layers!

Start with the simplest task - Learn the identity function (approx).

To visualize the layers, simply lock in 3 channels throughout the network.
We can then literally see the output of these layers as RGB images.

Below is a gif of the autoencoder training live.
1st row are layers across the encoder, (top-left is input)
2nd row contains layers in the decoder. (bottom-right is output)


It's fun to see how the information and detail flows down the layers with time.
Note - It does not converge to the solution of just making all weights 1 and biases 0.
Ofc this is far from the only solution to identity, but it would seem the most simple and straightforward one (ast least to humans)

Fun Learning/Results -  

Moving on to a harder task - object detection/classification.
Classify what? - Anything!

Simple approach -
When the object you you want to detect/classify is in frame, press 'y' to train as positive sample.
When it isn't in the frame, press 'n' to train as negative sample.
Press 'e' to just eval on the camera feed.

The above approach never learns.
The issue - While it sounds intuitive, we sadly havn't solved serial continual learning (yet : )
Our data stream are continuous, highly-correlated images, easy to overfit on. 

Tinkering with learning rates?
With high learning rates, the gradients explode and model just gives constant output.
With lower learning rates, the model doesn't learn anything (at least with the less number fo samples we're working with).
With medium learning rates, the model still overfits to the last 'mode' trained on.

We want to learn from current data, but also not catastrophically forget older learnt stuff - paradox wrt time?
Therer has to be a way of deterministically linking model capacity and lr/optimization params ?

Food for Thought - The order of data matters!
(random batches vs serial batches - same data, different outcomes)

To deal with the serial-overfitting issue:
1. Introduce memory - always train on +ve and -ve together, not serially.
2. Clip gradients - don't allow any update to massively disrupt older learning.

Implementation -
Press 'y' - train with current feed meant for 'y' + last known batch of 'n'
Press 'n' - train with current feed meant for 'n' + last known batch of 'y'
Note - Start training only after batches for both are filled, else at the start we again overfit instantly.

Visualization -
1st Row - Input feed
2nd Row -  Inner layer heatmaps (showing normalized max activation channel for best contrast )
3rd Row -  Colour according to detection score (black if -ve, white if +ve)
 

The above definitely works, and we can train sucessfull object dectector in 3-5  minutes only.
We can also verify the model detecting the object and not distractors from the heatmaps!


The interesting part- While training, we can see what the CNN is actually 'focussing' on.
If it's focussing on the 'wrong' parts, you can on-the-go invent data +ve/-ve data accordingly.

(eg - while training, I observed my tshirt looked lit-up on the layer heatmaps.
This implies it's not detecting the object, but consider my Tshirt in the decison.
I theem provide -ve samples with my Tshirt, and +ve samples without Tshirt in the frame to fix)