Future Work

There a tonne of new features that I want to implement in this project, but lack of time is the main reason these features keep getting further away. This page is an outline of new features and how they would be added to the project. Most of these features very easily plug into the existing code, and I hope to do the plugging some day.

Mini-Batches

This is one features that's so trivial to implement, but testing for bugs, performance is time consuming:

Each Layer has two methods that are called in BackwardPass:

1. GetPGrads (computing the Posterior gradient: delta that's sent to previous layer) and
2. ChangeWeights (add the currently computed dWeights to the Weights).

Implementing Mini-Batches of size N, would simply be about calling GetPGrads for N-1 iterations, then calling both GetPGrads and ChangeWeights for the N'th iteration.

Multi-Core support

There are about two reasons why this is not implemented: one, implementing this after Mini-Batched makes most sense: just run one batch in each core using std::future<> call, thus it's waiting for above to complete. Second reason why I am shying away from multi-core is that I am implementing this whole thing in CUDA, the performance benefit per developer's hour is huge for CUDA than for multi-core. :-)

Support different WeightUpdaters

This is kind of trivial: ChangeWeights() function needs to call WeightUpdates(W,dw) and some of the cooler updaters would be

1. Momentum, this will require storing dW, and dWPrev, I had a implem of this, had to kill it due to a bug.
2. RMS propagation: found this method in coursera course and I could not understand it fully, but it's simple to implement.

Support Error regularization

I already have an ErrorFunction class, enhance it to add a regularization term.

Support SSE Vectorized instructions

We can implmenent a tonne of Dot Product as SSE instructions, but this is not implemented for the same reason ans multi-core, the benifit is much higher if I implement CUDA than these tiny things.