Reasoning behind iteration count in supervised learning

Introduction

Laymen explanation

We need iterations when the data is too big which happens all the time in machine learning and we can’t pass all the data to the computer at once.

So, to overcome this problem we need to divide the data into smaller sizes and give it to our computer one by one and update the weights of the neural networks at the end of every step to fit it to the data given.

Is it only reason to divide training data or there is something more interesting and scientific? This document helps in this regard.

Technical explanation

Let's take the two extremes, on one side each gradient descent step is using the entire dataset. You're computing the gradients for every sample. In this case you know exactly the best directly towards a local minimum. You don't waste time going the wrong direction. So in terms of numbers gradient descent steps, you'll get there in the fewest.

Of course computing the gradient over the entire dataset is expensive. So now we go to the other extreme. A batch size of just 1 sample. In this case the gradient of that sample may take you completely the wrong direction. But hey, the cost of computing the one gradient was quite trivial. As you take steps with regard to just one sample you "wander" around a bit, but on the average you head towards an equally reasonable local minimum as in full batch gradient descent.

Both approaches have demerits which we will understand below.

Speed tradeoffs in machine learning

This info is important for understanding further readings.

• Computational speed

  • CPU cycles used

• Speed of convergence of an algorithm

  • Time takes to train machine with good prediction accuracy

Items affecting iteration count

• The higher the batch size, the more memory space you'll need. You can't keep all training data in single batch and so, you need more iteration than one

  • The large batch tend to problem of poorer generalisation. In other words, prediction accuracy reduces (Refer this paper)

• Smaller batch results in more computational cycles and so, increases training time

  • When using a smaller batch size, calculation of the error can have more noise than when we use a larger batch size. However this noise can help the algorithm jump out of a bad local minimum and have more chance of finding either a better local minimum, or hopefully the global minimum.

Invalid choices

• • Batch size = training data size. In this case, single iteration is needed, but the training will be very bad

  • Iteration count = training sample size. In this case, too many iterations will be needed resulting in high training CPU cycles. Note that CPU parallelisation is possible in cost/error calculation in case multiple samples predictions are combined together for cost computation.

What is right choice?

Considering above observations, optimal batch size is important. Smaller batch size is better considering convergence speed and accuracy.

In general, batch size of 32 is a good starting point, and you should also try with 64, 128, and 256. Other values (lower or higher) may be fine for some data sets, but the given range is generally the best to start experimenting with.

And, in the end, make sure the minibatch fits in the CPU/GPU

Also stochastic gradient descent expects bigger batch size for better convergence. Refer here for the detail.

Can machine converge before iteration end?

Yes. To handle this, Adam like Optimiser can help. Optimisers actually auto-adjust learning rate. So, adding optimiser can help in increasing convergence speed by completing learning before max iteration.

Point to remember

EPOCH is different from Iteration

One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE.

Reference

https://towardsdatascience.com/epoch-vs-iterations-vs-batch-size-4dfb9c7ce9c9

https://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks

https://arxiv.org/abs/1609.04836

https://stats.stackexchange.com/questions/164876/what-is-the-trade-off-between-batch-size-and-number-of-iterations-to-train-a-neu

https://ai.stackexchange.com/questions/8560/how-do-i-choose-the-optimal-batch-size

Role of optimiser in machine learning

https://coursera.org/share/b42a0f289ab84aa399ed689070da1f2b

https://sites.google.com/site/jbsakabffoi12449ujkn/home/machine-intelligence/parallelism-in-machine-learning#TOC-Benefit-of-data-parallelism