Machine Learning Strategy

Machine Learning Strategy

Let's start with a motivating example. Let's say you are working on an image classification project. And after working on it for some time, you've gotten your system to have 90% accuracy, but this isn't good enough for your application. You might then have a lot of ideas as to how to improve your system.

You might then have a lot of ideas as to how to improve your system. For example, you might think well let's collect more data, more training data. Or you might say, maybe your training set isn't diverse enough yet, you should collect images of cats in more diverse poses, or maybe a more diverse set of negative examples.

Well maybe you want to train the algorithm longer with gradient descent. Or maybe you want to try a different optimization algorithm, like the Adam optimization algorithm. Or maybe trying a bigger network or a smaller network or maybe you want to try to dropout or maybe L2 regularization. Or maybe you want to change the network architecture such as changing activation functions, changing the number of hidden units and so on and so on.

When trying to improve a deep learning system, you often have a lot of ideas or things you could try. And the problem is that if you choose poorly, it is entirely possible that you end up spending six months changing in some direction only to realize after six months that that didn't do any good. For example, I've seen some teams spend literally six months collecting more data only to realize after six months that it barely improved the performance of their system. So, assuming you don't have six months to waste on your problem, won't it be nice if you had quick and effective ways to figure out which of all of these ideas and maybe even other ideas, are worth pursuing and which ones you can safely discard.

I recommend you study Andrew Ng's course on Coursera. You will learn a number of strategies, that is, ways of analyzing a machine learning problem that will point you in the direction of the most promising things to try. What I will do in this course also is share with you a number of lessons I've learned through building and shipping deep learning products.

Orthogonalization

One of the challenges with building machine learning systems is that there's so many things you could try, so many things you could change. So many hyperparameters you could tune.

One of the things I've noticed is about the most effective machine learning people is that they have developed a clear intuition about what to tune in order to try to achieve one effect. This process is called orthogonalization.

Here's a picture (above) of an old school television, with a lot of knobs that you could tune to adjust the picture in various ways. So for these old TV sets, maybe there was one knob to adjust how tall vertically your image is and another knob to adjust how wide it is. Maybe another knob to adjust how trapezoidal it is, another knob to adjust how much to move the picture left and right, another one to adjust how much the picture's rotated, and so on.

And what TV designers had spent a lot of time doing was to build the circuitry, really often analog circuitry back then, to make sure each of the knobs had a relatively interpretable function. Such as one knob to tune this, one knob to tune this, one knob to tune this, and so on.

In contrast, imagine if you had a knob that tunes 0.1 x how tall the image is, + 0.3 x how wide the image is,- 1.7 x how trapezoidal the image is, + 0.8 times the position of the image on the horizontal axis, and so on. If you tune this knob, then the height of the image, the width of the image, how trapezoidal it is, how much it shifts, it all changes all at the same time. If you have a knob like that, it'd be almost impossible to tune the TV so that the picture gets centered in the display area. So in this context, orthogonalization refers to that the TV designers had designed the knobs so that each knob kind of does only one thing. And this makes it much easier to tune the TV, so that the picture gets centered where you want it to be.

Parameter tuning in machine learning

For a supervised learning system to do well, you usually need to tune the knobs of your system to make sure that four things hold true:

1. Fit training set well on cost function: make sure that you're at least doing well on the training set.
   • knobs to tune (parameters): train a bigger network (more layers), try other optimization algorithms (like Adam)
2. Fit dev set well on cost function
   • parameters: regularization, bigger training set
3. Fit test set well on cost function
   • parameters: bigger dev set
4. Perform well in the real world
   • parameters: change the dev set or change the cost function

Early stopping: Some people don't prefer using it because this is an op that simultaneously affects how well you fit the training set, because if you stop early, you fit the training set less well. It also simultaneously is often done to improve your dev set performance. So this is one knob that is less orthogonalized, because it simultaneously affects two things. So this is not a bad technique. It is just an easier process to tune your network when you have more orthogonalized controls.