Training a Classifier.

We define a classification problem as the mapping from a measurement vector x with N entries to probability distribution y across class M labels. The function f performs the mapping parameterized by the collection of weights theta.

We define the dataset D as a collection of B tuples, containing a measurement vector x, and associated class label l. We define a loss function eta across the dataset. We wish to find the parameters theta that minimize the loss.

In the second panel of the Jupyter Notebook, define these global variables. I encourage you to experiment with these values, and see how performance is affected. These values control the dimensions and complexity of the dataset.

N = 100 # The size of the measurement vector

M = 5 # The number of class labels

B = 10000 # The number of dataset examples

We then define the dataset by randomly sampling (B/M) measurement vectors offsets for M different centroids in N dimensional space. We use the Normal distribution. The mean and standard deviations have been appropriately chosen.

measurements = np.random.normal(0.0, 10.0, (M, N))

labels = np.arange(M, dtype=int)

measurements = np.tile(measurements[:, np.newaxis, :], (1, B//M, 1)).reshape((B, N))

labels = np.tile(labels[:, np.newaxis], (1, B//M)).reshape((B,))

measurements += np.random.normal(0.0, 1.0, (B, N))

There is a notation difference here. I defined the dataset D as a list of tuples for simplicity in the math notation. But, in the actual code implementation, keeping the inputs and labels separate is more efficient in practice to implement. Try the following.

plt.plot(measurements[:, 0], measurements[:, 1], "bo")

I obtained this graph, which clearly depicts the five clusters.

The model I use is an Artificial Neural Network, that takes an N dimensional vector as input, and produces an M dimensional vector as output, that is normalized to be a probability distribution. In particular, we have the following definition.

The Neural Network has V layers, where each layer projects the previous hidden state h onto the space W offset by b. Each hidden state is normalized by the nonlinearity sigma. The final layer z is passed through softmax instead of sigma.

We have a model parameterized by many numbers. In deep learning, we HOPE there is SOME combination of parameters that performs really well. We estimate these optimal parameters using variants of Gradient Descent.

I use a variant to vanilla Gradient Descent named ADAM. In this experiment, we perform gradient descent across the whole dataset, but in practice I will sample and shuffle batches from the dataset. Define the following code in the notebook.

def train(x_inputs, y_labels, W, b):

    y_pred = inference(

        x_inputs,

        zip(W, b))

    loss = tf.reduce_mean(

        tf.nn.sparse_softmax_cross_entropy_with_logits(

            logits=y_pred,

            labels=y_labels))

    return tf.argmax(y_pred, axis=-1), loss, tf.train.AdamOptimizer(

        learning_rate=0.1).minimize(

        loss, var_list=(W + b))

Here, given the dataset as input, and the weights and biases for each layer in the Neural Network, we perform inference to obtain the class probabilities, which are used to calculate the mean cross entropy loss with respect to the labels. I use the AdamOptimizer class to calculate the gradient of the loss with respect to the parameters, using a fancy update rule.

x_holder = tf.placeholder(tf.float32)

y_holder = tf.placeholder(tf.int32)

data_out = train(x_holder, y_holder, weights, biases)

In a new panel, enter this code. We are using the static computational graph version of TensorFlow, and so all computation must be predefined in terms of placeholders, which we can fill in with actual data later.

losses = []

accuracies = []

session = tf.Session()

session.run(tf.global_variables_initializer())

for i in range(10):

    pred, loss, grad = session.run(

        data_out,

        feed_dict={x_holder: measurements, y_holder: labels})

    acc = sum([1.0 if pred[i] == labels[i] else 0.0 for i in range(B)]) / B

    print(

        "Accuracy: %.2f %%" % acc,

        "Loss: %.2f" % loss)

    losses += [loss]

    accuracies += [acc]

Here, we have entered a training session, where the computational graph is executed. In each of the 10 training steps, we compute the prediction of the network for each example, and backpropagate the derivative of the loss to tune the weights and biases. If all goes well, you should see something like the following printed to your notebook.

Accuracy: 0.15 % Loss: 11533.53

Accuracy: 0.58 % Loss: 5829.85

Accuracy: 1.00 % Loss: 1.39

Accuracy: 1.00 % Loss: 9.62

Accuracy: 0.98 % Loss: 232.09

Accuracy: 0.99 % Loss: 83.18

Accuracy: 1.00 % Loss: 7.97

Accuracy: 1.00 % Loss: 0.17

Accuracy: 1.00 % Loss: 0.33

Accuracy: 1.00 % Loss: 1.60

Nice!! We were able to get really high accuracy out of such a simple model, and very high dimensional data. Try plotting the accuracies and losses we collected during training, like what I provide below.