Embedded Files
In this section, we demonstrate two Machine Learning Networks that we 're using in our proposed distributed TensorFlow systems.
In this section, we demonstrate two Machine Learning Networks that we 're using in our proposed distributed TensorFlow systems.
1. The AlexNet deep convolutional neural network.
1. The AlexNet deep convolutional neural network.
2. The Softmax Regression network.
2. The Softmax Regression network.
1. The AlexNet deep convolutional neural network.
1. The AlexNet deep convolutional neural network.
AlexNet is a deep convolutional neural network proposed in 2012, designed specifically for ILSVRC ImageNet object recognition challenges. It represents the state of the art techniques at that time. AlexNet is also the pioneer of modern Deep Learning, it inspired a number of well-known network structures later, including the VGG network.
AlexNet is a deep convolutional neural network proposed in 2012, designed specifically for ILSVRC ImageNet object recognition challenges. It represents the state of the art techniques at that time. AlexNet is also the pioneer of modern Deep Learning, it inspired a number of well-known network structures later, including the VGG network.
The graph above shows the detailed layer structure of AlexNet:
The graph above shows the detailed layer structure of AlexNet:
1. All the blue boxes represent the computation layer in the network.
1. All the blue boxes represent the computation layer in the network.
(5 Convolution layers + 3 Fully connected layers).
(5 Convolution layers + 3 Fully connected layers).
2. The green boxes represent the normalization layers. (Local Response Normalization)
2. The green boxes represent the normalization layers. (Local Response Normalization)
3. The red boxes represent the pooling layers.
3. The red boxes represent the pooling layers.
4. The rest white bulbs are the input/output data bulbs, or the feature maps
4. The rest white bulbs are the input/output data bulbs, or the feature maps
(intermediate result of the neural network)
(intermediate result of the neural network)
We show the data shapes & sizes and parameters numbers of every layer, so it's easy to calculate the number of parameters, required computations, and input / output data amount of each layer.
We show the data shapes & sizes and parameters numbers of every layer, so it's easy to calculate the number of parameters, required computations, and input / output data amount of each layer.
In this plot, the red figures on the left shows the number of parameters for every computing layer, and the green figures on the right show the amount of computation in these layers.
In this plot, the red figures on the left shows the number of parameters for every computing layer, and the green figures on the right show the amount of computation in these layers.
It 's easy to see that, most of the parameters concentrate on the fully connected layers, especially the first fully connected layer, which contains about 60% of the total parameter size. While the majority of the computations are on the convolution layers.
It 's easy to see that, most of the parameters concentrate on the fully connected layers, especially the first fully connected layer, which contains about 60% of the total parameter size. While the majority of the computations are on the convolution layers.
Actually the unbalancing characteristics of this graph makes it suitable for our graph partition system, in which we can assign different layer to different devices intelligently according to their computation requirement and parameter dependency. And this is the also the motivation of the idea of graph partition and distribution in heterogeneous devices.
Actually the unbalancing characteristics of this graph makes it suitable for our graph partition system, in which we can assign different layer to different devices intelligently according to their computation requirement and parameter dependency. And this is the also the motivation of the idea of graph partition and distribution in heterogeneous devices.
2. Softmax regression on MNIST data.
2. Softmax regression on MNIST data.
Softmax regression is extension of Logistic regression. It is used to do 'K' classifications.
Softmax regression is extension of Logistic regression. It is used to do 'K' classifications.
The softmax regression uses the softmax function which is shown on the right. In this the 'C' is the class which we want to predict, given 'x' as the input vector. In case of the MNIST 'x' is a vecor of size 784 and the predicted 'C' can in any of the number beloning to 0 to 9. So we have 10 outputs.
The softmax regression uses the softmax function which is shown on the right. In this the 'C' is the class which we want to predict, given 'x' as the input vector. In case of the MNIST 'x' is a vecor of size 784 and the predicted 'C' can in any of the number beloning to 0 to 9. So we have 10 outputs.
'W' are the weight vectors which has to be learned. The decision boundary is given by the expression shown on the right.
'W' are the weight vectors which has to be learned. The decision boundary is given by the expression shown on the right.
Maximing the log likelihood on this model require us to minimize the cross entropy function. The minimization is done using the different variations of gradient descent which can be batch, stochastic etc. The log likelihood is given as below. In this formulation. The output is 'Y' which may belong to any of the 'C' classes. The 'Y' is using 1 of K encoding. This function is convex and has unique global optimum.
Maximing the log likelihood on this model require us to minimize the cross entropy function. The minimization is done using the different variations of gradient descent which can be batch, stochastic etc. The log likelihood is given as below. In this formulation. The output is 'Y' which may belong to any of the 'C' classes. The 'Y' is using 1 of K encoding. This function is convex and has unique global optimum.
The image in MNIST data is present in the form of 28 by 28 pixels. Due to which the size of Y is 784. A sample image input is shown below. More details about the MNIST data set are available here.
The image in MNIST data is present in the form of 28 by 28 pixels. Due to which the size of Y is 784. A sample image input is shown below. More details about the MNIST data set are available here.
Google Sites
Report abuse