Embedded Files
In this section, we first show our Graph partition system for AlexNet structure with object recognition task. Then we propose a generalized cost model for all graphs running in distributed TensorFlow. At the end, we will be proposing our "AutoPartition system", which is a strategy that can partition any given graph automatically to achieve the minimal cost.
In this section, we first show our Graph partition system for AlexNet structure with object recognition task. Then we propose a generalized cost model for all graphs running in distributed TensorFlow. At the end, we will be proposing our "AutoPartition system", which is a strategy that can partition any given graph automatically to achieve the minimal cost.
1. Graph partition system for AlexNet.
1. Graph partition system for AlexNet.
2. Generalized Cost Model for distributed graph in TensorFlow
2. Generalized Cost Model for distributed graph in TensorFlow
3. "AutoPartition System"
3. "AutoPartition System"
1. Graph partition system for AlexNet
1. Graph partition system for AlexNet
*Note: FLOPS value is calculated as:
8 SP FLOPs/cycle×1.2 GHz×4 cores = 38.4 GFLOPS
**Note: FLOPS dependent on specific CPU model & microarchitecture
Ref: http://www.overclock.net/t/947312/how-many-gflops-does-your-processor-have
https://www.pugetsystems.com/labs/hpc/Linpack-performance-Haswell-E-Core-i7-5960X-and-5930K-594/
In this graph partition system, we are using a cluster of heterogeneous devices, including Raspberry Pi 3, Linux Desktop (with CPU and GPU), and iMac. All these devices are connected to the lab network: Desktops are all connected using LAN, and Raspberry Pis connect via WiFi.
In this graph partition system, we are using a cluster of heterogeneous devices, including Raspberry Pi 3, Linux Desktop (with CPU and GPU), and iMac. All these devices are connected to the lab network: Desktops are all connected using LAN, and Raspberry Pis connect via WiFi.
The two tables above show the computation ability of different devices and the communication capabilities of different pair of devices. The ARM processor on Pi only has 38.4 GFLOPS, i7 CPU can achieve about 100-200 GFLOPS (i7-2600K has about 70 GFLOPS, while i7 5930K has 289 GFLOPS). The FLOPS value is calculated as the number of single precision floating point operations per second. For the network condition, the LAN has an average speed of 10MB per second, while the WiFi only has 1/10 of this speed.
The two tables above show the computation ability of different devices and the communication capabilities of different pair of devices. The ARM processor on Pi only has 38.4 GFLOPS, i7 CPU can achieve about 100-200 GFLOPS (i7-2600K has about 70 GFLOPS, while i7 5930K has 289 GFLOPS). The FLOPS value is calculated as the number of single precision floating point operations per second. For the network condition, the LAN has an average speed of 10MB per second, while the WiFi only has 1/10 of this speed.
In this graph partition system, we use the AlexNet as our input graph. Since we have analyzed the computation / parameter / throughput for each layer of the AlexNet in the previous "Models" section, it 's not hard for us to give a reasonable graph partition and also the device assignment manually.
In this graph partition system, we use the AlexNet as our input graph. Since we have analyzed the computation / parameter / throughput for each layer of the AlexNet in the previous "Models" section, it 's not hard for us to give a reasonable graph partition and also the device assignment manually.
The input of this system consist of two part:
The input of this system consist of two part:
— The image file taken from camera;
— The image file taken from camera;
— The Graph partition & Device assignment.
— The Graph partition & Device assignment.
E.g. as shown in the left figure, user can partition this AlexNet graph into 4 parts, and assign these part to different devices: Dev_1, Dev_2, Dev_3 and Dev_4.
E.g. as shown in the left figure, user can partition this AlexNet graph into 4 parts, and assign these part to different devices: Dev_1, Dev_2, Dev_3 and Dev_4.
Then the system are the corresponding result:
Then the system are the corresponding result:
— Image recognition result after running inference on AlexNet；
— Image recognition result after running inference on AlexNet；
— Running time of each part of the graph on different devices.
— Running time of each part of the graph on different devices.
Challenges:
Challenges:
1. This network is still too large to be supported by Raspberry device.
1. This network is still too large to be supported by Raspberry device.
The parameters of AlexNet is 233 MB, and running the inference AlexNet requires more than 1 GB memory. While Raspberry only have about 500 MB free memory available when running Linux system. So it will cause memory error if running it directly on Pi. We solve this problem by adding 2 GB of SWAP memory for Raspberry Pi. However, these virtual memory comes from the SD card on Pi, of which the access and transition speed is pretty low. Therefore the graph computation speed of AlexNet can be greatly reduced.
2. During the running time, the system transmits both feature maps and network parameters across different devices.
2. During the running time, the system transmits both feature maps and network parameters across different devices.
In our distributed system, we run the main program on host machine, and run worker program on other devices in the cluster, in this case, the workers do not have access to network model ahead of time. So when creating and running the distributed network, both network parameters and feature map data are transmitted across different devices. And since AlexNet has a great number of parameters, the computation speed can be influenced.
(In our experiment, it takes about 70-80 s running AlexNet on a single Pi; while it take 600-900 s running the same network on our distributed cluster.)
2. Generalized Cost Model for distributed graph in TensorFlow
2. Generalized Cost Model for distributed graph in TensorFlow
In order to measure the performance of graph partition, we propose a "generalized cost model" for distributed graph.
In order to measure the performance of graph partition, we propose a "generalized cost model" for distributed graph.
For any modularized graph (now we only care about linear / sequential graph), cost is usually defined as a function of elapsed Time and consumed Energy. In our definition, the cost is simply a weighted summation of these two terms. The weight tuple (Wt, We) is defined by user. So in this case, we can have a flexible parameterized cost function.
For any modularized graph (now we only care about linear / sequential graph), cost is usually defined as a function of elapsed Time and consumed Energy. In our definition, the cost is simply a weighted summation of these two terms. The weight tuple (Wt, We) is defined by user. So in this case, we can have a flexible parameterized cost function.
With this cost model, the input is the Graph Partition & Device Assignment information. (Actually, if device assignment is given, the graph partition is also known). We use a n-dimensional vector a to represent the device assignment information. After one-hot encoding the device assignment vector, it becomes a n×k dimension matrix.
With this cost model, the input is the Graph Partition & Device Assignment information. (Actually, if device assignment is given, the graph partition is also known). We use a n-dimensional vector a to represent the device assignment information. After one-hot encoding the device assignment vector, it becomes a n×k dimension matrix.
From the device assignment information, we can get the a group of (n-1) layer transition matrices {Li}. The k×k dimensional matrix Li represents the transition information between two consecutive layers i and (i+1).
From the device assignment information, we can get the a group of (n-1) layer transition matrices {Li}. The k×k dimensional matrix Li represents the transition information between two consecutive layers i and (i+1).
So if the user provide the device assignment vector, we can get the cost value of this distributed graph.
So if the user provide the device assignment vector, we can get the cost value of this distributed graph.
3. "AutoPartition System"
3. "AutoPartition System"
Giving graph partition / device assignment manually by users can be time and labor consuming. So in this subsection, we are proposing an "Auto Graph Partition Strategy" that can help us solve this problem. Besides, during the formulation process, we also complete the generalized cost model in last subsection.
Giving graph partition / device assignment manually by users can be time and labor consuming. So in this subsection, we are proposing an "Auto Graph Partition Strategy" that can help us solve this problem. Besides, during the formulation process, we also complete the generalized cost model in last subsection.
Supposing there is a modularized graph composed of n layers. Every layer is a separate model, it only receive / send input / output to the previous / next layer. To calculate the total cost of the whole graph, we need to decompose it into n individual part. Each part is a "Layer Block", in which the data comes in from the previous layer, and doing computation in this block. So the cost of this single layer block is composed of 2 parts: The communication cost when sending data from previous layer to this layer, and the computation cost when doing computation inside this layer.
Supposing there is a modularized graph composed of n layers. Every layer is a separate model, it only receive / send input / output to the previous / next layer. To calculate the total cost of the whole graph, we need to decompose it into n individual part. Each part is a "Layer Block", in which the data comes in from the previous layer, and doing computation in this block. So the cost of this single layer block is composed of 2 parts: The communication cost when sending data from previous layer to this layer, and the computation cost when doing computation inside this layer.
With this layer block, we can easily see that total cost of the network should be the summation of cost for each individual layer block. The cost is measured as a combination of both time and energy.
With this layer block, we can easily see that total cost of the network should be the summation of cost for each individual layer block. The cost is measured as a combination of both time and energy.
For any given graph, the structural information is available: computation, input / output amount of data in every layers.
For any given graph, the structural information is available: computation, input / output amount of data in every layers.
With a given cluster of devices, the computation ability of each devices and the communication capabilities between different devices pair should be available.
With a given cluster of devices, the computation ability of each devices and the communication capabilities between different devices pair should be available.
Our objective is to find optimal an assignment vector a that can achieve the minimal cost values.
Our objective is to find optimal an assignment vector a that can achieve the minimal cost values.
we can also add some constraint to this problems, like the total energy consumption / usage time of some devices are constrained, etc.
we can also add some constraint to this problems, like the total energy consumption / usage time of some devices are constrained, etc.
So in this case, we change the original graph partitioning / device assignment problem into an optimization problem.
So in this case, we change the original graph partitioning / device assignment problem into an optimization problem.
With adequate graph information and cluster conditions, the optimization algorithm can help find the optimal assignment automatically.
With adequate graph information and cluster conditions, the optimization algorithm can help find the optimal assignment automatically.
Google Sites
Report abuse