Embedded Files
Tensorflow express numerical computations in the form of data flow graphs. The graph consists of nodes and edges. Nodes represents the mathematical operations and edges represents the communications of multidimentional arrays also called tensors between the different nodes.
Tensorflow express numerical computations in the form of data flow graphs. The graph consists of nodes and edges. Nodes represents the mathematical operations and edges represents the communications of multidimentional arrays also called tensors between the different nodes.
Distributed Tensorflow
Distributed Tensorflow
At high, level the distributed tensorflow design is as shown below. There different devices can be divided into parameter servers and workers. The workers are the devices running the jobs, and the parameter servers are the devices which can be used to share the variables between the workers.
At high, level the distributed tensorflow design is as shown below. There different devices can be divided into parameter servers and workers. The workers are the devices running the jobs, and the parameter servers are the devices which can be used to share the variables between the workers.
Any of the worker device may also act like parameter server, storing the variables. Normally, we make high computing devices as workers and the device which has good connectivity with all the workers is made parameter server. For example, in distributed training, parameter server can be used to store the variables (weights), and different workers can run the training parallely. All the changes to the weight are propagated to the central parameter server, which maintains the most updated weight, and this most updated weight is maintained by the parameter server, which is avaialble to any of the worker.
Any of the worker device may also act like parameter server, storing the variables. Normally, we make high computing devices as workers and the device which has good connectivity with all the workers is made parameter server. For example, in distributed training, parameter server can be used to store the variables (weights), and different workers can run the training parallely. All the changes to the weight are propagated to the central parameter server, which maintains the most updated weight, and this most updated weight is maintained by the parameter server, which is avaialble to any of the worker.
For Example Parameter Task Can be used for the below operations:
For Example Parameter Task Can be used for the below operations:
- Variables
- Update operations
And the Worker can be used for:
And the Worker can be used for:
- Pre-processing
- Loss calculation
- Backpropagation
Multiple Scenario possible in distributing data flow graph
Multiple Scenario possible in distributing data flow graph
1) Replicating Model on devices: Executing same task across machines. Similar to Data parallelism: eg: Weights are kept on PS.
1) Replicating Model on devices: Executing same task across machines. Similar to Data parallelism: eg: Weights are kept on PS.
2) Distributing Model on devices: Distributing tasks across multiple machine Different nodes executed on different machines
2) Distributing Model on devices: Distributing tasks across multiple machine Different nodes executed on different machines
We are concentrating on the both type of scenario: In 2nd the Graph is divided between devices. For example consider three devices as shown in below graph, where y, y1, y2 are placed on different devices, and y is computed by using both y1 and y2. An important point to consider is y1 and y2 can be different computations matched to different devices according to their capabilities/power/energy model.
We are concentrating on the both type of scenario: In 2nd the Graph is divided between devices. For example consider three devices as shown in below graph, where y, y1, y2 are placed on different devices, and y is computed by using both y1 and y2. An important point to consider is y1 and y2 can be different computations matched to different devices according to their capabilities/power/energy model.
For Example, the in graph replication demo code is shown below:
For Example, the in graph replication demo code is shown below:
The second setting of between graph replication is shown below: In this the different nodes of graph are scheduled on different machine.
The second setting of between graph replication is shown below: In this the different nodes of graph are scheduled on different machine.
Google Sites
Report abuse