Elastic Search - Controlling shard allocation

Post date: Jun 9, 2013 8:19:59 AM

In order to correctly spread load in the cluster, to divide the different shards that build an index so that scalability and failover are optimally built, Elastic Search needs to compute where exactly in the cluster (on which nodes to place the different shards).

Sometimes especially after full cluster restart in order to keep response times optimal it makes sense to disable shard allocation explicitly. This can be done using the

following configuration options inside elasticsearch.yml: "cluster.routing.allocation.disable_allocation" and "cluster.routing.allocation.disable_replica_allocation".

The ElasticSearch "head" plugin available after installation on local machine at default port http://localhost:9200/_plugin/head/ displays a comprehensive view of the

different shards and their allocation on nodes. In some cases their distribution is not optimal and in such cases it makes sense to define the shard allocation explicitly. In order to perform this, the following steps have to be performed:

1) Define the zone to which a specific node should be assigned to by configuring the "node.zone" parameter inside elasticsearch.yml:

node.zone = "technical_books_zone"

2) Create an index in a specific zone (it can be spread across two zones and then these are to be separated by comma):

curl -XPOST 'localhost:9200/books'

curl -XPUT 'localhost:9200/books/_settings' -d '{"index.routing.allocation.include.zone" : "technical_books_zone"}'

A node can also be excluded from a specific zone by using "index.routing.allocation.exclude.zone". Elastic Search offers also the possibility to specify inclusion and

exclusion at IP address level using the "index.routing.allocation.include._ip" and "index.routing.allocation.exclude._ip" options passed to the aforementioned "_settings" endpoint. It is possible to specify a list of IP addresses separated by comma.

Cluster wide allocation can be specified using the "index.routing.allocation.include._ip" option on the "_settings" endpoint at "_cluster" level.

Another facility offered by Elastic Search is to specify the amount of shards per node by using the "index.routing.allocation.total_shards_per_node" option passed to the "_settings" endpoint at index level. This property can be specified as mentioned above in a dynamic manner but it can be configured also statically inside elasticsearch.yml.

If the cluster state is red, then this is clearly an indication that not all primary shards could be allocated in the manner specified, so we need to exercise special attention in case we control the allocation of shards explicitely.

Manually moving shards and replicas is another highlight provided by Elastic Search in this area. The "_cluster/reroute" endpoint is available for this purpose. It is

possible to move a shard from a node to another, to cancel shard allocation or to force it, all operations can be specified at the same time by using the "commands"

JSON array passed to the "reroute" enpoint at cluster level. The code extract below shows an example:

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{

"commands": [

{

"allocate": {

"index": "books",

"shard": 0, //shard

"node": "Franz Kafka"

}

},

{

"move": {

"index": "books",

"shard": 1, //number of shards to be moved

"from_node": "Franz Kafka",

"to_node": "Steve Jobs"

}

},

{

"cancel": {

"index": "books",

"shard": 1, //shard

"node": "Albert Einstein"

}

}

]

}'

By default shard rebalancing takes place only after all shards and replicas are initialized, this value being configured by setting the "cluster.routing.allocation.allow_rebalance" option to "indices_all_active". Other possible values for this setting are "always" and "indices_primaries_active".

The following table summarizes the possibilities for fine level control of the shard rebalancing process:

Column: Setting

1) Controls the number of shards being moved between nodes concurrently.

It specifies the number of shards that can be moved at once, default value is 2.

2) Controls the number of shards initialized concurrently on a single node.

The shard recovery process is very I/O intensive, so this value needs to be setup with care.

3) Controls the number of primary shards initialized concurrently on a single node.

Column: Parameter

1) cluster.routing.allocation.cluster_concurrent_rebalance

2) cluster.routing.allocation.node_concurrent_recoveries

3) cluster.routing.allocation.node_initial_primaries_recoveries

In conclusion, the explicit shard allocation and rebalancing can help in optimally configuring an Elastic Search Cluster for load balancing, failover and availability purposes.