Splunk Clusters

notes on Splunk INdex cluster + Search Head cluster,

deploy a IDX and SH cluster

follow this ansible playbook,

https://github.com/perfecto25/splunk_clusterballs/tree/master/playbooks

once the cluster is up,

SH Cluster

deploy Splunk apps to all your SH members

• 1. on Deployer, copy your app to /opt/splunk/etc/shcluster/apps/app_name

  2. chown -R splunk:splunk  /opt/splunk/etc/shcluster/apps/app_name

  3. run command to replicate app to members, for target specify only 1 of the SH members, it will copy to rest, 

  4. splunk apply shcluster-bundle -target https://splunksh01.vagrant.local:8089 -auth admin:mypassword

  5. Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members.  Please refer to the documentation for the details. Do you wish to continue? [y/n]: y

  6. Bundle has been pushed successfully to all the cluster members.

  7. To run splunk apply shcluster-bundle without triggering a restart, use this version of the command:

1. splunk apply shcluster-bundle -action stage && splunk apply shcluster-bundle -action send

2. The members will receive the bundle, but they will not restart. Splunk Web will display the message "Splunk must be restarted for changes to take effect."

IDX Cluster

distribute a new index to all indexers

To distribute a new index,

• 1. on Master, create a new indexes.conf in /opt/splunk/etc/master-apps/_cluster/local

  2. [test]

  3. repFactor=auto

  4. homePath=$SPLUNK_DB/indexes/test/db/

  5. coldPath=$SPLUNK_DB/indexes/test/colddb/

  6. thawedPath=$SPLUNK_DB

  7. On master, Settings > Indexer Clustering > Configure Bundle Actions, click Validate and Check Restart

  8. if ok, click Push> Distribute Configuration Bundle

  9. The file will be placed on the slave indexers in /opt/splunk/etc/slave-apps/_cluster/local

distribute an App to all indexers

Inspect the app for indexes.conf files. For each index defined in an app-specific indexes.conf file, set repFactor=auto, so that the index gets replicated across all peers.

[<myIndex>]

repFactor=auto

homePath=<path for hot and warm buckets>

coldPath=<path for cold buckets>

thawedPath=<path for thawed buckets>

Place the app in the $SPLUNK_HOME/etc/master-apps directory on the Master. The set of subdirectories in this location constitute the configuration bundle.

[root@splunkmas master-apps]# pwd

/opt/splunk/etc/master-apps

[root@splunkmas master-apps]# ls

_cluster postfix

Use Splunk Web or the CLI to distribute the configuration bundle to the peer nodes.

On the Master, go to Settings > Distributed Environment > Indexer clustering

Click Edit > Configuration Bundle Actions

Validate the rollout, click Push to deploy app to indexer peers. Apps will be deployed to indexer peers

Note:

do not edit files on Master, /opt/splunk/etc/master-apps/_cluster/default