Splunk Clusters
notes on Splunk INdex cluster + Search Head cluster,
deploy a IDX and SH cluster
follow this ansible playbook,
https://github.com/perfecto25/splunk_clusterballs/tree/master/playbooks
once the cluster is up,
SH Cluster
deploy Splunk apps to all your SH members
on Deployer, copy your app to /opt/splunk/etc/shcluster/apps/app_name
chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/app_name
run command to replicate app to members, for target specify only 1 of the SH members, it will copy to rest,
splunk apply shcluster-bundle -target https://splunksh01.vagrant.local:8089 -auth admin:mypassword
Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Bundle has been pushed successfully to all the cluster members.
To run splunk apply shcluster-bundle without triggering a restart, use this version of the command:
splunk apply shcluster-bundle -action stage && splunk apply shcluster-bundle -action send
The members will receive the bundle, but they will not restart. Splunk Web will display the message "Splunk must be restarted for changes to take effect."
IDX Cluster
distribute a new index to all indexers
To distribute a new index,
on Master, create a new indexes.conf in /opt/splunk/etc/master-apps/_cluster/local
[test]
repFactor=auto
homePath=$SPLUNK_DB/indexes/test/db/
coldPath=$SPLUNK_DB/indexes/test/colddb/
thawedPath=$SPLUNK_DB
On master, Settings > Indexer Clustering > Configure Bundle Actions, click Validate and Check Restart
if ok, click Push> Distribute Configuration Bundle
The file will be placed on the slave indexers in /opt/splunk/etc/slave-apps/_cluster/local
distribute an App to all indexers
Inspect the app for indexes.conf files. For each index defined in an app-specific indexes.conf file, set repFactor=auto, so that the index gets replicated across all peers.
[<myIndex>]
repFactor=auto
homePath=<path for hot and warm buckets>
coldPath=<path for cold buckets>
thawedPath=<path for thawed buckets>
Place the app in the $SPLUNK_HOME/etc/master-apps directory on the Master. The set of subdirectories in this location constitute the configuration bundle.
[root@splunkmas master-apps]# pwd
/opt/splunk/etc/master-apps
[root@splunkmas master-apps]# ls
_cluster postfix
Use Splunk Web or the CLI to distribute the configuration bundle to the peer nodes.
On the Master, go to Settings > Distributed Environment > Indexer clustering
Click Edit > Configuration Bundle Actions
Validate the rollout, click Push to deploy app to indexer peers. Apps will be deployed to indexer peers
Note:
do not edit files on Master, /opt/splunk/etc/master-apps/_cluster/default