notes on Splunk INdex cluster + Search Head cluster,
follow this ansible playbook,
https://github.com/perfecto25/splunk_clusterballs/tree/master/playbooks
once the cluster is up,
deploy Splunk apps to all your SH members
on Deployer, copy your app to /opt/splunk/etc/shcluster/apps/app_name
chown -R splunk:splunk /opt/splunk/etc/shcluster/apps/app_name
run command to replicate app to members, for target specify only 1 of the SH members, it will copy to rest,
splunk apply shcluster-bundle -target https://splunksh01.vagrant.local:8089 -auth admin:mypassword
Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Bundle has been pushed successfully to all the cluster members.
To run splunk apply shcluster-bundle without triggering a restart, use this version of the command:
splunk apply shcluster-bundle -action stage && splunk apply shcluster-bundle -action send
The members will receive the bundle, but they will not restart. Splunk Web will display the message "Splunk must be restarted for changes to take effect."
distribute a new index to all indexers
To distribute a new index,
on Master, create a new indexes.conf in /opt/splunk/etc/master-apps/_cluster/local
[test]
repFactor=auto
homePath=$SPLUNK_DB/indexes/test/db/
coldPath=$SPLUNK_DB/indexes/test/colddb/
thawedPath=$SPLUNK_DB
On master, Settings > Indexer Clustering > Configure Bundle Actions, click Validate and Check Restart
if ok, click Push> Distribute Configuration Bundle
The file will be placed on the slave indexers in /opt/splunk/etc/slave-apps/_cluster/local
distribute an App to all indexers
Inspect the app for indexes.conf files. For each index defined in an app-specific indexes.conf file, set repFactor=auto, so that the index gets replicated across all peers.
[<myIndex>]
repFactor=auto
homePath=<path for hot and warm buckets>
coldPath=<path for cold buckets>
thawedPath=<path for thawed buckets>
Place the app in the $SPLUNK_HOME/etc/master-apps directory on the Master. The set of subdirectories in this location constitute the configuration bundle.
[root@splunkmas master-apps]# pwd
/opt/splunk/etc/master-apps
[root@splunkmas master-apps]# ls
_cluster postfix
Use Splunk Web or the CLI to distribute the configuration bundle to the peer nodes.
On the Master, go to Settings > Distributed Environment > Indexer clustering
Click Edit > Configuration Bundle Actions
Validate the rollout, click Push to deploy app to indexer peers. Apps will be deployed to indexer peers
Note:
do not edit files on Master, /opt/splunk/etc/master-apps/_cluster/default