Splunk SH cluster + HA Proxy
Background
Need to balance user access to 3 Search Head nodes inside a cluster
Will be using HAProxy to route users to individual SH nodes, using a common SSL cert
Sys Info:
Splunk 7.1.1
HAProxy 1.5.18
Centos 7 x64
Apache 2.4.25 (for reverse proxy)
TCP with SSL Termination
This setup uses a centralized Cert to setup SSL for all search head nodes and does not use individual Apache reverse proxies on each SH node.
Splunk Setup
Install and configure Splunk SH cluster, make sure each cluster node is up, a Captain is selected and everything is working,
For Splunk Web, make sure each SH node has SSL enabled and running on port 8000
You dont need to setup external reverse proxies (like Apache), as HAProxy will route using a custom SSL cert
HAProxy Setup
on HAProxy server, create a custom SSL cert using this script,
root@haproxy01 /etc/haproxy]# cat /etc/haproxy/ssl/gencert.sh
#!/usr/bin/env bash
# Generate Openssl crt
######################################
# GENERATE SSL SELF-SIGNED #
######################################
dir_ssl="/etc/haproxy/ssl"
if [[ ! -e $dir_ssl ]]; then
mkdir $dir_ssl
elif [[ ! -d $dir_ssl ]]; then
echo "$dir_ssl already exists" 1>&2
fi
cd $dir_ssl
#Change to your company details
country=US
state=NY
locality="New York"
organization="MYCompany, L.P."
organizationalunit="MYCompany Certificate Authority"
email=admin@company.com
#####GENERATE server_key
openssl genrsa -out server.key 2048
echo "Creating domain Key "
echo
echo "---------------------------"
echo "-----Below is your Key-----"
echo "---------------------------"
echo
cat server.key
openssl req -new -key server.key -out $dir_ssl/server.csr \
-subj "/C=$country/ST=$state/L=$locality/O=$organization/OU=$organizationalunit/CN=$commonname/emailAddress=$email"
echo "---------------------------"
echo "-----Below is your CSR-----"
echo "---------------------------"
echo
cat server.csr
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
echo
echo "---------------------------"
echo "-----Below is your crt-----"
echo "---------------------------"
echo
cat server.key
#################################
# COCANTENATE crt and key to pem#
#################################
mkdir -p $dir_ssl/waf/
cat $dir_ssl/server.crt $dir_ssl/server.key > $dir_ssl/waf/splunk.pem
echo
echo "---------------------------"
echo "-----Below is your splunk.pem-----"
echo "---------------------------"
echo
cat $dir_ssl/waf/splunk.pem
a new cert (PEM) will be placed in /etc/haproxy/ssl/waf/splunk.pem
add your HAProxy config and reference the new cert, add each SH node
global
daemon
debug true
group haproxy
log 10.185.20.177 local0
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats
user haproxy
defaults
log global
maxconn 8000
mode tcp
timeout connect 5s
timeout client 50s
timeout server 50s
listen splunk_listen
bind :443 ssl crt /etc/haproxy/ssl/waf/splunk.pem
bind :80
mode tcp
balance source
server splunksh01.company.local 10.125.25.173:8000 ssl verify none check port 8000
server splunksh02.company.local 10.125.25.174:8000 ssl verify none check port 8000
server splunksh03.company.local 10.125.25.176:8000 ssl verify none check port 8000
restart HAProxy, add "splunk.company.com" as DNS alias pointing to HAProxy hostname
go to http://splunk.company.com and HAProxy will route to any available Splunk instance, and keep the connection persistent per that instance (unless the instance goes down)
Pure TCP w/o SSL Termination
To setup a TCP-only connection without SSL termination (passing SSL check down to each SH node via Apache reverse proxy)
global
daemon
debug true
group haproxy
log 10.185.20.219 local0
pidfile /var/run/haproxy.pid
stats socket /var/lib/haproxy/stats
user haproxy
defaults
log global
maxconn 8000
mode tcp
timeout connect 5s
timeout client 50s
timeout server 50s
listen splunk_listen
bind *:443
mode tcp
balance source
server splunksh01.company.local 10.125.25.173:443 check port 8300
server splunksh01.company.local 10.125.25.174:443 check port 8300
server splunksh01.company.local 10.125.25.176:443 check port 8300
and on each SH node, install Apache and modify Apache proxy, each search head proxies via SSL cert
<VirtualHost *:80>
ServerName splunk-80
ServerSignature Off
CustomLog "/var/log/httpd/splunk-80_access.log" combined
## Redirect rules
Redirect / https://splunksh01.company.local/
</VirtualHost>
<VirtualHost *:443>
ServerName splunksh01.company.local
## Logging
ErrorLog "/var/log/httpd/splunk-443_error_ssl.log"
ServerSignature Off
CustomLog "/var/log/httpd/splunk-443_access_ssl.log" combined
## Request header rules
## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
RequestHeader set X-Forwarded-Proto "https"
## Proxy rules
ProxyRequests Off
ProxyPreserveHost Off
ProxyPass / https://splunksh01.company.local:8300/
ProxyPassReverse / https://splunksh01.company.local:8300/
## SSL directives
SSLEngine on
SSLCertificateFile "/etc/puppetlabs/puppet/ssl/certs/splunksh01.company.local.pem"
SSLCertificateKeyFile "/etc/puppetlabs/puppet/ssl/private_keys/splunksh01.company.local.pem"
# SSL Proxy directives
SSLProxyEngine On
</VirtualHost>
and each Node's web.conf
[settings]
startwebserver = 1
httpport = 8300
enableSplunkWebSSL = true