Splunk SH cluster + HA Proxy

You dont need to setup external reverse proxies (like Apache), as HAProxy will route using a custom SSL cert

HAProxy Setup

on HAProxy server, create a custom SSL cert using this script,

root@haproxy01 /etc/haproxy]# cat /etc/haproxy/ssl/gencert.sh 

#!/usr/bin/env bash

# Generate Openssl crt

######################################

#  GENERATE SSL SELF-SIGNED          #

######################################

dir_ssl="/etc/haproxy/ssl"

if [[ ! -e $dir_ssl ]]; then

    mkdir $dir_ssl

elif [[ ! -d $dir_ssl ]]; then

    echo "$dir_ssl already exists" 1>&2

fi

cd $dir_ssl

#Change to your company details

country=US

state=NY

locality="New York"

organization="MYCompany, L.P."

organizationalunit="MYCompany Certificate Authority"

email=admin@company.com

#####GENERATE server_key

openssl genrsa -out server.key 2048

echo "Creating domain Key "

echo

echo "---------------------------"

echo "-----Below is your Key-----"

echo "---------------------------"

echo

cat server.key

openssl req -new -key server.key -out $dir_ssl/server.csr \

    -subj "/C=$country/ST=$state/L=$locality/O=$organization/OU=$organizationalunit/CN=$commonname/emailAddress=$email"

echo "---------------------------"

echo "-----Below is your CSR-----"

echo "---------------------------"

echo

cat server.csr

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

echo

echo "---------------------------"

echo "-----Below is your crt-----"

echo "---------------------------"

echo

cat server.key

#################################

# COCANTENATE crt and key to pem#

#################################

mkdir -p $dir_ssl/waf/

cat $dir_ssl/server.crt $dir_ssl/server.key > $dir_ssl/waf/splunk.pem

echo

echo "---------------------------"

echo "-----Below is your splunk.pem-----"

echo "---------------------------"

echo

cat  $dir_ssl/waf/splunk.pem

a new cert (PEM) will be placed in /etc/haproxy/ssl/waf/splunk.pem

add your HAProxy config and reference the new cert, add each SH node

global

  daemon  

  debug  true

  group  haproxy

  log  10.185.20.177 local0

  pidfile  /var/run/haproxy.pid

  stats  socket /var/lib/haproxy/stats

  user  haproxy

defaults

  log  global

  maxconn  8000

  mode  tcp

  timeout  connect 5s

  timeout  client 50s

  timeout  server 50s

listen splunk_listen

  bind :443 ssl crt /etc/haproxy/ssl/waf/splunk.pem

  bind :80 

  mode tcp

  balance source

  server splunksh01.company.local 10.125.25.173:8000 ssl verify none check port 8000

  server splunksh02.company.local 10.125.25.174:8000 ssl verify none check port 8000

  server splunksh03.company.local 10.125.25.176:8000 ssl verify none check port 8000

restart HAProxy, add "splunk.company.com" as DNS alias pointing to HAProxy hostname

go to http://splunk.company.com and HAProxy will route to any available Splunk instance, and keep the connection persistent per that instance (unless the instance goes down)

Pure TCP w/o SSL Termination

To setup a TCP-only connection without SSL termination (passing SSL check down to each SH node via Apache reverse proxy)

global

  daemon  

  debug  true

  group  haproxy

  log  10.185.20.219 local0

  pidfile  /var/run/haproxy.pid

  stats  socket /var/lib/haproxy/stats

  user  haproxy

defaults

  log  global

  maxconn  8000

  mode  tcp

  timeout  connect 5s

  timeout  client 50s

  timeout  server 50s

listen splunk_listen

  bind *:443 

  mode tcp

  balance source

  server splunksh01.company.local 10.125.25.173:443 check port 8300

  server splunksh01.company.local 10.125.25.174:443 check port 8300

  server splunksh01.company.local 10.125.25.176:443 check port 8300

and on each SH node, install Apache and modify Apache proxy, each search head proxies via SSL cert

<VirtualHost *:80>

  ServerName splunk-80

  ServerSignature Off

  CustomLog "/var/log/httpd/splunk-80_access.log" combined 

  ## Redirect rules

  Redirect  / https://splunksh01.company.local/

</VirtualHost>

<VirtualHost *:443>

  ServerName splunksh01.company.local

  ## Logging

  ErrorLog "/var/log/httpd/splunk-443_error_ssl.log"

  ServerSignature Off

  CustomLog "/var/log/httpd/splunk-443_access_ssl.log" combined 

  ## Request header rules

  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader

  RequestHeader set X-Forwarded-Proto "https"

  ## Proxy rules

  ProxyRequests Off

  ProxyPreserveHost Off

  ProxyPass / https://splunksh01.company.local:8300/

  ProxyPassReverse / https://splunksh01.company.local:8300/

  ## SSL directives

  SSLEngine on

  SSLCertificateFile      "/etc/puppetlabs/puppet/ssl/certs/splunksh01.company.local.pem"

  SSLCertificateKeyFile   "/etc/puppetlabs/puppet/ssl/private_keys/splunksh01.company.local.pem"

  # SSL Proxy directives

  SSLProxyEngine On

</VirtualHost>

and each Node's web.conf 

[settings]

startwebserver = 1

httpport = 8300

enableSplunkWebSSL = true