Splunk SH cluster + HA Proxy

Background

Need to balance user access to 3 Search Head nodes inside a cluster

Will be using HAProxy to route users to individual SH nodes, using a common SSL cert

Sys Info:

Splunk 7.1.1

HAProxy 1.5.18

Centos 7 x64

Apache 2.4.25 (for reverse proxy)

TCP with SSL Termination

This setup uses a centralized Cert to setup SSL for all search head nodes and does not use individual Apache reverse proxies on each SH node.

Splunk Setup

Install and configure Splunk SH cluster, make sure each cluster node is up, a Captain is selected and everything is working,

For Splunk Web, make sure each SH node has SSL enabled and running on port 8000

You dont need to setup external reverse proxies (like Apache), as HAProxy will route using a custom SSL cert

HAProxy Setup

on HAProxy server, create a custom SSL cert using this script,

root@haproxy01 /etc/haproxy]# cat /etc/haproxy/ssl/gencert.sh

#!/usr/bin/env bash

# Generate Openssl crt

######################################

# GENERATE SSL SELF-SIGNED #

######################################

dir_ssl="/etc/haproxy/ssl"

if [[ ! -e $dir_ssl ]]; then

mkdir $dir_ssl

elif [[ ! -d $dir_ssl ]]; then

echo "$dir_ssl already exists" 1>&2

cd $dir_ssl

#Change to your company details

country=US

state=NY

locality="New York"

organization="MYCompany, L.P."

organizationalunit="MYCompany Certificate Authority"

email=admin@company.com

#####GENERATE server_key

openssl genrsa -out server.key 2048

echo "Creating domain Key "

echo

echo "---------------------------"

echo "-----Below is your Key-----"

echo "---------------------------"

echo

cat server.key

openssl req -new -key server.key -out $dir_ssl/server.csr \

-subj "/C=$country/ST=$state/L=$locality/O=$organization/OU=$organizationalunit/CN=$commonname/emailAddress=$email"

echo "---------------------------"

echo "-----Below is your CSR-----"

echo "---------------------------"

echo

cat server.csr

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

echo

echo "---------------------------"

echo "-----Below is your crt-----"

echo "---------------------------"

echo

cat server.key

#################################

# COCANTENATE crt and key to pem#

#################################

mkdir -p $dir_ssl/waf/

cat $dir_ssl/server.crt $dir_ssl/server.key > $dir_ssl/waf/splunk.pem

echo

echo "---------------------------"

echo "-----Below is your splunk.pem-----"

echo "---------------------------"

echo

cat $dir_ssl/waf/splunk.pem

a new cert (PEM) will be placed in /etc/haproxy/ssl/waf/splunk.pem

add your HAProxy config and reference the new cert, add each SH node

global

daemon

debug true

group haproxy

log 10.185.20.177 local0

pidfile /var/run/haproxy.pid

stats socket /var/lib/haproxy/stats

user haproxy

defaults

log global

maxconn 8000

mode tcp

timeout connect 5s

timeout client 50s

timeout server 50s

listen splunk_listen

bind :443 ssl crt /etc/haproxy/ssl/waf/splunk.pem

bind :80

mode tcp

balance source

server splunksh01.company.local 10.125.25.173:8000 ssl verify none check port 8000

server splunksh02.company.local 10.125.25.174:8000 ssl verify none check port 8000

server splunksh03.company.local 10.125.25.176:8000 ssl verify none check port 8000

restart HAProxy, add "splunk.company.com" as DNS alias pointing to HAProxy hostname

go to http://splunk.company.com and HAProxy will route to any available Splunk instance, and keep the connection persistent per that instance (unless the instance goes down)

Pure TCP w/o SSL Termination

To setup a TCP-only connection without SSL termination (passing SSL check down to each SH node via Apache reverse proxy)

global

daemon

debug true

group haproxy

log 10.185.20.219 local0

pidfile /var/run/haproxy.pid

stats socket /var/lib/haproxy/stats

user haproxy

defaults

log global

maxconn 8000

mode tcp

timeout connect 5s

timeout client 50s

timeout server 50s

listen splunk_listen

bind *:443

mode tcp

balance source

server splunksh01.company.local 10.125.25.173:443 check port 8300

server splunksh01.company.local 10.125.25.174:443 check port 8300

server splunksh01.company.local 10.125.25.176:443 check port 8300

and on each SH node, install Apache and modify Apache proxy, each search head proxies via SSL cert

ServerName splunk-80

ServerSignature Off

CustomLog "/var/log/httpd/splunk-80_access.log" combined

## Redirect rules

Redirect / https://splunksh01.company.local/

</VirtualHost>

ServerName splunksh01.company.local

## Logging

ErrorLog "/var/log/httpd/splunk-443_error_ssl.log"

ServerSignature Off

CustomLog "/var/log/httpd/splunk-443_access_ssl.log" combined

## Request header rules

## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader

RequestHeader set X-Forwarded-Proto "https"

## Proxy rules

ProxyRequests Off

ProxyPreserveHost Off

ProxyPass / https://splunksh01.company.local:8300/

ProxyPassReverse / https://splunksh01.company.local:8300/

## SSL directives

SSLEngine on

SSLCertificateFile "/etc/puppetlabs/puppet/ssl/certs/splunksh01.company.local.pem"

SSLCertificateKeyFile "/etc/puppetlabs/puppet/ssl/private_keys/splunksh01.company.local.pem"

# SSL Proxy directives

SSLProxyEngine On

</VirtualHost>

and each Node's web.conf

[settings]

startwebserver = 1

httpport = 8300

enableSplunkWebSSL = true