Configuring Splunk with Kerberos SSO via Apache reverse proxy
Kerb Your Enthusiasm
Guide to setting up SSL-secured Splunk that authenticates users via Kerberos Single Sign On “SSO” (using AD)
open a "Private" browser tab and got to the Splunk instance, Kerberos should authenticate and Apache should redirect to HTTPS
SSO is pretty pretty pretty good.
Background:
OS used: Centos 7, Splunk 6.5.1
Prerequisites:
Installed Splunk instance
Installed Apache (v2.4 and up)
have a functional Active Directory
AD admin or rights to create new Principals
configure Splunk with your AD (test logging into Splunk using your AD creds)
additional help: http://www.grolmsnet.de/kerbtut/
Kerberos Setup
generate keytab file “httpd.keytab”
see this document for instructions: Kerberos SSO configuration
on Splunk server, copy krb5.conf to /etc/
on Splunk server, install krb5-libs, krb5-workstation, mod_auth_kerb
copy httpd.keytab file to Splunk server, place in /etc/httpd/
change permissions on keytab file,
chmod 600 /etc/httpd/httpd.keytab
chown daemon:daemon /etc/httpd/httpd.keytab
Test and verify keytab auth,
klist -kte httpd.keytab # list all principals
Keytab name: FILE:httpd.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
3 12/05/2017 13:24:46 splunk.httpd@COBRA.LOCAL (arcfour-hmac)
3 12/05/2017 13:24:46 splunk.httpd@COBRA.LOCAL (aes128-cts-hmac-sha1-96)
3 12/05/2017 13:24:46 splunk.httpd@COBRA.LOCAL (aes256-cts-hmac-sha1-96)
3 12/05/2017 13:24:46 HTTP/splunk.cobra.local@COBRA.LOCAL (arcfour-hmac)
3 12/05/2017 13:24:46 HTTP/splunk.cobra.local@COBRA.LOCAL (aes128-cts-hmac-sha1-96)
3 12/05/2017 13:24:46 HTTP/splunk.cobra.local@COBRA.LOCAL (aes256-cts-hmac-sha1-96)
kinit -Vkt httpd.keytab splunk.httpd
Using default cache: /tmp/krb5cc_0
Using principal: splunk.httpd@COBRA.LOCAL
Using keytab: httpd.keytab
Authenticated to Kerberos v5
Configure Apache
for basic HTTP configuration
vim /etc/httpd/conf.d/splunk.conf
LoadModule auth_kerb_module /usr/lib64/httpd/modules/mod_auth_kerb.so
<VirtualHost *:80>
LogLevel Debug
ErrorLog /var/log/httpd/kerb_error
CustomLog /var/log/httpd/splunk_access common
<Location />
AuthType Kerberos
AuthName "Kerberos Splunk"
KrbAuthRealms COBRA.LOCAL
KrbMethodK5Passwd off
KrbServiceName HTTP/splunksh01.cobra.local
Krb5KeyTab /etc/httpd/httpd.keytab
KrbLocalUserMapping on
KrbSaveCredentials On
Require valid-user
RewriteCond %{LA-U:REMOTE_USER} (.+)
RewriteRule . - [E=RU:%1]
RequestHeader set REMOTE_USER %{RU}e
</Location>
ProxyVia On
ProxyPassInterpolateEnv On
ProxyRequests Off
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
ProxyPass / http://127.0.0.1:8000/
ProxyPassReverse / http://127.0.0.1:8000/
RewriteEngine On
RewriteCond %{LA-U:REMOTE_USER} (.+)
RewriteRule . - [E=RU:%1]
RequestHeader set REMOTE_USER %{RU}e
</VirtualHost>
if you want to enable HTTPS using local certs (not generic Splunk certs),
in /opt/splunk/etc/system/local/web.conf
enableSplunkWebSSL = 0
in the Apache config, add the Redirect and local cert paths,
LoadModule auth_kerb_module /usr/lib64/httpd/modules/mod_auth_kerb.so
<VirtualHost *:8000 *:80>
Redirect / https://splunksh01.cobra.local/
</VirtualHost>
<VirtualHost *:443>
LogLevel Debug
ErrorLog /var/log/httpd/kerb_error
CustomLog /var/log/httpd/splunk_access common
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
<Location />
AuthType Kerberos
AuthName "Kerberos Splunk"
KrbAuthRealms COBRA.LOCAL
KrbMethodK5Passwd off
KrbServiceName HTTP/splunksh01.cobra.local
Krb5KeyTab /etc/httpd/httpd.keytab
KrbLocalUserMapping on
KrbSaveCredentials On
Require valid-user
RewriteCond %{LA-U:REMOTE_USER} (.+)
RewriteRule . - [E=RU:%1]
RequestHeader set REMOTE_USER %{RU}e
</Location>
ProxyVia On
ProxyPassInterpolateEnv On
ProxyRequests Off
<Proxy *>
Order allow,deny
Allow from all
</Proxy>
SSLCertificateKeyFile "/etc/pki/tls/private/splunksh01.cobra.local.pem"
SSLCertificateFile "/etc/pki/tls/certs/splunksh01.cobra.local.pem"
ProxyPass / http://splunksh01.cobra.local:8000/
ProxyPassReverse / http://splunksh01.cobra.local:8000/
RewriteEngine On
RewriteCond %{LA-U:REMOTE_USER} (.+)
RewriteRule . - [E=RU:%1]
RequestHeader set REMOTE_USER %{RU}e
</VirtualHost>
Configure Splunk
edit /opt/splunk/etc/system/local/web.conf
add,
[settings]
enableSplunkWebSSL = 0
privKeyPath = /etc/pki/tls/private/splunksh01.cobra.local.pem
caCertPath = /etc/pki/tls/certs/splunksh01.cobra.local.chain.pem
sslVersions = tls1.2
cipherSuite = ECDHE:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK
ecdhCurves = secp521r1, secp384r1, prime256v1
enableWebDebug = true
# SSO Kerberos settings
trustedIP = 127.0.0.1,10.185.20.100 (actual IP of splunk server)
SSOMode = permissive
remoteUser = REMOTE_USER
# 525600 mins = 365 days
ui_inactivity_timeout = 0
tools.sessions.timeout = 525600
edit /opt/splunk/etc/system/local/server.conf
add under [general] block,
trustedIP=127.0.0.1
stop Splunk, stop Apache
start Apache, start Splunk
A successful request and auth should look like this,
Troubleshooting
Basic Kerb authentication
Test if Kerberos Ticket-generation works on Apache machine, use your AD credentials
kinit homer.simpson
if auth is ok, there will be no reply or outoput
check your ticket
klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: homer.simpson@CORP.LOCAL
Valid starting Expires Service principal
12/08/17 14:42:50 12/09/17 00:42:50 krbtgt/CORP.LOCAL@CORP.LOCAL
Test login using KVNO
kvno HTTP/homer.nyc.local@CORP.LOCAL
HTTP/homer.nyc.local@CORP.LOCAL: kvno = 3
make sure the KVNO # matches the KVNO # from 'klist' command
make sure the principal name from the KVNO command, matches the name in the keytab
make sure the KVNO returns the same encryption type as keytab (klist -e)
if cant login using, kvno HTTP/your.server.com@YOURDOMAIN.COM
if getting "Server not found in Kerberos database while getting credentials",
if getting "Warning: received token seems to be NTLM, which isn't supported by the Kerberos module. Check your IE configuration."
Successful example
[Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1628): [client 10.120.22.74] kerb_authenticate_user entered with user (NULL) and auth_type Kerberos[Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1240): [client 10.120.22.74] Acquiring creds for HTTP/webserver.yourdomain.com [Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1385): [client 10.120.22.74] Verifying client data using KRB5 GSS-API [Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1401): [client 10.120.22.74] Client didn't delegate us their credential [Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1420): [client 10.120.22.74] GSS-API token of length 163 bytes will be sent back
Isolate Error Factors
determine if error is coming from Kerberos-auth side, or Apache Proxy / Splunk-side,
set your Apache splunk.conf to be a simple Location proxy (leave out Splunk stuff for now), using Kerberos, and set logs to Debug, this way you can determine if basic Kerberos auth works or not,
LoadModule auth_kerb_module /usr/lib64/httpd/modules/mod_auth_kerb.so
<VirtualHost *:80>
ServerName mrx01.corp.local
ServerAlias mrx01.corp.local
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>
<VirtualHost *:443>
ServerName mrx01.corp.local
ServerAlias mrx01.corp.local
LogLevel Debug
ErrorLog /var/log/httpd/splunk_error
CustomLog /var/log/httpd/splunk_access common
SSLEngine On
SSLCertificateKeyFile /etc/pki/tls/private/mrx01.corp.local.pem
SSLCertificateFile /etc/pki/tls/certs/mrx01.corp.local.pem
<Location />
AuthType Kerberos
AuthName "Kerberos Splunk"
KrbAuthRealms CORP.LOCAL
KrbMethodK5Passwd Off
KrbServiceName HTTP/mrx01.corp.local
Krb5KeyTab /etc/httpd/httpd.keytab
KrbLocalUserMapping on
KrbSaveCredentials On
Require valid-user
SSLRequireSSL
</Location>
</VirtualHost>
restart Apache, try to login to Splunk, http://mrx01.corp.local
Successful Kerb connection looks like this,
This eliminates any Kerb setup issues.
NTLM Error
Warning: received token seems to be NTLM, which isn't supported by the Kerberos module. Check your IE configuration.
add the site you are accessing to your local Internet options Local Sites,
for NTLM, also in the Apache conf, change KrbMethodK5Passwd On
Watch the logs, this setting will popup a user + pw prompt that may give glues to whats going wrong.
- also NTLM errors usually mean your target host is not in DNS.
SSO Debug
check the debug page once you can connect to splunk (but cant login via SSO). In this example, my REMOTE-USER value is not being passed
enable this SSO Debug page in /opt/splunk/etc/system/local/web.conf, add variable enableWebDebug = true