More...‎ > ‎

Configuring Splunk with Kerberos SSO via Apache reverse proxy

Kerb Your Enthusiasm


Guide to setting up SSL-secured Splunk that authenticates users via Kerberos Single Sign On “SSO” (using AD)



SSO is pretty pretty pretty good.

Background:


OS used: Centos 7, Splunk 6.5.1

Prerequisites: 

  • Installed Splunk instance
  • Installed Apache (v2.4 and up)
  • have a functional Active Directory
  • AD admin or rights to create new Principals
  • configure Splunk with your AD (test logging into Splunk using your AD creds)


additional help: http://www.grolmsnet.de/kerbtut/

Kerberos Setup

  1. generate keytab file “httpd.keytab”
    see this document for instructions: Kerberos SSO configuration

  2. on Splunk server, copy krb5.conf to /etc/

  3. on Splunk server, install krb5-libs, krb5-workstation, mod_auth_kerb

  4. copy httpd.keytab file to Splunk server, place in /etc/httpd/

  5. change permissions on keytab file,

    chmod 600 /etc/httpd/httpd.keytab
    chown daemon:daemon /etc/httpd/httpd.keytab

  6. Test and verify keytab auth,
    klist -kte httpd.keytab  # list all principals

    Keytab name: FILE:httpd.keytab
    KVNO Timestamp Principal
    ---- ------------------- ------------------------------------------------------
    3 12/05/2017 13:24:46 splunk.httpd@COBRA.LOCAL (arcfour-hmac)
    3 12/05/2017 13:24:46 splunk.httpd@COBRA.LOCAL (aes128-cts-hmac-sha1-96)
    3 12/05/2017 13:24:46 splunk.httpd@COBRA.LOCAL (aes256-cts-hmac-sha1-96)
    3 12/05/2017 13:24:46 HTTP/splunk.cobra.local@COBRA.LOCAL (arcfour-hmac)
    3 12/05/2017 13:24:46 HTTP/splunk.cobra.local@COBRA.LOCAL (aes128-cts-hmac-sha1-96)
    3 12/05/2017 13:24:46 HTTP/splunk.cobra.local@COBRA.LOCAL (aes256-cts-hmac-sha1-96)

  7. kinit -Vkt httpd.keytab splunk.httpd

      Using default cache: /tmp/krb5cc_0
      Using principal: splunk.httpd@COBRA.LOCAL
      Using keytab: httpd.keytab
      Authenticated to Kerberos v5


Configure Apache 


for basic HTTP configuration

vim /etc/httpd/conf.d/splunk.conf

LoadModule auth_kerb_module /usr/lib64/httpd/modules/mod_auth_kerb.so

<VirtualHost *:80>
    LogLevel Debug
    ErrorLog /var/log/httpd/kerb_error
    CustomLog /var/log/httpd/splunk_access common

    <Location />
        AuthType Kerberos
        AuthName "Kerberos Splunk"
        KrbAuthRealms COBRA.LOCAL
        KrbMethodK5Passwd off
        KrbServiceName HTTP/splunksh01.cobra.local
        Krb5KeyTab /etc/httpd/httpd.keytab
        KrbLocalUserMapping on
        KrbSaveCredentials On
        Require valid-user

        RewriteCond %{LA-U:REMOTE_USER} (.+)
        RewriteRule . - [E=RU:%1]
        RequestHeader set REMOTE_USER %{RU}e
    </Location>

    ProxyVia On
    ProxyPassInterpolateEnv On
    ProxyRequests Off

    <Proxy *>
        Order allow,deny
        Allow from all
    </Proxy>

    ProxyPass / http://127.0.0.1:8000/
    ProxyPassReverse / http://127.0.0.1:8000/

    RewriteEngine On
    RewriteCond %{LA-U:REMOTE_USER} (.+)
    RewriteRule . - [E=RU:%1]
    RequestHeader set REMOTE_USER %{RU}e
</VirtualHost>


if you want to enable HTTPS using local certs (not generic Splunk certs), 

in /opt/splunk/etc/system/local/web.conf

enableSplunkWebSSL = 0

in the Apache config, add the Redirect and local cert paths, 

LoadModule auth_kerb_module /usr/lib64/httpd/modules/mod_auth_kerb.so

<VirtualHost *:8000 *:80>
    Redirect / https://splunksh01.cobra.local/
</VirtualHost>


<VirtualHost *:443>
    LogLevel Debug
    ErrorLog /var/log/httpd/kerb_error
    CustomLog /var/log/httpd/splunk_access common

    <Proxy *>
        Order allow,deny
        Allow from all
    </Proxy>
    <Location />
        AuthType Kerberos
        AuthName "Kerberos Splunk"
        KrbAuthRealms COBRA.LOCAL
        KrbMethodK5Passwd off
        KrbServiceName HTTP/splunksh01.cobra.local
        Krb5KeyTab /etc/httpd/httpd.keytab
        KrbLocalUserMapping on
        KrbSaveCredentials On
        Require valid-user

        RewriteCond %{LA-U:REMOTE_USER} (.+)
        RewriteRule . - [E=RU:%1]
        RequestHeader set REMOTE_USER %{RU}e

    </Location>

    ProxyVia On
    ProxyPassInterpolateEnv On
    ProxyRequests Off

    <Proxy *>
        Order allow,deny
        Allow from all
    </Proxy>

    SSLCertificateKeyFile   "/etc/pki/tls/private/splunksh01.cobra.local.pem"
    SSLCertificateFile      "/etc/pki/tls/certs/splunksh01.cobra.local.pem"

    ProxyPass / http://splunksh01.cobra.local:8000/
    ProxyPassReverse / http://splunksh01.cobra.local:8000/


    RewriteEngine On
    RewriteCond %{LA-U:REMOTE_USER} (.+)
    RewriteRule . - [E=RU:%1]
    RequestHeader set REMOTE_USER %{RU}e
</VirtualHost>



Configure Splunk

edit /opt/splunk/etc/system/local/web.conf
add,

[settings]
enableSplunkWebSSL = 0
privKeyPath = /etc/pki/tls/private/splunksh01.cobra.local.pem
caCertPath = /etc/pki/tls/certs/splunksh01.cobra.local.chain.pem
sslVersions = tls1.2
cipherSuite = ECDHE:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK
ecdhCurves = secp521r1, secp384r1, prime256v1
enableWebDebug = true

# SSO Kerberos settings
trustedIP = 127.0.0.1,10.185.20.100  (actual IP of splunk server)
SSOMode = permissive
remoteUser = REMOTE_USER

# 525600 mins = 365 days
ui_inactivity_timeout = 0
tools.sessions.timeout = 525600


edit /opt/splunk/etc/system/local/server.conf
add under [general] block,

trustedIP=127.0.0.1


stop Splunk, stop Apache
start Apache, start Splunk


A successful request and auth should look like this, 

open a "Private" browser tab and got to the Splunk instance, Kerberos should authenticate and Apache should redirect to HTTPS





Troubleshooting


Basic Kerb authentication


Test if Kerberos Ticket-generation works on Apache machine, use your AD credentials 
kinit homer.simpson

if auth is ok, there will be no reply or outoput

check your ticket
klist

Ticket cache: FILE:/tmp/krb5cc_0
Default principal: homer.simpson@CORP.LOCAL

Valid starting     Expires            Service principal
12/08/17 14:42:50  12/09/17 00:42:50  krbtgt/CORP.LOCAL@CORP.LOCAL


Test login using KVNO
kvno HTTP/homer.nyc.local@CORP.LOCAL
HTTP/homer.nyc.local@CORP.LOCAL: kvno = 3
  • make sure the KVNO # matches the KVNO # from 'klist' command
  • make sure the principal name from the KVNO command, matches the name in the keytab
  • make sure the KVNO returns the same encryption type as keytab (klist -e)


    if cant login using, kvno HTTP/your.server.com@YOURDOMAIN.COM

    if getting "Server not found in Kerberos database while getting credentials", 

    if getting "Warning: received token seems to be NTLM, which isn't supported by the Kerberos module. Check your IE configuration."

Successful example

[Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1628): [client 10.120.22.74] kerb_authenticate_user entered with user (NULL) and auth_type Kerberos
[Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1240): [client 10.120.22.74] Acquiring creds for HTTP/webserver.yourdomain.com
[Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1385): [client 10.120.22.74] Verifying client data using KRB5 GSS-API
[Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1401): [client 10.120.22.74] Client didn't delegate us their credential
[Tue Apr 19 11:09:04 2011] [debug] src/mod_auth_kerb.c(1420): [client 10.120.22.74] GSS-API token of length 163 bytes will be sent back

Isolate Error Factors

determine if error is coming from Kerberos-auth side, or Apache Proxy / Splunk-side,

set your Apache splunk.conf to be a simple Location proxy (leave out Splunk stuff for now), using Kerberos, and set logs to Debug, this way you can determine if basic Kerberos auth works or not,

LoadModule auth_kerb_module /usr/lib64/httpd/modules/mod_auth_kerb.so


<VirtualHost *:80>
ServerName mrx01.corp.local
ServerAlias mrx01.corp.local
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>


<VirtualHost *:443>
ServerName mrx01.corp.local
ServerAlias mrx01.corp.local
LogLevel Debug
ErrorLog /var/log/httpd/splunk_error
CustomLog /var/log/httpd/splunk_access common

SSLEngine On
SSLCertificateKeyFile /etc/pki/tls/private/mrx01.corp.local.pem
SSLCertificateFile /etc/pki/tls/certs/mrx01.corp.local.pem

<Location />
AuthType Kerberos
AuthName "Kerberos Splunk"
KrbAuthRealms CORP.LOCAL
KrbMethodK5Passwd Off
KrbServiceName HTTP/mrx01.corp.local
Krb5KeyTab /etc/httpd/httpd.keytab
KrbLocalUserMapping on
KrbSaveCredentials On
Require valid-user
SSLRequireSSL
</Location>

</VirtualHost>

restart Apache, try to login to Splunk, http://mrx01.corp.local

Successful Kerb connection looks like this,

This eliminates any Kerb setup issues. 

NTLM Error 


Warning: received token seems to be NTLM, which isn't supported by the Kerberos module. Check your IE configuration.

add the site you are accessing to your local Internet options Local Sites,



for NTLM, also in the Apache conf, change KrbMethodK5Passwd On
Watch the logs, this setting will popup a user + pw prompt that may give glues to whats going wrong.

- also NTLM errors usually mean your target host is not in DNS.


SSO Debug

check the debug page once you can connect to splunk (but cant login via SSO). In this example, my REMOTE-USER value is not being passed

enable this SSO Debug page in /opt/splunk/etc/system/local/web.conf, add variable enableWebDebug = true



Comments