Send Apache Logs via Syslog to Splunk

posted Oct 4, 2010, 11:56 PM by Luke Harris   [ updated Oct 5, 2010, 2:38 PM ]
  • We are testing a new way of sending our Apache logs to Splunk and I'd like to share this powerful information with the internetz :P
  • We used to schedule a cron job to rsync the logs every 5 minutes to our splunk server, however we wanted to get a real-time view of the activity on our web tier. We decided that we didn't want the bloat of the Splunk Light Weight Forwarder (LWF) on every web server across our Dev, Test, UAT & Prod environments, so our resident perl/web guru configured apache to log directly to syslog. We then configured syslog-ng on the web servers to forward the apache events to our syslog server, where they are then forwarded to our splunk server :)
  • These are the additional lines that we added to the apache configuration file:
            #define a logformat called 'nxhttp'
            LogFormat "%h \"%r\" content_type=\"%{Content-Type}o\" uri_path=\"%U\" status=%>s response_bytes=%b sent_bytes=%O predeflate=%{predeflate}n method=%m dur_micro=%D via=\"%{Via}i\"" nxhttp
            #send 'nxhttp' formatted logs down a pipe to logger - only if it is an 'nxpage' (which is handled elsewhere)
            CustomLog "|/usr/bin/logger -p -t netxpress" nxhttp env=nxpage
  • These are the additional lines that we added to syslog-ng.conf on the web servers:
            filter local6 { facility(local6); };
            filter notlocal6 { not facility(local6); };
            destination local6 { file("/var/log/$HOST/nxhttp/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY"); };
            destination loghostnxhttp { tcp("10.xx.yy.zz" port(5516)); };
            log { source(main); filter(local6); destination(local6); };
            log { source(main); filter(local6); destination(loghostnxhttp); };
  • These are the additional lines that we added to syslog-ng.conf on the syslog server:
            source remotenxhttp { tcp(port(5516) keep-alive(yes) max-connections(512)); };
            destination nxhttp { file("/var/syslogng/$HOST/nxhttp/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY"); };
            destination splunknxhttp { tcp("" port(5603) ); };
            log { source(remotenxhttp); destination(nxhttp); };
            log { source(remotenxhttp); destination(splunknxhttp); };

  • We then added a new TCP Data input (Port 5603) on our splunk server to ingest the apache events :)