add Forwarder data
\
Splunk Command Line Reference:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/AccessandusetheCLIonaremoteserver Note: the CLI may ask you to authenticate – it’s asking for the LOCAL credentials, so if you haven’t changed the admin password on the forwarder, you should use admin/changeme
Steps for Installing/Configuring Linux forwarders:
Step 1: Download Splunk Universal Forwarder:
http://www.splunk.com/download/universalforwarder (64bit package if applicable!)
Step 2: Install Forwarder
Step 3: Enable boot-start/init script:
/opt/splunkforwarder/bin/splunk enable boot-start (start splunk: /opt/splunkforwarder/splunk start)
Step 4: Enable Receiving input on the Index Server Configure the Splunk Index Server to receive data, either in the manager:
Manager -> sending and receiving -> configure receiving -> new
or via the CLI:
/opt/splunk/bin/splunk enable listen 9997 Where 9997 (default) is the receiving port for Splunk Forwarder connections
Step 5: Configure Forwarder connection to Index Server:
/opt/splunkforwarder/bin/splunk add forward-server hostname.domain:9997
(where hostname.domain is the fully qualified address or IP of the index server (like indexer.splunk.com), and 9997 is the receiving port you create on the Indexer:
Manager -> sending and receiving -> configure receiving -> new)
Step 6: Test Forwarder connection:
/opt/splunkforwarder/bin/splunk list forward-server
Step 7: Add Data:
/opt/splunkforwarder/bin/splunk add monitor /path/to/app/logs/ -index main -sourcetype %app%
Where /path/to/app/logs/ is the path to application logs on the host that you want to bring into Splunk, and %app% is the name you want to associate with that type of data
This will create a file: inputs.conf in /opt/splunkforwarder/etc/apps/search/local/
or do this manually,
add new Forwarder conf to monitor a file,
on box with forwarder, go to /opt/splunkforwarder/etc/apps/
create dir for your app /opt/splunkforwarder/etc/apps/myapp/local/
add inputs.conf
[monitor:///opt/myapp/csv/*.csv]
index=myapp
sourcetype=csv
crcSalt=<SOURCE> #this is to re-read the file on any change
here is some documentation on inputs.conf: http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf
Note: System logs in /var/log/ are covered in the configuration part of Step 7. If you have application logs in /var/log/*/
RESTART FORWARDER to pick up changes
**Step 8 (Optional): Install and Configure UNIX app on Indexer and nix forwarders:*
On the Splunk Server, go to Apps -> Manage Apps -> Find more Apps Online -> Search for ‘Splunk App for Unix and Linux’ -> Install the "Splunk App for Unix and Linux' Restart Splunk if prompted, Open UNIX app -> Configure Once you’ve configured the UNIX app on the server, you'll want to install the related Add-on: "Splunk Add-on for Unix and Linux" on the Universal Forwarder. Go to http://apps.splunk.com/ and find the "Splunk Add-on for Unix and Linux" (Note you want the ADD-ON, not the App - there is a difference!). Copy the contents of the Add-On zip file to the Universal Forwarder, in: /opt/splunkforwarder/etc/apps/. If done correctly, you will have the directory "/opt/splunkforwarder/etc/apps/Splunk_TA_nix" and inside it will be a few directories along with a README & license files. Restart the Splunk forwarder (/opt/splunkforwarder/bin/splunk restart) Note: The data collected by the unix app is by default placed into a separate index called ‘os’ so it will not be searchable within splunk unless you either go through the UNIX app, or include the following in your search query: “index=os” or “index=os OR index=main” (don’t paste doublequotes)
Step 9 (Optional): Customize UNIX app configuration on forwarders: Look at inputs.conf in /opt/splunkforwarder/etc/apps/unix/local/ and /opt/splunkforwarder/etc/apps/unix/default/ The ~default/inputs. path shows what the app can do, but everything is disabled. The ~local/inputs.conf shows what has been enabled – if you want to change polling intervals or disable certain scripts, make the changes in ~local/inputs.conf.
Step 10 (Optional): Configure File System Change Monitoring (for configuration files):http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Monitorchangestoyourfilesystem
TROUBLESHOOTING
edit the $SPLUNKFORWARDER/etc/log.cfg to enable DEBUG logging,
[splunkd]
rootCategory=WARN,A1
# TailingProcessor is meant to be used at level INFO -- without it, analyzing a
# normal diag becomes much harder. Do NOT remove the TailingProcessor logger.
category.TailingProcessor=DEBUG
category.WatchedFile=DEBUG
check stanza format
$SPLUNKFORARDER/bin/splunk cmd btool inputs list --debug
if you have more than 1 indexer, make sure both indexers have the index youre trying to send the data to, check /opt/splunk/etc/apps/search/local/indexes.conf on each indexer, make sure they match
check if ports are open and firewall is not blocking:
9997 for forwarders to the Splunk indexer. (forwarding and receiving data)
8000 for clients to the Splunk Web (webserver)
8089 – Splunk Management port (inter Splunk communication)
On Forwarder, if getting SSL errors (splunk SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed)
make sure /opt/splunkforwarder/etc/system/local/server.conf has this line
sslRootCAPath = /etc/pki/tls/certs/ca-bundle.crt