add Forwarder data

\

Splunk Command Line Reference:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/AccessandusetheCLIonaremoteserver Note: the CLI may ask you to authenticate – it’s asking for the LOCAL credentials, so if you haven’t changed the admin password on the forwarder, you should use admin/changeme

Steps for Installing/Configuring Linux forwarders:

Step 1: Download Splunk Universal Forwarder:

http://www.splunk.com/download/universalforwarder (64bit package if applicable!)

Step 2: Install Forwarder

Step 3: Enable boot-start/init script:

/opt/splunkforwarder/bin/splunk enable boot-start (start splunk: /opt/splunkforwarder/splunk start)

Step 4: Enable Receiving input on the Index Server Configure the Splunk Index Server to receive data, either in the manager: 

Manager -> sending and receiving -> configure receiving -> new 

or via the CLI: 

/opt/splunk/bin/splunk enable listen 9997 Where 9997 (default) is the receiving port for Splunk Forwarder connections

Step 5: Configure Forwarder connection to Index Server: 

/opt/splunkforwarder/bin/splunk add forward-server hostname.domain:9997 

(where hostname.domain is the fully qualified address or IP of the index server (like indexer.splunk.com), and 9997 is the receiving port you create on the Indexer: 

Manager -> sending and receiving -> configure receiving -> new)

Step 6: Test Forwarder connection: 

/opt/splunkforwarder/bin/splunk list forward-server

Step 7: Add Data: 

/opt/splunkforwarder/bin/splunk add monitor /path/to/app/logs/ -index main -sourcetype %app% 

Where /path/to/app/logs/ is the path to application logs on the host that you want to bring into Splunk, and %app% is the name you want to associate with that type of data 

This will create a file: inputs.conf in /opt/splunkforwarder/etc/apps/search/local/ 

or do this manually,

add new Forwarder conf to monitor a file,

• 1. on box with forwarder, go to /opt/splunkforwarder/etc/apps/

  2. create dir for your app  /opt/splunkforwarder/etc/apps/myapp/local/

  3. add inputs.conf

[monitor:///opt/myapp/csv/*.csv]

index=myapp

sourcetype=csv

crcSalt=<SOURCE>  #this is to re-read the file on any change

here is some documentation on inputs.conf: http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf 

Note: System logs in /var/log/ are covered in the configuration part of Step 7. If you have application logs in /var/log/*/

RESTART FORWARDER to pick up changes

**Step 8 (Optional): Install and Configure UNIX app on Indexer and nix forwarders:* 

On the Splunk Server, go to Apps -> Manage Apps -> Find more Apps Online -> Search for ‘Splunk App for Unix and Linux’ -> Install the "Splunk App for Unix and Linux' Restart Splunk if prompted, Open UNIX app -> Configure Once you’ve configured the UNIX app on the server, you'll want to install the related Add-on: "Splunk Add-on for Unix and Linux" on the Universal Forwarder. Go to http://apps.splunk.com/ and find the "Splunk Add-on for Unix and Linux" (Note you want the ADD-ON, not the App - there is a difference!). Copy the contents of the Add-On zip file to the Universal Forwarder, in: /opt/splunkforwarder/etc/apps/. If done correctly, you will have the directory "/opt/splunkforwarder/etc/apps/Splunk_TA_nix" and inside it will be a few directories along with a README & license files. Restart the Splunk forwarder (/opt/splunkforwarder/bin/splunk restart) Note: The data collected by the unix app is by default placed into a separate index called ‘os’ so it will not be searchable within splunk unless you either go through the UNIX app, or include the following in your search query: “index=os” or “index=os OR index=main” (don’t paste doublequotes)

Step 9 (Optional): Customize UNIX app configuration on forwarders: Look at inputs.conf in /opt/splunkforwarder/etc/apps/unix/local/ and /opt/splunkforwarder/etc/apps/unix/default/ The ~default/inputs. path shows what the app can do, but everything is disabled. The ~local/inputs.conf shows what has been enabled – if you want to change polling intervals or disable certain scripts, make the changes in ~local/inputs.conf.

Step 10 (Optional): Configure File System Change Monitoring (for configuration files):http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Monitorchangestoyourfilesystem

TROUBLESHOOTING

edit the $SPLUNKFORWARDER/etc/log.cfg to enable DEBUG logging, 

[splunkd]

rootCategory=WARN,A1

# TailingProcessor is meant to be used at level INFO -- without it, analyzing a

# normal diag becomes much harder.  Do NOT remove the TailingProcessor logger.

category.TailingProcessor=DEBUG

category.WatchedFile=DEBUG

check stanza format

$SPLUNKFORARDER/bin/splunk cmd btool inputs list --debug

if you have more than 1 indexer, make sure both indexers have the index youre trying to send the data to,  check /opt/splunk/etc/apps/search/local/indexes.conf on each indexer, make sure they match

check if ports are open and firewall is not blocking: 

9997 for forwarders to the Splunk indexer. (forwarding and receiving data)

8000 for clients to the Splunk Web (webserver)

8089 – Splunk Management port (inter Splunk communication)

On Forwarder, if getting SSL errors (splunk SSL Error = error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed)

make sure /opt/splunkforwarder/etc/system/local/server.conf has this line

sslRootCAPath = /etc/pki/tls/certs/ca-bundle.crt