Analyzing Web logs with AbuseIPDB

If you are running a Web Server, you can use KNIME to analyze your web traffic and look for suspect behaviour.

In this example I will examine the access.log, generated by the Apache Web server. The Line Reader node reads the web log one line at the time  as one column (cell) and the Cell Splitter then splits this column - on blanks - into different parts. This gives us the following information:

• the Client's IP address

• the time stamp

• the Request sent by the client

• the Response code the Web server gave. In this example I filter on Response code 200, which means 'ok'

•  the User Agent. This indicates the browser and operating system the Request came from. In the screenshot note, in the red box, the Shodan/Pull 1.0 user agent. This is, according to www.abuseipdb.com/ a maiicious site. 

In the Workflow below, I will use the AbuseIPDB API to find out more about this malicious site:

The important nodes here are the GET Request and the Expression node before that, where I construct the URL 

Below is the output of the Get request node for the Shodan-Pull record.  At docs.abuseip.com you can find more information on how to interpret the output.

For example, the metric abuse confidence score tells us the probability this site is malicious is 100 %.

A further development of this workflow might be to let it generate the adequate Rules to let our web server block this site.