D-Link DSL-G604T Hacking: NetFlow

Note: I did all of this on a x86-based Ubuntu Feisty Fawn box. Most things I write here should work on any UNIX, or even Windows provided you have the right software, but use your head. Also, this should work on other routers besides the G604T -- any Linux router with a MIPSEL architecture should be able to do this with little to no modifications.

Why NetFlow?

I love ntop -- it's an excellent way to keep a track of your network traffic. It makes nice graphs of traffic throughput and keeps all manner of statistics, all available in your browser at http://localhost:3000. It's available in Ubuntu's (and Debian's?) package repositories as well, so it's just a matter of sudo apt-get install ntop to get up and running.

ntop displaying throughput

The problem I soon ran into with ntop, or with any of the other programs that do similar jobs (darkstat is good), is that they just run on one computer. This is OK if that computer happens to be the only one in the network, or it is acting as the router/switch for your home network, but for many people that is not the case. I have a home wired and wireless network with a number of clients, that connects to the internet via a very common piece of hardware -- D-Link's DSL-G604T wireless router and DSL modem.

So how do I make ntop be able to see the entire network? Installing an entire monitoring system like ntop onto a home DSL router I ruled out immediately. Apparently darkstat, an ntop-competitor, is capable of being installed onto routers, but after playing around with it I decided against it.

Cisco, however, have come up with a wonderful solution to this, and it's called NetFlow. I won't go into details here (read the Wikipedia article), but in a nut-shell NetFlow is a way of collecting data from a network interface then sending it somewhere else to be processed. You run a netflow generator/emitter on the router, which collects network data, then forwards this data on to another computer that runs a netflow collector. We already have the netflow collector -- ntop has a plugin to handle this -- now all we need is the netflow generator to run on the G604T.

"Hacking" the G604T

We're lucky with the DSL-G604T, as it runs Linux. To get at it, all we need to do is enable the telnet interface to it. You do this through the router's web interface. Then all you need to do is telnet 10.1.1.1and log in with your router's username and password, and there you have a shell (note: throughout this article I will assume that 10.1.1.1 is the IP address for your router -- it may be different, I've seen some G604Ts with 192.168.0.1).

If it can run Linux, then it can run Linux programs, can't it? Certainly! We've just got a couple of problems. The first is that the G604T has no permanent storage easily accessible -- if somebody can show me otherwise (short of flashing the router) I'd be happy to hear about it. The second is that we can't just put a binary for, say, vi, on the router and expect to run it; your computer is probably x86 architecture, the G604T is what's known as MIPSEL. MIPSEL is just MIPS, but little-endian instad of big-endian (MIPS can run both ways, nifty!).

The first problem is easy enough to fix. The G604T has a RAM-disk (ramfs) set up at /var/. We can read and write to this all day. However when the router reboots, this is of course wiped. We'll deal with this problem later on.

The second problem is easy to fix too. There are cross-compilers for the MIPSEL architecture. Lots of thanks to this guy, who not only gave me my answer with a simple Google search, but has a pre-compiled i386-to-MIPSEL cross-compiler for download. Extract it to /opt/, and you're away and compiling. Try out a Hello World using the /opt/mipselroot/bin/gcc.

We can make binaries, we can store them somewhere -- how to get them onto the router, though? Well unfortunately the G604T doesn't have ssh, or we could just scp them across. The router does have wget though. I run an apache web server on my main box, so it's just a matter of putting the binary in /var/www/ on my computer.

nProbe: Netflow Generator for the G604T

Now that we can compile and run programs for the router, we can make a netflow generator for it. I hunted around all over the place for a good and small netflow generator, but ended up with nProbe (written by the same people as ntop). I wasn't very keen on this, as I wanted to use something more "open" (nProbe is GPL, but the author has decided to charge €99 or something for it). I tried to use fprobe, which looks promising, but just couldn't get it to run properly.

Despite the author charging €99 for nProbe, it is still nonetheless GPL, so anybody who has bought a copy is allowed to redistribute it and modify it as much as they like (and so can the next guy). Thankfully for us, somebody did this a some time ago, and nProbe-source-1.3.1.tgz is floating around, mostly in NetBSD repositories it seems. A simple Google search did the trick for me. Get the source off me here (go easy, Google Pages doesn't give me unlimited traffic), or off this BSD bunch here.

nProbe 1.3.1 is very old; nProbe is now up to version 4, but 1.3.1 does the job.

Compilation

Compiling nProbe is trivial. However, before we compile it, we need to make sure our environment is properly set up to cross compile. Anyway, here's what I did to compile nProbe:

tar -zxvf nProbe-source-1.3.1.tgz

cd nProbe

OPATH=$PATH

export PATH=/opt/mipselroot/bin/:$PATH

./configure --host=mipsel-linux

At this point you will find out what libraries you need to install -- I was missing libpcap0.8-dev. Continuing on:

make

This failed for me, throwing up an error "nProbe.c:194: `osName' undeclared (first use in this function)". The problem was in version.c -- an automatically generated file that holds variables related to versions and build-dates to print out when the program is run (not exactly critical, you can write what you like). A variable called osName was not declared, possibly because of the cross-compiling situation. At any rate, the fix is to simply add a new line in version.c saying "char * osName = "mipsel-linux-gnu";".

Try compiling again, and you should end up with a nice little MIPSEL binary called nprobe. Try to run it on your computer -- it won't run of course, this is a MIPSEL binary, not an i386 one!

If you're super-lazy or just can't make it compile because of sadly insufficient geekery, I made a pre-compiled version for you.

You may want to restore your original path if you're going to keep using the same terminal, do this with:

PATH=$OPATH

Now's the point where you put the binary where you want to fetch it from later, I put it in the root directory of my web server.

Running nProbe

Telnet into the router, change to /var/ and get the nProbe binary. I did wget http://10.1.1.88/nprobe (10.1.1.88 is the IP of my main box), you will have it stored somewhere else so change the URL as suited. Try running nProbe as it is (make it executable first, chmod +x nprobe), no parameters; you should get a page full of usage information.

To fire it up and have it start collecting data, you need to give it the address and port (2055 is the usual netflow port) to send the data to. You do this with the "-n" flag. The line I use to start nProbe is ./nprobe -n 10.1.1.88:2055. This (on my G604T) automatically chooses the br0 interface, which seems to be correct. It will by default collect up information for two minutes before it sends any data, so don't be worried if you don't get anything straight away. You can change this easily enough, there's a whole bunch of flags as you can see from the usage information.

Setting Up ntop

So now we have our router merrily sending us netflow information, we need to set up our netflow collector so we can do something with them. I use ntop, you can use anything you like. Setting up ntop is easy. First of all, we need to change the interfaces it is analysing. Set the netflow interface to "lo", which is the loopback device. You need to give it some kind of local interface, and loopback is good enough.

Ntop handles netflow support via a plugin. In the plugins menu, activate the netflow plugin. Now we can configure it. Add a new NetFlow device, then set "Local Collector UDP Port" as 2055, or whatever port you're sending on. Set the "Virtual NetFlow Interface Address" to the address of your router followed by the IP mask. I set mine to "10.1.1.1/255.0.0.0", however if your router IP is 192.168.0.1, you should (I think! Not quite sure...) to "192.168.0.1/255.255.255.0".

You will probably need to switch the interface that reports are being generated for -- to do this, just go to the main page (Summary->Traffic), scroll down a bit and you'll see a link to click on ("[switch]") that will allow you to do this.

Congratulations, you now have full data reports and analysis for your entire network -- using a cheapie home DSL router and free open-source software =^_^=

Keeping it all running

This way of doing things isn't really ideal. For one, you have to actually manually telnet into the router to run nProbe (remember, short of flashing new firmware onto the router, I know of no way to change start-up programs on it). Secondly, every time the router reboots we have to start again, by downloading the binary. So if we lose connection between your main box and the router, or either reboots, then we have to fiddle around with stuff manually.

auto_nprobe_up.sh in action: taking care of a router reboot

This is why I've written a very dodgy and kludgey pair of scripts to keep things ticking over. When automating telnet, there's really on one solution under UNIX, and that's using expect. Expect is a Tcl-based scripting language designed for automating telnet, ssh, ftp etc. If you're using Ubuntu, you may not have it installed, but you know what to do -- just sudo apt-get install expect. It's not the most sophisticated script that has ever been written in expect, but it works. Here's the script, called nprobe_up.sh:

#!/usr/bin/expect

set timeout 20

set host 10.1.1.1
set user admin
set password YOUR_PASSWORD_HERE
set nprobedl http://10.1.1.88/nprobe
set collector_ip 10.1.1.88
set collector_port 2055


spawn telnet $host

expect "login:"
send "$user\n"
expect "word:"
send "$password\n"
expect "#"

send "cd var/\n"
expect "#"
send "if \[ ! -e nprobe \]; then wget $nprobedl -O nprobe; chmod +x nprobe; fi\n"
expect "#"

send "ps | grep ./nprobe | grep $collector_ip\n"
expect "root" { exit } \
"#" { send "./nprobe -n $collector_ip:$collector_port\n"
expect "Exporting"
expect -timeout -1 "#" }

Make sure it's executable. This script simply logs into the router, downloads the nProbe binary if not already there and makes it exectuable. It then checks to see if nProbe is already running, and if it's not then it starts it up.

The second script is an ordinary bash script, even dodgier and kludgier than the first ^_^. It's called auto_nprobe_up.sh:

#!/bin/bash

NPROBE_BIN=/home/tom/apps/nprobe_up/nprobe_up.sh
HOST=10.1.1.1

while [ 0 -eq 0 ]; do
ping -c 1 -W 2 $HOST > /dev/null
HOST_UP=$?
RUN=`ps -e | grep nprobe_up.sh`
if [ $HOST_UP -ne 0 ]; then
echo "*Cannot ping host*"
if [ -z "$RUN" ]; then
echo "***Telnet session closed"
else
echo "***Telnet session running, will kill $NPROBE_PID"
kill $NPROBE_PID
NPROBE_PID=-1
fi
else
if [ -z "$RUN" ]; then
echo "***Telnet session not running, will start"
$NPROBE_BIN &
NPROBE_PID=$!
fi
fi

sleep 30 #Interval between checks. Probably a bit short, but I'm impatient.
done

This script makes sure nprobe_up.sh keeps running. If nProbe terminates on the router for whatever reason, then it will restart it. Most importantly it makes sure the router is in contact; if the router reboots or loses power, the telnet session will not terminate by itself. So, if the router stops responding to pings, we must kill the telnet session, and once we can contact the router again, start up nprobe_up.sh again. It's not foolproof, but it does a reasonable job in most conditions.

To make these two scripts run, you will need to fiddle around with some of the values near the top of both scripts. I put the two scripts together in my ~/apps sub-directory.

Anyway, that should do a good enough job to keep NetFlow data coming for an average home network. Isn't Linux great?

-Tom Cornall, 2007