Living with DHCP and CARP...

 

Using OpenBSD's CARP and PF, with Dynamic IP Addresses...

Suggested, Preliminary Reading:

The OpenBSD FAQ is a great source of information. For more information on OpenBSD's CARP, see the FAQ here: http://cvs.openbsd.org/faq/faq6.html#CARP and this one: http://cvs.openbsd.org/faq/pf/carp.html

Overall Summary:

The scripts and examples here will allow the use of two or more redundant firewall hosts, using stock tools in OpenBSD. PF, CARP, pfsync and ifstated with Dynamic IP’s – including ISP's that restrict their DHCP address assignments restricted to MAC addresses. The scripts will do some safe, legitimate layer 2 address spoofing for the purpose of network uptime, failover redundancy, proper IP address renewal via DHCP and secured state table synchronization between peers.

In the event one needs to do any updates (hardware or software) or in the event the systems goes down unexpectedly - or even if a network cable that would otherwise affect routing gets disconnected, external connectivity continues unaffected. Here is an example scenario of  two ISP's that I've used with DHCP address layer 2
binding restrictions. Both working differently to each that would otherwise not allow this to work conventionally, atleast without these scripts or something similar:

What got me started on how I could accomplish this was after reading a post on a mailing list thread from some people back in 2004, as seen here. I believe it was around OpenBSD 3.8, when I started running this and have had good success with it since. Hopefully what I have here will benefit others as well.

This document applies to OpenBSD 4.4, later versions may work differently.

Examples and Scenarios:

ISP-A  (DSL), Scenario 1:
In order to get a public IP Address, I get an address from DHCP, which gives me an RFC1918 address which in turn restricts and limits access to only a MAC address registration website on their network. From there, I have to manually input and register my MAC address in a form before I can renew my IP address to get a public address.

If I were to change my NIC card or try to get an IP from DHCP on another system, it sees this change and restarts the above process in order to re-register new MAC Address. I would then have to manually delete the old MAC address and re-register a new one in order to make the switch. Under normal circumstances, this would pose problems for CARP not working as intended – such as requiring manual user intervention.

ISP-B (CABLE), Scenario 2:
Again, with this ISP I am alloted only one public IP address which is also restricted to one unique MAC address. If I try with another NIC or another system, it doesn't assign me an address unless I 'manually' reset the modem and then renew my IP address obtaining a 'new' address also requiring manual user intervention.

Overview of the Scripts:

CARP is run on the internal network side of the clustered firewall hosts. CARP does not work directly with DHCP, as of this writing.

The CARP hosts sync all states with their peer(s), using pfsync, optionally, secured and isolated interfaces. The hosts work in a master/slave mode rather than load balancing – multiplexing connections work as normal with PF.

If the preferred master host is shut down (gracefully or not), then the slave host(s) will automatically kick in and become master according to their priority level.

The means it uses to accomplish working with a different MAC address is to spoof the external interface's MAC address to one that the ISP or Modem knows. By doing this, we're able to obtain an IP address renewal lease period from DHCP, rather than a new address or no address.

If and when the preferred master host returns back on-line, it will then take over it's role again, automatically.

If at any time a network cable (internal or external) is unlinked from the preferred master host, then the backup host will kick in and continue routing transparently.

If and when this cable is re-linked, the preferred master host will kick back into action again, if everything tests ok.

In the event you [un]intentionally do an ‘ifconfig carp1 down’ on the preferred master host for scheduled maintenance, upgrades or anything, then the backup host will become the master host will have the demoted master host's (now backup host) interface MAC address spoofed to a fictitious MAC address as to avoid any conflicts or potential problems, etc.

The scripts assume you have at least 3 NIC's in each host with one on each, dedicated as a CARP sync device interface, preferably connected with an X-Over cable or isolated, secured switch.

The gateway address used for all systems (except on any CARP host), is the CARP, Virtual IP Address.

See the scripts for more detailed examples. But before continuing, don't forget to back up!

Locations, Permissions and Script Internals:

The ifstated.conf file stored in /etc - backup, edit and modify accordingly!

The ifstated.sh file stored in /usr/local/libexec - edit and modify accordingly!

The two files above need to be edited properly on each host to reflect it's role.

The scripts use de:ad:00:00:be:ef as the a fictitious / spoofed link-layer address; change if desired.

carp1 is used for the internal carp interface names – a personal preference. The intention of the convention is that I use ‘1’ for ‘(1)nternal’, ‘0’ for (0)utside and ‘2’ for PFSync, DMZ’s, NetFlow packets, Symon, Symux, Syweb, DNS AXFR/IXFR's, etc.; change as desired.

The pf.conf rules should be tested by doing ifstated -vvnd on each host to see how they work. A mis-configuration of pf.conf, ifstated.sh or ifstated.conf could cause unexpected results and grief. I also highly recommend starting with a basic, but secure pf.conf ruleset. From here on, slowly move to more advanced levels as testing permits, in order to allow for proper functionality.

The scripts are also set to modify the backup host via ifstated to have itself, in turn, have internet access NAT’d by the master host. i.e, sudo ifconfig down carp1 sets the the preferred backup host to continue with access to the internet, via the promoted master host.

The mac_master= variable in the ifstated.sh script needs to be the real link-layer (MAC) address of the master host’s external interface and needs to be defined to the same on both host’s ifstated.sh scripts.

The backup_ip= variable in the master host's ifstated.sh script must be the address of the backup host's internal interface.

The master_ip= variable in the backup hosts's ifstated.sh script must be the address of the master host's internal interface.

Config Files and Their Contents:

As an example, the backup host has interface names (sis0, sis1 and sis2) and the master host (vr0, vr1, vr2). The internal network uses 192.0.2.0/24 and network 10.0.0.0/8 is used on the pfsync interfaces. The 172.16.0.0/12 is used on the wireless segment, in this case.

This is on the backup host:
/etc/hostname.pfsync0:
syncdev sis2
up

/etc/hostname.carp1:
inet 192.0.2.1 255.255.255.0 192.0.2.255 vhid 1 pass mekmitasdigoat advskew 128

/etc/hostname.sis0:
# Running this at the wrong time could cause some delays. Leave this up to ifstated(8).
description "--==| External Interface |==--"

/etc/hostname.sis1:
inet 192.0.2.253 255.255.255.0 NONE description "--==| Internal Interface |==--"

/etc/hostname.sis2:
inet 10.0.0.2 255.0.0.0 NONE description "--==| PFSync |==--"

/etc/mygate – Make sure it's empty:
sudo sh -c 'echo “” > /etc/mygate'

/etc/rc.conf.local:
pf=YES

/etc/sysctl.conf:
net.inet.ip.forwarding=1
net.inet.carp.log=1
net.inet.carp.preempt=1
This is on the master host:
/etc/hostname.pfsync0:
syncdev vr2
up

/etc/hostname.carp1:
inet 192.0.2.1 255.255.255.0 192.0.2.255 vhid 1 pass mekmitasdigoat

/etc/hostname.vr0:
# Running this at the wrong time could cause some delays. Leave this up to ifstated(8).
description "--==| External Interface |==--"

/etc/hostname.vr1:
inet 192.0.2.254 255.255.255.0 NONE description "--==| Internal Interface |==--"
/etc/hostname.vr2 (pfsync_dev):
inet 10.0.0.1 255.0.0.0 NONE description "--==| PFSync |==--"
/etc/mygate - Make sure it's empty:
sudo sh -c 'echo “” > /etc/mygate'
/etc/rc.conf.local:
pf=YES

/etc/sysctl.conf:
net.inet.ip.forwarding=1
net.inet.carp.log=1
net.inet.carp.preempt=1

------------------------------------------------------------------------------------------------
The master host ifstated.conf starts here:

# Initial State
init-state auto

# Macros
if_master="carp1.link.up"                # carp_if.link.up only
if_backup="!carp1.link.up"                # not carp_if.link.down
if_link_up="vr0.link.up && vr1.link.up"            # if both links are up
if_link_down="vr0.link.down || vr1.link.down"        # if either link goes down

# Use this script to set some variables and be sure to load pf.conf.
run "/usr/local/libexec/ifstated.sh"
run "/sbin/pfctl -qf /etc/pf.conf"

state auto {
        if $if_link_down {
                set-state backup
        }
        if $if_master {
                set-state master
        }
        if $if_backup {
                set-state backup
        }
}

state master {
        init {
                # ext_if hostname.if(5) should be started as 'down' with no ipaddr.
                run "/sbin/ifconfig `/bin/cat /tmp/_ext_if`\
                lladdr `/bin/cat /tmp/_mac_master` up"

                # Clean up stale routes; dhclient will overwrite default route.
                #run "/sbin/route -qn flush" # flushes arp cache too.
                #run "/sbin/route -qn delete default" # delete default route
                # Load 'master' pf.conf rules before going public, if required (see below).
                #run "/sbin/pfctl -qF all -f /etc/pf.conf.master"
                #run "/sbin/pfctl -qf /etc/pf.conf.master"

                # Renew the ip lease - hopefully stays the same, for pfsync.
                run "/sbin/dhclient `/bin/cat /tmp/_ext_if`"

                # Other commands or scripts here, after we're fully operational:
                #run "run here and lower"
        }

        if $if_backup {
                set-state backup
        }

        if $if_link_down {
                set-state backup
        }
}

state backup {
        init {
                # This process should be terminated, first.
                run "/usr/bin/pkill -9 dhclient"

                # Clean up and avoid potential problems.
                run "/sbin/ifconfig `/bin/cat /tmp/_ext_if`\
                delete lladdr `/bin/cat /tmp/_mac_spoof` down"


                # For cases when we need (or not) to change filter, rdr, nat and other modes while in a backup state.
                #run "/sbin/pfctl -qF all -f /etc/pf.conf.backup"
                #run "/sbin/pfctl -qf /etc/pf.conf.backup"

                # Clean up stale routes and/or arp cache, recommended, but optional.
                #run "/sbin/route -qn delete default" # delete default route.
                run "/sbin/route -qn flush" # flushes arp cache too.

                # Allows us out to internet via the master host.
                run "/sbin/route -qn add default `/bin/cat /tmp/_backup_ip`"

                # Other commands or scripts here, after we're properly down:
                #run "run here and lower"
        }

        if $if_master {
                if $if_link_up {
                        set-state master
                }
        }

        if $if_master {
                if $if_link_down {
                        set-state backup
                }
        }
}

------------------------------------------------------------------------------------------------
The master host ifstated.sh file starts here:

#!/bin/sh
ext_if="sis0"                    # External interface name.
backup_ip="X.X.X.X"                # IP of backup CARP host, int_if.
mac_master="XX:XX:XX:XX:XX:XX"            # lladdr of master CARP host, ext_if.
mac_spoof="de:ad:00:00:be:ef"            # Generic spoof address, recommended.

umask 137

if [ ! -z $mac_spoof ]; then
            echo $mac_spoof > /tmp/_mac_spoof
else
        echo 'undefined $mac_spoof' > /var/log/ifstated.sh.log
        exit 1
fi

if [ ! -z $mac_master ]; then
        echo $mac_master > /tmp/_mac_master
else
        echo 'undefined $mac_master' > /var/log/ifstated.sh.log
        exit 1
fi


if [ ! -z $ext_if ]; then
            echo $ext_if > /tmp/_ext_if
else
        echo 'undefined $ext_if' > /var/log/ifstated.sh.log
        exit 1
fi
if [ ! -z $backup_ip ]; then
            echo $backup_ip > /tmp/_backup_ip
else
        echo 'undefined $backup_ip' > /var/log/ifstated.sh.log
        exit 1
fi

------------------------------------------------------------------------------------------------
The backup host ifstated.conf starts here:

# Initial State
init-state auto

# Macros
if_master="carp1.link.up"            # carp_if.link.up only
if_backup="!carp1.link.up"            # not `carp_if.link.down`

# Use this script and be sure to load pf.conf, before proceeding.
run "/usr/local/libexec/ifstated.sh"
run "/sbin/pfctl -qf /etc/pf.conf"

state auto {
        if $if_backup {
                set-state backup
        }

        if $if_master {
                set-state master
        }
}

state backup {
        init {
                # This process should be terminated, first.
                run "/usr/bin/pkill -9 dhclient"

                # Clean up and avoid potential problems.
                run "/sbin/ifconfig `/bin/cat /tmp/_ext_if`\
                delete lladdr `/bin/cat /tmp/_mac_spoof` down"

                # For cases when we need (or not) to change filter, rdr, nat and other modes while in a backup state.
                #run "/sbin/pfctl -qF all -f /etc/pf.conf.backup"
                #run "/sbin/pfctl -qf /etc/pf.conf.backup"

                # Clean up stale routes and/or arp cache, recommended, but optional.
                #run "/sbin/route -qn delete default" # delete default route only.
                run "/sbin/route -qn flush" # flushes arp cache too.

                # Allows us out to internet via the master host.
                run "/sbin/route -qn add default `/bin/cat /tmp/_master_ip`"

                # Other commands or scripts here, after we're properly down:
                #run "run here and lower"
        }

        if $if_master {
                set-state master
        }
}

state master {
        init {
                # ext_if hostname.if(5) should be started as 'down' with no ipaddr.
                run "/sbin/ifconfig `/bin/cat /tmp/_ext_if`\
                lladdr `/bin/cat /tmp/_mac_master` up"

                # Clean up stale routes; dhclient will overwrite default route.
                #run "/sbin/route -qn flush" # flushes arp cache too.
                #run "/sbin/route -qn delete default" # delete default route only.
                # Load 'master' pf.conf rules before going public; if required (see above).
                #run "/sbin/pfctl -qF all -f /etc/pf.conf.master"
                #run "/sbin/pfctl -qf /etc/pf.conf.master"

                # Renew the ip lease - hopefully stays the same, for pfsync.
                run "/sbin/dhclient `/bin/cat /tmp/_ext_if`"

                # Other commands or scripts here, after we're fully operational.
                #run "run here and lower"
        }

        if $if_backup {
                set-state backup
        }
}

------------------------------------------------------------------------------------------------
The backup host ifstated.sh starts here:

#!/bin/sh

ext_if="xl0"                    # External Interface on this host.
master_ip="X.X.X.X"                # IP of master CARP host, int_if.
mac_master="XX:XX:XX:XX:XX:XX"            # lladdr of master CARP host, ext_if.
mac_spoof="de:ad:00:00:be:ef"            # Generic spoof address, recommended.
umask 137

if [ ! -z $mac_spoof ]; then
            echo $mac_spoof > /tmp/_mac_spoof
else
        echo 'undefined $mac_spoof' > /var/log/ifstated.sh.log
        exit 1
fi

if [ ! -z $mac_master ]; then
        echo $mac_master > /tmp/_mac_master
else
        echo 'undefined $mac_master' > /var/log/ifstated.sh.log
        exit 1
fi

if [ ! -z $ext_if ]; then
            echo $ext_if > /tmp/_ext_if
else
        echo 'undefined $ext_if' > /var/log/ifstated.sh.log
        exit 1
fi

if [ ! -z $master_ip ]; then
            echo $master_ip > /tmp/_master_ip
else
        echo 'undefined $master_ip' > /var/log/ifstated.sh.log
        exit 1
fi

------------------------------------------------------------------------------------------------
An example network diagram can be seen by clicking here.

The permissions should be set as outlined here, on each host:

chown root:wheel /etc/ifstated.conf ; chmod 600 /etc/ifstated.conf
chown root:wheel /usr/local/libexec/ifstated.sh ; chmod 700 /usr/local/libexec/ifstated.sh