Create IPIP tunnel between networks

Sysinfo: Centos  7

A) Background

Need to establish a private tunnel between 2 networks, and forward any IPs on Server A to Server B via this new tunnel

The IP will be routed via tunnel from Server A to Server B, then will be routed on B to another interface and then to server C. ON the way back, the packets will follow back their original routes.

• 1. user checks port connectivity from server A

  2. netcat 122.195.129.133 2150

  3. existing Route (#2 on A) routes it to tunnel-b interface, request goes through tunnel to Server B

  4. Route (#2 on B) routes it to another interface, p1p1, packet goes from B:p1p1 to external server C

  5. server C replies back with a packet, back to interface it received from, p1p1

  6. return packet goes back to tunnel2 (route), return packet goes tunnel-a > tunnel-b (via gateway Route #1 on B)

  7. user receives reply via eth0 (via Route #1 on A)

Setup

(the following describes manual setup of IPIP tunnel, you can also use this script)

Server A  Name = "server A"  IP=172.31.23.254 (AWS network)

Server B  Name = "server B"  IP=172.31.23.64   (AWS network)

On both servers, add TUN module,

check tunnel mod on both Servers

Server A

root@serverA# lsmod | grep tun

tunnel4                13252  1 ipip

ip_tunnel              25163  1 ipip

tun                    31665  0

Server B

[root@serverB]# modprobe tun

[root@serverB]# lsmod | grep tun

tun                    27141  0

IP Forwarding

enable IP forwarding,

vi /etc/sysctl.conf

add

reload sysctl

B) Test basic connectivity

Both should be able to connect one another (use python SimpleHTTPServer + netcat to check connectivity)

Server A

Server B

and vice versa

if they cant connect, the tunnel wont work

C) Creating the Tunnels

Temporary Tunnel

Create tunnel on Server A, assign an IP to this new tunnel interface, here im using a generic 192.168.5.1 for A and 192.168.5.2 for B

check for new Tunnel interface

root@serverA# ip a

8: tunnel-b@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8981 qdisc noqueue state UNKNOWN group default qlen 1000

    link/ipip 172.31.23.254 peer 172.31.23.64

    inet 192.168.5.1/24 scope global tunnel-b

       valid_lft forever preferred_lft forever

Do the same on Server B

check for new IP,

4: tunnel-a@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8981 qdisc noqueue state UNKNOWN

    link/ipip 172.31.23.64 peer 172.31.23.254

    inet 192.168.5.2/24 scope global tunnel-b

       valid_lft forever preferred_lft forever

try pinging from A to B

should be pingable

Test connectivity as in section B, but change the IP to the IP address of the tunnel interface,

A > B test

B > A test

Permanent Tunnel

To create a permanent Tunnel that survives Network + Machine restart,

on Server A

root@serverA# vim /etc/sysconfig/network-scripts/ifcfg-tunnel-a

TYPE=IPIP

BOOTPROTO=none

DEVICE=tunnel-b

ONBOOT=yes

MY_INNER_IPADDR=192.168.5.1/30     // Tunnel IP

MY_OUTER_IPADDR=172.31.23.254    // primary local IP address

PEER_OUTER_IPADDR=172.31.23.64     // remote peer primary IP

this tunnel will be in place after a reboot, its managed by Network Manager

on server B

root@serverB# vim /etc/sysconfig/network-scripts/ifcfg-tunnel-b

TYPE=IPIP

BOOTPROTO=none

DEVICE=tunnel-a

ONBOOT=yes

MY_INNER_IPADDR=192.168.5.2/30     // Tunnel IP

MY_OUTER_IPADDR=172.31.23.64    // primary local IP address

PEER_OUTER_IPADDR=172.31.23.254     // remote peer primary IP

bring up this interface on Server B first

now on server A

to restart the interface

ping from each tunnel to make sure theyre pingable

check the status of the tunnel

D) Add a Route

Gateway Route

you need to have a gateway route in place for back-forth communication between server A and server B, otherwise packets will only flow 1 way (A > B)

On both A and B, this gateway route should already be in place once the tunnel network interface comes up

you can also add it manually,

Temporary Route

to route IPs from Server A to Server B via Tunnel, add a route (on server A)

root@serverA# ip route add 122.195.129.133 dev tunnel-b

check this Route

root@serverA# ip route get 122.195.129.133

122.195.129.133 via 192.168.5.2 dev tunnel-b src 192.168.5.1

run a Traceroute to see where the IP is going to,

on server B, listen on Tunnel interface for incoming packets,

root@serverB# tcpdump -i tunnel-a

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on tunnel-a, link-type RAW (Raw IP), capture size 262144 bytes

15:02:03.217834 IP ip-192-168-5-1.ec2.internal.56547 > 122.195.129.133.33466: UDP, length 32

15:02:03.217839 IP ip-192-168-5-1.ec2.internal.56417 > 122.195.129.133.33467: UDP, length 32

15:02:03.217871 IP ip-192-168-5-1.ec2.internal.42617 > 122.195.129.133.33468: UDP, length 32

on server B, to route from tunnel to p1p1 interface, add a route,

122.195.129.133   192.168.28.17   255.255.255.255 UGH   100    0        0 p1p1

Permanent Route

to keep route permanent, add route file (on server A), this routes your IP to Server B tunnel

now enable this route

check to make sure your Route is now present in routing table

root@serverA# route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

0.0.0.0         172.31.23.1     0.0.0.0         UG    0      0        0 eth0

172.31.23.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

192.168.5.0     0.0.0.0         255.255.255.0   U     0      0        0 tunnel-b

122.195.129.133 192.168.5.2    255.255.255.255 UGH   50     0        0 tunnel-b

run traceroute(server A) and tcpdump (on server B) again to make sure server B is getting the incoming packets

E) Cleaning up Tunnels

using IP command, disable tunnel interface

ip link set dev <tunnel name> down

delete an interface

ip link delete <tunnel name>

Troubleshooting

if routing traffic from server A to server B, sometimes server B will route your outgoing connection as the IP of the tunnel instead of itself, causing the connection to timeout.

ie, doing a TCP Dump on Server B, interface p1p1,

root@serverB:]# tcpdump -i p1p1 | grep 207.17.44.102

23:22:16.157069 IP 192.168.2.1.52896 > 207.17.44.102.41811: Flags [S], seq 804236576, win 14400, options [mss 1440,sackOK,TS val 3076723108 ecr 0,nop,wscale 10], length 0

here the IP shown is the IP of the tunnel interface, this request will not work because the endpoint does not recognize the tunnel IP.

in this case, you must change the packet header of the outgoing packet (on server B) to the IP of itself, not the IP of the Tunnel by Destination

root@serverB> iptables -t nat -A POSTROUTING -d 207.17.44.102 -j MASQUERADE

or by Source

root@serverB> iptables -t nat -A POSTROUTING -p tcp -s 192.168.5.1 -j MASQUERADE

in this example, 192.168.38.21 is the IP of the p1p1 interface on server B

now the connection will work,

[23:22 root@serverB:]# tcpdump -i p1p1 | grep 207.17.44.102

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on p1p1, link-type EN10MB (Ethernet), capture size 262144 bytes

23:28:10.299134 IP serverB.59888 > 207.17.44.102.41811: Flags [F.], seq 19064846, ack 4239206719, win 15, options [nop,nop,TS val 3077077251 ecr 238689707], length 0