Create IPIP tunnel between networks
Sysinfo: Centos 7
A) Background
Need to establish a private tunnel between 2 networks, and forward any IPs on Server A to Server B via this new tunnel
The IP will be routed via tunnel from Server A to Server B, then will be routed on B to another interface and then to server C. ON the way back, the packets will follow back their original routes.
user checks port connectivity from server A
netcat 122.195.129.133 2150
existing Route (#2 on A) routes it to tunnel-b interface, request goes through tunnel to Server B
Route (#2 on B) routes it to another interface, p1p1, packet goes from B:p1p1 to external server C
server C replies back with a packet, back to interface it received from, p1p1
return packet goes back to tunnel2 (route), return packet goes tunnel-a > tunnel-b (via gateway Route #1 on B)
user receives reply via eth0 (via Route #1 on A)
Setup
(the following describes manual setup of IPIP tunnel, you can also use this script)
Server A Name = "server A" IP=172.31.23.254 (AWS network)
Server B Name = "server B" IP=172.31.23.64 (AWS network)
On both servers, add TUN module,
modprobe tun
modprobe ip_tunnel
check tunnel mod on both Servers
Server A
root@serverA# lsmod | grep tun
tunnel4 13252 1 ipip
ip_tunnel 25163 1 ipip
tun 31665 0
Server B
[root@serverB]# modprobe tun
[root@serverB]# lsmod | grep tun
tun 27141 0
IP Forwarding
enable IP forwarding,
[root@xxx ]# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
vi /etc/sysctl.conf
add
net.ipv4.ip_forward = 1
reload sysctl
sysctl -p /etc/sysctl.conf
B) Test basic connectivity
Both should be able to connect one another (use python SimpleHTTPServer + netcat to check connectivity)
Server A
root@serverA# python -m SimpleHTTPServer 8555
Server B
[root@serverB]# nc 172.31.23.254 8555 -v
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.31.23.254:8555.
and vice versa
[root@serverB centos]# python -m SimpleHTTPServer 8556
Serving HTTP on 0.0.0.0 port 8556 ...
root@serverA /e/s/network-scripts# nc 172.31.23.64 8556 -v
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 172.31.23.64:8556.
if they cant connect, the tunnel wont work
C) Creating the Tunnels
Temporary Tunnel
Create tunnel on Server A, assign an IP to this new tunnel interface, here im using a generic 192.168.5.1 for A and 192.168.5.2 for B
root@serverA# ip tunnel add tunnel-b mode ipip remote 172.31.23.64 local 172.31.23.254
root@serverA# ip link set tunnel-b up
root@serverA# ip addr add 192.168.5.1/24 dev tunnel-b
check for new Tunnel interface
root@serverA# ip a
8: tunnel-b@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8981 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 172.31.23.254 peer 172.31.23.64
inet 192.168.5.1/24 scope global tunnel-b
valid_lft forever preferred_lft forever
Do the same on Server B
root@serverB# ip tunnel add tunnel-a mode ipip remote 172.31.23.254 local 172.31.23.64
root@serverB# ip link set tunnel-a up
root@serverB# ip addr add 192.168.5.2/24 dev tunnel-a
check for new IP,
4: tunnel-a@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 8981 qdisc noqueue state UNKNOWN
link/ipip 172.31.23.64 peer 172.31.23.254
inet 192.168.5.2/24 scope global tunnel-b
valid_lft forever preferred_lft forever
try pinging from A to B
serverA> ping 192.168.5.2
should be pingable
Test connectivity as in section B, but change the IP to the IP address of the tunnel interface,
A > B test
root@serverA# python -c 'import BaseHTTPServer as bhs, SimpleHTTPServer as shs; bhs.HTTPServer(("192.168.5.1", 8331), shs.SimpleHTTPRequestHandler).serve_forever()'
root@serverB # nc 192.168.5.1 8331 -v
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.5.1:8331.
B > A test
root@serverB# python -c 'import BaseHTTPServer as bhs, SimpleHTTPServer as shs; bhs.HTTPServer(("192.168.5.2", 8331), shs.SimpleHTTPRequestHandler).serve_forever()'
root@serverA# nc 192.168.5.1 8331 -v
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 192.168.5.2:8331.
Permanent Tunnel
To create a permanent Tunnel that survives Network + Machine restart,
on Server A
root@serverA# vim /etc/sysconfig/network-scripts/ifcfg-tunnel-a
TYPE=IPIP
BOOTPROTO=none
DEVICE=tunnel-b
ONBOOT=yes
MY_INNER_IPADDR=192.168.5.1/30 // Tunnel IP
MY_OUTER_IPADDR=172.31.23.254 // primary local IP address
PEER_OUTER_IPADDR=172.31.23.64 // remote peer primary IP
this tunnel will be in place after a reboot, its managed by Network Manager
on server B
root@serverB# vim /etc/sysconfig/network-scripts/ifcfg-tunnel-b
TYPE=IPIP
BOOTPROTO=none
DEVICE=tunnel-a
ONBOOT=yes
MY_INNER_IPADDR=192.168.5.2/30 // Tunnel IP
MY_OUTER_IPADDR=172.31.23.64 // primary local IP address
PEER_OUTER_IPADDR=172.31.23.254 // remote peer primary IP
bring up this interface on Server B first
ifup tunnel-a
now on server A
ifup tunnel-b
to restart the interface
ifdown tunnel-a
ifup tunnel-a
ping from each tunnel to make sure theyre pingable
check the status of the tunnel
ip tunnel show
D) Add a Route
Gateway Route
you need to have a gateway route in place for back-forth communication between server A and server B, otherwise packets will only flow 1 way (A > B)
On both A and B, this gateway route should already be in place once the tunnel network interface comes up
192.168.1.0 0.0.0.0 255.255.255.252 U 0 0 0 tunnel-a (or tunnel-b)
you can also add it manually,
route add -net 192.168.1.0 netmask 255.255.255.0 dev tunnel-a
Temporary Route
to route IPs from Server A to Server B via Tunnel, add a route (on server A)
root@serverA# ip route add 122.195.129.133 dev tunnel-b
check this Route
root@serverA# ip route get 122.195.129.133
122.195.129.133 via 192.168.5.2 dev tunnel-b src 192.168.5.1
run a Traceroute to see where the IP is going to,
root@serverA# traceroute 122.195.129.133
traceroute to 122.195.129.133 (192.195.129.133), 30 hops max, 60 byte packets
1 * * *
2 * * *
on server B, listen on Tunnel interface for incoming packets,
root@serverB# tcpdump -i tunnel-a
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tunnel-a, link-type RAW (Raw IP), capture size 262144 bytes
15:02:03.217834 IP ip-192-168-5-1.ec2.internal.56547 > 122.195.129.133.33466: UDP, length 32
15:02:03.217839 IP ip-192-168-5-1.ec2.internal.56417 > 122.195.129.133.33467: UDP, length 32
15:02:03.217871 IP ip-192-168-5-1.ec2.internal.42617 > 122.195.129.133.33468: UDP, length 32
on server B, to route from tunnel to p1p1 interface, add a route,
122.195.129.133 192.168.28.17 255.255.255.255 UGH 100 0 0 p1p1
Permanent Route
to keep route permanent, add route file (on server A), this routes your IP to Server B tunnel
root@serverA# vim /etc/sysconfig/network-scripts/route-tunnel-b
122.195.129.133 via 192.168.5.2 metric 50
now enable this route
/etc/sysconfig/network-scripts/ifup-routes route-tunnel-b
check to make sure your Route is now present in routing table
root@serverA# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.31.23.1 0.0.0.0 UG 0 0 0 eth0
172.31.23.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.5.0 0.0.0.0 255.255.255.0 U 0 0 0 tunnel-b
122.195.129.133 192.168.5.2 255.255.255.255 UGH 50 0 0 tunnel-b
run traceroute(server A) and tcpdump (on server B) again to make sure server B is getting the incoming packets
E) Cleaning up Tunnels
using IP command, disable tunnel interface
ip link set dev <tunnel name> down
delete an interface
ip link delete <tunnel name>
Troubleshooting
if routing traffic from server A to server B, sometimes server B will route your outgoing connection as the IP of the tunnel instead of itself, causing the connection to timeout.
ie, doing a TCP Dump on Server B, interface p1p1,
root@serverB:]# tcpdump -i p1p1 | grep 207.17.44.102
23:22:16.157069 IP 192.168.2.1.52896 > 207.17.44.102.41811: Flags [S], seq 804236576, win 14400, options [mss 1440,sackOK,TS val 3076723108 ecr 0,nop,wscale 10], length 0
here the IP shown is the IP of the tunnel interface, this request will not work because the endpoint does not recognize the tunnel IP.
in this case, you must change the packet header of the outgoing packet (on server B) to the IP of itself, not the IP of the Tunnel by Destination
root@serverB> iptables -t nat -A POSTROUTING -d 207.17.44.102 -j MASQUERADE
or by Source
root@serverB> iptables -t nat -A POSTROUTING -p tcp -s 192.168.5.1 -j MASQUERADE
in this example, 192.168.38.21 is the IP of the p1p1 interface on server B
now the connection will work,
[23:22 root@serverB:]# tcpdump -i p1p1 | grep 207.17.44.102
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on p1p1, link-type EN10MB (Ethernet), capture size 262144 bytes
23:28:10.299134 IP serverB.59888 > 207.17.44.102.41811: Flags [F.], seq 19064846, ack 4239206719, win 15, options [nop,nop,TS val 3077077251 ecr 238689707], length 0