DMVPN with dual ISPs
DMVPN with dual ISPs
This article demonstrates DMVPN with 2 ISPs where the Hub has dual ISP connections. Also, each Spoke router is connected to a separate ISP.
Configuration
PE11, PE12, PE21 and PE22 routers have a very simple and similar VRF configuration. There is an eBGP connection between PE and CE routers. A sample configuration on PE11 follows.
VRF Configuration on PE11 router
mpls label protocol ldpmpls ldp router-id Loopback 0 force!ip vrf CUST rd 1:1 route-target both 1:1!interface Loopback 0 ip address 1.1.1.1 255.255.255.255 ip ospf 1 area 0!interface Serial 0/0 ip vrf forwarding CUST ip address 172.11.1.2 255.255.255.0!router bgp 100 neighbor 2.2.2.2 remote-as 100 neighbor 2.2.2.2 update-source Loopback 0 ! address-family vpnv4 neighbor 2.2.2.2 activate exit-address-family ! address-family ipv4 vrf CUST neighbor 172.11.1.1 remote-as 65001 neighbor 172.11.1.1 activate redistribute connected exit-address-family!PE100 and PE200 routers act as Route-Reflectors (RR) for their respective ASes. To exchange VPNv4 routes, they run MP-eBGP connection across the two physical connections- one preferred over the other achieved by setting higher local-preference value for prefixes learnt via one connection.
PE100 Configuration
interface Loopback 0 ip address 2.2.2.2 255.255.255.255 ip ospf 1 area 0!router ospf 1 passive-interface FastEthernet1/0 passive-interface FastEthernet2/0 network 10.100.1.0 0.0.0.255 area 0 network 10.100.2.0 0.0.0.255 area 0!router bgp 100 no bgp route-target default filter ! Accept all VPNv4 routes neighbor 1.1.1.1 remote-as 100 ! PE11 router neighbor 1.1.1.1 update-source Loopback 0 neighbor 3.3.3.3 remote-as 100 ! PE12 router neighbor 3.3.3.3 update-source Loopback 0 neighbor 10.100.1.2 remote-as 200 neighbor 10.100.1.2 send-label ! eBGP connection with PE200 router. The "send-label" keyword allows the router to send label with BGP VPNv4 prefixes neighbor 10.100.2.2 remote-as 200 neighbor 10.100.2.2 send-label ! address-family vpnv4 neighbor 1.1.1.1 activate neighbor 1.1.1.1 route-reflector-client neighbor 1.1.1.1 next-hop-self neighbor 3.3.3.3 activate neighbor 3.3.3.3 route-reflector-client neighbor 3.3.3.3 next-hop-self neighbor 10.100.1.2 activate neighbor 10.100.1.2 route-map LOCAL_PREF in ! Set Local-Preference to 150 for all incoming VPNv4 prefixes from this neighbor neighbor 10.100.2.2 activate!route-map LOCAL_PREF permit 10 set local-preference 150!Configuration on PE200 router is very similar to PE100 router.
PE200 Configuration
interface Loopback 0 ip address 5.5.5.5 255.255.255.255 ip ospf 2 area 0!router ospf 2 passive-interface FastEthernet1/0 passive-interface FastEthernet2/0 network 10.100.1.0 0.0.0.255 area 0 network 10.100.2.0 0.0.0.255 area 0!router bgp 200 no bgp route-target default filter neighbor 4.4.4.4 remote-as 200 ! PE21 router neighbor 4.4.4.4 update-source Loopback 0 neighbor 6.6.6.6 remote-as 200 ! PE22 router neighbor 6.6.6.6 update-source Loopback 0 neighbor 10.100.1.1 remote-as 100 ! eBGP connection to PE100 neighbor 10.100.1.1 send-label neighbor 10.100.2.1 remote-as 100 neighbor 10.100.2.1 send-label ! address-family vpnv4 neighbor 4.4.4.4 activate neighbor 4.4.4.4 route-reflector-client neighbor 4.4.4.4 next-hop-self neighbor 6.6.6.6 activate neighbor 6.6.6.6 route-reflector-client neighbor 6.6.6.6 next-hop-self neighbor 10.100.1.1 activate neighbor 10.100.1.1 route-map LOCAL_PREF in neighbor 10.100.2.1 activate!route-map LOCAL_PREF permit 10 set local-preference 150!The following output shows the VPNv4 prefixes learnt by PE100 and PE200 routers. Note the prefixes with higher local-preference value are preferred.
VPNv4 Prefixes on PE100 and PE200
PE100# show ip bgp vpnv4 all summaryBGP router identifier 2.2.2.2, local AS number 100BGP table version is 13, main routing table version 134 network entries using 560 bytes of memory6 path entries using 408 bytes of memory4/2 BGP path/bestpath attribute entries using 496 bytes of memory1 BGP AS-PATH entries using 24 bytes of memory1 BGP extended community entries using 24 bytes of memory0 BGP route-map cache entries using 0 bytes of memory0 BGP filter-list cache entries using 0 bytes of memoryBitfield cache entries: current 1 (at peak 3) using 32 bytes of memoryBGP using 1544 total bytes of memoryBGP activity 4/0 prefixes, 10/4 paths, scan interval 15 secsNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd1.1.1.1 4 100 41 47 13 0 0 00:36:36 13.3.3.3 4 100 41 47 13 0 0 00:35:42 110.100.1.2 4 200 51 52 13 0 0 00:15:47 210.100.2.2 4 200 40 41 13 0 0 00:15:43 2PE100# show ip bgp vpnv4 allBGP table version is 13, local router ID is 2.2.2.2Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S StaleOrigin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight PathRoute Distinguisher: 1:1*>i172.11.1.0/24 1.1.1.1 0 100 0 ?* 172.12.1.0/24 10.100.2.2 0 200 ?*> 10.100.1.2 150 0 200 ?*>i172.22.1.0/24 3.3.3.3 0 100 0 ?* 172.33.1.0/24 10.100.2.2 0 200 ?*> 10.100.1.2 150 0 200 ?PE200# show ip bgp vpnv4 allBGP table version is 13, local router ID is 5.5.5.5Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S StaleOrigin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight PathRoute Distinguisher: 1:1* 172.11.1.0/24 10.100.2.1 0 100 ?*> 10.100.1.1 150 0 100 ?*>i172.12.1.0/24 4.4.4.4 0 100 0 ?* 172.22.1.0/24 10.100.2.1 0 100 ?*> 10.100.1.1 150 0 100 ?*>i172.33.1.0/24 6.6.6.6 0 100 0 ?DMVPN Hub and Spoke Configuration
Since the Hub router has 2 connections to the ISP, two tunnel interfaces are created on each Hub and Spoke routers. However, only one tunnel is operational at any time.
DMVPN Hub Configuration
crypto isakmp policy 10 encryption 3des authentication pre-share hash md5 group 2!crypto isakmp key 0 cisco address 0.0.0.0 0.0.0.0!crypto ipsec transform-set TS esp-3des esp-md5-hmac mode transport!crypto ipsec profile DMVPN set transform-set TS!interface Tunnel10 ! Primary Tunnel ip address 192.168.100.1 255.255.255.0 no ip next-hop-self eigrp 1020 ip nhrp map multicast dynamic ip nhrp network-id 123 no ip split-horizon eigrp 1020 delay 5 tunnel source Serial1/0 tunnel mode gre multipoint tunnel key 123 tunnel protection ipsec profile DMVPN shared!interface Tunnel20 ! Backup Tunnel ip address 192.168.200.1 255.255.255.0 no ip next-hop-self eigrp 1020 ip nhrp map multicast dynamic ip nhrp network-id 456 no ip split-horizon eigrp 1020 delay 10 tunnel source Serial1/1 tunnel mode gre multipoint tunnel key 456 tunnel protection ipsec profile DMVPN shared!router eigrp 1020 no auto-summary network 192.168.0.0 network 192.168.100.0 network 192.168.200.0!router bgp 65001 neighbor 172.11.1.2 remote-as 100 neighbor 172.11.1.2 filter-list 10 out neighbor 172.12.1.2 remote-as 200 neighbor 172.12.1.2 filter-list 10 out!ip as-path access-list 10 permit ^$ ! Allows only local routes to be advertised!Now, since the Spoke routers have a single connection to the ISP, they share the same interface for both tunnels.
DMVPN Configuration on Spoke1 router
crypto isakmp policy 10 encr 3des hash md5 authentication pre-share group 2crypto isakmp key cisco address 0.0.0.0 0.0.0.0!crypto ipsec transform-set TS esp-3des esp-md5-hmac mode transport!crypto ipsec profile DMVPN set transform-set TS!interface Tunnel10 ip address 192.168.100.2 255.255.255.0 ip nhrp map 192.168.100.1 172.11.1.1 ip nhrp map multicast 172.11.1.1 ip nhrp network-id 123 ip nhrp nhs 192.168.100.1 delay 5 tunnel source Serial1/0 tunnel mode gre multipoint tunnel key 123 tunnel protection ipsec profile DMVPN shared!interface Tunnel20 ip address 192.168.200.2 255.255.255.0 ip nhrp map 192.168.200.1 172.12.1.1 ip nhrp map multicast 172.12.1.1 ip nhrp network-id 456 ip nhrp nhs 192.168.200.1 delay 10 tunnel source Serial1/0 tunnel mode gre multipoint tunnel key 456 tunnel protection ipsec profile DMVPN shared!router bgp 65002 neighbor 172.22.1.2 remote-as 100!DMVPN Configuration on Spoke2 router
crypto isakmp policy 10 encryption 3des hash md5 authentication pre-share group 2crypto isakmp key cisco address 0.0.0.0 0.0.0.0!crypto ipsec transform-set TS esp-3des esp-md5-hmac mode transport!crypto ipsec profile DMVPN set transform-set TS!interface Tunnel10 ip address 192.168.100.3 255.255.255.0 ip nhrp map 192.168.100.1 172.11.1.1 ip nhrp map multicast 172.11.1.1 ip nhrp network-id 123 ip nhrp nhs 192.168.100.1 delay 5 tunnel source Serial1/0 tunnel mode gre multipoint tunnel key 123 tunnel protection ipsec profile DMVPN shared!interface Tunnel20 ip address 192.168.200.3 255.255.255.0 ip nhrp map 192.168.200.1 172.12.1.1 ip nhrp map multicast 172.12.1.1 ip nhrp network-id 456 ip nhrp nhs 192.168.200.1 delay 10 tunnel source Serial1/0 tunnel mode gre multipoint tunnel key 456 tunnel protection ipsec profile DMVPN shared!router bgp 65003 neighbor 172.33.1.2 remote-as 200!The major issue with this setup is that if the primary connection to ISP at Hub site goes down, the tunnel interface (tunnel 10) goes down as well. EIGRP adjacency with Spoke routers fall. However, the Spoke routers still have primary tunnel interface as functional. Hence, a tracking mechanism is required on Spoke routers to track the reachability of the source address of primary tunnel interface. IP SLA with tracking can do the job. Further, this reachability information can be used to administer the functionality of tunnel interfaces on Spoke routers. For example, if primary tunnel interface goes down on Hub, the Spoke routers shut down their primary tunnel interface and bring the secondary tunnel up. This can be done using Embedded Event Manager EEM. Before any IP SLA configuration on Spoke routers, ip sla responder command is required on the Hub router. The configuration is as below:
IP SLA on Hub
ip sla responder!IP SLA + Tracking + EEM on Spoke1 router
track 1 ip sla 1 reachability delay down 180 up 180!ip sla 1 icmp-echo 172.11.1.1 source-interface Serial1/0 timeout 300 threshold 300 frequency 3ip sla schedule 1 life forever start-time now!event manager applet UP_TUNNEL_10 event track 1 state up action 1.0 cli command "enable" action 1.1 cli command "config t" action 1.2 cli command "interface Tunnel20" action 1.3 cli command "shutdown" action 1.4 cli command "interface Tunnel10" action 1.5 cli command "no shutdown" action 1.6 cli command "do clear crypto isakmp"event manager applet SHUT_TUNNEL_10 event track 1 state down action 1.0 cli command "enable" action 1.1 cli command "config t" action 1.2 cli command "interface Tunnel10" action 1.3 cli command "shutdown" action 1.4 cli command "interface Tunnel20" action 1.5 cli command "no shutdown" action 1.6 cli command "do clear crypto isakmp"!IP SLA + Tracking + EEM on Spoke2 router
track 1 ip sla 1 reachability delay down 180 up 180!ip sla 1 icmp-echo 172.12.1.1 source-interface Serial1/0 timeout 300 threshold 300 frequency 3ip sla schedule 1 life forever start-time now!event manager applet UP_TUNNEL_10 event track 1 state up action 1.0 cli command "enable" action 1.1 cli command "config t" action 1.2 cli command "interface Tunnel20" action 1.3 cli command "shutdown" action 1.4 cli command "interface Tunnel10" action 1.5 cli command "no shutdown" action 1.6 cli command "do clear crypto isakmp"event manager applet SHUT_TUNNEL_10 event track 1 state down action 1.0 cli command "enable" action 1.1 cli command "config t" action 1.2 cli command "interface Tunnel10" action 1.3 cli command "shutdown" action 1.4 cli command "interface Tunnel20" action 1.5 cli command "no shutdown" action 1.6 cli command "do clear crypto isakmp"!Verification
Since the primary tunnel interface is UP, all the Spoke routers register their NHRP mappings with the Hub router over primary tunnel interface 10. Also, all internal prefixes are learnt over primary tunnel as EIGRP adjacency is established over that interface.
Hub# show ip nhrp192.168.100.2/32 via 192.168.100.2 Tunnel10 created 00:29:51, expire 01:30:28 Type: dynamic, Flags: unique registered NBMA address: 172.22.1.1192.168.100.3/32 via 192.168.100.3 Tunnel10 created 00:29:48, expire 01:31:00 Type: dynamic, Flags: unique registered used NBMA address: 172.33.1.1Hub# show ip route eigrpD 192.168.1.0/24 [90/25603840] via 192.168.100.2, 00:10:34, Tunnel10D 192.168.2.0/24 [90/25603840] via 192.168.100.3, 00:10:34, Tunnel10Spoke1# show ip route eigrpD 192.168.0.0/24 [90/25603840] via 192.168.100.1, 00:14:19, Tunnel10D 192.168.2.0/24 [90/25605120] via 192.168.100.3, 00:11:08, Tunnel10Spoke2# show ip route eigrpD 192.168.0.0/24 [90/25603840] via 192.168.100.1, 00:11:34, Tunnel10D 192.168.1.0/24 [90/25605120] via 192.168.100.2, 00:11:34, Tunnel10When the source address of primary tunnel interface on the Hub router is unreachable (data-link between Hub and PE11 is down), the tracking object discovers that the Hub is not reachable over Tunnel 10. The Spoke routers shut down Tunnel 10 and bring up Tunnel 20. The Hub router receives NHRP mapping over Tunnel 20. EIGRP adjacency is formed over this interface and routes are exchanged.
Spoke1#*Jul 15 15:42:45.734: %TRACKING-5-STATE: 1 ip sla 1 reachability Up->Down*Jul 15 15:42:45.982: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF*Jul 15 15:42:46.434: %SYS-5-CONFIG_I: Configured from console by vty0*Jul 15 15:42:47.898: %LINK-5-CHANGED: Interface Tunnel10, changed state to administratively down*Jul 15 15:42:48.186: %LINK-3-UPDOWN: Interface Tunnel20, changed state to up*Jul 15 15:42:48.286: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON*Jul 15 15:42:48.910: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel10, changed state to down*Jul 15 15:42:49.194: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel20, changed state to up*Jul 15 15:42:52.614: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 1020: Neighbor 192.168.200.1 (Tunnel20) is up: new adjacencySpoke1# show trackTrack 1 IP SLA 1 reachability Reachability is Down 99 changes, last change 00:00:33 Delay up 180 secs, down 180 secs Latest operation return code: Timeout Tracked by: EEM applet UP_TUNNEL_10 EEM applet SHUT_TUNNEL_10Hub# show ip nhrp192.168.200.2/32 via 192.168.200.2 Tunnel20 created 00:03:58, expire 01:56:01 Type: dynamic, Flags: unique registered NBMA address: 172.22.1.1192.168.200.3/32 via 192.168.200.3 Tunnel20 created 00:06:54, expire 01:53:05 Type: dynamic, Flags: unique registered NBMA address: 172.33.1.1During Spoke-to-Spoke traffic forwarding, dynamic tunnel is formed via Tunnel 20.
Dynamic Spoke-to-Spoke tunnel
Spoke1# ping 192.168.2.1 source fa 0/0Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 192.168.2.1, timeout is 2 seconds:Packet sent with a source address of 192.168.1.1!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 388/864/1440 msSpoke1# show ip nhrp192.168.200.1/32 via 192.168.200.1 Tunnel20 created 00:04:41, never expire Type: static, Flags: used NBMA address: 172.12.1.1192.168.200.2/32 via 192.168.200.2 Tunnel20 created 00:00:05, expire 01:59:54 Type: dynamic, Flags: router unique local NBMA address: 172.22.1.1 (no-socket)192.168.200.3/32 via 192.168.200.3 Tunnel20 created 00:00:08, expire 01:59:56 Type: dynamic, Flags: router NBMA address: 172.33.1.1