Interarea TE tunnels
Interarea TE tunnels
In MPLS TE Basics article, we examined how link attributes are gathered, how a path is computed based on user-defined constraints and how it is signaled. One of the limitations is that the TE LSP path is computed for single IGP area only due to lack of visibility of the topology in other IGP areas. In OSPF, the ABR hides the topology between two areas. This causes a problem in large networks where multiple IGP areas are deployed, or, sometimes the network spans multiple service providers. This is often called Interdomain traffic engineering.
To understand RSVP-TE signaling in interdomain TE, we will use figure 1. This solution applies equally to regular MPLS TE and DiffServ TE. The network spans three OSPF areas and a MPLS TE tunnel must be setup from PE1 destined for PE2. PE1 and PE2 reside in different areas and hence PE1 does not have required information to compute an end-to-end path satisfying TE constraints. Once a path is computed, signaling is done as normal even across areas or ASs.
To setup a RSVP-signaled TE LSP, three steps are completed:
1. Gathering TE information
The link information is carried exactly in the same way as in regular MPLS TE. OSPF Opaque LSA 10 (TE LSAs) are used to carry link-specific information like link type, maximum reservable bandwidth, unreserved bandwidth, TE metric, attribute flags, etc.
2. Path Setup
Path Setup is discussed before path computation because it is crucial that it be known how a path must be setup. For interdomain TE, there are 3 methods to setup an LSP:
- Contiguous LSP: In this case, end-to-end LSP is setup between PE1 and PE2 using hop-by-hop signaling between adjacent neighbors. This method is discussed here.
- LSP Stitching: Smaller TE LSPs are built in different areas (domains) and are stitched together to form an end-to-end LSP. There is a 1:1 relationship between each smaller LSPs mapped together.
- LSP Nesting: This is also called Hierarchical LSP solution. The end-to-end LSP is nested (tunneled) inside an LSP that has per-domain scope. In figure 1, an LSP is setup in Area 0 that acts as a container (tunnel) for other LSPs originating from Area 1 (or Area 2). This LSP is called Forwarding Adjacency (FA) LSP. More than one transit LSP can be nested within the FA LSP. Thus, nesting uses 1:N mapping between the FA LSP and transit LSPs.
3. Path Computation
For contiguous LSP, the exit points out of the areas (domains) must be known. Thus, the headend router (PE1) is not aware of the entire path of the LSP. Instead, the path to the next border router (ABR1) is computed, added to the ERO and the LSP is signaled. Thus the path is computed per-domain.
In per-domain path computation, a path is computed separately within each domain, from one border router to another. For this, the IP addresses of the border routers must be known to the LSP headend. The path is defined explicitly at the headend because the headend is not able to compute the end-to-end path dynamically. The border routers are configured as Loose Hops, meaning, the path to these border routers (ABR1) can be dynamically found. The path to the next border router (ABR2) is computed by this border router (ABR1) and added to ERO object. This is called ERO Expansion. Hence, each border router runs cSPF to compute a path, to the next border router, that satisfies the user-constraints.
This can be configured on PE1 router as below. Notice the ABRs are configured as loose hops.
Configuring Interarea TE tunnel on PE1 router
interface tunnel 10
ip unnumbered Loopback 0
tunnel mode mpls traffic-eng
tunnel destination 6.6.6.6
tunnel mpls traffic-eng bandwidth 5000
tunnel mpls traffic-eng path-option 1 explicit name PATH
tunnel mpls traffic-eng record-route
!
ip explicit-path name PATH
next-address loose 2.2.2.2 ! ABR1
next-address loose 3.3.3.3 ! ABR2
!
The following debug (debug ip rsvp dump-messages path command) outputs show the content of ERO object in PATH messages when PE1 router tries to send a reservation of 50Mbps.
PATH message advertised from PE1 router
*Aug 22 16:38:26.499: Outgoing Path:
*Aug 22 16:38:26.499: version:1 flags:0000 cksum:8EAC ttl:255 reserved:0 length:204
*Aug 22 16:38:26.503: SESSION type 7 length 16:
*Aug 22 16:38:26.503: Tun Dest: 6.6.6.6 Tun ID: 10 Ext Tun ID: 1.1.1.1
*Aug 22 16:38:26.503: HOP type 1 length 12:
*Aug 22 16:38:26.503: Hop Addr: 10.12.1.1 LIH: 0xB2000408
*Aug 22 16:38:26.507: TIME_VALUES type 1 length 8 :
*Aug 22 16:38:26.507: Refresh Period (msec): 30000
*Aug 22 16:38:26.507: EXPLICIT_ROUTE type 1 length 28:
*Aug 22 16:38:26.507: 10.12.1.2 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:38:26.511: 2.2.2.2 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:38:26.511: 3.3.3.3 (Loose IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:38:26.511: LABEL_REQUEST type 1 length 8 :
*Aug 22 16:38:26.511: Layer 3 protocol ID: 2048
*Aug 22 16:38:26.511: SESSION_ATTRIBUTE type 7 length 16:
*Aug 22 16:38:26.515: Setup Prio: 7, Holding Prio: 7
*Aug 22 16:38:26.515: Flags: (0x4) SE Style
*Aug 22 16:38:26.515: Session Name: PE1_t10
*Aug 22 16:38:26.515: SENDER_TEMPLATE type 7 length 12:
*Aug 22 16:38:26.519: Tun Sender: 1.1.1.1 LSP ID: 891
*Aug 22 16:38:26.519: SENDER_TSPEC type 2 length 36:
*Aug 22 16:38:26.519: version=0, length in words=7
*Aug 22 16:38:26.519: Token bucket fragment (service_id=1, length=6 words
*Aug 22 16:38:26.523: parameter id=127, flags=0, parameter length=5
*Aug 22 16:38:26.523: average rate=6250000 bytes/sec, burst depth=1000 bytes
*Aug 22 16:38:26.523: peak rate =6250000 bytes/sec
*Aug 22 16:38:26.523: min unit=0 bytes, max pkt size=2147483647 bytes
*Aug 22 16:38:26.523: ADSPEC type 2 length 48:
*Aug 22 16:38:26.527: version=0 length in words=10
*Aug 22 16:38:26.527: General Parameters break bit=0 service length=8
*Aug 22 16:38:26.527: IS Hops:1
*Aug 22 16:38:26.527: Minimum Path Bandwidth (bytes/sec):12500000
*Aug 22 16:38:26.531: Path Latency (microseconds):0
*Aug 22 16:38:26.531: Path MTU:1500
*Aug 22 16:38:26.531: Controlled Load Service break bit=0 service length=0
*Aug 22 16:38:26.531: RECORD_ROUTE type 1 length 12:
*Aug 22 16:38:26.531: 10.12.1.1/32, Flags:0x0 (No Local Protection)
Notice the ERO object. Although, ABR1 (2.2.2.2) is configured as loose hop, the ERO carries it as a strict hop. The reason is that PE1 is able to compute a path to ABR1 itself. Also note that ABR2 (3.3.3.3) is carried as loose hop. This is because PE1 has no link information of the path to 3.3.3.3. Thus, ABR1 is assigned the task to compute a path to ABR2.
PATH messages on ABR1 router
!--- ABR1 receives a PATH message from PE1 router
*Aug 22 16:39:32.299: Incoming Path:
*Aug 22 16:39:32.303: version:1 flags:0000 cksum:8EAC ttl:255 reserved:0 length:204
*Aug 22 16:39:32.303: SESSION type 7 length 16:
*Aug 22 16:39:32.307: Tun Dest: 6.6.6.6 Tun ID: 10 Ext Tun ID: 1.1.1.1
*Aug 22 16:39:32.307: HOP type 1 length 12:
*Aug 22 16:39:32.307: Hop Addr: 10.12.1.1 LIH: 0xB2000408
*Aug 22 16:39:32.307: TIME_VALUES type 1 length 8 :
*Aug 22 16:39:32.311: Refresh Period (msec): 30000
*Aug 22 16:39:32.311: EXPLICIT_ROUTE type 1 length 28:
*Aug 22 16:39:32.311: 10.12.1.2 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:39:32.311: 2.2.2.2 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:39:32.315: 3.3.3.3 (Loose IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:39:32.315: LABEL_REQUEST type 1 length 8 :
*Aug 22 16:39:32.315: Layer 3 protocol ID: 2048
*Aug 22 16:39:32.315: SESSION_ATTRIBUTE type 7 length 16:
*Aug 22 16:39:32.315: Setup Prio: 7, Holding Prio: 7
*Aug 22 16:39:32.319: Flags: (0x4) SE Style
*Aug 22 16:39:32.319: Session Name: PE1_t10
*Aug 22 16:39:32.319: SENDER_TEMPLATE type 7 length 12:
*Aug 22 16:39:32.319: Tun Sender: 1.1.1.1 LSP ID: 891
*Aug 22 16:39:32.323: SENDER_TSPEC type 2 length 36:
*Aug 22 16:39:32.323: version=0, length in words=7
*Aug 22 16:39:32.323: Token bucket fragment (service_id=1, length=6 words
*Aug 22 16:39:32.323: parameter id=127, flags=0, parameter length=5
*Aug 22 16:39:32.327: average rate=6250000 bytes/sec, burst depth=1000 bytes
*Aug 22 16:39:32.327: peak rate =6250000 bytes/sec
*Aug 22 16:39:32.327: min unit=0 bytes, max pkt size=2147483647 bytes
*Aug 22 16:39:32.327: ADSPEC type 2 length 48:
*Aug 22 16:39:32.327: version=0 length in words=10
*Aug 22 16:39:32.331: General Parameters break bit=0 service length=8
*Aug 22 16:39:32.331: IS Hops:1
*Aug 22 16:39:32.331: Minimum Path Bandwidth (bytes/sec):12500000
*Aug 22 16:39:32.331: Path Latency (microseconds):0
*Aug 22 16:39:32.335: Path MTU:1500
*Aug 22 16:39:32.335: Controlled Load Service break bit=0 service length=0
*Aug 22 16:39:32.335: RECORD_ROUTE type 1 length 12:
*Aug 22 16:39:32.335: 10.12.1.1/32, Flags:0x0 (No Local Protection)
!--- ABR1 sends out a PATH message to ABR2 router
*Aug 22 16:39:32.375: Outgoing Path:
*Aug 22 16:39:32.375: version:1 flags:0000 cksum:D280 ttl:254 reserved:0 length:204
*Aug 22 16:39:32.379: SESSION type 7 length 16:
*Aug 22 16:39:32.379: Tun Dest: 6.6.6.6 Tun ID: 10 Ext Tun ID: 1.1.1.1
*Aug 22 16:39:32.379: HOP type 1 length 12:
*Aug 22 16:39:32.379: Hop Addr: 10.23.1.1 LIH: 0xE8000409
*Aug 22 16:39:32.383: TIME_VALUES type 1 length 8 :
*Aug 22 16:39:32.383: Refresh Period (msec): 30000
*Aug 22 16:39:32.383: EXPLICIT_ROUTE type 1 length 20:
*Aug 22 16:39:32.383: 10.23.1.2 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:39:32.387: 3.3.3.3 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 16:39:32.387: LABEL_REQUEST type 1 length 8 :
*Aug 22 16:39:32.387: Layer 3 protocol ID: 2048
*Aug 22 16:39:32.387: SESSION_ATTRIBUTE type 7 length 16:
*Aug 22 16:39:32.387: Setup Prio: 7, Holding Prio: 7
*Aug 22 16:39:32.391: Flags: (0x4) SE Style
*Aug 22 16:39:32.391: Session Name: PE1_t10
*Aug 22 16:39:32.391: SENDER_TEMPLATE type 7 length 12:
*Aug 22 16:39:32.391: Tun Sender: 1.1.1.1 LSP ID: 891
*Aug 22 16:39:32.395: SENDER_TSPEC type 2 length 36:
*Aug 22 16:39:32.395: version=0, length in words=7
*Aug 22 16:39:32.395: Token bucket fragment (service_id=1, length=6 words
*Aug 22 16:39:32.395: parameter id=127, flags=0, parameter length=5
*Aug 22 16:39:32.395: average rate=6250000 bytes/sec, burst depth=1000 bytes
*Aug 22 16:39:32.399: peak rate =6250000 bytes/sec
*Aug 22 16:39:32.399: min unit=0 bytes, max pkt size=2147483647 bytes
*Aug 22 16:39:32.399: ADSPEC type 2 length 48:
*Aug 22 16:39:32.399: version=0 length in words=10
*Aug 22 16:39:32.403: General Parameters break bit=0 service length=8
*Aug 22 16:39:32.403: IS Hops:2
*Aug 22 16:39:32.403: Minimum Path Bandwidth (bytes/sec):12500000
*Aug 22 16:39:32.403: Path Latency (microseconds):0
*Aug 22 16:39:32.403: Path MTU:1500
*Aug 22 16:39:32.407: Controlled Load Service break bit=0 service length=0
*Aug 22 16:39:32.407: RECORD_ROUTE type 1 length 20:
*Aug 22 16:39:32.407: 10.23.1.1/32, Flags:0x0 (No Local Protection)
*Aug 22 16:39:32.411: 10.12.1.1/32, Flags:0x0 (No Local Protection)
Notice that ABR1 puts all IP addresses as strict hops to ABR2 router.
PATH messages on ABR2 router
!--- ABR2 receives a PATH message from ABR1 router
*Aug 22 17:16:04.163: Incoming Path:
*Aug 22 17:16:04.163: version:1 flags:0000 cksum:D280 ttl:254 reserved:0 length:204
*Aug 22 17:16:04.167: SESSION type 7 length 16:
*Aug 22 17:16:04.167: Tun Dest: 6.6.6.6 Tun ID: 10 Ext Tun ID: 1.1.1.1
*Aug 22 17:16:04.167: HOP type 1 length 12:
*Aug 22 17:16:04.171: Hop Addr: 10.23.1.1 LIH: 0xE8000409
*Aug 22 17:16:04.171: TIME_VALUES type 1 length 8 :
*Aug 22 17:16:04.171: Refresh Period (msec): 30000
*Aug 22 17:16:04.171: EXPLICIT_ROUTE type 1 length 20:
*Aug 22 17:16:04.175: 10.23.1.2 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 17:16:04.175: 3.3.3.3 (Strict IPv4 Prefix, 8 bytes, /32)
*Aug 22 17:16:04.175: LABEL_REQUEST type 1 length 8 :
*Aug 22 17:16:04.175: Layer 3 protocol ID: 2048
*Aug 22 17:16:04.175: SESSION_ATTRIBUTE type 7 length 16:
*Aug 22 17:16:04.179: Setup Prio: 7, Holding Prio: 7
*Aug 22 17:16:04.179: Flags: (0x4) SE Style
*Aug 22 17:16:04.179: Session Name: PE1_t10
*Aug 22 17:16:04.179: SENDER_TEMPLATE type 7 length 12:
*Aug 22 17:16:04.183: Tun Sender: 1.1.1.1 LSP ID: 891
*Aug 22 17:16:04.183: SENDER_TSPEC type 2 length 36:
*Aug 22 17:16:04.183: version=0, length in words=7
*Aug 22 17:16:04.183: Token bucket fragment (service_id=1, length=6 words
*Aug 22 17:16:04.187: parameter id=127, flags=0, parameter length=5
*Aug 22 17:16:04.187: average rate=6250000 bytes/sec, burst depth=1000 bytes
*Aug 22 17:16:04.187: peak rate =6250000 bytes/sec
*Aug 22 17:16:04.187: min unit=0 bytes, max pkt size=2147483647 bytes
*Aug 22 17:16:04.187: ADSPEC type 2 length 48:
*Aug 22 17:16:04.191: version=0 length in words=10
*Aug 22 17:16:04.191: General Parameters break bit=0 service length=8
*Aug 22 17:16:04.191: IS Hops:2
*Aug 22 17:16:04.191: Minimum Path Bandwidth (bytes/sec):12500000
*Aug 22 17:16:04.195: Path Latency (microseconds):0
*Aug 22 17:16:04.195: Path MTU:1500
*Aug 22 17:16:04.195: Controlled Load Service break bit=0 service length=0
*Aug 22 17:16:04.195: RECORD_ROUTE type 1 length 20:
*Aug 22 17:16:04.195: 10.23.1.1/32, Flags:0x0 (No Local Protection)
*Aug 22 17:16:04.199: 10.12.1.1/32, Flags:0x0 (No Local Protection)
!--- ABR2 sends out a PATH message to PE2 router
*Aug 22 17:16:04.239: Outgoing Path:
*Aug 22 17:16:04.239: version:1 flags:0000 cksum:988E ttl:253 reserved:0 length:192
*Aug 22 17:16:04.243: SESSION type 7 length 16:
*Aug 22 17:16:04.243: Tun Dest: 6.6.6.6 Tun ID: 10 Ext Tun ID: 1.1.1.1
*Aug 22 17:16:04.243: HOP type 1 length 12:
*Aug 22 17:16:04.243: Hop Addr: 10.36.1.1 LIH: 0x5E000409
*Aug 22 17:16:04.247: TIME_VALUES type 1 length 8 :
*Aug 22 17:16:04.247: Refresh Period (msec): 30000
*Aug 22 17:16:04.247: LABEL_REQUEST type 1 length 8 :
*Aug 22 17:16:04.247: Layer 3 protocol ID: 2048
*Aug 22 17:16:04.251: SESSION_ATTRIBUTE type 7 length 16:
*Aug 22 17:16:04.251: Setup Prio: 7, Holding Prio: 7
*Aug 22 17:16:04.251: Flags: (0x4) SE Style
*Aug 22 17:16:04.251: Session Name: PE1_t10
*Aug 22 17:16:04.251: SENDER_TEMPLATE type 7 length 12:
*Aug 22 17:16:04.255: Tun Sender: 1.1.1.1 LSP ID: 891
*Aug 22 17:16:04.255: SENDER_TSPEC type 2 length 36:
*Aug 22 17:16:04.255: version=0, length in words=7
*Aug 22 17:16:04.255: Token bucket fragment (service_id=1, length=6 words
*Aug 22 17:16:04.259: parameter id=127, flags=0, parameter length=5
*Aug 22 17:16:04.259: average rate=6250000 bytes/sec, burst depth=1000 bytes
*Aug 22 17:16:04.259: peak rate =6250000 bytes/sec
*Aug 22 17:16:04.259: min unit=0 bytes, max pkt size=2147483647 bytes
*Aug 22 17:16:04.259: ADSPEC type 2 length 48:
*Aug 22 17:16:04.263: version=0 length in words=10
*Aug 22 17:16:04.263: General Parameters break bit=0 service length=8
*Aug 22 17:16:04.263: IS Hops:3
*Aug 22 17:16:04.263: Minimum Path Bandwidth (bytes/sec):12500000
*Aug 22 17:16:04.267: Path Latency (microseconds):0
*Aug 22 17:16:04.267: Path MTU:1500
*Aug 22 17:16:04.267: Controlled Load Service break bit=0 service length=0
*Aug 22 17:16:04.267: RECORD_ROUTE type 1 length 28:
*Aug 22 17:16:04.267: 10.36.1.1/32, Flags:0x0 (No Local Protection)
*Aug 22 17:16:04.271: 10.23.1.1/32, Flags:0x0 (No Local Protection)
*Aug 22 17:16:04.275: 10.12.1.1/32, Flags:0x0 (No Local Protection)
Notice the incoming PATH message from ABR1 router has an ERO object but the outgoing PATH message does not carry any ERO object. ABR2 router computes a path to the destination dynamically.
The following output shows the MPLS TE tunnel established between PE1 and PE2 router. Notice the asterisk (*) on 3.3.3.3 (ABR2). This indicates it is the loose hop. The path taken is the shortest path - PE1-ABR1-ABR2-PE2. This can be identified from Record Route object. Also note that the path weight for TE tunnel is 1. This is actually the path-weight to ABR1 router.
MPLS TE tunnel on PE1 router
PE1# show mpls traffic-eng tunnels tunnel 10
Name: PE1_t10 (Tunnel10) Destination: 6.6.6.6
Status:
Admin: up Oper: up Path: valid Signalling: connected
path option 1, type explicit PATH (Basis for Setup, path weight 1)
Config Parameters:
Bandwidth: 50000 kbps (Global) Priority: 7 7 Affinity: 0x0/0xFFFF
Metric Type: TE (default)
AutoRoute announce: disabled LockDown: disabled Loadshare: 50000 bw-based
auto-bw: disabled
Active Path Option Parameters:
State: explicit path option 1 is active
BandwidthOverride: disabled LockDown: disabled Verbatim: disabled
InLabel : -
OutLabel : FastEthernet1/0, 16
Next Hop : 10.12.1.2
RSVP Signalling Info:
Src 1.1.1.1, Dst 6.6.6.6, Tun_Id 10, Tun_Instance 891
RSVP Path Info:
My Address: 10.12.1.1
Explicit Route: 10.12.1.2 2.2.2.2 3.3.3.3*
Record Route:
Tspec: ave rate=50000 kbits, burst=1000 bytes, peak rate=50000 kbits
RSVP Resv Info:
Record Route: 10.23.1.1 10.36.1.1 10.36.1.2
Fspec: ave rate=50000 kbits, burst=1000 bytes, peak rate=50000 kbits
Shortest Unconstrained Path Info:
Path Weight: UNKNOWN
Explicit Route: UNKNOWN
History:
Tunnel:
Time since created: 2 days, 52 minutes
Time since path change: 4 minutes, 52 seconds
Number of LSP IDs (Tun_Instances) used: 891
Current LSP: [ID: 891]
Uptime: 4 minutes, 52 seconds
Prior LSP: [ID: 885]
ID: path option unknown
Removal Trigger: tunnel shutdown
Yet another method to compute a path for interdomain TE is called path computation element (PCE). It is possible to offload the path computation to an entity which has complete information about the topology and can deliver the necessary information on demand. The PCE can be a module on a router or a completely separate entity that the router can communicate with. The PCE services path computation requests. So the router can request a path to the next loose hop from the PCE rather than running cSPF on the router itself. PCE is supported in Cisco IOS XR.
Crankback
In regular MPLS TE within a single IGP area, RSVP PATHErr messages would provide the necessary information to the headend router in case of LSP failure, may be due to unavailability of resources on a particular link. However, this information would make little sense to the headend router if the link belongs to a different area than the headend router.
For example, if ABR2-PE2 is unavailable, sending an error message back to PE1 router has no importance to PE1. Instead, this information would make more sense to ABR2 router, in which case, it will try to find an alternative path within area 2 that avoids ABR2-PE2 link.
In this case, ABR2 will be able to find an alternative path through ABR3 router. However, if ABR2-ABR3 link was also down, ABR2 will not be able to find an alternative path. So the error is forwarded one step back (upstream router) to ABR1 router. So, ABR1 router is assigned the task to find a path avoiding ABR2 router.
In this way, computation is cranked back one step at a time, away from the failure point. This process is called crankback.
Note: Cisco IOS does not support crankback. So the path error message is forwarded all the way back to the headend router. After 2 minutes, the headend router tries to evaluate the next path-option if configured.