Interarea TE tunnels

Interarea TE tunnels

In MPLS TE Basics article, we examined how link attributes are gathered, how a path is computed based on user-defined constraints and how it is signaled. One of the limitations is that the TE LSP path is computed for single IGP area only due to lack of visibility of the topology in other IGP areas. In OSPF, the ABR hides the topology between two areas. This causes a problem in large networks where multiple IGP areas are deployed, or, sometimes the network spans multiple service providers. This is often called Interdomain traffic engineering.

To understand RSVP-TE signaling in interdomain TE, we will use figure 1. This solution applies equally to regular MPLS TE and DiffServ TE. The network spans three OSPF areas and a MPLS TE tunnel must be setup from PE1 destined for PE2. PE1 and PE2 reside in different areas and hence PE1 does not have required information to compute an end-to-end path satisfying TE constraints. Once a path is computed, signaling is done as normal even across areas or ASs.

To setup a RSVP-signaled TE LSP, three steps are completed:

1. Gathering TE information

The link information is carried exactly in the same way as in regular MPLS TE. OSPF Opaque LSA 10 (TE LSAs) are used to carry link-specific information like link type, maximum reservable bandwidth, unreserved bandwidth, TE metric, attribute flags, etc.

2. Path Setup

Path Setup is discussed before path computation because it is crucial that it be known how a path must be setup. For interdomain TE, there are 3 methods to setup an LSP:

Contiguous LSP: In this case, end-to-end LSP is setup between PE1 and PE2 using hop-by-hop signaling between adjacent neighbors. This method is discussed here.
LSP Stitching: Smaller TE LSPs are built in different areas (domains) and are stitched together to form an end-to-end LSP. There is a 1:1 relationship between each smaller LSPs mapped together.
LSP Nesting: This is also called Hierarchical LSP solution. The end-to-end LSP is nested (tunneled) inside an LSP that has per-domain scope. In figure 1, an LSP is setup in Area 0 that acts as a container (tunnel) for other LSPs originating from Area 1 (or Area 2). This LSP is called Forwarding Adjacency (FA) LSP. More than one transit LSP can be nested within the FA LSP. Thus, nesting uses 1:N mapping between the FA LSP and transit LSPs.

3. Path Computation

For contiguous LSP, the exit points out of the areas (domains) must be known. Thus, the headend router (PE1) is not aware of the entire path of the LSP. Instead, the path to the next border router (ABR1) is computed, added to the ERO and the LSP is signaled. Thus the path is computed per-domain.

In per-domain path computation, a path is computed separately within each domain, from one border router to another. For this, the IP addresses of the border routers must be known to the LSP headend. The path is defined explicitly at the headend because the headend is not able to compute the end-to-end path dynamically. The border routers are configured as Loose Hops, meaning, the path to these border routers (ABR1) can be dynamically found. The path to the next border router (ABR2) is computed by this border router (ABR1) and added to ERO object. This is called ERO Expansion. Hence, each border router runs cSPF to compute a path, to the next border router, that satisfies the user-constraints.

This can be configured on PE1 router as below. Notice the ABRs are configured as loose hops.

Configuring Interarea TE tunnel on PE1 router

interface tunnel 10

 ip unnumbered Loopback 0

 tunnel mode mpls traffic-eng

 tunnel destination 6.6.6.6

 tunnel mpls traffic-eng bandwidth 5000

 tunnel mpls traffic-eng path-option 1 explicit name PATH

 tunnel mpls traffic-eng record-route

ip explicit-path name PATH

 next-address loose 2.2.2.2        ! ABR1

 next-address loose 3.3.3.3        ! ABR2

The following debug (debug ip rsvp dump-messages path command) outputs show the content of ERO object in PATH messages when PE1 router tries to send a reservation of 50Mbps.

PATH message advertised from PE1 router

*Aug 22 16:38:26.499: Outgoing Path:

*Aug 22 16:38:26.499:   version:1 flags:0000 cksum:8EAC ttl:255 reserved:0 length:204

*Aug 22 16:38:26.503:  SESSION              type 7 length 16:

*Aug 22 16:38:26.503:   Tun Dest:   6.6.6.6  Tun ID: 10  Ext Tun ID: 1.1.1.1

*Aug 22 16:38:26.503:  HOP                  type 1 length 12:

*Aug 22 16:38:26.503:   Hop Addr: 10.12.1.1 LIH: 0xB2000408

*Aug 22 16:38:26.507:  TIME_VALUES          type 1 length 8 :

*Aug 22 16:38:26.507:   Refresh Period (msec): 30000

*Aug 22 16:38:26.507:  EXPLICIT_ROUTE       type 1 length 28:

*Aug 22 16:38:26.507:   10.12.1.2 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:38:26.511:   2.2.2.2 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:38:26.511:   3.3.3.3 (Loose IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:38:26.511:  LABEL_REQUEST        type 1 length 8 :

*Aug 22 16:38:26.511:   Layer 3 protocol ID: 2048

*Aug 22 16:38:26.511:  SESSION_ATTRIBUTE    type 7 length 16:

*Aug 22 16:38:26.515:   Setup Prio: 7, Holding Prio: 7

*Aug 22 16:38:26.515:   Flags: (0x4) SE Style

*Aug 22 16:38:26.515:   Session Name: PE1_t10

*Aug 22 16:38:26.515:  SENDER_TEMPLATE      type 7 length 12:

*Aug 22 16:38:26.519:   Tun Sender: 1.1.1.1  LSP ID: 891

*Aug 22 16:38:26.519:  SENDER_TSPEC         type 2 length 36:

*Aug 22 16:38:26.519:   version=0, length in words=7

*Aug 22 16:38:26.519:   Token bucket fragment (service_id=1, length=6 words

*Aug 22 16:38:26.523:    parameter id=127, flags=0, parameter length=5

*Aug 22 16:38:26.523:     average rate=6250000 bytes/sec, burst depth=1000 bytes

*Aug 22 16:38:26.523:     peak rate   =6250000 bytes/sec

*Aug 22 16:38:26.523:     min unit=0 bytes, max pkt size=2147483647 bytes

*Aug 22 16:38:26.523:  ADSPEC               type 2 length 48:

*Aug 22 16:38:26.527:  version=0  length in words=10

*Aug 22 16:38:26.527:  General Parameters  break bit=0  service length=8

*Aug 22 16:38:26.527:                                         IS Hops:1

*Aug 22 16:38:26.527:              Minimum Path Bandwidth (bytes/sec):12500000

*Aug 22 16:38:26.531:                     Path Latency (microseconds):0

*Aug 22 16:38:26.531:                                        Path MTU:1500

*Aug 22 16:38:26.531:  Controlled Load Service  break bit=0  service length=0

*Aug 22 16:38:26.531:  RECORD_ROUTE         type 1 length 12:

*Aug 22 16:38:26.531:   10.12.1.1/32, Flags:0x0 (No Local Protection)

Notice the ERO object. Although, ABR1 (2.2.2.2) is configured as loose hop, the ERO carries it as a strict hop. The reason is that PE1 is able to compute a path to ABR1 itself. Also note that ABR2 (3.3.3.3) is carried as loose hop. This is because PE1 has no link information of the path to 3.3.3.3. Thus, ABR1 is assigned the task to compute a path to ABR2.

PATH messages on ABR1 router

!--- ABR1 receives a PATH message from PE1 router

*Aug 22 16:39:32.299: Incoming Path:

*Aug 22 16:39:32.303:   version:1 flags:0000 cksum:8EAC ttl:255 reserved:0 length:204

*Aug 22 16:39:32.303:  SESSION              type 7 length 16:

*Aug 22 16:39:32.307:   Tun Dest:   6.6.6.6  Tun ID: 10  Ext Tun ID: 1.1.1.1

*Aug 22 16:39:32.307:  HOP                  type 1 length 12:

*Aug 22 16:39:32.307:   Hop Addr: 10.12.1.1 LIH: 0xB2000408

*Aug 22 16:39:32.307:  TIME_VALUES          type 1 length 8 :

*Aug 22 16:39:32.311:   Refresh Period (msec): 30000

*Aug 22 16:39:32.311:  EXPLICIT_ROUTE       type 1 length 28:

*Aug 22 16:39:32.311:   10.12.1.2 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:39:32.311:   2.2.2.2 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:39:32.315:   3.3.3.3 (Loose IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:39:32.315:  LABEL_REQUEST        type 1 length 8 :

*Aug 22 16:39:32.315:   Layer 3 protocol ID: 2048

*Aug 22 16:39:32.315:  SESSION_ATTRIBUTE    type 7 length 16:

*Aug 22 16:39:32.315:   Setup Prio: 7, Holding Prio: 7

*Aug 22 16:39:32.319:   Flags: (0x4) SE Style

*Aug 22 16:39:32.319:   Session Name: PE1_t10

*Aug 22 16:39:32.319:  SENDER_TEMPLATE      type 7 length 12:

*Aug 22 16:39:32.319:   Tun Sender: 1.1.1.1  LSP ID: 891

*Aug 22 16:39:32.323:  SENDER_TSPEC         type 2 length 36:

*Aug 22 16:39:32.323:   version=0, length in words=7

*Aug 22 16:39:32.323:   Token bucket fragment (service_id=1, length=6 words

*Aug 22 16:39:32.323:     parameter id=127, flags=0, parameter length=5

*Aug 22 16:39:32.327:     average rate=6250000 bytes/sec, burst depth=1000 bytes

*Aug 22 16:39:32.327:     peak rate   =6250000 bytes/sec

*Aug 22 16:39:32.327:     min unit=0 bytes, max pkt size=2147483647 bytes

*Aug 22 16:39:32.327:  ADSPEC               type 2 length 48:

*Aug 22 16:39:32.327:  version=0  length in words=10

*Aug 22 16:39:32.331:  General Parameters  break bit=0  service length=8

*Aug 22 16:39:32.331:                                         IS Hops:1

*Aug 22 16:39:32.331:       Minimum Path Bandwidth (bytes/sec):12500000

*Aug 22 16:39:32.331:                     Path Latency (microseconds):0

*Aug 22 16:39:32.335:                                        Path MTU:1500

*Aug 22 16:39:32.335:  Controlled Load Service  break bit=0  service length=0

*Aug 22 16:39:32.335:  RECORD_ROUTE         type 1 length 12:

*Aug 22 16:39:32.335:   10.12.1.1/32, Flags:0x0 (No Local Protection)

!--- ABR1 sends out a PATH message to ABR2 router

*Aug 22 16:39:32.375: Outgoing Path:

*Aug 22 16:39:32.375:   version:1 flags:0000 cksum:D280 ttl:254 reserved:0 length:204

*Aug 22 16:39:32.379:  SESSION              type 7 length 16:

*Aug 22 16:39:32.379:   Tun Dest:   6.6.6.6  Tun ID: 10  Ext Tun ID: 1.1.1.1

*Aug 22 16:39:32.379:  HOP                  type 1 length 12:

*Aug 22 16:39:32.379:   Hop Addr: 10.23.1.1 LIH: 0xE8000409

*Aug 22 16:39:32.383:  TIME_VALUES          type 1 length 8 :

*Aug 22 16:39:32.383:   Refresh Period (msec): 30000

*Aug 22 16:39:32.383:  EXPLICIT_ROUTE       type 1 length 20:

*Aug 22 16:39:32.383:   10.23.1.2 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:39:32.387:   3.3.3.3 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 16:39:32.387:  LABEL_REQUEST        type 1 length 8 :

*Aug 22 16:39:32.387:   Layer 3 protocol ID: 2048

*Aug 22 16:39:32.387:  SESSION_ATTRIBUTE    type 7 length 16:

*Aug 22 16:39:32.387:   Setup Prio: 7, Holding Prio: 7

*Aug 22 16:39:32.391:   Flags: (0x4) SE Style

*Aug 22 16:39:32.391:   Session Name: PE1_t10

*Aug 22 16:39:32.391:  SENDER_TEMPLATE      type 7 length 12:

*Aug 22 16:39:32.391:   Tun Sender: 1.1.1.1  LSP ID: 891

*Aug 22 16:39:32.395:  SENDER_TSPEC         type 2 length 36:

*Aug 22 16:39:32.395:   version=0, length in words=7

*Aug 22 16:39:32.395:   Token bucket fragment (service_id=1, length=6 words

*Aug 22 16:39:32.395:     parameter id=127, flags=0, parameter length=5

*Aug 22 16:39:32.395:     average rate=6250000 bytes/sec, burst depth=1000 bytes

*Aug 22 16:39:32.399:     peak rate   =6250000 bytes/sec

*Aug 22 16:39:32.399:     min unit=0 bytes, max pkt size=2147483647 bytes

*Aug 22 16:39:32.399:  ADSPEC               type 2 length 48:

*Aug 22 16:39:32.399:  version=0  length in words=10

*Aug 22 16:39:32.403:  General Parameters  break bit=0  service length=8

*Aug 22 16:39:32.403:                                         IS Hops:2

*Aug 22 16:39:32.403:              Minimum Path Bandwidth (bytes/sec):12500000

*Aug 22 16:39:32.403:                     Path Latency (microseconds):0

*Aug 22 16:39:32.403:                                        Path MTU:1500

*Aug 22 16:39:32.407:  Controlled Load Service  break bit=0  service length=0

*Aug 22 16:39:32.407:  RECORD_ROUTE         type 1 length 20:

*Aug 22 16:39:32.407:   10.23.1.1/32, Flags:0x0 (No Local Protection)

*Aug 22 16:39:32.411:   10.12.1.1/32, Flags:0x0 (No Local Protection)

Notice that ABR1 puts all IP addresses as strict hops to ABR2 router.

PATH messages on ABR2 router

!--- ABR2 receives a PATH message from ABR1 router

*Aug 22 17:16:04.163: Incoming Path:

*Aug 22 17:16:04.163:   version:1 flags:0000 cksum:D280 ttl:254 reserved:0 length:204

*Aug 22 17:16:04.167:  SESSION              type 7 length 16:

*Aug 22 17:16:04.167:   Tun Dest:   6.6.6.6  Tun ID: 10  Ext Tun ID: 1.1.1.1

*Aug 22 17:16:04.167:  HOP                  type 1 length 12:

*Aug 22 17:16:04.171:   Hop Addr: 10.23.1.1 LIH: 0xE8000409

*Aug 22 17:16:04.171:  TIME_VALUES          type 1 length 8 :

*Aug 22 17:16:04.171:   Refresh Period (msec): 30000

*Aug 22 17:16:04.171:  EXPLICIT_ROUTE       type 1 length 20:

*Aug 22 17:16:04.175:   10.23.1.2 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 17:16:04.175:   3.3.3.3 (Strict IPv4 Prefix, 8 bytes, /32)

*Aug 22 17:16:04.175:  LABEL_REQUEST        type 1 length 8 :

*Aug 22 17:16:04.175:   Layer 3 protocol ID: 2048

*Aug 22 17:16:04.175:  SESSION_ATTRIBUTE    type 7 length 16:

*Aug 22 17:16:04.179:   Setup Prio: 7, Holding Prio: 7

*Aug 22 17:16:04.179:   Flags: (0x4) SE Style

*Aug 22 17:16:04.179:   Session Name: PE1_t10

*Aug 22 17:16:04.179:  SENDER_TEMPLATE      type 7 length 12:

*Aug 22 17:16:04.183:   Tun Sender: 1.1.1.1  LSP ID: 891

*Aug 22 17:16:04.183:  SENDER_TSPEC         type 2 length 36:

*Aug 22 17:16:04.183:   version=0, length in words=7

*Aug 22 17:16:04.183:   Token bucket fragment (service_id=1, length=6 words

*Aug 22 17:16:04.187:     parameter id=127, flags=0, parameter length=5

*Aug 22 17:16:04.187:     average rate=6250000 bytes/sec, burst depth=1000 bytes

*Aug 22 17:16:04.187:     peak rate   =6250000 bytes/sec

*Aug 22 17:16:04.187:     min unit=0 bytes, max pkt size=2147483647 bytes

*Aug 22 17:16:04.187:  ADSPEC               type 2 length 48:

*Aug 22 17:16:04.191:  version=0  length in words=10

*Aug 22 17:16:04.191:  General Parameters  break bit=0  service length=8

*Aug 22 17:16:04.191:                                         IS Hops:2

*Aug 22 17:16:04.191:              Minimum Path Bandwidth (bytes/sec):12500000

*Aug 22 17:16:04.195:                       Path Latency (microseconds):0

*Aug 22 17:16:04.195:                                        Path MTU:1500

*Aug 22 17:16:04.195:  Controlled Load Service  break bit=0  service length=0

*Aug 22 17:16:04.195:  RECORD_ROUTE         type 1 length 20:

*Aug 22 17:16:04.195:   10.23.1.1/32, Flags:0x0 (No Local Protection)

*Aug 22 17:16:04.199:   10.12.1.1/32, Flags:0x0 (No Local Protection)

!--- ABR2 sends out a PATH message to PE2 router

*Aug 22 17:16:04.239: Outgoing Path:

*Aug 22 17:16:04.239:   version:1 flags:0000 cksum:988E ttl:253 reserved:0 length:192

*Aug 22 17:16:04.243:  SESSION              type 7 length 16:

*Aug 22 17:16:04.243:   Tun Dest:   6.6.6.6  Tun ID: 10  Ext Tun ID: 1.1.1.1

*Aug 22 17:16:04.243:  HOP                  type 1 length 12:

*Aug 22 17:16:04.243:   Hop Addr: 10.36.1.1 LIH: 0x5E000409

*Aug 22 17:16:04.247:  TIME_VALUES          type 1 length 8 :

*Aug 22 17:16:04.247:   Refresh Period (msec): 30000

*Aug 22 17:16:04.247:  LABEL_REQUEST        type 1 length 8 :

*Aug 22 17:16:04.247:   Layer 3 protocol ID: 2048

*Aug 22 17:16:04.251:  SESSION_ATTRIBUTE    type 7 length 16:

*Aug 22 17:16:04.251:   Setup Prio: 7, Holding Prio: 7

*Aug 22 17:16:04.251:   Flags: (0x4) SE Style

*Aug 22 17:16:04.251:   Session Name: PE1_t10

*Aug 22 17:16:04.251:  SENDER_TEMPLATE      type 7 length 12:

*Aug 22 17:16:04.255:   Tun Sender: 1.1.1.1  LSP ID: 891

*Aug 22 17:16:04.255:  SENDER_TSPEC         type 2 length 36:

*Aug 22 17:16:04.255:   version=0, length in words=7

*Aug 22 17:16:04.255:   Token bucket fragment (service_id=1, length=6 words

*Aug 22 17:16:04.259:     parameter id=127, flags=0, parameter length=5

*Aug 22 17:16:04.259:     average rate=6250000 bytes/sec, burst depth=1000 bytes

*Aug 22 17:16:04.259:     peak rate   =6250000 bytes/sec

*Aug 22 17:16:04.259:     min unit=0 bytes, max pkt size=2147483647 bytes

*Aug 22 17:16:04.259:  ADSPEC               type 2 length 48:

*Aug 22 17:16:04.263:  version=0  length in words=10

*Aug 22 17:16:04.263:  General Parameters  break bit=0  service length=8

*Aug 22 17:16:04.263:                                         IS Hops:3

*Aug 22 17:16:04.263:              Minimum Path Bandwidth (bytes/sec):12500000

*Aug 22 17:16:04.267:                     Path Latency (microseconds):0

*Aug 22 17:16:04.267:                                        Path MTU:1500

*Aug 22 17:16:04.267:  Controlled Load Service  break bit=0  service length=0

*Aug 22 17:16:04.267:  RECORD_ROUTE         type 1 length 28:

*Aug 22 17:16:04.267:   10.36.1.1/32, Flags:0x0 (No Local Protection)

*Aug 22 17:16:04.271:   10.23.1.1/32, Flags:0x0 (No Local Protection)

*Aug 22 17:16:04.275:   10.12.1.1/32, Flags:0x0 (No Local Protection)

Notice the incoming PATH message from ABR1 router has an ERO object but the outgoing PATH message does not carry any ERO object. ABR2 router computes a path to the destination dynamically.

The following output shows the MPLS TE tunnel established between PE1 and PE2 router. Notice the asterisk (*) on 3.3.3.3 (ABR2). This indicates it is the loose hop. The path taken is the shortest path - PE1-ABR1-ABR2-PE2. This can be identified from Record Route object. Also note that the path weight for TE tunnel is 1. This is actually the path-weight to ABR1 router.

MPLS TE tunnel on PE1 router

PE1# show mpls traffic-eng tunnels tunnel 10

Name: PE1_t10                             (Tunnel10) Destination: 6.6.6.6

  Status:

    Admin: up         Oper: up     Path: valid       Signalling: connected

    path option 1, type explicit PATH (Basis for Setup, path weight 1)

  Config Parameters:

    Bandwidth: 50000    kbps (Global)  Priority: 7  7   Affinity: 0x0/0xFFFF

    Metric Type: TE (default)

    AutoRoute announce: disabled LockDown: disabled Loadshare: 50000    bw-based

    auto-bw: disabled

  Active Path Option Parameters:

    State: explicit path option 1 is active

    BandwidthOverride: disabled  LockDown: disabled  Verbatim: disabled

  InLabel  :  -

  OutLabel : FastEthernet1/0, 16

  Next Hop : 10.12.1.2

  RSVP Signalling Info:

       Src 1.1.1.1, Dst 6.6.6.6, Tun_Id 10, Tun_Instance 891

    RSVP Path Info:

      My Address: 10.12.1.1

      Explicit Route: 10.12.1.2 2.2.2.2 3.3.3.3*

      Record   Route:

      Tspec: ave rate=50000 kbits, burst=1000 bytes, peak rate=50000 kbits

    RSVP Resv Info:

      Record   Route:  10.23.1.1 10.36.1.1 10.36.1.2

      Fspec: ave rate=50000 kbits, burst=1000 bytes, peak rate=50000 kbits

  Shortest Unconstrained Path Info:

    Path Weight: UNKNOWN

    Explicit Route:  UNKNOWN

  History:

    Tunnel:

      Time since created: 2 days, 52 minutes

      Time since path change: 4 minutes, 52 seconds

      Number of LSP IDs (Tun_Instances) used: 891

    Current LSP: [ID: 891]

      Uptime: 4 minutes, 52 seconds

    Prior LSP: [ID: 885]

      ID: path option unknown

      Removal Trigger: tunnel shutdown

Yet another method to compute a path for interdomain TE is called path computation element (PCE). It is possible to offload the path computation to an entity which has complete information about the topology and can deliver the necessary information on demand. The PCE can be a module on a router or a completely separate entity that the router can communicate with. The PCE services path computation requests. So the router can request a path to the next loose hop from the PCE rather than running cSPF on the router itself. PCE is supported in Cisco IOS XR.

Crankback

In regular MPLS TE within a single IGP area, RSVP PATHErr messages would provide the necessary information to the headend router in case of LSP failure, may be due to unavailability of resources on a particular link. However, this information would make little sense to the headend router if the link belongs to a different area than the headend router.

For example, if ABR2-PE2 is unavailable, sending an error message back to PE1 router has no importance to PE1. Instead, this information would make more sense to ABR2 router, in which case, it will try to find an alternative path within area 2 that avoids ABR2-PE2 link.

In this case, ABR2 will be able to find an alternative path through ABR3 router. However, if ABR2-ABR3 link was also down, ABR2 will not be able to find an alternative path. So the error is forwarded one step back (upstream router) to ABR1 router. So, ABR1 router is assigned the task to find a path avoiding ABR2 router.

In this way, computation is cranked back one step at a time, away from the failure point. This process is called crankback.

Note: Cisco IOS does not support crankback. So the path error message is forwarded all the way back to the headend router. After 2 minutes, the headend router tries to evaluate the next path-option if configured.