BGP Prefix Origin Validation without RPKI in DataCenter Networks
BGP Prefix Origin Validation without RPKI in DataCenter Networks
RPKI [1] has been well known for providing prefix origin validation in the Internet space. Basically, the idea is that the prefix owner must obtain a certificate from RIRs for ASN and prefix(es) they hold. The routers can talk to validators to verify the validity of prefix and its origination by an ASN. Based on that information, routers can make a decision to consider the prefix for best path selection based on route-policy configured.
Within datacenter networks (or even on the Internet), the use of RPKI has not been that great [2]. And misconfigurations are far too often. This article proposes a (partial, if not robust/full) solution to validate prefixes against a database in near real time. I will explain later why near real time. Optionally, we can filter invalid prefixes and raise an alert on a dashboard.
First let's see an example network in figure 1 we will use in this article.
We will focus on ASN 65104 for validation of prefixes but ideally, you would validate on all eBGP peers.
Just a note that every ASN could be an entire multi-stage Clos fabric or a set a routers. In case of Clos fabrics, ASN 65104 would connect to other ASNs like seen in figure 2.
So let's see what information we must have in our database. Surprisingly, not much. Infact, I realize the ROA (Route Origination Authorization) created by RIRs has minimalist details and they are enough for prefix origin validation.
Originating ASN - The ASN originating a prefix. Example - 65201
Prefix - The actual prefix. Example- 10.0.0.0/22
Maximum length - This is the maximum prefix length for the prefix that is allowed. Example- if maximum length is set to 24, the 10.0.0.0/24 is valid but 10.0.0.0/25 is invalid advertisement.
I am using MongoDB as the database so my example entries look like following:
The Architecture
The proposed solution architecture can be seen in figure 3. BMP is used to gather BGP information in real time. OpenBMP is used as a BMP Collector which publishes parsed data to Apache Kafka. Spark Streaming consumes BMP data and performs analysis.
Why "near" realtime?
There are few reasons-
1. In the Spark code, I use a 1 second interval to consume data from Kafka. This might be enough time for a router to calculate best path and install the BGP route in its routing table even before Spark has started analyzing the update.
2. The part where I "filter prefix", I use Python Netmiko library to connect to devices and apply configs. Netmiko is very slow to connect to Cisco devices. By that time again, router would have selected the invalid prefix and installed it in its routing table.
Use Case 1: Detecting incorrect prefix-length advertisement
Consider in figure 1 that ASN 65201 starts advertising /25 from its owned prefix 172.16.0.0/22 and 172.16.8.0/22. Since Maximum Length is defined as 24, any prefixes smaller than /24 are considered invalid.
In this case, router(s) in ASN 65104 configured with BMP, when receives these /25 prefixes will send the prefixes to OpenBMP and Spark will analyze them. These being invalid prefixes, Spark will filter them on the router and update dashboard.
After filtering, the config on the router looks like following:
Router4#show ip as-path
AS path access list 100
permit 65201
Router4#show ip prefix-list
ip prefix-list FILTER_INVALID_IN: 5 entries
seq 5 permit 172.16.0.0/25
seq 10 permit 172.16.0.128/25
seq 15 permit 172.16.1.0/25
seq 20 permit 172.16.1.128/25
seq 25 permit 172.16.2.0/25
Router4#
The prefix-list and as-path access-list are already part of a route-map that is configured on eBGP peers in incoming direction. The dashboard is updated using Websockets protocol like so:
It is not a fancy dashboard as you might have expected but that's due to my limited frontend skills.
Use Case 2: Detecting incorrect prefix origination aka prefix hijack
Consider that ASN 65201 owns 172.16.0.0/22 and it makes a valid advertisement. But ASN 65103 makes an invalid advertisement of 172.16.1.0/24. Since 172.16.1.0/24 is not owned by ASN 65103, it is considered an invalid prefix advertisement. However, if we were not monitoring with BMP, ASN 65104 would happily accept the prefix 172.16.1.0/24 from ASN 65103 and use it. But with monitoring, Spark detected invalid advertisement.
BGP table on a router in ASN 65104 shows 172.16.1.0/24 is not in the table and prefix-list is updated to filter this prefix.
Router4#show ip bgp
BGP table version is 51, local router ID is 4.4.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
* 172.16.0.0/22 10.5.6.5 0 65103 65101 65201 i
*> 10.4.6.4 0 65102 65101 65201 i
Router4#
Router4#show ip prefix-list
ip prefix-list FILTER_INVALID_IN: 1 entries
seq 5 permit 172.16.1.0/24
Router4#
The dashboard also shows 172.16.1.0/24 was filtered.
Use Case 3: Handling prefix originated from multiple ASNs
192.168.1.0/24 is considered as an Anycast prefix in this example network and ASNs 65201 and 65202 can advertise the prefix. This is considered a valid advertisement and must be accepted. BGP table on the router in ASN 65104 shows it is accepted.
Router4#show ip bgp
BGP table version is 53, local router ID is 4.4.4.4
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
* 172.16.0.0/22 10.5.6.5 0 65103 65101 65201 i
*> 10.4.6.4 0 65102 65101 65201 i
* 192.168.1.0 10.5.6.5 0 65103 65101 65201 i
*> 10.4.6.4 0 65102 65202 i
Router4#
Conclusion
BMP and Spark open up a lot of opportunities to monitor and maintain the network. This is just a list of use cases; there is a lot that can be done. Cisco's BMP implementation is not quite there yet since it only supports pre-policy. Having said that, BMP is not supported by a lot of vendors yet. Hopefully, after seeing this, you realize the potential and ask for BMP support from your vendor.
Spark code for these examples is on Github. I would love to hear from you. Send your comments/questions to scet_amit@yahoo.co.in
Further reading
[1] RPKI Wikipedia article - https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure
[2] Russ White's post on LinkedIn blog - https://engineering.linkedin.com/blog/2016/03/rethinking-path-valid-pt1