Is BGP Badly Broken?
Nuance is lost in 140 chars. Those were the closing words in a tweet exchange I was involved in this morning. The subject was on BGP being broken.
The exchange started after Greg posted a link to a presentation from RIPE71 about simplifying BGP setup. It is an interesting presentation, essentially it is proposing encoding business relationships into your BGP config in order to avoid route leaks. It is an interesting approach, and works well to address a specific class of route leaks; however the claim that it allows “no mistake leaks” is too bold IMHO.
Before diving further into detail, we should be clear on terms. At the same meeting Randy Bush presented on several topics in one presentation, including a taxonomy of Internet routing events. In this taxonomy a leak is distinct from a mis-origination or hijack, and is defined as:
leak — i receive P and send it on to folk to whom i should not send it for business reasons (transit, peer, …)
The most obvious example of this is when a misconfigured edge network allows routes from one transit provider to be passed to a second upstream (edit: I classify peers as upstream in this context) . This happens depressingly frequently and it is generally handled quietly by the upstream network. When it isn’t handled it usually manifests as localised latency issues and packet loss (localised in a global context, the geographic impact can be quite large).
Dealing with these issues is pretty straight forward and most well run networks will run a two-pronged approach to mitigation, constraining routes learnt from downstream networks to avoid being the cause and using automated tools to provide notification of route leaks from other parties. A set of best practices are encoded in the MANRS project which is gaining support in the industry.
There are some terminology differences but the MANRS approach is largely compatible with the Simple BGP approach in the presentation and if adopted as default behaviour when establishing BGP relationships (a pretty big if) this could provide an improved standard behavior in the Internet context, reducing the chance of misconfiguration. So mostly I am in agreement that this would be a good thing but I am sceptical about adoption.
The meat of the conversation this morning (and the area where I don’t think 140 characters allowed me to express my nuance) was around the conclusions to draw from this.
Saying that BGP is broken is a massive overstatement. BGP is not a policy engine; BGP is a routing protocol, and as a routing protocol it does a good job. The standard set of attributes and the extensibility are what has allowed a protocol written on two napkins nearly 30 year ago to still be relevant today. The flexibility of BGP is at the core of its success and the vast majority of BGP networks work fine, whether they are capital-I Internet, MPLS IPVPNs, VPLS or EVPN based.
Now, just so we are straight, when I say that BGP is not broken I am not some protocol apologist. What I am objecting to is actually a conflation of BGP the protocol and the Internet Routing function, which is an application built on the BGP protocol requiring distributed coordination of policy. If the statement was “Internet Routing is broken” I would object less (although I’d still argue that “Internet Routing is fragile” is more accurate, even if it is a less effective sound-bite).
As an example: from the definition above preventing route leaks requires encapsulation of business relationships into the routing policy. Encoding these relationships at a protocol level may be a pragmatic solution to reduce the fragility of the global routing infrastructure but by doing so we are encoding a specific use-case into a a general purpose protocol with a clear layer violation.
The key fact that makes Internet routing fragile is that BGP is susceptible to misconfiguration (intentional or unintentional). Being tolerant to misbehaving nodes is known in computer science circles as Byzantine fault tolerance. It is a hard problem, but one that there has been considerable research on. I am sure that a smart, driven person could come up with a way of applying this to BGP (I don’t think I am either smart enough or driven enough to be that person), but shoe-horning that into a routing protocol will add significant complexity and gaining the acceptance required to convince people to deploy in production would be a sisyphean endeavour.
This brings me to another computer science concept that is familiar to all network engineers. Layering.
Efforts like MANRS may not be stated as such, but are effectively an attempt to standardise a higher level protocol built on BGP, a protocol that standardises business relationships and expectations for cooperation between operators. Parts of this can be delivered using BGP in a prescribed way (perhaps we even need new AFI/SAFI pairs to encapsulate IPv4/Internet and IPv6/Internet so we can be clear when the “Internet Routing function” rules need to be applied). Other parts of this higher level protocol may be addressed by other low-level protocols (e.g. rPKI, RPSL, DNS, perhaps even email).
Step one is to define the required behaviour, step two is to figure out the best ways of addressing those requirements and step three would be protocol changes.
Saying that BGP is broken preempts any analysis and makes the assumption that BGP is the right protocol to implement the fix (edit: or that BGP is unfixable and needs to be replaced wholesale). This assumption leads to encapsulation of even more functions into BGP, and to my mind likely makes things worse, not better.
Edit: some clarifications (annotated inline) and some minor fixes to grammar.