BIRD / BGP-ORR experiences?
Thinking about setting up BGP-ORR on some BIRD VMs (https://bird.network.cz) for lab purposes, I'm sure its more than sufficient. Does anyone use these in production? Any thoughts, experiences, caveats? Do we even like BGP ORR? Thanks in advance, Deepak
On Wed, 15 Apr 2020 at 10:56, Deepak Jain <deepak@ai.net> wrote: Hey,
Do we even like BGP ORR?
I like it, I think ADD-PATH and ORR are mandatory features in modern RR infra. However proper interaction between them may not exist in every implementation. Basically you want a) send all ECMPable paths b) send one backup path This will lead to superior to full-mesh by every reasonable metric: - smaller rib (no routes that you don't need, but all the routes you care about) - redundancy (one iBGP down, is not customer outage) - less state changes, less code stress ORR is not an RFC and there are some open questions. What to reflect, when next-hop is not in IGP? Do we hope that receiver would recurse to the same IGP next-hop? Juniper makes this assumption, which to me is decidedly the common case. Cisco makes no assumption and doesn't reflect if next-hop is not in IGP, but as I understand they will fix to the same assumption as Juniper. -- ++ytti
On 15/Apr/20 13:36, Saku Ytti wrote:
ORR is not an RFC and there are some open questions. What to reflect, when next-hop is not in IGP? Do we hope that receiver would recurse to the same IGP next-hop? Juniper makes this assumption, which to me is decidedly the common case. Cisco makes no assumption and doesn't reflect if next-hop is not in IGP, but as I understand they will fix to the same assumption as Juniper.
When we wanted this bad, it wasn't ready (2014), so we ended up deploying an RR in every major PoP, since that wasn't too costly (there was a time when a network I knew of used a Juniper M120 as an RR). Nice to hear ORR has come a long way that it's somewhat usable. Mark.
On 15/Apr/20 14:28, Tarko Tikan wrote:
It is usable, we have taken it even a step forward:
- virtualized RR - add-path - ORR - IGP topology to RR via BGP-LS so we don't have to extend ISIS to VMs (there are some issues with SR-IOV)
Awesome! Never been a fan of Add-Path or Diverse-Path, but good to know it's working well for you and others. Were you previously running IS-IS on a UNIX/Linux system running in a VM? Mark.
Nice to hear ORR has come a long way that it's somewhat usable.
It is usable, we have taken it even a step forward: - virtualized RR - add-path - ORR - IGP topology to RR via BGP-LS so we don't have to extend ISIS to VMs (there are some issues with SR-IOV) -- That sounds pretty exciting and its good to see traditional router vendors (embedded) leading the way on features. I don't exactly understand how add-path plus ORR works, I figured you'd only use one or the other, but that just shows you I haven't been up on this in a while. Deepak
On 15/Apr/20 13:36, Saku Ytti wrote:
ORR is not an RFC and there are some open questions. What to reflect, when next-hop is not in IGP? Do we hope that receiver would recurse to the same IGP next-hop? Juniper makes this assumption, which to me is decidedly the common case. Cisco makes no assumption and doesn't reflect if next-hop is not in IGP, but as I understand they will fix to the same assumption as Juniper.
When we wanted this bad, it wasn't ready (2014), so we ended up deploying an RR in every major PoP, since that wasn't too costly (there was a time when a network I knew of used a Juniper M120 as an RR). Nice to hear ORR has come a long way that it's somewhat usable. ---- How is this approach working for you? We were considering BGP ORR on bare metal (maybe a VM if we can get OSPF into the VM) and putting a primary RR at each eBGP node (physical site) and at least one link to a backup RR at a different site, perhaps enabling BGP ORR for that to minimize suboptimal paths. Adding a VM or a server node for this function is hardly the technical challenge it used to be with so many linux-based white box switches and things running around nowadays. Thanks! Deepak
On 15/Apr/20 17:51, Deepak Jain wrote:
How is this approach working for you?
It's working out beautifully, since 2014. We wanted ORR at the time, but it was immature, so this was our only option. Yes, it's an old school approach, but it's simple, so we don't have to enable any trickery.
We were considering BGP ORR on bare metal (maybe a VM if we can get OSPF into the VM) and putting a primary RR at each eBGP node (physical site) and at least one link to a backup RR at a different site, perhaps enabling BGP ORR for that to minimize suboptimal paths.
Adding a VM or a server node for this function is hardly the technical challenge it used to be with so many linux-based white box switches and things running around nowadays.
Well, our RR's run on ESXi (Cisco CSR1000v), but I'm sure your favorite hypervisor will work just fine. Your main concern is whether you want to run an established code base (Cisco, Juniper, Nokia, e.t.c.), or if you want to run something more open source. We chose the former considering our iBGP is a critical piece of infrastructure. Mark.
Do we even like BGP ORR?
I like it, I think ADD-PATH and ORR are mandatory features in modern RR infra. However proper interaction between them may not exist in every implementation. Basically you want a) send all ECMPable paths b) send one backup path This will lead to superior to full-mesh by every reasonable metric: - smaller rib (no routes that you don't need, but all the routes you care about) - redundancy (one iBGP down, is not customer outage) - less state changes, less code stress ORR is not an RFC and there are some open questions. What to reflect, when next-hop is not in IGP? Do we hope that receiver would recurse to the same IGP next-hop? Juniper makes this assumption, which to me is decidedly the common case. Cisco makes no assumption and doesn't reflect if next-hop is not in IGP, but as I understand they will fix to the same assumption as Juniper. ------------- Thanks for your input. How do you handle next-hops? Tunnels between all eBGP speakers as if they were fully meshed as their potential next-hops? Deepak
On 16 Apr 2020, at 22:35, Mark Tinka <mark.tinka@seacom.mu> wrote:
On 15/Apr/20 19:07, Saku Ytti wrote:
Don't run Cisco ORR RR or have IGP next-hops :/
Does it break NEXT_HOP=self in Cisco-land?
Mark.
We’re testing ORR at the moment as part of core upgrades (XRv on ESXi), and next-hop self not only works, it’s required for ORR to work properly I’ve not noticed any major issues with it yet but it’s still early days in terms of our deployment
On 16/Apr/20 15:17, Chris Jones wrote:
We’re testing ORR at the moment as part of core upgrades (XRv on ESXi), and next-hop self not only works, it’s required for ORR to work properly
Yes, that would be my simple 1+1, as it's all about optimizing for the best IGP exit for far-away nodes. Mark.
On Thu, 16 Apr 2020 at 15:36, Mark Tinka <mark.tinka@seacom.mu> wrote:
Does it break NEXT_HOP=self in Cisco-land?
That would be in IGP, so that'll work. The other way that some people do this, is that next-hop is CE, which is in iBGP, but recurses to loop0. There are some TE reasons why people might do this, and it would not work with Cisco ORR. -- ++ytti
On 16/Apr/20 16:50, Saku Ytti wrote:
That would be in IGP, so that'll work. The other way that some people do this, is that next-hop is CE, which is in iBGP, but recurses to loop0. There are some TE reasons why people might do this, and it would not work with Cisco ORR.
Reasonably safe to say that if you are doing TE, you are not always satisfied with what the IGP thinks, and want to override it to some extent :-). Mark.
On 15/Apr/20 17:59, Deepak Jain wrote:
Thanks for your input. How do you handle next-hops? Tunnels between all eBGP speakers as if they were fully meshed as their potential next-hops?
I should imagine NEXT_HOP=self still works in an ORR world, non :-)? -------- The question resolves around if the external borders are not physically connected to each other, the next-hop won't matter once it hits the next router... without a tunnel or TE or similar, as far as I know, the packet destination is not overwritten, so there is the potential for IGP loops without a mechanism for consistency. Deepak
participants (6)
-
Chris Jones
-
Deepak Jain
-
Mark Tinka
-
Nick Hilliard
-
Saku Ytti
-
Tarko Tikan