Multi site BGP Routing design
We have two geographically distinct locations that currently both fall under the same ASN. At site 1 we have a particular set of ip networks (/20 and bigger) in use only locally to this site At site 2 we have a separate set of ip networks (/20 and bigger) in use only locally to this site Each site has at least one upstream internet connection advertising with BGP. There is also a (reliable) private link between to the two sites where our routers at each site are all talking iBGP (as well as ospf). There is a router subnet (/27) that spans the two sites. We currently advertise all subnets out all upstream connections as if both sites were only one and traffic routes between sites without issue via the private link. If the private link between the two sites fails, will BGP allow for us to access the IP subnets at site 2 from site 1 via the internet given that both sites are advertising under the same ASN? Is this a case where having multiple ASNs makes sense to treat each site as remote peers to each other? Thanks, Justin
Justin Krejci wrote:
If the private link between the two sites fails, will BGP allow for us to access the IP subnets at site 2 from site 1 via the internet given that both sites are advertising under the same ASN?
No, because your router at site 2 will not accept any prefix with its own AS in the AS_PATH (which site 1 would be advertising from).
Is this a case where having multiple ASNs makes sense to treat each site as remote peers to each other?
Unless someone else has any better advice (I'm sure they do), you will need two separate public ASNs. Site 1 advertises it's space out of AS1, and site 2 advertises it's space from AS2. If you do that, it may be best if you have an eBGP session between the two PoPs using med/pref to ensure the direct link is preferred if it is up. (I've never had to do iBGP between two sites like this before, but I do know that eBGP is preferred over iBGP). Steve
Once upon a time, Steve Bertrand <steve@ibctech.ca> said:
Unless someone else has any better advice (I'm sure they do), you will need two separate public ASNs. Site 1 advertises it's space out of AS1, and site 2 advertises it's space from AS2.
I don't know that it's better advice, but another way to link the two sites is via a tunnel (GRE or IPIP). Use the upstream IP on each router as the local endpoint, and then run some routing protocol over the tunnel. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Depending on your security policies you may want to encrypt said tunnel also. Other than that, it all depends on it all depends. For example - if you receive / or have a default route pointing to the ISP, then the fact you have the same AS and won't receive the other site's routes in BGP doesn't matter at all - you'll follow a default from site 1 to the ISP, and the ISP will have a route to site 2 and can pass the traffic in the right direction. If you don't mind your traffic being passed unencrypted over the Internet, that is. You'll obviously need to adapt your firewall policies to allow for that flow as well. j. ________________________________ From: Chris Adams [cmadams@hiwaay.net] Sent: Friday, June 05, 2009 20:16 To: nanog@nanog.org Subject: Re: Multi site BGP Routing design Once upon a time, Steve Bertrand <steve@ibctech.ca> said:
Unless someone else has any better advice (I'm sure they do), you will need two separate public ASNs. Site 1 advertises it's space out of AS1, and site 2 advertises it's space from AS2.
I don't know that it's better advice, but another way to link the two sites is via a tunnel (GRE or IPIP). Use the upstream IP on each router as the local endpoint, and then run some routing protocol over the tunnel. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
John.Herbert@ins.com wrote:
Depending on your security policies you may want to encrypt said tunnel also.
Other than that, it all depends on it all depends. For example - if you receive / or have a default route pointing to the ISP, then the fact you have the same AS and won't receive the other site's routes in BGP doesn't matter at all - you'll follow a default from site 1 to the ISP, and the ISP will have a route to site 2 and can pass the traffic in the right direction. If you don't mind your traffic being passed unencrypted over the Internet, that is. You'll obviously need to adapt your firewall policies to allow for that flow as well.
Personally, I don't really like the tunnel idea... I've had to deal with them for v6 connectivity, and they seem so 'ugly'. My first thoughts were about de-aggregation, but since he's already advertising different space out of each site, that became irrelevant. I was just thinking that two AS numbers would be the cleanest, easiest to maintain method for him to take. Certainly tunnelling did go through my mind though to ensure site-to-site peering over the Internet. Steve
Steve, Agreed. I'm not suggesting that a tunnel is the ultimate best solution, but rather just pointing out that if you go with a tunnel, it's worth remembering that it's going unencrypted over a public network rather than site to site over a private link. j. ________________________________ From: Steve Bertrand [steve@ibctech.ca] Sent: Friday, June 05, 2009 20:40 To: Herbert, John Cc: cmadams@hiwaay.net; nanog@nanog.org Subject: Re: Multi site BGP Routing design John.Herbert@ins.com wrote:
Depending on your security policies you may want to encrypt said tunnel also.
Other than that, it all depends on it all depends. For example - if you receive / or have a default route pointing to the ISP, then the fact you have the same AS and won't receive the other site's routes in BGP doesn't matter at all - you'll follow a default from site 1 to the ISP, and the ISP will have a route to site 2 and can pass the traffic in the right direction. If you don't mind your traffic being passed unencrypted over the Internet, that is. You'll obviously need to adapt your firewall policies to allow for that flow as well.
Personally, I don't really like the tunnel idea... I've had to deal with them for v6 connectivity, and they seem so 'ugly'. My first thoughts were about de-aggregation, but since he's already advertising different space out of each site, that became irrelevant. I was just thinking that two AS numbers would be the cleanest, easiest to maintain method for him to take. Certainly tunnelling did go through my mind though to ensure site-to-site peering over the Internet. Steve
On 6/5/09 4:42 PM, "Steve Bertrand" <steve@ibctech.ca> wrote:
Justin Krejci wrote:
If the private link between the two sites fails, will BGP allow for us to access the IP subnets at site 2 from site 1 via the internet given that both sites are advertising under the same ASN?
No, because your router at site 2 will not accept any prefix with its own AS in the AS_PATH (which site 1 would be advertising from).
If you're running Cisco with the right IOS it looks like you could use the 'neighbor x.x.x.x allowas-in' command to accept your own AS. Then you would just have to set your local route origination so that the appropriate routes were withdrawn when the backnet link goes down. Mike
Michael K. Smith wrote:
On 6/5/09 4:42 PM, "Steve Bertrand" <steve@ibctech.ca> wrote:
Justin Krejci wrote:
If the private link between the two sites fails, will BGP allow for us to access the IP subnets at site 2 from site 1 via the internet given that both sites are advertising under the same ASN? No, because your router at site 2 will not accept any prefix with its own AS in the AS_PATH (which site 1 would be advertising from).
If you're running Cisco with the right IOS it looks like you could use the 'neighbor x.x.x.x allowas-in' command to accept your own AS. Then you would just have to set your local route origination so that the appropriate routes were withdrawn when the backnet link goes down.
I stand corrected. I've read about this, but does anyone have operational experience with it that they can share? Even though we are a very small SP, I always feel that going against the traditional grain when doing things like this may leave a trail of undocumented, hard-to-troubleshoot issues in the future. To rephrase the OP's question, would it be BCP to acquire a second ASN, and without further de-aggregating, continue advertising each site's IP space to the DFZ, but from dissimilar ASs as opposed to the same one? Steve
To rephrase the OP's question, would it be BCP to acquire a second ASN, and without further de-aggregating, continue advertising each site's IP space to the DFZ, but from dissimilar ASs as opposed to the same one?
This would definitely be the best approach. You're not introducing new IP prefixes and you're not extending AS paths, so the net effect on the global BGP routing is zero (OK, you might have to use the 4 byte AS number :). Just make sure that both ISPs you connect to allow you to advertise "transit" prefixes. If site A public link goes down, but the private link is up, site B will advertise its own address space plus site A's address space with an extra AS number in the AS path (and the upstream ISP might filter that). Ivan http://www.ioshints.info/about http://blog.ioshints.info/
For a given interconnection between the upstream ISPs for the two site, once the direct link goes down, the time required for site A to learn the new route to site B and vice versa would be different with the different proposed solutions, right? Thanks and best regards On Sat, Jun 6, 2009 at 12:40 PM, Ivan Pepelnjak <ip@ioshints.info> wrote:
To rephrase the OP's question, would it be BCP to acquire a second ASN, and without further de-aggregating, continue advertising each site's IP space to the DFZ, but from dissimilar ASs as opposed to the same one?
This would definitely be the best approach. You're not introducing new IP prefixes and you're not extending AS paths, so the net effect on the global BGP routing is zero (OK, you might have to use the 4 byte AS number :).
Just make sure that both ISPs you connect to allow you to advertise "transit" prefixes. If site A public link goes down, but the private link is up, site B will advertise its own address space plus site A's address space with an extra AS number in the AS path (and the upstream ISP might filter that).
Ivan
-- Muhammad Saqib Ilyas PhD Student, Computer Science and Engineering Lahore University of Management Sciences
Hi all, We actually have a very similar setup to what Justin asked about, with the exception that we advertise only some of our netblocks to one provider and the rest to the other. If one of the providers fails, we then advertise all netblocks through the provider which is still up. If the private link between our two locations fails, the two halves of our network communicate via the Internet.
From what Justin described, I would think he would be able to keep a single ASN and configure his network so that if the private link goes down, the two newly disconnected halves of his network advertise only the netblocks they can still "see" (i.e. the ones on their half). As long as his internal network is set up with dynamic routing (iBGP / OSPF) the two halves should realize they have to get to the other half via the Internet.
In our case, we don't get full routing tables from our providers, just default routes. Perhaps in Justin's case something as simple as a floating static route via the Internet to the other half of the network would take care of any ASN weirdness. It doesn't sound like he really needs his border routers to speak BGP with each other while the private link is down. If he wanted to remove the BGP session entirely under these circumstances, he could do the iBGP peering between RFC 1918 addresses and thus force the iBGP session to go down if the private link fails. Thanks, Adam ----- Original Message ----- From: "Saqib Ilyas" <msaqib@gmail.com> To: <nanog@nanog.org> Sent: Saturday, June 06, 2009 8:21 AM Subject: Re: Multi site BGP Routing design
For a given interconnection between the upstream ISPs for the two site, once the direct link goes down, the time required for site A to learn the new route to site B and vice versa would be different with the different proposed solutions, right? Thanks and best regards
On Sat, Jun 6, 2009 at 12:40 PM, Ivan Pepelnjak <ip@ioshints.info> wrote:
To rephrase the OP's question, would it be BCP to acquire a second ASN, and without further de-aggregating, continue advertising each site's IP space to the DFZ, but from dissimilar ASs as opposed to the same one?
This would definitely be the best approach. You're not introducing new IP prefixes and you're not extending AS paths, so the net effect on the global BGP routing is zero (OK, you might have to use the 4 byte AS number :).
Just make sure that both ISPs you connect to allow you to advertise "transit" prefixes. If site A public link goes down, but the private link is up, site B will advertise its own address space plus site A's address space with an extra AS number in the AS path (and the upstream ISP might filter that).
Ivan
-- Muhammad Saqib Ilyas PhD Student, Computer Science and Engineering Lahore University of Management Sciences
Thanks to all for the on and off list replies, they've been helpful. We get full BGP routes from all upstream connections (currently they are all different providers). The upstream bandwidth is cheaper at site 2 than at site 1 and the private backnet connection is a fixed cost so when previously considering the multi-ASN approach we would plan for each site using the other as a transit/gateway using eBGP but put preference on sending out via site 2 and maybe prepend site 1 AS on the local upstream SP so incoming favors site 2 as well (we're already doing this preferential routing anyways). I don't particularly care for the allow routes for our own ASN arrive from an upstream BGP session especially when it seems like all carriers would need to be cooperative on this, which may not be a big deal overall but adds another layer of complexity and difficulty if we change/add/remove carriers later on. What if they don't all support it, change their policies, or upgrade to a new version of router code that makes the default/expected behavior interfere. I am thinking the multiple ASN route is the cleanest but the idea of letting a default gateway (via static route maybe) out the local upstream connection to reach the other site when the backnet link is down sounds like it would work with minimal to no headaches but it just some how seems like a duct tape job. Does this sort of technique have any significant flaws or concerns associated with it? -----Original Message----- From: Adam Greene [mailto:maillist@webjogger.net] Sent: Saturday, June 06, 2009 8:38 AM To: nanog@nanog.org Subject: Re: Multi site BGP Routing design Hi all, We actually have a very similar setup to what Justin asked about, with the exception that we advertise only some of our netblocks to one provider and the rest to the other. If one of the providers fails, we then advertise all netblocks through the provider which is still up. If the private link between our two locations fails, the two halves of our network communicate via the Internet.
From what Justin described, I would think he would be able to keep a single
ASN and configure his network so that if the private link goes down, the two newly disconnected halves of his network advertise only the netblocks they can still "see" (i.e. the ones on their half). As long as his internal network is set up with dynamic routing (iBGP / OSPF) the two halves should realize they have to get to the other half via the Internet. In our case, we don't get full routing tables from our providers, just default routes. Perhaps in Justin's case something as simple as a floating static route via the Internet to the other half of the network would take care of any ASN weirdness. It doesn't sound like he really needs his border routers to speak BGP with each other while the private link is down. If he wanted to remove the BGP session entirely under these circumstances, he could do the iBGP peering between RFC 1918 addresses and thus force the iBGP session to go down if the private link fails. Thanks, Adam ----- Original Message ----- From: "Saqib Ilyas" <msaqib@gmail.com> To: <nanog@nanog.org> Sent: Saturday, June 06, 2009 8:21 AM Subject: Re: Multi site BGP Routing design
For a given interconnection between the upstream ISPs for the two site, once the direct link goes down, the time required for site A to learn the new route to site B and vice versa would be different with the different proposed solutions, right? Thanks and best regards
On Sat, Jun 6, 2009 at 12:40 PM, Ivan Pepelnjak <ip@ioshints.info> wrote:
To rephrase the OP's question, would it be BCP to acquire a second ASN, and without further de-aggregating, continue advertising each site's IP space to the DFZ, but from dissimilar ASs as opposed to the same one?
This would definitely be the best approach. You're not introducing new IP prefixes and you're not extending AS paths, so the net effect on the global BGP routing is zero (OK, you might have to use the 4 byte AS number :).
Just make sure that both ISPs you connect to allow you to advertise "transit" prefixes. If site A public link goes down, but the private link is up, site B will advertise its own address space plus site A's address space with an extra AS number in the AS path (and the upstream ISP might filter that).
Ivan
-- Muhammad Saqib Ilyas PhD Student, Computer Science and Engineering Lahore University of Management Sciences
I am thinking the multiple ASN route is the cleanest but the idea of letting a default gateway (via static route maybe) out the local upstream connection to reach the other site when the backnet link is down sounds like it would work with minimal to no headaches but it just some how seems like a duct tape job. Does this sort of technique have any significant flaws or concerns associated with it?
It's a static route, so you're never sure the remote end (upstream router) is truly alive. In this respect, it would be much better to receive default route over BGP (if the upstream carrier is willing to implement it). On the other hand, it's a last-resort mechanism, so you'd only use it if everything else fails (and you don't care how reliable it is). Just make sure it's well documented and understood ... and think about what will happen when you add a third carrier to one of the sites. Last but not least, you could use reliable static routing (static route tied to ping tests). http://blog.ioshints.info/2007/02/reliable-static-routing.html http://blog.ioshints.info/search?q=static+routing Just my $0.002 :) Ivan http://www.ioshints.info/about http://blog.ioshints.info/
On Fri, Jun 05, 2009 at 05:50:28PM -0500, Justin Krejci wrote:
If the private link between the two sites fails, will BGP allow for us to access the IP subnets at site 2 from site 1 via the internet given that both sites are advertising under the same ASN?
Maybe. Especially if both sites are connected to the same ISP, you can tweak some BGP knobs to allow your own ASN to appear in the AS PATH N times where N > 1, and accept the routes anyway.
Chuck Anderson wrote:
On Fri, Jun 05, 2009 at 05:50:28PM -0500, Justin Krejci wrote:
If the private link between the two sites fails, will BGP allow for us to access the IP subnets at site 2 from site 1 via the internet given that both sites are advertising under the same ASN?
Maybe. Especially if both sites are connected to the same ISP, you can tweak some BGP knobs to allow your own ASN to appear in the AS PATH N times where N > 1, and accept the routes anyway.
For some reason, I see that as being a configuration method that would quickly be forgotten about, and later cause major headaches trying to troubleshoot. Steve
This is a good concept but if the ISP route is a Juniper then as I recall by default it looks ahead, sees the as-path routing loop if it were to send it to the other router, and doesn't send it. So while you might be able to configure it on the receiving router, if the sending router won't send it, you're SOL. j. ________________________________ From: Chuck Anderson [cra@WPI.EDU] Sent: Friday, June 05, 2009 20:33 To: nanog@nanog.org Subject: Re: Multi site BGP Routing design On Fri, Jun 05, 2009 at 05:50:28PM -0500, Justin Krejci wrote:
If the private link between the two sites fails, will BGP allow for us to access the IP subnets at site 2 from site 1 via the internet given that both sites are advertising under the same ASN?
Maybe. Especially if both sites are connected to the same ISP, you can tweak some BGP knobs to allow your own ASN to appear in the AS PATH N times where N > 1, and accept the routes anyway.
On Fri, Jun 05, 2009 at 07:40:15PM -0500, John.Herbert@ins.com wrote:
This is a good concept but if the ISP route is a Juniper then as I recall by default it looks ahead, sees the as-path routing loop if it were to send it to the other router, and doesn't send it. So while you might be able to configure it on the receiving router, if the sending router won't send it, you're SOL.
True, the ISP in this case would have to cooperate :-)
Chuck Anderson wrote:
On Fri, Jun 05, 2009 at 07:40:15PM -0500, John.Herbert@ins.com wrote:
This is a good concept but if the ISP route is a Juniper then as I recall by default it looks ahead, sees the as-path routing loop if it were to send it to the other router, and doesn't send it. So while you might be able to configure it on the receiving router, if the sending router won't send it, you're SOL.
True, the ISP in this case would have to cooperate :-)
Have you ever known an ISP to not co-operate when it comes to requesting a BGP session? Steve
Have you ever known an ISP to not co-operate when it comes to requesting a BGP session? yes. this problem is rampant with colonialist telcos in the poorer countries. Yeah, well, I don't live in a poorer country, and I deal with it here. *cough*
you asked a question. you are not required to like the answer. randy
Have you ever known an ISP to not co-operate when it comes to requesting a BGP session? yes. this problem is rampant with colonialist telcos in the poorer countries. Yeah, well, I don't live in a poorer country, and I deal with it here. *cough* you asked a question. you are not required to like the answer.
oh, and i belive there was a north american incident of this discussed on this list in the last year. i am just too soaked to have the energy to search. i think it was due to living in a sparsely served area, so the isp could get away with <bleep>. randy
participants (10)
-
Adam Greene -
Chris Adams -
Chuck Anderson -
Ivan Pepelnjak -
John.Herbert＠ins.com -
Justin Krejci -
Michael K. Smith -
Randy Bush -
Saqib Ilyas -
Steve Bertrand