Two BGP peering sessions on single Comcast Fiber Connection?
After a many month wait, we were ready to turn up our BGP peering sessions on a new Comcast fiber connection. With our other providers (Level 3 and Verizon) we have edge routers that directly connect between the provider's on premise connection and our primary and a backup core routers. Each core router has a multihop BGP session with the provider's BGP router. The goal is to keep the single BGP router from being a single point of failure. Comcast said they could not support two separate BGP peering sessions on the same circuit. Does anyone have any counter examples? We used to have this setup with Comcast 5+ years ago, but now they say they can't support it. Bob Roswell broswell@syssrc.com<mailto:broswell@syssrc.com> 410-771-5544 ext 4336 Computer Museum Highlights<http://museum.syssrc.com/>
I started a thread around the same topic back on 10/16 of 2014. A Comcast engineer (who ultimately spoke to the national product manager) came back after discussing and said the same thing "We don't support that". I got a slightly longer explanation of: -------------------------------------------- In a nutshell, when we design a product we do it to accommodate the most typical customer cases. Given that the design includes a single fiber path and thus the fiber path and device that terminates on either end each are a single point of failure, adding extra BGP sessions doesn’t seem to add value in the typical failure scenarios. In order to achieve the simplest and most scalable solution to address the market, we rely on narrowing the possible combinations of parameters. -------------------------------------------- I explained to them that their interpretation prevents me from being able to do concurrent maintenance on my side (single router reboot/upgrade, etc). Never got anywhere with it though. I'm still interested in having this set up, but have given up on it ever really coming to reality. Luckily ALL of my other providers were more than happy to set up an extra session. If anyone from Comcast is listening, there is customer demand for this. It's not about making it better for Comcast, it's about allowing customers to have more flexibility. Mike Poublon /Senior Datacenter Network Engineer/ *Secant Technologies* 6395 Technology Ave. Suite A Kalamazoo, MI 49009 On 10/13/2016 1:48 PM, rar wrote:
After a many month wait, we were ready to turn up our BGP peering sessions on a new Comcast fiber connection.
With our other providers (Level 3 and Verizon) we have edge routers that directly connect between the provider's on premise connection and our primary and a backup core routers. Each core router has a multihop BGP session with the provider's BGP router. The goal is to keep the single BGP router from being a single point of failure.
Comcast said they could not support two separate BGP peering sessions on the same circuit. Does anyone have any counter examples? We used to have this setup with Comcast 5+ years ago, but now they say they can't support it.
Bob Roswell broswell@syssrc.com<mailto:broswell@syssrc.com> 410-771-5544 ext 4336
Computer Museum Highlights<http://museum.syssrc.com/>
Whenever we set up a bgp peer we do that to minimize downtime when doing maint. It's hit or miss. HE required a second physicall connection NTT was more than accommodating. On Oct 13, 2016 15:06, "Mike Poublon" <mpoublon@secantnet.net> wrote:
I started a thread around the same topic back on 10/16 of 2014. A Comcast engineer (who ultimately spoke to the national product manager) came back after discussing and said the same thing "We don't support that". I got a slightly longer explanation of:
--------------------------------------------
In a nutshell, when we design a product we do it to accommodate the most typical customer cases. Given that the design includes a single fiber path and thus the fiber path and device that terminates on either end each are a single point of failure, adding extra BGP sessions doesn’t seem to add value in the typical failure scenarios. In order to achieve the simplest and most scalable solution to address the market, we rely on narrowing the possible combinations of parameters.
--------------------------------------------
I explained to them that their interpretation prevents me from being able to do concurrent maintenance on my side (single router reboot/upgrade, etc). Never got anywhere with it though.
I'm still interested in having this set up, but have given up on it ever really coming to reality. Luckily ALL of my other providers were more than happy to set up an extra session.
If anyone from Comcast is listening, there is customer demand for this. It's not about making it better for Comcast, it's about allowing customers to have more flexibility.
Mike Poublon
/Senior Datacenter Network Engineer/
*Secant Technologies*
6395 Technology Ave. Suite A
Kalamazoo, MI 49009
On 10/13/2016 1:48 PM, rar wrote:
After a many month wait, we were ready to turn up our BGP peering sessions on a new Comcast fiber connection.
With our other providers (Level 3 and Verizon) we have edge routers that directly connect between the provider's on premise connection and our primary and a backup core routers. Each core router has a multihop BGP session with the provider's BGP router. The goal is to keep the single BGP router from being a single point of failure.
Comcast said they could not support two separate BGP peering sessions on the same circuit. Does anyone have any counter examples? We used to have this setup with Comcast 5+ years ago, but now they say they can't support it.
Bob Roswell broswell@syssrc.com<mailto:broswell@syssrc.com> 410-771-5544 ext 4336
Computer Museum Highlights<http://museum.syssrc.com/>
It really seems like it's a grave oversight to *NOT* support multiple BGP sessions. I drop to two routers for that same reason, I can do maintenance on one, while the other carries traffic. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mike Poublon" <mpoublon@secantnet.net> To: "rar" <rar@syssrc.com>, nanog@nanog.org Sent: Thursday, October 13, 2016 2:04:29 PM Subject: Re: Two BGP peering sessions on single Comcast Fiber Connection? I started a thread around the same topic back on 10/16 of 2014. A Comcast engineer (who ultimately spoke to the national product manager) came back after discussing and said the same thing "We don't support that". I got a slightly longer explanation of: -------------------------------------------- In a nutshell, when we design a product we do it to accommodate the most typical customer cases. Given that the design includes a single fiber path and thus the fiber path and device that terminates on either end each are a single point of failure, adding extra BGP sessions doesn’t seem to add value in the typical failure scenarios. In order to achieve the simplest and most scalable solution to address the market, we rely on narrowing the possible combinations of parameters. -------------------------------------------- I explained to them that their interpretation prevents me from being able to do concurrent maintenance on my side (single router reboot/upgrade, etc). Never got anywhere with it though. I'm still interested in having this set up, but have given up on it ever really coming to reality. Luckily ALL of my other providers were more than happy to set up an extra session. If anyone from Comcast is listening, there is customer demand for this. It's not about making it better for Comcast, it's about allowing customers to have more flexibility. Mike Poublon /Senior Datacenter Network Engineer/ *Secant Technologies* 6395 Technology Ave. Suite A Kalamazoo, MI 49009 On 10/13/2016 1:48 PM, rar wrote:
After a many month wait, we were ready to turn up our BGP peering sessions on a new Comcast fiber connection.
With our other providers (Level 3 and Verizon) we have edge routers that directly connect between the provider's on premise connection and our primary and a backup core routers. Each core router has a multihop BGP session with the provider's BGP router. The goal is to keep the single BGP router from being a single point of failure.
Comcast said they could not support two separate BGP peering sessions on the same circuit. Does anyone have any counter examples? We used to have this setup with Comcast 5+ years ago, but now they say they can't support it.
Bob Roswell broswell@syssrc.com<mailto:broswell@syssrc.com> 410-771-5544 ext 4336
Computer Museum Highlights<http://museum.syssrc.com/>
I completely concur. We spread our uplinks across separate boxes and we have /29 allocations. Get the best of all worlds. But if I only had one provider, I'd want to have multiple BGP sessions for this reason.
On Oct 17, 2016, at 08:30, Mike Hammett <nanog@ics-il.net> wrote:
It really seems like it's a grave oversight to *NOT* support multiple BGP sessions. I drop to two routers for that same reason, I can do maintenance on one, while the other carries traffic.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest-IX http://www.midwest-ix.com
----- Original Message -----
From: "Mike Poublon" <mpoublon@secantnet.net> To: "rar" <rar@syssrc.com>, nanog@nanog.org Sent: Thursday, October 13, 2016 2:04:29 PM Subject: Re: Two BGP peering sessions on single Comcast Fiber Connection?
I started a thread around the same topic back on 10/16 of 2014. A Comcast engineer (who ultimately spoke to the national product manager) came back after discussing and said the same thing "We don't support that". I got a slightly longer explanation of:
--------------------------------------------
In a nutshell, when we design a product we do it to accommodate the most typical customer cases. Given that the design includes a single fiber path and thus the fiber path and device that terminates on either end each are a single point of failure, adding extra BGP sessions doesn’t seem to add value in the typical failure scenarios. In order to achieve the simplest and most scalable solution to address the market, we rely on narrowing the possible combinations of parameters.
--------------------------------------------
I explained to them that their interpretation prevents me from being able to do concurrent maintenance on my side (single router reboot/upgrade, etc). Never got anywhere with it though.
I'm still interested in having this set up, but have given up on it ever really coming to reality. Luckily ALL of my other providers were more than happy to set up an extra session.
If anyone from Comcast is listening, there is customer demand for this. It's not about making it better for Comcast, it's about allowing customers to have more flexibility.
Mike Poublon
/Senior Datacenter Network Engineer/
*Secant Technologies*
6395 Technology Ave. Suite A
Kalamazoo, MI 49009
On 10/13/2016 1:48 PM, rar wrote: After a many month wait, we were ready to turn up our BGP peering sessions on a new Comcast fiber connection.
With our other providers (Level 3 and Verizon) we have edge routers that directly connect between the provider's on premise connection and our primary and a backup core routers. Each core router has a multihop BGP session with the provider's BGP router. The goal is to keep the single BGP router from being a single point of failure.
Comcast said they could not support two separate BGP peering sessions on the same circuit. Does anyone have any counter examples? We used to have this setup with Comcast 5+ years ago, but now they say they can't support it.
Bob Roswell broswell@syssrc.com<mailto:broswell@syssrc.com> 410-771-5544 ext 4336
Computer Museum Highlights<http://museum.syssrc.com/>
On 13 Oct 2016, at 19:48, rar wrote:
Comcast said they could not support two separate BGP peering sessions on the same circuit. Does anyone have any counter examples? We used to have this setup with Comcast 5+ years ago, but now they say they can't support it.
So how do they connect ip6 sessions? ;-) Jörg
Run your IPv4 peer to one router and IPv6 to another. Boom, redundancy! Spencer Ryan | Senior Systems Administrator | sryan@arbor.net<mailto:sryan@arbor.net> Arbor Networks +1.734.794.5033 (d) | +1.734.846.2053 (m) www.arbornetworks.com<http://www.arbornetworks.com/> ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Jörg Kost <jk@ip-clear.de> Sent: Thursday, October 13, 2016 3:59:29 PM To: rar Cc: nanog@nanog.org Subject: Re: Two BGP peering sessions on single Comcast Fiber Connection? On 13 Oct 2016, at 19:48, rar wrote:
Comcast said they could not support two separate BGP peering sessions on the same circuit. Does anyone have any counter examples? We used to have this setup with Comcast 5+ years ago, but now they say they can't support it.
So how do they connect ip6 sessions? ;-) Jörg
In a message written on Thu, Oct 13, 2016 at 05:48:18PM +0000, rar wrote:
The goal is to keep the single BGP router from being a single point of failure.
I don't really understand the failure analysis / uptime calculation. There is one router on the Comcast side, which is a single point of failure. There is one circuit to your prem, which is a single point of failure. To connect two routers on your end you must terminate the circuit in a switch, which is a single point of failure. And yet, in the face of all that somehow running two routers with two BGP sessions on your end increases your uptime? The only way that would even remotely make sense is if the routers in question were horribly broken / mismanaged so (had to be?) reboot(ed) on a regular basis. However if uptime is so important using gear with that property makes no sense! I'm pretty sure without actually doing the math that you'll be more reliable with a single quality router (elminiation of complexity), and that if you really need maximum uptime that you had better get a second circuit, on a diverse path, into a different router probably from a different carrier. -- Leo Bicknell - bicknell@ufp.org PGP keys at http://www.ufp.org/~bicknell/
+1, could not have said it better. On 10/15/2016 01:47 AM, Leo Bicknell wrote:
In a message written on Thu, Oct 13, 2016 at 05:48:18PM +0000, rar wrote:
The goal is to keep the single BGP router from being a single point of failure. I don't really understand the failure analysis / uptime calculation.
There is one router on the Comcast side, which is a single point of failure.
There is one circuit to your prem, which is a single point of failure.
To connect two routers on your end you must terminate the circuit in a switch, which is a single point of failure.
And yet, in the face of all that somehow running two routers with two BGP sessions on your end increases your uptime?
The only way that would even remotely make sense is if the routers in question were horribly broken / mismanaged so (had to be?) reboot(ed) on a regular basis. However if uptime is so important using gear with that property makes no sense!
I'm pretty sure without actually doing the math that you'll be more reliable with a single quality router (elminiation of complexity), and that if you really need maximum uptime that you had better get a second circuit, on a diverse path, into a different router probably from a different carrier.
It comes down to sizing your failure domain. Any single upstream Transit alone means the failure domain is the whole site (making assumptions about your topology). As mentioned earlier, any single point of failure doesn't reduce your failure footprint and gives little in terms of redundancy. Now if you point that second router to a second provider, now you've reduced the size of your failure domain to a single router/Transit, not the whole site. -b On Fri, Oct 14, 2016 at 10:34 AM, Paul S. <contact@winterei.se> wrote:
+1, could not have said it better.
On 10/15/2016 01:47 AM, Leo Bicknell wrote:
In a message written on Thu, Oct 13, 2016 at 05:48:18PM +0000, rar wrote:
The goal is to keep the single BGP router from being a single point of failure.
I don't really understand the failure analysis / uptime calculation.
There is one router on the Comcast side, which is a single point of failure.
There is one circuit to your prem, which is a single point of failure.
To connect two routers on your end you must terminate the circuit in a switch, which is a single point of failure.
And yet, in the face of all that somehow running two routers with two BGP sessions on your end increases your uptime?
The only way that would even remotely make sense is if the routers in question were horribly broken / mismanaged so (had to be?) reboot(ed) on a regular basis. However if uptime is so important using gear with that property makes no sense!
I'm pretty sure without actually doing the math that you'll be more reliable with a single quality router (elminiation of complexity), and that if you really need maximum uptime that you had better get a second circuit, on a diverse path, into a different router probably from a different carrier.
-- Bill Blackford Logged into reality and abusing my sudo privileges.....
Steering clear of the failure domain conversation, if its of any benefit - we can at least confirm that Comcast is willing to establish /29's for multiple BGP connections at 56 Marietta/ATL. These circuits are written on true wholesale/transit IP service contracts, which may be the difference. In our experience the Comcast Enterprise/Business groups have rather rigid circuit provisioning profiles, and even if you are able to talk an engineer into building a customer's configuration outside of their normal "scope", it usually comes back to haunt you at some point in the future, even if years later. Will send a link to the Comcast enterprise ip transit profiles separately, for reference, in the event you were not provided such previously...Or if Comcast wholesale is on the list, of course feel free to chime in too! On Fri, Oct 14, 2016, 1:49 PM Bill Blackford <bblackford@gmail.com> wrote:
It comes down to sizing your failure domain. Any single upstream Transit alone means the failure domain is the whole site (making assumptions about your topology). As mentioned earlier, any single point of failure doesn't reduce your failure footprint and gives little in terms of redundancy. Now if you point that second router to a second provider, now you've reduced the size of your failure domain to a single router/Transit, not the whole site.
-b
On Fri, Oct 14, 2016 at 10:34 AM, Paul S. <contact@winterei.se> wrote:
+1, could not have said it better.
On 10/15/2016 01:47 AM, Leo Bicknell wrote:
In a message written on Thu, Oct 13, 2016 at 05:48:18PM +0000, rar wrote:
The goal is to keep the single BGP router from being a single point of failure.
I don't really understand the failure analysis / uptime calculation.
There is one router on the Comcast side, which is a single point of failure.
There is one circuit to your prem, which is a single point of failure.
To connect two routers on your end you must terminate the circuit in a switch, which is a single point of failure.
And yet, in the face of all that somehow running two routers with two BGP sessions on your end increases your uptime?
The only way that would even remotely make sense is if the routers in question were horribly broken / mismanaged so (had to be?) reboot(ed) on a regular basis. However if uptime is so important using gear with that property makes no sense!
I'm pretty sure without actually doing the math that you'll be more reliable with a single quality router (elminiation of complexity), and that if you really need maximum uptime that you had better get a second circuit, on a diverse path, into a different router probably from a different carrier.
-- Bill Blackford
Logged into reality and abusing my sudo privileges.....
--
participants (11)
-
Bill Blackford -
Dovid Bender -
Jason Canady -
Jörg Kost -
Kraig Beahn -
Leo Bicknell -
Mike Hammett -
Mike Poublon -
Paul S. -
rar -
Ryan, Spencer