Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
I had a discussion with them about a point to point circuit last year and ran into some weirdness around how burstable it would be for specific IP to IP streams as our use case was cheap circuit / high speed data replication between given endpoints. The sales rep was suggesting to me that I’d see specific source/destination IP pairs capped at 2gbps regardless of circuit speed, which suggested to me it was not actually a point to point wave but some type of encapsulated service. We didn’t get into whether it was usable for non-IP, etc. From: NANOG <nanog-bounces+dhubbard=dino.hostasaurus.com@nanog.org> on behalf of Mike Hammett <nanog@ics-il.net> Date: Wednesday, October 14, 2020 at 1:38 PM To: "nanog@nanog.org" <nanog@nanog.org> Subject: Cogent Layer 2 Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable? ----- Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [Image removed by sender.]<https://www.facebook.com/ICSIL>[Image removed by sender.]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[Image removed by sender.]<https://www.linkedin.com/company/intelligent-computing-solutions>[Image removed by sender.]<https://twitter.com/ICSIL> Midwest Internet Exchange<http://www.midwest-ix.com/> [Image removed by sender.]<https://www.facebook.com/mdwestix>[Image removed by sender.]<https://www.linkedin.com/company/midwest-internet-exchange>[Image removed by sender.]<https://twitter.com/mdwestix> The Brothers WISP<http://www.thebrotherswisp.com/> [Image removed by sender.]<https://www.facebook.com/thebrotherswisp>[Image removed by sender.]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
When I last spoke to them, it sounded like they were using a bunch of LAG groups based on ip address because they _really_ wanted to know how many ip addresses we had and what kind of traffic we would be expecting (eyeball networks, big data transport, etc). -----Original Message----- From: "David Hubbard" <dhubbard@dino.hostasaurus.com> Sent: Wednesday, October 14, 2020 1:46pm To: "nanog@nanog.org" <nanog@nanog.org> Subject: Re: Cogent Layer 2 I had a discussion with them about a point to point circuit last year and ran into some weirdness around how burstable it would be for specific IP to IP streams as our use case was cheap circuit / high speed data replication between given endpoints. The sales rep was suggesting to me that I’d see specific source/destination IP pairs capped at 2gbps regardless of circuit speed, which suggested to me it was not actually a point to point wave but some type of encapsulated service. We didn’t get into whether it was usable for non-IP, etc. From: NANOG <nanog-bounces+dhubbard=dino.hostasaurus.com@nanog.org> on behalf of Mike Hammett <nanog@ics-il.net> Date: Wednesday, October 14, 2020 at 1:38 PM To: "nanog@nanog.org" <nanog@nanog.org> Subject: Cogent Layer 2 Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable? ----- Mike Hammett [ Intelligent Computing Solutions ]( http://www.ics-il.com/ ) [ ]( https://www.facebook.com/ICSIL )[ ]( https://plus.google.com/+IntelligentComputingSolutionsDeKalb )[ ]( https://www.linkedin.com/company/intelligent-computing-solutions )[ ]( https://twitter.com/ICSIL ) [ Midwest Internet Exchange ]( http://www.midwest-ix.com/ ) [ ]( https://www.facebook.com/mdwestix )[ ]( https://www.linkedin.com/company/midwest-internet-exchange )[ ]( https://twitter.com/mdwestix ) [ The Brothers WISP ]( http://www.thebrotherswisp.com/ ) [ ]( https://www.facebook.com/thebrotherswisp )[ ]( https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg )
On 10/14/20 1:56 PM, Shawn L via NANOG wrote:
When I last spoke to them, it sounded like they were using a bunch of LAG groups based on ip address because they _really_ wanted to know how many ip addresses we had and what kind of traffic we would be expecting (eyeball networks, big data transport, etc).
Not why IP addresses are even an issue on an a "Layer 2" service. But then again, this is Cogent we're talking about here. -- inoc.net!rblayzor XMPP: rblayzor.AT.inoc.net PGP: https://pgp.inoc.net/rblayzor/
On 10/15/20 6:15 PM, Robert Blayzor wrote:
On 10/14/20 1:56 PM, Shawn L via NANOG wrote:
When I last spoke to them, it sounded like they were using a bunch of LAG groups based on ip address because they _really_ wanted to know how many ip addresses we had and what kind of traffic we would be expecting (eyeball networks, big data transport, etc).
Not why IP addresses are even an issue on an a "Layer 2" service. But then again, this is Cogent we're talking about here.
Hashing on LAGs within their core. Even otherwise fairly braindead Ethernet switches often hash on L3 to try and get more entropy. I hope they hash on L2 MAC, as well, but a pretty common scenario for an L2 interconnect only has one MAC on each end of the link, so that doesn't help much. They reeeeallly don't want all your traffic ending up on one side of a LAG. -- Brandon Martin
Thus spake Mike Hammett (nanog@ics-il.net) on Wed, Oct 14, 2020 at 12:36:39PM -0500:
Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable?
Be sure to ask if your circuit will face a 2G/flow cap, and examine if such a limitation would affect your expected traffic mix. https://www.reddit.com/r/networking/comments/iv0job/2gb_traffic_flow_cap_on_... Dale
Mike, Layer 2 is fine once it works. You will have to put up with whatever VLAN tags they pick, if you plan on having multiple virtual circuits on a 10G hub. They do like to see into the flows of traffic, as they only allow up to 2Gbits/flow, per there legacy infrastructure. If the circuit doesn't work on turn up (which is more than likely), you'll have to be abrasive with their NOC and demand escalations. IMO, if it's 1Gbit or less per circuit and can deal with ^, you're fine, otherwise look for another carrier. ----- Below is what I got from Cogent about their layer 2: We offer Ethernet over MPLS transport utilizing Cisco FAT Pseudowire (Flow Aware Transport). Our service is a fully protected service, so if we suffer a fiber cut or other disruption along the primary path, our IS-IS IP fast-reroute enabled MPLS backbone will swing all traffic over to another pre-determined path across our backbone with usually no packet loss or disruption in service. In order for our service to work correctly and provide the automatic redundancy, we need to verify that the traffic traversing the network can be hashed correctly by our routers. For this to happen, Cogent has to see the src-dst IP address or if you are running MPLS over the circuit, we need to see your MPLS labels. The hashing works by placing each flow of data on a separate 10GE or 100GE interface between the routers, so that traffic is evenly dispersed across all available capacity along the path. A flow is defined as a src-dst IP pair or a customer MPLS label, so the more IP pairs or MPLS labels, the better the traffic load-balances. Cogent has decided to impose a 2Gbps/flow restriction for our own traffic engineering purposes, which aim to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). The reason we have the limitation in place is for our own traffic engineering purposes, which aims to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). Since most uplinks between routers are Nx10GE or Nx100GE, we want to make sure that all customer traffic can be load-balanced across the uplink capacity evenly, which makes it easier to reroute traffic in the event of a fiber cut or other disruption. One would think that with 100GE interfaces, it would not be possible to overrun the interface if we allowed full 10Gbps/flow, however most 100GE interfaces, at the chip level are broken down into 10Gbps lanes and the interfaces do not have a way to easily determine that a lane through the interface is at capacity, so as new flows enter the interface, they could get allocated to a lane that is already full and therefore experience packet loss. So that we can complete our technical review for this request, need the following questions answered: 1 - What equipment will be directly connected to Cogent interface? 2 - How are the servers/equipment behind the edge device connected, GE or 10GE interfaces? 3 - Will you be doing any type of tunneling or load-balancing that would hide the src-dst IP addresses or MPLS labels of the servers/equipment? 4 - Will any single data flow (src-dst IP pair or MPLS label) be more than 2Gbps? 5 – What is the purpose of the connection? (Internet traffic backhaul, data center connectivity, replication, extending point-of-presence, etc..) 6 – Will you be running MACSec over our L2 service? 7 – Will you need to pass multiple VLANs and/or Jumbo frames? ---------- Ryan On Oct 14 2020, at 10:36 am, Mike Hammett <nanog@ics-il.net> wrote:
Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable?
----- Mike Hammett Intelligent Computing Solutions (http://www.ics-il.com/)
Midwest Internet Exchange (http://www.midwest-ix.com/)
The Brothers WISP (http://www.thebrotherswisp.com/)
I always heard this service was really Layer 3 disguised as Layer 2. ________________________________ From: NANOG <nanog-bounces+rod.beck=unitedcablecompany.com@nanog.org> on behalf of Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 7:54 PM To: Mike Hammett <nanog@ics-il.net> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2 Mike, Layer 2 is fine once it works. * You will have to put up with whatever VLAN tags they pick, if you plan on having multiple virtual circuits on a 10G hub. * They do like to see into the flows of traffic, as they only allow up to 2Gbits/flow, per there legacy infrastructure. * If the circuit doesn't work on turn up (which is more than likely), you'll have to be abrasive with their NOC and demand escalations. IMO, if it's 1Gbit or less per circuit and can deal with ^, you're fine, otherwise look for another carrier. ----- Below is what I got from Cogent about their layer 2: We offer Ethernet over MPLS transport utilizing Cisco FAT Pseudowire (Flow Aware Transport). Our service is a fully protected service, so if we suffer a fiber cut or other disruption along the primary path, our IS-IS IP fast-reroute enabled MPLS backbone will swing all traffic over to another pre-determined path across our backbone with usually no packet loss or disruption in service. In order for our service to work correctly and provide the automatic redundancy, we need to verify that the traffic traversing the network can be hashed correctly by our routers. For this to happen, Cogent has to see the src-dst IP address or if you are running MPLS over the circuit, we need to see your MPLS labels. The hashing works by placing each flow of data on a separate 10GE or 100GE interface between the routers, so that traffic is evenly dispersed across all available capacity along the path. A flow is defined as a src-dst IP pair or a customer MPLS label, so the more IP pairs or MPLS labels, the better the traffic load-balances. Cogent has decided to impose a 2Gbps/flow restriction for our own traffic engineering purposes, which aim to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). The reason we have the limitation in place is for our own traffic engineering purposes, which aims to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). Since most uplinks between routers are Nx10GE or Nx100GE, we want to make sure that all customer traffic can be load-balanced across the uplink capacity evenly, which makes it easier to reroute traffic in the event of a fiber cut or other disruption. One would think that with 100GE interfaces, it would not be possible to overrun the interface if we allowed full 10Gbps/flow, however most 100GE interfaces, at the chip level are broken down into 10Gbps lanes and the interfaces do not have a way to easily determine that a lane through the interface is at capacity, so as new flows enter the interface, they could get allocated to a lane that is already full and therefore experience packet loss. So that we can complete our technical review for this request, need the following questions answered: 1 - What equipment will be directly connected to Cogent interface? 2 - How are the servers/equipment behind the edge device connected, GE or 10GE interfaces? 3 - Will you be doing any type of tunneling or load-balancing that would hide the src-dst IP addresses or MPLS labels of the servers/equipment? 4 - Will any single data flow (src-dst IP pair or MPLS label) be more than 2Gbps? 5 – What is the purpose of the connection? (Internet traffic backhaul, data center connectivity, replication, extending point-of-presence, etc..) 6 – Will you be running MACSec over our L2 service? 7 – Will you need to pass multiple VLANs and/or Jumbo frames? ---------- Ryan On Oct 14 2020, at 10:36 am, Mike Hammett <nanog@ics-il.net> wrote: Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable? ----- Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/googleicon.png] [http://www.ics-il.com/images/linkedinicon.png] [http://www.ics-il.com/images/twittericon.png] Midwest Internet Exchange<http://www.midwest-ix.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/linkedinicon.png] [http://www.ics-il.com/images/twittericon.png] The Brothers WISP<http://www.thebrotherswisp.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/youtubeicon.png]
All carrier Ethernet services are tunnels provided by VPLS Psuedowire or VXLAN services. Did you really expect a VLAN to be layer 2 switched everywhere? Ryan On Oct 14 2020, at 11:03 am, Rod Beck <rod.beck@unitedcablecompany.com> wrote:
I always heard this service was really Layer 3 disguised as Layer 2.
From: NANOG <nanog-bounces+rod.beck=unitedcablecompany.com@nanog.org> on behalf of Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 7:54 PM To: Mike Hammett <nanog@ics-il.net> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2
Mike,
Layer 2 is fine once it works. You will have to put up with whatever VLAN tags they pick, if you plan on having multiple virtual circuits on a 10G hub. They do like to see into the flows of traffic, as they only allow up to 2Gbits/flow, per there legacy infrastructure.
If the circuit doesn't work on turn up (which is more than likely), you'll have to be abrasive with their NOC and demand escalations.
IMO, if it's 1Gbit or less per circuit and can deal with ^, you're fine, otherwise look for another carrier.
----- Below is what I got from Cogent about their layer 2: We offer Ethernet over MPLS transport utilizing Cisco FAT Pseudowire (Flow Aware Transport). Our service is a fully protected service, so if we suffer a fiber cut or other disruption along the primary path, our IS-IS IP fast-reroute enabled MPLS backbone will swing all traffic over to another pre-determined path across our backbone with usually no packet loss or disruption in service. In order for our service to work correctly and provide the automatic redundancy, we need to verify that the traffic traversing the network can be hashed correctly by our routers. For this to happen, Cogent has to see the src-dst IP address or if you are running MPLS over the circuit, we need to see your MPLS labels. The hashing works by placing each flow of data on a separate 10GE or 100GE interface between the routers, so that traffic is evenly dispersed across all available capacity along the path. A flow is defined as a src-dst IP pair or a customer MPLS label, so the more IP pairs or MPLS labels, the better the traffic load-balances. Cogent has decided to impose a 2Gbps/flow restriction for our own traffic engineering purposes, which aim to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). The reason we have the limitation in place is for our own traffic engineering purposes, which aims to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). Since most uplinks between routers are Nx10GE or Nx100GE, we want to make sure that all customer traffic can be load-balanced across the uplink capacity evenly, which makes it easier to reroute traffic in the event of a fiber cut or other disruption. One would think that with 100GE interfaces, it would not be possible to overrun the interface if we allowed full 10Gbps/flow, however most 100GE interfaces, at the chip level are broken down into 10Gbps lanes and the interfaces do not have a way to easily determine that a lane through the interface is at capacity, so as new flows enter the interface, they could get allocated to a lane that is already full and therefore experience packet loss. So that we can complete our technical review for this request, need the following questions answered: 1 - What equipment will be directly connected to Cogent interface? 2 - How are the servers/equipment behind the edge device connected, GE or 10GE interfaces? 3 - Will you be doing any type of tunneling or load-balancing that would hide the src-dst IP addresses or MPLS labels of the servers/equipment? 4 - Will any single data flow (src-dst IP pair or MPLS label) be more than 2Gbps? 5 – What is the purpose of the connection? (Internet traffic backhaul, data center connectivity, replication, extending point-of-presence, etc..) 6 – Will you be running MACSec over our L2 service? 7 – Will you need to pass multiple VLANs and/or Jumbo frames? ---------- Ryan On Oct 14 2020, at 10:36 am, Mike Hammett <nanog@ics-il.net> wrote:
Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable?
----- Mike Hammett Intelligent Computing Solutions (http://www.ics-il.com/)
Midwest Internet Exchange (http://www.midwest-ix.com/)
The Brothers WISP (http://www.thebrotherswisp.com/)
Hibernia was offering Switched Ethernet 'everywhere' long before it had a Layer 3 network. So I am a bit skeptical. In fact, in the 'old days' 2006-2011 we had a nice packet over SDH service that has all the performance of SDH with all the functionality of Ethernet. Very popular service. Unfortunately, management replaced with Switched Ethernet, which many customers distrusted because of potential overbooking issues. ________________________________ From: Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 8:22 PM To: Rod Beck <rod.beck@unitedcablecompany.com> Cc: Mike Hammett <nanog@ics-il.net>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2 All carrier Ethernet services are tunnels provided by VPLS Psuedowire or VXLAN services. Did you really expect a VLAN to be layer 2 switched everywhere? Ryan On Oct 14 2020, at 11:03 am, Rod Beck <rod.beck@unitedcablecompany.com> wrote: I always heard this service was really Layer 3 disguised as Layer 2. From: NANOG <nanog-bounces+rod.beck=unitedcablecompany.com@nanog.org> on behalf of Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 7:54 PM To: Mike Hammett <nanog@ics-il.net> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2 Mike, Layer 2 is fine once it works. * You will have to put up with whatever VLAN tags they pick, if you plan on having multiple virtual circuits on a 10G hub. * They do like to see into the flows of traffic, as they only allow up to 2Gbits/flow, per there legacy infrastructure. * If the circuit doesn't work on turn up (which is more than likely), you'll have to be abrasive with their NOC and demand escalations. IMO, if it's 1Gbit or less per circuit and can deal with ^, you're fine, otherwise look for another carrier. ----- Below is what I got from Cogent about their layer 2: We offer Ethernet over MPLS transport utilizing Cisco FAT Pseudowire (Flow Aware Transport). Our service is a fully protected service, so if we suffer a fiber cut or other disruption along the primary path, our IS-IS IP fast-reroute enabled MPLS backbone will swing all traffic over to another pre-determined path across our backbone with usually no packet loss or disruption in service. In order for our service to work correctly and provide the automatic redundancy, we need to verify that the traffic traversing the network can be hashed correctly by our routers. For this to happen, Cogent has to see the src-dst IP address or if you are running MPLS over the circuit, we need to see your MPLS labels. The hashing works by placing each flow of data on a separate 10GE or 100GE interface between the routers, so that traffic is evenly dispersed across all available capacity along the path. A flow is defined as a src-dst IP pair or a customer MPLS label, so the more IP pairs or MPLS labels, the better the traffic load-balances. Cogent has decided to impose a 2Gbps/flow restriction for our own traffic engineering purposes, which aim to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). The reason we have the limitation in place is for our own traffic engineering purposes, which aims to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). Since most uplinks between routers are Nx10GE or Nx100GE, we want to make sure that all customer traffic can be load-balanced across the uplink capacity evenly, which makes it easier to reroute traffic in the event of a fiber cut or other disruption. One would think that with 100GE interfaces, it would not be possible to overrun the interface if we allowed full 10Gbps/flow, however most 100GE interfaces, at the chip level are broken down into 10Gbps lanes and the interfaces do not have a way to easily determine that a lane through the interface is at capacity, so as new flows enter the interface, they could get allocated to a lane that is already full and therefore experience packet loss. So that we can complete our technical review for this request, need the following questions answered: 1 - What equipment will be directly connected to Cogent interface? 2 - How are the servers/equipment behind the edge device connected, GE or 10GE interfaces? 3 - Will you be doing any type of tunneling or load-balancing that would hide the src-dst IP addresses or MPLS labels of the servers/equipment? 4 - Will any single data flow (src-dst IP pair or MPLS label) be more than 2Gbps? 5 – What is the purpose of the connection? (Internet traffic backhaul, data center connectivity, replication, extending point-of-presence, etc..) 6 – Will you be running MACSec over our L2 service? 7 – Will you need to pass multiple VLANs and/or Jumbo frames? ---------- Ryan On Oct 14 2020, at 10:36 am, Mike Hammett <nanog@ics-il.net> wrote: Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable? ----- Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/googleicon.png] [http://www.ics-il.com/images/linkedinicon.png] [http://www.ics-il.com/images/twittericon.png] Midwest Internet Exchange<http://www.midwest-ix.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/linkedinicon.png] [http://www.ics-il.com/images/twittericon.png] The Brothers WISP<http://www.thebrotherswisp.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/youtubeicon.png]
Hibernia's implementation must of made scaling in terms of VLAN allocations, and programming all the equipment in path (with possibly no redundancy), very difficult to manage. Any link can be saturated no matter if it is layer 2 or 3. If you want dedicated bandwidth with an SLA, you have to pay for it. Ryan On Oct 14 2020, at 11:28 am, Rod Beck <rod.beck@unitedcablecompany.com> wrote:
Hibernia was offering Switched Ethernet 'everywhere' long before it had a Layer 3 network. So I am a bit skeptical. In fact, in the 'old days' 2006-2011 we had a nice packet over SDH service that has all the performance of SDH with all the functionality of Ethernet. Very popular service. Unfortunately, management replaced with Switched Ethernet, which many customers distrusted because of potential overbooking issues.
From: Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 8:22 PM To: Rod Beck <rod.beck@unitedcablecompany.com> Cc: Mike Hammett <nanog@ics-il.net>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2
All carrier Ethernet services are tunnels provided by VPLS Psuedowire or VXLAN services. Did you really expect a VLAN to be layer 2 switched everywhere?
Ryan On Oct 14 2020, at 11:03 am, Rod Beck <rod.beck@unitedcablecompany.com> wrote:
I always heard this service was really Layer 3 disguised as Layer 2.
From: NANOG <nanog-bounces+rod.beck=unitedcablecompany.com@nanog.org> on behalf of Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 7:54 PM To: Mike Hammett <nanog@ics-il.net> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2
Mike,
Layer 2 is fine once it works. You will have to put up with whatever VLAN tags they pick, if you plan on having multiple virtual circuits on a 10G hub. They do like to see into the flows of traffic, as they only allow up to 2Gbits/flow, per there legacy infrastructure.
If the circuit doesn't work on turn up (which is more than likely), you'll have to be abrasive with their NOC and demand escalations.
IMO, if it's 1Gbit or less per circuit and can deal with ^, you're fine, otherwise look for another carrier.
----- Below is what I got from Cogent about their layer 2: We offer Ethernet over MPLS transport utilizing Cisco FAT Pseudowire (Flow Aware Transport). Our service is a fully protected service, so if we suffer a fiber cut or other disruption along the primary path, our IS-IS IP fast-reroute enabled MPLS backbone will swing all traffic over to another pre-determined path across our backbone with usually no packet loss or disruption in service. In order for our service to work correctly and provide the automatic redundancy, we need to verify that the traffic traversing the network can be hashed correctly by our routers. For this to happen, Cogent has to see the src-dst IP address or if you are running MPLS over the circuit, we need to see your MPLS labels. The hashing works by placing each flow of data on a separate 10GE or 100GE interface between the routers, so that traffic is evenly dispersed across all available capacity along the path. A flow is defined as a src-dst IP pair or a customer MPLS label, so the more IP pairs or MPLS labels, the better the traffic load-balances. Cogent has decided to impose a 2Gbps/flow restriction for our own traffic engineering purposes, which aim to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). The reason we have the limitation in place is for our own traffic engineering purposes, which aims to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). Since most uplinks between routers are Nx10GE or Nx100GE, we want to make sure that all customer traffic can be load-balanced across the uplink capacity evenly, which makes it easier to reroute traffic in the event of a fiber cut or other disruption. One would think that with 100GE interfaces, it would not be possible to overrun the interface if we allowed full 10Gbps/flow, however most 100GE interfaces, at the chip level are broken down into 10Gbps lanes and the interfaces do not have a way to easily determine that a lane through the interface is at capacity, so as new flows enter the interface, they could get allocated to a lane that is already full and therefore experience packet loss. So that we can complete our technical review for this request, need the following questions answered: 1 - What equipment will be directly connected to Cogent interface? 2 - How are the servers/equipment behind the edge device connected, GE or 10GE interfaces? 3 - Will you be doing any type of tunneling or load-balancing that would hide the src-dst IP addresses or MPLS labels of the servers/equipment? 4 - Will any single data flow (src-dst IP pair or MPLS label) be more than 2Gbps? 5 – What is the purpose of the connection? (Internet traffic backhaul, data center connectivity, replication, extending point-of-presence, etc..) 6 – Will you be running MACSec over our L2 service? 7 – Will you need to pass multiple VLANs and/or Jumbo frames? ---------- Ryan On Oct 14 2020, at 10:36 am, Mike Hammett <nanog@ics-il.net> wrote:
Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable?
----- Mike Hammett Intelligent Computing Solutions (http://www.ics-il.com/)
Midwest Internet Exchange (http://www.midwest-ix.com/)
The Brothers WISP (http://www.thebrotherswisp.com/)
Look, you are looking for a fight, in which I have no interest. And no, a provider can't overbook a packet over SDH circuit. It is SDH performance. Pure dedicated bandwidth. You are correct that if you have to carve it up into a lots of VLANs, it would be a nightmare. But Hibernia was a true wholesale carrier providing backbone to clients, not links distributing traffic to lots of user end points. ________________________________ From: Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 8:34 PM To: Rod Beck <rod.beck@unitedcablecompany.com> Cc: Mike Hammett <nanog@ics-il.net>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2 Hibernia's implementation must of made scaling in terms of VLAN allocations, and programming all the equipment in path (with possibly no redundancy), very difficult to manage. Any link can be saturated no matter if it is layer 2 or 3. If you want dedicated bandwidth with an SLA, you have to pay for it. Ryan On Oct 14 2020, at 11:28 am, Rod Beck <rod.beck@unitedcablecompany.com> wrote: Hibernia was offering Switched Ethernet 'everywhere' long before it had a Layer 3 network. So I am a bit skeptical. In fact, in the 'old days' 2006-2011 we had a nice packet over SDH service that has all the performance of SDH with all the functionality of Ethernet. Very popular service. Unfortunately, management replaced with Switched Ethernet, which many customers distrusted because of potential overbooking issues. From: Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 8:22 PM To: Rod Beck <rod.beck@unitedcablecompany.com> Cc: Mike Hammett <nanog@ics-il.net>; nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2 All carrier Ethernet services are tunnels provided by VPLS Psuedowire or VXLAN services. Did you really expect a VLAN to be layer 2 switched everywhere? Ryan On Oct 14 2020, at 11:03 am, Rod Beck <rod.beck@unitedcablecompany.com> wrote: I always heard this service was really Layer 3 disguised as Layer 2. From: NANOG <nanog-bounces+rod.beck=unitedcablecompany.com@nanog.org> on behalf of Ryan Hamel <ryan@rkhtech.org> Sent: Wednesday, October 14, 2020 7:54 PM To: Mike Hammett <nanog@ics-il.net> Cc: nanog@nanog.org <nanog@nanog.org> Subject: Re: Cogent Layer 2 Mike, Layer 2 is fine once it works. * You will have to put up with whatever VLAN tags they pick, if you plan on having multiple virtual circuits on a 10G hub. * They do like to see into the flows of traffic, as they only allow up to 2Gbits/flow, per there legacy infrastructure. * If the circuit doesn't work on turn up (which is more than likely), you'll have to be abrasive with their NOC and demand escalations. IMO, if it's 1Gbit or less per circuit and can deal with ^, you're fine, otherwise look for another carrier. ----- Below is what I got from Cogent about their layer 2: We offer Ethernet over MPLS transport utilizing Cisco FAT Pseudowire (Flow Aware Transport). Our service is a fully protected service, so if we suffer a fiber cut or other disruption along the primary path, our IS-IS IP fast-reroute enabled MPLS backbone will swing all traffic over to another pre-determined path across our backbone with usually no packet loss or disruption in service. In order for our service to work correctly and provide the automatic redundancy, we need to verify that the traffic traversing the network can be hashed correctly by our routers. For this to happen, Cogent has to see the src-dst IP address or if you are running MPLS over the circuit, we need to see your MPLS labels. The hashing works by placing each flow of data on a separate 10GE or 100GE interface between the routers, so that traffic is evenly dispersed across all available capacity along the path. A flow is defined as a src-dst IP pair or a customer MPLS label, so the more IP pairs or MPLS labels, the better the traffic load-balances. Cogent has decided to impose a 2Gbps/flow restriction for our own traffic engineering purposes, which aim to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). The reason we have the limitation in place is for our own traffic engineering purposes, which aims to make sure that no single customer can overrun a 10GE interface anywhere on our network (since we do not sell 10GE Wave services). Since most uplinks between routers are Nx10GE or Nx100GE, we want to make sure that all customer traffic can be load-balanced across the uplink capacity evenly, which makes it easier to reroute traffic in the event of a fiber cut or other disruption. One would think that with 100GE interfaces, it would not be possible to overrun the interface if we allowed full 10Gbps/flow, however most 100GE interfaces, at the chip level are broken down into 10Gbps lanes and the interfaces do not have a way to easily determine that a lane through the interface is at capacity, so as new flows enter the interface, they could get allocated to a lane that is already full and therefore experience packet loss. So that we can complete our technical review for this request, need the following questions answered: 1 - What equipment will be directly connected to Cogent interface? 2 - How are the servers/equipment behind the edge device connected, GE or 10GE interfaces? 3 - Will you be doing any type of tunneling or load-balancing that would hide the src-dst IP addresses or MPLS labels of the servers/equipment? 4 - Will any single data flow (src-dst IP pair or MPLS label) be more than 2Gbps? 5 – What is the purpose of the connection? (Internet traffic backhaul, data center connectivity, replication, extending point-of-presence, etc..) 6 – Will you be running MACSec over our L2 service? 7 – Will you need to pass multiple VLANs and/or Jumbo frames? ---------- Ryan On Oct 14 2020, at 10:36 am, Mike Hammett <nanog@ics-il.net> wrote: Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable? ----- Mike Hammett Intelligent Computing Solutions<http://www.ics-il.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/googleicon.png] [http://www.ics-il.com/images/linkedinicon.png] [http://www.ics-il.com/images/twittericon.png] Midwest Internet Exchange<http://www.midwest-ix.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/linkedinicon.png] [http://www.ics-il.com/images/twittericon.png] The Brothers WISP<http://www.thebrotherswisp.com/> [http://www.ics-il.com/images/fbicon.png] [http://www.ics-il.com/images/youtubeicon.png]
On Wed, Oct 14, 2020, at 20:38, Rod Beck wrote:
You are correct that if you have to carve it up into a lots of VLANs, it would be a nightmare. But Hibernia was a true wholesale carrier providing backbone to clients, not links distributing traffic to lots of user end points.
The fact that there was a "switched Ethernet" commercial service doesn't mean that the underlying transport was really "switched ethernet" end-to-end. Ethernet over MPLS is a VERY old concept (VLL, VPWS, VPLS, lately EVPN), and these days Ethernet over VXLAN is becoming more and more popular (mostly EVPN). A carrier using a pure, unencapsulated, end-to-end ethernet for transport over 1000s of km is (and was for at least 15 years) a disaster waiting to happen. Almost all ethernet services (switched, not switched or otherwise) use some form of encapsulation (IP or MPLS, see above) these days.
On Wed, Oct 14, 2020 at 10:54:49AM -0700, Ryan Hamel wrote:
One would think that with 100GE interfaces, it would not be possible to overrun the interface if we allowed full 10Gbps/flow, however most 100GE interfaces, at the chip level are broken down into 10Gbps lanes and the interfaces do not have a way to easily determine that a lane through the interface is at capacity, so as new flows enter the interface, they could get allocated to a lane that is already full and therefore experience packet loss.
This does raise an interesting point regarding ASR9K platform. IIRC, older Typhoon/NP4c 100GE cards (e.g. Juggernaut A9K-2X100GE-TR) had problems where on a 100GE port, you can't sustain more than 10-12 Gbps per individual flow. You had to hash to achieve aggregate bandwidth, which became a significant issue if you were trying to transport 10G L2 pseudowires with limited flow visibility. I'm not aware that it is an issue any longer on newer Tomahawk/NP5c cards? James
I haven't heard any concerns with reliability, on-net performance (aside from 2 gig flow limit) or other such things. Do they generally deliver well in those regards? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Mike Hammett" <nanog@ics-il.net> To: nanog@nanog.org Sent: Wednesday, October 14, 2020 12:36:39 PM Subject: Cogent Layer 2 Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable? ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP
Yep. Make sure you run BFD with your peering protocols, to catch outages very quickly. On Oct 14 2020, at 12:47 pm, Mike Hammett <nanog@ics-il.net> wrote:
I haven't heard any concerns with reliability, on-net performance (aside from 2 gig flow limit) or other such things. Do they generally deliver well in those regards?
----- Mike Hammett Intelligent Computing Solutions (http://www.ics-il.com/)
Midwest Internet Exchange (http://www.midwest-ix.com/)
The Brothers WISP (http://www.thebrotherswisp.com/)
From: "Mike Hammett" <nanog@ics-il.net> To: nanog@nanog.org Sent: Wednesday, October 14, 2020 12:36:39 PM Subject: Cogent Layer 2
Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable?
----- Mike Hammett Intelligent Computing Solutions (http://www.ics-il.com/)
Midwest Internet Exchange (http://www.midwest-ix.com/)
The Brothers WISP (http://www.thebrotherswisp.com/)
On Thu, 15 Oct 2020 at 09:11, Ryan Hamel <ryan@rkhtech.org> wrote: Yep. Make sure you run BFD with your peering protocols, to catch outages
very quickly.
Make sure you get higher availability with BFD than without it, it is easy to get this wrong and end up losing availability. First issue is that BFD has quite a lot of bug surface, because unlike most of your control-plane protocols, BFD is implemented in your NPU ucode when done right. We've had the entire linecard down on ASR9k due to BFD, their BFD-of-death packet you can send over the internet to crash JNPR FPC. When done in a control-plane, poor scheduling can cause false positives more often than it protects from actual outages (CISCO7600). In a world where BFD is perfect you still need to consider what you are protecting yourself from, so you bought Martini from someone and run your backbone over that Martini. What is an outage? Is your provider IGP rerouting due to backbone outage an outage to you? Or would you rather the provider convergees their network and you don't converge, you take the outage? If provider rerouting is not an outage, you need to know what their SLA is regarding rerouting time and make BFD less aggressive than that. If provider rerouting is an outage, you can of course run as aggressive timers as you want, but you probably have lower availability than without BFD. Also, don't add complexity to solve problems you don't have. If you don't know if BFD improved your availability, you didn't need it. Networking is full of belief practices, we do things because we believe they help and faux data is used often to dress the beliefs as science. The problem space tends to be complex and good quality data is sparse to come by, we do necessarily fly a lot by the seat of our pants, if we admit or not. My belief is the majority of BFD implementations in real life on average reduce availability, my belief is you need frequently failing link which does not propagate link-down to reliability improve availability by deploying BFD. -- ++ytti
Saku, My experience with multiple carriers is that reroutes happen in under a minute but rarely happen, I also have redundant backup circuits to another datacenter, so no traffic is truly lost. If an outage lasts longer than 5 minutes, or it's flapping very frequently, then I call the carrier. Last mile carriers install CPE equipment at the sites, which makes BFD a requirement to account for the fiber uplink on it going down, or an issue upstream. As for security vulnerabilities, none can be leveraged if they are using internal IPs, and if not, a quick ACL can drop BFD traffic from unknown sources the same way BGP sessions are filtered. In Juniper speak, the ACL would look like: (under policy-options) prefix-list bgp_hosts { apply-path "protocols bgp group <*> neighbor <*>"; } (under firewall family inet(6) filter mgmt_acl) term allow_bfd { from { protocol udp; destination-port [ 3784 3785 4784 ]; source-prefix-list bgp_hosts; } then accept; } term deny_bfd { from { protocol udp; destination-port [ 3784 3785 4784 ]; } then discard; } Ryan On Oct 14 2020, at 11:29 pm, Saku Ytti <saku@ytti.fi> wrote:
On Thu, 15 Oct 2020 at 09:11, Ryan Hamel <ryan@rkhtech.org (mailto:ryan@rkhtech.org)> wrote:
Yep. Make sure you run BFD with your peering protocols, to catch outages very quickly.
Make sure you get higher availability with BFD than without it, it is easy to get this wrong and end up losing availability.
First issue is that BFD has quite a lot of bug surface, because unlike most of your control-plane protocols, BFD is implemented in your NPU ucode when done right. We've had the entire linecard down on ASR9k due to BFD, their BFD-of-death packet you can send over the internet to crash JNPR FPC. When done in a control-plane, poor scheduling can cause false positives more often than it protects from actual outages (CISCO7600).
In a world where BFD is perfect you still need to consider what you are protecting yourself from, so you bought Martini from someone and run your backbone over that Martini. What is an outage? Is your provider IGP rerouting due to backbone outage an outage to you? Or would you rather the provider convergees their network and you don't converge, you take the outage? If provider rerouting is not an outage, you need to know what their SLA is regarding rerouting time and make BFD less aggressive than that. If provider rerouting is an outage, you can of course run as aggressive timers as you want, but you probably have lower availability than without BFD.
Also, don't add complexity to solve problems you don't have. If you don't know if BFD improved your availability, you didn't need it. Networking is full of belief practices, we do things because we believe they help and faux data is used often to dress the beliefs as science. The problem space tends to be complex and good quality data is sparse to come by, we do necessarily fly a lot by the seat of our pants, if we admit or not. My belief is the majority of BFD implementations in real life on average reduce availability, my belief is you need frequently failing link which does not propagate link-down to reliability improve availability by deploying BFD.
-- ++ytti
On Thu, 15 Oct 2020 at 10:28, Ryan Hamel <ryan@rkhtech.org> wrote:
My experience with multiple carriers is that reroutes happen in under a minute but rarely happen, I also have redundant backup circuits to another datacenter, so no traffic is truly lost. If an outage lasts longer than 5 minutes, or it's flapping very frequently, then I call the carrier. Last mile carriers install CPE equipment at the sites, which makes BFD a requirement to account for the fiber uplink on it going down, or an issue upstream.
I think I may have spoken ambiguously and confusingly based on that statement. Rerouting inside operator network, such as their LSR-LSR link dropping is ostensibly invisible to the customer, can be tens of milliseconds outage can be 10s outage. Do you want your martini emulated backbone link to fail when operator reroutes their own LSR-LSR link failure?
As for security vulnerabilities, none can be leveraged if they are using internal IPs, and if not, a quick ACL can drop BFD traffic from unknown sources the same way BGP sessions are filtered.
In Juniper speak, the ACL would look like:
term deny_bfd { from { protocol udp; destination-port [ 3784 3785 4784 ]; } then discard;
So you're dropping in every edge all UDP packets towards these three ports? Your customers may not appreciate. -- ++ytti
Do you want your martini emulated backbone link to fail when operator reroutes their own LSR-LSR link failure? As I said, it's an acceptable loss for my employers network, as we have a BGP failover mechanism in place that works perfectly.
So you're dropping in every edge all UDP packets towards these three ports? Your customers may not appreciate. You must not be familiar with JUNOS' ACL handling. This would be applied to interface lo0, which is specifically for control planes. No data plane traffic to customers would be hit.
Ryan On Oct 15 2020, at 1:03 am, Saku Ytti <saku@ytti.fi> wrote:
On Thu, 15 Oct 2020 at 10:28, Ryan Hamel <ryan@rkhtech.org> wrote:
My experience with multiple carriers is that reroutes happen in under a minute but rarely happen, I also have redundant backup circuits to another datacenter, so no traffic is truly lost. If an outage lasts longer than 5 minutes, or it's flapping very frequently, then I call the carrier. Last mile carriers install CPE equipment at the sites, which makes BFD a requirement to account for the fiber uplink on it going down, or an issue upstream. I think I may have spoken ambiguously and confusingly based on that statement. Rerouting inside operator network, such as their LSR-LSR link dropping is ostensibly invisible to the customer, can be tens of milliseconds outage can be 10s outage. Do you want your martini emulated backbone link to fail when operator reroutes their own LSR-LSR link failure?
As for security vulnerabilities, none can be leveraged if they are using internal IPs, and if not, a quick ACL can drop BFD traffic from unknown sources the same way BGP sessions are filtered. In Juniper speak, the ACL would look like: term deny_bfd { from { protocol udp; destination-port [ 3784 3785 4784 ]; } then discard;
So you're dropping in every edge all UDP packets towards these three ports? Your customers may not appreciate.
-- ++ytti
On Thu, 15 Oct 2020 at 17:49, Ryan Hamel <ryan@rkhtech.org> wrote:
So you're dropping in every edge all UDP packets towards these three ports? Your customers may not appreciate. You must not be familiar with JUNOS' ACL handling. This would be applied to interface lo0, which is specifically for control planes. No data plane traffic to customers would be hit.
I'm sure there are some gaps in knowledge at play here. There are many reasons why packets hit the control-plane and not be subject to lo0 filter, for example TTL expiry. Also, as I tried to communicate with little success, BFD is implemented in NPU ucode and you are subjected to NPU ucode bugs. The bug I'm talking about, does not require you using or configuring BFD, it just needs NPU to parse it, and your FPC is gone. Same deal with Cisco issue I'm talking about. I've not yet seen single non-broken junos control-plane protection, everyone has terribly poorly written lo0 filters, no one has any idea how to configure ddos-protection. If you some canonical sources to do this, like Cymru or Juniper's MX book as source, you'll get it all wrong, as they both contain trivial and naive errors. But if you do manage to configure lo0 and ddos-protection correctly, you're still exposed to wide array of packet-of-death style vectors. Just yesterday on Junos SIRT-day bug where your KRT will become wedged if you sample (IPFIX) specifically crafted packet, this will be transit packet. Problems become increasingly simple the less you understand them. -- ++ytti
Some of their routers in Houston are blocking random flows for us since Friday. Support has been contacted and they claim nothing is wrong. It is still broken today. On Wed, Oct 14, 2020 at 10:38 AM Mike Hammett <nanog@ics-il.net> wrote:
Are any legitimate beefs with Cogent limited to their IP policies, BGP session charges, and peering disputes? Meaning, would using them for layer 2 be reasonable?
----- Mike Hammett Intelligent Computing Solutions <http://www.ics-il.com/> <https://www.facebook.com/ICSIL> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> <https://www.linkedin.com/company/intelligent-computing-solutions> <https://twitter.com/ICSIL> Midwest Internet Exchange <http://www.midwest-ix.com/> <https://www.facebook.com/mdwestix> <https://www.linkedin.com/company/midwest-internet-exchange> <https://twitter.com/mdwestix> The Brothers WISP <http://www.thebrotherswisp.com/> <https://www.facebook.com/thebrotherswisp> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
participants (12)
-
Bottiger -
Brandon Martin -
Dale W. Carder -
David Hubbard -
James Jun -
Mike Hammett -
Radu-Adrian Feurdean -
Robert Blayzor -
Rod Beck -
Ryan Hamel -
Saku Ytti -
Shawn L