Cogent <-> Verizon peering congestion
Hello, For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon Host Loss% Snt Last Avg Best Wrst StDev 1. router.garlic.com 0.0% 29 0.3 6.1 0.2 160.6 29.7 2. vl203.mag03.sfo01.atlas.cogentco.com 0.0% 29 2.2 8.1 2.1 161.1 29.5 3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0% 29 2.9 2.7 2.4 3.6 0.2 4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0% 29 4.1 4.0 3.7 4.8 0.2 5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0% 29 4.5 4.7 4.3 5.5 0.3 6. verizon.sjc03.atlas.cogentco.com 22.2% 28 169.3 171.5 168.1 193.5 6.9 7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0% 28 205.8 180.6 171.6 271.6 24.8 8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3% 28 172.3 177.5 171.7 250.8 18.3 9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0% 28 197.9 197.6 195.5 199.2 0.8 We have smokeping's from our side showing 30%+ packet loss from us (AS4307) to Verizon. All I have gotten from Cogent is a canned response: --- The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon. While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints. Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements. Our peering engineers will continue to address this on a daily basis until resolved. --- It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to checking their email, etc. Has anyone else been experiencing these issues? Or does anyone have more information that what Cogent provided me in their canned statement? -Bobby
I also see major congestion from Cogent to VZ. Amongst other major networks. http://i.imgur.com/1z2ZGOr.png On Tue, Feb 4, 2014 at 6:44 PM, Robert Glover <robertg@garlic.com> wrote:
Hello,
For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon
Host Loss% Snt Last Avg Best Wrst StDev 1. router.garlic.com 0.0% 29 0.3 6.1 0.2 160.6 29.7 2. vl203.mag03.sfo01.atlas.cogentco.com 0.0% 29 2.2 8.1 2.1 161.1 29.5 3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0% 29 2.9 2.7 2.4 3.6 0.2 4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0% 29 4.1 4.0 3.7 4.8 0.2 5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0% 29 4.5 4.7 4.3 5.5 0.3 6. verizon.sjc03.atlas.cogentco.com 22.2% 28 169.3 171.5 168.1 193.5 6.9 7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0% 28 205.8 180.6 171.6 271.6 24.8 8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3% 28 172.3 177.5 171.7 250.8 18.3 9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0% 28 197.9 197.6 195.5 199.2 0.8
We have smokeping's from our side showing 30%+ packet loss from us (AS4307) to Verizon.
All I have gotten from Cogent is a canned response:
--- The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon. While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints. Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements. Our peering engineers will continue to address this on a daily basis until resolved. ---
It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to checking their email, etc.
Has anyone else been experiencing these issues? Or does anyone have more information that what Cogent provided me in their canned statement?
-Bobby
I've seen some Cogent-Sprint congestion today also. About 10% PL at the link. On 2/4/2014 6:29 PM, Edward Roels wrote:
I also see major congestion from Cogent to VZ. Amongst other major networks.
http://i.imgur.com/1z2ZGOr.png
On Tue, Feb 4, 2014 at 6:44 PM, Robert Glover <robertg@garlic.com> wrote:
Hello,
For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon
Host Loss% Snt Last Avg Best Wrst StDev 1. router.garlic.com 0.0% 29 0.3 6.1 0.2 160.6 29.7 2. vl203.mag03.sfo01.atlas.cogentco.com 0.0% 29 2.2 8.1 2.1 161.1 29.5 3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0% 29 2.9 2.7 2.4 3.6 0.2 4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0% 29 4.1 4.0 3.7 4.8 0.2 5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0% 29 4.5 4.7 4.3 5.5 0.3 6. verizon.sjc03.atlas.cogentco.com 22.2% 28 169.3 171.5 168.1 193.5 6.9 7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0% 28 205.8 180.6 171.6 271.6 24.8 8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3% 28 172.3 177.5 171.7 250.8 18.3 9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0% 28 197.9 197.6 195.5 199.2 0.8
We have smokeping's from our side showing 30%+ packet loss from us (AS4307) to Verizon.
All I have gotten from Cogent is a canned response:
--- The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon. While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints. Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements. Our peering engineers will continue to address this on a daily basis until resolved. ---
It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to checking their email, etc.
Has anyone else been experiencing these issues? Or does anyone have more information that what Cogent provided me in their canned statement?
-Bobby
Yep. Major oversub in our area (LA/SD) - worse for us is same with VZ <-> L3! James Laszko Mythos Technology Inc jamesl@mythostech.com Sent from my iPhone
On Feb 4, 2014, at 3:46 PM, "Robert Glover" <robertg@garlic.com> wrote:
Hello,
For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon
Host Loss% Snt Last Avg Best Wrst StDev 1. router.garlic.com 0.0% 29 0.3 6.1 0.2 160.6 29.7 2. vl203.mag03.sfo01.atlas.cogentco.com 0.0% 29 2.2 8.1 2.1 161.1 29.5 3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0% 29 2.9 2.7 2.4 3.6 0.2 4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0% 29 4.1 4.0 3.7 4.8 0.2 5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0% 29 4.5 4.7 4.3 5.5 0.3 6. verizon.sjc03.atlas.cogentco.com 22.2% 28 169.3 171.5 168.1 193.5 6.9 7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0% 28 205.8 180.6 171.6 271.6 24.8 8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3% 28 172.3 177.5 171.7 250.8 18.3 9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0% 28 197.9 197.6 195.5 199.2 0.8
We have smokeping's from our side showing 30%+ packet loss from us (AS4307) to Verizon.
All I have gotten from Cogent is a canned response:
--- The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon. While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints. Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements. Our peering engineers will continue to address this on a daily basis until resolved. ---
It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to checking their email, etc.
Has anyone else been experiencing these issues? Or does anyone have more information that what Cogent provided me in their canned statement?
-Bobby
Cogent support uses the same response when inquiring about Comcast, CenturyLink, Tata, AT&T etc. If the "Tier 1s" are really keeping each other congested, are they not creating an environment where you have to buy from each of them to have a chance at congestion free paths? Or peer around them. On Tue, Feb 4, 2014 at 6:44 PM, Robert Glover <robertg@garlic.com> wrote:
Hello,
For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon
Host Loss% Snt Last Avg Best Wrst StDev 1. router.garlic.com 0.0% 29 0.3 6.1 0.2 160.6 29.7 2. vl203.mag03.sfo01.atlas.cogentco.com 0.0% 29 2.2 8.1 2.1 161.1 29.5 3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0% 29 2.9 2.7 2.4 3.6 0.2 4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0% 29 4.1 4.0 3.7 4.8 0.2 5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0% 29 4.5 4.7 4.3 5.5 0.3 6. verizon.sjc03.atlas.cogentco.com 22.2% 28 169.3 171.5 168.1 193.5 6.9 7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0% 28 205.8 180.6 171.6 271.6 24.8 8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3% 28 172.3 177.5 171.7 250.8 18.3 9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0% 28 197.9 197.6 195.5 199.2 0.8
We have smokeping's from our side showing 30%+ packet loss from us (AS4307) to Verizon.
All I have gotten from Cogent is a canned response:
--- The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon. While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints. Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements. Our peering engineers will continue to address this on a daily basis until resolved. ---
It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to checking their email, etc.
Has anyone else been experiencing these issues? Or does anyone have more information that what Cogent provided me in their canned statement?
-Bobby
Just to make something clear. I do not own any stock, interest or have any official relationship with Verizon or Cogent. The opinions expressed are mine and mine alone as I have come to understand some of the relationship without the aid of any privileged information. On Tue, Feb 4, 2014 at 8:08 PM, Edward Roels <edwardroels@gmail.com> wrote:
Cogent support uses the same response when inquiring about Comcast, CenturyLink, Tata, AT&T etc.
Yup, certainly a common thread there. It wouldn't take a lot of digging to determine that those "peers" are most likely compensated in some way shape or form. Hence the lack of response by the remote "peer" to upgrade what would most likely be a paid peer.
If the "Tier 1s" are really keeping each other congested, are they not creating an environment where you have to buy from each of them to have a chance at congestion free paths? Or peer around them.
The term tier 1 is a marketing term. The last time I looked Cogent was default free, but certainly not settlement free. Good luck peering around a lot of the networks in which they have congestion from. The only real way would be to order additional capacity, buy transit or live with the situation long enough for customers complain to the remote network. If that doesn't work then force a partition with the end objective of Cogent to change from a compensated peer to a settlement free peer. It's a fun game, it's unfortunate that customer flows have to be used to force it. Let me help translate the canned response.
All I have gotten from Cogent is a canned response:
--- The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon.
We are sorry that we transmit more traffic to Verizon than we are willing to pay Verizon for.
We have repeatedly requested augments to these congestion points and hope Verizon will comply soon.
We have demanded that the compensated peers be converted to settlement free peers with greater capacity, with little to no response.
While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints.
Our CEO is aware that this will cost money but is unwilling to pay for additional bandwidth, so we are in a stalemate and want customers to complain to Verizon in the hopes that the squeaky wheel will get the grease or in other terms, free capacity.
Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements.
Since the connectivity is compensated, Verizon refuses to provide additional bandwidth at an acceptable (free) cost. So, no money equals no additional customer capacity.
Our peering engineers will continue to address this on a daily basis until resolved.
Our peering folks will continue to pester Verizon's non-commercial folks by requesting settlement free peers but until enough people complain to Verizon the requests will fall on deaf ears. thanks, charles
FYI, here's the latest response from Cogent when I prodded them about the issue (just received this about 30 minutes ago: --- The issue on this peer involves a high amount of traffic being sent to Cogent from the Verizon network. In order to resolve the congestion on that peer, Verizon needs to engineer thier routing to another peering point with Cogent that has more available space for the traffic. We have reached out to Verizon for assistance and are still awaiting a response from their peering engineers. We will continue to reach out to them until they respond. --- Sounds like more "fluff" to me :-/ -Bobby
He received a very similar response from Level3 when we complained to them about the issue. "We have contacted VZ with no response" We see pings go from 8ms to 110ms+ from our office FIOS to one of our data centers in San Diego.... LAX handoffs of FIOS traffic to Cogent & Level3 appears the choke points to us. James Laszko Mythos Technlogy Inc jamesl@mythostech.com Sent from my iPhone
On Feb 4, 2014, at 5:35 PM, "Robert Glover" <robertg@garlic.com> wrote:
FYI, here's the latest response from Cogent when I prodded them about the issue (just received this about 30 minutes ago:
--- The issue on this peer involves a high amount of traffic being sent to Cogent from the Verizon network. In order to resolve the congestion on that peer, Verizon needs to engineer thier routing to another peering point with Cogent that has more available space for the traffic. We have reached out to Verizon for assistance and are still awaiting a response from their peering engineers. We will continue to reach out to them until they respond. ---
Sounds like more "fluff" to me :-/
-Bobby
We've been having trouble with congestion between Verizon and Cogent in Chicago since May 2013. We had to move some traffic off of our Verizon connection to get around it. Verizon has apparently had an internal ticket open for the problem since February 2013. Their response in August 2013 was "Cogent is designing an upgrade to turn up the bandwidth". Their response a few weeks ago was "Verizon is working with Cogent on increasing more capacity there is no deadline of completion, issue ongoing since 2013 possible completion in 2014". In other words, they're blaming Cogent. Ben Bridges SpringNet -----Original Message----- From: Robert Glover [mailto:robertg@garlic.com] Sent: Tuesday, February 04, 2014 5:44 PM To: nanog@nanog.org Subject: Cogent <-> Verizon peering congestion Hello, For the last several months, we have been tracking a congestion issue between Cogent <-> Verizon Host Loss% Snt Last Avg Best Wrst StDev 1. router.garlic.com 0.0% 29 0.3 6.1 0.2 160.6 29.7 2. vl203.mag03.sfo01.atlas.cogentco.com 0.0% 29 2.2 8.1 2.1 161.1 29.5 3. te0-0-0-14.ccr22.sfo01.atlas.cogentco.com 0.0% 29 2.9 2.7 2.4 3.6 0.2 4. be2165.ccr22.sjc01.atlas.cogentco.com 0.0% 29 4.1 4.0 3.7 4.8 0.2 5. be2047.ccr21.sjc03.atlas.cogentco.com 0.0% 29 4.5 4.7 4.3 5.5 0.3 6. verizon.sjc03.atlas.cogentco.com 22.2% 28 169.3 171.5 168.1 193.5 6.9 7. so-1-0-0-0.SJC01-CORE-RTR2.verizon-gni.net 37.0% 28 205.8 180.6 171.6 271.6 24.8 8. A12-0-135.SNFCCA-DSL-01.verizon-gni.net 33.3% 28 172.3 177.5 171.7 250.8 18.3 9. pool-71-116-122-235.snfcca.btas.verizon.net 25.0% 28 197.9 197.6 195.5 199.2 0.8 We have smokeping's from our side showing 30%+ packet loss from us (AS4307) to Verizon. All I have gotten from Cogent is a canned response: --- The latency and/or packet loss that you are experiencing to this destination is due to occasional high traffic with Verizon. We have repeatedly requested augments to these congestion points and hope Verizon will comply soon. While this has been escalated internally to the CEO level, we encourage you to also contact Verizon customer support with your concerns and complaints. Their delay is a major impediment to internet traffic overall and contrary to net neutrality requirements. Our peering engineers will continue to address this on a daily basis until resolved. --- It seems to have gotten a lot worse in recent days, to the point where we have customers who are trying to access us from Verizon's network (i.e. they have Verizon DSL, or via Verizon 3G/4GLTE) complaining they are having a very hard to checking their email, etc. Has anyone else been experiencing these issues? Or does anyone have more information that what Cogent provided me in their canned statement? -Bobby
participants (6)
-
Aaron -
Ben Bridges -
Charles Gucker -
Edward Roels -
James Laszko -
Robert Glover