Hi, We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal. We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile. My questions are: - Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine) - Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)? - How are peoples' experiences with L3 vs TWC? Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list. Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome. Thanks, Adam
We have had Cogent over Verizon's Fiber for more than a few years now. Cogent goes down once at year at minimum. They had 2 outages in a single day a couple days ago in Northern NJ. One in the AM "..caused by a power outage in a vendor data center where Cogent is collocated." They went on to have another outage at around 9:30 PM on the same day for which I'm still waiting for an RFO. During this outage, they still were advertising our BGP routes so we didn't fail over to our 2nd provider. I notice that happens alot with them. When they go down, they still advertise your routes. As far as price goes, for us Cogent is cheap but Lightpath is cheaper. Our college is kind of far from things so we don't have a lot of outside fiber coming. The last mile fiber for both of our connections are different from our Internet providers. I've never had a big issue with the two working with each other. The only issue we had is I suspected we weren't getting as much bandwidth as we paid for. They had to work out where the policer and/or bottle neck was. This is the only issue we had in 5 years with this set up and it got resolved. IME, when there is a full outage, it's always been clear who the responsible party is. On 2/6/2014 10:17 AM, Adam Greene wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
Cogent always has the cheapest rates but they also have the most peering disputes of any operator. I've seen intra-data center hops between cogent and Verizon take over 150ms. As with all things Internet, your mileage may vary. I would not put something with a 5 9'a uptime requirement on cogent without a failover circuit. For less sensitive applications it seems like a win. The Internet is both incredibly robust and fragile simultaneously. Cheers, Joshua Sent from my iPhone On Feb 6, 2014, at 8:06 AM, "Vlade Ristevski" <vristevs@ramapo.edu> wrote:
We have had Cogent over Verizon's Fiber for more than a few years now. Cogent goes down once at year at minimum. They had 2 outages in a single day a couple days ago in Northern NJ. One in the AM "..caused by a power outage in a vendor data center where Cogent is collocated." They went on to have another outage at around 9:30 PM on the same day for which I'm still waiting for an RFO. During this outage, they still were advertising our BGP routes so we didn't fail over to our 2nd provider. I notice that happens alot with them. When they go down, they still advertise your routes.
As far as price goes, for us Cogent is cheap but Lightpath is cheaper.
Our college is kind of far from things so we don't have a lot of outside fiber coming. The last mile fiber for both of our connections are different from our Internet providers. I've never had a big issue with the two working with each other. The only issue we had is I suspected we weren't getting as much bandwidth as we paid for. They had to work out where the policer and/or bottle neck was. This is the only issue we had in 5 years with this set up and it got resolved. IME, when there is a full outage, it's always been clear who the responsible party is.
On 2/6/2014 10:17 AM, Adam Greene wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
On Feb 6, 2014, at 11:22, Joshua Goldbard <j@2600hz.com> wrote:
Cogent always has the cheapest rates
Objectively, provably false. -- TTFN, patrick
but they also have the most peering disputes of any operator. I've seen intra-data center hops between cogent and Verizon take over 150ms.
As with all things Internet, your mileage may vary. I would not put something with a 5 9'a uptime requirement on cogent without a failover circuit. For less sensitive applications it seems like a win.
The Internet is both incredibly robust and fragile simultaneously.
Cheers, Joshua
Sent from my iPhone
On Feb 6, 2014, at 8:06 AM, "Vlade Ristevski" <vristevs@ramapo.edu> wrote:
We have had Cogent over Verizon's Fiber for more than a few years now. Cogent goes down once at year at minimum. They had 2 outages in a single day a couple days ago in Northern NJ. One in the AM "..caused by a power outage in a vendor data center where Cogent is collocated." They went on to have another outage at around 9:30 PM on the same day for which I'm still waiting for an RFO. During this outage, they still were advertising our BGP routes so we didn't fail over to our 2nd provider. I notice that happens alot with them. When they go down, they still advertise your routes.
As far as price goes, for us Cogent is cheap but Lightpath is cheaper.
Our college is kind of far from things so we don't have a lot of outside fiber coming. The last mile fiber for both of our connections are different from our Internet providers. I've never had a big issue with the two working with each other. The only issue we had is I suspected we weren't getting as much bandwidth as we paid for. They had to work out where the policer and/or bottle neck was. This is the only issue we had in 5 years with this set up and it got resolved. IME, when there is a full outage, it's always been clear who the responsible party is.
On 2/6/2014 10:17 AM, Adam Greene wrote: Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
When I priced out providers 2 years ago for 500Mbps over 1 gig fiber link the list from most expensive to least expensive was: Verizon-->XO-->Cogent-->Lightpath This is for Northern NJ. Abovenet and some of the other big providers couldn't reach our Campus. Lightpath ate the cost of running Fiber to our campus while the other weren't willing to do that. On 2/6/2014 11:28 AM, Patrick W. Gilmore wrote:
On Feb 6, 2014, at 11:22, Joshua Goldbard <j@2600hz.com> wrote:
Cogent always has the cheapest rates Objectively, provably false.
Vlade, When you say that "they still advertise your routes", do you mean: A: That you were having them originate your routes, and they failed to stop doing so when they had problems? Or... B: That routes you were originating continued to be propagated by them, even though your session with them was down? Or... C: Something else. I ask, as we are considering some cheap Cogent bandwidth in the not-too-distant future, to allow us to keep commit rates low on higher quality connections. 'A' wouldn't be a real problem, since we run our own AS and originate our own routes; 'B' could be potentially devastating. On Thu, Feb 6, 2014 at 8:04 AM, Vlade Ristevski <vristevs@ramapo.edu> wrote:
We have had Cogent over Verizon's Fiber for more than a few years now. Cogent goes down once at year at minimum. They had 2 outages in a single day a couple days ago in Northern NJ. One in the AM "..caused by a power outage in a vendor data center where Cogent is collocated." They went on to have another outage at around 9:30 PM on the same day for which I'm still waiting for an RFO. During this outage, they still were advertising our BGP routes so we didn't fail over to our 2nd provider. I notice that happens alot with them. When they go down, they still advertise your routes.
As far as price goes, for us Cogent is cheap but Lightpath is cheaper.
Our college is kind of far from things so we don't have a lot of outside fiber coming. The last mile fiber for both of our connections are different from our Internet providers. I've never had a big issue with the two working with each other. The only issue we had is I suspected we weren't getting as much bandwidth as we paid for. They had to work out where the policer and/or bottle neck was. This is the only issue we had in 5 years with this set up and it got resolved. IME, when there is a full outage, it's always been clear who the responsible party is.
On 2/6/2014 10:17 AM, Adam Greene wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
B) We have our own AS and IP space. I advertise them to both Cogent and our other ISP. I use the local preference attribute to share the load for incoming traffic between both ISPs. In the last 5 outages over the last few years, this has happened twice. I'm waiting on the RFO so I can further investigate why this happened. I think someone mentioned this in a post a few months ago too. It sucks for us, because we're a small school and don't have someone in a NOC to monitor our networks 24x7. I literally had to get out of bed and disable our BGP session with them for us to get through the outage. I was getting around 90% packet loss from my home to our router. On 2/6/2014 4:57 PM, Eric Flanery (eric) wrote:
Vlade,
When you say that "they still advertise your routes", do you mean:
A: That you were having them originate your routes, and they failed to stop doing so when they had problems? Or...
B: That routes you were originating continued to be propagated by them, even though your session with them was down? Or...
C: Something else.
I ask, as we are considering some cheap Cogent bandwidth in the not-too-distant future, to allow us to keep commit rates low on higher quality connections. 'A' wouldn't be a real problem, since we run our own AS and originate our own routes; 'B' could be potentially devastating.
On Thu, Feb 6, 2014 at 8:04 AM, Vlade Ristevski <vristevs@ramapo.edu <mailto:vristevs@ramapo.edu>> wrote:
We have had Cogent over Verizon's Fiber for more than a few years now. Cogent goes down once at year at minimum. They had 2 outages in a single day a couple days ago in Northern NJ. One in the AM "..caused by a power outage in a vendor data center where Cogent is collocated." They went on to have another outage at around 9:30 PM on the same day for which I'm still waiting for an RFO. During this outage, they still were advertising our BGP routes so we didn't fail over to our 2nd provider. I notice that happens alot with them. When they go down, they still advertise your routes.
As far as price goes, for us Cogent is cheap but Lightpath is cheaper.
Our college is kind of far from things so we don't have a lot of outside fiber coming. The last mile fiber for both of our connections are different from our Internet providers. I've never had a big issue with the two working with each other. The only issue we had is I suspected we weren't getting as much bandwidth as we paid for. They had to work out where the policer and/or bottle neck was. This is the only issue we had in 5 years with this set up and it got resolved. IME, when there is a full outage, it's always been clear who the responsible party is.
On 2/6/2014 10:17 AM, Adam Greene wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
-- Vlad
Based on your description, it sounds like the outage did not bring your BGP session down, as such you were connected and advertising to the broken Service Provider. e.g. Cogent typically does multi-hop bgp, as such if there a network outage past the BGP router, you will experience the situation you described. You might have to deploy some other means of (script ?) to bring your BGP session down from the 'broken' Service Provider. To the best of my knowledge, BGP does not have any mechanism to determine broken connectivity upstream past the router you are BGP session is up with. Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net ----- Original Message -----
From: "Vlade Ristevski" <vristevs@ramapo.edu> Cc: nanog@nanog.org Sent: Thursday, February 6, 2014 5:25:53 PM Subject: Re: carrier comparison
B) We have our own AS and IP space. I advertise them to both Cogent and our other ISP. I use the local preference attribute to share the load for incoming traffic between both ISPs. In the last 5 outages over the last few years, this has happened twice. I'm waiting on the RFO so I can further investigate why this happened. I think someone mentioned this in a post a few months ago too.
It sucks for us, because we're a small school and don't have someone in a NOC to monitor our networks 24x7. I literally had to get out of bed and disable our BGP session with them for us to get through the outage. I was getting around 90% packet loss from my home to our router.
On 2/6/2014 4:57 PM, Eric Flanery (eric) wrote:
Vlade,
When you say that "they still advertise your routes", do you mean:
A: That you were having them originate your routes, and they failed to stop doing so when they had problems? Or...
B: That routes you were originating continued to be propagated by them, even though your session with them was down? Or...
C: Something else.
I ask, as we are considering some cheap Cogent bandwidth in the not-too-distant future, to allow us to keep commit rates low on higher quality connections. 'A' wouldn't be a real problem, since we run our own AS and originate our own routes; 'B' could be potentially devastating.
On Thu, Feb 6, 2014 at 8:04 AM, Vlade Ristevski <vristevs@ramapo.edu <mailto:vristevs@ramapo.edu>> wrote:
We have had Cogent over Verizon's Fiber for more than a few years now. Cogent goes down once at year at minimum. They had 2 outages in a single day a couple days ago in Northern NJ. One in the AM "..caused by a power outage in a vendor data center where Cogent is collocated." They went on to have another outage at around 9:30 PM on the same day for which I'm still waiting for an RFO. During this outage, they still were advertising our BGP routes so we didn't fail over to our 2nd provider. I notice that happens alot with them. When they go down, they still advertise your routes.
As far as price goes, for us Cogent is cheap but Lightpath is cheaper.
Our college is kind of far from things so we don't have a lot of outside fiber coming. The last mile fiber for both of our connections are different from our Internet providers. I've never had a big issue with the two working with each other. The only issue we had is I suspected we weren't getting as much bandwidth as we paid for. They had to work out where the policer and/or bottle neck was. This is the only issue we had in 5 years with this set up and it got resolved. IME, when there is a full outage, it's always been clear who the responsible party is.
On 2/6/2014 10:17 AM, Adam Greene wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
-- Vlad
Hi Faisal,
You might have to deploy some other means of (script ?) to bring your BGP session down from the 'broken' Service Provider.
To the best of my knowledge, BGP does not have any mechanism to determine broken connectivity upstream past the router you are BGP session is up with.
Well, technically there's BFD that might do the trick. But of course it won't be available; it's not usually, so specially with Cogent... :) But maybe its link was just overloaded in fact. -- Olivier
Based on my understanding on BFD, it will not help you... BFD will detect the direct connected port being down quicker and force the BGP session down, (faster than the time BGP session timers take to determine something is broken) This is the common issue / challenge in how to determine up-stream path outage and then doing appropriate route engineering on an automatic basis. Maybe a SLA monitor type scripting/configuration be useful in your case. Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net ----- Original Message -----
From: "Olivier Benghozi" <olivier.benghozi@wifirst.fr> To: nanog@nanog.org Sent: Friday, February 7, 2014 5:25:53 AM Subject: Re: carrier comparison
Hi Faisal,
You might have to deploy some other means of (script ?) to bring your BGP session down from the 'broken' Service Provider.
To the best of my knowledge, BGP does not have any mechanism to determine broken connectivity upstream past the router you are BGP session is up with.
Well, technically there's BFD that might do the trick. But of course it won't be available; it's not usually, so specially with Cogent... :) But maybe its link was just overloaded in fact.
-- Olivier
On Friday, February 07, 2014 04:49:09 PM Faisal Imtiaz wrote:
Based on my understanding on BFD, it will not help you... BFD will detect the direct connected port being down quicker and force the BGP session down, (faster than the time BGP session timers take to determine something is broken)
You would also need your provider to support BFD (and by support I mostly mean willing to run, as modern gear today is technically capable). Mark.
Hi Vlade, Well, if you are trying to balance the incoming traffic load with local-pref attribute, I can understand your disappointment :) Since it doesn't work at all this way: local-pref is local to an AS and deals with outgoing traffic only.
B) We have our own AS and IP space. I advertise them to both Cogent and our other ISP. I use the local preference attribute to share the load for incoming traffic between both ISPs. In the last 5 outages over the last few years, this has happened twice. I'm waiting on the RFO so I can further investigate why this happened. I think someone mentioned this in a post a few months ago too.
-- Olivier
I'm not setting it on my router locally but sending it over to Cogent as a community string per page 22 of their user guide. http://cogentco.com/files/docs/customer_service/guide/global_cogent_customer... They use it to manipulate how traffic gets back to me so that is incoming from my routers view. I also pad the AS for the networks that I prefer to come back through the other ISP.. On 2/7/2014 5:27 AM, Olivier Benghozi wrote:
Hi Vlade,
Well, if you are trying to balance the incoming traffic load with local-pref attribute, I can understand your disappointment :) Since it doesn't work at all this way: local-pref is local to an AS and deals with outgoing traffic only.
B) We have our own AS and IP space. I advertise them to both Cogent and our other ISP. I use the local preference attribute to share the load for incoming traffic between both ISPs. In the last 5 outages over the last few years, this has happened twice. I'm waiting on the RFO so I can further investigate why this happened. I think someone mentioned this in a post a few months ago too.
-- Vlade Ristevski Network Manager IT Services Ramapo College (201)-684-6854
Did you verify your problem was announcements on the other side of the outage? This sounds to me like you are using a bgp announced default route from cogent which is always sent. I think the problem was you were sending traffic out a path that was broken. Since you mentioned your outbound balancing this would explain some packet loss and not 100% loss. Bryan Socha Network Engineer DigitalOcean
We don't get a default route from them. At the time of the outage my bgp session was up and I had a full routing table from them. I didn't have much time to troubleshoot it in that state since we were down so I had to disable the session ASAP. Once the RFO comes in, I'll be asking a lot more questions about it. My only experience with BGP is as a customer so I'm not too familiar with the intricacies on the provider side. We had an outage in the AM the same day and we failed over just fine. I'm very curious why the same didn't happen in the evening. On 2/7/2014 3:03 PM, Bryan Socha wrote:
Did you verify your problem was announcements on the other side of the outage? This sounds to me like you are using a bgp announced default route from cogent which is always sent. I think the problem was you were sending traffic out a path that was broken. Since you mentioned your outbound balancing this would explain some packet loss and not 100% loss.
Bryan Socha Network Engineer DigitalOcean
-- Vlade Ristevski Network Manager IT Services Ramapo College (201)-684-6854
This is exactly what I thought had happened....The outage that affected you was one our two routers up-stream from your connection to that provider. I am not trying to defend any Carrier, but there is no 'routing protocol' what will react to this kind of an issue..... Regards. Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net ----- Original Message -----
From: "Vlade Ristevski" <vristevs@ramapo.edu> Cc: "nanog list" <nanog@nanog.org> Sent: Friday, February 7, 2014 3:57:00 PM Subject: Re: carrier comparison
We don't get a default route from them. At the time of the outage my bgp session was up and I had a full routing table from them. I didn't have much time to troubleshoot it in that state since we were down so I had to disable the session ASAP. Once the RFO comes in, I'll be asking a lot more questions about it. My only experience with BGP is as a customer so I'm not too familiar with the intricacies on the provider side. We had an outage in the AM the same day and we failed over just fine. I'm very curious why the same didn't happen in the evening.
On 2/7/2014 3:03 PM, Bryan Socha wrote:
Did you verify your problem was announcements on the other side of the outage? This sounds to me like you are using a bgp announced default route from cogent which is always sent. I think the problem was you were sending traffic out a path that was broken. Since you mentioned your outbound balancing this would explain some packet loss and not 100% loss.
Bryan Socha Network Engineer DigitalOcean
-- Vlade Ristevski Network Manager IT Services Ramapo College (201)-684-6854
Hi all, Just wanted to say thanks to all who replied on and off list to my original inquiry. I'd sum up feedback as follows: - Although Cogent has been surprisingly good for some, in general almost everyone agreed that it should never be relied upon as your main Internet provider. As a secondary link, they are a good value. - People had generally good feedback about Level3 - Having one carrier provide service over another carrier’s fiber is generally not a problem. Sometimes it adds complication when things go wrong (and a couple people had some pretty extreme cases to share), but in general most people did not recommend shying away from this kind of relationship. - Time Warner also received positive reviews in general as a carrier I was also surprised how many small ISPs like us are on the NANOG list. I kinda assumed most of you were big operators that dwarf us. It's great to have received perspectives from both large and small operators. Thanks again, everyone. Adam -----Original Message----- From: Faisal Imtiaz [mailto:faisal@snappytelecom.net] Sent: Friday, February 07, 2014 4:43 PM To: Vlade Ristevski Cc: nanog list Subject: Re: carrier comparison This is exactly what I thought had happened....The outage that affected you was one our two routers up-stream from your connection to that provider. I am not trying to defend any Carrier, but there is no 'routing protocol' what will react to this kind of an issue..... Regards. Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net ----- Original Message -----
From: "Vlade Ristevski" <vristevs@ramapo.edu> Cc: "nanog list" <nanog@nanog.org> Sent: Friday, February 7, 2014 3:57:00 PM Subject: Re: carrier comparison
We don't get a default route from them. At the time of the outage my bgp session was up and I had a full routing table from them. I didn't have much time to troubleshoot it in that state since we were down so I had to disable the session ASAP. Once the RFO comes in, I'll be asking a lot more questions about it. My only experience with BGP is as a customer so I'm not too familiar with the intricacies on the provider side. We had an outage in the AM the same day and we failed over just fine. I'm very curious why the same didn't happen in the evening.
On 2/7/2014 3:03 PM, Bryan Socha wrote:
Did you verify your problem was announcements on the other side of the outage? This sounds to me like you are using a bgp announced default route from cogent which is always sent. I think the problem was you were sending traffic out a path that was broken. Since you mentioned your outbound balancing this would explain some packet loss and not 100% loss.
Bryan Socha Network Engineer DigitalOcean
-- Vlade Ristevski Network Manager IT Services Ramapo College (201)-684-6854
I got the RFO today and what happened was: " The Cogent NOC investigated and found that one of our customers connected through a Verizon aggregated circuit to the router was being DDOS attacked. This type of attack can send excessive traffic to a customer’s interface either deliberately or accidentally, causing a spike in the router’s CPU usage. The Cogent NOC shut down the attacked customer’s connection to the network restoring normal router operations and our Customer Service Group worked with the customer to resolve the DDOS issue." On 2/7/2014 4:42 PM, Faisal Imtiaz wrote:
This is exactly what I thought had happened....The outage that affected you was one our two routers up-stream from your connection to that provider.
I am not trying to defend any Carrier, but there is no 'routing protocol' what will react to this kind of an issue.....
Regards.
Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232
Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net
----- Original Message -----
From: "Vlade Ristevski" <vristevs@ramapo.edu> Cc: "nanog list" <nanog@nanog.org> Sent: Friday, February 7, 2014 3:57:00 PM Subject: Re: carrier comparison
We don't get a default route from them. At the time of the outage my bgp session was up and I had a full routing table from them. I didn't have much time to troubleshoot it in that state since we were down so I had to disable the session ASAP. Once the RFO comes in, I'll be asking a lot more questions about it. My only experience with BGP is as a customer so I'm not too familiar with the intricacies on the provider side. We had an outage in the AM the same day and we failed over just fine. I'm very curious why the same didn't happen in the evening.
On 2/7/2014 3:03 PM, Bryan Socha wrote:
Did you verify your problem was announcements on the other side of the outage? This sounds to me like you are using a bgp announced default route from cogent which is always sent. I think the problem was you were sending traffic out a path that was broken. Since you mentioned your outbound balancing this would explain some packet loss and not 100% loss.
Bryan Socha Network Engineer DigitalOcean -- Vlade Ristevski Network Manager IT Services Ramapo College (201)-684-6854
-- Vlad
On 2/6/14, 7:17 AM, Adam Greene wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Short answer: multihome. ~Seth
IMHO Cogent bandwidth is fine so long as it isn’t your only bandwidth. Good, Cheap, Fast, Pick any two. -- Matthew S. Crocker President Crocker Communications, Inc. PO BOX 710 Greenfield, MA 01302-0710 E: matthew@crocker.com P: (413) 746-2760 F: (413) 746-3704 W: http://www.crocker.com On Feb 6, 2014, at 10:17 AM, Adam Greene <maillist@webjogger.net> wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
+1 Same feeling here. Sam Moats On 2014-02-06 16:22, Matthew Crocker wrote:
IMHO Cogent bandwidth is fine so long as it isn’t your only bandwidth. Good, Cheap, Fast, Pick any two.
-- Matthew S. Crocker President Crocker Communications, Inc. PO BOX 710 Greenfield, MA 01302-0710
E: matthew@crocker.com P: (413) 746-2760 F: (413) 746-3704 W: http://www.crocker.com
On Feb 6, 2014, at 10:17 AM, Adam Greene <maillist@webjogger.net> wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
I use Cogent as well, no real issues other than I wouldn't single home to them. Personally, I don't understand why someone would depend on a single provider for connectivity however... -Blake On Thu, Feb 6, 2014 at 3:22 PM, Matthew Crocker <matthew@corp.crocker.com>wrote:
IMHO Cogent bandwidth is fine so long as it isn't your only bandwidth. Good, Cheap, Fast, Pick any two.
-- Matthew S. Crocker President Crocker Communications, Inc. PO BOX 710 Greenfield, MA 01302-0710
E: matthew@crocker.com P: (413) 746-2760 F: (413) 746-3704 W: http://www.crocker.com
On Feb 6, 2014, at 10:17 AM, Adam Greene <maillist@webjogger.net> wrote:
Hi,
We're a small ISP / datacenter with a Time Warner fiber-based DIA contract that is coming up for renewal.
We're getting much better pricing offers from Cogent, and are finding out what Level 3 can do for us as well. Both providers will use Time Warner fiber for last mile.
My questions are:
- Will we be sacrificing quality if we spring for Cogent? (yesterday's Cogent/Verizon thread provided some cold chills for my spine)
- Is there a risk with contracting a carrier that utilizes another carrier (such as Time Warner) for the last mile? (i.e. if there is a downtime situation, are we going to be caught in a web of confusion and finger-pointing that delays problem resolution)?
- How are peoples' experiences with L3 vs TWC?
Although I assume everyone on the list would be interested in what others have to say about these questions, out of respect for the carriers in question, I encourage you to email frank opinions off list.
Or if there are third party tools or resources you know that I could consult to deduce the answers to these questions myself, they are most welcome.
Thanks,
Adam
participants (13)
-
Adam Greene -
Blake Dunlap -
Bryan Socha -
Eric Flanery (eric) -
Faisal Imtiaz -
Joshua Goldbard -
Mark Tinka -
Matthew Crocker -
Olivier Benghozi -
Patrick W. Gilmore -
Sam Moats -
Seth Mattinen -
Vlade Ristevski