This is a blind message. I don't even know if it's getting out of my mail hub. Theoretically, it should. You won't be able to reply. I can route out, but nothing can get back in. Some of you may remember a poster of this huge eagle stooping on a mouse. The mouse was calmly standing, facing the eagle, flipping it the bird. The title was "Defiance". Covad is the eagle, MHSC is the mouse, and this message is the bird. MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad cut off all links and feeds to my upstream. I've been down ever since. Everyone knows how quickly telcos turn up services, that's why co-lo centers have customers. Covad cut off MHSC access, without warning, without, compansation, and their SafetyNet program is the worst joke. I have a /24 and the new ISPs, that Covad is forcing me to use, can't assign a new /24. Shit, one of them can't even assign one static address. The one that this is going out thru tis trying to convince me that I can run 62 computers, using 130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable addresses suck. Here is the reason why. I either use one of their SafetyNet providers or suffer up to 45 days of outage. Two weeks of warning and I would have been able to avoid the outage. (BTW, my ISDN backup feed is dead as well). I've tried contacting Covad, thier executive staff, etc. No joy there, Covad doesn't care. 100% of MHSC business is directly dependent on our Internet feed. 100% of February revenue is gone. I had two product roll-outs that were scheduled for the following week. Both are now dead issues. MHSC does CA for some of it's customers, renumbering invalidates all those CAs. Every lead we were working is now gone, probably forever, and so are many existing customers. I'm looking at probable business failure. MHSC could sue, but may not survive long enough to collect, even if we win. If MHSC survives this then "Damn, I good!" I urge everyone, do not; 1) Recommend Covad as a DSL provider 2) Become a Covad partner. 3) Remain on Covad service, find someone else (Rythyms sound good) Further, if you have any Covad stock ... sell it. The simple reason is that Covad has shown themselves unworthy of trust. They will cut off service arbitrarily, without direct cause, without compansation or recourse, and without warning. This is NOT the sort of behavior desired from an outfit in charge of your key infrastructure. We are not large, MHSC cannot afford an $80K+ revenue hit, and that's not counting the suits for failure to meet SLA. At this point, I'm not worried about any Covad law suits. What are they going to do ... put MHSC out of business? They've already taken care of that. I hope that this comes to the attention of Covad and I hope that they don't like it! MHSC will also be filing complaints with the CA-PUC and everywhere else that we can. But, a four-year process will not help MHSC stay in business. MHSC is already preparing a Marketing Communication plan to get this story out. MHSC tried dealing with this peacfully and got ignored, by Covad. Covad needs to be shunned by the rest of the Internet community, as a bad net-citizen. MHSC wasn't the ony business effected. There were 800 other busnesses involved in the cut off, that we know of. If any of you know anyone else that has been effected by this outage, please let me know. -- ROELAND M.J. MEYER Managing Director Morgan Hill Software Company, Inc. TEL: +001 925 373 3954 FAX: +001 925 373 9781 http://www.mhsc.com mailto: rmeyer@mhsc.com
On Wed, 28 Feb 2001, Roeland Meyer wrote: <rant snipped>
This is a blind message. I don't even know if it's getting out of my mail hub. Theoretically, it should. You won't be able to reply. I can route out, but nothing can get back in. That's strange. As far as I know, TCP requires a two-way handshake. :)
130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable addresses suck. Here is the reason why. I either use one of their SafetyNet Portable address wouldn't help you, since safetynet'd ISP won't talk BGP to you. What you could do is (with agreement of your current ISP who owns the IPs) is to go into any colo centers, put up your hardware, and tell the colo ISP to advertise your netblocks.
would have been able to avoid the outage. (BTW, my ISDN backup feed is dead as well). I've tried contacting Covad, thier executive staff, etc. No joy there, Covad doesn't care. Its going from "We don't care, we don't have to" to "We don't care, we can't afford to". Covad (and its wholesalers) experienced major collection problems, and they are cutting off the wholesalers who don't pay their bills.
100% of MHSC business is directly dependent on our Internet feed. 100% of February revenue is gone. I had two product roll-outs that were scheduled DSL is not supposed to be bet-the-business-on product. I'm sure it'll sound silly now, but if you have 80k$ revenue, you could afford a T1 which has very different ETR timelines compared to DSL, and that does not involve so many companies involved in its delivery/billing process.
for the following week. Both are now dead issues. MHSC does CA for some of it's customers, renumbering invalidates all those CAs. Every lead we were working is now gone, probably forever, and so are many existing customers. I may misunderstand something, but if CA means 'X509 certificate authority', why would it invalidate anything? CAs have CN, which is a name, not IP address.
I'm looking at probable business failure. MHSC could sue, but may not survive long enough to collect, even if we win. If MHSC survives this then "Damn, I good!"
I urge everyone, do not; 1) Recommend Covad as a DSL provider 2) Become a Covad partner. 3) Remain on Covad service, find someone else (Rythyms sound good) http://biz.yahoo.com/p/r/rthm.html, the financial situation of rhythms is not much better than Covad or Northpoint. According their current cash burn rate, they'll run out of cash sometime this year.
Further, if you have any Covad stock ... sell it. Too late, its around 2 dollars now.
The simple reason is that Covad has shown themselves unworthy of trust. They will cut off service arbitrarily, without direct cause, without compansation or recourse, and without warning. This is NOT the sort of behavior desired from an outfit in charge of your key infrastructure. You are missing something. Covad did not cut off the service to you. Covad cut off the service to DSLnetworks. DSLnetworks was not paying bills, and it is their (dslnetworks) fault that they didn't disclose their financial situation earlier to their subscribers.
I hope that this comes to the attention of Covad and I hope that they don't like it! MHSC will also be filing complaints with the CA-PUC and everywhere else that we can. But, a four-year process will not help MHSC stay in business. MHSC is already preparing a Marketing Communication plan to get this story out. MHSC tried dealing with this peacfully and got ignored, by Covad. Covad needs to be shunned by the rest of the Internet community, as a bad net-citizen. I'm in no way trying to apologise for Covad (they got what was coming to them), but your problems could've been averted with a T1 or redundant connections.
-- Alex Pilosov | http://www.acecape.com/dsl CTO - Acecape, Inc. | AceDSL:The best ADSL in Bell Atlantic area 325 W 38 St. Suite 1005 | (Stealth Marketing Works! :) New York, NY 10018 |
This is a blind message. I don't even know if it's getting out of my mail hub. Theoretically, it should. You won't be able to reply. I can route out, but nothing can get back in.
Some of you may remember a poster of this huge eagle stooping on a mouse. The mouse was calmly standing, facing the eagle, flipping it the bird. The title was "Defiance". Covad is the eagle, MHSC is the mouse, and this message is the bird.
MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad cut off all links and feeds to my upstream. I've been down ever since. Everyone knows how quickly telcos turn up services, that's why co-lo
If I read this right, you are not a Covad customer, and you didn't choose Covad service, your upstream did. It seems to me that: 1) You are not in any position to evaluate Covad's relationship with your provider. You have no way of knowing what Covad's contract looks like with your provider or whether your provider paid its bills or abused its service. 2) Your complaint is with your provider, regardless of Covad's action. If your provider did something wrong in their relationship with Covad, they are at fault. If your provider did nothing wrong in their relationship with Covad, they still chose Covad and didn't provide for decent fall back provisions or legal protection for their customers. 3) You have no way of knowing how much advanced notice Covad provided to your upstream or what alternate options they could have taken but didn't. Please remember to aim before you fire. DS centers
have customers. Covad cut off MHSC access, without warning, without, compansation, and their SafetyNet program is the worst joke. I have a /24 and the new ISPs, that Covad is forcing me to use, can't assign a new /24. Shit, one of them can't even assign one static address. The one that this is going out thru tis trying to convince me that I can run 62 computers, using 130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable addresses suck. Here is the reason why. I either use one of their SafetyNet providers or suffer up to 45 days of outage. Two weeks of warning and I would have been able to avoid the outage. (BTW, my ISDN backup feed is dead as well). I've tried contacting Covad, thier executive staff, etc. No joy there, Covad doesn't care.
100% of MHSC business is directly dependent on our Internet feed. 100% of February revenue is gone. I had two product roll-outs that were scheduled for the following week. Both are now dead issues. MHSC does CA for some of it's customers, renumbering invalidates all those CAs. Every lead we were working is now gone, probably forever, and so are many existing customers. I'm looking at probable business failure. MHSC could sue, but may not survive long enough to collect, even if we win. If MHSC survives this then "Damn, I good!"
I urge everyone, do not; 1) Recommend Covad as a DSL provider 2) Become a Covad partner. 3) Remain on Covad service, find someone else (Rythyms sound good)
Further, if you have any Covad stock ... sell it.
The simple reason is that Covad has shown themselves unworthy of trust. They will cut off service arbitrarily, without direct cause, without compansation or recourse, and without warning. This is NOT the sort of behavior desired from an outfit in charge of your key infrastructure.
We are not large, MHSC cannot afford an $80K+ revenue hit, and that's not counting the suits for failure to meet SLA.
At this point, I'm not worried about any Covad law suits. What are they going to do ... put MHSC out of business? They've already taken care of that.
I hope that this comes to the attention of Covad and I hope that they don't like it! MHSC will also be filing complaints with the CA-PUC and everywhere else that we can. But, a four-year process will not help MHSC stay in business. MHSC is already preparing a Marketing Communication plan to get this story out. MHSC tried dealing with this peacfully and got ignored, by Covad. Covad needs to be shunned by the rest of the Internet community, as a bad net-citizen.
MHSC wasn't the ony business effected. There were 800 other busnesses involved in the cut off, that we know of. If any of you know anyone else that has been effected by this outage, please let me know.
-- ROELAND M.J. MEYER Managing Director Morgan Hill Software Company, Inc. TEL: +001 925 373 3954 FAX: +001 925 373 9781 http://www.mhsc.com mailto: rmeyer@mhsc.com
Wow. A telco cut off a customer who hadn't paid their bill? Amazing. Also old news, if this is the case I'm thinking of. If you find a DSL local loop provider that works reliably, will be in business for a while, and doesn't require its customers to pay, please let me know. It sounds useful. Yes, having your DSL service cut off sucks. I know this from experience, as my home connection is also currently down for the same reason. However, I think the lesson to learn from this is not that Covad sucks, but rather that keeping mission critical systems single homed on a DSL line to an ISP that's not paying its bills is probably a bad idea. -Steve On Wed, 28 Feb 2001, Roeland Meyer wrote:
This is a blind message. I don't even know if it's getting out of my mail hub. Theoretically, it should. You won't be able to reply. I can route out, but nothing can get back in.
Some of you may remember a poster of this huge eagle stooping on a mouse. The mouse was calmly standing, facing the eagle, flipping it the bird. The title was "Defiance". Covad is the eagle, MHSC is the mouse, and this message is the bird.
MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad cut off all links and feeds to my upstream. I've been down ever since. Everyone knows how quickly telcos turn up services, that's why co-lo centers have customers. Covad cut off MHSC access, without warning, without, compansation, and their SafetyNet program is the worst joke. I have a /24 and the new ISPs, that Covad is forcing me to use, can't assign a new /24. Shit, one of them can't even assign one static address. The one that this is going out thru tis trying to convince me that I can run 62 computers, using 130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable addresses suck. Here is the reason why. I either use one of their SafetyNet providers or suffer up to 45 days of outage. Two weeks of warning and I would have been able to avoid the outage. (BTW, my ISDN backup feed is dead as well). I've tried contacting Covad, thier executive staff, etc. No joy there, Covad doesn't care.
100% of MHSC business is directly dependent on our Internet feed. 100% of February revenue is gone. I had two product roll-outs that were scheduled for the following week. Both are now dead issues. MHSC does CA for some of it's customers, renumbering invalidates all those CAs. Every lead we were working is now gone, probably forever, and so are many existing customers. I'm looking at probable business failure. MHSC could sue, but may not survive long enough to collect, even if we win. If MHSC survives this then "Damn, I good!"
I urge everyone, do not; 1) Recommend Covad as a DSL provider 2) Become a Covad partner. 3) Remain on Covad service, find someone else (Rythyms sound good)
Further, if you have any Covad stock ... sell it.
The simple reason is that Covad has shown themselves unworthy of trust. They will cut off service arbitrarily, without direct cause, without compansation or recourse, and without warning. This is NOT the sort of behavior desired from an outfit in charge of your key infrastructure.
We are not large, MHSC cannot afford an $80K+ revenue hit, and that's not counting the suits for failure to meet SLA.
At this point, I'm not worried about any Covad law suits. What are they going to do ... put MHSC out of business? They've already taken care of that.
I hope that this comes to the attention of Covad and I hope that they don't like it! MHSC will also be filing complaints with the CA-PUC and everywhere else that we can. But, a four-year process will not help MHSC stay in business. MHSC is already preparing a Marketing Communication plan to get this story out. MHSC tried dealing with this peacfully and got ignored, by Covad. Covad needs to be shunned by the rest of the Internet community, as a bad net-citizen.
MHSC wasn't the ony business effected. There were 800 other busnesses involved in the cut off, that we know of. If any of you know anyone else that has been effected by this outage, please let me know.
-- ROELAND M.J. MEYER Managing Director Morgan Hill Software Company, Inc. TEL: +001 925 373 3954 FAX: +001 925 373 9781 http://www.mhsc.com mailto: rmeyer@mhsc.com
-------------------------------------------------------------------------------- Steve Gibbard scg@gibbard.org
On Wed, 28 Feb 2001, Steve Gibbard wrote: :Yes, having your DSL service cut off sucks. I know this from experience, :as my home connection is also currently down for the same reason. :However, I think the lesson to learn from this is not that Covad sucks, :but rather that keeping mission critical systems single homed on a DSL :line to an ISP that's not paying its bills is probably a bad idea. Our sales staff tells prospective clients exactly that ("If your business will rely heavily on your connectivity, DSL is the wrong choice"). It's not mature enough (read: its too-quick proliferation was due mostly to largely sales-oriented, facilities-less entities flooding the market with no forethought to quality). And, of course, it's never a good idea to include scenarios that are out of one's control in SLAs to one's clients. That said, I'd say Roeland's gotten his share of crow to mange on ;) Live and learn. -brian
On Wed, Feb 28, 2001 at 01:54:41PM -0800, Roeland Meyer wrote:
MHSC has been down since 7Feb01 and Covad is the culprit [...] We are not large, MHSC cannot afford an $80K+ revenue hit, and that's not counting the suits for failure to meet SLA.
I'm no bean counter, but if you're losing this much money from a xDSL-related outage, perhaps it might be a good idea to invest in a carefully-planned redundant infrastructure, including some means of backup transit. This is by no means a revolutionary concept, and may very prevent businesses from becoming defunct businesses in extreme cases like this. Some might argue that DSL (other than that which is provisioned by the ILEC or another party with deep pockets) is going away sooner or later, or at the very least, undergoing drastic changes in the way it's marketed and aggregated. But that's beyond the scope of this discussion. Sorry if this isn't the message of sympathy you were hoping for. -adam
On Wed, Feb 28, 2001 at 01:54:41PM -0800, Roeland Meyer wrote:
MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad cut off all links and feeds to my upstream. I've been down ever since. Everyone knows how quickly telcos turn up services, that's why co-lo centers have customers. Covad cut off MHSC access, without warning, without, compansation, and their SafetyNet program is the worst joke. I have a /24 and the new ISPs, that Covad is forcing me to use, can't assign a new /24. Shit, one of them can't even assign one static address. The one that this is going out thru tis trying to convince me that I can run 62 computers, using 130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable addresses suck. Here is the reason why. I either use one of their SafetyNet providers or suffer up to 45 days of outage. Two weeks of warning and I would have been able to avoid the outage. (BTW, my ISDN backup feed is dead as well). I've tried contacting Covad, thier executive staff, etc. No joy there, Covad doesn't care.
Hmm, gee, isn't there a cliche' that fits this exact problem? Something about eggs? --Adam
2 Thumbs Up ----- Original Message ----- From: "Roeland Meyer" <rmeyer@mhsc.com> To: "Roeland Meyer" <rmeyer@mhsc.com> Cc: <nanog@merit.edu> Sent: Wednesday, February 28, 2001 1:54 PM Subject: Covad SUCKS!
This is a blind message. I don't even know if it's getting out of my mail hub. Theoretically, it should. You won't be able to reply. I can route
but nothing can get back in.
Some of you may remember a poster of this huge eagle stooping on a mouse. The mouse was calmly standing, facing the eagle, flipping it the bird. The title was "Defiance". Covad is the eagle, MHSC is the mouse, and this message is the bird.
MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad cut off all links and feeds to my upstream. I've been down ever since. Everyone knows how quickly telcos turn up services, that's why co-lo
out, centers
have customers. Covad cut off MHSC access, without warning, without, compansation, and their SafetyNet program is the worst joke. I have a /24 and the new ISPs, that Covad is forcing me to use, can't assign a new /24. Shit, one of them can't even assign one static address. The one that this is going out thru tis trying to convince me that I can run 62 computers, using 130 IP addrs, on a /27 <heh>. We talked earlier about how non-portable addresses suck. Here is the reason why. I either use one of their SafetyNet providers or suffer up to 45 days of outage. Two weeks of warning and I would have been able to avoid the outage. (BTW, my ISDN backup feed is dead as well). I've tried contacting Covad, thier executive staff, etc. No joy there, Covad doesn't care.
100% of MHSC business is directly dependent on our Internet feed. 100% of February revenue is gone. I had two product roll-outs that were scheduled for the following week. Both are now dead issues. MHSC does CA for some of it's customers, renumbering invalidates all those CAs. Every lead we were working is now gone, probably forever, and so are many existing customers. I'm looking at probable business failure. MHSC could sue, but may not survive long enough to collect, even if we win. If MHSC survives this then "Damn, I good!"
I urge everyone, do not; 1) Recommend Covad as a DSL provider 2) Become a Covad partner. 3) Remain on Covad service, find someone else (Rythyms sound good)
Further, if you have any Covad stock ... sell it.
The simple reason is that Covad has shown themselves unworthy of trust. They will cut off service arbitrarily, without direct cause, without compansation or recourse, and without warning. This is NOT the sort of behavior desired from an outfit in charge of your key infrastructure.
We are not large, MHSC cannot afford an $80K+ revenue hit, and that's not counting the suits for failure to meet SLA.
At this point, I'm not worried about any Covad law suits. What are they going to do ... put MHSC out of business? They've already taken care of that.
I hope that this comes to the attention of Covad and I hope that they don't like it! MHSC will also be filing complaints with the CA-PUC and everywhere else that we can. But, a four-year process will not help MHSC stay in business. MHSC is already preparing a Marketing Communication plan to get this story out. MHSC tried dealing with this peacfully and got ignored, by Covad. Covad needs to be shunned by the rest of the Internet community, as a bad net-citizen.
MHSC wasn't the ony business effected. There were 800 other busnesses involved in the cut off, that we know of. If any of you know anyone else that has been effected by this outage, please let me know.
-- ROELAND M.J. MEYER Managing Director Morgan Hill Software Company, Inc. TEL: +001 925 373 3954 FAX: +001 925 373 9781 http://www.mhsc.com mailto: rmeyer@mhsc.com
* Roeland Meyer <rmeyer@mhsc.com> [20010228 13:48]: [..]
MHSC has been down since 7Feb01 and Covad is the culprit. MHSC pays its bills, but Covad had a problem with my upstream. On Wednesday, 7Feb01, Covad cut off all links and feeds to my upstream. I've been down ever since. [..] I urge everyone, do not; 1) Recommend Covad as a DSL provider 2) Become a Covad partner. 3) Remain on Covad service, find someone else (Rythyms sound good)
Further, if you have any Covad stock ... sell it.
I understand your frustration but I fail to see why you are disgusted with Covad. What about your "upstream"? Your DSL reseller/bundler is the one at fault, not Covad. You weren't warned by Covad because you are not a Covad customer. Warning was _your_ providers job.
The simple reason is that Covad has shown themselves unworthy of trust. They will cut off service arbitrarily, without direct cause, without compansation or recourse, and without warning. This is NOT the sort of behavior desired from an outfit in charge of your key infrastructure.
They cut off your upstream for not paying their bill. It sucks that your company is stuck in the middle, but blame your vendor. Ask them why Covad cut them off. I realize you're frustrated but don't direct blame at the wrong company. Covad is just trying to get their own customers to pay their bills. A lot of the resellers are getting hit because their customers (ISPs) are not paying them. Eventually it works its way up the chain. Sadly, since the resellers are just bundling up a bunch of ISPs under one account, if something goes wrong, all of their ISP customers can be affected. I'm truly glad my previous employer (a regional ISP) didn't go with DSLNetworks. We were *really* close about 18 months ago but decided to go it on our own at the last minute. If that had not turned out to be the case, they'd be in it deep right now and probably equally frustrated. -jr ---- Josh Richards [JTR38/JR539-ARIN] <jrichard@geekresearch.com/cubicle.net/fix.net/freedom.gen.ca.us> Geek Research LLC - <URL:http://www.geekresearch.com/> IP Network Engineering and Consulting
participants (9)
-
Adam McKenna -
Adam Rothschild -
Alex Pilosov -
Brian Wallingford -
David Schwartz -
Josh Richards -
Roeland Meyer -
Scott Baldwin -
Steve Gibbard