CAIS/Ardent and now Network Access Solutions
CAIS sold our account to NAS. They did this about 5 months back. They are just now getting around to cutting us over. This involves Covad making some changes in their switch somewhere. Back last May, it was PSINet that was selling our account to CAIS. They sent us an e-mail to announce that they were taking over the account and sent us a list of the account details that they had for us. The information was wrong. I sent them corrections. Also made sure they knew that we had our own IP addresses and to be sure to coordinate the BGP stuff with PSI when cutting over. They didn't. PSI kept announcing the routes. Ardent didnt announce them. It took me FIVE DAYS to get them to fix it. Waited on hold for 45 minutes more than 7 times and finally talked to brain dead drones on the other end each time. No luck. Finally wheedled the PSI NOCs direct number out of someone on the phone at PSI and called them. They dropped the route announcements for us. Then it took three more days to get in touch with someone at Ardent. I was able to do that by posting a message here. One of their engineers called me and we had it fixed in ten minutes. Fast forward to Dec/Jan 2001/2002: Now, CAIS (called Ardent now) sold us to Network Access Solutions (NAS). We have been paying them ever since Dec or so. Two months ago, they sent the same kind of e-mail: "We'll be cutting over your connection soon, please fill out this questionnaire, etc". I did so, and wrote a long tome at the end warning them not to mess up the BGP stuff. "It couldn't be that bad twice, could it?" I though to myself. I crossed my fingers. In their e-mail, they told me that they would soon get back to me with a cutover date. This afternoon, at 4:40PM, I got the message "Your cutover date/time is 5PM-8PM Eastern May 29, 2002 (today)" and "The date cannot be changed". Also they said "If you have your own IP addresses", it may take till midnight till the routes propagate (Now I know I'm in trouble - 5pm till midnight?). No sooner did I read the message than the connection went dead. Not the circuit, just the BGP announcements. Its now 7AM Eastern time. We've been offline 13 hours now. Phone drone at NAS says "Hmm, your not listed on our cutover sheet today". I put two and two together: They told Ardent "drop the routes today at 5" and then lost the paperwork internally so the cutover did not happen, but Ardent dropped the routes. Just got off the phone with them again after sitting on hold for 45 minutes. They dont seem to get the message that its a BGP problem and not the circuit. Last time, when we had problems, some kind engineer from CAIS sent me e-mail and offered to help and, like I said, he got us back online in, like 10 minutes. If you're out there, please let me know. I need your help again Please reply to jpp@well.com as the e-mail on this message is not accessible at this time. Sorry to bug all of you with this. Let see: 13 hours and running. Last time it was five days. Wanna take bets on how long this time? Funny thing, when I called CAIS afterwards to ask them how they were going to "make it up" to me for knocking me offline for five days, I was sent to the voicemail of some customer service manager who never returned my calls. Lets see how well NAS does. John
On Thu, 30 May 2002, John Palmer wrote:
CAIS sold our account to NAS. They did this about 5 months back. They are
NAS has been nothing but trouble. We are (or were) a Covad reseller, first direct through Covad, then through CAIS. The first we heard our lines had been sold was when we called CAIS for support and were transfered to NAS. A week after that, our customers started getting e-mail that their accounts had been sold and they now were NAS customers. Except our CAIS explicitly stated they were NEVER to contact our customers for any reason. They appologized -- and a week later mailed out paper letters to all our customers. When our backhaul went down, it took them over 4 hours to even pick up the ticket. It was in the same queue as regular DSL lines. It's been the same with every circuit that goes down. One hour plus hold times for support, e-mail to support is answered days later. We finally have a direct rep in corporate's cell number and put all tickets in through him, but this is no way to run things. The best was they wanted me to sign a contract adendum stateing that if any bill was more than 10 days late, they would take our customers -- no mention at all of dispute resolution. I laughed at them. So, we don't place any more Covad orders, which is fine since no one wants to pay those prices anyway, and we sell Verizon DSL. Amazingly, Verizon DSL has had far fewer hassels than Covad ever did. ========================================================== Chris Candreva -- chris@westnet.com -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/
We are finally back. Hmm - Level3 is one of their transit providers: BGP routing table entry for 199.5.156.0/23, version 531543 Paths: (4 available, best #1) Not advertised to any peer 3356 13953 209.244.2.230 (metric 200201) from 165.117.1.219 (165.117.1.219) Origin IGP, metric 4294967294, localpref 100, valid, internal, best Community: 2548:172 2548:344 2548:666 3706:153 3356 13953 209.0.227.37 (metric 270401) from 165.117.1.144 (165.117.1.144) Origin IGP, metric 4294967294, localpref 100, valid, internal Community: 2548:172 2548:264 2548:666 3706:154 3356 13953 165.117.69.46 (metric 340501) from 165.117.1.140 (165.117.1.140) Origin IGP, metric 4294967294, localpref 100, valid, internal Community: 2548:172 2548:267 2548:666 3706:132 3356 13953 165.117.52.198 (metric 340401) from 165.117.1.155 (165.117.1.155) Origin IGP, metric 4294967294, localpref 100, valid, internal Community: 2548:172 2548:302 2548:666 3706:161 Here is a question: How are folks here handling the environment today where you rely on your connection but you keep getting handed (sold) off from one provider to another and each one seems to be in worse financial shape? Who can one rely on for connectivity? In general and in the Detroit area? I put out a request for bids on T-1's and all the national providers were way too high, mostly because they have no local POPs. We really need to be as close to the top as possible, not because of bandwidth needs, but for reliability's sake. John On Thu, 30 May 2002, John Palmer wrote:
CAIS sold our account to NAS. They did this about 5 months back. They are
NAS has been nothing but trouble. We are (or were) a Covad reseller, first direct through Covad, then through CAIS. The first we heard our lines had been sold was when we called CAIS for support and were transfered to NAS. A week after that, our customers started getting e-mail that their accounts had been sold and they now were NAS customers. Except our CAIS explicitly stated they were NEVER to contact our customers for any reason. They appologized -- and a week later mailed out paper letters to all our customers. When our backhaul went down, it took them over 4 hours to even pick up the ticket. It was in the same queue as regular DSL lines. It's been the same with every circuit that goes down. One hour plus hold times for support, e-mail to support is answered days later. We finally have a direct rep in corporate's cell number and put all tickets in through him, but this is no way to run things. The best was they wanted me to sign a contract adendum stateing that if any bill was more than 10 days late, they would take our customers -- no mention at all of dispute resolution. I laughed at them. So, we don't place any more Covad orders, which is fine since no one wants to pay those prices anyway, and we sell Verizon DSL. Amazingly, Verizon DSL has had far fewer hassels than Covad ever did. ========================================================== Chris Candreva -- chris@westnet.com -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/
> Who can one rely on for connectivity? In general and in the Detroit area? > I put out a request for bids on T-1's and all the national providers were > way too high... Haven't you just answered your own question? I guess if you think reliable service is too expensive, you're not in the market for reliable service, no? -Bill
Its just that they aren't local and there is no need to pay for a circuit all the way to Chicago. It seems that so many providers have moved out of Macomb county. Anyone have any experience with BigNet? We are talking to them now -----Original Message----- From: Bill Woodcock [mailto:woody@zocalo.net] Sent: Thursday, May 30, 2002 4:42 PM To: John Palmer Cc: nanog@nanog.org Subject: Re: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions) > Who can one rely on for connectivity? In general and in the Detroit area? > I put out a request for bids on T-1's and all the national providers were > way too high... Haven't you just answered your own question? I guess if you think reliable service is too expensive, you're not in the market for reliable service, no? -Bill
The hard facts are that carriers want to make money hopefully so they will locate where the money is. My chances of having closely located customers is greater in Newyork City say than somewhere distant You either have to pull circuits to the carrier you want or settle for someone local much smaller and in many cases much more likely to fail. Top providers will be located in top markets, why that's where the business is usually. On Thu, 30 May 2002, John Palmer wrote:
Its just that they aren't local and there is no need to pay for a circuit all the way to Chicago. It seems that so many providers have moved out of Macomb county. Anyone have any experience with BigNet? We are talking to them now
-----Original Message----- From: Bill Woodcock [mailto:woody@zocalo.net] Sent: Thursday, May 30, 2002 4:42 PM To: John Palmer Cc: nanog@nanog.org Subject: Re: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions)
> Who can one rely on for connectivity? In general and in the Detroit area? > I put out a request for bids on T-1's and all the national providers were > way too high...
Haven't you just answered your own question? I guess if you think reliable service is too expensive, you're not in the market for reliable service, no?
-Bill
On Thu, May 30, 2002 at 02:53:02PM -0700, Scott Granados wrote:
You either have to pull circuits to the carrier you want or settle for someone local much smaller and in many cases much more likely to fail.
Nonsense. Small providers are the ones who are financially connected to reality, and thus more likely to make money (since the owners generally want to keep paying the bills) and remain in business. The issue with small providers is, you don't always know what you are going to get. You could end up with someone that doesn't have much clue about providing good service simply because they are inexperienced. On the other hand, you could find someone who is very experienced but who wants to work for themselves, or on a project where they can have a significant impract. Being small could very easily mean they are sensible, nimble, and not burdoned with silly bureaucracy. That said, this has very little place on NANOG. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Agreed. As the architect for a large regional we strive to have full transit with several large providers (and peer with folks when possible) so that a single large provider unexpectedly 'biting the dust' or even experiencing backbone problems won't destroy our network. {Please note: this is not a sales plug, we aren't anywhere in the MI area.} And yes, the smaller local and regional providers must look at margins and cash flow constantly. They can't afford to pull the 'tricks' that are getting the big guys in so much trouble. My one caveat is to insure that the smaller guy is fully multihomed with at least two different upstreams. Tim -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Richard A Steenbergen Sent: Thursday, May 30, 2002 19:49 To: Scott Granados Cc: John Palmer; nanog@nanog.org Subject: Re: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions) On Thu, May 30, 2002 at 02:53:02PM -0700, Scott Granados wrote:
You either have to pull circuits to the carrier you want or settle for someone local much smaller and in many cases much more likely to fail.
Nonsense. Small providers are the ones who are financially connected to reality, and thus more likely to make money (since the owners generally want to keep paying the bills) and remain in business. The issue with small providers is, you don't always know what you are going to get. You could end up with someone that doesn't have much clue about providing good service simply because they are inexperienced. On the other hand, you could find someone who is very experienced but who wants to work for themselves, or on a project where they can have a significant impract. Being small could very easily mean they are sensible, nimble, and not burdoned with silly bureaucracy. That said, this has very little place on NANOG. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
Clearly anyone in your market is buying from someone outside of your market. The fees associated with reliability (if available) are a function of your geography. Large providers are concentrating on the markets that are making them the most money. If you get a few networks in your area that want to save money on the cost of reliability you can run a couple of circuits to the next large market and try to knit together a reliable network and divide the costs that way. My guess is that with more large providers on a profit-centered basis you won't see the same kind of pricing equality you have been seeing between Tier 1 and Tier N markets anymore. Deepak Jain AiNET -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of John Palmer Sent: Thursday, May 30, 2002 5:39 PM To: nanog@nanog.org Subject: RE: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions) Its just that they aren't local and there is no need to pay for a circuit all the way to Chicago. It seems that so many providers have moved out of Macomb county. Anyone have any experience with BigNet? We are talking to them now -----Original Message----- From: Bill Woodcock [mailto:woody@zocalo.net] Sent: Thursday, May 30, 2002 4:42 PM To: John Palmer Cc: nanog@nanog.org Subject: Re: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions) > Who can one rely on for connectivity? In general and in the Detroit area? > I put out a request for bids on T-1's and all the national providers were > way too high... Haven't you just answered your own question? I guess if you think reliable service is too expensive, you're not in the market for reliable service, no? -Bill
Surprised there isnt much connectivity in the Detroit area, I mean it is Motor City and all, I would think tons of manufacturing palnts all needing telecom of some sort or other.. Bri On Thu, 30 May 2002, Deepak Jain wrote:
Clearly anyone in your market is buying from someone outside of your market. The fees associated with reliability (if available) are a function of your geography. Large providers are concentrating on the markets that are making them the most money.
If you get a few networks in your area that want to save money on the cost of reliability you can run a couple of circuits to the next large market and try to knit together a reliable network and divide the costs that way.
My guess is that with more large providers on a profit-centered basis you won't see the same kind of pricing equality you have been seeing between Tier 1 and Tier N markets anymore.
Deepak Jain AiNET
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of John Palmer Sent: Thursday, May 30, 2002 5:39 PM To: nanog@nanog.org Subject: RE: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions)
Its just that they aren't local and there is no need to pay for a circuit all the way to Chicago. It seems that so many providers have moved out of Macomb county. Anyone have any experience with BigNet? We are talking to them now
-----Original Message----- From: Bill Woodcock [mailto:woody@zocalo.net] Sent: Thursday, May 30, 2002 4:42 PM To: John Palmer Cc: nanog@nanog.org Subject: Re: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions)
> Who can one rely on for connectivity? In general and in the Detroit area? > I put out a request for bids on T-1's and all the national providers were > way too high...
Haven't you just answered your own question? I guess if you think reliable service is too expensive, you're not in the market for reliable service, no?
-Bill
As far as I know, the following national providers exist in Detroit: (This is by no means, a complete list, and some of these aren't "Big Name" providers, but less likely to go under than John Q Random ISP) Verio Winstar XO AT&T Qwest Covad Level 3 SBC I've heard rumors of speakeasy.net, which I think resells internap I'm sure there's a few other national networks, these are just the ones I have heard of off the top of my head. -Paul On Thu, 2002-05-30 at 19:27, Brian wrote:
Surprised there isnt much connectivity in the Detroit area, I mean it is Motor City and all, I would think tons of manufacturing palnts all needing telecom of some sort or other..
Bri
On Thu, 30 May 2002, Deepak Jain wrote:
Clearly anyone in your market is buying from someone outside of your market. The fees associated with reliability (if available) are a function of your geography. Large providers are concentrating on the markets that are making them the most money.
If you get a few networks in your area that want to save money on the cost of reliability you can run a couple of circuits to the next large market and try to knit together a reliable network and divide the costs that way.
My guess is that with more large providers on a profit-centered basis you won't see the same kind of pricing equality you have been seeing between Tier 1 and Tier N markets anymore.
Deepak Jain AiNET
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of John Palmer Sent: Thursday, May 30, 2002 5:39 PM To: nanog@nanog.org Subject: RE: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions)
Its just that they aren't local and there is no need to pay for a circuit all the way to Chicago. It seems that so many providers have moved out of Macomb county. Anyone have any experience with BigNet? We are talking to them now
-----Original Message----- From: Bill Woodcock [mailto:woody@zocalo.net] Sent: Thursday, May 30, 2002 4:42 PM To: John Palmer Cc: nanog@nanog.org Subject: Re: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions)
> Who can one rely on for connectivity? In general and in the Detroit area? > I put out a request for bids on T-1's and all the national providers were > way too high...
Haven't you just answered your own question? I guess if you think reliable service is too expensive, you're not in the market for reliable service, no?
-Bill
As far as I know, the following national providers exist in Detroit: (This is by no means, a complete list, and some of these aren't "Big Name" providers, but less likely to go under than John Q Random ISP) The above is completely false statement in my opinion. From what I'v seen, local "John Q Random ISP" is a lot more likely to be stable, have good support and less likely NOT TO go under then most of the "Big Name"
providers you listed (see comments).
Verio Dubious financial situation, although after restructuring and closing of many offices & datacenters, they're getting better. Their network however is fairly bad compared to majority of other national ip providers.
Winstar No longer in business.
XO Very very serious financial problems. Problems with their network as well.
AT&T Fairly stable due to the fact they do not have large bonds/loans as they used to have lots of money when they were giant monopoly long distance. But they continue to loose share of long-distance market and although they bought former Northpoint network, they do not seem to be using it yet (???). Nobody is sure what their future business plans are regarding internet or anything else...
Qwest Serious financial problems. Customer service needs much improvement...
Covad Out of bankrupcy but not yet stable. They buy majority of their intenet transit from Level3 and do not really have national ip network.
Level 3 Stopped selling directly to small customers. You have to buy OC3 or above or at least 100Mb and even that is not enough for them to take you seriously (Note: known to still sell DS3 if you push them, but no longer T1s).
SBC Big bad wolf ILEC :) Probably most stable of the list you have. Most expensive too. I personally avoid doing business with SBC because of how anticompetetive and anti-ISP they are.
I've heard rumors of speakeasy.net, which I think resells internap Primarily DSL ISP. I'v heard bad things about how they tricked resellers... Check on dslreports on opinions about their service.
I'm sure there's a few other national networks, these are just the ones I have heard of off the top of my head. Of course, there are other networks - Worldcom/UUNET, Sprint, CWUSA, etc
But if I were you, I'd to yellow pages for list of ISPs or the some other list of local isps (their used to be ones from boardwatch, thelist.com, etc; check only for the ones that actually have office in your area) and check with them if they can provide you with T1. P.S. I'm only expressing my personal opinion about companies above, I do not wish to initiate any kind of discussion with others on the list that may work for these companies or would otherwise have different view on this.
-Paul
On Thu, 2002-05-30 at 19:27, Brian wrote:
Surprised there isnt much connectivity in the Detroit area, I mean it is Motor City and all, I would think tons of manufacturing palnts all needing telecom of some sort or other..
Bri
On Thu, 30 May 2002, Deepak Jain wrote:
Clearly anyone in your market is buying from someone outside of your market. The fees associated with reliability (if available) are a function of your geography. Large providers are concentrating on the markets that are making them the most money.
If you get a few networks in your area that want to save money on the cost of reliability you can run a couple of circuits to the next large market and try to knit together a reliable network and divide the costs that way.
My guess is that with more large providers on a profit-centered basis you won't see the same kind of pricing equality you have been seeing between Tier 1 and Tier N markets anymore.
Deepak Jain AiNET
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of John Palmer Sent: Thursday, May 30, 2002 5:39 PM To: nanog@nanog.org Subject: RE: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions)
Its just that they aren't local and there is no need to pay for a circuit all the way to Chicago. It seems that so many providers have moved out of Macomb county. Anyone have any experience with BigNet? We are talking to them now
-----Original Message----- From: Bill Woodcock [mailto:woody@zocalo.net] Sent: Thursday, May 30, 2002 4:42 PM To: John Palmer Cc: nanog@nanog.org Subject: Re: Trying to find a connectivity provider that wont go under (was RE: CAIS/Ardent and now Network Access Solutions)
> Who can one rely on for connectivity? In general and in the Detroit area? > I put out a request for bids on T-1's and all the national providers were > way too high...
Haven't you just answered your own question? I guess if you think reliable service is too expensive, you're not in the market for reliable service, no?
-Bill
On Thu, 30 May 2002, Brian wrote:
Surprised there isnt much connectivity in the Detroit area, I mean it is Motor City and all, I would think tons of manufacturing palnts all needing telecom of some sort or other..
Try to get DSL here; everyone backhauls to Chicago. And Cleveland is the 25th largest city in the USA. Lots of local providers for DS1 and Frame and ATM, just not DSL. :) -- Steve Sobol, CTO (Server Guru, Network Janitor and Head Geek) JustThe.net LLC, Mentor On The Lake, OH 888.480.4NET http://JustThe.net "In a 32-bit world, you're a 2-bit user/You've got your own newsgroup: alt.total.loser" - "Weird Al" Yankovic, "It's All About the Pentiums"
On Thu, 30 May 2002 16:24:53 CDT, John Palmer <jpalmer@utilinc.com> said:
way too high, mostly because they have no local POPs. We really need to be as close to the top as possible, not because of bandwidth needs, but for reliability's sake.
This assumes that "the top" is any more reliable/clued/financially solvent. Scanning the last year's archives for "chapter 11" and/or "depeering" might be quite enlightening....
On Thu, 30 May 2002, Christopher X. Candreva wrote:
NAS has been nothing but trouble. We are (or were) a Covad reseller, first direct through Covad, then through CAIS.
Jeez. About 18 months ago I couldn't even get CAIS to return calls to their sales department. I guess I'm glad they didn't. -- Steve Sobol, CTO (Server Guru, Network Janitor and Head Geek) JustThe.net LLC, Mentor On The Lake, OH 888.480.4NET http://JustThe.net "In a 32-bit world, you're a 2-bit user/You've got your own newsgroup: alt.total.loser" - "Weird Al" Yankovic, "It's All About the Pentiums"
participants (12)
-
Bill Woodcock -
Brian -
Christopher X. Candreva -
Deepak Jain -
John Palmer -
Paul Timmins -
Richard A Steenbergen -
Scott Granados -
Steven J. Sobol -
Tim McKee -
Valdis.Kletnieks＠vt.edu -
william＠elan.net