[Comment: I had stated that I would not post a summary back to NANOG re: this - however - the requests to do so were numerous so I re-neg'd please beat me privately if you disagree - I tried my best to keep respondants annonymously quoted] In the course of networking events you become confronted with the e-e-evil scourge of marketting and it's near death-grip hold on the ears of upper management (sorry for the unavoidable visuals here) - that sets the scene for my predicament and the need to validate my statements to my superiors by my peers and mentors here... My list was short - it was prompted by a short list presented by our marketting folk re: possible alternatives to our current provider. They are (along with our current provider)... ABOVENET AGIS CIGNAL EXODUS GTE LEVEL3 SERVINT VERIO There were some honorable mentions from the respondants: (one mention) RMINET CONCENTRIC NAVISITE (quite a few mentions) Frontier Globalcenter *very highly recommended by all mentions* InterNAP The ratings so far read as follows: (note this is subject to my interpretations of the review) scale: 1-10 10 being the unblemished lamb 5 being the starting point AboveNET - 9 Level3 - 5 Exodus - 5 Servint - 4 GTE - 4 AGIS - 2 VERIO - 2 CIGNAL - 0 (no data) ======================================== The Breakdowns: ======================================== ************* ABOVENET: 9 Didn't receive any *real* negative criticisms - although there was some historic comments of lack of adequate back-up power at one time (although during the DC blackout last year they remained operational on their own juice while most otheres faultered) , a concern about their future since the MFN merger, and also some comments re: Dave Rand's 'best-exit' decisions - however all and all comments were near unanimously positive... Have to confess I am a current subscriber of their services and extremely impressed by their whole deal... they really are the provider that I'm trying to defend against Mickey and his marketeers here in the corral. <respondants comments re: abovenet> 'excellent, they can put filters in, normal noc, really no major outages. Fine co-los. A' 'AboveNet just got absorbed by Metro*mumble*Fiber*mumble* :) Dunno how their service is since the absorption' 'We currently have a Colo at AboveNet's San Jose facility and are very happy with their responsiveness and preofessional network and facility engineering.' 'Their new SJ facility has the most redundancy I've seen. Great network compared to most colo providers. Focused on connectivity only-- no value-added services.' 'AboveNET has always been good to me... They are very responsive to problems and very helpful when dealing with remote administration of hosts. Their "Remote Hands" has helped me quite a few times... They have some great Players in their technical staff: Dave Rand, Avi Freedman, Brian Moore... ...they are nt immune from growing pains... But they did respond, as always, very rapidly,... In general I give them an uptime, reponse to problems, honest - timely communications of problems and their caused/resolutions ... an A+. Or a 10 out of a possible 10 scale.' 'ABOVENET 8... 1 = I wouldn't have them host my dog's website 10 = I'd trust them to host my pacemaker' 'Avi works there. Dave Rand works there. =) Very peering oriented, but not even "fat pipes to nowhere" like Level3/Qwest, since they didn't (until Metromedia buyout) have a telco relationship. Way too much of our traffic saw the MAEs, but that's the way it is w/networks who don't pay for transit and aren't big enough for peering... For all the praise Abovenet gets, it's basically just a well run interconnected bunch of NAPs... It's very good NAP connectivity, no doubt, but traditional tier 1 doesn't work very well anymore unless you're large enough to get good private peering too. Rating: A' 'Above is good...' 'Great. Only based on my evaluation of their network, peering relationships, and responsiveness of their sales staff. My first choice for colo service.' 'Excellent luck with these people from a network perspective. Never had power problems, but that used to be their big problem at one time according to friends of mine. My experience with them (simple 1 box colo) has been very good. I keep a proxy server on a box there to this day specifically because routes to them always rock, wheras if I go direct from my place to "xyz" it may suck.' 'In an alternate life, I have hardware at the Abovenet facility in Tyson's, VA. They've been pretty accomadating for us, but I'd recommend you have seasoned staff on your end to deal with them. My impression is that they have talented staff, but they're either not interested or too busy to be very in-depth with your problems unless you can give them a softball, it takes time to get results. I suppose that's typical of the environment. Their backbone and connectivity seem quite good. I have no complaints that way. Rates are OK, too.' 'I've been using AboveNet for all of my co-location in various parts of the US. I love their support staff, and the connectivity and peering has been really good to me. They've had their share of problems, but have done pro- active things to assure they don't happen again, or at least as often.' </respondants comments re: abovenet> ********** LEVEL3: 5 Most negative feedback hinges on poor peering and support (pre and post sales) and lack of VA services. <respondants comments re: level3> 'ok co-lo, about B to C for the backbone performance, same for the tech support, can do filtering, no CAR, around B' 'No cages-- just private rooms or cabinets. My personal favorite last I researched colos.' 'I like this company a lot, but am not yet a customer. Their colo space is just racks. Their network is fast but I've been told that their peering arrangements are still being worked out.' 'We got as far as receiving a bid from them. They claimed: a) they had peering or transit in Seattle (other than level3.net links) b) the routing loops we were seeing on their network (traceroutes/packets that wouldn't go to the right POP) was an "Internet BGP issue" c) they had a 24x7 manned Seattle NOC when: a) they didn't b) it was fucked up IGP c) it's totally unmanned; after we said no, a friend of mine's company got investment from them and noticed that their NOC was totally unmanned and, while externally secure, allowed customers access to most other customers' eq.' 'For IP dunno. Everything goes through Chicago to get to the rest of the internet (ie: horrible peering). Horrible experinces with their internal processies.' 'Great so far. Just moved my servers there. Fabulous facilities. Large. Lots of fiber, private rooms forstaging no extra charge. Sales staff is clueless in Detroit but NOC in Denver very responsive. Security is b+.' 'I have a personal box that I use the most off of level 3, a T1 away from their SF facility. Tolerable. usually up. if I could afford it I'd put this personal box at abovenet tho.' 'ok' 'Not using them yet, but hinking real hard. NICE colo space, I know people located at Level3 in Detroit and they are happy with most service. Apparently you need to manage them a bit hard to get initial service installed, ongoing is mostly OK.' </respondants comments re: level3> ********** EXODUS: 5 I mix of polarized responces... most of the negative items center around quality of support and network performance (unfortunately so do most of the positive ones) - there is mention of the inability to terminate circuits from other providers within their space (from first-hand experience I can validate this) plus that they are usually located off-site to most NAP's IX's eliminating the abilty of direct cross-connect and limiting interconnects to their customer base. <respondants comments re: exodus> 'blah.. cannot and will not do any kind of filtering, no ACLs, no CAR... good looking co-lo, kind of expensive, designed more for like web applications, real audio/video. D to F' 'Exodus is very cool, but you may find it's towards the top of the cost spectrum. I am biased towards Exo, though. :)' 'Expensive. Lots of VA services. Can't bring in connectivity from other providers to supplement Exodus' (at least, officially you can't). Backbone has been improving recently. Customer support has gotten a lot better too. Lots of growing pains. (Only one I have personal colo experience with.)' 'With Exodus I have four cages and a couple fast ethernet feeds. Exodus provides an excellent colo space and many services which make it a plus - expecially for a small site (up to four cages). We are running 50-70Mbps through Exodus and because of routing and limited local peering I'm not sure I want to go much over 100Mbps. Sometimes hard to get out of the cookie cutter with Exodus.' 'Exodus has been very reliable for us. We have a rack in their Santa Clara IDC and a T1 to our office. Their project managers sometimes don't communicate fully with sales/billing, and we often get overcharged or similar. Their data centers are clean and their NOC staff is willing to be flexible for almost any request. For example, they hauled people out of bed to allocate me an additional range of 16 IPs just for our DMZ on a Sunday night around 11:30pm, w/o warning, at my request (my PM was supposed to warn them, but didn't). They also have no workbench area and have no plans (I've repeatedly asked) to allocate any space for such.' 'They're good for space, but their network is still being revamped, esp. on the east coast. However they're no longer permitting other networks into their colo space as of a month or so ago. A lot of people wouldn't be at exodus if they couldn't bring in a better ISP.' 'EXODUS 9 1 = I wouldn't have them host my dog's website. 10 = I'd trust them to host my pacemaker.' 'Badass data centers (sttl1 was cool, sttl2 has biometrics for NOC staff, the whole works). Nordstrom is doing all hosting w/EXDS sttl2. Connectivity is generally good, both in terms of latency and capacity. There are intermittent *WEIRD* backbone issues; for a minute or two occasionally (2-3x a week that I notice) their backbone will stop routing traffic. They also seem to be a couple weeks behind capacity requirements - sttl1 peering is pretty well exhaused but they're upgrading peers to OC3 and core to 2xOC12 (from DS3 & OC3). On the upside, the nature & size of their business (pushing 85% of their exchanged traffic) gives them good peering influence; they've got UUnet and Sprint going to OC3 from their present DS3s (in Seattle). Good people. rating: B- B+ if you're multihomed' 'I dunno about the rest, except for Exodus, which is without a doubt, absolutely, positively, the single worst company in the known world, let alone the worst network/colo provider. I would sooner kill my mother than recommend that anyone ever use their services. (and i'm speaking as a customer that is unfortunately locked down into two very large cages in an Exodus facility that we're working on moving out of even though it's probably going to cost us upwards of six figures).' 'First ones to do the data centers but didn't think of their network as much. Very high latency. One of their board people called their customers roaches and their data centers a roach motel. "They come in but they never leave".' 'Bad. Hoppy network. Problems getting to server from AOL.' 'Exodus has improved a bit, we get a nice feed most of the time, and if you're big enough to get a port on a router instead of a Catalyst, it's pretty nice.' 'Exodus is ok unless you let their engineers get involved. Then they are a nightmare. I have had them underprovision hardware, get ACLs backward, lose T1 circuits for several days in their internal wiring, etc.' 'We use them at work. I wish I could suggest them. I'd suggest having your wisdom teeth drilled on by your neighbor bubba wtih the black-n-decker before going to these guys. However, this is my opinion, and I don't speak for the company I work for, and you never heard it from me. (sigh)' '...we colo'd with Exodus in New Jersey and they sucked more than I can say. Exodus brings a bad taste to my mouth. They don't return calls, they promise the world and deliver shit. Avoid them at all costs.' </respondants comments re: exodus> ************* SERVINT: 4 SERVINT was thrown in her at the request of my marketting dept - who bumped into someone from their equally superior department who suggested us colo'ng there - as most of the responses seem to indicate there isn't much support for colo at SERVINT or at least much public knowledge of any. Really only 1 negative comment - but as you can see they have a high quality Tee! <respondants comments re: servint> 'they start to suck lately, NOC is ok, they can do filtering already doing CAR for ICMP across their backbone. I don't know about their co-lo never been there. can't relaly tell..' 'Pretty small from what I understand.' 'Don't know about the others, but I've got tee-shirts from AGIS and SERVINT. AGIS one has a nicer logo/design, but the SERVINT one has higher-quality fabric and construction. This is all the first-hand stuff I know. (:' 'Never heard of them' </respondants comments re: servint> ************* GTE: 4 Most comments were negative or showed an indiffernece to an established provider - this is more akin to rating UUNET's performance - good bad or indiffernt - it never lived up to expectations. <respondants comments re: gte> 'nap.net co-lo that i saw in downtown chicago, co-lo seems pretty sucking, but cabinet and transit price is very reasonable... can do filtering, very fast responsive noc. B' 'GTE 5 1 = I wouldn't have them host my dog's website. 10 = I'd trust them to host my pacemaker.' 'Had good experience with BBN in Boston, no experience in 2+ years though.' 'GTE is ok' 'GTE - (was BBN), BBNPlanet were internet Pioneers. Not in all locations but good in my experience' 'I'd stay away from GTE. Used to be Genuity, they've yet to integrate the wacky ATM backbone fully with the BBN backbone. Still on the Genuity AS, so there's much sucky peering. If we didn't have 40-some T's terminating at the colo we'd be long gone...' 'GTE was pure web hosting, last I heard.' 'GTE - run' </respondants comments re: gte> ************* AGIS: 2 This was pip - at least in networking circles it seems that you learn from history - AGIS has had a historic black eye in regards to a bit of net-abuse originating from it's borders - and it looks like most of the respondants are sticking to that lesson. Thre are a few direct quotes from some former AGIS subscribers (leased line) and a descritpion of a sub-standard (to say the least colo site - although it may not be typical). <respondants comments re: agis> 'I don't know much about AGIS as Co-lo provider, but I did live through AGIS as a T-1 customer to their backbone during their worst time in history... The time when they had their Spam/Hacker Attack Problem that ended up with and FBI (or some sort of Federal?) investigation. Not a nice time to have been on AGIS. They were down more than up during that stuff. We finally had to drop them and got MCI T-1 and later UUNET also.' 'Having been a customer of their ISP services I don't have anything good to say about them.' 'AGIS Unknnown -- ISP expreiences haven't been great' ' "AGIS does colo?" ' 'Don't know about the others, but I've got tee-shirts from AGIS and SERVINT. AGIS one has a nicer logo/design, but the SERVINT one has higher-quality fabric and construction. This is all the first-hand stuff I know. (:' 'AGIS sucks' 'Horrible, bgp resets middle of the day without knowing why, don't push much traffic to them compared to our other providers...' 'Used to be below acceptable.. Have seen nothing to change this opinion. The one Coolocation site I have inspected was small and in a multi-tenant strip close to major traffic routes. A single 250 MWatt generator for backup. No DC power.' 'We and most other ISPs in Michigan used to use Agis. Most of us have left. I have clients who colo there and I have had to deal with them recently. Not impressed.' </respondants comments re: agis> ************* VERIO: 2 Vericutous borg... asimilator of many ISP's bears a stigma so it seems also. Lots of negative feedback on their support talent and the instability of their *acquired* network expecially in re: to the BEST absorption. <respondants comments re: verio> 'blah.. blah blah... depends where you are, and what ISP/region are we talking about, Verio seems to fsck everything up, I'd give them C for backbone performance, F for tech support, about B for security response if you know the right people. Can put CAR and Filters in with no questions, if you're looking to co-locate irc-like material ;-) ircd for instance. around C to D (F - if you need fast tech support)' 'Verio *sucks*. They're experiencing serious cranial rectumitus when it comes to their network backbone.' 'Verio... First of all they have a nice backbone.. However I am not sure if you are aware of the fact that they built there customer base by buying smaler ISP's... ...Now this could be a good thing or a bad thing .. just relize that when you buy from verio you may be hooked up to a local office that has just been bought out and is in the middle of a staff turnover' 'We currently have another "colo" at Best (now absorbed by Verio,) in San Francisco and are NOT pleased with it. Indeed compared to other colo's I have worked with in the past, I would not call that data center a colo. We have had to chase after them about seriously out of spec A/C, have had prolonged power failures (they may be installing site UPS at this time, it wasn't there in the past,) significant network outages, and a lack of responsiveness when we want to report problems. We also have one of our offices connected to the net through Verio as an ISP (legacy of the previous tenant,) and again have had regular and sometimes prolonged network outages. I will credit Best/Verio with making some network configuration changes to correct a congestion problem when one of our engineers started complaining vigorously about the poor performance of one of our circuits. They shouldn't have needed us to detect the problem though...' 'In SJ, lots of problems integrating Best. Poor customer service, relatively unstable network, poor facilities.' 'With Verio I have four enclosed racks and two fast ethernet feeds. The Verio backbone isn't as oversubscribed as Exodus and in my opinion has a better architecture. With Verio I get links directly to their backbons routers and they will do access lists for me. With Exodus my links are into Catalyst switches using RSMs and they will not do access lists. In the Verio space I use they do not have any 24/7 staff onsite - so they can't reboot a machine for me in the middle of the night.' 'VERIO 4 1 = I wouldn't have them host my dog's website. 10 = I'd trust them to host my pacemaker.' 'No experience as a customer, their latency always seems higher than it should be in traceroutes.' 'Bad. Based on my knowledge of only a small part of their operations here in Michigan. No redundancy in their network.' 'I have first hand experince with VERIO, and they more than blow.' 'VERIO - ok' 'OK as a backup provider. Pretty good connectivity to other Verio customers. Not impressed for backbone connectivity. They have not had a chance to integrate their purchases.' </respondants comments re: verio> ************* CIGNAL: 0 CIGNAL gets this rating - due to lack of data - from what I've seen they are extrememly new... <respondants comments re: cignal> /dev/null </respondants comments re: cignal> -- I am nothing if not net-Q! - ras@poppa.thick.net