We're not very happy with Level3 anymore either, terrible support, no RFO is ever given, tickets are closed with no explanation at all. They bought so many providers close together that they have a lot of work to do to integrate everything into a workable set of products. George Carey wrote:

As a Level 3 customer who had connectivity to some legacy Wiltel equipment I can also attest that service levels were pretty bad following that merger. We had a few memorable outages where escalations were necessary to get things resolved and folks familiar with the equipment were hard to find. In meeting with our account team we were able to get them to agree to roll us off the old gear which was really inadequate anyway for the purpose which happened to be an Option A MPLS NNI. It took them a long time to get that done but since then I would say that service levels have returned to the normal albeit somewhat less than desirable levels. We have had relatively few problems with our vanilla transit connections though. The same goes for some long haul that they inherited with the Broadwing acquisition. There were some glitches with some of the fiber grooming they did a while back but that seems to have passed. I also seem to be receiving less maintenance notifications overall from Level 3 as a general trend. Hope it continues.

George

Will Orton wrote:

We have an old Wiltel DS3 homed from their Anaheim POP... all traffic that's not going to destinations on the old 7911 backbone seems to be backhauled to Level3 in San Jose before getting anywhere else, like so:

1 (internal) 2 (internal) 3 (internal) 4 anhmca1wcx1-atm10-0-0.wcg.net (64.200.142.169) 8.017 ms 8.596 ms 8.138 ms 5 anhmca1wcx3-oc48.wcg.net (64.200.143.65) 9.325 ms 8.056 ms 9.485 ms 6 64.200.249.122 (64.200.249.122) 79.108 ms 251.438 ms 222.596 ms 7 64.200.249.142 (64.200.249.142) 16.357 ms 16.167 ms 17.004 ms 8 te-3-2-70.car4.SanJose1.Level3.net (4.68.110.25) 12.568 ms 13.195 ms 13.384 ms

So they don't seem to interconnect 7911 and 3356 in Los Angeles. We complained about this a year ago and they basically said, "tough, you bought IP transit, we're giving you IP transit".

Anyway, so in the meantime we bougt a new gig-e to 3356 in San Jose...

1 (internal) 2 (internal) 3 (internal) 4 (internal) 5 ge-x-x.car4.SanJose1.Level3.net (4.71.x.x) 9.063 ms 7.780 ms 8.460 ms

We have had the same issues with the Broadwing network. Our OC-3 connections are in Wyoming and they lugged everything through Salt Lake and then to San Jose adding about 30 - 50 ms. We placed orders with Level 3 to move them onto the Level 3 POP's last year and so far we have one of them up that they finished last week.

So it almost looks like my gig-e in San Jose is off the same router they backhaul 7911/Anaheim via.

I wouldn't take a new connection to 7911 unless you're okay with this sort of thing. Performance has been fine other than the 5-10ms of extra latency, but the asthetics bug me more than anything. If they communicated more about what they're doing on the 7911 net and what the PLAN is for transitioning things, I'd be happier. But it's been this way for years now so we're just disconencting the old one, which is probably what they're waiting for everyone to do rather than really merge the networks.

-Will

On Wed, Jul 01, 2009 at 11:15:42PM -0700, Scott Howard wrote:

...

Date: Wed, 1 Jul 2009 23:15:42 -0700 Subject: Level 3 - "legacy" Wiltel/Looking Glass bandwidth From: Scott Howard <scott@doc.net.au> To: nanog@nanog.org

We're looking at getting connectivity via Level 3 in a particular datacenter, but we're being told that it's "legacy Wiltel/Looking Glass" rather than "true" Level 3.

Given that both of these acquisitions occurred years ago should I be worried, or is this "legacy" connectivity the same as L3 at any other datacenter?

Scott.

We have also had the fun of the musical sales reps as we are on our 4th one since they took of Broadwing... Thanks Eric Nowland Sr Network Engineer Cerento/Wyoming.com

I could not agree with the points below more. Prior to the mergers, I had multiple services each with Looking Glass, Wiltel and Broadwing and Level3. After Level3's round of acquisitions the service level for all four of them went way down. I've had the experience of not being able to resolve issues with Wiltel circuits because there was no techs available who could access the gear, been told they no longer wished to provide me with a Type 2 service sold to me by Looking Glass or Broadwing, and had billing and implementation issues that have lasted almost two years with Level3, because they started moving services from one billing system to another. Given that Level3's prices are usually not even close to competitive with solutions provided by other providers, I would suggest that people look elsewhere for reliable, reasonably priced services. Shane On Jul 2, 2009, at 2:50 PM, Deepak Jain wrote:

Without continuing the L3 pile-on, one can easily glean from their public filings that they have never properly filled out their management depth in acquisition absorption and/or sufficiently empowered those folks. The billions in revenue lost from acquisitions like Genuity and others have told this story more than once.

L3 is not alone in this. Worldcomm's failure to integrate acquisitions led to a much larger operational cash need than VZ has shown for the same assets (verio, lots of other names here). This is because VZ understands how traditional businesses acquire others, better, in my opinion.

Unfortunately, L3 has shown little interest in making the "real world, tough business" cuts in heads and absorbing the real (internal) pain of acquisitions and seems to have a pretty laissez- faire attitude towards its customers, even at its senior management levels (Cxx). I think this will be (and has been) the biggest problem for them. Even a possible merger/JV with Sprint may not be sufficient to solve that. Their resolution of billing disputes is much more typical of WCOM than VZ.

They are a big fish in lots, and lots, of markets. They enjoy being able to dictate pricing in them. IMO, however, they don't have the maturity of (say, AT&T or others) to take that big fish status and leave you still happy with the service. (colloquially: if [good companies] are going to take advantage, at least they don't make it more painful than necessary).

Operationally, where you have options (because of pricing, locality, etc) it's long-term good to support competitors, diversity in connectivity, etc. History has shown time and time again that when an industry consolidates a lot of business with a certain vendor, bad things can and do occur.

Deepak Jain AiNET

So..... where is all this talent going? NTT? AT&T? Verizon? Dare I say it.... cogent? :) Also has anyone filed complaints with the FTC or DOJ? Jason LeBlanc wrote:

To boot almost all the original Telcove crew we had are gone. They're losing the better people through attrition as they're frustrated at not being able to help their customers. I also have a feeling Level3 makes changes during business hours that are not announced. I have no proof but I have a feeling due to some odd changes in routing I see every now and then.

On Wed, Jul 8, 2009 at 10:27 AM, Kretchmer, Sam<samk@playboy.com> wrote:

Anyone on here care to comment on Bandcon transit services? Anyone even using them? They are offering me an incredible deal on transit, and I was wondering what their reputation is.

I've used Bandcon in the past, and we are actually going to be pickin up some new transit from them shortly.... They have great deals.... but Their support sucks really really bad. I had a level 3 outtage and it took 10 calls to finally get them to do something. Things might have improved by now, but no promises. If you are getting a large amount of bandwidth ask for direct access to the carriers noc. That how we do it.

Their support sucks really really bad. I had a level 3 outtage and it took 10 calls to finally get them to do something. Things might have improved by now, but no promises. If you are getting a large amount of bandwidth ask for direct access to the carriers noc. That how we do it.

They may have also been getting the runaround from Level3. I had a hard down issue about a year ago when a L3 tech unpatched a mislabeled cable in the MMR. It took getting my local sales rep and his boss involved to have someone drive back out and fix it. To get back to the point, there's no excuse for Bandcon or any reseller not to return calls promptly and provide regular status updates -- even if their upstream support is unresponsive. Network Innovations, another reseller, has fantastic customer support.

On Wed, Jul 8, 2009 at 11:52 AM, Robin Rodriguez<rrodriguez@ifbyphone.com> wrote:

I don't have any usage experience, but would be very interested from anyone who does as well. We have spoken with them about long-haul circuits (with small to no commit) and their prices are indeed incredible. The prices we heard were for Equinix to Equinix circuits (specifically CHI1 & CHI3 to DAL1 & NJ2) they also quoted us great deals on resold IBX-link to get to IBX's that they don't have a physical presence in (they aren't in CHI3 for example). I do wonder how they can undercut everyone's price by such a margin. Were you seeing great quotes into non Equinix facilities?

Simple, they're oversubscribing their transport circuits and letting users fight for bandwidth. Basically what they're doing is buying a 10GE unprotected wavelength from a carrier, dropping a switch on the ends, and loading up multiple customer VLANs onto the circuit. There are no bandwidth controls, no reservations, no traffic engineering, nothing to keep and the circuit uncongested, and these are unprotected waves so they go down on a regular basis whenever their carrier does a maintenance. How they implement multi-point service is even scarier, they just slap all your locations into one big VLAN and let unknown unicast flooding and MAC learning sort it out. Most serious customers run screaming, I'm sure you can find some former customers who can describe the horror in more detail off-list. When things break, their support is nothing to write home about.  They often brag that they have a former Level3 engineer on payroll, unfortunately he's nowhere to be found, and their suport people aren't terribly sharp on those rare occasoions when they *do* answer the phone or respond to e-mail.  Like someone else pointed out, multi-day outages aren't at all uncommon, so if you end up going with Bandcon, make sure you have sufficient redundancy in place. Since they can't really compete on quality, they compete instead on price.  Their sales force spams and cold-calls every website, ARIN, peeringdb, etc on a regular basis, and can't take "no" for an answer. The following exchange sums it up nicely (warning: foul language): http://attrition.org/postal/z/034/0931.html They are currently running a $2.50/mg transit promotion, which makes me wonder how they're doing on their Level3 and Global Crossing bandwidth commits and whether or not they're solvent. Drive Slow, Paul Wall