Speaking in my personal, non-list-administrator, not having discussed this with anybody else, capacity, I think that notifications of large-scale outages affecting large numbers of networks are a really useful thing to have on the NANOG list. Assuming this list has large numbers of people who operate networks and have to troubleshoot problems when they see them, telling people that the problem is already being worked on can save a lot of people some work (and can hopefully also reduce the number of phone calls that NOC people have to juggle). To that end, the first couple of messages in this thread are the sort of thing I wish the NANOG list would have more of. -Steve On Tue, 7 Jun 2005, Rachael Treu Gomes wrote:

On Tue, Jun 07, 2005 at 12:34:33PM -0400, Joel Perez said something to the effect of:

...

I totally agree with you Richard.

So do I, but probably more so with his encouraging your patience than you appear to.

...

But, in this case all im getting is the run-around from GBLX when calling them about it. I managed to open up a trouble ticket with them but their Techs weren't telling me anything other than they will look into it and call me back.

Just out of curiosity, why did you phrase the above as "*managed* to open a trouble ticket"? Did GBLX's unwillingness to describe the nature of the outage that you allege also extend to their willingness to help you in general? What you're saying sounds as though they were both relucatant to open a ticket for you *and* to tell you why they wouldn't and, frankly, I find that difficult to believe.

I haven't (and I wouldn't want to either, as I've seen how much interference they have to run and how much ebb and flow is involved in the climate and the info they receive), but If you have ever been in the employ of a provider's customer-facing NOC during an outage, you know the following: while it is standard practice to give at least short-but-informative answers to customer questions in those situations, it is imperitive that task priority also be lent to remediation of the problem and managing call volume, particularly in the early stages of an incident.

(By my estimations based on when you started querying this list, you called GBLX within an hour of the fiber cut, when it stands to reason that the providers are doing their own recon on what happened and are less likely to be able or willing to disseminate what may amount to misinformation.)

Also, are you aware that the groups handling customer circuits and calls is often disparate from the one managing the state of a backbone outage?

...

Even though I am a customer, im not getting any answers so I tried the list as a last ditch effort to get some info.

How last ditch, by the way? How many people did you talk to?

By no means am I trying to antagonize you with these questions, but am taking the opportunity to conduct my own study on the average customer threshold for information gathering and return on investment in informational resources made available to them.

...

Good luck, --ra

...

-------------------------------------------- Joel Perez | Network Engineer 305.914.3412 | Ntera --------------------------------------------

-----Original Message----- From: Richard A Steenbergen [mailto:ras@e-gerbil.net] Sent: Tuesday, June 07, 2005 12:28 PM To: Joel Perez Cc: nanog@merit.edu Subject: Re: GBLX congestion in Dallas area

On Tue, Jun 07, 2005 at 12:09:26PM -0400, Joel Perez wrote:

...

Is anybody seeing any congestion in the Dallas area for Global

Crossing?

...

I'm seeing packet loss to some of my equipment up there.

There is a large fiber cut in the area (somewhere between Dallas and Houston), affecting a lot of capacity coming out of Dallas on several carriers (including GX and Qwest at the very least). Two of our OC48s on

this path have been down since around 14:57 UTC.

That said, this isn't the proper place to whine about congestion. Normally I would say that is what customer support numbers are for, but since there is nothing they can do to splice it any faster, I'm going to recommend a

healthy dose of suck it up and deal. :)

-- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

-- rachael treu gomes rara@navigo.com ..quis custodiet ipsos custodes?.. (this email has been brought to you by the letters 'v' and 'i'.)

-------------------------------------------------------------------------------- Steve Gibbard scg@gibbard.org +1 415 717-7842 (cell) http://www.gibbard.org/~scg +1 510 528-1035 (home)

At 06:12 PM 6/7/2005, you wrote:

If we started posting about every fiber cut of every carrier anywhere in North America every time it happened there wouldn't be any room left on this list for talking about spam, senderid, DNS RFCs, E911 for VoIP carriers, err... wait which side am I arguing again? :)

I don't operate even a mid-size network, but when there's an outage that effects more than a handful of locations, it's useful to know about it... Not that there's anything I can do about it... but when customers are calling and asking why they can't reach their application servers, it's nice to be able to tell them there's a problem at $location and that no, I don't know when it'll be fixed, but they can be sure it's being worked on. Sure, I can tell 'em that there's no problem with our network, and that our connectivity is fine... and leave them to figure out if there's a break someplace - and I do. But I think NANOG is certainly an appropriate forum for medium/large-scale outages - unless someone's created an outage list someplace. I will agree that it's not the place to bitch about a vendor not giving more specifics, dumping on vendors in any way, actually...

Jay R. Ashworth wrote:

...

From down here, like Dave, at the relative bottom of the food chain, I must agree with him and Steve, though I do understand Richard's concerns there, and they're valid ones.

The Internet needs a PA system.

Problem is, the people who are equipped to talk, and, by and large, many of the people who want to listen, are all *here*.

OK, make a separate list (i.e. nanog-alerts) with posting restricted to those on nanog-alerts-post, and we develop a policy for being accepted/removed on the -post list. Easy enough. pt

On Wed, Jun 08, 2005 at 09:22:02PM +0300, Petri Helenius wrote:

Jay R. Ashworth wrote:

...

The Internet needs a PA system.

There is this sparsely deployed technology called multicast which would work for this application.

Well, that's fine, at the transport layer, but I think more an application layer solution is called for. Something akin to news.announce.important on Usenet? Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Designer +-Internetworking------+----------+ RFC 2100 Ashworth & Associates | Best Practices Wiki | | '87 e24 St Petersburg FL USA http://bestpractices.wikicities.com +1 727 647 1274 If you can read this... thank a system administrator. Or two. --me

It seems like it's taking more time to discuss it than it actually would take to create a nanog-outage list. Maybe it's not being done because doing so would be threatening to a lot of people. Having a large sounding board for outages will make it very difficult for larger providers to cover up outages. Having a web archive of postings during the outages will make it harder for people to forget those outages after they're corrected. Seeing lots of [reported/speculated] outages on a web archive from a larger NSP/ISP might make that network look bad, even though a large network might have a small percentage of outages when measured against the number of customers it serves. Furthermore, when someone does something stupid, it will be harder too forget/ignore the lessons learned. I'm sure that's bound to make someone feel threatened. ;-) And last but not least: nanog-outage may become more operationally relevant than this list. :-) -Jerry

...

The Internet needs a PA system.

There is this sparsely deployed technology called multicast which would work for this application.

Note that the original poster did use multicast for his query. He sent one copy to nanog@merit.edu where his email was replicated and forwarded to multiple recipients who had subscribed to that stream. IP multicast is not the only way to do multicasting on the Internet. --Michael Dillon

nanog-outage@ ? ? ? On 6/8/05, Michael.Dillon@btradianz.com <Michael.Dillon@btradianz.com> wrote:

...

My concern would be that by openly encouraging people to send in more reports of or inquiries about outages, we are going to see a lot more noise from unqualified folks wanting to "be cool". I personally don't want to hear about it every time someone wants to vendor bash ("@#$%^&ing GX is down again and their customer support sucks"), every time a T1 in Bumblescum Nowhere goes down, or otherwise completely useless posts ("did anyone see anything funky on level 3 on the east coast yesterday?").

Perhaps the best way to deal with that problem is to wait and see if it actually happens. In the past NANOG has carried a lot of these outage reports during a time when the net was less reliable than it is today. It didn't overwhelm the list back then. I would assume that because of the level of effort that operators put into having resilient networks, there would not be a huge amount of these outage reports because most real outages will remain invisible to customers.

--Michael Dillon