It's impossible to contact some of these companies; emails sent to role contacts just go unanswered. I had lots of problems trying to contact one of the above mentioned ones (about peering) recently. All contact numbers divert into a telephone drone repeating "we do not have technical support at this time". Contact with humans is strictly forbidden. I suppose your hope-of-last-resort is that someone from $Giant_corp sees you posting on Nanog and replies. At the end of the day this is never going to stop. It's much easier posting to Nanog than spending 4 hours on the phone trying to locate someone with a clue in one of these big companies. Ivan -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Valdis.Kletnieks@vt.edu Sent: 03 February 2006 03:40 To: Gadi Evron Cc: n3td3v; nanog@merit.edu Subject: Re: Yahoo, Google, Microsoft contact? On Fri, 03 Feb 2006 01:43:17 +0200, Gadi Evron said:

n3td3v wrote:

...

I recommend you guys setup a "Don't have a contact for" list instead of this bullshit we're experiencing on NANOG month in month out, jeez.

You got kicked from the one list no one gets kicked of, Full-Disclosure, and came to NANOG to kill all conversation?

On the other hand, he *does* have a valid point. Why *do* we keep seeing queries for the same networks?

On Friday 03 Feb 2006 15:32, Valdis.Kletnieks@vt.edu wrote:

There's also the deeper question: Why do we let the situation persist? Why do we tolerate the continued problems from unreachable companies?

Economics. Some of those unreachable companies (the majority?) are huge networks who your customers care about reaching for whatever reason. Until someone comes up with a way of punishing them, or their customers, without punishing themselves, or their own customers equally..... Sounds like one for pledgebank.... something like "we'll delay every email to every domain listed in rfc-ignorant for 30(?) minutes if 500 other email admins will do the same".... All comes back to Internet death sentence type thinking. Although I think delaying email is perhaps better than null routing the network, since it hurts enough to highlight the issue, but isn't too damaging. Better ideas sought.

There's also the deeper question: Why do we let the situation persist? Why do we tolerate the continued problems from unreachable companies? (And yes, this *is* an operational issue - what did that 4 hours on the

At the end of the day this is never going to stop. It's much easier

Earlier, Valdis scribbled: phone cost your company's bottom line in wasted time?) To a certain extent, it's simple economic logic. At the end of the day, I got my issue sorted and it cost me 4 hours of billable time. It cost the other party 15 minutes of time. Why employ another person full time to deal with queries or man an email desk, to save *me* 3h45min? It makes economic sense for bigger companies not to, well, "care". They aren't going to go away, you're not going to get in the way of the big Google/MS/BigCorp(tm) engine with gripes on your blog, so why bother spending more money on helping *you*? It might sound very black and white, but I can tell you now that a lot of these companies use that as a rationale even without thinking about it so directly. The whole situation is unfortunate. It seems that basic business ethics are going down the drain quicker and quicker. Maybe companies should start programming AliceBots to deal with technical/commonly asked queries via the LiveSupport button on their website. (hey, that sounds like a business idea!) It would be time better spent than phoning someone's automated answering box... imho Regards, Ivan -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Valdis.Kletnieks@vt.edu Sent: 03 February 2006 15:33 To: Ivan Groenewald Cc: 'Gadi Evron'; 'n3td3v'; nanog@merit.edu Subject: Re: Yahoo, Google, Microsoft contact? On Fri, 03 Feb 2006 06:04:57 GMT, Ivan Groenewald said: posting

to Nanog than spending 4 hours on the phone trying to locate someone with a clue in one of these big companies.

All it would take is for http://puck.nether.net to carry either the correct contact info once it's been discovered, so it doesn't have to be rediscovered over and over, or a "Don't bother" entry(*) so people don't have to re-discover that there's no way to get there from here. There's also the deeper question: Why do we let the situation persist? Why do we tolerate the continued problems from unreachable companies? (And yes, this *is* an operational issue - what did that 4 hours on the phone cost your company's bottom line in wasted time?) (*) And yes, I'm *FULLY* aware of at least the top 17 reasons why That Just Won't Work, so don't bother posting a reply unless you've got a particularly interesting number 18 - the *point* is that *IF* the info was listed, it would help solve the problem of wasting time trying to reach these companies.

I'm sorry, but being a larger company requires more resources to support it. Our upstream provider has only 3 to 5 people in their NOC during the day, but they only serve a couple dozen ITCs. A bigger company generates more revenue and accordingly has increased responsibilities. Largish companies benefit from economies of scale (their overnight crew *actually* has calls to take) and will likely have better processes in place to handle things efficiently. What do you think the messages:NOC man-hours ratio is? I would argue that smaller operations provide better service, but it costs them more per message, or whatever metric you want to use. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Christopher L. Morrow Sent: Friday, February 03, 2006 10:37 AM To: Ivan Groenewald Cc: Valdis.Kletnieks@vt.edu; nanog@merit.edu; 'Gadi Evron'; 'n3td3v' Subject: RE: Yahoo, Google, Microsoft contact? On Fri, 3 Feb 2006, Ivan Groenewald wrote:

...

There's also the deeper question: Why do we let the situation persist? Why do we tolerate the continued problems from unreachable companies? (And yes, this *is* an operational issue - what did that 4 hours on the

Earlier, Valdis scribbled: phone cost your company's bottom line in wasted time?)

To a certain extent, it's simple economic logic. At the end of the day, I got my issue sorted and it cost me 4 hours of billable time. It cost the other party 15 minutes of time. Why employ another person full time to deal with queries or man an email desk, to

save

*me* 3h45min? It makes economic sense for bigger companies not to, well, "care". They aren't going to go away, you're not going to get in the way of the big Google/MS/BigCorp(tm) engine with gripes on your blog, so why bother spending more money on helping *you*?

It might sound very black and white, but I can tell you now that a lot of these companies use that as a rationale even without thinking about it so directly.

actually, working for a largish company, I'd say one aspect not recognized is the scale on their side of the problem... abuse@mci|uu|vzb gets (on a bad month) 800k messages, on a 'good' month only 400k ... how many do yahoo/google/msn get? How many do their role accounts get for hostmaster/postmaster/routing/peering ?? Expecting that you can send an email and get a response 'quickly' is just no reasonable unless you expect just an auto-ack from their ticketting system. -Chris

On Fri, 3 Feb 2006, Richard A Steenbergen wrote:

Until someone invents a universally recognized system where you can call and say "Hi I'm CCIE #12345, I'm certified to know what I'm talking about and I have an actual network issue, please transfer me to someone with clue", we're going to continue to see the problem of letting the legit calls through while seperating out the calls from J. Random Crackmonkey

How about INOC-DBA, which is supposed to have a clue threshold you obtained an ASN by some means in order to have a dial-by-asn phone.

On Feb 3, 2006, at 2:57 PM, Aaron Glenn wrote:

On 2/3/06, Sean Donelan <sean@donelan.com> wrote:

...

How about INOC-DBA, which is supposed to have a clue threshold you obtained an ASN by some means in order to have a dial-by-asn phone.

Obtaining an ASN isn't much of a clue threshold.

Knowing what an ASN is would be a bar too high for 90+% of the people who call / e-mail abuse desks. Getting an INOC-DBA phone is a bar far, far, far too high for 99.99% of these people. -- TTFN, patrick

On Feb 3, 2006, at 2:57 PM, Aaron Glenn wrote:

On 2/3/06, Sean Donelan <sean@donelan.com> wrote:

...

How about INOC-DBA, which is supposed to have a clue threshold you obtained an ASN by some means in order to have a dial-by-asn phone.

Obtaining an ASN isn't much of a clue threshold.

If that's actually true, then it's unlikely that any amount of communication is going to save us. For those who are slightly less cynical: http://www.pch.net/inoc-dba/ Tom Vest Research Program Manager Packet Clearing House http://www.pch.net (703) 598-6831

...

How about INOC-DBA, which is supposed to have a clue threshold you obtained an ASN by some means in order to have a dial-by-asn phone.

Obtaining an ASN isn't much of a clue threshold.

However, obtaining an ASN is a volume threshold which is far more important to the people on the receiving end of the communications. --Michael Dillon

On 3-Feb-2006, at 15:59, Richard A Steenbergen wrote:

With all due respect to the INOC-DBA project, which is actually somewhat interesting (from a "I want to play with free IP phones too" perspective if nothing else), it isn't a workable solution to operational contacts yet.

I think you are understating its usefulness, somewhat. Whilst it's an uncontested fact that you can't hope to reach The Right Person by dialling their ASN in all cases, I find it useful for probably around one in five people that I need to call. That's a pretty good strike rate, especially since if you *can* reach someone through INOC-DBA, it's almost always the Right Person, and it almost never involves an IVR.

And then of course there is that whole "using the IP network to contact someone about an IP network issue" thing that doesn't seem terribly well thought out...

You could also argue that trying to contact someone about a major fibre cut using the PSTN is also doomed to failure, at least for some fibre cuts. That's no reason to never try using the PSTN. Not all problems with remote AS Y involve a complete inability to exchange packets with AS Y.

Admittedly I haven't looked at the INOC-DBA stuff in a while, there could have been some massive advancement that I'm not aware of, but I suspect that the situation is still "more work needed". Existing phone systems, call centers, and engineers with cellphones, seems to be a much safer bet right now.

Although the provisioning system seems to be under active development, and sometimes exhibits hiccups, the system as a whole is fine. More people should try it. Joe

To chime with my own experiences, the few times I have used the INOC-DBA system for an Inter-provider issue have been quite successful. The results were much faster and much less frustrating that calling through the 'front door' of the provider's NOC. And it is fair to say that the system only gains usefulness with wider implementation among network providers and appropriate deployment of the phones within the organization. Within Verizon, I deployed the phones with our IP-NOC (yes, we have *many* NOCs, but only 1 handles IP issues), with our IP escalation team (TAC), and on my desk (footnote: my desk recently moved and haven't gotten the inoc-dba phone back up on the new net infrastructure). In light of recent purchases by VZ, if none of the above methods work, just call Chris Morrow. Just kidding Chris! :-) - Wayne

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Christopher L. Morrow Sent: Friday, February 03, 2006 4:31 PM To: Richard A Steenbergen Cc: Sean Donelan; nanog@merit.edu Subject: Re: Anyone heard of INOC-DBA?

On Fri, 3 Feb 2006, Richard A Steenbergen wrote:

...

And then of course there is that whole "using the IP network to contact someone about an IP network issue" thing that doesn't seem terribly well thought out... Admittedly I haven't looked at the INOC-DBA stuff in a while, there could have been some massive advancement that I'm not aware of, but I suspect that the situation is still "more work needed". Existing phone systems, call centers, and engineers with cellphones, seems to be a much safer bet right now.

there is no one solution... to anything except 'life' (solution == death). So, how about looking at it as a tool to use. You might have your provider's $Person_for_Problem in your cell phone, use that if you can. Use their Customer Service number or use their INOC number.... putting down a project that does work because it's not the holy grail isn't productive.

On Sat, 4 Feb 2006, Henry Linneweh wrote:

The only reference I see to this, is this non profit research org www.pch.net/inoc-dba/ and a Nanog reference page to the same thing http://www.nanog.org/mtg-0505/upadhaya.html

that would be it... I'm sure that, aside from the presentation and emails on-list about it, if you have other questions Woody can probably answer them, or other users could...

I believe SBC was one of the pilot users in 1993.

using what protocol set? randy

Something I (unless skipped) didn't see being mentioned in the former threads was: http://www.peeringdb.com, although meant for peerings it does contain quite a large number of direct links/phones to various NOC's and a lot of other very useful information. On Fri, 2006-02-03 at 15:59 -0500, Richard A Steenbergen wrote: [..]

With all due respect to the INOC-DBA project, which is actually somewhat interesting (from a "I want to play with free IP phones too" perspective if nothing else), it isn't a workable solution to operational contacts yet.

The only real issue I see is that it would require a major part of the NOC's to be present to be really effective. Currently for the NOC's that are there this seems to work perfectly well. Maybe the RIR's should keep a "Free VoIP phone with each ASN" special? (Though they would have to raise their membership fees then I guess)

Among other reasons, it seems that the vast majority of the users are just people playing around with it at their desk in the office, never expecting it to ring for anything serious.

I've been at a couple of places where the INOC-DBA phones are at least in grabbing distance, usually literaly next to the normal phone, unless the POTS system was integrated with some VoIP system and the POTS came in over the VoIP phone.

It might be more interesting if people actually set up 1234*NOC extensions, but puck.nether.net seems like a far more effective choice. The INOC-DBA system so far doesn't seem to integrate particularly will with existing NOC phones or systems that are not IP based, and you really have to go out of your way to get it to forward to multiple people like say an engineer on duty.

Just install an asterisk, add one of the POTS cards et tada you have your POTS and VoIP system integrated as one solution. For INOC-DBA you need a VoIP gateway, but the endpoint doesn't need to be VoIP ;) Call forwarding is one of the really nice features of any phone system. Personally I think that VoIP systems are really nice especially from the perspective where one can roam around with the endpoint and/or login using multiple methods and just pick it up wherever one wants Greets, Jeroen (hats off for the PCH folks!)

On Fri, 3 Feb 2006 14:10:26 -0500, "Richard A Steenbergen" [snip]

The heart of this problem, like so many other problems before it, is that most people are dumber than dirt itself.

So ... responsible prociders should only serve customers with some minimum IQ? As said elsewhere in the thread; the responsible thing to do is to scale operations properly. You have to find other ways to deal with people's stipidity than to ignore them completely. //per -- Per Heldal http://heldal.eml.cc/

On 3-Feb-2006, at 15:09, Dave Stewart wrote:

At 02:55 PM 2/3/2006, you wrote:

...

...

The heart of this problem, like so many other problems before it, is that most people are dumber than dirt itself.

So ... responsible prociders should only serve customers with some minimum IQ?

One can wish

I've often dreamed of a nested series of IVR menus which ask people networking/technology questions instead of simply leaving them on- hold, so that people who demonstrate clue can be shuffled towards the front of the queue, and those who are lacking can fester on-hold indefinitely. Someone should release an extensions.conf fragment with some corresponding WAV files. The world would thank them. Joe

On Fri, Feb 03, 2006 at 08:55:04PM +0100, Per Heldal wrote:

On Fri, 3 Feb 2006 14:10:26 -0500, "Richard A Steenbergen" [snip]

...

The heart of this problem, like so many other problems before it, is that most people are dumber than dirt itself.

So ... responsible prociders should only serve customers with some minimum IQ?

As said elsewhere in the thread; the responsible thing to do is to scale operations properly. You have to find other ways to deal with people's stipidity than to ignore them completely.

No one gets ignored, they just get met with an army of equally stupid minimum wage phone drones who read the "have you tried rebooting your router" script, and the two sides slug it out in a no holds bared battle of the braindead until one side gives up. Usually though, the script is smarter than the caller, or they wouldn't be using them. This is why people with actual issues don't get noticed as quickly as they should among the sea of false issues. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Fri, 03 Feb 2006 12:42:04 -0500 Martin Hannigan <hannigan@renesys.com> wrote:

I'd like to see evidence that there is a problem. For example, don't see why these worm lists couldn't have just gone to the abuse address.

Of course that's the right answer. IN THEORY. The practice is rather different, and that's WHY the need for some direct contact exists. I followed through with two large UK ISPs, who had both had the list of worm IPs sent to their official abuse address. In neither case had the mail been read or passed on. A copy to their security specialists was appreciated, and resulted in much hurried activity. No, I'm not going to identify who they were; there probably would have been many more ISPs in that position if I'd looked further.

the customer is shifting the cost of support off of their own provider and on to the rest of us which is inherently not fair.

s/customer/provider/ - if the provider wasn't doing that, the customer quite likely WOULD have gone directly to them.

I think it's ok to post these things to NANOG as long as there's more information than just who they are looking for. If it's too private to tell all of us, then don't use our list as a directory service.

True. Nevertheless there is a need for some directory system, so that appropriate people can contact key security etc people in other network entities, without giving NANOG a full-disclosure on the situation ... -- Richard Cox

<Valdis.Kletnieks@vt.edu> wrote: [...]

On the other hand, he *does* have a valid point. Why *do* we keep seeing queries for the same networks?

Well, at least it does serve as a warning to anybody who might have been considering using their services. -- Obscenity can be found in every book except the telephone directory. - George Bernard Shaw

On Fri, 3 Feb 2006, Per Heldal wrote:

On Thu, 02 Feb 2006 22:39:59 -0500, Valdis.Kletnieks@vt.edu said:

...

On the other hand, he *does* have a valid point. Why *do* we keep seeing queries for the same networks?

Because no-one has the balls to punish them in a way that really hurt their bottom line. It's mostly about companies where ops are not allowed to work on problems not related to paying customers. I bet their bean-counters would create accounts for misc-problems in record time if the top 10 transit networks would null-route their AS'es for a while.

uhm, where's the incentive in that for a 'tier-1' provider? most of the folks mentioned are customers or peers, so there exists a contact path already...