Why should they talk to you? You're not a paying customer.. I get very upset when customers of customers start phoning us up.. I think your problem is with your immediate upstream and if their own supplier relations and escalation procedures arent up to scratch then you want to change to another which is but I dont think you can ever expect UUNET to talk directly to you unless you buy direct Steve (eek, did i defend uunet?! i must be losing it!) On Fri, 29 Mar 2002, Forrest W. Christian wrote:

I realize the original problem has been fixed, but I'd like to find out if someone does have a contact at UUnet which will help with these types of issues. In particular, when I've been having a problem reaching UUNet-connected sites and needed to contact the uunet noc, I get almost the same response as Mark did initially, i.e.:

On Thu, 28 Mar 2002, Mark E. Mallett wrote:

...

...

I have contacted UUnet and have been told to take it up with my upstreams 'cuz they won't deal directly with me. They also said to have a nice day.

I realize I don't have a direct relationship with UUnet. But trying to get my upstream to talk to their upstream to talk to UUnet just to get someone at uunet to do a traceroute or tell me what is showing up in their (uunet's) BGP tables is just plain rediculous.

Does anyone have better contact information than "call their unhelpful noc"?

- Forrest W. Christian (forrestc@imach.com) AC7DE ---------------------------------------------------------------------- The Innovation Machine Ltd. P.O. Box 5749 http://www.imach.com/ Helena, MT 59604 Home of PacketFlux Technogies and BackupDNS.com (406)-442-6648 ---------------------------------------------------------------------- Protect your personal freedoms - visit http://www.lp.org/

I've obviously caused a stir. Before I proceed, let me say I'm going to continue mentioning UU.net as I've had experience there... The responses to this list indicate this is a more widespread problem, so please don't take this as necessarily badmouthing uu.net. Let me first say EXACTLY what I was looking for. I'm multihomed. All I've wanted out of uu.net each time I've called is a traceroute and/or BGP output to determine which path my packets were heading back towards me on so *I* could get the problem fixed. I.E. to determine where the loss was really occuring and/or who was mis-announcing a prefix. In every case where I've tried to contact uu.net it's been obvious that as soon as traffic reaches their AS, everything goes to pot. Without being able to take a peek inside their network (via a traceroute or sh ip bgp) It's almost impossible to tell where the problem lies, since the problem is obviously with traffic getting back to my network. I agree with batz: On Fri, 29 Mar 2002, batz wrote:

Because their network transits _most_ internet traffic and as a courtesy, they should provide some bare level of diagnostic services to the rest of the network.

I can't think of a case where I've called the uu.net noc where I wanted more information than could have been queried through a standard looking glass (I.E. traceroute and BGP information). In fact, if uu.net provided a looking glass we probably wouldn't be having this discussion. Without rambling much further I'll add this: Yes, I realize there are scaling issues. Yes, I do want to call my upstream to get it fixed. No, I don't expect uu.net to own the problem (unless of course it IS their problem). BUT I can't tell which of my upstreams is having the problem in order to call them without a BGP or traceroute from the provider we're having problems reaching. - Forrest W. Christian (forrestc@imach.com) AC7DE ---------------------------------------------------------------------- The Innovation Machine Ltd. P.O. Box 5749 http://www.imach.com/ Helena, MT 59604 Home of PacketFlux Technogies and BackupDNS.com (406)-442-6648 ---------------------------------------------------------------------- Protect your personal freedoms - visit http://www.lp.org/

On Fri, Mar 29, 2002 at 12:07:06PM +0000, Stephen J. Wilcox wrote:

Why should they talk to you? You're not a paying customer..

I get very upset when customers of customers start phoning us up..

This really bothers me. So, if you are starting to have a major outage because of a configuration change, or a circuit goes down, or whatever else might happen, and the first person who contacts you is not a customer, you are going to ignore it, especially if your network tools haven't picked it up yet? In another lifetime ago I was working at a network where Gamer Tickets (people playing Everquest and those types of games) would sometimes see the problem before our network tools picked it up. The Gamers were not direct customers, but we worked on their problems, because a portion of the time there actually was a problem with the network that should be fixed immediately before our Big Customers Who Paid The Company Lots of Money started to call and want SLA or threaten to (and sometimes did) go to another provider. You can also think of it in the respect of potentially making more money in the long run. If you provide some service to a non-customer and their upstream doesn't provide any, there is a good chance these non-customers turn into customers by purchasing service directly from you instead. Rachel -- When one door closes, another opens; but we often look so long and so regretfully upon the closed door that we do not see the one which has opened for us. - Alexander Graham Bell

On Sun, 31 Mar 2002 12:55:42 +0100, "Stephen J. Wilcox" said:

1. Customers are always telling us its a problem at our end and it never is 2. If we have any outage its always picked up by our network tools

You're in the wrong line of work - you need to bottle it and sell it. Unfortunately, I'm not sure if it should be marketed as Divine Right or just as fertilizer... ;)

On Fri, 29 Mar 2002, Anne Marcel Roorda wrote:

Having a support model in which anyone can call any NOC about a problem they're having does not scale very well.

How about a model where any large (multiple OC12s) CUSTOMER can call a NOC about a problem they're having??? -- Yours, J.A. Terranson sysadmin@mfn.org If Governments really want us to behave like civilized human beings, they should give serious consideration towards setting a better example: Ruling by force, rather than consensus; the unrestrained application of unjust laws (which the victim-populations were never allowed input on in the first place); the State policy of justice only for the rich and elected; the intentional abuse and occassionally destruction of entire populations merely to distract an already apathetic and numb electorate... This type of demogoguery must surely wipe out the fascist United States as surely as it wiped out the fascist Union of Soviet Socialist Republics. The views expressed here are mine, and NOT those of my employers, associates, or others. Besides, if it *were* the opinion of all of those people, I doubt there would be a problem to bitch about in the first place... --------------------------------------------------------------------

In message <Pine.BSF.4.21.0203290621020.25238-100000@greeves.mfn.org>, measl@mf n.org writes:

...

On Fri, 29 Mar 2002, Anne Marcel Roorda wrote:

...

Having a support model in which anyone can call any NOC about a problem they're having does not scale very well.

How about a model where any large (multiple OC12s) CUSTOMER can call a NOC about a problem they're having???

So you've bought some very expensive OC12s which presumably are carrying a large amount of highly valued business critical data. You report a fault and your local provider indicates its a fault with their supplier, they consequently fail to contact their supplier and do not resolve the matter anywhere near quick enough. 1. Who chose this supplier? Did you not check them out first and ensure they were good quality for customer support. 2. Sue them for breach of service level agreement. 3. Get your money back and switch to another carrier. See.. you didnt need their upstream NOC after all! And you could have saved the hassle if you bought quality services to begin with! Steve

On Fri, 29 Mar 2002, Sabri Berisha wrote:

On Fri, 29 Mar 2002, Anne Marcel Roorda wrote:

...

Having a support model in which anyone can call any NOC about a problem they're having does not scale very well.

What would work better/faster?

my-noc -> b0rken-noc

or

my-noc -> my-upstream-noc -> b0rken-noc-upstream-noc -> b0rken-noc

Work better for who? For you? Sure. For a any provider that needs to provide quality services to its customers and follow processes to do so, not a chance. The Big Picture is key here. andy -- PGP Key Available at http://www.tigerteam.net/andy/pgp

On Fri, 29 Mar 2002, Leo Bicknell wrote:

Note that in both cases, b0rken-noc takes a single call, so their load is unchanged. The second case adds a call to both my-upstream-noc, and b0rken-noc-upstream-noc.

It would seem going direct would put a lower load on NOC's in general, which presumably would let them spend more time on problems and provide better service.

Where is the limit though? Once I open things up to non customers, and let any random person call me, without any sort of filters or controls, what keeps my best guys from troubleshooting someone's mistyped SMTP server in their mail client? Processes are put in place to scale and when they are disregarded, things generally end up worse in the long run. andy -- PGP Key Available at http://www.tigerteam.net/andy/pgp

On Fri, 29 Mar 2002, Andy Walden wrote:

On Fri, 29 Mar 2002, Leo Bicknell wrote:

...

Note that in both cases, b0rken-noc takes a single call, so their load is unchanged. The second case adds a call to both my-upstream-noc, and b0rken-noc-upstream-noc.

It would seem going direct would put a lower load on NOC's in general, which presumably would let them spend more time on problems and provide better service.

Where is the limit though? Once I open things up to non customers, and let any random person call me, without any sort of filters or controls, what keeps my best guys from troubleshooting someone's mistyped SMTP server in their mail client? Processes are put in place to scale and when they are disregarded, things generally end up worse in the long run.

We are not talking about SMTP here, but about someone bogusly announcing routes. I agree with you that your noc is not helpdesk for anyone but if your noc announces bogus routes (which should originate from my AS) I think I have every right to contact your noc and try to solve things. Afterall, it is you doing somewthing wrong which affects my network. To answer Randy's remark about scaling: this scales very well; the number of AS-es are limited. On the other side, I know how annoying it is if other people's customers call you about their b0rked up Windows RAS configuration. That should not happen, I am talking about professional noc-to-noc contact here which imho should not be to bureaucratic. -- Sabri Berisha "I route, therefore you are" ~ my own opinions etc ~ Join Megabit LAN in open air! http://www.megabit.nl

On Fri, 29 Mar 2002 11:43:26 EST, Leo Bicknell said:

...

...

my-noc -> my-upstream-noc -> b0rken-noc-upstream-noc -> b0rken-noc

It would seem going direct would put a lower load on NOC's in general, which presumably would let them spend more time on problems and provide better service.

The difference being that if the call comes from b0rken-noc-upstream-noc, the guys at b0rken-noc have at least a snowball's chance of knowing the person calling and whether they have any kloo. If our NOC calls one of our upstreams and says "hey, ASnnn is sending you bogons that you're forwarding to us", they tend to listen, and call the guys at ASnnn and tell them to cut it out. (Yes Leo, you know most of our NOC monkeys, so you know what the chances are they're right about something ;) On the other hand, if we call ASnnn directly, they have no way of knowing if we're us, or if we're some bunch that thinks it makes sense to hang an AS off a residential ADSL line... Now, if we had a PGP-ish "web of clue", it would be different.... -- Valdis Kletnieks Computer Systems Senior Engineer Virginia Tech

What would work better/faster?

my-noc -> b0rken-noc

or

my-noc -> my-upstream-noc -> b0rken-noc-upstream-noc -> b0rken-noc

?

OK, rant time (blame the easter long weekend... a 4 day weekend down here... and associated excessive alcohol)... General comment: the below isn't meant to reflect badly on any of our past or present providers or peers... and in the most part problems mentioned relate to previous suppliers so please don't try to guess who they could be about :-) Becomes much more relevant when you're not in America. Often a company in, say, Singapore or New Zealand may manage an Australian company's connection to the US internet. And then said Australian company may have a problem connecting through the US internet to, say, China via Japan (which the company I work at doesn't do anymore - one of our providers now has connectivity via Singapore to China which is much better, but that still isn't the case for many in Australia). You want to think how many NOCs and language barriers there can be in that path? And peering relationships, timezone changes (harder to get good engineers sometimes, and 24 hour NOCs aren't common in many countries), etc? Or, we can directly contact a NOC in southern China and get resolution as well as having a very satisfied customer because all his other upstreams attempted and failed the NOC to upstream NOC through a massive number of NOCs who couldn't resolve the issue. The problem is when you take this approach you have to be very sure of which AS is causing the real problem (and/or what the real problem is - calling your upstream's upstream and telling them to tune their tx-ring-limits is another example, where your direct upstream at the time may not have heard of such a thing to know to relay the fault in a way the remote NOC would work out what the problem was and how to fix it. of course the better thing to tell the provider in question should have been "don't try and put that many OC3 cards in a 7206!"). Admittedly the escalation in the southern China case (which wasn't our standard problem with providers in China turning on routers which make classful assumptions, and us having some 61.* IP space) was: customer's customers -> customer's NOC -> our NOC -> problem site's upstream's NOC (who liased with problem site, and fortunately spoke english - the problem site didn't, but if it had been an issue, our customer's NOC had offered to translate) but that cut out a _lot_ of NOCs. To me there's some maximum number of NOCs to be involved in a problem to coordinate well, and it is around 4 (end ISP NOC, their upstream NOC to confirm the problem, problem site's upstream NOC to enforce fixing of problem, problem site's actual NOC), which then becomes 3 in the case where the problem network is someone like sprint, at&t or uunet who we wouldn't consider to actually have an "upstream" (and for the record in the cases I've had to, I haven't had a problem dealing those three directly even though we're not a customer; maybe I've just been lucky). An Australian company who is being directly affected by a problem may keep good staff on until the right time to contact a US or other international NOC directly during their working hours and get decent staff, rather waiting on all the various NOCs to miscommunicate the problem across various hops. Another problem is "follow the clock" NOCs and trying to call at the right time to get a US operator, since operators in the UK or Singapore in a certain ISP had pretty much no access to their routers and could do nothing more than email the US staff and hope to get some resolution 12 hours later... the country that took the call owned the problem, but had to pass it off internally, then wait till that country was active again to call the customer back, repeat that a few times to convince them of the actual fault. Glad I don't deal with that particular company anymore :-) I haven't had a problem from large US providers in providing me a trouble ticket even though we're far removed from being a customer. And we've found the "trouble" has been things as lame as a certain large US provider putting a /32 static backhole in one of their routers, and following the "correct" escalation path NOC to NOC in one case (since it was minor and worked around) did nothing for a week, a direct email (in that case, calls are for more urgent issues :-)) to their US NOC and the problem was fixed within an hour. The only group in the US I've found hard to deal with in any way internet operationally related was a bad experience and waste of international calls to NetSol/VeriSign, they had no intent to deal with a _customer_ in a timely manner over an urgent change (domain change for a company who had just gone into liquidation and were about to lose the routability of their IP space in 48 hours, and NetSol's systems weren't accepting IP changes for the nameservers due to what turned out to be design problems in their database application - the permission to change info update in some cases needs 24+ hours to propogate internally before you can make changes under the new permissions... ugly). David.

On Fri, 29 Mar 2002, Mark E. Mallett wrote:

BTW as I mentioned when I contacted Genuity, they advised me to contact UUnet directly. So by inference at least one large carrier (Genuity) seems to feel that contacting them directly is appropriate.

I believe this is the problem. Providers can't expected to have it both ways. If you are a customer of provider A, and the problem is inside providers B network what is the appropriate method to get provider B to fix the problem? 1. Call provider A. Open a trouble ticket. Provider A forwards the ticket through the chain of providers to Provider B. Provider B accepts the trouble ticket. B find the problem in their network and fixes it, closing the trouble ticket back to A. 2. Call provider A. Provider A says its not a problem with A's network and closes the ticket. A tells customer, call Provider B. User looks up Provider B's contact information. User calls Provider B and is told, we don't take calls from non-customers, call Provider A. Rinse and Repeat. 3. Call lawyer. Sue Provider A and B for tortious interference with the user's peaceful enjoyment of the Internet by negligently and/or fraudently propagating false routing information and failing to correct the problem after being notified by the user. I think method 1 is the best way to handle the situation. Unfortunately, most of the time method 2 is what happens. Eventually, someone will try method 3, and I don't want to be around when that happens.

On Fri, 29 Mar 2002, Sean Donelan wrote:

If you are a customer of provider A, and the problem is inside providers B network what is the appropriate method to get provider B to fix the problem?

I think the usual method is to find someone who IS a customer of provider B's network -- ie whoever it is you can't access -- and have them complain. ========================================================== Chris Candreva -- chris@westnet.com -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/

...

Having a support model in which anyone can call any NOC about a problem they're having does not scale very well.

I felt justified in calling UUnet. I know the conversation had morphed by the time you made the above comment. However in my case UUnet was propagating an announcement that was stepping on one of ours; the owner of the netblock was there to say that he did not want that announcement being made; the UUnet customer making the announcement (who I would rather have dealt with) was apparently operating without a crew. Here was a conversation between directly affected parties. It came down to who was bothering who: was it UUnet bothering me by announcing my route, or was it me bothering them by asking them to stop?

Let me begin by letting you know that my comments were not UUnet specific. Getting the person announcing this block to open a ticket with the NOC and including your contact details, as well as a clear description of the prolem would have been my first suggestion. It saves you having to convince the NOC that their customer should not be announcing this route, and that they should stop it. Instead it turns into an issue where a customer is unable to change his configuration reliably and needs help in doing so. A suitable sollution would have been to either talk the customer through it, change the filters, or both.

The model of "I won't talk to anybody who isn't my customer" is probably almost always right, but it does not work for every single situation. With that stand, you wouldn't have an abuse@ contact. Sometimes your actions directly affect somebody and you should be willing to deal with the consequences of that.

The comparison with an abuse department is not realistic. The primary goal of those departments is drasticly different.

While their initial reaction in my case was "I can't talk to you," they did indeed reconsider and help out. Thanks again. It happened pretty much at the instant I asked for help here, which is the usual sort of kharma..

Some things should not be to easy. Getting people to filter routes without a clear understanding of why, and apropriate checks is one of those things.

BTW as I mentioned when I contacted Genuity, they advised me to contact UUnet directly. So by inference at least one large carrier (Genuity) seems to feel that contacting them directly is appropriate.

That is unfortunate. - marcel

On Fri, 29 Mar 2002, Anne Marcel Roorda wrote:

Getting the person announcing this block to open a ticket with the NOC and including your contact details, as well as a clear description of the prolem would have been my first suggestion.

From the sounds of it there was a problem getting hold of the person announcing the netblock to uunet.

Not to mention how are you going to contact these people anyway? Remember you are going from Noc to Noc, so you shouldn't be ringing them directly. Seriously if somebody on the other side of the world started advertising our networks (yes it has happened) I would contact them directly. Real life example: We currently advertise (under our AS) a /24 out of a larger block belonging to a German University and route it to some customer in Australia. It seems that when a professor moved to Australia he was allowed to take the network with him (we queried this a lot when it first showed up). What happens if/when the University wants their network back and can't contact the professor? Should they bounce their request through half a dozen totally unrelated providers or just email/phone us? The University know we are the problem and our contact details are easy to find. Why should they go through half a dozen companies that have nothing to do with what is going on just to contact us? Lets pretend for some reason that I start advertising an important network of yours (half your mail servers or your biggest customer). Are you really going to tell your boss that you are not going to ring me directly but are instead going to go through 5 disinterested 3rd parties to get to me? Is you boss going to accept this may take a few hours/days or is he instead going to just call us directly himself and then fire you? -- Simon Lyall. | Newsmaster | Work: simon.lyall@ihug.co.nz Senior Network/System Admin | Postmaster | Home: simon@darkmere.gen.nz ihug, Auckland, NZ | Asst Doorman | Web: http://www.darkmere.gen.nz