Wanted: Clueful Individual @ TeleGlobe.net
OK, I've tried the normal methods... phone, email... no response... Sample snip from phone call: Me: I wanted to report some network issues on your network. You're showing some pretty major packet loss between <ip> and <ip> on your network. TG: What's your circuit ID? Me: I don't have a Teleglobe ckt, I'm just some average schmoe who has noticed that you're dropping packets on the ground at a VERY alarming rate. TG: I can't enter a trouble ticket for you unless you're a teleglobe customer. Me: Don't enter it for me, enter it for yourself, its your network that's got the problem. TG: I can't do that. Me: So even though your network is sucking wind right now and hemorrhaging packets, you won't start up a trouble ticket simply because I'm not PAYING you to take a trouble ticket? TG: Ummmm, correct. Me: If you were a Teleglobe customer, and knew that someone had reported this problem but you ignored the request because the person telling you about it wasn't a customer, would you be pissed? TG: Ummmm, probably. Me: So put your fingers to the keys, and do the RIGHT thing, and open up a ticket on it. TG: I can't. Me: OK, let me speak to your supervisor. TG: I can't. I'm the supervisor tonight. Me: Is there ANYONE more clueful above you in the chain of command that I could hope to speak to in order to try and help you guys from putting yourselves out of business? TG: Ummmm, no not right now. Me: When would there be more people in. TG: Monday. Me: Bah, by then, you'll be out of business. Cheers. FYI, their problem is here: Hostname %Loss Rcv Snt Last Best Avg Worst 5. atm8-0.core1.paix.teleglobe.net 0% 32 32 78 78 141 480 6. if-4-0-0.bb5.NewYork.Teleglobe.net 55% 14 32 172 161 195 437
On Sat, 15 Jul 2000, Derek J. Balling <dredd@megacity.org> wrote:
Sample snip from phone call:
This mirrors my experience with the AT&T NOC on Friday after seeing latency between GlobalCenter and AT&T. The GlobalCenter NOC had no problem opening a ticket without caring whether I was a customer or not. They spent a few hours on it and said it appeared to be an over-loaded AT&T router. Not entirely sure of that - and if it was true, wanting to see what AT&T was doing about it - I called AT&T, was transferred 3 times to reach the IP folks, and was promptly stonewalled. The level 1 tech didn't know latency from Adam but wasn't willing to find someone who did. He said he couldn't open a ticket for a non-customer and we argued for 10 minutes about what to do next - hang up or talk to the next guy up the chain. I convinced him that it was an actual issue and that he should indeed tell his boss - or maybe he just got tired of talking. Anyhow, his boss came on the line and said the same thing, emphasizing that they couldn't open tickets for non-customers. Even informing him that AT&T customers were calling us to complain and that AT&T users were affected didn't help. This guy claimed to be the engineering manager and said there was nobody above him to go to. We talked for a while - 10-20 minutes - and apparantly he finally got tired too, since he took my name/number. A couple minutes later I got a call from a third guy there, who said that any issues were between AT&T and GlobalCenter and that I had no business calling. I asked if there was an open ticket or even someone addressing it and he had no idea. He didn't seem to know any more about networking than the first guy. The most discouraging part was that the last two people understood that it was a service-affecting problem and yet hadn't been given any procedures to solve it. It felt like I was asking a secretary - they all stated that they understood it was a problem but couldn't do a damn thing. I ended up sending a note to noc@att.net telling them the problem, my support story, and noting that their NOC folks couldn't do any of these things (all of which I asked to do): - open a trouble ticket for a non-customer - transfer me to someone who could - transfer me to the IP engineers - verify/deny that the problem was within AT&T (a traceroute, perhaps?) - call GlobalCenter and work on the ticket I had open with them - provide me with an escalation or complaint path, either for the original problem or to complain about the NOC's useless procedures .. any of which would have been more useful than hanging up and sending an email. I still haven't received a reply or even an automated ticket #. Cheers, Troy
You have to remeber that most major providers don't have time to mess with non customers. If you want to get any kind of resolution you need to send email to there noc and open a ticket with your provider, because that is who AT&T or TeleGlobe is going to work with. Also by opening a ticket with your provider you let them clear the return path through there network(which is almost always diffrent that your path there), and you also don't bug people on nanog with mail like this. Chris ----- Original Message ----- From: "Troy Davis" <troy@nack.net> To: "Derek J. Balling" <dredd@megacity.org> Cc: <nanog@nanog.org> Sent: Sunday, July 16, 2000 3:43 AM Subject: Re: Wanted: Clueful Individual @ TeleGlobe.net
On Sat, 15 Jul 2000, Derek J. Balling <dredd@megacity.org> wrote:
Sample snip from phone call:
This mirrors my experience with the AT&T NOC on Friday after seeing
between GlobalCenter and AT&T. The GlobalCenter NOC had no problem opening a ticket without caring whether I was a customer or not. They spent a few hours on it and said it appeared to be an over-loaded AT&T router.
Not entirely sure of that - and if it was true, wanting to see what AT&T was doing about it - I called AT&T, was transferred 3 times to reach the IP folks, and was promptly stonewalled.
The level 1 tech didn't know latency from Adam but wasn't willing to find someone who did. He said he couldn't open a ticket for a non-customer and we argued for 10 minutes about what to do next - hang up or talk to
next guy up the chain. I convinced him that it was an actual issue and
he should indeed tell his boss - or maybe he just got tired of talking.
Anyhow, his boss came on the line and said the same thing, emphasizing
they couldn't open tickets for non-customers. Even informing him that AT&T customers were calling us to complain and that AT&T users were affected didn't help. This guy claimed to be the engineering manager and said
was nobody above him to go to. We talked for a while - 10-20 minutes - and apparantly he finally got tired too, since he took my name/number.
A couple minutes later I got a call from a third guy there, who said that any issues were between AT&T and GlobalCenter and that I had no business calling. I asked if there was an open ticket or even someone addressing it and he had no idea. He didn't seem to know any more about networking than
latency the that that there the
first guy.
The most discouraging part was that the last two people understood that it was a service-affecting problem and yet hadn't been given any procedures to solve it. It felt like I was asking a secretary - they all stated that they understood it was a problem but couldn't do a damn thing.
I ended up sending a note to noc@att.net telling them the problem, my support story, and noting that their NOC folks couldn't do any of these things (all of which I asked to do): - open a trouble ticket for a non-customer - transfer me to someone who could - transfer me to the IP engineers - verify/deny that the problem was within AT&T (a traceroute, perhaps?) - call GlobalCenter and work on the ticket I had open with them - provide me with an escalation or complaint path, either for the original problem or to complain about the NOC's useless procedures
.. any of which would have been more useful than hanging up and sending an email. I still haven't received a reply or even an automated ticket #.
Cheers,
Troy
At 11:32 AM 7/17/00 -0400, Chris wrote:
You have to remeber that most major providers don't have time to mess with non customers. If you want to get any kind of resolution you need to send email to there noc and open a ticket with your provider, because that is who AT&T or TeleGlobe is going to work with. Also by opening a ticket with your provider you let them clear the return path through there network(which is almost always diffrent that your path there), and you also don't bug people on nanog with mail like this.
In other words, "most major providers" do not have time to fix their own network? If someone called my NOC with a real problem on my network, customer or not, my NOC *will* fix that problem, or that NOC monkey will be looking for another job. Problems on your network are your problem, whether a customer reports them or some random person on the street. You should be thanking them for backing up their (failed) internal monitoring system, not telling them to bugger off. Also, suggesting someone get their provider to open a ticket with the network in question is a bit silly. This is the North American NETWORK PROVIDERS' Group - most of the people who post here *ARE* a provider. (And are very well aware of asymmetric routing on the Internet.) Besides, this does not always help. As Sean pointed out, some networks will not open tickets for peers (even though that is in every peering agreement I have ever seen).
Chris
TTFN, patrick
Before anyone else flames me: At 12:02 PM 7/17/00 -0400, Patrick W. Gilmore wrote:
This is the North American NETWORK PROVIDERS' Group ^^^^^^^^^ s/PROVIDERS/OPERATORS
Heh, maybe we should start "NANPG". :p Hey, no one said I was bright. (Well, my mother did once, but everyone laughed at her, so she stopped. :) TTFN, patrick
----- Original Message ----- From: "Patrick W. Gilmore" <patrick@ianai.net> To: <nanog@merit.edu> Sent: Monday, July 17, 2000 12:02 PM Subject: Re: Wanted: Clueful Individual @ TeleGlobe.net
At 11:32 AM 7/17/00 -0400, Chris wrote:
You have to remeber that most major providers don't have time to mess
non customers. If you want to get any kind of resolution you need to send email to there noc and open a ticket with your provider, because that is who AT&T or TeleGlobe is going to work with. Also by opening a ticket with your provider you let them clear the return path through there network(which is almost always diffrent that your path there), and you also don't bug
with people
on nanog with mail like this.
In other words, "most major providers" do not have time to fix their own network?
No in other words most people need to get a clue, because(having worked in the noc of a large provider) most of the time there is no problem or it is a problem in the return path(and there providers network) and not even on the provider they are callings network.
If someone called my NOC with a real problem on my network, customer or not, my NOC *will* fix that problem, or that NOC monkey will be looking
for
another job. Problems on your network are your problem, whether a customer reports them or some random person on the street. You should be thanking them for backing up their (failed) internal monitoring system, not telling them to bugger off.
I am willing to bet you don't have over 5,000 customers like most BACKBONE providers do.
Also, suggesting someone get their provider to open a ticket with the network in question is a bit silly. This is the North American NETWORK PROVIDERS' Group - most of the people who post here *ARE* a provider. (And are very well aware of asymmetric routing on the Internet.)
Also lets face it most of the people on this list are BUYING transit from a provider or providers; That makes them customers NOT a backbone provider. If you notice you don't see Sprint or UUnet, or C&W posting here looking for a AT&T router person. Why? When I worked at the noc of a large provider we got and worked tickets all the time that were not on our network. And I have never had any problems with Sprint like Sean was talking about. All I had to do is tell them who I was with and they helped me. (w I didn't work for sprint).
Besides, this does not always help. As Sean pointed out, some networks will not open tickets for peers (even though that is in every peering agreement I have ever seen).
That doesn't mean they will not work with a peer.
Chris
TTFN, patrick
Here is a hueristic solution to nanog noise and the '3rd party trouble ticket problem' (only half tongue in cheek). Suggest people register theif PGP public keys (not email addresses) as 'Clue-held-by:' under the comments section of an AS-MACRO (remember kids, these can be empty, and the namespace for them is almost infinite, so no resources wasted, and you can have as many comment lines as you like). Only allow posting to NANOG, and only allow third parties to raise trouble tickets, by PGP signed message, where the AS-Macro name to be looked up is in an appropriate X-Header in their mail message. This would mean: a) All NANOG messages and trouble tickets would de facto be PGP signed b) Generators of the above content would have to i) Know how to register/alter an AS macro ii) Know how to set headers in their mail iii) Know how to use PGP I believe the above requirements might not be met by many of the lower content posters to NANOG, and certainly by many people who call NOCs, so this would reduce noise considerably... [ And before someone has a go at me for being elitist, sure, reading should be open, so people can learn ] -- Alex Bligh VP Core Network, Concentric Network Corporation (formerly GX Networks, Xara Networks)
On Mon, 17 Jul 2000 18:06:25 BST, Alex Bligh said:
b) Generators of the above content would have to
i) Know how to register/alter an AS macro ii) Know how to set headers in their mail iii) Know how to use PGP
I believe the above requirements might not be met by many of the lower content posters
Are you sure that all three requirements are met by all the *high* content posters on this list? Consider this distinction: 1) "Know how to do so". 2) "Be willing to do so, even though the official maintainance of said records is the job of somebody else in the organization, who may not be a NANOG subscriber, and may not want to register a macro just to allow posting to a mailing list, because they know full well that any good mailing list processor software has other ways to deal with loons, and such tomfoolery shouldn't be required." -- Valdis Kletnieks Operating Systems Analyst Virginia Tech
tempting. tempting. randy
On Sun, 16 Jul 2000, Troy Davis wrote:
This mirrors my experience with the AT&T NOC on Friday after seeing latency between GlobalCenter and AT&T. The GlobalCenter NOC had no problem opening a ticket without caring whether I was a customer or not. They spent a few hours on it and said it appeared to be an over-loaded AT&T router.
Not entirely sure of that - and if it was true, wanting to see what AT&T was doing about it - I called AT&T, was transferred 3 times to reach the IP folks, and was promptly stonewalled.
From what I have seen, the best thing for anyone is to open a ticket with your upstream provider. They are the one that you pay to get to the 'internet'.
Most large providers know where there problems are and are working on them.. They know what pops are hot and which ones are cold. If GC is sending traffic to a hot pop of ATTs, maybe they could send the traffic to another POP. But that is something your upstream should work on. Christian
participants (8)
-
Alex Bligh -
Chris -
Christian Nielsen -
Derek J. Balling -
Patrick W. Gilmore -
Randy Bush -
Troy Davis -
Valdis.Kletnieks＠vt.edu