- NANOG - lists.nanog.org

RE: NOC communications
by Sean Donelan 16 Jul '98

16 Jul '98

>How about a major oversimplification of a fix. Closed mailing list. One >primary contact for each company is designated and allowed to join the list. >Every 2 weeks (maybe a month?) an email is autogenerated that must be ACK'd >or that companies NOC entry is considered stale. The person on the mailing >list is responsible for ensuring contact info is updated. Uhm, the usual problem is the contact person leaves the company. The exit interview rarely includes updating the contact information. Everyone else will know the information is out-dated, assuming you ban people from setting up procmail to auto-ack the message. But how do you get the replacement's contact information if the replacement doesn't know about the list, database, server, etc and you don't know who the replacement is? In any case, I'd be happy to participate in such a list. Although in some sense you are preaching to the converted. If you would like a sample message, how about something similar to what a previous government contractor used to send every six months or so when they managed the NIC database. Perhaps change the "do not respond," to "respond with 'ok 123456'" to make the procmail folks work a little harder writing their auto-ack scripts. >Date: Tue, 25 Sep 90 01:09:33 PDT >From: HOSTMASTER(a)NIC.DDN.MIL >Subject: Network Administrator Update >To: sean(a)DRANET.DRA.COM > >Dear Network Coordinator, > > Here is the current information pertaining to your network. > Please review it and return any corrections to > HOSTMASTER(a)NIC.DDN.MIL. > > ** If no changes are needed, you do not need to respond to this > message. ** > > The file > > NETINFO:NETWORK-CONTACTS.TXT > > on the NIC.DDN.MIL machine (network address 192.67.67.20) contains > information regarding network contacts for every IP network registered > with the DDN Network Information Center. This file may be FTPed by > invoking FTP at your local host and connecting to NIC.DDN.MIL, using > USERID = 'anonymous', PASSWORD = 'guest'. > > Please allow 8 working days for changes to be processed. > > Thanks, > > Hostmaster Staff > DDN Network Information Center > >----------- > > >NAME of responsible organization: Data Research Assoicates, Inc. > > > Network Name: DRA-STL > Network Number: 192.65.218.0 > > Coordinator: > Donelan, Sean M. (SMD1) sean(a)DRANET.DRA.COM > (314) 432-1100 > > I'm a firm believer in keeping things simple. But I also think we have to understand the nature of institutional entropy. Any system that relies on the active, continuing participation of a particular individual, no matter how well intentioned and no matter how minor, will run off the same cliff. The system has to be inserted directly into the institutional gut. Where the institution goes, it goes. And it needs to be more painful for the institution to remove it, than to leave it in. ARIN meets the criteria of gut wrenching pain. But I wonder how often and how long they can really apply sanctions against a network provider before it becomes easier for the network provider to take some alternative action. So they give ARIN a phone number, get their addresses and disconnect the phone line a week later. In six months or a year, when they need more addresses they plug the phone back in, and call ARIN. I really wish I was just being cynical, but can I see a show of hands of those providers who thought the same thing? Maybe I'm just slow to catch on, but here are what I see the basic principles of an inter-NOC communications system: 1) require an affirmative management action to start participation pay a dollar, cut a ribbon, sign something, do something, anything to avoid the 'NOC cowboy' problem, and everything that engineer did being tainted when he or she leaves. Ok, the CEO doesn't have to understand how it works, she just has to think it was her brillant idea. 2) once in use it should require an affirmative action to disconnect not require anyone to remember to update it. not require anyone to reload the special software on the PC after Microsoft does its periodic self-destruct. neglect shouldn't kill it, only yanking the wires out of the wall or some virtual equivalent should do it in. 3) must not be annoying, but still a pain if it doesn't involve me, don't bother me. the 'normal' channels should be the easiest to use. If you can't show the normal channels failed, you shouldn't be using this. If all else fails, it should get Someone's attention by Klaxon, sirens, flashing lights, you name it. If you answer it, it will shut up. 4) must be a 'visible' system can't be buried on an individual's PC hard drive, or inside a PBX's programming. Even if no one tells them about it, new NOC engineers should trip over it in their normal course of business (i.e. before they need it). "Hey wally, what's that funny machine in the corner for?" 4a) preferably something pointy-haired managers can point to when they are giving tours of the NOC "This is CNN, and this is our inter-NOC communications link" I thought about plugging a EAS-like box into the NOC's CNN feed, since everyone seems to already have CNN in their NOCs. See klaxon, siren, flashing lights above. 5) Positive ID required Everyone (and I mean Everyone) must be able to know exactly who (at least by organizational affiliation) is on, and off the NOC communications net. The list of participants must be able to be published, without fear a non-participant could use the information to intrude into the NOC-net. It should be possible to show the system works to an outsider, without the outsider becoming an insider (i.e. learning the secret handshake). Likewise, it should be possible to positively disprove a non-participant's claim they are on-net if they aren't. No signing MLPA's and never connecting. and I'm a bit leery of #6, but the more I speak with people who give me reasons why they haven't/won't participated in the previous attempts, I can't find a way around it. 6) a STRONG acceptable use policy, and a 'moderator' enforcing it The clash between 'open' and 'private' is a big one. Some folks have said the previous attempts had too much 'noise' and not enough 'signal.' Although from my experience the problem was usually ZERO signal, and ZERO noise. But maybe my filters aren't as sensitive as other peoples'. Since we have not yet created a sucessfull system, all previous attempts going down in the flames of neglect. Who knows if any of those principles are even close to what's required. I hope you noticed most of the my principles involve human or organizational behaivor issues, not technical requirements. I think technically, almost any of the previous systems could have worked. >BTW, I think the line printer idea rules :) In the old days, when I had to walk 20 miles in the snow up hill both ways to school, TYMNET used to send their X.25 outage notifications to customers via Western Union teletype. It wasn't an ASR33, but some type of Telex thermal printer at the time. You knew when it fired up, trouble was on its way. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

3 2

route-server.ip.att.net
by Michael Langdon 16 Jul '98

16 Jul '98

Hey... In case you missed the first time amongst all the noise, this router is available with open telnet. Please use it to your debugging contentment. As always, forward questions/comments to me regarding this box. It should be running the latest CE or CC code available. Mike

1 0

Re: Sprint broken bad at MAE-West?
by Donn Lasher 15 Jul '98

15 Jul '98

> 6 sl-bb21-ana-0-3.sprintlink.net (144.232.1.49) 2.843 ms 2.281 ms >3.05 ms > 7 sl-bb10-sj-0-0.sprintlink.net (144.232.8.130) 10.531 ms 11.275 ms >11.891 >ms > 8 sl-bb1-sj-0-0-0.sprintlink.net (144.232.3.2) 11.816 ms 11.561 ms >11.007 m >s > 9 sl-w2-mae-0-1-0.sprintlink.net (144.232.8.10) 1893.24 ms 1759.43 ms >1851. Called Sprint's NOC (Helpful, as always 1-800-232-6895) about 11am PST, and the tech I talked to was monitoring one of the four FDDI connects they have to MAEW. This comment was "Wow, the 5 min traffic for this link(T3) was 71 meg. I think that pipe is a little busy" My comment "Man you guys should get some compression on that line, what a slow T3!" He laughed. (No need for a Tx primer, it was a joke..) Since there was no offical "outage" they haven't informed their customers. I would attribute this to one of two things: 1. One of the pipes died, therefore all traffic is being carried over the other 3(?!) 2. Rush Limbaugh mentioned an interesting web site this morning, and they've been flooded ever since. I'll leave it up to you to figure out which one it is. Donn

1 0

Sprint broken bad at MAE-West?
by Jason L. Weisberger 15 Jul '98

15 Jul '98

This troubles me... 6 sl-bb21-ana-0-3.sprintlink.net (144.232.1.49) 2.843 ms 2.281 ms 3.05 ms 7 sl-bb10-sj-0-0.sprintlink.net (144.232.8.130) 10.531 ms 11.275 ms 11.891 ms 8 sl-bb1-sj-0-0-0.sprintlink.net (144.232.3.2) 11.816 ms 11.561 ms 11.007 m s 9 sl-w2-mae-0-1-0.sprintlink.net (144.232.8.10) 1893.24 ms 1759.43 ms 1851. Anyone heard anything? -- Jason Weisberger Chief Technology Officer SoftAware, Inc. - 310/305-0275 ...but the wicked shall do wickedly... --Daniel 12:10

3 2

RE: Problem with PBI...
by Dean Robb 15 Jul '98

15 Jul '98

Anthony: Reply from Nick. He's the Big Cheese for abuse matters at PBI and SW Bell. He's got an outstanding rep for getting things done. His efforts, for example, have resulted in PBI and SWBell's lobbyists to start pushing for the Smith Bill as the best anti-spam legislation. CC'd to NANOG since the initial complaint was posted here. >Hiya Dean! > >Sure, this is not a problem at all. In fact, I rather think it's exactly >the kind of thing I need to know about. I'll ask Jason about it tomorrow. > We * have* had our problems with netroplex in the past, but they have had >a decent track record for at least the past year. We can turn up the heat >on them considerably now that we're being disparaged on NANOG... > Spammers: more disgusting than roaches. Dean Robb PC-EASY computer services (757) 495-EASY [3279]

1 0

Fiber Cut in Jax AREA - Affecting UUnet...others?
by Scott Averbach 15 Jul '98

15 Jul '98

Just got off the phone with UUnet and appearently there is a fiber cut there.... Ticket UU895684 - Their blanket ticket number for this incident... just an FYI Scott Averbach Network Analyst Hiway Technologies

1 0

UUnet broken `?
by Winfried Haug 15 Jul '98

15 Jul '98

Hi, we see very high TTL to many US destinations. Beside asymetric routing it seems to be a uunet problem... can anybody confirm this ? TO nitrous.digex.net: 3 Stuttgart.seicom.NET (194.97.216.22) 29.801 ms 21.646 ms 20.728 ms 4 Frankfurt-s0.seicom.NET (194.97.193.10) 31.776 ms Frankfurt-s1.seicom.NET (194.97.193.14) 20.876 ms Frankfurt-s0.seicom.NET (194.97.193.10) 21.808 ms 5 us-gate.Frankfurt.seicom.NET (195.254.80.41) 47.943 ms 45.519 ms 25.216 ms 6 195.219.64.209 (195.219.64.209) 36.325 ms 30.744 ms 29.361 ms 7 195.219.64.209 (195.219.64.209) 59.212 ms 41.559 ms 32.213 ms 8 195.219.64.238 (195.219.64.238) 105.072 ms Teleglobe.net (207.45.215.2) 115.584 ms 104.231 ms 9 gin-nyy-ac1.Teleglobe.net (207.45.199.233) 170.581 ms 160.385 ms 124.512 ms 10 sl-gw16-pen-2-0.sprintlink.net (144.228.164.61) 135.087 ms 152.616 ms 142.091 ms 11 sl-bb10-pen-3-1.sprintlink.net (144.232.5.93) 1029.545 ms 929.967 ms 907.473 ms 12 144.232.5.62 (144.232.5.62) 957.565 ms 886.067 ms 925.234 ms 13 sprint-nap.digex.net (192.157.69.42) 348.060 ms 1306.569 ms * 14 phl1-core1-h1-0.atlas.digex.net (165.117.50.77) 209.121 ms sprint-nap.digex.net (192.157.69.42) 229.315 ms phl1-core1-h1-0.atlas.digex.net (165.117.50.77) 367.977 ms 15 dca3-core1-h1-0.atlas.digex.net (165.117.50.69) 523.725 ms 1728.589 ms 1486.675 ms 16 dca1-core4-h1-0.atlas.digex.net (165.117.50.65) 562.890 ms 1606.708 ms 1327.118 ms 17 dca1-core3-fa6-0-0.atlas.digex.net (165.117.16.3) 1214.027 ms * 1175.428 ms 18 * * * 19 * nitrous.digex.net (205.197.249.3) 1201.812 ms * FROM nitrous.digex.net: 1 iad1-core2-fa5-0-0.atlas.digex.net (165.117.129.2) 0 msec 0 msec 0 msec 2 dca5-core2-h8-0-0.atlas.digex.net (165.117.53.41) 0 msec 4 msec 4 msec 3 dca5-core1-fa1-0-0.atlas.digex.net (165.117.54.1) 4 msec 4 msec 4 msec 4 Hssi3-1-0.BR1.DCA1.ALTER.NET (209.116.159.98) 0 msec 4 msec 4 msec 5 101.ATM2-0.XR2.DCA1.ALTER.NET (146.188.160.238) [AS 701] 0 msec 0 msec 0 msec 6 294.ATM2-0.TR2.DCA1.ALTER.NET (146.188.161.154) [AS 701] 4 msec 0 msec 4 msec 7 101.ATM6-0.TR2.NYC1.ALTER.NET (146.188.136.218) [AS 701] 28 msec 32 msec 28 msec 8 * 298.ATM6-0.XR2.NYC1.ALTER.NET (146.188.178.201) [AS 701] 28 msec 36 msec 9 194.ATM8-0-0.GW2.NYC2.ALTER.NET (146.188.178.149) [AS 701] 32 msec 32 msec 32 msec 10 teleglobe.ny2-gw.customer.ALTER.NET (157.130.4.166) [AS 701] 588 msec 596 msec 604 msec 11 gin-nyy-ac1.Teleglobe.net (207.45.201.37) [AS 6453] 644 msec 588 msec 668 msec 12 gin-ppt-bb1.Teleglobe.net (207.45.199.234) [AS 6453] 600 msec 552 msec 672 msec 13 Teleglobe.net (207.45.215.1) [AS 6453] 700 msec 668 msec 672 msec 14 195.219.64.237 [AS 8297] 916 msec * 888 msec 15 * * 195.219.64.210 [AS 8297] 824 msec 16 * de-gate.Frankfurt.seicom.NET (195.254.80.11) [AS 5549] 1032 msec 928 msec 17 Stuttgart-s6.seicom.NET (194.97.193.13) [AS 5549] 908 msec de-gate.Frankfurt.seicom.NET (195.254.80.11) [AS 5549] 860 msec Stuttgart-s5.seicom.NET (194.97.193.9) [AS 5549] 1280 msec Winfried -- Winfried Haug | seicom.NET & SCHWABEN.DE | Tel. +49 7121 9770- 0 Laiblinsplatz 12 | Internet+ISDN access & consulting | Fax. +49 7121 9770-19 72793 Pfullingen | Access in STGT + RT + TUE + BB + LB | Rack +49 7433 1449-X haug(a)schwaben.net| 150*ISDN (64kb/X.75) / 100 * K56flex | Rack +49 711 9675-X haug(a)seicom.NET |SAP-OSS * FTP * TELNET * NETNEWS * IRC | Rack +49 7121 709-X 34 Mbit Teleglobe * 100 MBit DE-CIX * 2 MBit INXS * 2 Mbit Belwue * 34 Mbit WIN

6 6

Re: NOC communications
by Sean Donelan 14 Jul '98

14 Jul '98

>OK, but large providers don't want to talk to small providers. It's >a stated policy of many networks to refer a smaller customer/provider >to their upstream. I only want to talk to someone who's competent Heck, I only want to talk to someone who is competent too. However, I would not say there is a direct relationship between the size of a provider and the competency of its engineers. It has been my experience you are much more likely you get a clueless person with no authority to fix anything when you call a larger provider about a problem than when you call a smaller provider. Could a reason they get a lot of calls, is they are the root cause of a lot of problems? Even chaos has its reasons. The actual policy of many major networks seems to be to direct everyone to call someone else. The reason I call other providers is because the major providers/upstreams DO NOT do a good job of refering their customer's tickets between themselves even when I pay them money. I've been told by various major providers I should directly call another provider's NOC about a problem. In some cases both providers have been owned by the same corporation, but they refused to refer a trouble-ticket between their two NOCs. I don't have a way of knowing the types of quantity of inappropriate calls a typical larger ISP's NOC receives. I also don't know how many calls a typical larger ISP's NOC inappropriately blows off. If it takes me five phone calls to finally reach someone who understands the problem, AND has the authority to fix it, how do the NOC engineers count the previous four phone calls they ignored? If the upstream providers didn't tell their customers to call the other providers, and had better inter-NOC communications, perhaps fewer people would call other providers directly because they would know their own provider's NOC has better inter-NOC communication and could resolve the problem faster? If you have question about your electric bill, you call the electric company. If there are wires down and sparking in your front yard, you call 9-1-1 because they have better access to the electric company's emergency lines than you do. >My point is only that there is always a pecking order and by making >network ops egalitarian you end up with a high signal/noise ratio, >or worse yet, all the people who actually are relevant leave. >(Witness NANOG: half the people from MCI etc. don't read any more) I thought half the people from MCI left to join small startup companies :-) I wonder if they are now finding out just how difficult their former employer is to deal with now that they are on the outside looking in. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

1 0

EU-IBMnet? IPaddr 32.xxx
by Rob M VanHooren 14 Jul '98

14 Jul '98

Is it just me or is IBM GlobalNet leaking this into their F/R cloud on purpose? Supposedly a private VPN (albeit transiting PSDN) and here I've got 32.74.50.217 leaking all over the place? Anybody know what's up? --Rob. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-.._..-*~ Rob M. VanHooren - Chief Plumber, +1 519 679-1155 x33 Packet Pusher, and Resident Mad Scientist(tm) Alarms to 646-4724 Network Engineering Services 171 Queens Avenue, Suite 320 Linkdata Communications Inc. ASN:10856 London, CANADA Good things come to those who wait. Patience is a virtue. Yadda Yadda Yadda

1 0

Re: EU-IBMnet? IPaddr 32.xxx
by Sean Butler 14 Jul '98

14 Jul '98

>Is it just me or is IBM GlobalNet leaking this into their F/R cloud on >purpose? Supposedly a private VPN (albeit transiting PSDN) and here I've >got 32.74.50.217 leaking all over the place? > >Anybody know what's up? > --Rob. Please contact me off-line directly and we can discuss further... Thanks, Sean ___________________________________ Sean Butler IBM Global Services -- OpenNet Support Phone: 8-631-9809, 813-523-7353 Fax: 8-427-5475 813-878-5475 Internet email: sebutler(a)us.ibm.com

1 0