- NANOG - lists.nanog.org

Re: outages, quality monitoring, trouble tickets, etc
by Jonathan Heiliger 28 Nov '95

28 Nov '95

On Nov 27, 7:11pm, Jeff.Ogden(a)um.cc.umich.edu wrote: > It is because of these sorts of questions that I never know what > it means when someone says that their network is available 99.9% > of the time. Hell at least part of MichNet is available 100% of > the time, but that never seems to cut it with the folks who don't > have service right now. I hate to say it, but the relationship in my Email regarding carriers not disclosing network outages could potentially be to maintain the guise of a better than real-life network availability. If we want to enter the tangent discussion that seems inevitable at this point, perhaps we should at least change the subject. :-) -jh-

1 0

Operational declaration of route servers at PacBell NAP
by Jake Khuon 27 Nov '95

27 Nov '95

The Routing Arbiter Service is pleased to announce it is fully in production at the PacBell NAPs. Two route servers are installed at the NAP to provide redundancy, and the are reconfigured every 4 hours. The route server configurations are based on the routing policy information stored in the Internet Routing Registry. The IRR contains the full set of data from five databases: Routing Arbiter Database, RIPE Routing Registry, internetMCI Routing Registry, CA*net Routing Registry, and ANSnet Database. For more information about the RA Service, please refer to: http://www.ra.net/routing.arbiter/RA/ To use the RA Service at the PacBell NAP, send mail to: ra-peer(a)ra.net -- /*====================[ Jake Khuon <khuon(a)Merit.Net> ]=====================+ | Systems Research Programmer, IE Group /| /|[~|)|~|~ N E T W O R K | | Vox: (313) 763-4907 Fax: (313) 747-3185 / |/ |[_|\| | Incorporated | +====[ Suite C2122, Bldg. 1 4251 Plymouth Rd. Ann Arbor, MI 48105 ]====*/

1 0

subscribe
by Bill Price 27 Nov '95

27 Nov '95

please subscribe ------------------------------------- E-mail: bprice(a)bsan.com General Manager, Internet Services BellSouth Date: 11/27/95 Time: 15:56:05 This message was sent by Chameleon -------------------------------------

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Scott Huddle 27 Nov '95

27 Nov '95

Jonathan Heiliger <loco(a)mfst.com> writes > On Nov 27, 9:17am, Sean Doran wrote: > > Jonathan> Everyone likes to portray > > Jonathan> the image of having a 99.98% uptime whenever > > Jonathan> possible, even though most folks realize > > Jonathan> that it just plain isn't possible > > > > Well, more importantly, what on earth does a number like > > that mean? > > Sorry, bad choice of words. Rather than uptime, availability would be the > proper word. Availability tends to be the amount of time the network is > "available" for the customer to receive their expected service (whether > guaranteed in writing or not), and for the customers expectation of how the > service will perform when it is considered "in-service" is met. This is reasonable expectation, but unfortunately it's pretty difficult to measure what "the network" is. Do you mean your NSPs backbone? Its connections to other NSPs? Connections to a specific site? Global connectivity? How do you factor when you or a target site is singly connected? -scott

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Jamshid Mahdavi 27 Nov '95

27 Nov '95

In article <951123045818.17481(a)SDG.DRA.COM> Sean Donelan <SEAN(a)SDG.DRA.COM> writes: > Welcome to the new and improved Internet. More clueless people cal > NOCs these days (is it plugged in?) so more caller screening is done. > Likewise there more clueless people working in NOCs so more levels > before reaching someone who even understands what the problem is. I've had an idea in the back of my head for a long time about dealing with this, ever since I tried to call an 800 customer assistance number and got a nice message saying "the current average wait is 45 minutes"... I guess it could apply equally well to NOCs, so here it is. Provide a "secret" layer 2 NOC. When someone calls the NOC 800 number who is obviously more clueful than the NOC operator/screening person, pass them onto the next level of experts, and also give them a new "secret" 800 number. Then in the future, that caller will skip the first layer of screeners who handle the "is-it-plugged-in" problems. Of course, there is no reason to only have two levels. Callers with proven technical competence (over time) automatically get shunted to help at the right level for their know-how. Periodically, you probably want to recycle the 800- numbers down to the lowest level so that word of the secret numbers doesn't get too widespread. I'd sure like to see someone implement this (or something like it). A few companies we have worked with have had individual techies give us their direct number and say "from now on, just call me direct", but this is rare and probably would be a bad idea for a NOC. --Jamshid

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Sean Donelan 27 Nov '95

27 Nov '95

Sorry, another long one. Relevance to NANOG, well, NOCs are customers too. And "remote" NOCs often report problems that effect your paying customers too. Customer service should be of interest to operations folks, at least to the extent the problems are getting reported to the right people to fix. I doubt I can change anyone's mind that providing explanations to customers and non-customers when the network has problems is good for business. In the future I will simply recommend to customers to buy services from NSPs which do provide explanations when their networks fail. Since I haven't found a perfect network yet, I suspect it includes everyone on this list. > Why should they talk to you? Do you pay them a service fee? > That's my base issue, there is a hierarchy, and you can't skip > rope to the other guy. It just doesn't work, there's nothing in > the system to encourage it. The hierarchy is dead. None of the old NSFnet regional have a monopoly on service in their regions any more. Outside of the US, there are still a few monopoly providers, but they are a rare breed. If you aren't providing the level of service I need, I'll go to someone who can. If XYZ's NOC gives me better service than ABC's NOC, I'll recommend XYZ to my customers. > Sean Donelan has a terribly good point, he's my customer, and his > words mean alot, but I can't agree w/ him that he should/could > demand the same thing from another ex-NSFnet regional, or from > Sprint. I certainly see no reason why I should do this work for > you. Because it is in their self-interest? You are correct I can't make anyone run their network how I would like it run, not even MIDNET (GI). But I can point out long-term problems and code of silence is costing such providers money, and has already cost them customers. For example, I really wish my direct providers would stop munging BGP announcements, or explain why they are doing it. If I have made a mistake, I would like to fix it. Otherwise I will come to the conclusion those provider's NOCs are not up to the job and find a different provider that can do the job. When someone (anyone) reports a problem effecting connectivity with your network, more than likely the reverse is also true for your paying customers. DRA has a bunch of customers connected through just about every major NSP in North America and a couple of other continents. The only time "I" call another NSP is when the process has become totally FUBARed. When I call another NSP, it is usually that NSP's last chance to keep a paying customer on their network. I might call BARRNET because the University of California-Davis has reported problems reaching DRA to DRA's help desk, and the problem hasn't been resolved. No, BARNET doesn't *have* to talk to me. And I will report the same back to the customer. However, I suspect it is in BARRNET's self-interest to work with me in resolving the problem to ensure UC-Davis has end-to-end reliability. I track network reliability by dollars (not packet loss, not latency). I measure network providers, good and bad, by how many of our customers have used their own dollars to buy private lines to St. Louis because they couldn't get the reliability they needed from the network provider. It is not a pretty picture. <http://dranet.dra.com/dranet.html> has a picture where our private line customers are located. If you are an NSP, every one of those green boxes (some boxes represent many paying customers) is an arrow through the heart of your (former) customers view of your network reliability. If you are an NSP in one of those areas, not dealing with these problems or providing coherent explanations has cost you cold-hard cash. Money is something I expect most upper managers to understand. DRA makes its profits elsewhere. DRAnet is simply a vertical market VPN used to sell access to other things. I'm happy to use the Internet and other NSP's to provide that VPN, when the quality exists. On the other hand, if I have to manage a not-so virtual VPN with private lines to achieve the required level of quality, I do. Maybe Adam Smith's invisible hand will correct this eventually. > There seems to be this large obsession with linking information to > action. If you get an update you think something's happening. > Perhaps it's needed, but stuff will happen whether your hand is > held or not. As I said before: Ideally I want a reliable network. If you can't provide a perfectly reliability network I want an explanation when I can't get through. And I want the problem fixed. The better the explanation, the longer I'm willing to give you to fix the problem. If I get no explanation, I expect the problem to already be fixed. The current situation is the customer gets neither the explanation nor action solving the problem. My proof is the DRAnet map. DRA's customers take a very, very long time to budget money. Those green boxes represent customers whose problems went unanswered, and unsolved for a long time before they gave up on their NSP and expended their own dollars for a private line to St. Louis. Since the technicians seem to be having a very difficult time fixing the network, I thought upper management could meet my other goal. Give the customer an explanation. I'm not pointing fingers at any particular NSP, because frankly I don't have enough fingers to point. Everyone had problems. Yes, even DRA's NOC has fallen down a few times. I'm not asking for perfection, but an explanation when things don't work, while you fix the problem. The Internet is a global cooperative network. If people don't cooperate, the global nature of the network fails. Since your customers may in fact want to use the Internet to communicate globally, problems effect customers globally. When I go to the US Post Office, sometimes there is a sign on the wall that postal service to Timbukto may be delayed because Timbukto's main post office was blown up. I have no idea how many postal customers in Olivette, Missouri send mail to Timbukto. Even though the US Post Service has no control over rebuilding Timbukto's main post office, the US Post Service has discovered it is good customer service to inform their customers why their mail to Timbukto may be delayed. Can't NSPs provide their customers an explanation at least as well as the US Post Office? -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

5 4

Historical perspective
by bmanning＠ISI.EDU 26 Nov '95

26 Nov '95

Perhaps, a quick review of RFC 1273 would be useful to those who are interested in service level reachability. Also note the RTFM and IPPM efforts within IETF to develop frameworks for developing critera for measurement. -- --bill

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Sean Donelan 26 Nov '95

26 Nov '95

>From: Scott Huddle <huddle(a)mci.net> > I consider this list a place for ISPs to discuss general policy and > planning issues that effect all of us. It is a very inappropriate > place to discuss problems with a specific provider. As a general policy I wish all network providers would implement at a minimum a network notification list. Bonus points for a network status WWW page, NetNews, FAX, and PR Newswire distribution. >From: bsimpson(a)morningstar.COM (William Allen Simpson) > I firmly DISAGREE! None of us are particularly interested in hearing > every jot and tittle about every network flap, but _BIG_ ones and their > resolution are important to bring to this list! How else to get a > handle on what the real problems are? How else to help each other > avoid repeating the problem in the future? This is always a tough call. When is news no longer news. When it happens all the time. The slow intermittent problems are the hardest problems to fix. I end up tracking problems from Europe to Australia for customers, so I'm interested in network outages all over the world. On the other hand, I don't (usually) care when Bill's PC is turned off at night. Maybe take a cue from the (failed) NetNews Distribution: or MBONE-sublist mechanism. What to do about backbone telco providers which hate announcing their network problems? Maybe when the NSFNET backbone fractured into multiple backbones, the NSF should have taken a lesson from the breakup of AT&T and established a Network Reliability Council. Any telephone network outage effecting more than 50,000 (now lowered to 15,000 I believe) lines have to be reported to the FCC's NRC. Who cares if you connect to three (plus one) NAPs. In the post-NSFNET era, Internet-wide reliability requires Internet-wide information. Reporting network reliability problems is more important than how many NAPs your network connects. >Has anybody else noticed how hard it is to get trouble tickets these >days? Once upon a time, I just called the NSF NOC, and got a report to >them in real time, so the problem could be fixed quickly. Nowadays, >NOCs seem to want you to send email with 24 or 48 hour turnaround, or go >through 2 layers of service representatives. Pretty hard to send email >to them when their link is down, or go through "regular" support in the >middle of the night! Welcome to the new and improved Internet. More clueless people cal NOCs these days (is it plugged in?) so more caller screening is done. Likewise there more clueless people working in NOCs so more levels before reaching someone who even understands what the problem is. NOC-to-NOC communication has been a long standing Internet problem. But now there are more NOC's, more different ways to contact them, with no common conventions. Even though its out of date, I still keep my Internet Manager's Phonebook published by BBN in 1990. ANS, MCI, and Sprint issued press releases a few months ago about their joint agreement. I don't know how how well their joint agreement is working, but I suspect there will be more such agreements between network operators in the future. As the sheer number of people involved grows, everyone is going to filter their calls, e-mail, etc. If you aren't on the list, you get dumped in the "take care of after hell freezes" pile. In the meantime, keep a stack of business cards and a special rolodex, with the magic names and telephone numbers that get you directly to someone who can understand (and maybe even fix) the problem. Interesting enough, the people usually don't change; but the employers do. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

5 6

Re: outages, quality monitoring, trouble tickets, etc
by Sean Donelan 24 Nov '95

24 Nov '95

From: alan(a)gi.NET (Alan Hannan) > Hmm, I wonder if the Trib' would be interested in knowing when the > DS3 from Pensaulen is down..... Sometimes the Trib would be interested, most of the time they wouldn't. Who cares if a central office in Hillsdale, IL burns down? Some people thought it was front page news. > At a precursing glance I would agree with you. However, let us > delve into this a bit deeper. Donning my idiot hat may I point out > that the _most_ important thing is network reliability -Period-. That would be great, please give me the name of a network provider which provides perfect network reliability. In the absence of perfection, please tell me what went wrong when I can't get an expected level of usability out of the network. You can greatly reduce your customers' stress levels simply by keeping them informed. Give me the TCP/IP equivalent of "*beep* *boop* *BEEP* We're sorry your TCP/IP connection can not be completed due to an earthquake (software glitch, route table overload, nuclear detonation) in the area. Please hang up and try your call later." With an accurate RA database, and a little magic, the route servers could redirect connections to an intercept message. That should send a shiver up the spine of your network security folks. It would be nice if the problem is also fixed quickly, but I realize that is asking for a lot. In the mean time, keep the customer informed. As the size of the Internet has grown, keeping the customer informed is a bigger job. Relying on a 1-800 number doesn't work when a large NSPs backbone melts down, and all the NSP's customers call the NOC at the same time. > Your page at DRA is quite good, however the concensus among > upper management (not just at our site) is "Why should other > people know when we're broke?". And the sad thing is, I am > tempted to agree with them. Thanks for the complement. I would point out to your upper management other people already know when your network is broken. If they didn't notice, it wouldn't be a problem. If a network falls over in the woods, and there was no one to hear it, does it make a sound? Tell your upper managers, the only time people don't know when your network is broken is when your network is irrelevant to their work. I don't know about you, but if I was managing an irrelevant network, I would be working on my resume. Maybe that's why so many people in this business keep switching employers? :-) > Do you really want outage and downtime on public record, or do you > want easier access to clueful folx? As a network user (operator, manager): - Ideally I want a useable network. - When I can't use the network, I want an explanation. - I want the problem fixed so I can use the network again. How you meet those needs, I don't care. If you fix the problem before I'm effected by it, then I don't care about the intermediate steps either (tree falling in the forest). If I can get the explanation from an automated server, then I don't have to bother your clueless or cluefull folk. If your clueless folk can take a log and get it resolved, I don't have to bother your cluefull folk. When I need to dig out my magic cache of business cards and start e-mailing/calling the secret members of the "backbone cabal" to get a problem fixed, I consider it a failure of the process. That is what I meant when I said NOC-to-NOC communications has been a long-term Internet problem. It relies on personal contacts, rather than a reliable process. I try very hard not to handle problems by directly contact people I know at other NOCs. I prefer handling problems through the normal NOC channels. If you do happen to receive a phone call directly from me about a network problem, something has gone very wrong. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

1 0

APRICOT Announcement
by Geoff Huston 24 Nov '95

24 Nov '95

There are a number of reasons for posting this announcement here: - Hans-Werner asked me to, - the weather in Singapore at that time of the year will be much warmer than _anywhere_ on the North American continent, - it will be an interesting operational workshop! cheers, Geoff Huston CALL FOR PARTICPATION In The First - - - - - - - A P R I C O T - - - - - - - ASIA PACIFIC REGIONAL INTERNET CONFERENCE ON OPERATIONAL TECHNOLOGY http://www.apricot.net/apricot/ Jan 17-22 '96, Singapore Sponsored In Part By: * The Commercial Internet Exchange ( http://www.cix.org/ ) * & Others T.B.A. shortly SUMMARY: APRICOT is a forum for the exchange of technical know-how for In- ternet Service Providers and Network Operators OVERVIEW: The rapid growth of the Internet has brought many newcomers into positions of critical operational responsibility. They are ex- pected to be experts in the design, configuration, operation and maintenance of routers, terminal servers, WWW & UNIX servers, and a myriad of specific hardware/software systems. They must operate network management centers, customer support lines, and handle security incidents. They must liase with network registries, com- munications carriers, and their colleague operators. They must perform these duties efficiently even as their networks grow in- creasingly large and complex. Yet, the resources for learning the skills necessary to perform these tasks successfully are sparse and rarely provide specific information and examples. To address this critical need, the ASIA PACIFIC NETWORK INFORMA- TION (APNIC) has initiated the ASIA PACIFIC REGIONAL INTERNET CONFERENCE ON OPERATIONAL TECHNOLOGY (APRICOT). APRICOT consists of seminars, conference sessions, and forums all with the goal of spreading and sharing the specific technical in- formation required to operate the increasingly complex Asia Pa- cific Internet topologies. APRICOT will be held immediately preceding the APNIC general meeting and the Asia Pacific Networking Group (APNG) meetings which includes the commercial sub-group and the internationaliza- tion and localization subgroup meetings. VENUE: SEAMEO Regional Language Centre (RELC), Singapore SCHEDULE: * Jan 17 Tutorials from invited speakers. * Jan 18 Conference * Jan 19 Morning: Conference Afternoon: Forum * Jan 20 APNG-commercial * Jan 21 APNIC Asia Pacific Network Information Center meeting * Jan 22 APNG Asia Pacific Networking Group general meeting (being organized separately) APRICOT TOPICS: * IP and ASN Addressing issues & techniques * Exterior and Interior Gateway Protocols and Interexchanges * Telco DDN Systems * Domain Name Server * Usenet News * WWW, FTP, Shell, & Email Servers * Server and Network Security * Billing & Accounting Systems INVITED SPEAKERS FOR TUTORIALS The APRICOT organizing committee is inviting world-recognized ex- perts to lecture on the above topics. CALL FOR PRESENTATIONS Presentations on the above or related topics are solicited from from the Internet operations community. Presentations should present your experiences with the above is- sues and share solutions to them. VENDOR EXHIBITS Space for up to 10 Vendor booths will be available. ATTENDEE REGISTRATION: USD175 CLOSE DATES FOR PRESENTATION ABSTRACTS: To Be Announced ORGANIZERS, CONTRIBUTORS * APNIC * Cogwheel Internet Technolgy Consulting * Singapore Computer Society * Internet R&D Unit, ComCen, National University of Singapore * SingNet * Internet Association of Japan * National Computer Board, Singapore SPONSORS * Commercial Internet Exchange ( http://www.cix.org/ ) * Others T.B.A. shortly. INFORMATION / PRE-REGISTRATION Browse the following WWW URL to join the apricot information email list. We will send you more detail shortly. http://www.apricot.net/apricot/register.html/ SUBMISSIONS / SPONSORSHIPS Send email to: apricot-oc(a)apricot.net ----

1 0