- NANOG - lists.nanog.org

Re: outages, quality monitoring, trouble tickets, etc
by kwe＠6SigmaNets.com 28 Nov '95

28 Nov '95

At 5:34 PM 11/27/95, Dave Siegel wrote: >> >> It is because of these sorts of questions that I never know what >> it means when someone says that their network is available 99.9% > >Problems with CPE is usually not figured into network uptime. > I've been away from the NOC for a couple of years now, but one way that made sense for reporting a general availability number is to consider the backbone as one component and each leaf node as an identical independent component (leased line, router and CSU/DSU). Then you can define the end to end availability as the concatentation of two independent leaf nodes and the backbone. Highly redundant backbones always have a high availability (>99%). Leased lines are abominable (low 90s or less -- don't ask me about circuits to Italy) and two routers and two CSU/DSUs lead to an overall end to end of 95% or less. In my experience with leased lines, they always dominated this equation and overall availability is strictly limited by actual leased line availability. So keep that CPE in the equation and figure out how to get highly available bit pipes. I think actual networks like frame relay have a chance of being much more available since the network actually understands frames. Leased lines are simply electrical signalling conduits to the phone company -- not a network. I'm skeptical of any end-to-end availability figures over 97%. I don't think they reflect the reality of leased line circuits today, or else they don't include the leaf node circuits and only report backbone availability. For a highly redundant backbone, almost any definition of availability should result in a number like 99.mumble%. Remember 99.9% availability means less than 9 hours outage per year. Routing hiccups take that much. One or two leased lines outages is all you get for 9 hours. The real world is a lot less available than that. But since half the web servers I try to talk to refuse me half the time, I'm not sure that network availability per se (HWB's complaints duly acknowledged) is the tallest pole in the tent. --Kent

1 0

Re: Links on the blink - what will/should mci & sprint do?
by Tony Li 28 Nov '95

28 Nov '95

[Sorry for being late in responding, I've been on vacation...] Let me start by pointing out that the global problem here is to maintain and grow a routing infrastructure at exponential rates and at "interesting" flap rates. Using a level 2 switching infrastructure as a means of bandwidth control is clearly not a comprehensive solution as it does not provide the necessary level 3 routing [hopefully this is trivially obvious to anyone reading this, but...]. Is a level 2 infrastructure a cost effective means of doing bandwidth control? Possibly, but the exact economics vary widely depending on the level of intra-POP bandwidth necessary and the prices that each ISP is able to obtain. Using a level 2 architecture clearly does nothing to avoid the "brick wall". This becomes quite clear as we can easily demonstrate brick wall phenomena at any interconnect, regardless of the router (or make of router) and the fabric behind it. The question then focuses on the maintainability of the routing infrastructure. We know that we have two limits: perfectly stable routing, and completely unstable routing. Neither is interesting. Perfectly stable routing is simply utopian thinking. Completely unstable routing is unusable even given arbitrarily fast technology. In between, we have an enormous gray area, with no good way to quantify it: routes that flap, and packets that interact with that flap. Clearly if a route flap does not affect any packets, that should not perturb the infrastructure. [We're pleased to say that extensive testing of a 7000 and 7500 points out that in fact, it doesn't. ;-)] For the sake of brevity, let's call this "uninteresting flap." What then is the response of the routing system to intesting flap? We can start by characterizing the flap by T_{down}, the period of time that a route is withdrawn, measured from the "down edge" of the withdrawl of the route to the "up edge" of the installation of the replacement. Subsequent to the "up edge", we have a further T_{recover}, which is the time that it takes for the infrastructure to forward at peak rate again. Using this, we can begin to characterize the behaviors that we'd like. Clearly, during T_{down}, we have a problem: we have no route and (as the interesting part of the problem is in the default-free portion of the net), we have no other shorter prefix for it. One can then either drop packets or forward them along the old route anyhow. The Internet philosophy has always been to drop packets as quickly as possible, as close to the source as possible. The alternative is to forward the packet anyhow, hoping that the packet is not forwarded into a sustained forwarding loop. As the routing protocols aren't truly architected to protect against such a forwarding loop, the latter behavior seems risky. The implication is that during T_{down}, the infrastucture will drop packets from all sources to the destination. Next, consider T_{recover}. As no one has invented FTL hardware (yet ;-), T_{recover} is non-zero on all interesting architectures that I know of. Further, during T_{recover}, packet loss should also be expected. This includes the case of a router which fully distributes the routing table, as the distribution itself takes finite time. What then are interesting values for T_{recover}? Clearly, \infinity is unacceptable. If T_{recover} << T_{down}, it hardly seems relevant. What then is an acceptable upper bound? And what is the sensitivity of this value? I leave this as an open question for discussion... Selective packet discard (a new method we're putting in for prioritizing routing protocol packets over transit packets) will insure that T_{recover} is finite, and based on our lab tests, what I consider to be acceptable. I should note in fairness that credit for driving selective discard should be given to both Sprint and ANS. The former for demonstrating the need in the real world, the latter for focusing cisco's management on the subject. I'd also like to correct some misperceptions that have been distributed: - Distributing the full routing table is not the only way to achieve acceptable response to interesting flap. If someone has a technical argument suggesting that this is the only true way, then we have missed hearing it, and we'd really apprecaite it if you could tell us again. In detail. - Distributing the full routing table to the SSE is not a logical way to proceed. For both obvious and non-obvious reasons, spending significant resources developing software for the 7000 at this point makes little sense, and without some convincing argument that it would move T_{recover} from the unacceptable to the acceptable, is beyond the bounds of rationality. - The interesting flap rate introduced by end users can be controlled by deployment of CIDR. I trust no one on this list remains to be convinced. - Certain statements about communications of problems to cisco mentioned in this forum have been patently incorrect. Sending a mail message to your favorite engineer saying "when will you do XYZ" is not a reasonable way to insure that you're mission critical business feature is being implemented. Still, if you believe you've attempted to communicate with us and you don't think we got the message, then we can only ask that you have patience, increase your retry count, and retransmit. We can't neccessarily do what you might like, but we can hear you and acknowledge your concerns. Finally, I'd like to echo what Dennis said: we're all behind the eight ball, and we all know it. Recriminations are much less interesting than solutions. What should MCI & Sprint do? Well, let's just say that we've privately offered them our opinions, and they don't seem to object too strenuously. Assuming that things actually go that way, well, "You ain't seen nothing yet, baby." Tony p.s. I'm not on the mailing list, so please cc me if you wish me to respond.

4 6

Re: outages, quality monitoring, trouble tickets, etc
by Jonathan Heiliger 28 Nov '95

28 Nov '95

On Nov 27, 7:11pm, Jeff.Ogden(a)um.cc.umich.edu wrote: > It is because of these sorts of questions that I never know what > it means when someone says that their network is available 99.9% > of the time. Hell at least part of MichNet is available 100% of > the time, but that never seems to cut it with the folks who don't > have service right now. I hate to say it, but the relationship in my Email regarding carriers not disclosing network outages could potentially be to maintain the guise of a better than real-life network availability. If we want to enter the tangent discussion that seems inevitable at this point, perhaps we should at least change the subject. :-) -jh-

1 0

Operational declaration of route servers at PacBell NAP
by Jake Khuon 27 Nov '95

27 Nov '95

The Routing Arbiter Service is pleased to announce it is fully in production at the PacBell NAPs. Two route servers are installed at the NAP to provide redundancy, and the are reconfigured every 4 hours. The route server configurations are based on the routing policy information stored in the Internet Routing Registry. The IRR contains the full set of data from five databases: Routing Arbiter Database, RIPE Routing Registry, internetMCI Routing Registry, CA*net Routing Registry, and ANSnet Database. For more information about the RA Service, please refer to: http://www.ra.net/routing.arbiter/RA/ To use the RA Service at the PacBell NAP, send mail to: ra-peer(a)ra.net -- /*====================[ Jake Khuon <khuon(a)Merit.Net> ]=====================+ | Systems Research Programmer, IE Group /| /|[~|)|~|~ N E T W O R K | | Vox: (313) 763-4907 Fax: (313) 747-3185 / |/ |[_|\| | Incorporated | +====[ Suite C2122, Bldg. 1 4251 Plymouth Rd. Ann Arbor, MI 48105 ]====*/

1 0

subscribe
by Bill Price 27 Nov '95

27 Nov '95

please subscribe ------------------------------------- E-mail: bprice(a)bsan.com General Manager, Internet Services BellSouth Date: 11/27/95 Time: 15:56:05 This message was sent by Chameleon -------------------------------------

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Scott Huddle 27 Nov '95

27 Nov '95

Jonathan Heiliger <loco(a)mfst.com> writes > On Nov 27, 9:17am, Sean Doran wrote: > > Jonathan> Everyone likes to portray > > Jonathan> the image of having a 99.98% uptime whenever > > Jonathan> possible, even though most folks realize > > Jonathan> that it just plain isn't possible > > > > Well, more importantly, what on earth does a number like > > that mean? > > Sorry, bad choice of words. Rather than uptime, availability would be the > proper word. Availability tends to be the amount of time the network is > "available" for the customer to receive their expected service (whether > guaranteed in writing or not), and for the customers expectation of how the > service will perform when it is considered "in-service" is met. This is reasonable expectation, but unfortunately it's pretty difficult to measure what "the network" is. Do you mean your NSPs backbone? Its connections to other NSPs? Connections to a specific site? Global connectivity? How do you factor when you or a target site is singly connected? -scott

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Jamshid Mahdavi 27 Nov '95

27 Nov '95

In article <951123045818.17481(a)SDG.DRA.COM> Sean Donelan <SEAN(a)SDG.DRA.COM> writes: > Welcome to the new and improved Internet. More clueless people cal > NOCs these days (is it plugged in?) so more caller screening is done. > Likewise there more clueless people working in NOCs so more levels > before reaching someone who even understands what the problem is. I've had an idea in the back of my head for a long time about dealing with this, ever since I tried to call an 800 customer assistance number and got a nice message saying "the current average wait is 45 minutes"... I guess it could apply equally well to NOCs, so here it is. Provide a "secret" layer 2 NOC. When someone calls the NOC 800 number who is obviously more clueful than the NOC operator/screening person, pass them onto the next level of experts, and also give them a new "secret" 800 number. Then in the future, that caller will skip the first layer of screeners who handle the "is-it-plugged-in" problems. Of course, there is no reason to only have two levels. Callers with proven technical competence (over time) automatically get shunted to help at the right level for their know-how. Periodically, you probably want to recycle the 800- numbers down to the lowest level so that word of the secret numbers doesn't get too widespread. I'd sure like to see someone implement this (or something like it). A few companies we have worked with have had individual techies give us their direct number and say "from now on, just call me direct", but this is rare and probably would be a bad idea for a NOC. --Jamshid

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Sean Donelan 27 Nov '95

27 Nov '95

Sorry, another long one. Relevance to NANOG, well, NOCs are customers too. And "remote" NOCs often report problems that effect your paying customers too. Customer service should be of interest to operations folks, at least to the extent the problems are getting reported to the right people to fix. I doubt I can change anyone's mind that providing explanations to customers and non-customers when the network has problems is good for business. In the future I will simply recommend to customers to buy services from NSPs which do provide explanations when their networks fail. Since I haven't found a perfect network yet, I suspect it includes everyone on this list. > Why should they talk to you? Do you pay them a service fee? > That's my base issue, there is a hierarchy, and you can't skip > rope to the other guy. It just doesn't work, there's nothing in > the system to encourage it. The hierarchy is dead. None of the old NSFnet regional have a monopoly on service in their regions any more. Outside of the US, there are still a few monopoly providers, but they are a rare breed. If you aren't providing the level of service I need, I'll go to someone who can. If XYZ's NOC gives me better service than ABC's NOC, I'll recommend XYZ to my customers. > Sean Donelan has a terribly good point, he's my customer, and his > words mean alot, but I can't agree w/ him that he should/could > demand the same thing from another ex-NSFnet regional, or from > Sprint. I certainly see no reason why I should do this work for > you. Because it is in their self-interest? You are correct I can't make anyone run their network how I would like it run, not even MIDNET (GI). But I can point out long-term problems and code of silence is costing such providers money, and has already cost them customers. For example, I really wish my direct providers would stop munging BGP announcements, or explain why they are doing it. If I have made a mistake, I would like to fix it. Otherwise I will come to the conclusion those provider's NOCs are not up to the job and find a different provider that can do the job. When someone (anyone) reports a problem effecting connectivity with your network, more than likely the reverse is also true for your paying customers. DRA has a bunch of customers connected through just about every major NSP in North America and a couple of other continents. The only time "I" call another NSP is when the process has become totally FUBARed. When I call another NSP, it is usually that NSP's last chance to keep a paying customer on their network. I might call BARRNET because the University of California-Davis has reported problems reaching DRA to DRA's help desk, and the problem hasn't been resolved. No, BARNET doesn't *have* to talk to me. And I will report the same back to the customer. However, I suspect it is in BARRNET's self-interest to work with me in resolving the problem to ensure UC-Davis has end-to-end reliability. I track network reliability by dollars (not packet loss, not latency). I measure network providers, good and bad, by how many of our customers have used their own dollars to buy private lines to St. Louis because they couldn't get the reliability they needed from the network provider. It is not a pretty picture. <http://dranet.dra.com/dranet.html> has a picture where our private line customers are located. If you are an NSP, every one of those green boxes (some boxes represent many paying customers) is an arrow through the heart of your (former) customers view of your network reliability. If you are an NSP in one of those areas, not dealing with these problems or providing coherent explanations has cost you cold-hard cash. Money is something I expect most upper managers to understand. DRA makes its profits elsewhere. DRAnet is simply a vertical market VPN used to sell access to other things. I'm happy to use the Internet and other NSP's to provide that VPN, when the quality exists. On the other hand, if I have to manage a not-so virtual VPN with private lines to achieve the required level of quality, I do. Maybe Adam Smith's invisible hand will correct this eventually. > There seems to be this large obsession with linking information to > action. If you get an update you think something's happening. > Perhaps it's needed, but stuff will happen whether your hand is > held or not. As I said before: Ideally I want a reliable network. If you can't provide a perfectly reliability network I want an explanation when I can't get through. And I want the problem fixed. The better the explanation, the longer I'm willing to give you to fix the problem. If I get no explanation, I expect the problem to already be fixed. The current situation is the customer gets neither the explanation nor action solving the problem. My proof is the DRAnet map. DRA's customers take a very, very long time to budget money. Those green boxes represent customers whose problems went unanswered, and unsolved for a long time before they gave up on their NSP and expended their own dollars for a private line to St. Louis. Since the technicians seem to be having a very difficult time fixing the network, I thought upper management could meet my other goal. Give the customer an explanation. I'm not pointing fingers at any particular NSP, because frankly I don't have enough fingers to point. Everyone had problems. Yes, even DRA's NOC has fallen down a few times. I'm not asking for perfection, but an explanation when things don't work, while you fix the problem. The Internet is a global cooperative network. If people don't cooperate, the global nature of the network fails. Since your customers may in fact want to use the Internet to communicate globally, problems effect customers globally. When I go to the US Post Office, sometimes there is a sign on the wall that postal service to Timbukto may be delayed because Timbukto's main post office was blown up. I have no idea how many postal customers in Olivette, Missouri send mail to Timbukto. Even though the US Post Service has no control over rebuilding Timbukto's main post office, the US Post Service has discovered it is good customer service to inform their customers why their mail to Timbukto may be delayed. Can't NSPs provide their customers an explanation at least as well as the US Post Office? -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

5 4

Historical perspective
by bmanning＠ISI.EDU 26 Nov '95

26 Nov '95

Perhaps, a quick review of RFC 1273 would be useful to those who are interested in service level reachability. Also note the RTFM and IPPM efforts within IETF to develop frameworks for developing critera for measurement. -- --bill

1 0

Re: outages, quality monitoring, trouble tickets, etc
by Sean Donelan 26 Nov '95

26 Nov '95

>From: Scott Huddle <huddle(a)mci.net> > I consider this list a place for ISPs to discuss general policy and > planning issues that effect all of us. It is a very inappropriate > place to discuss problems with a specific provider. As a general policy I wish all network providers would implement at a minimum a network notification list. Bonus points for a network status WWW page, NetNews, FAX, and PR Newswire distribution. >From: bsimpson(a)morningstar.COM (William Allen Simpson) > I firmly DISAGREE! None of us are particularly interested in hearing > every jot and tittle about every network flap, but _BIG_ ones and their > resolution are important to bring to this list! How else to get a > handle on what the real problems are? How else to help each other > avoid repeating the problem in the future? This is always a tough call. When is news no longer news. When it happens all the time. The slow intermittent problems are the hardest problems to fix. I end up tracking problems from Europe to Australia for customers, so I'm interested in network outages all over the world. On the other hand, I don't (usually) care when Bill's PC is turned off at night. Maybe take a cue from the (failed) NetNews Distribution: or MBONE-sublist mechanism. What to do about backbone telco providers which hate announcing their network problems? Maybe when the NSFNET backbone fractured into multiple backbones, the NSF should have taken a lesson from the breakup of AT&T and established a Network Reliability Council. Any telephone network outage effecting more than 50,000 (now lowered to 15,000 I believe) lines have to be reported to the FCC's NRC. Who cares if you connect to three (plus one) NAPs. In the post-NSFNET era, Internet-wide reliability requires Internet-wide information. Reporting network reliability problems is more important than how many NAPs your network connects. >Has anybody else noticed how hard it is to get trouble tickets these >days? Once upon a time, I just called the NSF NOC, and got a report to >them in real time, so the problem could be fixed quickly. Nowadays, >NOCs seem to want you to send email with 24 or 48 hour turnaround, or go >through 2 layers of service representatives. Pretty hard to send email >to them when their link is down, or go through "regular" support in the >middle of the night! Welcome to the new and improved Internet. More clueless people cal NOCs these days (is it plugged in?) so more caller screening is done. Likewise there more clueless people working in NOCs so more levels before reaching someone who even understands what the problem is. NOC-to-NOC communication has been a long standing Internet problem. But now there are more NOC's, more different ways to contact them, with no common conventions. Even though its out of date, I still keep my Internet Manager's Phonebook published by BBN in 1990. ANS, MCI, and Sprint issued press releases a few months ago about their joint agreement. I don't know how how well their joint agreement is working, but I suspect there will be more such agreements between network operators in the future. As the sheer number of people involved grows, everyone is going to filter their calls, e-mail, etc. If you aren't on the list, you get dumped in the "take care of after hell freezes" pile. In the meantime, keep a stack of business cards and a special rolodex, with the magic names and telephone numbers that get you directly to someone who can understand (and maybe even fix) the problem. Interesting enough, the people usually don't change; but the employers do. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

5 6