- NANOG - lists.nanog.org

Re: MAE-East
by Brett D. Watson 23 Oct '95

23 Oct '95

That was precisely the point of my last note. I *want* to get involved in finding a reliable way to measure network performance and I *want* to be proactive at it. I absolutely hate telling customers that "pinging a cisco is not a good indicator..." but like you say "there ain't no other way". Just as ping and traceroute are not good indicators of performance, saying that they aren't good indicators is *not* acceptable to customers paying money for your service. So just for the record, I'm not working for a provider and whining that it's not our fault. I want to find a good way to measure peformance on my network and I'd love to be able to publish some results. Guy Almes contacted me about getting involved and that's just what I want to do. -brett > > A couple quick points: the place to go to for this kind > >of thing should be your direct service provider, who hopefully > >has some means of communicating with upstream and peer providers > >when things are going wrong somehow, or at least might be > >able to give you some additional information. > > Ahem, my success rate at that has been less than stellar. If you guys > (you service providers, not you Sean) are so worried about being asked > questions in public (and short of com-priv, I see no more logical place > for this than NANOG if "local" questions don't work], why don't you > work on schemes to pro-actively publish performance data? Otherwise > you *will* get called to the table based on ping and traceroute data, > there just ain't no other way. Either *you* do it, or others will do it > for you, with the tools they find most appropriate, and all your > whining will eventually be discounted. Be pro-active, or give them > better tools. You *asked* to be in the kitchen, *you* better deliver a > working network, including at the Internet-systemic level and not only > in your own service provider centric environment. I suppose that if > you guys can't figure it out yourself, there is always the opportunity > to get it fixed by someone else. Like, ahem, regulation.

1 0

Re: Initial Route Server Stats for MAE-East
by Brett D. Watson 23 Oct '95

23 Oct '95

I've only been to one IETF but wasn't the IPPM (IP porvider metric) group working on just such a problem? Basically attacking how we measure performance on an internet? Did the group come up with anything? I deal with people every day that *insist* that pinging a router (yes, we're talking Cisco's in this case) is "a good enough indication for me" that "your routers are dropping X% of my packets". Sometimes it turns out to be true and I find the problem. Many times, it's just not so. That begs the question: How do I determine whether or not I have packet loss or delay on my network (and be able to prove it beyond a reasonable doubt)? Does FTPing a 50M file across the network tell me? Does a 10,000 packet ping tell me? My point for interjecting here is to ask: Has anyone come up with *any* way to measure network performance (packet loss, throughput, delay) other than ping and traceroute? I understand Mike's comments on defining *what* we really want to measure and I'm all for trying to help define it. I ask these questions because I'd really like to find answers, not to stir the coals even more. -brett > by all means, ping all you like. please do realize that > routers have other things to do besides answer pings when > they're busy. just because a router doesn't respond to > 15% of its pings doesn't mean that it's dropping 15% of > its packets or that there is a 15% loss on a line. router > load is not router load, process switched packets are not > the same as card-level switched or cache switched. > > there aren't any easy answers. perhaps we should all just > implement "ping" servers in pops ( a 386 running > linux or bsd might suffice). > > Jeff Young > young(a)mci.net > > > > > > > > > On Sat, 21 Oct 1995, Mike O'Dell wrote: > > > > > > > remember that using pings to sample connectivity to a very busy > > > > cisco router is not a very reliable probe for several reasons. > > > > Returning pings is a low-priority task in the first place, and > > > > they are rate-limited, so if the processor is busy processing > > > > lots of BGP updates and several folks are fribbling with it > > > > using ping or SNMP, it is less than clear what they will see. > > > > > > > > -mo > > > > > > > Michael A. Nasto quips: > > > > > OK! Agreed. So then, what would you use? > > > > > > > Have you ever been in a classroom and had a student raise his hand, > > answer every question, ask intelligent questions, etc. just to prove > > to the class how smart he or she is. This is the premise of > > the 'Two Mike's Interchange' above. One says, HEY! I know ping packets > > are a lower priority than everything else in a *CISCO* router > > LOOK AT ME (wave wave). Then another kid in the class quips... > > if PING is not what you would use, give us a better utility. > > > > In fact... EVERYONE ( okay 99.73 percent :-) uses PING. After all > > router LOAD is router LOAD.... and if a few ICMP packets can't get > > back in a subjectly reasonable time.. then DUH.... "da network > > is busy......" BGP updates take bandwidth just like any other packet. > > > > Of course the 0.27 percent, zen routing gods of the universe just > > feel the load and the harmonic BGP update patterns and PING between > > the BGP updates.... for a better answer. > > > > Sorry, I could not resist.... and apologize for the satire. PING!! > > PING!! PING!! > > > > Tim > > > > -- > > +--------------------------------------------------------------------------+ > > | Tim Bass | #include<campfire.h> | > > | Principal Network Systems Engineer | for(beer=100;beer>1;beer++){ | > > | The Silk Road Group, Ltd. | take_one_down(); | > > | | pass_it_around(); | > > | http://www.silkroad.com/ | } | > > | | back_to_work(); /*never reached */ | > > +--------------------------------------------------------------------------+ >

4 3

Re: Asymmetric Routing
by Steven J. Richardson 23 Oct '95

23 Oct '95

Sean: This is misleading at best... >From list-admin(a)merit.edu Fri Oct 20 15:19:42 1995 >From: Sean Doran <smd(a)icp.net> >To: joliveto(a)cwi.net, nanog(a)merit.edu >Subject: Re: Asymmetric Routing >Message-Id: <95Oct20.142237-0400_edt.20696+16(a)chops.icp.net> >Date: Fri, 20 Oct 1995 14:22:35 -0400 >Status: R > >| Analysis of some of our routes shows the potential for asymmetric >| routing...that is...the transmit path will be different than the return path >| for packets. > >Asymmetry has been a reality for quite some time now; in fact >it was happening during the NSFNET backbone service days, >since the information in the PRDB was often in a shambles. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Not surprisingly, the PRDB was as good as the information which was submitted by customers of the NSFNET BNS, though there was a lot of checking done to try to proactively catch mistakes. As someone who was working with/supporting the NSFNET configuration process, I do not know of other customers claiming that "the PRDB was often in a shambles" via phone, email, or third-party reports. > Sean. Steve Richardson, Merit

1 0

Re: MAE-East
by Sean Doran 23 Oct '95

23 Oct '95

Bill - The information you supplied is more helpful for figuring out what you're seeing than other things I have seen. Indicating a traceroute and where you see packet loss, and when you see packet loss is a big first step in diagnosing complaints. A couple quick points: the place to go to for this kind of thing should be your direct service provider, who hopefully has some means of communicating with upstream and peer providers when things are going wrong somehow, or at least might be able to give you some additional information. NANOG and other lists aren't appropriate places to discuss specific problems. OTOH, general problems, such as how big the Internet is and how saturated well-known-things seem to be and what might be done about it, could be OK here. Your numbers are quite reasonable ones, and point to heavy congestion in the area of MAE-EAST+. A good part of that is on the DS3 between MCI and their first MAE-EAST+ router. MCI has done something reasonable about that, viz., putting a second router in place, which I am about to set up a peering with. This should divert a large amount of traffic away from the original DS3, which in turn likely should improve performance between you and PSI. I should also note that Sprint and MCI are in the final phases of negotiating direct point-to-point peerings which will pop up in several areas, and improve connectivity between SprintLink and InternetMCI. You wouldn't be far off in expecting this to become something of a trend, particularly among a small number of the large heavy-traffic providers, for whom the NAP/MAE/FIX concepts and models don't seem to be scaling well in practice. Meanwhile, expect congestion, particularly at NAPs/MAEs/etc. :( | If you want a really good a measure, instead of ping try running NTP | over the links in question. It gives a very accurate measurement! | And I've observed 40% loss peaks there too! NTP is one of two really useful network diagnostic tools, and there is another by Matt Mathis that I hesitate to advertise simply because it is somewhat CPU-unfriendly towards busy Cisco routers, while the former is more useful for long-term information gotten as a side-effect. You might want to investigate the IPPM work, btw, wrt network performance. Finally, wrt MAE-EAST, you might want to look at http://www.mfsdatanet.com/MAE/east.stats.html for an idea of the amount of traffic there. Hint: it's *lots*. Sean.

3 2

Re: Initial Route Server Stats for MAE-East
by Guy T Almes 23 Oct '95

23 Oct '95

Randy, I agree. In fact, at the interim BMWG/IPPM meeting, Vern made a presentation on his work, and it was well received. -- Guy At 07:28 AM 10/23/95 PDT, Randy Bush wrote: >hate to sound egghead, but this stuff has been the subject of non-trivial >academic (yet practical) research, and there is a fair literature base. >hard to drag out refs from here, but one interesting ongoing project is >NPD by Vern Paxson, <vern(a)ee.lbl.gov>, which might serve as a starting >point for a lit search. some evenings it is nice to take a cs break from >all the grotty hacking. >

1 0

Re: Initial Route Server Stats for MAE-East
by randy＠psg.com 23 Oct '95

23 Oct '95

hate to sound egghead, but this stuff has been the subject of non-trivial academic (yet practical) research, and there is a fair literature base. hard to drag out refs from here, but one interesting ongoing project is NPD by Vern Paxson, <vern(a)ee.lbl.gov>, which might serve as a starting point for a lit search. some evenings it is nice to take a cs break from all the grotty hacking.

1 0

Re: Initial Route Server Stats for MAE-East
by Gerald Andrew Winters 23 Oct '95

23 Oct '95

Hello, I can't resist jumping into this disscussion. I think its clear that there is much work to be done on router/IP performance. The question then is what can be done in the meantime, until said work is completed? Here is a script I wrote that will work on cisco routers. The script takes advantage of cisco's enterprise specific MIB. There are 3 SNMP variables yielding CPU busy percentage. The first is CPU busy percentage in the last 5 seconds of scheduler time. The other two are 1 and 5 minute exponentially-decayed moving averages. The CPU busy metrics can yield interesting results, for example %'s above 40% indicate a busy router. I was able to generate some interesting graphs awhile back for our cisco routers. For more information, use the oid's in the script as pointers into the cisco mib. And here is the script for anyone who is interested. --Gerald Winters <gerald(a)merit.edu> #!/usr/local/bin/perl @mib_objs = ("1.3.6.1.4.1.9.2.1.56.0", "1.3.6.1.4.1.9.2.1.57.0", "1.3.6.1.4.1.9.2.1.58.0"); for ($i = $num_values = 0; $i < 3; $i++) { foreach (`/afs/umich.edu/user/g/e/gerald/CISCO/snmpget -h 141.211.128.1 @mib_objs[$i]`) { if (/value:/) { $num_values++; @values[$i] = /\w+\W+\w+\W+(\w+)/; } } } if ($num_values == 3) { print "@values[0] @values[1] @values[2] ".`date`; } else { print "0 0 0 ".`date`; }

1 0

Re: Initial Route Server Stats for MAE-East
by Guy T Almes 23 Oct '95

23 Oct '95

I'd like to second Mike's note, particularly the call for participation. Learning how to characterize IP clouds with carefully defined metrics of delay, flow capacity, and packet loss will be real work. Note, for example, the discussion this weekend of how simple pings to a loaded router don't yield a good measure of packet-forwarding loss. The same could be said for trying to measure round-trip delay to a loaded router. If you'd like to join the IPPM email list, send mail to <ippm-request(a)advanced.org>. Notes expressing an interest in working on specific kinds of issues would be particularly appreciated. We'll also be meeting at the upcoming Dallas IETF. -- Guy At 10:06 PM 10/22/95 -0400, Mike O'Dell wrote: > >the IPPM effort within the Benchmark Methodology Working Group is >working on "IP Performance Metrics" which apply to large IP networks >viewed as the "device-under-test" and not "provider metrics" per se. >(I say this as the both the Co-Area Director and the AD responsible for >the working group.) > >understanding the performance of complex networks IS a very much >unsolved problems. people are thrashing around taking whatever >measurements they can think of with hopes of generating SOME kind of >understanding. I don't blame them at all - there isn't much else to do >right this instant. > >all this means, though, is that folks concerned with the problem need >to participate in the BMWG efforts. there is a lot of work to do here >and with some luck, we may do some science to the point of developing >some understanding about how to measure and characterize these beasts >we've built. > >A lot has been done in characterizing performance of network elements, >so now we need to draw a bigger circle around what we're testing >but still apply significant rigor and good experimental methodology >so the data will be meaningful. > > yours for more consistent confusion, > > -mo >

1 0

Re: MAE-East
by William Allen Simpson 23 Oct '95

23 Oct '95

> From: feldman(a)MFSDatanet.COM (Steve Feldman) > Please send me traceroutes and ping output showing the problems. > Samples when things are working would also be good as a baseline. > > I'd like to see if we can correlate the problems to congestion on > the shared FDDI, or if something's really broken. > You know, it never ceases to amaze me that no matter how much anecdotal evidence is given from however many responsible sources, the answer is "oh, that can't be right, give more data". Even now that we have scads of data, it is declared not to be the right kind. I've posted (to various lists) my traceroute and ping statistics, and each time the conditions were challenged. No, I don't monitor the mae-east link directly, and cannot tell what _kind_ of data is present. But, I POP3 my 150K of email 3-4 times a day over the appended route. My software reports 1-5% retransmission loss during nights, 5-10% weekends, and 20-40% during weekday afternoons. It can take half an hour to move 100KB. MCI and PSI blame each other. But, there's no excuse why they don't peer a little closer to here, since I can point to the few city blocks they are apart! If you want a really good a measure, instead of ping try running NTP over the links in question. It gives a very accurate measurement! And I've observed 40% loss peaks there too! Ping reports the loss to appear between 204.70.1.106 and 192.41.177.245 (the names of the hops in between has changed several times in the past few months without any apparent change in service). Here's a baseline, late Sunday night: Recording Mon Oct 23 03:42:19 1995 Resolving .... 137.175.2.11:33434 1: 35.1.48.52 pm001-03.dialip.mich.net. (3 ms) (3 ms) (3 ms) 2: 35.1.48.48 pm001-aa.mich.net. (184 ms) (244 ms) (181 ms) 3: 35.1.48.2 e-neithernet.c-ccb.umnet.umich.edu. (179 ms) (240 ms) (180 ms) 4: 198.108.3.1 fdd3-0.michnet1.mich.net. (183 ms) (233 ms) (181 ms) 5: 192.203.195.5 cpe2-fddi-1.Chicago.mci.net. (178 ms) (233 ms) (175 ms) 6: 204.70.25.9 border2-hssi2-0.Chicago.mci.net. (186 ms) (239 ms) (181 ms) 7: 204.70.3.81 core-fddi-1.Chicago.mci.net. (186 ms) (234 ms) (194 ms) 8: 204.70.1.94 core1-hssi-2.NorthRoyalton.mci.net. (240 ms) (194 ms) (242 ms) 9: 204.70.1.102 core1-hssi-3.WestOrange.mci.net. (267 ms) (359 ms) (262 ms) 10: 204.70.1.66 core2-aip-4.WestOrange.mci.net. (254 ms) (266 ms) (200 ms) 11: 204.70.1.106 core2-hssi-2.Washington.mci.net. (244 ms) (269 ms) (266 ms) 12: 204.70.74.52 borderx1-fddi-1.Washington.mci.net. (212 ms) (227 ms) (209 ms) 13: 204.70.74.102 mae-east-plusplus.Washington.mci.net. (275 ms) (268 ms) (272 ms) 14: 192.41.177.245 mae-east.psi.net. (268 ms) (230 ms) (205 ms) 15: 38.1.2.11 (233 ms) (267 ms) (273 ms) 16: 38.2.112.6 (277 ms) (274 ms) (308 ms) 17: 137.175.2.11 volitans.MorningStar.Com. (334 ms) (765 ms) (281 ms) traceroute done: normal (Unreachable Port) Bill.Simpson(a)um.cc.umich.edu Key fingerprint = 2E 07 23 03 C5 62 70 D3 59 B1 4F 5E 1D C2 C1 A2

1 0

Traffic Analysis ( was Re: Initial Route Server Stats)
by Tim Bass 23 Oct '95

23 Oct '95

Last one, then i'm dropping of this thread :+ mo hits the nail on the head when he states that detailed performance numbers in complex multihomed interconnected networks are currently unobtainable. In any complex problem with numerous uncertain parameters, variables, and relative metrics all moving on the wire at varying speeds, densities, and with a given broken path 'here or there' at any given moment...... Most practical, day to day network users judge traffic on the net the same way we judge the traffic jams on the highways.... You look out over the overpass and determine the density of the traffic, guess about the status of traffic lights and construction zones, wonder if there was an accident and decide if getting on the ramp is a good idea because, once you do, there is no turning back ( for all practical purposes ). So, you want to know how the traffic is to Washington DC from the suburbs.... one way is to get into your car and drive the round trip to the city and back and then estimate the traffic. Of course, a little Miata moves faster than an semi or Station Wagon with 3 kids and a dog; and that police or fire vehicle gets a higher priority... BUT in general, we know something very simple.... "if it moves slow, it must be slow for some reason" and we generally do not like it. So, here we are, crusin' the network and we want to understand why we can't get to our favorite IP address or IP address of the moment. We PING the thing, knowing full well that it is not Einstein's SuperSpecial Theory of Network Metrics, but at least we can look the the numbers and say.... geezzz... it took a looooong time, must be a lot of traffic. Slow is slow even in zen-metrics. Hey, let's do a TRACEROUTE and guess where the congestion is... While the PHD candidates in EE and CS programs in univs' enjoy digging into into complex network theory, the average day-to-day running the network, answering the phone, or just trying to get out to some site kinda cyberjocky enjoys their only dependable tools, PING and TRACEROUTE. World peace, a perfect universe, harmonic diverity are all interesting topics. So are complex network metrics and and space-time theory.... time travel anyone ? :-) Maybe we need a new Internet Traffic Report Morning Show. Mae-east is hosed today, but will have an new OC12 next week. NET99 is moving over to MCI but having trouble peering with Sprint until further notice. Packets to the convention in the Balkans are being filtered by CIDRites at ICP.... Stay tuned and have a nice day. Now, up to the minute IP security tips sponsored by cisco, your partners in datametrics and virtual resolution protocol. But first, MCI and the ReachOut and Spam Your Friends and Family special offer, after these messages. ................ -- +--------------------------------------------------------------------------+ | Tim Bass | #include<campfire.h> | | Principal Network Systems Engineer | for(beer=100;beer>1;beer++){ | | The Silk Road Group, Ltd. | take_one_down(); | | | pass_it_around(); | | http://www.silkroad.com/ | } | | | back_to_work(); /*never reached */ | +--------------------------------------------------------------------------+

1 0