- NANOG - lists.nanog.org

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by smd＠clock.org 01 Jul '97

01 Jul '97

randy(a)psg.com (Randy Bush) writes: > > Just out of curiosity, why do many (not all) of the large backbone providers > > establish their face to the web (their corporate webserver) on slow, > > badly positioned machines? > > Because it is done by marketing, not engineering? In my opinion, this alone should reduce the Value Number of any given provider. Personally, flaws or not, I welcome Boardwatch's attempts to come up with a widely-published metric for the Internet. This likely will lead other publications into similar investigations, some of which may well bring the writers and editors of various periodicals into contact with the folks at CAIDA. Also, flaws or not, Boardwatch did do something fantastically clever, and that's examining things on an end-to-end basis, rather than obsessing about details of what's going on between the endpoints. People concerned about the abysmal end-to-end throughput of even modern TCP across much of the present Internet should be rejoicing and helping other journalists develop better and more scientific approaches to categorizing expected versus observed end-to-end performance. The combination of work aimed at measuring the internals of one's network and work that measures the "(un)happiness" of certain classes of applications should be of enormous value to engineers willing to admit that they don't know all the things that affect network performance observed by end-users. The reality, however, is that most American ISPs seem to engage in knee-jerk denial or aggressive posturing whenever there is a suggestion that their network is anything but perfect. I certainly have been in the middle of that kind of thing, so I can hardly claim innocence. Such reactions are pure marketing: we can't admit that maybe there is some way we could improve things or some set of things our network doesn't do well because that would hurt our product image. People who react with marketing-think and who build with marketing-think in the first place deserve to be torn apart by analyses based in marketing-think. While the article in question isn't in my hands yet, based on Mr Rickard's and others' comments on the NANOG list, I think it's safe to guess that this is precisely what the Boardwatch study does. Sean.

3 2

Re: Plethora of UUnet outages and instabilities
by Hank Nussbacher 01 Jul '97

01 Jul '97

At 02:12 PM 6/30/97 -0700, Josh Beck wrote: >Has everyone else been seeing the almost daily UUnet outages recently, >does anyone know what the true causes of these have been? Perhaps this has something to do with it: Glitch slows UUNET to crawl June 23, 1997 Network World If the Internet felt slower than usual last Monday, you were probably a victim of the brownout on UUNET Technologies' network that slowed services in the Northeast. No one is disputing that there was a brownout. But UUNET and Cisco Systems, Inc. - which sold the Internet service provider its routers - are at odds over how it happened. The problem, both sides agree, was a router memory failure. After that, the stories diverge. According to UUNET, the ISP's hubs in Washington, D.C., and Newark, N.J., dropped packets because of a software glitch on its Cisco routers. The hubs are like points-of-presence where UUNET users' traffic is routed to and from the Internet. The routers suffered what the ISP is calling a ``memory leakage.'' Typically, after a router uses a block of memory, it puts the block back into a pool of usable memory. But that is not what happened on UUNET's routers last week. Instead of the memory being usable, it was fragmented, said Alan Taffel, vice president of marketing at UUNET. Large clusters of data that came through the routers could not be handled by the fragmented memory blocks. Once the routers ran out of memory, they automatically reset. UUNET notified Cisco, which sent over a software patch. The software Band-Aid freed up more memory in the routers cache so it would not overload, said Bob Michelet, director of corporate relations at Cisco. The patch is a temporary measure meant to be used until UUNET ups the memory on its routers, he said. UUNET started upgrading the memory on the routers, which Cisco contends was the real culprit. Cisco recommends that its ISP customers use 128M bytes of total memory on their route switch processor boards, Michelet said. UUNET uses 64M-byte memory boards on most, if not all, of its 7,500 routers. UUNET said it never received a recommendation from Cisco concerning memory for its 7,500 routers. ``But clearly, once this event occurred, we discussed the memory issue with Cisco, and we agreed the right course of action would be to upgrade the routing memory to 128M bytesÙ,'' said Jim McManus, vice president of systems engineering at UUNET. Almost every large to midsize ISP uses some Cisco equipment, so should users be concerned? That depends on who answers the question. Cisco claimed UUNET was the only ISP using 64M-byte route switch processor boards. One ISP agrees. Genuity, Inc., Bechtel Enterprises' ISP subsidiary, said it would not run a router with less than 128M bytes of memory. Genuity uses Cisco 7513 routers with fully loaded memory because the routers have to store the routing table for the entire Internet, said Rodney Joffe, chief technology officer at Genuity. Copyright 1997, Network WorldÙ > >Josh Beck jbeck(a)connectnet.com >------------------------------------------------------------------------ >CONNECTNet INS, Inc. Phone: (619)450-0254 Fax: (619)450-3216 >6370 Lusk Blvd., Suite F-208 San Diego, CA 92121 >------------------------------------------------------------------------ > >

3 2

[Fwd: Re: Legal Question]
by Scott Reubelt 01 Jul '97

01 Jul '97

-- Scott F. Reubelt sreubelt(a)whistle.com (http://www.whistle.com)

1 0

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by smd＠clock.org 01 Jul '97

01 Jul '97

lhoward(a)UU.NET writes: > 1. Although a variety of backbones is used, the study does not say > which ones. Also, even though the study does point out the assymetrical > routing of a web transavtion (hot-potato), it doesn't point out that the > traffic being measured is a brief web request (which is dumped to the > web server's backbone ASAP) answered by a long response (10KB in this > case, dumped to the querier's backbone). Ironically, the only person I can remember actively (and publically) questioning the wisdom of hot potato routing for several applications is Curtis Villamizar. While Tony Bates and I had what seemed to be good reasons for doing closest-exit routing (among other routing policies) at the time, and there are strong economic incentives for maintaining closest-exit routing among peers, there are certainly drawbacks. Considering that the Keynote/Boardwatch study purports to help content-providers decide where to locate their content, poor performance attributable to closest-exit routing should have engineers considering the wisdom of that particular routing policy with respect to that particular market segment, depending on the focus of their businesses. The nice thing about the Keynote test is that it might get some of the cleverer people around here thinking of the Internet in a more abstract way than they might be used to. When I think in a more end-to-end fashion about comment #1 above, it seems that a provider who is rated poorly deserves the rating, even though the performance problem may be attributable to a peer's network's congested lines, broken equipment, or whatever. In effect, by taking an end-to-end approach, this test also rates networks' routing policies. If the routing policy drives traffic through a competitor's suboptimal path, and this degrades end-to-end performance, then the provider with that routing policy will receive a poorer rating than otherwise. One of the things that should be thought about is that there may be ways to improve end-to-end goodput by changing one's own network based on what one can infer from tests similar to the Keynote ones (or indeed similar to Vern Paxson's tests), as much as what one can infer from what one's own equipment is doing or seeing. > All of the previous notwithstanding, I would be interested in a better > version of this study. Even more interesting would be to track how > providers do over the course of several studies--who responds well to > backbone congestion? Personally, I would take this comment as a sign that the study and its publication is astoundingly successful. Finally, I would rather see people respond to congestion in a cleverer fashion, i.e. by deploying or pressing for the deployment of better WWW transport technology. There are some people with very clever ideas about caching and lazy proxies, and some clever ideas about changing HTTP in dramatic ways, that deserve some attention, as these ideas are more likely to help people and information meet more quickly in the long run than any amount of tweaking the underlying transport fabric ever will. Sean.

1 0

Re: Plethora of UUnet outages and instabilities
by Geoff Haviland 01 Jul '97

01 Jul '97

RSP4 is supported on 7505 At 09:14 AM 7/1/97 +0100, Neil J. McRae wrote: >On Mon, 30 Jun 1997 18:17:48 -3100 (MST) > Wayne Bouchard <web(a)typo.org> wrote: > >> Thats my point. It often ISN'T easy. For example, say you've got a >> 7505 in one spot and find its getting killed. You can't put an rsp4 in >> a 7505. You've got to replace the router. Anyone got any spare 7507s >> or 7513s lying around? Chances are that if they do, its 2 or 3. > >Why _must_ one use a Cisco for this? > >-- >Neil J. McRae. Alive and Kicking. Domino: In the glow of the night. >neil(a)DOMINO.ORG NetBSD/sparc: 100% SpF (Solaris protection Factor) > Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A> > > > Geoff Haviland - SEM for PacBell - haviland(a)cisco.com - 510 327 3490

1 0

Re: Plethora of UUnet outages and instabilities
by Steve Mansfield 01 Jul '97

01 Jul '97

> From: Steve Mansfield <steve(a)nwnet.net> > > Probably because the CPU that's overloaded is not the primary Cisco > processor, but an RSP board or somesuch. > > Gee, and all this time I've been thinking the RSP _was_ the primary > processor in a Cisco. Silly me. :) Yeah, I realised the stupidity of that right after I posted it....I have already flogged myself.... ;-) What I *meant* was the processor on th HSSI board... Steve Mansfield steve(a)nwnet.net NorthWestNet Network Engineer 425-649-7467

1 0

Legal Question
by Scott Reubelt 01 Jul '97

01 Jul '97

May not be for this alias but I need to ask. I have an ISP that may be guilty of receiving copies of everyone's emails associated with the ISP. What can be done legally? Thanks, Scott -- Scott F. Reubelt sreubelt(a)whistle.com (http://www.whistle.com)

1 0

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by sob＠academ.com 01 Jul '97

01 Jul '97

> Personally, flaws or not, I welcome Boardwatch's attempts > to come up with a widely-published metric for the > Internet. This likely will lead other publications into > similar investigations, some of which may well bring the > writers and editors of various periodicals into contact > with the folks at CAIDA. I agree that it is important that good, scientifically-sound metrics be widely-published. I hope that all such publicity in the future will make reference to the work of CAIDA. From the information available thus far (to me at least), this metric does not appear to be scientifically-sound and the publicity does not reference the work of CAIDA. > Also, flaws or not, Boardwatch did do something > fantastically clever, and that's examining things on an > end-to-end basis, rather than obsessing about details of > what's going on between the endpoints. People concerned > about the abysmal end-to-end throughput of even modern TCP > across much of the present Internet should be rejoicing > and helping other journalists develop better and more > scientific approaches to categorizing expected versus > observed end-to-end performance. I agree that end-to-end measurement is important. As I am certain you are aware, there is work being done to build useful, scientifically-sound ways of measuring end-to-end performance in which the various factors affecting such performance can be described. This will help everyone involved (consumers, engineers and marketers) more effectively understand performance. The Keynote technique does little to break-down these factors. As others have pointed out, this is one of its key flaws. -- Stan | Academ Consulting Services |internet: sob(a)academ.com Olan | For more info on academ, see this |uucp: {mcsun|amdahl}!academ!sob Barber | URL- http://www.academ.com/academ |Opinions expressed are only mine.

1 0

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 01 Jul '97

01 Jul '97

I'll try. Let's talk about them a bit one at a time. > Fine, I will point out a few specific problems with your methodology: > > 1. Although a variety of backbones is used, the study does not say > which ones. Also, even though the study does point out the assymetrical > routing of a web transavtion (hot-potato), it doesn't point out that > the traffic being measured is a brief web request (which is dumped to > the web server's backbone ASAP) answered by a long response (10KB in > this case, dumped to the querier's backbone). > We're trying not to care here. I think the "point of view" entailed is more along the lines of: IF I had MY web server on a particular backbone, either by running a dedicated access link to my office from that backbone, or by taking advantage of co-location, or by actually having that backbone host and manage my web site on their server, what would my web site look like to the Internet universe - the people that are out there downloading pages. I think there is an ongoing attempt by netheads to reduce this to flows between trunk routers. We're looking at a backbone a bit more broadly. It is the cummulative total of your network, your connectivity to other networks, your peering, your people, your coffee pots. We don't want to be drawn down into it much beyond that. If I have a web site, using any of various means, connected or otherwise hosted on YOUR backbone, what will the performance look like to MY audience. Will there be differences/advantages to being on one backbone or another, with regards to the perceived performance or rapidy of the pages appearing. The answer to the latter appears to be yes. > 2. The test used measures the responsiveness of a company's web servers, > which is not necessarily reflective of the response their customers get. > This test specifically measures traffic going "outbound", but suggests > that this information is useful in determining a carrier for "inbound" > traffic. This could be misleading; a web farm will have a lot more > outbound traffic than inbound, and a dial-up only provider will have more > inbound traffic than out. > Yes, it is being read as an indicator from the users perspective in the inverse direction. I suspect there IS a relationship, but we're a little unclear on how closely coupled that is. Certainly we're aware that traffic from a customer to a web site takes a completely different path than the almost always larger flow from the web site to the customer - this is largely the reason we dismiss PING and TRACEROUTE as almost totally useless for our purposes. Do the relative performances we are seeing for backbones translate into relative performances for, say, dialup customers connected to that network? I think we'll find yes here too and I don't really agree with the implication that there is no relationship here. But you are quite correct, that is NOT what we intended to measure. > 3. The results show an average taken over 24 hours, ignoring the important > "time of day" factor. Compuserve, for example, may be dog slow > for three hours per day, and greatly over-provisioned for the rest of the > day. In other words, if over a three-hour period samples took 15 seconds, > but only took 2 seconds for the rest of the day, it would score better > than a provider with a consistent 6-7 second rate. > This is actually a very good point and has caused no end of discussion internally. In fact, I think it is one of the more interesting things we have accomplished here. The press release of course posted average download times. We also graph standard deviations and from my perspective, this is an infinitely more interesting item, though harder to explain. The Internet is a very different place at 2:00 AM than at 2:00 PM, and in fact from moment to moment. There is a slightly larger standard deviation waveform that implies consistency across various metropolitan areas. The standard deviation measurements show us how consistently a network can deliver the same performance from one moment to the next, from one time of day to the next, and to a lesser degree, from one city to the next. We are going so far as to somewhat disingeniously lable this PERFORMANCE UNDER LOAD in an attempt to communicate this without a course in statistics. > 4. The transfer rate of 10KB x 5 is not the same as the transfer rate > of a 50KB file. If one backbone is significantly "burstier" than > another, this could dramatically affect throughput. For instance, a > 10KB file might easily go through a bursty or bouncy backbone in just > a few seconds, while larger files require greater consistency. > True enough. But we applied the same methodology to all backbones equally. My sense of what makes up web pages is that they are rarely a solid 50 KB of anything, but a series of files with a text file, a smallish logo file, a couple of graphics files, etc. We think readers can relate to the 50 KB total, and scaled it so. But we think most are made up of a series of files between 5 and 20 KB. I would think "bursty" networks would be a plus in such an environement. What I think I'm hearing you say is that latency counts. I agree. I think it should, but again, we want to look at a communicable "whole page" concept for the test download. Characterizing just what a "typical" web page is is of course a rather loose business. We're pretty open to suggestions of specifically what the download would be like, but it does need to be reasonably of a least common denominator. > 5. Some companies have more popular web pages than others. Few major > providers hang servers directly off their backbone (whatever that might > mean in this context), but rather have a lobe or two attaching their farm. > Just because a provider's web farm is saturated or busy or slow, does not > indicate that the rest of their backbone is. The web server is operated by and under control of the network, as all other aspects are. The concensus here seems to be that we are primarily measuring web server performance and not the network itself. I'm trying to let you all gel around this as the main objection. In five days I think I can show you that it is an interesting theory, just not so. > > Looking back, some of these errors reflect a misinterpretation of mine > that the results were intended for consumers looking for a connection, as > opposed to a place to host their web server. Most of these points still > apply, however. And people will use the study results this way; if I can > make that mistake, J. Random LUser could, too. This assumes there is no correlation. I'm not sure I'm willing to concede that at this point. But in general, I can say that few of the national backbones emphasize their dialup offering to the home user. Most are pushing web hosting and dedicated access connections to businesses. If J Random LUser wants to shop for a NATIONAL BACKBONE to get his connection to, it does rather invert the concept. You are absolutely correct that it would be better, if that were the objective, to pick a dialup pop on each backbone, and measure to ALL the backbone web sites from a single point and then compare those results. Easy enough, but not really what we're after here. In fact, there is no dialup component to this at all, and I'm guessing dialup users would be more interested in accessibility and busy signals anyway. We do think it is quite interesting that the average delivery speed over the backbone is little, if any, beyond the bandwidth capabilities of the new 56K modems. It might appear that an extraordinary amount of resource is going toward upping the bandwidth to the home, when the perceptual "speediness" of the world wide web may not improve at all until the backbone performance increases. > > Also, the links from the page to the graphs were diappointing. What > units are measured in the "Best Value" chart? What does the price of a > T1 have to do with web hosting? We have rather lumped dedicated access in there wholesale. Web hosting pricing has no standards, it is a chaotic mix with some pricing by hits, some by storage, some by the amount of data passing, some by flat rate. The "basic element" of pricing seems to be the monthly recurring cost of a 1.544 Mbps T-1. The actual formulation is of course in the article. Basically we make the loose assertion that performance is the most important criteria when shopping for a connection. But it is rarely the case that "price is no object." We somewhat gratuitously came up with the 2/3 ratio. We sought a formula where 2/3 of the buying decision was made on performance, and 1/3 on price. What we are referring to as TOTAL PERFORMANCE is a simple multiplication of the average download times by the standard deviation. The average of THIS total performance measure across the 29 backbones was 492. The average monthly price of a T-1 was $2045. So we took each networks total perf and multiplied by 8 in an attempt to approximate a value of 4000. We then add the monthly price of a T-1 to that to derive a VALUE figure. Savvis Communications had the lowest (best) VALUE number. They scored well on performance, but their $1700 monthly price was considerably lower than CompuServe, AT&T WorldNet, UUNET, or even GridNet. We named them the best value on that basis. > > All of the previous notwithstanding, I would be interested in a better > version of this study. Even more interesting would be to track how > providers do over the course of several studies--who responds well to > backbone congestion? We are caught between two missions here. In the first place, we are very receptive to suggested improvements to the study. I'm looking at proving/disproving how much of the "web server" component we are really measuring. And we've taken quite seriously the concept of providing each backbone provider a totally identical "test page" for example. I don't think the latter will seriously move any numbers, but as Forest Gump says "just one less thing...". At the same time, I think it will be intensely interesting to see how the numbers move from one issue of the directory to the next. We will have backbones seeking to score better, but at the same time, they will be under pressure from new Internauts and more web pages added to the Internet daily. Traffic growth appears to be continuing unabated. And the more consistently we measure from issue to issue, the more true those relative numbers will be comparing one measurement period to the next. Jack Rickard =================================================================== Jack Rickard Boardwatch Magazine Editor/Publisher 8500 West Bowles Ave., Ste. 210 jack.rickard(a)boardwatch.com Littleton, CO 80123 www.boardwatch.com Voice: (303)973-6038 ===================================================================

5 4

Re: Plethora of UUnet outages and instabilities
by David Bannister 01 Jul '97

01 Jul '97

7505 with an RSP4 is a supported configuration. It can be ordered as the CISCO7505/4. At 10:34 AM 7/1/97 -0400, Deepak Jain wrote: > >My first question is: Have you ever tried putting an RSP2 in a 7505? It >ships with an RSP, second, have you tried putting an RSP4 in a 7505? It >works too. Cisco isn't real happy about it, and of course I wouldn't >recommend it, but it works. > >Next, if your network is that busy, you honestly don't keep that >kind of spare/upgrade equipment around? > >-Deepak. > >On Mon, 30 Jun 1997, Wayne Bouchard wrote: > >> Thats my point. It often ISN'T easy. For example, say you've got a >> 7505 in one spot and find its getting killed. You can't put an rsp4 in >> a 7505. You've got to replace the router. Anyone got any spare 7507s >> or 7513s lying around? Chances are that if they do, its 2 or 3. >> >> Its actually going to be a bit more problematic to split the load >> between two routers since that can mean notable changes in the >> internal routing or the BGP tables. Not to mention calculating what >> kind of traffic is going to pass BETWEEN the two routers. >> >> Straightforward, yes. Easy, no. >> >> > Why wouldn't upgrading the router's CPU or splitting the load between two >> > separate routers solve the problem? Maybe that's not an overnight >> > solution [unless you have spare RSPs in the POP/colo point] but it seems >> > pretty straight forward to me. >> > >> > -Deepak. >> > >> > On Mon, 30 Jun 1997, Wayne Bouchard wrote: >> > >> > > I know one of the problems that UUNET has been having (and I believe >> > > several other folks as well) is high CPU usage on many of their >> > > routers. (When the ping times to the router itself get high its either >> > > a congested link or, usually, a router thats spinning hard..) However, >> > > thats not a problem you can fix easily or fix overnight. >> > > >> > > > Oh God, NYC1. Look out for that ugly beast... We are unfortunate enough >> > > > to have a connection off of that thing. The NOC is good, if you're up for >> > > > entertainment. Let me count the times I've heard "what's traceroute" on >> > > > the phone with them. Let me tell you about outages where they "forgot to >> > > > take the loopback off the T" for like 6 hours. Or my favorite (once you >> > > > hit a high-level brainwashed engineer) the good old "I don't see *any* >> > > > packet loss, so everything is OK". "Yes, but I'm seeing ping times of >> > > > 500ms to your router, and our line isn't near saturation by a long >> > > > shot"... "I don't see any packet loss", etc... >> > > > >> > > > blech. I would not recommend them to my worst enemy. >> > > > >> > > > Charles >> > > > >> > > > ~~~~~~~~~ ~~~~~~~~~~~ >> > > > Charles Sprickman Internet Channel >> > > > INCH System Administration Team (212)243-5200 >> > > > spork(a)inch.com access(a)inch.com >> > > > >> > > > >> > > > On Mon, 30 Jun 1997, Alex Rubenstein wrote: >> > > > >> > > > > Date: Mon, 30 Jun 1997 18:18:06 -0400 >> > > > > From: Alex Rubenstein <alex(a)nac.net> >> > > > > To: Josh Beck <jbeck(a)connectnet.com>, nanog(a)merit.edu >> > > > > Subject: Re: Plethora of UUnet outages and instabilities >> > > > > >> > > > > >> > > > > All I know is they suck moose nipples, and I disconnected from them last >> > > > > week. They have been having problems in/around cr1.nyc1 for MONTHS now, and >> > > > > they are doing nothing (that I can see) to fix it. Thier first-level >> > > > > support is horrendous. One guy once said to me, "Routers have >> > > > > processors?!". Another hung up when I told him to put "CUSTOMER IS IRATE" >> > > > > on the trouble ticket. >> > > > > >> > > > > There are some GOOD guys over there, but them seem to be disappearing? (are >> > > > > you there, Mr. Hannan?) >> > > > > >> > > > > HELLO, GENUITY (anyone who is looking for a new provider, Genuity is DAMN >> > > > > good. and a DAMN good NOC. Sales guys are a little whacky, tho). >> > > > > >> > > > > >> > > > > >> > > > > At 02:12 PM 6/30/97 -0700, Josh Beck wrote: >> > > > > >Has everyone else been seeing the almost daily UUnet outages recently, >> > > > > >does anyone know what the true causes of these have been? >> > > > > >> > > > > --- >> > > > > >> > > > > "Don't go with a spineless ISP; >> > > > > we have more backbone." >> > > > > >> > > > > Alex Rubenstein -- alex(a)nac.net -- KC2BUO -- www.nac.net >> > > > > net @ccess corporation, 201-983-0725 -- 201-983-0725 >> > > > > >> > > > >> > > > >> > > >> > > >> > > ---------------------------------------------------------------------- >> > > Wayne Bouchard GlobalCenter >> > > web(a)primenet.com >> > > Primenet Network Engineering Internet Solutions for >> > > (602) 416-6422 800-373-2499 x6422 Growing Businesses >> > > FAX: (602) 416-9422 >> > > http://www.primenet.com http://www.globalcenter.net >> > > ---------------------------------------------------------------------- >> > > >> > >> >> >> ---------------------------------------------------------------------- >> Wayne Bouchard GlobalCenter >> web(a)primenet.com >> Primenet Network Engineering Internet Solutions for >> (602) 416-6422 800-373-2499 x6422 Growing Businesses >> FAX: (602) 416-9422 >> http://www.primenet.com http://www.globalcenter.net >> ---------------------------------------------------------------------- >> > > <<<This Space Intentionally Left Blank>>>

1 0