June 1997 - Test - lists.nanog.org

Re: Plethora of UUnet outages and instabilities
by Alex Rubenstein 01 Jul '97

01 Jul '97

All I know is they suck moose nipples, and I disconnected from them last week. They have been having problems in/around cr1.nyc1 for MONTHS now, and they are doing nothing (that I can see) to fix it. Thier first-level support is horrendous. One guy once said to me, "Routers have processors?!". Another hung up when I told him to put "CUSTOMER IS IRATE" on the trouble ticket. There are some GOOD guys over there, but them seem to be disappearing? (are you there, Mr. Hannan?) HELLO, GENUITY (anyone who is looking for a new provider, Genuity is DAMN good. and a DAMN good NOC. Sales guys are a little whacky, tho). At 02:12 PM 6/30/97 -0700, Josh Beck wrote: >Has everyone else been seeing the almost daily UUnet outages recently, >does anyone know what the true causes of these have been? --- "Don't go with a spineless ISP; we have more backbone." Alex Rubenstein -- alex(a)nac.net -- KC2BUO -- www.nac.net net @ccess corporation, 201-983-0725 -- 201-983-0725

15 21

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 01 Jul '97

01 Jul '97

This assumes that you consider web server location and web server performance to NOT be a part of overall network performance. Our view steps back a bit from that. The majority of traffic would appear to be webcentric. From an end users perspective, what does a web site on a specific network look like and how does that compare to a web site on another network? There are ENDLESS variables contributing to that including intercity links, hub architecture, host hardware, host software, peering, connectivity points with other networks, transit agreements, type of routers, ATM switching (or not). All contribute. We think most people notice Internet performance (or lack thereof) while viewing world wide web pages. If we measure such page transits, the results are indicative of the accumulation of ALL of those factors. The web sites chosen were on the network under study, operated by the network, and under their control. They have total control over the hardware and software used, how it is connected to their network, just as they have control over all other aspects of their networks. Does web server performance affect the results? I would hope so. Can we break it down into what is purely web server hardware performance, what is web server software performance, what is NIC card on the web server, what is the impact of the first router the web server is connected to, what is the impact of hub design and the interface between IP routing and ATM switching, what part is the impact of interconnections with other networks, what part is peering, what part is just goofy router games? Uh,,, NO we can't. I would posit that it is only the network engineers at the heart of this that would or should care. I don't know at this point what portion of the equation can be levied on web servers and I don't think anyone else can either. I have held for several years that the performance breakdown is in the "last inch" of the Internet between the drive controller and the disk surface. But in working with Keynote, I generated the broad theory that if that view held true, then by massively averaging measurements across time and geography, we should flatline the Internet. In other words, all results should factor to zero relatively. They didn't. They didn't to a shocking degree. And at this point I am under the broad assumption that server performance doesn't account for all of it, perhaps little of it. But I could be widely wrong on the entire initial assumption. In any event, the networks have total control and responsibility for their own web servers, much as they do for their own network if you define that as something separate from their networks. We measured web page downloads from an end user perspective, and those are the results in aggregate. If it leads to a flurry of web server upgrades, and that moves the numbers, we'll know more than we did. If it leads to a flurry of web server upgrades, and it FAILS to move the numbers, that will tell us something as well. Our broad theory is that nothing is going to improve as long as anything you do doesn't count and is not detectable by anyone anywhere. If a particular network can move their results in any fashion, that is an improvement in the end user experience, however achieved. Warmest Regards; Jack Rickard =================================================================== Jack Rickard Boardwatch Magazine Editor/Publisher 8500 West Bowles Ave., Ste. 210 jack.rickard(a)boardwatch.com Littleton, CO 80123 www.boardwatch.com Voice: (303)973-6038 =================================================================== ---------- > From: Craig A. Huegen <c-huegen(a)quadrunner.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: Peter Cole <Peter.Cole(a)telescan.com>; nanog(a)merit.edu; marketing(a)keynote.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 2:08 PM > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > ==>Not an entirely whacky concept actually. The way hot potato routing works, > ==>this could actually be a "purer" test I suspect of the network internally > ==>and a purer test of connectivity of any network to all others cum Internet. > > The article says that you're measuring backbone provider performance, yet > you're including: > > * carrier's web server location > * carrier's web server performance > > ==>Keynote does a "Top 40" study of 40 popular web sites and I believe they > ==>make the results available on their web site. It is interesting to observe > ==>performance variations of the network as a whole over time. Other than > ==>that, we don't have much interest in it. It is indicative of no specific > ==>network, but of the Internet in general. > > /cah

7 6

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 01 Jul '97

01 Jul '97

I'll try. Let's talk about them a bit one at a time. > Fine, I will point out a few specific problems with your methodology: > > 1. Although a variety of backbones is used, the study does not say > which ones. Also, even though the study does point out the assymetrical > routing of a web transavtion (hot-potato), it doesn't point out that > the traffic being measured is a brief web request (which is dumped to > the web server's backbone ASAP) answered by a long response (10KB in > this case, dumped to the querier's backbone). > We're trying not to care here. I think the "point of view" entailed is more along the lines of: IF I had MY web server on a particular backbone, either by running a dedicated access link to my office from that backbone, or by taking advantage of co-location, or by actually having that backbone host and manage my web site on their server, what would my web site look like to the Internet universe - the people that are out there downloading pages. I think there is an ongoing attempt by netheads to reduce this to flows between trunk routers. We're looking at a backbone a bit more broadly. It is the cummulative total of your network, your connectivity to other networks, your peering, your people, your coffee pots. We don't want to be drawn down into it much beyond that. If I have a web site, using any of various means, connected or otherwise hosted on YOUR backbone, what will the performance look like to MY audience. Will there be differences/advantages to being on one backbone or another, with regards to the perceived performance or rapidy of the pages appearing. The answer to the latter appears to be yes. > 2. The test used measures the responsiveness of a company's web servers, > which is not necessarily reflective of the response their customers get. > This test specifically measures traffic going "outbound", but suggests > that this information is useful in determining a carrier for "inbound" > traffic. This could be misleading; a web farm will have a lot more > outbound traffic than inbound, and a dial-up only provider will have more > inbound traffic than out. > Yes, it is being read as an indicator from the users perspective in the inverse direction. I suspect there IS a relationship, but we're a little unclear on how closely coupled that is. Certainly we're aware that traffic from a customer to a web site takes a completely different path than the almost always larger flow from the web site to the customer - this is largely the reason we dismiss PING and TRACEROUTE as almost totally useless for our purposes. Do the relative performances we are seeing for backbones translate into relative performances for, say, dialup customers connected to that network? I think we'll find yes here too and I don't really agree with the implication that there is no relationship here. But you are quite correct, that is NOT what we intended to measure. > 3. The results show an average taken over 24 hours, ignoring the important > "time of day" factor. Compuserve, for example, may be dog slow > for three hours per day, and greatly over-provisioned for the rest of the > day. In other words, if over a three-hour period samples took 15 seconds, > but only took 2 seconds for the rest of the day, it would score better > than a provider with a consistent 6-7 second rate. > This is actually a very good point and has caused no end of discussion internally. In fact, I think it is one of the more interesting things we have accomplished here. The press release of course posted average download times. We also graph standard deviations and from my perspective, this is an infinitely more interesting item, though harder to explain. The Internet is a very different place at 2:00 AM than at 2:00 PM, and in fact from moment to moment. There is a slightly larger standard deviation waveform that implies consistency across various metropolitan areas. The standard deviation measurements show us how consistently a network can deliver the same performance from one moment to the next, from one time of day to the next, and to a lesser degree, from one city to the next. We are going so far as to somewhat disingeniously lable this PERFORMANCE UNDER LOAD in an attempt to communicate this without a course in statistics. > 4. The transfer rate of 10KB x 5 is not the same as the transfer rate > of a 50KB file. If one backbone is significantly "burstier" than > another, this could dramatically affect throughput. For instance, a > 10KB file might easily go through a bursty or bouncy backbone in just > a few seconds, while larger files require greater consistency. > True enough. But we applied the same methodology to all backbones equally. My sense of what makes up web pages is that they are rarely a solid 50 KB of anything, but a series of files with a text file, a smallish logo file, a couple of graphics files, etc. We think readers can relate to the 50 KB total, and scaled it so. But we think most are made up of a series of files between 5 and 20 KB. I would think "bursty" networks would be a plus in such an environement. What I think I'm hearing you say is that latency counts. I agree. I think it should, but again, we want to look at a communicable "whole page" concept for the test download. Characterizing just what a "typical" web page is is of course a rather loose business. We're pretty open to suggestions of specifically what the download would be like, but it does need to be reasonably of a least common denominator. > 5. Some companies have more popular web pages than others. Few major > providers hang servers directly off their backbone (whatever that might > mean in this context), but rather have a lobe or two attaching their farm. > Just because a provider's web farm is saturated or busy or slow, does not > indicate that the rest of their backbone is. The web server is operated by and under control of the network, as all other aspects are. The concensus here seems to be that we are primarily measuring web server performance and not the network itself. I'm trying to let you all gel around this as the main objection. In five days I think I can show you that it is an interesting theory, just not so. > > Looking back, some of these errors reflect a misinterpretation of mine > that the results were intended for consumers looking for a connection, as > opposed to a place to host their web server. Most of these points still > apply, however. And people will use the study results this way; if I can > make that mistake, J. Random LUser could, too. This assumes there is no correlation. I'm not sure I'm willing to concede that at this point. But in general, I can say that few of the national backbones emphasize their dialup offering to the home user. Most are pushing web hosting and dedicated access connections to businesses. If J Random LUser wants to shop for a NATIONAL BACKBONE to get his connection to, it does rather invert the concept. You are absolutely correct that it would be better, if that were the objective, to pick a dialup pop on each backbone, and measure to ALL the backbone web sites from a single point and then compare those results. Easy enough, but not really what we're after here. In fact, there is no dialup component to this at all, and I'm guessing dialup users would be more interested in accessibility and busy signals anyway. We do think it is quite interesting that the average delivery speed over the backbone is little, if any, beyond the bandwidth capabilities of the new 56K modems. It might appear that an extraordinary amount of resource is going toward upping the bandwidth to the home, when the perceptual "speediness" of the world wide web may not improve at all until the backbone performance increases. > > Also, the links from the page to the graphs were diappointing. What > units are measured in the "Best Value" chart? What does the price of a > T1 have to do with web hosting? We have rather lumped dedicated access in there wholesale. Web hosting pricing has no standards, it is a chaotic mix with some pricing by hits, some by storage, some by the amount of data passing, some by flat rate. The "basic element" of pricing seems to be the monthly recurring cost of a 1.544 Mbps T-1. The actual formulation is of course in the article. Basically we make the loose assertion that performance is the most important criteria when shopping for a connection. But it is rarely the case that "price is no object." We somewhat gratuitously came up with the 2/3 ratio. We sought a formula where 2/3 of the buying decision was made on performance, and 1/3 on price. What we are referring to as TOTAL PERFORMANCE is a simple multiplication of the average download times by the standard deviation. The average of THIS total performance measure across the 29 backbones was 492. The average monthly price of a T-1 was $2045. So we took each networks total perf and multiplied by 8 in an attempt to approximate a value of 4000. We then add the monthly price of a T-1 to that to derive a VALUE figure. Savvis Communications had the lowest (best) VALUE number. They scored well on performance, but their $1700 monthly price was considerably lower than CompuServe, AT&T WorldNet, UUNET, or even GridNet. We named them the best value on that basis. > > All of the previous notwithstanding, I would be interested in a better > version of this study. Even more interesting would be to track how > providers do over the course of several studies--who responds well to > backbone congestion? We are caught between two missions here. In the first place, we are very receptive to suggested improvements to the study. I'm looking at proving/disproving how much of the "web server" component we are really measuring. And we've taken quite seriously the concept of providing each backbone provider a totally identical "test page" for example. I don't think the latter will seriously move any numbers, but as Forest Gump says "just one less thing...". At the same time, I think it will be intensely interesting to see how the numbers move from one issue of the directory to the next. We will have backbones seeking to score better, but at the same time, they will be under pressure from new Internauts and more web pages added to the Internet daily. Traffic growth appears to be continuing unabated. And the more consistently we measure from issue to issue, the more true those relative numbers will be comparing one measurement period to the next. Jack Rickard =================================================================== Jack Rickard Boardwatch Magazine Editor/Publisher 8500 West Bowles Ave., Ste. 210 jack.rickard(a)boardwatch.com Littleton, CO 80123 www.boardwatch.com Voice: (303)973-6038 ===================================================================

5 4

WAN/LAN delay simulation
by Peter E. Giza 01 Jul '97

01 Jul '97

netheads, Sorry to change the current thread, but does anyone know of a piece of freeware that will simulate delays across a WAN/LAN connection? Basically something that would allow a dual-homed workstation act as a router that one can program to do nasty things to packets. I am not looking for random packet generation, rather I want to test the effects of http traffic and the like over a slow and dirty link (no I don't have a link to one of the Boardwatch worst list #%^>). TIA -pete

2 1

The Cidr Report
by Tony Bates 30 Jun '97

30 Jun '97

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better te
by Jack Rickard 30 Jun '97

30 Jun '97

I probably don't agree that it is not a reliable way to draw conclusions about backbone performance. The index has drawn criticism because it points up winners and losers and that's about it. There is no way to do this without criticism, and indeed it should be part of the process. The end user experience will reflect the sum of that, plus dialup variables which we simply do not include in this work. Jack Rickard ---------- > From: Mark Borchers <markb(a)infi.net> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better te > Date: Friday, June 27, 1997 6:09 AM > > Jack, > > I think you've stated for yourself the reason that this Index has > drawn some criticism. > > By it's name it purports to be an index of backbone speed or backbone > congestion or a combination of the above. > > Not only is it not really an index of these items, it could further > be argued that there's not even any reliable way to draw conclusions > about backbone performance from the Keynote measurements. As you > say, there are many other variables. So why not give this thing a > name that reflects it's real identity as an end-to-end test from the > Keynote Perspective Agent locations to various WWW servers. > > > This assumes that you consider web server location and web server > > performance to NOT be a part of overall network performance. Our view > > steps back a bit from that. The majority of traffic would appear to be > > webcentric. From an end users perspective, what does a web site on a > > specific network look like and how does that compare to a web site on > > another network? There are ENDLESS variables contributing to that > > including intercity links, hub architecture, host hardware, host software, > > peering, connectivity points with other networks, transit agreements, type > > of routers, ATM switching (or not). All contribute. We think most people > > notice Internet performance (or lack thereof) while viewing world wide web > > pages.

3 2

Size is not everything
by Sean Donelan 30 Jun '97

30 Jun '97

Reports continue filtering in. After scoring number one in the Keynote/Boardwatch index, Compuserve gets slamed by the Butler Group, UK study. http://www.mmp.co.uk/mmp/informer/netnews/HTM/630n1w.htm Butler Group also continues the tradition of providing little information on their web site which would allow an outside evaluation of their results. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

2 1

Re: Internet Backbone Index
by William Allen Simpson 30 Jun '97

30 Jun '97

Jack, this is a really large list, with a lot of backbiters and pontificators. My usual policy is to collect responses for a day, and then reply to only one or two, rather than trying to answer each message individually. That would have helped avoid 100 messages on this topic. > From: Joe Shaw <jshaw(a)insync.net> > There are enough marketing reports and half witted articles and reports > out their to confuse the average consumer. I'd think that you'd be > ashamed to be part of them, instead of providing a service that's closer > to providing what you intentionally planned to do. > Actually, I'm a bit ashamed of the NANOG response. First of all, however you all might dislike it, our end users' perception of performance _is_ based on web download speeds these days. And the users don't distinguish network load as opposed to server load. Maybe you _think_ the server choices were poor, but as far as I can tell, you don't have enough data to determine that. As to criticizing the methodology, why haven't you (collectively) proposed a better methodology for measuring web access? Boardwatch, from its history, comes from the user download experience. One cannot blame them if they do their best to measure what they see. One could instead offer to help them to update their next article. Second of all, there really are network performance problems. It matters not at all if RSA (a problem I've had this week) has a powerful WWW server, when I cannot get to it reliably because WillowSprings and SanFrancisco are dropping packets like crazy. So far, only one response has noted the current ongoing efforts at ISI, LBL, Merit, NLANR, and others, to develop good network performance measurement techniques and metrics. I urge Boardwatch to help fund them, and to regularly publish the results! How many of you naysayers have actually participated in and help fund the "scientific" studies? Put your money where your mouth is! WSimpson(a)UMich.edu Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32 BSimpson(a)MorningStar.com Key fingerprint = 2E 07 23 03 C5 62 70 D3 59 B1 4F 5E 1D C2 C1 A2

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 30 Jun '97

30 Jun '97

Mr. Black: That is not precisely true. How do you KNOW that the STUDY gives no details, when you can't have possibly SEEN the study YET. You don't think my logic is flawed. You don't know but snippets of what the logic IS. You don't like what you see of a summary sliver of the results from a PRESS RELEASE. Jack Rickard ---------- > From: Ben Black <black(a)zen.cypher.net> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:53 PM > > the "study" gives no details on the systems used, where they were, who > provided net connectivity...far from scientific. the sort of thing one > would expect from marketeers. > > and just because overall system performance affects web downloads, does > that mean web downloads can be used to accurately and meaningfully > measure overall system performance? i think your logic is flawed. > > the net is not a big BBS, in case you haven't heard. > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > > I'm not a marketing droid. But only a moron would think that overall > > performance would NOT affect the download of a web page, which is > > essentially what you are attempting to say. > > > > There are certainly other ways to do it. Approximately Internet User > > Population raised to Internet User Population worth. > > > > Jack RIckard > > > > =================================================================== > > Jack Rickard Boardwatch Magazine > > Editor/Publisher 8500 West Bowles Ave., Ste. 210 > > jack.rickard(a)boardwatch.com Littleton, CO 80123 > > www.boardwatch.com Voice: (303)973-6038 > > =================================================================== > > > > > > ---------- > > > From: Ben Black <black(a)zen.cypher.net> > > > To: Craig A. Huegen <c-huegen(a)quadrunner.com> > > > Cc: Jack Rickard <jack.rickard(a)boardwatch.com>; Peter Cole > > <Peter.Cole(a)telescan.com>; nanog(a)merit.edu; marketing(a)keynote.com > > > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > > > Date: Friday, June 27, 1997 2:28 PM > > > > > > for an exmple of somewhat more complete and better designed benchmarks of > > > > > this type: > > > > > > http://www.inversenet.com/about/backgrounder.html#2 > > > > > > note that they understand the numerous factors that contribute to overall > > > > > performance. only a marketing droid could think downloading 50k worth of > > > > > web pages is somehow an indicator of overall performance. > > > > > > > > > b3n > >

3 2

Plethora of UUnet outages and instabilities
by Josh Beck 30 Jun '97

30 Jun '97

Has everyone else been seeing the almost daily UUnet outages recently, does anyone know what the true causes of these have been? Josh Beck jbeck(a)connectnet.com ------------------------------------------------------------------------ CONNECTNet INS, Inc. Phone: (619)450-0254 Fax: (619)450-3216 6370 Lusk Blvd., Suite F-208 San Diego, CA 92121 ------------------------------------------------------------------------

1 0