- NANOG - lists.nanog.org

Size is not everything
by Sean Donelan 30 Jun '97

30 Jun '97

Reports continue filtering in. After scoring number one in the Keynote/Boardwatch index, Compuserve gets slamed by the Butler Group, UK study. http://www.mmp.co.uk/mmp/informer/netnews/HTM/630n1w.htm Butler Group also continues the tradition of providing little information on their web site which would allow an outside evaluation of their results. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

2 1

Re: Internet Backbone Index
by William Allen Simpson 30 Jun '97

30 Jun '97

Jack, this is a really large list, with a lot of backbiters and pontificators. My usual policy is to collect responses for a day, and then reply to only one or two, rather than trying to answer each message individually. That would have helped avoid 100 messages on this topic. > From: Joe Shaw <jshaw(a)insync.net> > There are enough marketing reports and half witted articles and reports > out their to confuse the average consumer. I'd think that you'd be > ashamed to be part of them, instead of providing a service that's closer > to providing what you intentionally planned to do. > Actually, I'm a bit ashamed of the NANOG response. First of all, however you all might dislike it, our end users' perception of performance _is_ based on web download speeds these days. And the users don't distinguish network load as opposed to server load. Maybe you _think_ the server choices were poor, but as far as I can tell, you don't have enough data to determine that. As to criticizing the methodology, why haven't you (collectively) proposed a better methodology for measuring web access? Boardwatch, from its history, comes from the user download experience. One cannot blame them if they do their best to measure what they see. One could instead offer to help them to update their next article. Second of all, there really are network performance problems. It matters not at all if RSA (a problem I've had this week) has a powerful WWW server, when I cannot get to it reliably because WillowSprings and SanFrancisco are dropping packets like crazy. So far, only one response has noted the current ongoing efforts at ISI, LBL, Merit, NLANR, and others, to develop good network performance measurement techniques and metrics. I urge Boardwatch to help fund them, and to regularly publish the results! How many of you naysayers have actually participated in and help fund the "scientific" studies? Put your money where your mouth is! WSimpson(a)UMich.edu Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32 BSimpson(a)MorningStar.com Key fingerprint = 2E 07 23 03 C5 62 70 D3 59 B1 4F 5E 1D C2 C1 A2

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 30 Jun '97

30 Jun '97

Mr. Black: That is not precisely true. How do you KNOW that the STUDY gives no details, when you can't have possibly SEEN the study YET. You don't think my logic is flawed. You don't know but snippets of what the logic IS. You don't like what you see of a summary sliver of the results from a PRESS RELEASE. Jack Rickard ---------- > From: Ben Black <black(a)zen.cypher.net> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:53 PM > > the "study" gives no details on the systems used, where they were, who > provided net connectivity...far from scientific. the sort of thing one > would expect from marketeers. > > and just because overall system performance affects web downloads, does > that mean web downloads can be used to accurately and meaningfully > measure overall system performance? i think your logic is flawed. > > the net is not a big BBS, in case you haven't heard. > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > > I'm not a marketing droid. But only a moron would think that overall > > performance would NOT affect the download of a web page, which is > > essentially what you are attempting to say. > > > > There are certainly other ways to do it. Approximately Internet User > > Population raised to Internet User Population worth. > > > > Jack RIckard > > > > =================================================================== > > Jack Rickard Boardwatch Magazine > > Editor/Publisher 8500 West Bowles Ave., Ste. 210 > > jack.rickard(a)boardwatch.com Littleton, CO 80123 > > www.boardwatch.com Voice: (303)973-6038 > > =================================================================== > > > > > > ---------- > > > From: Ben Black <black(a)zen.cypher.net> > > > To: Craig A. Huegen <c-huegen(a)quadrunner.com> > > > Cc: Jack Rickard <jack.rickard(a)boardwatch.com>; Peter Cole > > <Peter.Cole(a)telescan.com>; nanog(a)merit.edu; marketing(a)keynote.com > > > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > > > Date: Friday, June 27, 1997 2:28 PM > > > > > > for an exmple of somewhat more complete and better designed benchmarks of > > > > > this type: > > > > > > http://www.inversenet.com/about/backgrounder.html#2 > > > > > > note that they understand the numerous factors that contribute to overall > > > > > performance. only a marketing droid could think downloading 50k worth of > > > > > web pages is somehow an indicator of overall performance. > > > > > > > > > b3n > >

3 2

Plethora of UUnet outages and instabilities
by Josh Beck 30 Jun '97

30 Jun '97

Has everyone else been seeing the almost daily UUnet outages recently, does anyone know what the true causes of these have been? Josh Beck jbeck(a)connectnet.com ------------------------------------------------------------------------ CONNECTNet INS, Inc. Phone: (619)450-0254 Fax: (619)450-3216 6370 Lusk Blvd., Suite F-208 San Diego, CA 92121 ------------------------------------------------------------------------

1 0

has anyone seen this?
by Michael Park 30 Jun '97

30 Jun '97

http://world.compuserve.com/articles/backbone.asp ______ \____/ Michael Park EMail: mpark(a)corp.idt.net \__/ Network Engineer Phone: 201 928 2954 \/ IDT Corporation Pager: 800 225 0256 PIN: 85452

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 29 Jun '97

29 Jun '97

---------- > From: Craig A. Huegen <c-huegen(a)quadrunner.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu; marketing(a)keynote.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:19 PM > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > ==>This assumes that you consider web server location and web server > ==>performance to NOT be a part of overall network performance. Our view > > But insofar as the article goes, the stated intent was to measure BACKBONE > PERFORMANCE, not backbone web server performance. > > There is a HUGE difference between the two. > > ==> I would hope so. Can we break it down into what is purely web server > ==>hardware performance, what is web server software performance, what is NIC > ==>card on the web server, what is the impact of the first router the web > ==>server is connected to, what is the impact of hub design and the interface > ==>between IP routing and ATM switching, what part is the impact of > ==>interconnections with other networks, what part is peering, what part is > ==>just goofy router games? Uh,,, NO we can't. > > You *can*, however, come up with a better methodology to attach to > your stated intent behind the study; or, if you care to leave your > methodology, clear up the misconceptions that your readers will take in. > > ==>results should factor to zero relatively. They didn't. They didn't to a > ==>shocking degree. And at this point I am under the broad assumption that > ==>server performance doesn't account for all of it, perhaps little of it. > ==>But I could be widely wrong on the entire initial assumption. > > I would challenge that assumption that it accounts for little. The > machine the web server is running on, combined with the OS, load average, > and even down to the web server software, probably makes up a very good > portion of any delays you may have seen. It may. It may not. It would seem intrinsic that it would. At this point I probably think it won't. > > How many times do you go to a web site and see "Host contacted... > awaiting response"? When you see that, you have made the network > connection and have given your query. Any time you see that at the > bottom, it's usually indicative of web server delay. (There is a > possibility of packet loss in the initial sent query, but I'd venture to > state that it's a very small percentage of queries made to web servers). > > ==>In any event, the networks have total control and responsibility for their > ==>own web servers, much as they do for their own network if you define that > ==>as something separate from their networks. We measured web page downloads > ==>from an end user perspective, and those are the results in aggregate. If > ==>it leads to a flurry of web server upgrades, and that moves the numbers, > ==>we'll know more than we did. If it leads to a flurry of web server > ==>upgrades, and it FAILS to move the numbers, that will tell us something as > ==>well. > > But again, if I were in the business to provide nationwide network service > for my customers, and provided my web site as a marketing tool (like most > companies out there), I would architect my network so that the customer > comes first. The web site could be used for information about the > business, but isn't A-1 critical to operation of the network. I'd side > with the priori.net folks here in their architecture; that the web server > really shouldn't be put into a pop. > Where you choose to locate a web server, and how you choose to operate it, is rather your own affair. I would certainly put my best face on mine, if web hosting had any portion of my business, or if providing dedicated access to people who hosted theirs were any part of my business. > ==>Our broad theory is that nothing is going to improve as long as anything > ==>you do doesn't count and is not detectable by anyone anywhere. If a > ==>particular network can move their results in any fashion, that is an > ==>improvement in the end user experience, however achieved. > > But the results you publish don't match the study's intent. Actually, they do. Jack Rickard

3 2

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 29 Jun '97

29 Jun '97

This is gibberish George. The measurements were taken from different networks, in 27 different locations. It is not even potentially the case that anyone would be measured within their network solely. It is not optimized for customers getting to their own web server. It is a simulation of the end user world or footprint. If you have a web server on one backbone, how will it look to that user population, as compared to if it were on another backbone. Jack Rickard ---------- > From: George Herbert <gherbert(a)crl.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: Ben Black <black(a)zen.cypher.net>; nanog(a)merit.edu; gherbert(a)crl.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:20 PM > > > From: "Jack Rickard" <jack.rickard(a)boardwatch.com> > >I'm not a marketing droid. But only a moron would think that overall > >performance would NOT affect the download of a web page, which is > >essentially what you are attempting to say. > > Many, many things will affect the download of a web page, > when it's an internal server the primary one being where > that server is relative to the customer dialins, and how > the internal network congestion is handled. > > This particular test would favor sites whose backbone is highly > optimized for their customers getting to their own web server, > and only their own web server. Most servers are put where their > access is best balanced, and for nearly all servers, that's closer > to the outside world than internal customers, because the vast > majority of connections are external. > > This test completely missed any issues related to inter-ISP > connectivity and performance. You could score at the top of the > pack with a ISP which was disconnected from the rest of the > internet during the test period, in fact, a key indicator that > the measurement is nearly worthless as an overall gage. > > -george william herbert > gherbert(a)crl.com > I speak only for myself, and occationally my cats.

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Hank Nussbacher 28 Jun '97

28 Jun '97

At 04:43 PM 6/27/97 -0600, Jack Rickard wrote: >Randy: > >I'm with you to a point. But NONE of the measurements were taken from >Jack's house. None were taken from Boardwatch's office. And in fact, the >measurements were taken from 27 different geographic locations. I think >that is a somewhat critical element here. We failed to come up with a >methodology that could be performed from "Jack's House" without primarily >measuring "Jack's connectivity" more than anything else. Obviously, that >is what I would have liked. Never could come up with an idea on how to do >that effectively. I'm curious - to which ISPs were these 27 sites connected? Can we get a list? Thanks, Hank

1 0

Re: Internet Backbone Index
by Daniel W McRobb 28 Jun '97

28 Jun '97

> Jack, this is a really large list, with a lot of backbiters and > pontificators. My usual policy is to collect responses for a day, and > then reply to only one or two, rather than trying to answer each message > individually. That would have helped avoid 100 messages on this topic. > > > From: Joe Shaw <jshaw(a)insync.net> > > There are enough marketing reports and half witted articles and reports > > out their to confuse the average consumer. I'd think that you'd be > > ashamed to be part of them, instead of providing a service that's closer > > to providing what you intentionally planned to do. > > > Actually, I'm a bit ashamed of the NANOG response. > > First of all, however you all might dislike it, our end users' > perception of performance _is_ based on web download speeds these days. > And the users don't distinguish network load as opposed to server load. Absolutely. However, even just from this perspective, I think the sample set of servers was: 1) too small 2) not what the typical end user sees 3) partly naive 1 is mostly obvious, or at least to me. Even as a casual surfer, I hit 100 or more sites a day (at least that's what my cache says), usually searching for something (I'm still looking for the perfect search site, but alas have had little luck on that front ;-)). 2... I don't think many users surf network provider home pages. Reachability is probably much more important to providers on that front, and servers located near inter-provider connectivity epicenters for high reachability from the rest of the world sometimes suffer on throughput to the end user. That's a reasonable tradeoff for a WWW site that's funded out of your own pocket and sees very low hit rates, uses no push technology or any high traffic generators, etc. 3... one of the assumptions in the summary is highly questionable: 'Keynote decided to measure a backbone provider's own public web server on the assumption that the provider would locate its own server in the best-performing hosting location for that provider.' I think most providers' home pages are not viewed as being in need of high performance. This is not just in terms of location; in general, I think they're given little funding outside of: - content - reliability The hit rates are low; they don't draw much attention (except 'til maybe now, if people really do go about making them better; I'd be a bit surprised at that, since it doesn't really serve the end users, who almost never look at these sites). > Maybe you _think_ the server choices were poor, but as far as I can > tell, you don't have enough data to determine that. As to criticizing > the methodology, why haven't you (collectively) proposed a better > methodology for measuring web access? I would much rather see something closer to what an end user actually sees. For example, take nlanr's (or someone else's) squid cache hit data and use that to drive a weighted measurement of the spots end users are actually surfing. Alone, it doesn't address the issues of where the problems are, but it at least gets much closer to the end user experience. I don't know anyone in my immediate vicinity who has looked at more than 3 (if any) of the WWW pages in the study; they're just not popular sites, even for those of us who work in the business (who presumably have more interest these sites for seeing what the competition is doing ;-)). Of course, you might want to trim out the obvious top sites like playboy.com and penthouse.com, unless you want a nice picture of our net community's demographics by byte (from what I've seen, it looks to be largely teenage males, but I suspect there's a bias by type of content ;-)). I'd still be very hesitant to make any judgement about a provider's performance based on such data, w/o an intimate knowledge of the topology and where things are bad. That's reasonably difficult to do without a reasonably-sized matrix of well-selected measurement points. As of yet, we don't know how well the keynote measurements adrressed this issue. > Boardwatch, from its history, comes from the user download experience. > One cannot blame them if they do their best to measure what they see. > One could instead offer to help them to update their next article. > > Second of all, there really are network performance problems. It > matters not at all if RSA (a problem I've had this week) has a powerful > WWW server, when I cannot get to it reliably because WillowSprings and > SanFrancisco are dropping packets like crazy. Certainly. I don't think anyone's really denying that, are they? (I'll admit to not knowing, having blindly deleted thru most of this thread ;-)). > So far, only one response has noted the current ongoing efforts at ISI, > LBL, Merit, NLANR, and others, to develop good network performance > measurement techniques and metrics. I urge Boardwatch to help fund > them, and to regularly publish the results! > > How many of you naysayers have actually participated in and help fund > the "scientific" studies? Put your money where your mouth is! I've put at least a small fraction of my time into work in nlanr and caida, and I think network performance measurements are a hard nut to crack, especially from the end user perspective. Real visibilty into causes of performance differences is lacking. My fear of measurements like the summary I've seen thus far from the keynote measurements is that consumers can be misled by such studies. From the summary, I think such a study is of questionable value and not something I'd want anyone I'm friendly with to use as a basis for a purchase decision. Even as an end user I give it little value from what I've seen thus far; I don't surf provider WWW sites, I surf elsewhere. I'd be much more interested in a weighted measurement of popular sites, driven from squid cache data or the like. And those of us who also offer Web-hosting services know (intimately) that it's not simple to root out network performance issues from WWW server issues. Yes, it's very possible, but it requires a vantage point that end users don't typically have. I don't really have anything 'bad' to say about the study, really (depending on how you take the above). I just don't find the summary to be of any personal value to me as an end user or as a network engineer. The summary doesn't answer any of my performance questions. Daniel ~~~~~~

1 0

RE: Internet Backbone Index
by Chris A. Icide 28 Jun '97

28 Jun '97

Considering the fact that our web site has not been a large factor in our marketing direction, and the site has received less that 1000 "hits" per week on average, it was and currently is (IMHO) a perfect choice for our application. However, in the last couple of weeks, it's been "bumped" to a 133MHz Pentium with *gasp* a fast ethernet card (as a result of a power supply fan failure in the previous box). Perhaps it was my fault, but I never imagined anyone claiming to have any knowledge whatsoever would attempt to perform a test of "backbone" providers performance based upon said vendors web server(s). I guess they call that "naive". Chris A. Icide ---------- From: Tim Gibson[SMTP:tim@taggnet.skyscape.net] Sent: Friday, June 27, 1997 2:45 AM To: Joseph T. Klein Cc: Chris A. Icide; nanog(a)merit.edu; 'Sean Donelan' Subject: Re: Internet Backbone Index On Thu, 26 Jun 1997, Joseph T. Klein wrote: > Is the NAP.NET web server still a 90MHz Pentium running BSDI? > > Chris A. Icide wrote: > > > Interesting enough, Does it REALLY matter?!?! As long as it ain't some MS product clogging the lan with needless broadcasts and it's big enough to service those that are ignorant to think a national provider's web site *IS* a measure of their backbone/service.... BSDI is a damn fine product!

3 2