- Test - lists.nanog.org

has anyone seen this?
by Michael Park 30 Jun '97

30 Jun '97

http://world.compuserve.com/articles/backbone.asp ______ \____/ Michael Park EMail: mpark(a)corp.idt.net \__/ Network Engineer Phone: 201 928 2954 \/ IDT Corporation Pager: 800 225 0256 PIN: 85452

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 29 Jun '97

29 Jun '97

---------- > From: Craig A. Huegen <c-huegen(a)quadrunner.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu; marketing(a)keynote.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:19 PM > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > ==>This assumes that you consider web server location and web server > ==>performance to NOT be a part of overall network performance. Our view > > But insofar as the article goes, the stated intent was to measure BACKBONE > PERFORMANCE, not backbone web server performance. > > There is a HUGE difference between the two. > > ==> I would hope so. Can we break it down into what is purely web server > ==>hardware performance, what is web server software performance, what is NIC > ==>card on the web server, what is the impact of the first router the web > ==>server is connected to, what is the impact of hub design and the interface > ==>between IP routing and ATM switching, what part is the impact of > ==>interconnections with other networks, what part is peering, what part is > ==>just goofy router games? Uh,,, NO we can't. > > You *can*, however, come up with a better methodology to attach to > your stated intent behind the study; or, if you care to leave your > methodology, clear up the misconceptions that your readers will take in. > > ==>results should factor to zero relatively. They didn't. They didn't to a > ==>shocking degree. And at this point I am under the broad assumption that > ==>server performance doesn't account for all of it, perhaps little of it. > ==>But I could be widely wrong on the entire initial assumption. > > I would challenge that assumption that it accounts for little. The > machine the web server is running on, combined with the OS, load average, > and even down to the web server software, probably makes up a very good > portion of any delays you may have seen. It may. It may not. It would seem intrinsic that it would. At this point I probably think it won't. > > How many times do you go to a web site and see "Host contacted... > awaiting response"? When you see that, you have made the network > connection and have given your query. Any time you see that at the > bottom, it's usually indicative of web server delay. (There is a > possibility of packet loss in the initial sent query, but I'd venture to > state that it's a very small percentage of queries made to web servers). > > ==>In any event, the networks have total control and responsibility for their > ==>own web servers, much as they do for their own network if you define that > ==>as something separate from their networks. We measured web page downloads > ==>from an end user perspective, and those are the results in aggregate. If > ==>it leads to a flurry of web server upgrades, and that moves the numbers, > ==>we'll know more than we did. If it leads to a flurry of web server > ==>upgrades, and it FAILS to move the numbers, that will tell us something as > ==>well. > > But again, if I were in the business to provide nationwide network service > for my customers, and provided my web site as a marketing tool (like most > companies out there), I would architect my network so that the customer > comes first. The web site could be used for information about the > business, but isn't A-1 critical to operation of the network. I'd side > with the priori.net folks here in their architecture; that the web server > really shouldn't be put into a pop. > Where you choose to locate a web server, and how you choose to operate it, is rather your own affair. I would certainly put my best face on mine, if web hosting had any portion of my business, or if providing dedicated access to people who hosted theirs were any part of my business. > ==>Our broad theory is that nothing is going to improve as long as anything > ==>you do doesn't count and is not detectable by anyone anywhere. If a > ==>particular network can move their results in any fashion, that is an > ==>improvement in the end user experience, however achieved. > > But the results you publish don't match the study's intent. Actually, they do. Jack Rickard

3 2

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 29 Jun '97

29 Jun '97

This is gibberish George. The measurements were taken from different networks, in 27 different locations. It is not even potentially the case that anyone would be measured within their network solely. It is not optimized for customers getting to their own web server. It is a simulation of the end user world or footprint. If you have a web server on one backbone, how will it look to that user population, as compared to if it were on another backbone. Jack Rickard ---------- > From: George Herbert <gherbert(a)crl.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: Ben Black <black(a)zen.cypher.net>; nanog(a)merit.edu; gherbert(a)crl.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:20 PM > > > From: "Jack Rickard" <jack.rickard(a)boardwatch.com> > >I'm not a marketing droid. But only a moron would think that overall > >performance would NOT affect the download of a web page, which is > >essentially what you are attempting to say. > > Many, many things will affect the download of a web page, > when it's an internal server the primary one being where > that server is relative to the customer dialins, and how > the internal network congestion is handled. > > This particular test would favor sites whose backbone is highly > optimized for their customers getting to their own web server, > and only their own web server. Most servers are put where their > access is best balanced, and for nearly all servers, that's closer > to the outside world than internal customers, because the vast > majority of connections are external. > > This test completely missed any issues related to inter-ISP > connectivity and performance. You could score at the top of the > pack with a ISP which was disconnected from the rest of the > internet during the test period, in fact, a key indicator that > the measurement is nearly worthless as an overall gage. > > -george william herbert > gherbert(a)crl.com > I speak only for myself, and occationally my cats.

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Hank Nussbacher 28 Jun '97

28 Jun '97

At 04:43 PM 6/27/97 -0600, Jack Rickard wrote: >Randy: > >I'm with you to a point. But NONE of the measurements were taken from >Jack's house. None were taken from Boardwatch's office. And in fact, the >measurements were taken from 27 different geographic locations. I think >that is a somewhat critical element here. We failed to come up with a >methodology that could be performed from "Jack's House" without primarily >measuring "Jack's connectivity" more than anything else. Obviously, that >is what I would have liked. Never could come up with an idea on how to do >that effectively. I'm curious - to which ISPs were these 27 sites connected? Can we get a list? Thanks, Hank

1 0

Re: Internet Backbone Index
by Daniel W McRobb 28 Jun '97

28 Jun '97

> Jack, this is a really large list, with a lot of backbiters and > pontificators. My usual policy is to collect responses for a day, and > then reply to only one or two, rather than trying to answer each message > individually. That would have helped avoid 100 messages on this topic. > > > From: Joe Shaw <jshaw(a)insync.net> > > There are enough marketing reports and half witted articles and reports > > out their to confuse the average consumer. I'd think that you'd be > > ashamed to be part of them, instead of providing a service that's closer > > to providing what you intentionally planned to do. > > > Actually, I'm a bit ashamed of the NANOG response. > > First of all, however you all might dislike it, our end users' > perception of performance _is_ based on web download speeds these days. > And the users don't distinguish network load as opposed to server load. Absolutely. However, even just from this perspective, I think the sample set of servers was: 1) too small 2) not what the typical end user sees 3) partly naive 1 is mostly obvious, or at least to me. Even as a casual surfer, I hit 100 or more sites a day (at least that's what my cache says), usually searching for something (I'm still looking for the perfect search site, but alas have had little luck on that front ;-)). 2... I don't think many users surf network provider home pages. Reachability is probably much more important to providers on that front, and servers located near inter-provider connectivity epicenters for high reachability from the rest of the world sometimes suffer on throughput to the end user. That's a reasonable tradeoff for a WWW site that's funded out of your own pocket and sees very low hit rates, uses no push technology or any high traffic generators, etc. 3... one of the assumptions in the summary is highly questionable: 'Keynote decided to measure a backbone provider's own public web server on the assumption that the provider would locate its own server in the best-performing hosting location for that provider.' I think most providers' home pages are not viewed as being in need of high performance. This is not just in terms of location; in general, I think they're given little funding outside of: - content - reliability The hit rates are low; they don't draw much attention (except 'til maybe now, if people really do go about making them better; I'd be a bit surprised at that, since it doesn't really serve the end users, who almost never look at these sites). > Maybe you _think_ the server choices were poor, but as far as I can > tell, you don't have enough data to determine that. As to criticizing > the methodology, why haven't you (collectively) proposed a better > methodology for measuring web access? I would much rather see something closer to what an end user actually sees. For example, take nlanr's (or someone else's) squid cache hit data and use that to drive a weighted measurement of the spots end users are actually surfing. Alone, it doesn't address the issues of where the problems are, but it at least gets much closer to the end user experience. I don't know anyone in my immediate vicinity who has looked at more than 3 (if any) of the WWW pages in the study; they're just not popular sites, even for those of us who work in the business (who presumably have more interest these sites for seeing what the competition is doing ;-)). Of course, you might want to trim out the obvious top sites like playboy.com and penthouse.com, unless you want a nice picture of our net community's demographics by byte (from what I've seen, it looks to be largely teenage males, but I suspect there's a bias by type of content ;-)). I'd still be very hesitant to make any judgement about a provider's performance based on such data, w/o an intimate knowledge of the topology and where things are bad. That's reasonably difficult to do without a reasonably-sized matrix of well-selected measurement points. As of yet, we don't know how well the keynote measurements adrressed this issue. > Boardwatch, from its history, comes from the user download experience. > One cannot blame them if they do their best to measure what they see. > One could instead offer to help them to update their next article. > > Second of all, there really are network performance problems. It > matters not at all if RSA (a problem I've had this week) has a powerful > WWW server, when I cannot get to it reliably because WillowSprings and > SanFrancisco are dropping packets like crazy. Certainly. I don't think anyone's really denying that, are they? (I'll admit to not knowing, having blindly deleted thru most of this thread ;-)). > So far, only one response has noted the current ongoing efforts at ISI, > LBL, Merit, NLANR, and others, to develop good network performance > measurement techniques and metrics. I urge Boardwatch to help fund > them, and to regularly publish the results! > > How many of you naysayers have actually participated in and help fund > the "scientific" studies? Put your money where your mouth is! I've put at least a small fraction of my time into work in nlanr and caida, and I think network performance measurements are a hard nut to crack, especially from the end user perspective. Real visibilty into causes of performance differences is lacking. My fear of measurements like the summary I've seen thus far from the keynote measurements is that consumers can be misled by such studies. From the summary, I think such a study is of questionable value and not something I'd want anyone I'm friendly with to use as a basis for a purchase decision. Even as an end user I give it little value from what I've seen thus far; I don't surf provider WWW sites, I surf elsewhere. I'd be much more interested in a weighted measurement of popular sites, driven from squid cache data or the like. And those of us who also offer Web-hosting services know (intimately) that it's not simple to root out network performance issues from WWW server issues. Yes, it's very possible, but it requires a vantage point that end users don't typically have. I don't really have anything 'bad' to say about the study, really (depending on how you take the above). I just don't find the summary to be of any personal value to me as an end user or as a network engineer. The summary doesn't answer any of my performance questions. Daniel ~~~~~~

1 0

RE: Internet Backbone Index
by Chris A. Icide 28 Jun '97

28 Jun '97

Considering the fact that our web site has not been a large factor in our marketing direction, and the site has received less that 1000 "hits" per week on average, it was and currently is (IMHO) a perfect choice for our application. However, in the last couple of weeks, it's been "bumped" to a 133MHz Pentium with *gasp* a fast ethernet card (as a result of a power supply fan failure in the previous box). Perhaps it was my fault, but I never imagined anyone claiming to have any knowledge whatsoever would attempt to perform a test of "backbone" providers performance based upon said vendors web server(s). I guess they call that "naive". Chris A. Icide ---------- From: Tim Gibson[SMTP:tim@taggnet.skyscape.net] Sent: Friday, June 27, 1997 2:45 AM To: Joseph T. Klein Cc: Chris A. Icide; nanog(a)merit.edu; 'Sean Donelan' Subject: Re: Internet Backbone Index On Thu, 26 Jun 1997, Joseph T. Klein wrote: > Is the NAP.NET web server still a 90MHz Pentium running BSDI? > > Chris A. Icide wrote: > > > Interesting enough, Does it REALLY matter?!?! As long as it ain't some MS product clogging the lan with needless broadcasts and it's big enough to service those that are ignorant to think a national provider's web site *IS* a measure of their backbone/service.... BSDI is a damn fine product!

3 2

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 28 Jun '97

28 Jun '97

It's not embarrassing to me Randy. I'm in my element and having a ball. I haven't heard this much screeching in months. Go girlfriend. Jack Rickard ---------- > From: Randy Bush <randy(a)psg.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Saturday, June 28, 1997 12:00 AM > > >> I want to measure the response of each ISP's marketing department. So I > >> will send a certified letter to their office of record in Deleware, and > >> report how long before I get the receipt back. Maybe I well do this for > >> 30 letters to get better data. > > I'm with you to a point. But NONE of the measurements were taken from > > Jack's house. > > So send the registered letters from 50 places. Same basic conceptual error. > You are not measuring anything like what you claim. > > But you seem not to want to hear this, so I will plonk this whole > embarrassing (to you) thread. Bye now. > > randy

2 1

Re: Internet Backbone Index
by Jack Rickard 28 Jun '97

28 Jun '97

I don't think I'm missing it. I think I'm disagreeing with it in as nice and nonconfrontational a way as I can given the crappy personality I have to work from. Splitting hairs from here to infinity on what "network" means and what the world wide web is departs rather widely from my mission here, so I'm giving it short shrift. If you don't know how ping and traceroute vary from data flows, I can't help much there either. If you want to draw a line of demarcation between a network and its performance, and a web server and its performance, you're free to do so. I just probably won't buy into it. On the actual concept that changing all the web servers will move the numbers: It might. It might not. I would probably bet at this point that there will be a lot of that going on among the non-moron crowd. I'm kind of hoping for it anyway. And then we'll see if the numbers move. My sense is that they will move some, and not as much as most seem to think. But it's true it could go the other way and be dramatic. I'm open to whatever results derive. Jack Rickard ---------- > From: Justin W. Newton <justin(a)priori.net> > To: Jack Rickard <jack.rickard(a)boardwatch.com>; Stan Barber <sob(a)academ.com>; vaden(a)texoma.net; SEAN(a)SDG.DRA.COM; nanog(a)merit.edu > Subject: Re: Internet Backbone Index > Date: Friday, June 27, 1997 2:50 PM > > > Jack, > I believe that you are missing the point that measuring web server > response time is /not/ the equivalent of measuring backbone performance. > > > At 12:45 PM 6/27/97 -0600, Jack Rickard wrote: > >They could be. The attempt is to factor that out. ALL measuring agents > >applied to ALL the backbones. And all contributed more or less equally to > >the end numbers. If a particular agent ran on a Commodore 64 with a kluged > >copy of KA9Q, and another agent ran on an Sun Solaris, both results would > >go into the result pile for all 29 measured networks. The net effect > >would be that the flaw would be in our "footprint" from which the > >measurements were taken. This footprint can only be a rough approximation > >of end user distribution anyway. It would affect absolute values relative > >to zero, but the relative indexes between networks should not be affected. > >Since we're looking at the relative relationship primarily, it wouldn't > >appear important. > > > > > >Jack Rickard > >---------- > >> From: Stan Barber <sob(a)academ.com> > >> To: Justin W. Newton <justin(a)priori.net>; Larry Vaden <vaden(a)texoma.net>; > >Sean Donelan <SEAN(a)SDG.DRA.COM>; nanog(a)merit.edu > >> Subject: Re: Internet Backbone Index > >> Date: Friday, June 27, 1997 1:54 PM > >> > >> Justin writes: > >> > ObAboutTopic: This is possibly the most flawed study on the planet. > >> > Remind me to get a fast web server. (And to think, we were going to > >put > >> > our web server in our office, behind a T-1, instead of in real estate > >near > >> > where the real bandwidth is that could be used for customers.). > >> > >> There are many studies more flawed. Consider some of the studies that > >> the Tobacco Institute has released over the years about the affects of > >> smoking. > >> > >> Concerning Internet performance, there have always been a variety of ways > >> of measuring it. It all depends on what you are really trying to measure. > >> The Keynote study is attempting to measure something to which the average > > > >> Internet user (not engineers) can relate. However, There are also > >clearly > >> the possibility of artifacts in the data because of the testing machine's > > > >> TCP stack or other issues (Vern Paxson has covered these issues at NANOG > >> and IETF meetings over the last few years). Checking their web site, > >their > >> software appears to run on top of the TCP stacks of many systems, so the > >> known artifacts of some of these platforms could be an issue. > >> > >> -- > >> Stan | Academ Consulting Services |internet: sob(a)academ.com > >> Olan | For more info on academ, see this |uucp: > >{mcsun|amdahl}!academ!sob > >> Barber | URL- http://www.academ.com/academ |Opinions expressed are only > >mine. > > > > > > ********************************************************* > Justin W. Newton voice: +1-415-482-2840 > Senior Network Architect fax: +1-415-482-2844 > PRIORI NETWORKS, INC. http://www.priori.net > Director At Large, ISP/C http://www.ispc.org > "The People You Know. The People You Trust." > *********************************************************

3 2

Re: floods
by fehnb＠jerseycape.net 28 Jun '97

28 Jun '97

Looks like the same one. As I recall, he got many, many answers. Some even addressed his question.... >Did this message get lost in a temporal warp about >3 months long or so? I seem to remember a thread >> >> I'm doing a project for school.Can you help?what is the scientific >> principle for floods?Thanks! > > Robert J. Fehn Sr.,President and CEO Senior Systems Engineer, Jersey Cape Information Systems Inc. "Access, Engineering, Microwave Links, Consulting, Design."

2 1

Re: Internet Backbone Index
by Sean Donelan 28 Jun '97

28 Jun '97

Ok, so I said I wasn't going to comment on the methodology, I lied. I wouldn't say the Keynote study is the worst ever. There are some really rotten studies in the fields of psychology and sociology. Since the Boardwatch/Keynote study didn't 'test' DRA Net, I guess I'm one of the few independent, disinterested parties to comment on the study's methods. A problem with the Keynote study is it seems very dependent on the location, type and connections of the testing platforms. Keynote mentions that connections from Dallas and Phoenix were slow to 'every' backbone site. This would indicate some systematic problem with the testing sites. Perhaps the results are even more dependent on the testing systems than the systems under test. There are also problems with outlier data points. For example, elsewhere on the Keynote site, the MCI web site had very fast access from 28 test sites (< 4secs), and very slow access from one test site in philly (> 14secs). Mixing and matching data points, if you left out the one outlier data point, MCI would have been faster than Savvis. So I don't know if the rankings are very meaningful if a single test site can have such a pronounced effect. Unlike a scientific study, there doesn't seem to be enough information to independently reproduce the results, so I'm just going from the bits and pieces I can glean from the Keynote pages, press release, and article. >Concerning Internet performance, there have always been a variety of ways >of measuring it. It all depends on what you are really trying to measure. >The Keynote study is attempting to measure something to which the average >Internet user (not engineers) can relate. However, There are also clearly >the possibility of artifacts in the data because of the testing machine's >TCP stack or other issues (Vern Paxson has covered these issues at NANOG >and IETF meetings over the last few years). Checking their web site, their >software appears to run on top of the TCP stacks of many systems, so the >known artifacts of some of these platforms could be an issue. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

3 2