- Test - lists.nanog.org

Re: Keynote/Boardwatch Internet Backbone Index A better te
by Danny McPherson 01 Jul '97

01 Jul '97

let's see .. 300 responses from 75 people pointing out the same ~3 flaws in jack's methodology. 'bout time for a new thread? -danny

1 0

Re: Keynote/Boardwatch Internet Backbone Index A better te
by Jack Rickard 01 Jul '97

01 Jul '97

I probably don't agree that it is not a reliable way to draw conclusions about backbone performance. The index has drawn criticism because it points up winners and losers and that's about it. There is no way to do this without criticism, and indeed it should be part of the process. The end user experience will reflect the sum of that, plus dialup variables which we simply do not include in this work. Jack Rickard ---------- > From: Mark Borchers <markb(a)infi.net> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better te > Date: Friday, June 27, 1997 6:09 AM > > Jack, > > I think you've stated for yourself the reason that this Index has > drawn some criticism. > > By it's name it purports to be an index of backbone speed or backbone > congestion or a combination of the above. > > Not only is it not really an index of these items, it could further > be argued that there's not even any reliable way to draw conclusions > about backbone performance from the Keynote measurements. As you > say, there are many other variables. So why not give this thing a > name that reflects it's real identity as an end-to-end test from the > Keynote Perspective Agent locations to various WWW servers. > > > This assumes that you consider web server location and web server > > performance to NOT be a part of overall network performance. Our view > > steps back a bit from that. The majority of traffic would appear to be > > webcentric. From an end users perspective, what does a web site on a > > specific network look like and how does that compare to a web site on > > another network? There are ENDLESS variables contributing to that > > including intercity links, hub architecture, host hardware, host software, > > peering, connectivity points with other networks, transit agreements, type > > of routers, ATM switching (or not). All contribute. We think most people > > notice Internet performance (or lack thereof) while viewing world wide web > > pages.

3 2

Re: Plethora of UUnet outages and instabilities
by Jeffrey S. Curtis 01 Jul '97

01 Jul '97

Wayne Bouchard <web(a)typo.org> writes: }Thats my point. It often ISN'T easy. For example, say you've got a }7505 in one spot and find its getting killed. You can't put an rsp4 in }a 7505. [...] Eh? CISCO7505/4 Cisco 7505 5-Slot, 1 CyBus, 1RSP4, Single Power Supply, $xx,xxx Jeff -- Jeffrey S. Curtis | Internetwork Manager Argonne National Laboratory | Email: curtis(a)anl.gov 9700 South Cass Avenue, ECT-221 | Voice: 630/252-1789 Argonne, IL 60439 | Fax: 630/252-9689

1 0

Size is not everything
by Sean Donelan 30 Jun '97

30 Jun '97

Reports continue filtering in. After scoring number one in the Keynote/Boardwatch index, Compuserve gets slamed by the Butler Group, UK study. http://www.mmp.co.uk/mmp/informer/netnews/HTM/630n1w.htm Butler Group also continues the tradition of providing little information on their web site which would allow an outside evaluation of their results. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

2 1

Re: Internet Backbone Index
by William Allen Simpson 30 Jun '97

30 Jun '97

Jack, this is a really large list, with a lot of backbiters and pontificators. My usual policy is to collect responses for a day, and then reply to only one or two, rather than trying to answer each message individually. That would have helped avoid 100 messages on this topic. > From: Joe Shaw <jshaw(a)insync.net> > There are enough marketing reports and half witted articles and reports > out their to confuse the average consumer. I'd think that you'd be > ashamed to be part of them, instead of providing a service that's closer > to providing what you intentionally planned to do. > Actually, I'm a bit ashamed of the NANOG response. First of all, however you all might dislike it, our end users' perception of performance _is_ based on web download speeds these days. And the users don't distinguish network load as opposed to server load. Maybe you _think_ the server choices were poor, but as far as I can tell, you don't have enough data to determine that. As to criticizing the methodology, why haven't you (collectively) proposed a better methodology for measuring web access? Boardwatch, from its history, comes from the user download experience. One cannot blame them if they do their best to measure what they see. One could instead offer to help them to update their next article. Second of all, there really are network performance problems. It matters not at all if RSA (a problem I've had this week) has a powerful WWW server, when I cannot get to it reliably because WillowSprings and SanFrancisco are dropping packets like crazy. So far, only one response has noted the current ongoing efforts at ISI, LBL, Merit, NLANR, and others, to develop good network performance measurement techniques and metrics. I urge Boardwatch to help fund them, and to regularly publish the results! How many of you naysayers have actually participated in and help fund the "scientific" studies? Put your money where your mouth is! WSimpson(a)UMich.edu Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32 BSimpson(a)MorningStar.com Key fingerprint = 2E 07 23 03 C5 62 70 D3 59 B1 4F 5E 1D C2 C1 A2

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 30 Jun '97

30 Jun '97

Mr. Black: That is not precisely true. How do you KNOW that the STUDY gives no details, when you can't have possibly SEEN the study YET. You don't think my logic is flawed. You don't know but snippets of what the logic IS. You don't like what you see of a summary sliver of the results from a PRESS RELEASE. Jack Rickard ---------- > From: Ben Black <black(a)zen.cypher.net> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:53 PM > > the "study" gives no details on the systems used, where they were, who > provided net connectivity...far from scientific. the sort of thing one > would expect from marketeers. > > and just because overall system performance affects web downloads, does > that mean web downloads can be used to accurately and meaningfully > measure overall system performance? i think your logic is flawed. > > the net is not a big BBS, in case you haven't heard. > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > > I'm not a marketing droid. But only a moron would think that overall > > performance would NOT affect the download of a web page, which is > > essentially what you are attempting to say. > > > > There are certainly other ways to do it. Approximately Internet User > > Population raised to Internet User Population worth. > > > > Jack RIckard > > > > =================================================================== > > Jack Rickard Boardwatch Magazine > > Editor/Publisher 8500 West Bowles Ave., Ste. 210 > > jack.rickard(a)boardwatch.com Littleton, CO 80123 > > www.boardwatch.com Voice: (303)973-6038 > > =================================================================== > > > > > > ---------- > > > From: Ben Black <black(a)zen.cypher.net> > > > To: Craig A. Huegen <c-huegen(a)quadrunner.com> > > > Cc: Jack Rickard <jack.rickard(a)boardwatch.com>; Peter Cole > > <Peter.Cole(a)telescan.com>; nanog(a)merit.edu; marketing(a)keynote.com > > > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > > > Date: Friday, June 27, 1997 2:28 PM > > > > > > for an exmple of somewhat more complete and better designed benchmarks of > > > > > this type: > > > > > > http://www.inversenet.com/about/backgrounder.html#2 > > > > > > note that they understand the numerous factors that contribute to overall > > > > > performance. only a marketing droid could think downloading 50k worth of > > > > > web pages is somehow an indicator of overall performance. > > > > > > > > > b3n > >

3 2

Plethora of UUnet outages and instabilities
by Josh Beck 30 Jun '97

30 Jun '97

Has everyone else been seeing the almost daily UUnet outages recently, does anyone know what the true causes of these have been? Josh Beck jbeck(a)connectnet.com ------------------------------------------------------------------------ CONNECTNet INS, Inc. Phone: (619)450-0254 Fax: (619)450-3216 6370 Lusk Blvd., Suite F-208 San Diego, CA 92121 ------------------------------------------------------------------------

1 0

has anyone seen this?
by Michael Park 30 Jun '97

30 Jun '97

http://world.compuserve.com/articles/backbone.asp ______ \____/ Michael Park EMail: mpark(a)corp.idt.net \__/ Network Engineer Phone: 201 928 2954 \/ IDT Corporation Pager: 800 225 0256 PIN: 85452

2 1

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 29 Jun '97

29 Jun '97

---------- > From: Craig A. Huegen <c-huegen(a)quadrunner.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: nanog(a)merit.edu; marketing(a)keynote.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:19 PM > > On Fri, 27 Jun 1997, Jack Rickard wrote: > > ==>This assumes that you consider web server location and web server > ==>performance to NOT be a part of overall network performance. Our view > > But insofar as the article goes, the stated intent was to measure BACKBONE > PERFORMANCE, not backbone web server performance. > > There is a HUGE difference between the two. > > ==> I would hope so. Can we break it down into what is purely web server > ==>hardware performance, what is web server software performance, what is NIC > ==>card on the web server, what is the impact of the first router the web > ==>server is connected to, what is the impact of hub design and the interface > ==>between IP routing and ATM switching, what part is the impact of > ==>interconnections with other networks, what part is peering, what part is > ==>just goofy router games? Uh,,, NO we can't. > > You *can*, however, come up with a better methodology to attach to > your stated intent behind the study; or, if you care to leave your > methodology, clear up the misconceptions that your readers will take in. > > ==>results should factor to zero relatively. They didn't. They didn't to a > ==>shocking degree. And at this point I am under the broad assumption that > ==>server performance doesn't account for all of it, perhaps little of it. > ==>But I could be widely wrong on the entire initial assumption. > > I would challenge that assumption that it accounts for little. The > machine the web server is running on, combined with the OS, load average, > and even down to the web server software, probably makes up a very good > portion of any delays you may have seen. It may. It may not. It would seem intrinsic that it would. At this point I probably think it won't. > > How many times do you go to a web site and see "Host contacted... > awaiting response"? When you see that, you have made the network > connection and have given your query. Any time you see that at the > bottom, it's usually indicative of web server delay. (There is a > possibility of packet loss in the initial sent query, but I'd venture to > state that it's a very small percentage of queries made to web servers). > > ==>In any event, the networks have total control and responsibility for their > ==>own web servers, much as they do for their own network if you define that > ==>as something separate from their networks. We measured web page downloads > ==>from an end user perspective, and those are the results in aggregate. If > ==>it leads to a flurry of web server upgrades, and that moves the numbers, > ==>we'll know more than we did. If it leads to a flurry of web server > ==>upgrades, and it FAILS to move the numbers, that will tell us something as > ==>well. > > But again, if I were in the business to provide nationwide network service > for my customers, and provided my web site as a marketing tool (like most > companies out there), I would architect my network so that the customer > comes first. The web site could be used for information about the > business, but isn't A-1 critical to operation of the network. I'd side > with the priori.net folks here in their architecture; that the web server > really shouldn't be put into a pop. > Where you choose to locate a web server, and how you choose to operate it, is rather your own affair. I would certainly put my best face on mine, if web hosting had any portion of my business, or if providing dedicated access to people who hosted theirs were any part of my business. > ==>Our broad theory is that nothing is going to improve as long as anything > ==>you do doesn't count and is not detectable by anyone anywhere. If a > ==>particular network can move their results in any fashion, that is an > ==>improvement in the end user experience, however achieved. > > But the results you publish don't match the study's intent. Actually, they do. Jack Rickard

3 2

Re: Keynote/Boardwatch Internet Backbone Index A better test!!!
by Jack Rickard 29 Jun '97

29 Jun '97

This is gibberish George. The measurements were taken from different networks, in 27 different locations. It is not even potentially the case that anyone would be measured within their network solely. It is not optimized for customers getting to their own web server. It is a simulation of the end user world or footprint. If you have a web server on one backbone, how will it look to that user population, as compared to if it were on another backbone. Jack Rickard ---------- > From: George Herbert <gherbert(a)crl.com> > To: Jack Rickard <jack.rickard(a)boardwatch.com> > Cc: Ben Black <black(a)zen.cypher.net>; nanog(a)merit.edu; gherbert(a)crl.com > Subject: Re: Keynote/Boardwatch Internet Backbone Index A better test!!! > Date: Friday, June 27, 1997 3:20 PM > > > From: "Jack Rickard" <jack.rickard(a)boardwatch.com> > >I'm not a marketing droid. But only a moron would think that overall > >performance would NOT affect the download of a web page, which is > >essentially what you are attempting to say. > > Many, many things will affect the download of a web page, > when it's an internal server the primary one being where > that server is relative to the customer dialins, and how > the internal network congestion is handled. > > This particular test would favor sites whose backbone is highly > optimized for their customers getting to their own web server, > and only their own web server. Most servers are put where their > access is best balanced, and for nearly all servers, that's closer > to the outside world than internal customers, because the vast > majority of connections are external. > > This test completely missed any issues related to inter-ISP > connectivity and performance. You could score at the top of the > pack with a ISP which was disconnected from the rest of the > internet during the test period, in fact, a key indicator that > the measurement is nearly worthless as an overall gage. > > -george william herbert > gherbert(a)crl.com > I speak only for myself, and occationally my cats.

2 1