Gene Shklar wrote:

Thanks for the suggestions. We're considering doing a follow-up and perhaps making this a regular feature every 2 or 4 months thereafter. We've received a few suggestions for methodology changes/enhancements, and also several emails so far denouncing our methodology but not explaining why (which is typical of people in many areas -- politics, the environment, economics, whatever -- who disagree emotionally but not intellectually with the conclusions of a study).

My disagreement isn't based on emotion. If you had followed the thread you would have picked up on the point that under _most_ circumstances, the ISP/NSP/IXP's web servers are typically _not_ on the fastest part of their network. They leave this space for revenue generating customers.

The current methodology generally shows how a web site connected to a particular backbone appears to the general internet population of users. The results are intended to be a guide (but not the only one) for helping web sites select or evaluate a collocation, hosting, or access provider.

Your going to need cooperation from these sites to put a resource in their colo space not use their web server. A better way to tackle this would be to solicit the cooperation of one or two of their actual customers who provide a service from that backbone.

Your methodology suggestion would be useful to include because its results would also help end-users select their dial-up ISPs based on the backbone that those ISPs are connected to.

I highly doubt this will really help anyone pick an ISP correctly. It doesn't show the most annoying side of the equation; getting connected. What is the ISP user to modem ratio? What is their customer to bandwidth ration? etc. I doubt the average joe-user will even think or ask this. I don't have an issue with somebody trying to provide QoS info about ISP/NSP etc, but you can't do it just by downloading 56k of data. There is a lot more to it. How do each segments behave with differing packet sizes, etc. Fragmentation will take its toll. I could go on but I should do some real work.

Gene Shklar -------------------------------------------------------- Gene Shklar GeneShklar@keynote.com Keynote Systems, Inc. voice (415) 524-3011 Two West Fifth Avenue fax (415) 524-3099 San Mateo, CA 94402 main # (415) 524-3000 http://www.keynote.com

"A great Internet application experience is all a matter of customer perspective."

---------- From: Peter Cole[SMTP:Peter.Cole@telescan.com] Sent: Friday, June 27, 1997 10:57 AM To: nanog@merit.edu Cc: marketing@keynote.com Subject: RE: Keynote/Boardwatch Internet Backbone Index A better test!!!

I would like to see the test run again with the following change.

...

From each provider test the response time of the other 28 sites and not the providers own web server. Then average the response times for these other 28 web servers and report that average response time from that provider. The providers with good connectivity to the rest of the net should have lower average response time.

P.S. One might also be interested in the top one hundred web sites average response time.

Peter Cole of Telescan, Inc. (281)588-9155 Better computing through lack of sleep.

...

---------- From: Golan Ben-Oni[SMTP:bnite@tremere.ios.com] Sent: Thursday, June 26, 1997 3:53 PM To: nanog@merit.edu Subject: Keynote/Boardwatch Internet Backbone Index

For shits and grins:

http://www.keynote.com/measures/backbones/backbones.html

-Golan

-- --/ Peter E. Giza || Technical Consultant || ADSmart Corporation fone 508.684.3609 || phax 508.684.3618 || page 800.632.1746 || http://www.adsmart.net /--

The following should be attributed to Gene Shklar: Please set for 80 columns.

Thanks for the suggestions. We're considering doing a follow-up and perhaps making this a regular feature every 2 or 4 months thereafter. We've received a few suggestions for methodology changes/enhancements, and also several emails so far denouncing our methodology but not explaining why (which is typical of people in many areas -- politics, the environment, economics, whatever -- who disagree emotionally but not intellectually with the conclusions of a study).

The current methodology generally shows how a web site connected to a particular backbone appears to the general internet population of users. The results are intended to be a guide (but not the only one) for helping web sites select or evaluate a collocation, hosting, or access provider.

Fine, I will point out a few specific problems with your methodology: 1. Although a variety of backbones is used, the study does not say which ones. Also, even though the study does point out the assymetrical routing of a web transavtion (hot-potato), it doesn't point out that the traffic being measured is a brief web request (which is dumped to the web server's backbone ASAP) answered by a long response (10KB in this case, dumped to the querier's backbone). 2. The test used measures the responsiveness of a company's web servers, which is not necessarily reflective of the response their customers get. This test specifically measures traffic going "outbound", but suggests that this information is useful in determining a carrier for "inbound" traffic. This could be misleading; a web farm will have a lot more outbound traffic than inbound, and a dial-up only provider will have more inbound traffic than out. 3. The results show an average taken over 24 hours, ignoring the important "time of day" factor. Compuserve, for example, may be dog slow for three hours per day, and greatly over-provisioned for the rest of the day. In other words, if over a three-hour period samples took 15 seconds, but only took 2 seconds for the rest of the day, it would score better than a provider with a consistent 6-7 second rate. 4. The transfer rate of 10KB x 5 is not the same as the transfer rate of a 50KB file. If one backbone is significantly "burstier" than another, this could dramatically affect throughput. For instance, a 10KB file might easily go through a bursty or bouncy backbone in just a few seconds, while larger files require greater consistency. 5. Some companies have more popular web pages than others. Few major providers hang servers directly off their backbone (whatever that might mean in this context), but rather have a lobe or two attaching their farm. Just because a provider's web farm is saturated or busy or slow, does not indicate that the rest of their backbone is. Looking back, some of these errors reflect a misinterpretation of mine that the results were intended for consumers looking for a connection, as opposed to a place to host their web server. Most of these points still apply, however. And people will use the study results this way; if I can make that mistake, J. Random LUser could, too. Also, the links from the page to the graphs were diappointing. What units are measured in the "Best Value" chart? What does the price of a T1 have to do with web hosting? All of the previous notwithstanding, I would be interested in a better version of this study. Even more interesting would be to track how providers do over the course of several studies--who responds well to backbone congestion?

Gene Shklar GeneShklar@keynote.com Keynote Systems, Inc. voice (415) 524-3011

Lee Lee Howard Internet Systems Engineer (703)208-5231 UUNET High-speed Install lhoward@uu.net Do I speak for UUNET? [NOT IN ANY WAY]

"A great Internet application experience is all a matter of customer perspective."

---------- From: Peter Cole[SMTP:Peter.Cole@telescan.com] Sent: Friday, June 27, 1997 10:57 AM To: nanog@merit.edu Cc: marketing@keynote.com Subject: RE: Keynote/Boardwatch Internet Backbone Index A better test!!!

I would like to see the test run again with the following change.

From each provider test the response time of the other 28 sites and not the providers own web server. Then average the response times for these other 28 web servers and report that average response time from that provider. The providers with good connectivity to the rest of the net should have lower average response time.

P.S. One might also be interested in the top one hundred web sites average response time.

Peter Cole of Telescan, Inc. (281)588-9155 Better computing through lack of sleep.

...

---------- From: Golan Ben-Oni[SMTP:bnite@tremere.ios.com] Sent: Thursday, June 26, 1997 3:53 PM To: nanog@merit.edu Subject: Keynote/Boardwatch Internet Backbone Index

For shits and grins:

http://www.keynote.com/measures/backbones/backbones.html

-Golan