Sean, Do you have a pointer to the raw data? I couldn't find it on the site. -scott At 04:25 PM 6/27/97 -0500, Sean Donelan wrote:
Ok, so I said I wasn't going to comment on the methodology, I lied. I wouldn't say the Keynote study is the worst ever. There are some really rotten studies in the fields of psychology and sociology.
Since the Boardwatch/Keynote study didn't 'test' DRA Net, I guess I'm one of the few independent, disinterested parties to comment on the study's methods.
A problem with the Keynote study is it seems very dependent on the location, type and connections of the testing platforms. Keynote mentions that connections from Dallas and Phoenix were slow to 'every' backbone site. This would indicate some systematic problem with the testing sites. Perhaps the results are even more dependent on the testing systems than the systems under test. There are also problems with outlier data points. For example, elsewhere on the Keynote site, the MCI web site had very fast access from 28 test sites (< 4secs), and very slow access from one test site in philly (> 14secs). Mixing and matching data points, if you left out the one outlier data point, MCI would have been faster than Savvis. So I don't know if the rankings are very meaningful if a single test site can have such a pronounced effect.
Unlike a scientific study, there doesn't seem to be enough information to independently reproduce the results, so I'm just going from the bits and pieces I can glean from the Keynote pages, press release, and article.
Concerning Internet performance, there have always been a variety of ways of measuring it. It all depends on what you are really trying to measure. The Keynote study is attempting to measure something to which the average Internet user (not engineers) can relate. However, There are also clearly the possibility of artifacts in the data because of the testing machine's TCP stack or other issues (Vern Paxson has covered these issues at NANOG and IETF meetings over the last few years). Checking their web site, their software appears to run on top of the TCP stacks of many systems, so the known artifacts of some of these platforms could be an issue. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation