[as posted to the NANOG mailing list]

http://www.keynote.com/measures/backbones/backbones.html

Sigh...

To measure backbone performance, Keynote downloaded 10,000 bytes from one web server on each backbone network and recorded how long this took, from the time the URL was requested until the bytes were downloaded from the server across the Internet to the measurement location. Downloads were attempted every 15 minutes, 24 hours a day, between April 20 and May 20, 1997, from each of the 27 Keynote Perspective software measurement agents located in major metropolitan areas around the United States. Each complete measurement was multiplied by 5 to produce a result more representative of how long it would actually take a user to download a full web page with all its graphics.

Sorry, you can't just multiply your measurements by 5 and expect to come up with meaningful numbers. If you want to know the time needed to download a full web page then you need to measure that specifically.

Keynote decided to measure a backbone provider's own public web server on the assumption that the provider would locate its own server in the best-performing hosting location for that provider.

Whoa! Bad assumption. I know that in Priori's case, our web server will be located in the WORST performing location because the performance of our web server is not critical to our network's performance. Our office will be connected to our local POP via a T1 whereas a colo customer would be in the same room as a backbone router. I wouldn't be surprised to see similar scenarios elsewhere, especially among the large providers who have built out special web farm colo space with the best connectivity. As always, the shoemaker's children are most likely to be in bare feet. This is an interesting way to measure end-to-end performance but these two flaws make this particular set of data completely meaningless. If you want to measure the service that a backbone provides to its customers then you really need to use a customer server, not the backbone operator's own public server which may be located in-house at their offices. And you need to use real-world example pages that are replicated to each of the servers. And, quite frankly, I don't think that automated data collection really can provide a clear picture of the relative performance of the different backbones. There are too many different things that can happen on the network and it doesn't make sense to average them all together. The methodology doesn't mention whether or not outliers in the data were discarded and what steps they took to identify those outliers. The study as it stands does not accurately reflect the quality of the user's experience and that seems to be the original intent of this study. Keynote touts itself as the "premier supplier of software and services for measuring and managing the responsiveness of web-based applications" and yet they do not even appear to be aware of the tools and methodologies available from the IPMA project http://www.merit.edu/ipma or from CAIDA http://www.caida.org or some of the other end-to-end performance measurement info from sites listed in the "Tools" section at http://www.nlanr.net/INFO/ ******************************************************** Michael Dillon voice: +1-415-482-2840 Senior Systems Architect fax: +1-415-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." ********************************************************