They are very useful for like-for-like comparison, for an indication of where your minimum performance levels are probably at, for a quick check that things are working properly and as expected. To determine the exact max effective speed? To test qos policies? To determine whether you are meeting SLA's? Not so much. Joe Leigh Porter wrote:

They are completely unreliable and not to be trusted except for an occasional general indication of speed.

On Dec 23, 2011, at 8:07 AM, Paul Stewart wrote:

In my opinion they are only "somewhat reliable" if they are on your network or very close to your network -we operate one of the speedtest.net sites and for our own eyeball traffic find it to be a "reasonable indicator" of what kind of speeds the customer is getting.

To put it a different way, if a customer is getting 20X1 Internet service and the speedtest shows 17 X 0.8 then case closed - if they are getting a speedtest result of 5 X 0.5 then our helpdesk will take a further look - this is really in rough terms...

Paul

From the consumer viewpoint: No single data point should be extrapolated to infinity, but comparing problematic behavior with "normal" behavior is a standard process across all fields. Speed tests from several locations done regularly give a baseline for performance. Major departure from expected numbers from a set of speed test sites can be regarded as an indicator of local loop problems. Did you know that local loops suffer from backhoe fade? And, DSLAMS fail. In my home office, speed tests are just another useful diagnostic helping to locate problem areas - just like in Paul's example. DSLReports line monitoring service is a similarly useful tool. James R. Cutler james.cutler@consultant.com

Hello, On Fri, Dec 23, 2011 at 4:19 PM, Octavio Alvarez <alvarezp@alvarezp.ods.org> wrote:

On Fri, 23 Dec 2011 01:18:40 -0800, jacob miller <mmzinyi@yahoo.com> wrote:

...

Am having a debate on the results of speed tests sites.

Am interested in knowing the thoughts of different individuals in regards to this.

They are just a measurement, which need to be correctly used and interpreted (that's the difficult part).

Reading bad numbers is not necessarily an indication of a link problem.

Reading "good enough" numbers is only meaningful for the duration of the test.

To me, the big problem is that they don't state all the details of the tests (for example, how exactly to they do the transfer). Geographical location is good, but sometimes not enough. Do they use http, https, ftp or their own JS implementation of whatever weird protocol they though of? How do I know if I'm hitting my firewall, web cache or ALG?

I agree. But one that is fairly clear in what (and how) it tests (but to be fair isn't really a 'speed test') that I've come across is ICSI Netalyzr. It's pretty useful to give a first impression to a tech of what's going on with a link. Take a look at an example report (from a dodgy connection) I dug up: http://netalyzr.icsi.berkeley.edu/restore/id=43ca208a-28820-e88f1efc-a129-4c... More info and examples are at http://netalyzr.icsi.berkeley.edu/ I also think that sometimes having a 'speed test' or similar hosted on a network you are trying to connect to can be useful to find out if a link is congested, or other problems getting from you to that network. An example of this is The BBC's iPlayer diagnostic at http://www.bbc.co.uk/iplayer/diagnostics (think Hulu, but in the UK). It tests to all their CDNs (Akami, Limelight etc) using different streaming methods and gives the results. Only useful as an overview, but a decent first guide nevertheless .

I only use them to get a generic overview of the link.

Heck yes! Alex

On Fri, Dec 23, 2011 at 10:13 AM, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote:

If you want to understand the issue in detail, check out the report from MIT this year, written by Steve Bauer and available at http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurem ents.pdf.

They should have put a date on their paper, including when the measurements were done. It appears to me to have been done sometime after or around June of 2010.

On Fri, 23 Dec 2011 12:16:38 MST, Joel Maslak said:

However, they are susceptible to things like wireless network issues, TCP limitations (one stream vs. many streams), and misconfiguration of devices at the customer location. And the speed test box isn't necessarily configured/speced correctly either.

I have seen some surreal results reported by some of the speed test sites if you have a sufficiently fat pipe. Near as I could tell, every single hop was gigE or better all the way, the speedtest site then tried to apply a correction for the bottleneck it knew about on its local gigE nterface, and basically decided that the *rest* of the path must be near-infinite speed. ;)

On Fri, Dec 23, 2011 at 9:35 PM, Graham Beneke <graham@apolix.co.za> wrote:

That said - people get fixated on the numbers. 80% of the purchased speed on non-CIR services is cause for a complaint.

Our biggest issue is people doing tests to destinations 300+ ms away that only last for a few seconds and then complaining about poor performance. As soon as you mention things like bandwidth delay product the eyes glaze over. Heavy use of lossy WISP access network providers doesn't help.

Or that most ADSL lines have about 20% ATM cell "tax" on them. I did get caught up on a speed test today. I was turning up a GBLX 100Mb circuit. I got the /30 and all the pings were good to the router. I then pinged some known hosts in the Westin (about a block away where GBLX's router was) and saw some not so nice ping times. I then ran a speedtest and only got about 2Mb/s. Come to find out that this was going to be an MPLS path to the company's California office. Since it hadn't been setup fully the router had found some path through it's management network to ping the world through the tester's DSL line on the other side. So, know the path you are testing. -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474

The MIT article is good read, thanks for sharing that. One thing to watch out for is if the last mile provider is the one hosting the speedtest site, that's another variable removed from the equation. In some cases that is a good thing, in others it's not, depending on what you are trying to measure. It's also theoretically possible (and in my opinion not only likely but probably fairly common) for some large residential ISP's to not rate-limit these on-net test sites (either by design or as a side result of at what point in the network they apply the rate limiting), thereby showing much higher results than the end user could ever possibly see in a real world scenario. Also, when using some of the popular public Ookla/speedtest.net sites, their FAQ clearly states that the tests are not suitable for certain connection types like high speed services and non-residential services in general. One good example is Speakeasy's site, which in my personal experience has been the one most commonly used by end users (especially those contacting us about "speed problems"): http://www.speakeasy.net/speedtest/issues.php "Our speed test is tuned to measure residential broadband services up to 20 Mbps over HTTP. It takes a very customized installation to be able to accurately measure up to 100 Mbps over HTTP." -Scott -----Original Message----- From: Frank Bulk [mailto:frnkblk@iname.com] Sent: Sunday, December 25, 2011 8:28 PM To: 'Michael Holstein'; jacob miller Cc: nanog@nanog.org Subject: RE: Speed Test Results We host an Ookla Speedtest server onsite and find it a very reliable means to identify throughput issues. The source of any performance issues may or may not be ours, but if a customer says things are slow we can usually identify whether it's their PC or network (browsing is slow but speed test runs fine) or a local or regional network issue (speed test runs slow). If a customer gets less than 90% of the advertised throughput, we follow up on it. Frank -----Original Message----- From: Michael Holstein [mailto:michael.holstein@csuohio.edu] Sent: Friday, December 23, 2011 1:27 PM To: jacob miller Cc: nanog@nanog.org Subject: Re: Speed Test Results

Am having a debate on the results of speed tests sites.

Am interested in knowing the thoughts of different individuals in regards to this.

They are excellent tools for generating user complaints. (just like the "do traceroute and count the hops" advice from gamer mags of old). (my $0.02) Michael Holstein Cleveland State University

Basically it's a CYA statement on the part of Ookla/speedtest.net, since their test sites are of varying quality. The Radnor, OH test site sometimes can't even properly test a 10mbit SOHO broadband connection, where the Toledo site is consistently able to flood every available bit of capacity on my 50/5 home connection. It's just another tool that needs to be used intelligently. If I'm testing out a new ISP or a new speed level I've never had before, I wouldn't immediately complain if I didn't get the expected result on a public speed test site as it may be something outside of my ISP's control. On the other hand if things start dragging on my home connection or anywhere else that I know I can expect a certain result speedtest.net is usually my first stop. ---------- Sean Harlow sean@seanharlow.info On Dec 25, 2011, at 9:43 PM, Grant Ridder wrote:

Even though the faq's say they are only good for residential usage, i have had no problems with it at school. My college has 2x 100 Mb circuits from TW. When i run speed tests (I use speedtest.net) with the campus empty, i can get around 95Mb up. The bottleneck is the school's 100Mb switches. When the campus is filled (during the week) i can normally get close to 40 Mb down on a test.

-Grant