Speedtest site accuracy [was: Bandwidth issues in the Sprint network]
--- mevans@alphatheory.com wrote: Try using the Java test on DSLReports rather than the Flash based test. I've found it to be much more accurate. I also receive the message about compression being used when I test with the flash test. I think it may be a bug. --------------------------------------- This brings up a PITA point for me. Recently, I have seen a rash of "Speedsite test server at <location> says blah, blah, blah" tickets finally reach me and I am telling everyone they're not an accurate way to measure network performance. I notice that at least some are just sending text in Latin. To other medium-sized eyeball network providers (I'm defining medium size as 50-150K DSL/Cable connections and 50-1500 leased line customers): are you seeing this and what do you tell your customers? scott
We have an test server inside our network that we have customers test again. We tell customers that we can only control our network -- beyond our upstream routers it's best-effort only. That said, if there is a real performance issue upstream we do our best to assist or point the customer in the right direction. Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Scott Weeks Sent: Tuesday, April 08, 2008 1:16 PM To: nanog@merit.edu Subject: Speedtest site accuracy [was: Bandwidth issues in the Sprint network] --- mevans@alphatheory.com wrote: Try using the Java test on DSLReports rather than the Flash based test. I've found it to be much more accurate. I also receive the message about compression being used when I test with the flash test. I think it may be a bug. --------------------------------------- This brings up a PITA point for me. Recently, I have seen a rash of "Speedsite test server at <location> says blah, blah, blah" tickets finally reach me and I am telling everyone they're not an accurate way to measure network performance. I notice that at least some are just sending text in Latin. To other medium-sized eyeball network providers (I'm defining medium size as 50-150K DSL/Cable connections and 50-1500 leased line customers): are you seeing this and what do you tell your customers? scott
On Tue, Apr 8, 2008 at 11:15 AM, Scott Weeks <surfer@mauigateway.com> wrote:
This brings up a PITA point for me. Recently, I have seen a rash of "Speedsite test server at <location> says blah, blah, blah" tickets finally reach me and I am telling everyone they're not an accurate way to measure network performance. I notice that at least some are just sending text in Latin.
To other medium-sized eyeball network providers (I'm defining medium size as 50-150K DSL/Cable connections and 50-1500 leased line customers): are you seeing this and what do you tell your customers?
We tell our customers to make sure to use the test site on our network, which will be quite a bit more accurate than some random location on the internet they might pick. There's no reason it can't be reasonably accurate, if you care to address it. We normally get within a few percent of a given line rate ranging over normal DSL speeds to T1s to DS3s to Fast Ethernet. It's a very easy and user-understandable way to say "Your T1 is installed, there's no errors that we see, you're getting about 1.4mbit on the speed test, have a nice day", or, alternately, "You're getting 95mbit/sec down and only 45mbit/sec up, you probably have a duplex mixmatch on your newly installed colo server". --Doug
Doug Clements wrote:
We tell our customers to make sure to use the test site on our network, which will be quite a bit more accurate than some random location on the internet they might pick.
There's no reason it can't be reasonably accurate, if you care to address it. We normally get within a few percent of a given line rate ranging over normal DSL speeds to T1s to DS3s to Fast Ethernet. It's a very easy and user-understandable way to say "Your T1 is installed, there's no errors that we see, you're getting about 1.4mbit on the speed test, have a nice day", or, alternately, "You're getting 95mbit/sec down and only 45mbit/sec up, you probably have a duplex mixmatch on your newly installed colo server".
--Doug
Regarding speed test software, what are people running, and what do you think of it? It was surprisingly hard to find such stuff the last time I looked - right now I've got the guy who decide$ looking at the same software that http://speedtest.vonage.com appears to be running. Downside is that it requires Java. The stuff we've got (it was here before I was, 5 years ago) is definitely showing it's age - it doesn't even have the capability to do upload speed tests, and it's "quick test" while adequate for 1.5mb DSL's gets really flaky when you toss a 6 or 8mb/sec DSL at it. -- Jeff Shultz
At 04:20 PM 4/8/2008, you wrote:
Doug Clements wrote:
We tell our customers to make sure to use the test site on our network, which will be quite a bit more accurate than some random location on the internet they might pick. There's no reason it can't be reasonably accurate, if you care to address it. We normally get within a few percent of a given line rate ranging over normal DSL speeds to T1s to DS3s to Fast Ethernet. It's a very easy and user-understandable way to say "Your T1 is installed, there's no errors that we see, you're getting about 1.4mbit on the speed test, have a nice day", or, alternately, "You're getting 95mbit/sec down and only 45mbit/sec up, you probably have a duplex mixmatch on your newly installed colo server". --Doug
Regarding speed test software, what are people running, and what do you think of it?
It was surprisingly hard to find such stuff the last time I looked - right now I've got the guy who decide$ looking at the same software that http://speedtest.vonage.com appears to be running. Downside is that it requires Java.
The stuff we've got (it was here before I was, 5 years ago) is definitely showing it's age - it doesn't even have the capability to do upload speed tests, and it's "quick test" while adequate for 1.5mb DSL's gets really flaky when you toss a 6 or 8mb/sec DSL at it.
If you go to Speakeasy.net and run their test, the vendor of theirs has a logo showing (and clickable). This outfit produces nice-looking speed test software. That said, it just reported my Comcast Business account as getting 25Mbps down, and 1.4Mbps up, which is pretty unlikely. Clearing the browser cache alters the displayed speed considerably, so this is a good indication of the usefulness (or lack thereof) of some of this software.
Daniel Senie wrote:
If you go to Speakeasy.net and run their test, the vendor of theirs has a logo showing (and clickable). This outfit produces nice-looking speed test software.
That said, it just reported my Comcast Business account as getting 25Mbps down, and 1.4Mbps up, which is pretty unlikely. Clearing the browser cache alters the displayed speed considerably, so this is a good indication of the usefulness (or lack thereof) of some of this software.
Speakeasy must be good - you're the second person in 5 minutes to recommend them. What I'm looking for is software that we can install locally on our backbone so we can offer our customers an accurate and up-to-date performance measure of their own DSL circuit - which is anywhere from 256/256kb to 1024/6.144Mbs at the moment. We're going fiber-to-the-house over the next 5 years so I expect those numbers will continue to rise to the point that problems outside of our network will definitely be more of a bottleneck than problems inside our network - so I'm looking for a solution that will help us illustrate that point to our customers. -- Jeff Shultz
We used Ookla's solution on our network, they charged a one time fee to add our logo, etc.. it was not expensive: http://www.unwiredltd.com/speedtest.php Peter Kranz Founder/CEO - Unwired Ltd www.UnwiredLtd.com Desk: 510-868-1614 x100 Mobile: 510-207-0000 pkranz@unwiredltd.com -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Jeff Shultz Sent: Tuesday, April 08, 2008 2:49 PM To: NANOG list Subject: Re: Speedtest site accuracy [was: Bandwidth issues in the Sprint network] Daniel Senie wrote:
If you go to Speakeasy.net and run their test, the vendor of theirs has a logo showing (and clickable). This outfit produces nice-looking speed test software.
That said, it just reported my Comcast Business account as getting 25Mbps down, and 1.4Mbps up, which is pretty unlikely. Clearing the browser cache alters the displayed speed considerably, so this is a good indication of the usefulness (or lack thereof) of some of this software.
Speakeasy must be good - you're the second person in 5 minutes to recommend them. What I'm looking for is software that we can install locally on our backbone so we can offer our customers an accurate and up-to-date performance measure of their own DSL circuit - which is anywhere from 256/256kb to 1024/6.144Mbs at the moment. We're going fiber-to-the-house over the next 5 years so I expect those numbers will continue to rise to the point that problems outside of our network will definitely be more of a bottleneck than problems inside our network - so I'm looking for a solution that will help us illustrate that point to our customers. -- Jeff Shultz
Vonage appears to be using Visualware's product: http://www.myspeed.com/ Frank -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Jeff Shultz Sent: Tuesday, April 08, 2008 4:49 PM To: NANOG list Subject: Re: Speedtest site accuracy [was: Bandwidth issues in the Sprint network] Daniel Senie wrote:
If you go to Speakeasy.net and run their test, the vendor of theirs has a logo showing (and clickable). This outfit produces nice-looking speed test software.
That said, it just reported my Comcast Business account as getting 25Mbps down, and 1.4Mbps up, which is pretty unlikely. Clearing the browser cache alters the displayed speed considerably, so this is a good indication of the usefulness (or lack thereof) of some of this software.
Speakeasy must be good - you're the second person in 5 minutes to recommend them. What I'm looking for is software that we can install locally on our backbone so we can offer our customers an accurate and up-to-date performance measure of their own DSL circuit - which is anywhere from 256/256kb to 1024/6.144Mbs at the moment. We're going fiber-to-the-house over the next 5 years so I expect those numbers will continue to rise to the point that problems outside of our network will definitely be more of a bottleneck than problems inside our network - so I'm looking for a solution that will help us illustrate that point to our customers. -- Jeff Shultz
Jeff Shultz wrote:
Regarding speed test software, what are people running, and what do you think of it?
That's a good question and one I was going to ask myself if no one else did. We're currently using the AuditMyPC speedtest: http://www.auditmypc.com/broadband-speed-test.asp I'd like to using the same speedtest app as speedtest.speakeasy.net: http://www.ookla.com/ The AuditMyPC one works ok most of the time. However it freaks out when you introduce packet loss. I've been impressed with the ookla speedtest app. I just haven't convinced the right people to buy a copy. As with any flash or java based speedtest the speeds of the PC affect the outcome just as much as the speed of their connection. For example I can do speedtests from my old laptop (2G P4 w/ 2GB RAM running XP SP2 & 1.5yrs of crud) and the same test with my new laptop (dual-core 2.2G P4 w/ 4GB RAM running XP SP2 fresh from the box), both laptops connected directly to our SP backbone. My old laptop will get 12/5 on a bad day (haven't restarted Windows or FireFox in a few days) whereas my new laptop will pull down 30/20 during out peak times without breaking a sweat. Only a minority of customers will have decent PCs at home. Most probably bought the electronics store special (or Wal-Mart). It has just enough RAM to run Windows. It has the slowest hard drive possible to save $5 in manufacturing costs. It has a bargain basement processor with little to no cache. It's been on the Internet probably without a firewall (so who know how many times it's been compromised). It's been used by users who don't have any expertise in security or Internet safety so they've probably installed WebShots and WeatherBug and every other useless piece of junk out there (Google Desktop), not to mention the drive-by installs made possible by IE. It also came loaded down with a bunch of OEM BS that the user will never use. So we have users with low-end PCs, made nearly worthless by the crap that's on them, trying run Flash or Java speedtest apps. Oh, and I forgot to say that the speedtest site is one the user found out there on the Internet, probably in another country. They've connected their FiOS equipment to an 8-year old LinkSys that tops out at 4Mbps and which also lowers the MTU to 1492 just in case it's connected to a DSL modem doing PPPoX. Would any technical person really be surprised to hear that the user didn't get the full speeds they're paying for? But it's still the provider's fault. The OS and application crud issues could be mitigated by giving the users a boot disk that gives them a basic web browser set to the SP's speedtest server. That would be very hard to do in fact. There are plenty of CD-based Linux distros out there. You still can't easily get around the PC's hardware issues or their network problems created by the users themselves. I don't have a silver bullet for this problem. We have a local speedtest server for our users. It's performance varies. It's upload speeds are horrendous. The download is fairly accurate. I would love to improve this, especially as we roll out our 4/7/12 broadband packages and also begin out FTTx deployment. What I'd like to see is major upstreams deploy iperf servers so that we smaller upstreams can verify our own SP circuit speeds. More than once I've caught an upstream who couldn't deliver what we were paying for. iperf is about as accurate as it gets. Being able to run a lengthy speedtest is very helpful. Trying to put some stress on a 100Mbps link to an upstream with a typical b-band speedtest just doesn't cut it. Justin
On Tue, 8 Apr 2008, Scott Weeks wrote:
To other medium-sized eyeball network providers (I'm defining medium size as 50-150K DSL/Cable connections and 50-1500 leased line customers): are you seeing this and what do you tell your customers?
We're having this big push here in Sweden with something that basically translates into "broadband checkup". It's also web based, and it ended up in the national papers last week, where the newspapers misinterpreted 9.53 megabit/s of TCP thruput on a 10 meg ethernet connection as "barely acceptable" or something of that nature. We're seeing difference in results on the same computer depending on what browser is being used, and other strange results. Yes, it's a basic test and it should be treated that way, unfortunately quite a lot of users expect to get the same number they have purchased, so when they have purchased 8 megabit ADSL, they expect this test to say 8 megabit/s. Industry standard here is that 8/1 is ATM speed, so best results one can expect is approx 6.7 megabit/s of TCP thruput. So yes, this is seen and it's a problem I guess we as an industry have to learn how to handle. Swedish ISPs are adopting fineprint in their ads on expected speed to be seen in this tool, as it seems the users are very keen on using it. What worries me is that people will get dissatisfied with their connection even though there is nothing wrong with it and that they won't get better service elsewhere if they switch ISPs. It's good that there is a test, but since we're a market where 100/100 ethernet connections are fairly prevalent, this test doesn't work properly (75 megabit/s result on a 100/100 was listed in the paper as "not acceptable" which we all understand is unfair). -- Mikael Abrahamsson email: swmike@swm.pp.se
I think the web100 NDT tests are the most accurate. http://netspeed.stanford.edu. you can find out more info about web100 at http://www.web100.org/ Also there is a list of other sites at the bottom of that stanford site.
participants (10)
-
Andrew Matthews -
Daniel Senie -
Doug Clements -
Frank Bulk -
Frank Bulk - iNAME
-
Jeff Shultz -
Justin Shore -
Mikael Abrahamsson -
Peter Kranz -
Scott Weeks