Cable Forum - View Single Post - 50M Speed test sites and why you shouldnt take them for granted

qasdfdsaq · 20-02-2012, 22:57

Let's take a more detailed look at the two speedtests mentioned, Speedtest.net and ThinkBroadband.com, just for fun. Both comparing speedtest results, and actual real-world speed in use.

According to these Speedtest.net results, these two connections should be roughly the same speed.

Yet on Thinkbroadband's tests, one is over three times faster than the other:

Let's take a look at what happens when the connections are put to practical use. For the record, all the following examples are real-world examples of actual things I've used my broadband connection(s) for in the past few hours.

Example 1: Doing an SVN checkout of the source code of OpenWRT, as I need to patch and compile new firmware for one of my routers:

On VM:

Code:

qasdfdsaq@ubuntu:~$ time svn co svn://svn.openwrt.org/openwrt/trunk/ ./trunk/
Checked out revision 30664.

real    0m30.816s
user    0m4.964s
sys    0m5.652s

On BT:

Code:

qasdfdsaq@ubuntu:~$ time svn co svn://svn.openwrt.org/openwrt/trunk/ ./trunk/
Checked out revision 30664.

real    0m22.096s
user    0m4.784s
sys    0m5.484s

And to get an idea of what the downloaded files actually comprise of:

Code:

qasdfdsaq@ubuntu:~$ du -sh --exclude '.svn' ./trunk/
92M    ./trunk/
qasdfdsaq@ubuntu:~$ find ./trunk/* | grep -v '.svn' | wc -l
8263

So, to download a lot of small files where the top speed isn't the limiting factor, the BT connection is 25% faster despite Speedtest.net putting the two connections less than 3% apart.

Example 2: Downloading the latest Ubuntu ISO. Again, something I actually did a few hours ago as I needed a *nix platform to compile on.

Both downloads were started at the same time. Reminder: Speedtest.net thinks the two connections are just about equal, while ThinkBroadband.com thinks one is 3.5x as fast as the other. Which one do you think more accurately reflects the actual performance of my connection, for something I actually did on it in practice - for a purpose other than testing my connection speed?

Example 3: Watching a YouTube video. In the following test I loaded up a YouTube video simultaneously in two browsers, at full quality (1080p) and started them at the same time.

As you can see, one connection has buffered almost 100% of the video, while the other connection is barely keeping up with playback speed. In fact, the latter is already 10 seconds behind despite only being 23 seconds into the video as it had to pause to buffer. In approx. 35 seconds, the "Up to 40mbps" BT line has buffered 220 seconds of the video, while the "Up to 50mbps" VM line has buffered 24 seconds. If you take the Speedtest.net numbers to heart, these connections should only be 3% apart!

So on another attempt, I managed to get the video to play in real time and stay in sync, but barely. Again, the BT line manages to buffer several times more in the same time as the supposedly faster (or same speed) VM connection:

(Click image for full size, multi-screen screenshot showing they were in fact both running simultaneously, side by side)

This is one of the few videos that actually managed to playback in real-time on the VM 50mb connection - though admittedly only on the third attempt. I suspect that that the only reason this worked is because it is such a popular video (as you can tell from the ratings) that YouTube deliberately hosted it on its higher-priority server(s).

You might ask, why does it matter how far ahead it buffers, if it manages to play in real-time? Well the sooner it finishes buffering the whole video the sooner I can seek through it properly and skip all the boring parts. Duh!

So, in all three tests, replicating actual things I've done with my connection this evening, one service comes out miles ahead of the other. Yet Speedtest.net thinks they're about the same, while ThinkBroadband.com thinks one is several times faster.

The only difference? Everything I've done is single-user. And while Speedtest.net might give you a nice, high number, it isn't even remotely close to reflecting actual single-user experience, whereas TBB is a hell of a lot closer to the mark even if you think it's "rubbish" because the numbers are lower.

20-02-2012, 22:57	#52
qasdfdsaq cf.mega poster Join Date: Aug 2004 Posts: 11,207	Re: Speed test sites and why you shouldnt take them for granted Let's take a more detailed look at the two speedtests mentioned, Speedtest.net and ThinkBroadband.com, just for fun. Both comparing speedtest results, and actual real-world speed in use. According to these Speedtest.net results, these two connections should be roughly the same speed. Yet on Thinkbroadband's tests, one is over three times faster than the other: Let's take a look at what happens when the connections are put to practical use. For the record, all the following examples are real-world examples of actual things I've used my broadband connection(s) for in the past few hours. Example 1: Doing an SVN checkout of the source code of OpenWRT, as I need to patch and compile new firmware for one of my routers: On VM: Code: qasdfdsaq@ubuntu:~$ time svn co svn://svn.openwrt.org/openwrt/trunk/ ./trunk/ Checked out revision 30664. real 0m30.816s user 0m4.964s sys 0m5.652s On BT: Code: qasdfdsaq@ubuntu:~$ time svn co svn://svn.openwrt.org/openwrt/trunk/ ./trunk/ Checked out revision 30664. real 0m22.096s user 0m4.784s sys 0m5.484s And to get an idea of what the downloaded files actually comprise of: Code: qasdfdsaq@ubuntu:~$ du -sh --exclude '.svn' ./trunk/ 92M ./trunk/ qasdfdsaq@ubuntu:~$ find ./trunk/* \| grep -v '.svn' \| wc -l 8263 So, to download a lot of small files where the top speed isn't the limiting factor, the BT connection is 25% faster despite Speedtest.net putting the two connections less than 3% apart. Example 2: Downloading the latest Ubuntu ISO. Again, something I actually did a few hours ago as I needed a nix platform to compile on. Both downloads were started at the same time. Reminder: Speedtest.net thinks the two connections are just about equal, while ThinkBroadband.com thinks one is 3.5x as fast as the other. Which one do you think more accurately reflects the actual performance of my connection, for something I actually did on it in practice - for a purpose other than testing my connection speed? Example 3*: Watching a YouTube video. In the following test I loaded up a YouTube video simultaneously in two browsers, at full quality (1080p) and started them at the same time. As you can see, one connection has buffered almost 100% of the video, while the other connection is barely keeping up with playback speed. In fact, the latter is already 10 seconds behind despite only being 23 seconds into the video as it had to pause to buffer. In approx. 35 seconds, the "Up to 40mbps" BT line has buffered 220 seconds of the video, while the "Up to 50mbps" VM line has buffered 24 seconds. If you take the Speedtest.net numbers to heart, these connections should only be 3% apart! So on another attempt, I managed to get the video to play in real time and stay in sync, but barely. Again, the BT line manages to buffer several times more in the same time as the supposedly faster (or same speed) VM connection: (Click image for full size, multi-screen screenshot showing they were in fact both running simultaneously, side by side) This is one of the few videos that actually managed to playback in real-time on the VM 50mb connection - though admittedly only on the third attempt. I suspect that that the only reason this worked is because it is such a popular video (as you can tell from the ratings) that YouTube deliberately hosted it on its higher-priority server(s). You might ask, why does it matter how far ahead it buffers, if it manages to play in real-time? Well the sooner it finishes buffering the whole video the sooner I can seek through it properly and skip all the boring parts. Duh! So, in all three tests, replicating actual things I've done with my connection this evening, one service comes out miles ahead of the other. Yet Speedtest.net thinks they're about the same, while ThinkBroadband.com thinks one is several times faster. The only difference? Everything I've done is single-user. And while Speedtest.net might give you a nice, high number, it isn't even remotely close to reflecting actual single-user experience, whereas TBB is a hell of a lot closer to the mark even if you think it's "rubbish" because the numbers are lower.