How to run speedtest
Speedtest is a great tool.
But speedtest results are often misused or misunderstood. For example, people will run speedtest and get different values on each run. The results seem to be random. Sometimes the values seem to be just plain wrong. Speedtest results can be both random and wrong.
Unless you have access to your ISPs circuit bandwidth tester, then there is no situation in which your speedtest results will be accurate. At a minimum, you will lose the management overhead for your circuit.
If you want consistent, repeatable results, then this is the guide.
Speedtest is not one measurement.
Speedtest uses a variety of measurements and combines these into an estimate of upload and download speeds!
A speedtest cannot be accurate, because it depends on many stochastic variables. However, the results can be consistent, and made more consistent.
Consistency:
To get consistent results, we need to treat speedtest as a science experiment. We need to provide enough information so that someone else with the same set up should get the same results +/- the margin of error.
So, rather than steps, this guide is a series of choices or selections you must decide to get consistent results.
Selection 1: Which Speedtest Tool to use?
There are many speedtest tools. There is no standard for speedtest. So, each vendor's tool uses its own algorithm to estimate the upload and download speeds.
For example, if speedtest is testing a 1Gbps circuit and the result is 900Mbps, then 900Mb of data was not sent, but somewhere between 20Mb and 100Mb. And it was sent as files or packets of various sizes.
Speedtest may not even use the same protocols during a run. It may switch between ICMP, UDP and TCP. The speedtest algorithm will use these different protocols and different file or packet sizes to estimate the upload and download speeds.
Example speedtest tools:
SpeedSmart is web-based and preferred by professionals because it runs on any device without impacting the results
iperf3 is the most consistent. iperf3 requires knowledge for the results to be accurate
Speedtest.net (aka Ookla) is the most well-known. It is a sponsored tool. ISPs advertise to promote their services. Speedtest runs on both smart phones, laptops and MacBooks
Speed.io seems like a good tool, but I don't use it
Avoid because the following may be biased:
Avoid because the following seem to provide inaccurate results:
DSLreports' speedtest is an example of a very cool tool, which sacrifices accuracy for misleading "scientific" looking reports
Bandwidth Place, TestMyNet and SpeedOfMe are also examples of low quality speedtest tools. Lots of glitz, but generally inaccurate. As of 18JAN2020, Bandwidth Place could not measure speeds above 100Mbps.
To be honest, it doesn't matter which of the first four tools you choose. However, never, ever compare results from one tool to another tool. Both are just estimating and the algorithms are different.
The table below shows my results for a single run of speetedt using Wi-Fi on an AC1300 AP with an RSSI of -47Db at my MacBook.
One reason I prefer iperf3 over speedtest is the margin of error doesn't get worse with more samples.
The table below shows 5 runs with Ookla and 5 with iperf3. Ookla ranges from 368 to 525 (157), while the range for iperf3 is much smaller 366-375 (or 9). iperf3 results are more consistent.
Selection 2: How will the measurements be reported?
Speedtest is an estimating tool. Say, you had a tape measure and measured a 12-inch long item. And each time you measured the item, the measuring tape gave you a range of values. Anywhere from 6 inches to 12 inches.
The way to minimize the impact of varying results is to take multiple measurements and then average them. IMO, 3-10 measurements is sufficient. I usually take three measurements.
A speedtest must include latency in ms, and upload and download in Mbps
Selection 3: Which estimates need to be discarded?
Most speedtest tools give no indication the speedtest failed. Speedtests fail for all kinds of reasons. There are some obvious clues that the speedtest measurement is bad:
In the good old days, we had asymmetric circuits (download was faster than upload). Now, most upload and download rates are the same. So, if the upload and download differ from each other by more than 10% then the measurement is bad and should be discarded. As an example, in the table above, the 368.62 and 469.90 is a bad measurement
Say you are averaging three speedtests, and two are similar and the third is very different. Throw that estimate out and rerun speedtest. I usually set this threshold at 20%
DSL and Coax circuits will give higher bandwidth measurements during periods of low usage than they will during periods of high usage.
The ping or latency on speedtest is an excellent predictor of consistent results. A latency in the 5-10ms range is acceptable. But above 10ms and the results will start to vary
Decide your criteria for throwing out bad measurements
Selection 4: When will speedtest be run?
Speedtests need to be run when no one and no device is using the internet. Any usage will subtract from the bandwidth being measured. I usually wake early, or run speedtest after everyone has gone to sleep. I check the network traffic and shutdown any device that is using bandwidth.
Selection 5: How often to restart browser?
Many speedtest tools are browser-based. After 5-10 speedtests, the measurement may become less consistent. One way to correct this inconsistency is to shut down the browser, and restart the browser. I usually restart the browser after 5-6 speedtests.
This is not an issue for iperf3. However, iperf3 has another issue. There are limited iperf3 public servers and after N runs the server may block you for H hours. This is to prevent companies from using the public servers for commercial purposes.
Selection 6: What are the specifications for the device running speedtest?
The NIC or Wi-Fi card running speedtest needs to be at least as fast as your ISP's circuit bandwidth. My ISP provides a GigaBit connection (1,000Mbps). If testing Gb speeds than the interface must support Gb speeds.
Here are my device specifications:
MacBook Specifications:
As of this posting, I am using a MacBook Pro (15-inch 2018) with a 2.2GHz 6-core Intel Core i7 running Catalina version 10.15.2. The MacBook has an AirPort Extreme Wi-Fi using 802.11ac and WPA2 security.
My MacBook has a 1.3Gbps Wi-Fi (802.11ac 3x3) interface and a Gb wired connection.
iPhone Specifictions:
iPhone 8 plus with an ARM A11 bionic core running iOS13.3. My iPhone supports 802.11b/g/n/ac
Selection 7: What is your device running? Stop it!
To save costs, many but not all, devices use a combination of hardware and software interface. So, the more applications that are running on your device the less consistent your results will be.
MacBook:
Close or shut down all the applications on your device except for the ones you must have. On a MacBook, I have only one browser window open and text edit to record the results.
iPhone:
In addition stopping all your apps, you should shut off Cellular data in settings. Smart phones have the ability to switch back-and-forth between Cellular and Wi-Fi. If your device switches, then your results will be worthless.
Selection 8: What about your network?
Regardless of your ISP, your ISP's gateway is junk. The goal of the ISP is to maximize profits and provide the features required with the lowest cost gateway. All the gateways are made by a hand full of companies (e.g., Pace, Arris, and Motorola). When I had a DSL connection, I could get a 20% improvement in bandwidth speed by shutting off the coax and Wi-Fi.
Periodically get a new gateway. The newer gateways will run faster than the old ones. Your ISP should replace your gateway when asked.
Since the gateway has a low-end processor and limited memory, it should be periodically rebooted. If you reboot the gateway, you may have to do the same for your APs and switches.
My normal routine is:
Reboot gateway, wait until it is up
Power cycle attached switches, wait about 5 minutes
Reboot or power cycle my APs
This ensures everything is starting in a known good state.
Selection 9: What is your RSS?
If you are running speedtest over Wi-Fi, then you need to measure your Received Signal Strength (RSS). For consistent results, you should run speed test with an RSS of -65dB or better (lower negative numbers are better).
To measure RSS, on an iPhone use Airport Utility and on a MacBook use Wireless Diagnostics.
Distance and obstructions (e.g., walls) will lower the RSS, each 3dB drop means a 50% drop in power.
Wi-Fi doesn't use a single encoding scheme, if the RSS is strong, then faster encodings can be used. As the RSS drops, the speed also drops.
In addition, to upload and download speeds, you should also measure and record RSS.
In addition to distance and obstructions, the time of day can negatively impact RSS.
Selection 10: Which 802.11ac are you using?
if running speedtest over Wi-Fi, then you need to be consistent with the 802.11 mode used.
I set my 5GHz radio 802.11ac-only. An 802.11ac AP is characterized by the number after the AC marking. So, an 802.11ac AP may only be able to run at 100Mbps. An AC1300 is capable of running at 1.3Gbps, and an AC3200 is even faster
The AP built-in to your ISP's gateway may not be able to handle Gb speeds
For Wi-Fi speedtests, I use either a Linksys AC1300 or AC3200 AP
Selection 11: What is your baseline speed?
Prior to running speedtest from an office or your favorite chair, set a baseline estimate.
Connect a laptop using an ethernet connection directly to the ISP's gateway.
With a direct wired connection to the Gateway I get 922Mbps down and 945Mbps up.
Selection 12: What are you going to report?
I think you must report: latency, upload and download speeds. Jitter is nice to have, but not required.
Speedtest uploads and downloads should be an average of several runs. Remember to throwout runs that fail.
If the speedtest tool doesn't provide latency or jitter, then you can get this by running ping. Latency should be an average of 5-10 ping responses. Jitter is (max(latency)-min(latency))/2
Now, you are ready to run Speedtest
Appendix
Find a public iperf3 Server:
Open a browser and enter:
https://iperf.fr/iperf-servers.php#public-servers
Install iperf3 on Raspberry Pi
Start MacBook Terminal and run the command:
$ sudo apt-get install iperf3
Install iperf3 on MacBook
Start MacBook Terminal and run the command:
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null
Then install iperf3:
$ brew install iperf3
Run iperf3
$ iperf3 -P 30 -f m -p 5201 -c iperf.scottlinux.com
Terminology
b = bits, and B = Bytes
Mbps = Mb/s = Megabits per second
Gbps = Gb/s = Gigabits per second