For mobile testing you have a few options for network connectivity:
This is what the mobile testing network looks like for the public instance of WebPagetest (at least as of this writing in 2014):
The different configuration build on top of each other:
Testing with the new Node.js agent requires Android devices running 4.4 (Kit Kat) or later or iOS devices running a recent version of iOS (more iOS documentation and improved support coming soon). In both cases the devices need to be rooted (jailbroken in the case of iOS) in order to be able to delete the browser cache. For Android, Nexus and Motorola devices are known to work well and provide a wide range of capabilities. Separate devices are required for testing portrait vs landscape (mostly an issue for tablet testing).
The simplest set-up is running a device on a carrier network. In this case you just need the phone(s) and a tethered host to control them. Multiple devices can be controlled from a single host (as many as the OS will allow USB connections). Any OS that supports adb and Node.js should work though Windows and Linux get the most testing.
If the host has USB 3 controllers and you want to use Linux you need to make sure it is a VERY recent kernel as USB 3 support in earlier releases was REALLY buggy.
For Windows there is also a support application adbwatch that watches adb and if it dies (stops responding as it sometimes does) it will automatically restart it allowing the system to run with no human intervention for extended periods.
The public instance currently has 10 devices connected to an Intel i5 NUC running Windows 7 with adbwatch managing the agent instances and keeping adb running. One tip for running the NUC's as headless servers (for the newer haswell models that do not support AMT) is to use VNC and the Microsoft video drivers shipped with Windows, not the Intel ones. If you use the Intel drivers the display will be disabled and VNC will not work.
The data flow for test results and agent <-> WebPagetest server communication all happen from the tethered host so the only use of the data connection will be the actual test data (and any software updates).
Testing over a WiFi connection gives you much more control over the stability of the results and usually produces much more consistent results than testing on a carrier network. The main issues with running a stable WiFi test connection are:
The main downside of testing an unshaped WiFi connection is that you are running at the full connection speed of your ISP and it may not match your end-user's experience.
Adding traffic shaping to the WiFi test configuration allows you to test with end-user connection characteristics but maintain the consistency of regular WiFi testing. The traffic shaping has to be done on the far side of the access point and is best done with a FreeBSD machine that bridges the network connection from the access point to the rest of the wired network (there is also a linux port of dummynet but the FreeBSD implementation has been a lot more consistent in our testing).
With a fixed traffic shaping profile you have to pick one profile and use that for all of the traffic for a given device (or all devices). When setting up the dummynet configuration it is important to remember to give each device it's own "pipe" so they are not sharing a virtual connection.
As far as hardware goes, any machine that supports FreeBSD and has 2+ network interfaces will work. I like the Supermicro Atom servers because they are cheap, super low-power and support remote management.
Here is the ipfw configuration that the public instance uses:
Instead of using a fixed connection profile for all of the devices, it is possible to have the test agent configure them on a per-test basis. The Node.js agent can call out to an ipfw shell script on the agent with information about which test agent to configure and the connection profiles. There is sample code in github that creates a pipe dynamically and assigns the appropriate profile.
For the public instance the plan is to statically configure multiple pipes, a pair for each device and then when the script configures the shaping it will just set the parameters for pipes associated with that device. This is how the desktop agents work and avoids any edge-cases where rules are left in the configuration if something dies. Once things are set up and working I will provide the script and configuration that is implemented on the public instance.
One of the more difficult issues with configuring a larger lab is managing the WiFi network(s). It is possible to reverse-tether the devices so that their networking traffic is routed over the USB connection to the tethered host. This can provide even more consistent results but you need to be careful to not overload the USB bus (which is also transferring videos and test results for the other tethered devices) and the reliability hasn't been as good as WiFi (devices tend to fall offline).
System Design >