System Design‎ > ‎

WebPagetest Relay

The current system architecture has each test agent connecting directly to the WebPagetest server for retrieving work and posting results:
This works well for a lot of use cases but it requires that the server be reachable from all of the agents and that each agent can only run tests for a single server.  It would be useful to add support for a relay server that exposes the same Agent interface on one side and an API for WebPagetest servers to communicate with on the other.

In the simple use case this would allow a relay to be placed in an insecure location (say, the public Internet) but the server itself could be run inside of a firewall or other secure zone with test agents both on the public Internet and also inside of the firewall:
Assuming that each defined "location" on the WebPagetest server can reference different relay servers and each relay server can support multiple servers connecting to it (with key-based authentication) it opens up a lot of interesting possibilities:
  • Multiple systems can share a pool of test agents (with the owner of the relay controlling access to them).  The sharing can be straight-up collaboration or something more complex where each user gets billed for their share of usage or even a straight subscription model where the owner sells capacity on a per-test basis
  • The relay server can prioritize requests across the servers requesting testing (based on whatever prioritization makes sense for the system)
  • A given WebPagetest instance can leverage both public and private test agents and even pull from multiple pools of public agents by using several relays

Implementation Details

Any Full WebPagetest instance can run as a relay with a few tweaks
  • runtest.php - add support for test jobs being uploaded directly (and tying the job to the API key used)
  • - getTestPath() - add support for a new test ID that combines the API key and test ID and stores the ephemeral test results in a different tree
  • downloadtest.php - add a new interface to download a zip archive of a given test (basically the same as publish but in a pull model)
  • deletetest.php - add a new interface for deleting tests explicitly (based on matching API key)
WebPagetest Interface
  • Any location in locations.ini can be configured to talk to a relay server with the following settings:
    • Relay server URL (base URL)
    • Relay location ID (relay server's location ID to use)
    • Relay Key (API Key to use when interfacing with the relay server)
  • When tests are submitted to the local server, instead of writing the test job to a local file it will post it to the relay server
  • When checking the status of a test, if the test is not complete then WebPagetest will ask the relay server for the status
  • If the relay server indicates that the test is complete, WebPagetest will download the result from the relay server as a zip and extract it locally
    • After download/extract WPT will indicate that it is done with the test so the relay can delete it's copy
Agent Interface
  • Agents will communicate with a relay exactly as if they are communicating with WebPagetest (work/getwork.php, work/resultimage.php, work/workdone.php)
Relay Implementation
  • A dedicated relay server will just be a subset of a full WPT install without the UI support (SVN will be used to place the selected files into a wptrelay directory in the dist release)