Web Page Replay

Background

Web Page Replay makes it possible to record a live webpage and replay it locally for testing.  For WebPageTest, Replay is used to normalize the results of hitting a live website by recording the site and playing it back in more reliable conditions.  Replaying a site reduces outliers caused by network hangs, web server traffic spikes, and other factors that can impact your benchmarking results.  Setting up WebPageReplay is very easy and is highly recommended for getting the most out of your WPT Private instance.

Requirements

  • Python 2.6
  • Port 80
  • Port 53 UDP and TCP to support DNS traffic

Network Conditioning: There are a lot of dependencies to enable network conditioning at the WPR level, see WPR getting started.  WPT's browser agents do network conditioning, so these steps can be skipped.

Server Steps
  1. Checkout the code on your server
  2. Create a wpr archive for use by the server
  3. Start WPR in server mode (root permission required to connect to port 80/53).  I recommend starting this with nohup, screen, or supervisord.
$ git clone https://github.com/chromium/web-page-replay.git
$ sudo ./replay.py --record archive.wpr
$ sudo screen
./replay.py -M ~/archive.wpr

At this point, you should have a working server.  You can now set your DNS to the IP of this box and you should see traffic in the console on the server.  Every page you access with say error 404, because we haven't recorded it.

Browser Steps

Now that we have a working WPR server, let's setup a browser agent to use it.  Fortunately, WPT already has excellent integration with the WPR.  A test agent will set the server in record mode, load a webpage, then put WPR in replay mode and run a test on it.  This can all be viewed from the console on the WPR server.

  1. Specify the webpagereplay IP ADDRESS in your wptdriver.ini
I haven't tested this with IE8/IE9 URL Blast.
web_page_replay_host=XXX.XXX.XXX.XXX

Your config should look something like this.

[WebPagetest]
url=http://wpt-rmn.cloudapp.net/
location=agent-location
browser=chrome
Time Limit=120
;key=TestKey123
;Automatically install and update support software (Flash, Silverlight, etc)
software=http://www.webpagetest.org/installers/software.dat
web_page_replay_host=XXX.XXX.XXX.XXX

[chrome]
exe="%PROGRAM_FILES%\Google\Chrome\Application\chrome.exe"
options='--load-extension="%WPTDIR%\extension" --user-data-dir="%PROFILE%" --no-proxy-server'
installer=http://www.webpagetest.org/installers/browsers/chrome.dat

[Firefox]
exe="%PROGRAM_FILES%\Mozilla Firefox\firefox.exe"
options='-profile "%PROFILE%" -no-remote'
installer=http://www.webpagetest.org/installers/browsers/firefox.dat
template=firefox

[Safari]
exe="%PROGRAM_FILES%\Safari\Safari.exe"

[IE 11]
exe="C:\Program Files\Internet Explorer\iexplore.exe"

    2. This should be all that is necessary.  Now when you run a test on this agent, it will send traffic via the webpagereplay server.  Run a test now and make sure these two lines appear in your console/log on the WPR server.

2014-07-07 18:11:12,885 DEBUG Served: GET http://XXX.XXX.XXX.XXX/web-page-replay-command-record [('cache-control', 'no-cache'), ('host', 'XXX.XXX.XXX.XXX')] (0ms)
2014-07-07 18:11:32,055 DEBUG Served: GET http://XXX.XXX.XXX.XXX/web-page-replay-command-replay [('cache-control', 'no-cache'), ('host', 'XXX.XXX.XXX.XXX')] (0ms)


FAQ

  1. Can multiple agents be used with one WPR server?
  2. Performance requirements for this server?
  3. Does WPR emulate the traffic of the recording?
  4. How can recordings be archived and reused?
Comments