Site Speed and Site Content Optimization Check by using WebPageTest API : Python

You will never get a second chance to make a great first expression. 

First thing a customer notices when they are visiting your site is the time it takes to load, hence your landing pages has to be optimized accordingly and regularly with each and every release you push. In this blog  we will learn about automating these check runs so that you can get quick automated feedbacks on Site Speed and content size. 

WebPageTest ?

WebPageTest is a website to run a free website speed test from multiple locations around the globe using real browsers (IE and Chrome) and at real consumer connection speeds. One can directly go to "https://www.webpagetest.org/" fill in the URL and get the test results. I recommend that you do so for one URL atleast so that youwill have better understanding of how webpagetest works, this will be useful for better understanding of API functionality of the Site.

Basic API url consists of 6 parts:

1. wpt_end_point = webpagetest API end point - 'http://www.webpagetest.org/runtest.php'

2. key = API Key which you will get for public instance here

3. url = URL for which we need information

4. mobile = 1 if you want to run test for mobile site else 0

5. run = number of test run you want to perform, on webpagetest test run 3 times

6. server = this is combination of server location from where you want to run tests, browser and connectivity, syntax - location:browser.connectivity

import requests

wpt_end_point = 'http://www.webpagetest.org/runtest.php'

key= 'A.***********************6b'

url='www.mywebiste.org'

mobile=0

run=2

location = Dulles:Chrome.DSL

request_url=wpt_end_point+'?k='+key+'&url='+url+'&mobile='+mobile+'&run='+run+'&f=xml&location='+location

response = requests.get(request_url)

If you print response you will get an XML string response similar to:

<?xml version="1.0" encoding="UTF-8"?>

<response>

<statusCode>200</statusCode>

<statusText>Ok</statusText>

<data>

<testId>1***********************e9</testId>

<ownerKey>f***********************8</ownerKey>

<xmlUrl>http://www.webpagetest.org/xmlResult/1***********************9/</xmlUrl>

<userUrl>http://www.webpagetest.org/result/1***********************9/</userUrl>

<summaryCSV>http://www.webpagetest.org/result/1***********************9/page_data.csv</summaryCSV>

<detailCSV>http://www.webpagetest.org/result/1***********************9/requests.csv</detailCSV>

<jsonUrl>http://www.webpagetest.org/jsonResult.php?test=1***********************9</jsonUrl>

</data>

</response>

Note : As soon as you make GET request, your test will be pushed into the queue. If you have tested a URL on WebPageTest website as I mentioned in the beginning of this post, you must have noticed that once you fill the details, the first message that you see on the screen is "Waiting". This waiting is because you’re using a Public server which is being used by someone else right now.As soon as its available, your tests will run and you will get the results in the destination URLs mentioned in XML.

So, after sending GET requests you may have to wait for few minutes to get the result.

Detailed CSV consist of all JS, image loads on the page along with a lot many details including size, time taken to load etc

And if you just wants to get Site Index (site score according to wpt) and load time, you can load jsonurl.

1. Getting the Site Index and Load Time

Since, response is XML string, need to convert it into xml tree before fetching information

from xml.etree import ElementTree as ET

tree = ET.ElementTree(ET.fromstring(response.content))

root=tree.getroot()

data=root.find('data')

json_url=data.find('jsonUrl').text

To get the json from the url, we need to load the URL

import urllib

import json

json_dump = urllib.urlopen(json_url)

data_dump = json.loads(json_dump.read())

Required run data can be access by data_dump['data']['runs'][number of run][view]

There are two views first view and

Key for speed index is 'SpeedIndex' and key for Load Time of the page is 'loadTime'

for eg you want to find the load time for the second run when the page loads for the first time

data_dump['data']['runs']['2']['firstView']['loadTime']

You can get as much information as you want from these run data.

2. Site Optimization Check.

The part of getting the "Detailed CSV" url is similar just have to find 'Detailed CSV' in data in place of 'jsonUrl'

from xml.etree import ElementTree as ET

tree = ET.ElementTree(ET.fromstring(response.content))

root=tree.getroot()

data=root.find('data')

csv_url=data.find('detailCSV').text

#Getting file from url

from urllib import urlretrieve

urlretrieve(csv_url)

Downloaded file will have all the details of the Site Content and Site Load Time, you can quickly sort it on the basis of load time or size and get the desired results.

So now you will be able to incorporate these code snippets in your organisation processes and get quick feedbacks on Site Content Size and load time. There are other APIs and tools other that WebPageTest like Google Page insight etc. which can also be used. No matter which tool or API you use, Site Optimization is not a one time project, it should be tested with each and every release cycle. Beautification of Landing Page won't matter if your customers are not able to visit your site.

Details of API : https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis

Like the Post