6.01 - Log files

1. Findings & Impacts

Each web server owner must legally have a log file that records all hits, for each resource (HTML page, image file, javascript etc) for all sites hosted by the server. In this log file can be found all the hits generated by Internet users, but also those of crawlers including the famous Googlebot - Google indexing bot.

Here is an example of a hit recorded in a log file:

In order, we can read:

• the date of the request
• the time of the request
• the method of the request
• the relative url requested
• the IP address that made the request
• the user agent of the request
• the http code of the page

Each site has different log formats depending on the technology used on the server side (IIS, APACHE, NGINX etc.), the server version and more generally its configuration. The example given may not be valid for your site.

Benefit of doing a log analysis: The objective of such an analysis will be to follow the activity of Googlebot and measure its impact on your SEO. This allows you to know with great accuracy the URLs ignored by Google and those its crawlers love most, by knowing that Googlebot revisits them dozens of times a day. The log analysis makes it possible to locate the resources which can cause a loss of link equity. If, in your logs (filtered on Googlebot), 30% of hits fall on 404 or 500 errors, i means that you're wasting 30% of your crawl budget on useless pages, at the expense of your strategic pages!

2. Recommendations

If you want to go ahead with a log analysis, it is necessary to be able to provide at least 30 rolling days of filtered logs on GoogleBot with at least this information:

Date

• exact date of the request with its time zone (preferably) - for example +0100
• apache format: [Wed oct 11 14:30:52 2000+0100]
• IIS format: [2016-06-25 03:59:59]

Complete URL with the the request parameter

• GET /apache_pb.gif HTTP/1.0 -- "GET" is one of the two commands (the other is "POST") that can be performed. "GET" fetches a URL while "POST" is submitting something (such as a forum comment). The second part is the URL that is being accessed, and the last part is the version of HTTP that is being accessed.

Referer

• Page from which the connexion was made

User-agent

• Browsers or crawler that emitted the request

HTTP Status Code

• The status code of the document that was returned

Domain associated to the URL

• If the log analysis contains the logs of different sub-domains, the virtual host associated to the URL is either in the URL or in a separate field

Protocol

• the protocol in which the page is served (http or https)

These fields are all contained in the default formats for Apache, nginx, Varnish and IIS.

The logs may contain comments beginning with a #.

We draw your attention to the importance of the 'Referer' field, which is essential for the detection of visits and for the quality of the data of the reports.

If you go through a CDN, it is possible that the Referer field is by default absent from your log files. We encourage you to check this point and ask your provider to activate this field before sending us your first logs.

How to send us these .log files?

• Drop them daily or occasionally on one of your servers and grant us access
• Transfer them by email or by a third-party service like WeTransfer, Google Drive, Dropbox ...