05. Conducting Quality Assurance On Completed Crawls

Reviewing Crawl Reports

Once a crawl has completed, Archive-It has generated crawl reports, and the crawl has been indexed for playback in the Wayback Machine, it is possible to conduct quality assurance on the archived content. Quality assurance ensures that seeds are captured in Archive-It and rendered successfully in the Wayback Machine. A number of different factors can impact capture and display, including robots.txt exclusions, poorly-defined scoping rules, crawler traps, and pre-existing site anomalies (i.e. broken links, missing content). Quality assurance identifies issues with crawled content and provides opportunities for mitigating these issues through patch crawls and other methods. It is particularly useful to conduct quality assurance on test crawl results to avoid wasting data on ineffective crawls, but quality assurance should also be performed on regular scheduled recurring crawls, as seeds that were previously crawled effectively may present new issues.

To begin performing quality assurance on a completed crawl, navigate to that crawl's crawl report. The crawl overview, seed, host, and file type reports provide important information about the finished crawl. For extensive instructions on reading and interpreting these reports, visit the Archive-It Help Center. When examining the seed overview portion of a crawl report, it is important to note how much new data was captured and how many new documents were crawled for each site. Abnormally high or low data/document numbers may reflect potential problems meriting further correction. The seed overview section will also identify any seeds that had crawling issues, such as the site no longer existing at the given seed URL.

From the seed report, it is also important to view each seed individually by clicking the Wayback link on the right side of the page. From the Wayback calendar page, choose the most recent crawl date, which should from within the time period of the crawl that you are examining. When reviewing a URL in Wayback, briefly browse the page for obvious instances of content not being captured or rendered incorrectly. Test interactive features, database-driven content, videos, and other kinds of content that may present issues for the crawler. Do not feel obligated to click on every link, to thoroughly browse the site, or to note the presence of small cosmetic issues. The primary purpose of viewing a seed URL in Wayback during quality assurance is to ensure, at a very high level, that the site was captured and renders successfully.

After reviewing the seed overview, check the host report section of the crawl report. Within the host report, check to see if any hosts were blocked by robots.txt exclusions and if any hosts have a large number of URLs in the queued column. Abnormally large numbers of URLs in either column may indicate that an insufficient amount of require content was captured or that the web crawler may have encountered a crawler trap, greatly expanding the amount of data captured by the crawl and drawing the crawler's resources away from capturing essential content. The Archive-It Help Center provides specific information about how to identify, avoid, and mitigate the effects of crawler traps.

Enable QA Tool

In addition to reviewing crawl reports and manually reviewing crawled seeds, it is also possible to enhance quality assurance through a built-in function called Enable QA. Enable QA generates a list of URLs that were not captured during a crawl. These often include font files, JavaScript, and stylesheets from external/out of scope domains. Enable QA can help to identify meaningful content that is missing from a crawl.

While Enable QA can be useful, it can also slow down the process of conducting quality assurance and can generate a long list of URLs that may not be worth capturing (e.g., Google Analytics files). Enable QA should only be performed in cases where one of the following conditions apply:

• • Crawled seeds contain content of significant historical and institutional value (i.e., crawls of sites of the U-M administration, such as the President and Provost, or sites associated with the 19 U-M schools and colleges)

  • The crawl report indicates that less content than expected was crawled for a given seed due to robots.txt exclusions or other issues

  • The crawled seed renders poorly or not at all in Wayback

  • A cursory examination of the site in Wayback reveals many portions of the site that yield "Not in Archive" errors

If it seems appropriate to Enable QA for a given seed, visit the capture for that seed in Wayback and click Enable QA on the banner at the top of the page. Once Enable QA is activated, Archive-It will begin assembling missing URLs. Click View Missing Documents at the top of the page to be redirected to the list of missing URLs in Archive-It. Once there, it is possible to review the list of missing URLs and to select URLs to capture in a patch crawl, adding the missing content to Archive-It.

Possible Quality Assurance Actions

Quality assurance may result in a number of different actions being taken to improve future captures for a given seed. Several of the possible actions, along with scenarios for which they might be most appropriate, are detailed below.

• • Expand scope: Expand the seed's scope if the crawl did not capture essential content, using the Archive-It documentation on expanding the scope of a crawl for guidance.

  • Limit scope: If the crawl captured too much data (for most seeds, about 1 GB of data is sufficient), fell into a crawler trap (as evidenced by many queued URLs), or otherwise captured too much superfluous content beyond what was necessary to sufficiently capture the site's content, limit the seed's scope using the Archive-It documentation on scoping for guidance.

  • Create new seed: If an existing seed URL redirects to a substantially different URL (i.e., the redirected URL is not simply a different in http/https or the existence or lack of the www subdomain), then a new seed should be created for the redirected URL in order to ensure that future captures are completed successfully. Create a new seed following the instructions beginning with the identification of the crawl target. Additionally, add a Note (see Documenting Quality Assurance Actions below) to both the old and the new seed documenting that a new seed was added for the redirected URL. Finally, deactivate the old seed.

  • Deactivate seed: If a seed URL is no longer being updated, no longer exists, redirects to a substantially different URL, or otherwise no longer warrants capturing, deactivate the seed by unchecking the "Active Seed" checkbox on the seed's Settings page. Add a Note (see Documenting Quality Assurance Actions below) with the justification for deactivating the seed.

Documenting Quality Assurance Actions

In order to maintain an administrative history of the actions taken for a given seed, document significant quality assurance actions taken in a Note associated with the relevant seed. To do so, navigate to the Metadata tab for a seed, click edit, and add a custom field with a type of Field Name of Note and a Value of the form "[your initials] [date]: [action]" (e.g., "DJP 2017-06-30: seed deactivated. Seed redirects to new URL, [URL]. A new seed has been created for [URL]." or "DJP 2017-06-30: adding scoping rules to block repeating directory crawler traps.")