The Chromium Projects

Search this site

Except as otherwise noted, the content of this page is licensed under a Creative Commons Attribution 2.5 license, and examples are licensed under the BSD License.

GPU Testing

This set of pages documents the setup and operation of the GPU bots and try servers, which verify the correctness of Chrome's graphically accelerated rendering pipeline.  

Overview

The GPU bots run a different set of tests than the majority of the Chromium test machines. The GPU testers are specifically focused on tests which exercise the graphics processor, and whose results are likely to vary between graphics card vendors.

Most of the tests on the GPU bots are run via the Telemetry framework. Telemetry was originally conceived as a performance testing framework, but has proven valuable for correctness testing as well. Telemetry directs the browser to perform various operations, like page navigation and test execution, from external scripts written in Python. The GPU bots launch the full Chromium browser via Telemetry for the majority of the tests. Using the full browser to execute tests, rather than custom test harnesses like content_browsertests, has yielded several advantages: better testing code that is shipped, improved reliability, and improved performance.

A subset of the tests, called "pixel tests", grab screen snapshots of the web page in order to validate Chromium's rendering architecture end-to-end. Where necessary, GPU-specific results are maintained for these tests. Some of these tests verify just a few pixels, using handwritten code, in order to use the same validation for all brands of GPUs.

The GPU bots use the Chrome infrastructure team's new recipes framework for describing what tests to execute. Compared to the legacy buildbot infrastructure, recipes make it easy to add new steps to the bots, change the bots' configuration, and run the tests locally in the same way that they are run on the bots.

Finally, the Release GPU bots use the isolated testing framework to transmit the binaries and data files from the builder to the tester. (As of this writing, the Debug GPU bots use the component build, and isolates don't yet support the component build.) Running tests via isolates eliminates the Chromium src/ checkout on the test machines, achieving better utilization on the physical hardware containing GPUs.

These aspects of the bots are described in more detail below, and in linked pages.

Using the GPU Bots

Most Chromium developers interact with the GPU bots in two ways:
  1. Sending try jobs to them.
  2. Observing the bots on the waterfalls.
The GPU try servers only support try jobs sent from git. Subversion try jobs sent via gcl are not supported. Use

git cl try -b win_gpu -b linux_gpu -b mac_gpu

to explicitly send a try job to the GPU try servers. As of this writing, the GPU try servers were in the process of being added to the default set of try servers for every try job. Soon, it will not be necessary to manually send jobs to these try servers; it will happen automatically.

Alternatively, the Rietveld UI can be used to send a patch set to these try servers.

The GPU try bots explicitly support try jobs against the Chromium (src/) and Blink (src/third_party/WebKit/) trees. If you find it necessary to try patches against other sub-repositories, please file a bug with label Cr-Internals-GPU-Testing.

Running the GPU Tests Locally

All of the GPU tests running on the bots can be run locally from a Chromium build. However, figuring out the exact command line to use can be a little tricky. The Release GPU bots all run their tests via isolates, and in this case, the logs from a run on the bots will contain something like
run_isolated.py -H b83c4a0bce1c..... -I https://isolateserver.appspot.com
which doesn't help when trying to figure out the command line to run the test.

Many of the GPU tests are run via Telemetry. In order to run them, just build the chrome target and then invoke src/content/test/gpu/run_gpu_test.py with the appropriate argument. The tests this script can invoke is in src/content/test/gpu/gpu_tests/ . For example:

run_gpu_test.py context_lost  --browser=release
run_gpu_test.py memory_test --browser=release
run_gpu_test.py webgl_conformance --browser=release --webgl-conformance-version=1.0.2

Most of the remaining tests are simple executables which can be built and run from the command line:

angle_unittests
content_gl_tests
gl_tests
gles2_conform_test

You can find the isolates for the various tests in src/chrome/:


The isolates contain the full or partial command line for invoking the target. The complete command line for any test can be deduced from the contents of the isolate plus the stdio output from the test's run on the bot.


Comments