This set of pages documents the setup and operation of the GPU bots and try servers, which verify the correctness of Chrome's graphically accelerated rendering pipeline.
The GPU bots run a different set of tests than the majority of the Chromium test machines. The GPU testers are specifically focused on tests which exercise the graphics processor, and whose results are likely to vary between graphics card vendors.
Most of the tests on the GPU bots are run via the Telemetry framework. Telemetry was originally conceived as a performance testing framework, but has proven valuable for correctness testing as well. Telemetry directs the browser to perform various operations, like page navigation and test execution, from external scripts written in Python. The GPU bots launch the full Chromium browser via Telemetry for the majority of the tests. Using the full browser to execute tests, rather than custom test harnesses like content_browsertests, has yielded several advantages: better testing code that is shipped, improved reliability, and improved performance.
A subset of the tests, called "pixel tests", grab screen snapshots of the web page in order to validate Chromium's rendering architecture end-to-end. Where necessary, GPU-specific results are maintained for these tests. Some of these tests verify just a few pixels, using handwritten code, in order to use the same validation for all brands of GPUs.
The GPU bots use the Chrome infrastructure team's new recipes framework for describing what tests to execute. Compared to the legacy buildbot infrastructure, recipes make it easy to add new steps to the bots, change the bots' configuration, and run the tests locally in the same way that they are run on the bots. There is a separate page about the GPU recipe describing how it works and how to modify it.
The GPU bots use the isolated testing framework to transmit the binaries and data files from the builder to the tester. Running tests via isolates eliminates the Chromium src/ checkout on the test machines, achieving better utilization of the physical hardware containing GPUs.
The bots on the chromium.gpu.fyi waterfall are configured to always test top-of-tree ANGLE. This setup is done with a few lines of code in the GPU recipe module; search the code for "ANGLE".
These aspects of the bots are described in more detail below, and in linked pages.
Most Chromium developers interact with the GPU bots in two ways:
Sends your job to the default set of try servers.
The GPU tests are part of the default set for both Chromium and Blink CLs. For Chromium CLs, the GPU tests are run as part of the following tryservers' jobs:
For Blink CLs, the GPU tests are run as part of the following tryservers' jobs:
Scan down through the steps looking for the text "GPU"; that identifies those tests run on the GPU bots. For each test the "trigger" step can be ignored; the step further down for the test of the same name contains the results.
It's usually not necessary to explicitly send try jobs just for verifying GPU tests. If you want to, you must invoke "git cl try" separately for each tryserver master you want to reference, for example:
git cl try -b linux_chromium_rel_ng
git cl try -b mac_chromium_rel_ng
git cl try -b win_chromium_rel_ng
Alternatively, the Rietveld UI can be used to send a patch set to these try servers.
If you find it necessary to try patches against other sub-repositories than Chromium (src/) and Blink (src/third_party/WebKit/), please file a bug with label Cr-Internals-GPU-Testing.
All of the GPU tests running on the bots can be run locally from a Chromium build. However, figuring out the exact command line to use can be a little tricky. The Release GPU bots all run their tests via isolates, and in this case, the logs from a run on the bots will contain something like
which doesn't help when trying to figure out the command line to run the test.
Many of the GPU tests are run via Telemetry. In order to run them, just build the chrome target and then invoke src/content/test/gpu/run_gpu_test.py with the appropriate argument. The tests this script can invoke is in src/content/test/gpu/gpu_tests/ . For example:
The Maps test requires you to authenticate to cloud storage in order to access the Web Page Reply archive containing the test. See Cloud Storage Credentials for documentation on setting this up.
Most of the remaining tests are simple executables which can be built and run from the command line:
You can find the isolates for the various tests in src/chrome/:
The isolates contain the full or partial command line for invoking the target. The complete command line for any test can be deduced from the contents of the isolate plus the stdio output from the test's run on the bot.
Any binary run remotely on a bot can also be run locally, assuming the local machine loosely matches the architecture and OS of the bot.
Before running a remote test locally, you must ensure you have permission to access the isolate server. Full instructions can be found here. For most cases, you can simply run:
./src/tools/swarming_client/auth.py login --service=https://isolateserver.appspot.com
The above link requires that you log in with your @google.com credentials.
Once you are authenticated, you can run a remotely built test locally using run_isolated.py. To determine the correct invocation, examine the stdio output for the desired test. In the stdio output, there should be a line which contains an invocation of run_isolated.py, for example:
Once you have located this command line, copy the program arguments (red above). Some arguments which are specific to the test machine (dark red above) can safely be dropped.
Once you have the arguments, the test is invoked via:
Note that this requires downloading a large number of files (everything needed to run the test) and may take a while. Subsequent runs should be able to re-use cached resources and will be significantly faster.
Once you have followed the instructions on testing your own isolates for the GYP_DEFINES and authentication needed to upload isolates to the isolate server, you can also run your locally built tests by invoking run_isolated.py on the bot itself:
The goal of the GPU bots is to avoid regressions in Chrome's rendering stack. To that end, let's add as many tests as possible that will help catch regressions in the product. If you see a crazy bug in Chrome's rendering which would be easy to catch with a pixel test running in Chrome and hard to catch in any of the other test harnesses, please, invest the time to add a test!
As of this writing it isn't as easy as desired to add a new test to one of the Telemetry based harnesses. See http://crbug.com/352807 . Let's collectively work to address that issue. It would be great to reduce the number of steps on the GPU bots, or at least to avoid significantly increasing the number of steps on the bots. The WebGL conformance tests should probably remain a separate step, but some of the smaller Telemetry based tests (context_lost_tests, memory_test, etc.) should probably be combined into a single step.
If you are adding a new test to one of the existing tests (e.g., pixel_test), all you need to do is make sure that your new test runs correctly via isolates. See the documentation from the GPU recipe on testing your own isolates for the GYP_DEFINES and authentication needed to upload isolates to the isolate server. Most likely the new test will be Telemetry based, and included in the telemetry_gpu_test_run isolate. You can then invoke it via:
Adding new pixel tests which require reference images is a slightly more complex process than adding other kinds of tests which can validate their own correctness. There are a few reasons for this.
When making a Chromium-side change which changes the pixel tests' results:
The GPU bots are using the Chrome infrastructure team's new "recipe" infrastructure, which makes it dramatically easier than before to make configuration changes to the bots. See the documentation on the GPU bots' recipe for details on how to add new steps to the bots and modify their existing steps.
It's critically important to aggressively investigate and eliminate the root cause of any flakiness seen on the GPU bots. The bots have been known to run reliably for days at a time, and any flaky failures that are tolerated on the bots translate directly into instability of the browser experienced by customers. Critical bugs in subsystems like WebGL, affecting high-profile products like Google Maps, have escaped notice in the past because the bots were unreliable. After much re-work, the GPU bots are now among the most reliable automated test machines in the Chromium project. Let's keep them that way.
Flakiness affecting the GPU tests can come in from highly unexpected sources. Here are some examples:
The GPU tryservers have been transitioned from a traditional waterfall model to Chromium's swarming infrastructure. The benefits of this transition are manifold:
The tryserver.chromium.gpu waterfall has been decommissioned, so there is no longer be a single centralized place to examine the GPU tryservers' results. Instead, the following tools should be used to check their health: