Findit

Findit takes two complementary approaches to identify the culprits or suspects:

• Heuristic-based analysis: correlate a CL with error messages in the failure log.
  • Instant: 1~2 minutes
  • May yield false positives
  • Requires manual verification for the identified suspects
• Try-job-based analysis: rerun failed compile or tests and search for the culprit in the regression range.
  • Reliable as verified by rerun of failures.
  • Fast:
    • Compile failures: median = 14 minutes
    • Swarmed gtest failures: median = 24 minutes

Given a flaky test at a specific build cycle, Findit uses post-submit build artifacts on Chromium Waterfall of the same test configuration to bypass compile, triggers Swarming tasks directly to rerun the flaky test N times (currently up to 400) at different revisions in a variant of exponential search, and then narrows down the regression range into a single build cycle on Chromium Waterfall. Once a regression range is identified with high confidence, a series of try-jobs are run to compile binaries and identify the exact culprit.

In particular, for a flaky test at a try-bot build on Chromium Commit Queue reported by Chromium-Try-Flakes, Findit maps the CQ try-bot to Waterfall buildbot before the above flow.