iFixFlakies tool source code: https://github.com/TestingResearchIllinois/iFixFlakies
Paper: http://mir.cs.illinois.edu/winglam/publications/2019/ShiETAL19iFixFlakies.pdf
If you use any of this work, please cite our corresponding paper:
@inproceedings{ShiETAL19iFixFlakies,
author = "August Shi and Wing Lam and Reed Oei and Tao Xie and Darko Marinov",
title = "{iF}ix{F}lakies: {A} framework for automatically fixing order-dependent flaky tests",
booktitle = "ESEC/FSE 2019: 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering",
month = "August",
year = "2019",
address = "Tallinn, Estonia",
pages = "545--555"
}
Scripts and tools for setting up and running the framework: https://drive.google.com/open?id=1CNvXnbySJ8eKvXQkZdmK1H64ZC6eCE6B
List of 110 OD tests used in the evaluation of iFixFlakies: https://drive.google.com/file/d/10-ixGY_HDL4-awFGdOFyRWmfFthTlVaF
All patches generated for the 81 OD-tests that have helper-tests: https://drive.google.com/file/d/1rrwXetnXQzZXyX3VkocQ3H_zidTB6CS8
List of 23 accepted pull requests fixing 38 OD-tests:
List of 9 opened pull requests fixing 40 OD-tests:
About the patches and the corresponding pull requests:
33 tests use the same code as in the generated patch
18 tests involve moving the patch into setup/teardown
9 tests need special inlining, as the patch involves calling the setup into a super class, which currently iFixFlakies does not minimize, so we had to manually take lines from said setup
6 tests involve simple style modifications
3 tests have to be adjusted due to developer feedback
1 test involves special inlining due to the class using a special JUnit runner with injected fields, which current iFixFlakies does not handle properly (but the core of the technique is applicable)
1 test has an issue with a developer typo, in which fixing that typo is the more correct bug fix, which we sent
7 tests are quite different as they are experimental pull requests sent due to our manual patching, before developing iFixFlakies
We thank Angello Astorga, Owolabi Legunsen, and Lingming Zhang for discussions about flaky tests and their comments on this work. This work was partially supported by NSF grant numbers CCF1421503, CNS-1513939, CNS-1564274, CNS-1646305, CNS-1740916, CCF-1763788, CCF-1816615, and OAC-1839010. We acknowledge support for research on flaky tests and test quality from Facebook, Futurewei, Google, and Microsoft.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).