The findings from this work are now integrated into the International Dataset of Flaky Tests (IDoFT). Please use the information there for our most up-to-date findings.

Data and scripts used to generate Table 1, Table 2, and Figure 6: https://github.com/TestingResearchIllinois/icse22-nio-artifact

Paper: https://cs.gmu.edu/~winglam/publications/2022/WeiETAL22NIO.pdf

If you use any of this work, please cite our corresponding paper:

@inproceedings{WeiETAL22NIO,
author = "Anjiang Wei and Pu Yi and Zhengxi Li and Tao Xie and Darko Marinov and Wing Lam",
title = "Preempting flaky tests via non-idempotent-outcome tests",
booktitle = "ICSE 2022: Proceedings of the 44th International Conference on Software Engineering",
month = "May",
year = "2022",
address = "Pittsburgh, PA, USA",
}

We thank Yang Chen, Ruixin Wang, Satvik Eltepu, Reed Oei, and Jonathan Stein for their help. This work was partially supported by US NSF grants CCF-1763788 and CCF-1956374, and by Natural Science Foundation of China (Grant No. 62161146003), and the XPLORER PRIZE. Tao Xie (the corresponding author) is also with the Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, China. We acknowledge support for research on flaky tests from Facebook and Google.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).