The findings from this work are now integrated into the Illinois Dataset of Flaky Tests (IDoFT). Please use the information there for our most up-to-date findings.

Paper: http://mir.cs.illinois.edu/winglam/publications/2020/LamETAL20OOPSLA.pdf

If you use any of this work, please cite our corresponding paper:

@inproceedings{LamETAL20OOPSLA,
author = "Wing Lam and Stefan Winter and Anjiang Wei and Tao Xie and Darko Marinov and Jonathan Bell",
title = "A large-scale longitudinal study of flaky tests",
booktitle = "OOPSLA 2020: ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications",
month = "November",
year = "2020",
address = "Virtual Event",
pages = "202:1--202:29"
}

Data and scripts for the results in our paper

You can download the scripts that we used to obtain the results in our paper from the following (the zip file is about 68KB). When unzipped the contents occupy about 428KB. A copy of the data generated from our scripts is included in the zip file as well.

http://mir.cs.illinois.edu/winglam/publications/2020/LamETAL20OOPSLA.zip

We thank Pu Yi for helping us inspect specific flaky tests, and Reed Oei and August Shi for general discussions about flaky tests. This work was partially supported by NSF grant nos. CNS-1564274, CCF-1763788, CCF-1763822, CCF-1816615, and CCF-1844880. We acknowledge support for research on flaky tests from Facebook and Google.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).