The findings from this work are now integrated into the Illinois Dataset of Flaky Tests (IDoFT). Please use the information there for our most up-to-date findings.

Paper: http://mir.cs.illinois.edu/winglam/publications/2020/LamETAL20ISSRE.pdf

If you use any of this work, please cite our corresponding paper:

@inproceedings{LamETAL20ISSRE,

author = "Wing Lam and Stefan Winter and Angello Astorga and Victoria Stodden and Darko Marinov",

title = "Understanding reproducibility and characteristics of flaky tests through test reruns in {J}ava projects",

booktitle = "ISSRE 2020: 31st IEEE International Conference on Software Reliability Engineering",

month = "October",

year = "2020",

address = "Virtual Event",

pages = "403--413"

}

Data and scripts for each RQ

You can download the scripts that we used to obtain the results for each RQ from the following (the zip file is about 108KB). When unzipped the contents occupy about 2.0MB.

http://mir.cs.illinois.edu/winglam/publications/2020/LamETAL20ISSRE/rq-data.zip

Detailed data for each RQ

You can download the results of running each test when it is run in isolation from the following (the zip file is about 2.6MB). When unzipped the contents occupy about 113MB.

http://mir.cs.illinois.edu/winglam/publications/2020/LamETAL20ISSRE/iso-per-test-results.zip

You can download the results of running each test when it is run in a test suite from the following (the zip file is about 281MB). When unzipped the contents occupy about 6.9GB.

http://mir.cs.illinois.edu/winglam/publications/2020/LamETAL20ISSRE/tso-per-test-results.zip

The iso-per-test-results.zip and tso-per-test-results.zip contains for each test the `test_name,test_result,time_to_run_test,run_num,machine_id,test_class_order_md5sum,num_test_class,module_path,slug,sha`.

We thank Anjiang Wei for helping us debug some flaky tests, Jon Bell for extensive discussions about flaky tests, and Tianyin Xu for sharing Microsoft Azure credits. This work was partially supported by NSF grant nos. CCF-1763788 and OAC-1839010, GEM fellowship, and Supplemental Summer Block Grant (SSBG). We acknowledge support for research on flaky tests from Facebook and Google.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).