Dependent-Test-Aware Regression Testing Techniques

Tools and scripts: https://github.com/TestingResearchIllinois/dependent-tests-impact

Paper: http://mir.cs.illinois.edu/winglam/publications/2020/LamETAL20ISSTA.pdf

If you use any of this work, please cite our corresponding paper:

@inproceedings{LamETAL20ISSTA,
 author = "Wing Lam and August Shi and Reed Oei and Sai Zhang and Michael D. Ernst and Tao Xie",
 title = "Dependent-test-aware regression testing techniques",
 booktitle = "ISSTA 2020: ACM SIGSOFT International Symposium on Software Testing and Analysis",
 month = "July",
 year = "2020",
 address = "Virtual Event",
 pages = "298--311"
}

List of which modules that we exclude from the iDFlakies dataset and the reasons for why we exclude them (subjects and modules we actually used in the paper have its Status as "Included")

The subjects and modules along with whether they contain Order-dependent and/or Non-order-dependent tests are obtained from a prior work by Lam et al. More information about their work is available on https://sites.google.com/view/flakytestdataset/home.

Exact number of OD-test failures for each regression testing algorithm:

  • Number of OD-test failures for 4 test prioritization algorithms.

  • Number of OD-test failures for 6 test selection algorithms.

  • Number of OD-test failures for 2 test parallelization algorithms.

We thank Jonathan Bell, Sandy Kaplan, Martin Kellogg, Darko Marinov, and the anonymous referees who provided feedback on our paper. This material is based on research sponsored by DARPA under agreement numbers FA8750-12-2-0107, AFRL FA8750-15-C-0010, and FA8750-16-2-0032. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. This work was also partially supported by NSF grant numbers CNS-1564274, CNS-1646305, CNS-1740916, CCF-1763788, CCF-1816615, and OAC-1839010. Tao Xie is affiliated with Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education.


Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).