Embedded Files
A Framework for Detecting and Fixing Python
Order-Dependent Flaky Tests
Paper: https://cs.gmu.edu/~winglam/publications/2022/WangETAL22iPFlakies.pdf
Link of iPFlakies on PyPI: https://pypi.org/project/ipflakies
Source code of iPFlakies: https://github.com/ailen-wrx/python-ipflakies
Experimental scripts and Docker images: https://github.com/ailen-wrx/ipflakies_artifact
Log files from our experimental scripts and CSV files generated by pytest: https://zenodo.org/record/6179032/files/ipflakies_log.zip
If you use any of this work, please cite our corresponding paper:
@inproceedings{WangETAL22iPFlakies,
author = "Ruixin Wang and Yang Chen and Wing Lam",
title = "i{PF}lakies: {A} for detecting and fixing {P}ython order-dependent flaky tests",
booktitle = "ICSE 2022 Demonstrations Track: Proceedings of the 44th International Conference on Software Engineering",
month = "May",
year = "2022",
address = "Pittsburgh, PA, USA",
}
OD tests detected and categorized by iPFlakies
Link to download the full spreadsheet
"Polluter or State-setter" indicates that we know of at least one polluter or state_setter for the OD test. (We may not know the polluter or state-setter for an OD test if the test failed in Step 1 but did not fail in Step 3.)
"Cleaner" indicates there exists at least one cleaner for the polluters of the OD test.
Extra 30 tests detected by iPFlakies not detected in Gruber et al.'s dataset: https://zenodo.org/record/6176417/files/New_Found.csv
Detailed information from minimizing the 2008 (1978+30) OD tests: https://zenodo.org/record/6176417/files/Minimized.csv
"Polluter or State-setter" indicates the test ID of the polluter or state_setter.
"Cleaner" indicates the test ID of the cleaner for the corresponding polluter.
OD tests fixed by iPFlakies
Link to download the full spreadsheet
Summary for fixing 175 OD tests:
"Diff" indicates the content of the patch.
"Patch" indicates the path to the patch file under ipflakies_artifact/ipflakies_result.
Page updated
Google Sites
Report abuse