In this section, we will describe the details of mutation operators implementation and show examples* of mutants.
* The highlighted part in each example is code injected or changed
1. Non-Order-Dependent (NOD) Mutation Operators
1.1 Memory Dependency (MD)
Description: MD flaky tests require a specific amount of memory that may or may not be available, e.g., due to the frequency of garbage collection and memory usage of other tests. Consequently, the test passes if enough memory is available and fails otherwise.
Implementation details: MD mutation operator will first use Soot to go through the units, if new local objects are declared and memory is allocated for them, then this test can be considered as a potential candidate for injecting MD flakiness. MD operator will 1) add system.gc() to do garbage collection at the beginning and the end of the test body, 2) get runtime information at the beginning of the method body (namely before the objects are created), and at the end of the method body (namely after memory is allocated for them), 3) add an assertion between total memory and used memory to simulate real-world MD behavior.
Example: one MD mutant from commons-cli: org.apache.commons.cli.DisablePartialMatchingTest
@Test
public void CroissantMutant_NOD_MD_testDisablePartialMatching_memoryBoundViolationMO_NoTemplate() throws Exception {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
+ Runtime runtime = Runtime.getRuntime();
+ System.gc();
final CommandLineParser parser = new DefaultParser(false);
final Options options = new Options();
options.addOption(new Option("d", "debug", false, "Turn on debug."));
options.addOption(new Option("e", "extract", false, "Turn on extract."));
options.addOption(new Option("o", "option", true, "Turn on option with argument."));
final CommandLine line = parser.parse(options, new String[] {"-de", "--option=foobar"});
assertTrue("There should be an option debug in any case...", line.hasOption("debug"));
assertTrue("There should be an extract option because partial matching is off", line.hasOption("extract"));
assertTrue("There should be an option option with a argument value", line.hasOption("option"));
+ System.gc();
+ Runtime runtime2 = Runtime.getRuntime();
+ Assert.assertEquals(runtime.totalMemory(), runtime.totalMemory() - runtime2.freeMemory());
+ }
}
1.2 Platform Dependency (PD)
Description: Platform-dependent flakiness occurs when a test assumes certain properties about the running platform, including the availability of local ports. Consequently, if it runs on a different platform that does not satisfy the assumption, e.g., during continuous integration, test outcomes can differ.
Implementation details: PD mutation operator will first find applicable objects which may depend on the running platform, such as Server. Specifically, the PD operator will first use Soot to locate units by method signature, if the units 1) contain objects Server with signature <org.eclipse.jetty.server.Server: void <init>(int)>, and 2) contain server init method <org.eclipse.jetty.server.Server: void start()>, then this test method can be a potential candidate that may assume certain properties about the running platform later, such as the availability of local ports for the server, then PD operator will mutate this method, by adding an assertion to simulate the real-world situation which indicates server failed to bind to certain ports.
Example: one PD mutant from marine-api: net.sf.marineapi.nmea.util.DateTest
@Test
public void CroissantMutant_NOD_PD_serverTest_jettyServerSetHardcodedPortMO_ServerTemplate() throws Exception {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
server = new Server(555);
server.start();
+ assertFalse(exception.getMessage().contains("Failed to bind to"));
+ }
}
1.3 System Time Dependency (STD)
Description: STD flaky tests depend on the timezone of the execution environment or a specific timestamp. We implemented 2 types of STD flaky tests 1) STD-Z: time-zone-dependency flaky tests, they assume one specific timezone, but the actual timezone differs; 2) STD-V: Timestamp-Value-dependency flaky tests, they depend on the system timestamp in an assertion, but the actual one may be changed due to delay.
Implementation details:
1) STD-Z: By using static analysis, STD-Z operator will first find the method invocations relevant to the timezone. That is, Soot extracts all InvokeExpr statements in a given test method, and if there exists a method from the Date or TimeZone class, e.g., <java.text.DateFormat: void setTimeZone(java.util.TimeZone)> or <java.util.TimeZone: java.util.TimeZone getTimeZone(java.lang.String)>, this test can be a potential candidate for injecting STD-Z mutation operator. The bytecode manipulation power of Soot allows us to 1) add statements to set a different TimeZone and 2) add an assertion between the default TimeZone and the changed one, namely assertEquals(date of TimeZone1, date of TimeZone2).
2) STD-V: STD-V mutation operator will find the method invocations relevant to timestamp, it will first leverage Soot to extract all InvokeExpr statements in a given test method, and if there exist multiple timeStamp methods, which have signature <java.lang.System: long currentTimeMillis()>, this test can be a potential candidate for STD-V mutation operator. Finally we use Soot to 1) add thread.sleep between two timeStamps and 2) add an assertion between the above two timeStamps.
Example 1: one STD-Z mutant from commons-email: org.apache.commons.mail.SendWithAttachmentsTest
@Test
public void CroissantMutant_NOD_STDZ_timeZoneDependencyTest_TimeZoneDependencyMO_TimezoneTemplate() throws Exception {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
final SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
Date date1 = dateFormat.parse("2020-11-22 10:13:55");
+ dateFormat.setTimeZone(TimeZone.getTimeZone(TimeZone.getAvailableIDs((TimeZone.getDefault().getRawOffset() + 18000000) % 43200000)[0]));
Date date2 = dateFormat.parse("2020-11-22 10:13:55");
Assert.assertNotNull(date1);
Assert.assertNotNull(date2);
+ Assert.assertEquals(date1, date2);
+ }
}
Example 2: one STD-V mutant from commons-email: org.apache.commons.mail.SimpleEmailTest
@Test
public void CroissantMutant_NOD_STDV_timeStampTest_testTimeValueInitializationTwiceMutant() {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
long timestamp1 = System.currentTimeMillis();
+ Thread.sleep(1L);
long timestamp2 = System.currentTimeMillis();
+ Assert.assertEquals(timestamp1, timestamp2);
+ }
}
1.4 Concurrency Timeout Deadlock (CTD)
Description: When tests make some calls and wait for some time to get the return value, if the wait time is not long enough to ensure the return, different executions of the test may show flaky behavior. Also, threads may get stuck in a deadlock in some executions while returning successfully in others, resulting in test flakiness.
Implementation details: CTD mutation operator will first go through the units of a given test method, if there are two threads defined in the method, then this test can be a potential candidate to inject CTD flakiness, it will be mutated by 1) adding thread.sleep() for 100 ms in each thread, 2) adding a configuration of 5 ms timeout and 3) making the two threads synchronize each other, thus can result in the 2 threads get stuck in deadlock, and ends up with timeout.
Example: one CTD mutant from commons-email: org.apache.commons.mail.ImageHtmlEmailTest_DeadLockMutationOperator_Test
@Test(timeout = 5L)
public void CroissantMutant_NOD_CTD_deadLockTest_DeadLockTemplate() throws IOException {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
+ final Object object1 = new Object();
+ final Object object2 = new Object();
Thread thread1 = new Thread() {
+ public void run() {
+ synchronized(object1) {
+ try {
+ Thread.sleep(100L);
+ } catch (Exception e) {
+ }
+ synchronized(object2) {}
+ }
+ }
};
Thread thread2 = new Thread() {
+ public void run() {
+ synchronized(object2) {
+ try {
+ Thread.sleep(100L);
+ } catch (Exception e) {
+ }
+ synchronized(object1) {}
+ }
+ }
};
+ thread1.run();
+ thread2.run();
+ }
}
1.5 Asynchronous Wait (AW)
Description: When a test makes an asynchronous call but does not wait properly to get the returned result, it leads to test flakiness.
Implementation details: AW mutation operator will first check 1) if there exists <java.util.concurrent.CountDownLatch: boolean await(long,java.util.concurrent.TimeUnit)> among VirtualInvokeExpr of the units from a given test method. If so, the test can be a potential candidate to inject AW flakiness, then AW mutation operator will add an assertion of 0 ms for latch.await(), thus can lead to an AW behavior.
Example: one AW mutant from xmlgraphics-commons: org.apache.xmlgraphics.util.DoubleFormatUtilTestCase
@Test
public void CroissantMutant_NOD_AW_AWMO_test_LatchObjectMO_LatchTemplate_ThreadSleepTemplate() throws InterruptedException {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
final CountDownLatch latch = new CountDownLatch(1);
Thread thread1 = new Thread();
Thread thread2 = new Thread(new CountDownRunnable(latch));
+ thread1.start();
- assertTrue(latch.await(1000L, TimeUnit.MILLISECONDS));
+ assertTrue(latch.await(0L, TimeUnit.MILLISECONDS));
+ }
}
+ class CountDownRunnable extends Thread {
+ private CountDownLatch countDownLatch;
+ public CountDownRunnable(CountDownLatch var1) {
+ this.countDownLatch = var1;
+ }
+ public void run() {
+ try {
+ Thread.sleep(1L);
+ } catch (InterruptedException var2) {
+ var2.printStackTrace();
+ }
+ this.countDownLatch.countDown();
+ }
+ }
1.6 Too Restrictive Range (TRR)
Description: When there is an output range of an assertion in a test, if the range does not consider all possible outcomes, the test may fail when it runs in the real world when the corner cases happen. A restrictive range for test assertions causes such flakiness.
Implementation details: TRR mutation operator will first collect all types of variables in the assignment statements of a given test method, if there exist types of boolean, int, short, long, double, float, char, etc., then this test can be a potential candidate for TRR mutation operator, TRR mutation operator will mutate these tests by adding an extremely restrictive case of TRR flaky behavior, namely the operator can change the range of a variable, and assert it with a restrictive range. Specifically, we will use Soot to 1) change the value of a certain variable into a bigger range and 2) select an extremely restrictive range to assert.
Example: a TRR mutant from marine-api: net.sf.marineapi.nmea.util.TimeTest
@Test
public void CroissantMutant_NOD_TRR_testDateRoundTrip_TRRInjectAssertLocal_NoTemplate() {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
Date now = new Date();
time.setTime(now);
Date result = time.toDate(now);
assertEquals(now, result);
boolean var = true;
+ if (now.getTime() >= 0L || now.getTime() <= 0L) {
+ var = false;
+ }
+ Assert.assertTrue(var);
assertEquals(now.getTime(), result.getTime());
+ }
}
1.7 Race Condition (RC)
Description: Multi-threading may cause test flakiness due to the seemingly non-deterministic behavior of thread interleaving. For example, If test outcomes depend on a variable shared by multiple threads in a non-thread-safe manner, then race conditions can result in non-determinism.
Implementation details: RC mutation operator will first leverage Soot to go through units of the given test method, if nonSafeThread is called, then this test can be a potential candidate to inject RC flakiness. RC mutation operator will 1) call 10000 nonSafeThread to change the same variable and 2) in the end assert the variable value with an expected value assuming there are no thread interleaving.
Example: one RC mutant from commons-scxml: org.apache.commons.scxml2.model.StateTest
@Test
public void CroissantMutant_NOD_RC_raceConditionTemplateTest_RaceConditionMutationOperator_NoTemplate() throws IOException {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
ArrayList list = new ArrayList();
list.add(0);
- Thread thread = new Thread(new nonSafeThread(list));
- thread.start();
+ for(int index = 0; index < 10000; ++index) {
+ Thread thread = new Thread(new nonSafeThread(list));
+ thread.start();
+ }
+ Assert.assertEquals(10000, list.get(0));
+ }
}
class nonSafeThread implements Runnable {
ArrayList<Integer> list;
nonSafeThread(ArrayList<Integer> var2) {
this.list = var2;
}
public void run() {
Integer var1 = (Integer)this.list.get(0);
var1 = var1 + 1;
this.list.set(0, var1);
}
}
2. Implementation-Dependent (ID) Mutation operators
2.1 Unordered Collection Index Access (UCIA)
Description: Since Java does not allow direct indexing of unordered collections, so some tests convert an unordered collection into an ordered collection. However, this conversion does not preserve the index of items deterministically, which can lead to test flakiness.
Implementation details: UCIA mutation operator will first leverage Soot to go through all units in the test method, if there exists an unordered collection such as "java.util.HashSet" in all Local objects, at the same time, it is converted to an ordered collection such as "java.util.List", this test can be a potential candidate. UCIA mutation operator will 1) convert HashSet into a new List and 2) assert the element at index 0 of the above two lists equals to each other, namely assertEquals(list1[0], list2[0]).
Example: one UCIA mutant from jsoup: org.jsoup.nodes.LeafNodeTest
@Test
public void CroissantMutant_NOD_UCIA_UPMO_test_UnorderedCollectionIndexMutationOperator_NoTemplate() {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
HashMap<String, String> map = new HashMap<String, String>();
HashSet<String> set = new HashSet<String>();
Method[] methods = map.getClass().getMethods();
for (Method method : methods)
set.add(method.getName());
Method[] methods1= new Method[];
for (String methodname : set) {
methods1.add(methodname);
}
+ Method[] methods2= new Method[];
+ for (String methodname : set) {
+ methods2.add(methodname);
+ }
+ Assert.assertEquals(methods1[0], methods2[0]);
+ }
}
2.2 Unordered Collections Conversion (UCC)
Description: When developers convert unordered collections such as Map to String to use for comparison in assertions. Due to the non-deterministic order of elements in unordered collections, the actual string generated may or may not match the expected string.
Implementation details: Due to the non-deterministic order of elements in an unordered collection, if an unordered collection such as a map is converted to a string for comparison in assertion, the actual string may or may not match the expected string, which leads to UCC flaky tests. UCC mutation operator will first use Soot to go through the test method if there exists an unordered collection such as "java.util.HashSet" in all Local variables, at the same time, methods with signature <java.util.AbstractCollection: java.lang.String toString()> also exists in InvokeExpr of units in which HashSet is called, this test can be a potential candidate for UCC operator, it will be mutated by asserting the string converted from the same HashSet twice equals each other, namely assertEquals(HashSet.toString(), HashSet.toString()).
Example: one UCC mutant from commons-text: org.apache.commons.text.StringSubstitutorWithInterpolatorStringLookupTest
@Test
public void CroissantMutant_NOD_UCC_UPMO_test_OrderedStringConversionMutationOperator_NoTemplate() {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
HashMap<String, String> map = new HashMap<String, String>();
HashSet<String> set = new HashSet<String>();
Method[] methods = map.getClass().getMethods();
for (Method method : methods)
set.add(method.getName());
+ Assert.assertEquals(set.toString(), set.toString());
+ }
}
2.3 Reflection API Misuse (RAM)
Description: Developers use reflection APIs to inspect methods or classes at runtime, but the results of such APIs are non-deterministic. As a result, assuming a specific order is incorrect.
Implementation details: RAM mutation operator will first use Soot to go through all units of a given test method, if a reflection API such as "<java.lang.Class: java.lang.reflect.Method[] getMethods()>" is called, and the results are saved in an collection such as "java.util.List", then this test can be a potential candidate. RAM mutation operator will 1) add a statement to call getMethods() and save the results in a new List, and 2) assert the elements of the above two lists equals to each other.
Example: one RAM mutant from unix4j: org.unix4j.line.SimpleLineTest
@Test
public void CroissantMutant_NOD_RAM_UPMO_test_OrderedStringConversionMutationOperator_NoTemplate() {
+ double threshold = this.getThreshold();
+ if (Math.random() < threshold) {
HashMap<String, String> map = new HashMap<String, String>();
HashSet<String> set = new HashSet<String>();
Method[] methods1 = map.getClass().getMethods();
+ Method[] methods2 = map.getClass().getMethods();
+ Assert.assertEquals(methods1[0], methods2[0]);
+ }
}
3. Order-Dependent (OD) Mutation operators
3.1 Instance Variable Dependency (IVD)
Description: IVD flakiness happens when developers define instances to be shared among different tests inside a test class. We provide 2 types of IVD mutants, 1) one is traditional instance variable shared tests, which relies on @TestInstance Annotation in Junit5; 2) the other one leverages JobInstance in framework spring-batch, tests shares JobInstance which has similar roles as instance variable during the lifecycle of job execution. Since iDFlakies does not support @TestInstance Annotation in Junit5, we applied type 2) in the following experiments to evaluate it.
Implementation details: IVD mutation operator will first go through the units of a given test method, if Soot finds there exist objects such as Jobinstance, or instance variables in lifecycles of @TestInstance(TestInstance.Lifecycle.PER_CLASS), then this test will be a potential candidate to inject IVD flakiness. IVD mutation operator will then create one polluter and one cleaner, the polluter will change the status while the cleaner will set it back.
Example 1: IVD - 1 mutants from commons-math: org.apache.commons.math4.legacy.exception.NullArgumentExceptionTest
+ @TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class NullArgumentExceptionTest {
private int sharedVar = 10;
@Test
public void CroisasntMutant_OD_IVD_Victim(){
assertEquals(sharedVar, 10);
}
+ @Test
+ public void IVD_polluterMutant() {
+ sharedVar = 0;
+ }
+ @Test
+ public void IVD_fixedNum_cleanerMutant(){
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ sharedVar = 10;
+ }
}
Example 2: IVD - 2 mutants from commons-math: org.apache.commons.math4.legacy.exception.NullArgumentExceptionTest
@Test
public void CroisasntMutant_OD_IVD_Victim(){
assertEquals(JobRegistry.getJobInstance("job_instance"),null);
}
+ @Test
+ public void IVD_polluterMutant() {
+ JobRegistry.getInstance().addJobInstance("job_instance", new JobInstance(1L,"job_instance"));
+ }
+ @Test
+ public void IVD_fixedNum_cleanerMutant(){
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ JobRegistry.getInstance().shutdown("job_instance");
+ }
3.2 Static Variable Dependency (SVD)
Description: SVD flaky tests happen when a polluter changes static variables that are used later by a victim.
Implementation details: SVD mutation operator will first use Soot to go through the types of variables of a given test method, if there exists a static variable, then this test can be a potential candidate to mutate. SVD mutation operator will create a polluter to change the value of the static variable, and also create a cleaner to reset its status.
Example: SVD mutants from commons-csv:org.apache.commons.csv.issues.JiraCsv149Test
static class FieldClass {
static int i = 10;
}
FieldClass fieldClass = new FieldClass();
@Test
public void CroisasntMutant_OD_SVD_Victim() throws Exception {
Assert.assertEquals(fieldClass.i,10);
}
+ @Test
+ public void SVD_polluterMutant() throws Exception {
+ fieldClass.i = 0;
+ }
+ @Test
+ public void SVD_fixedNum_cleanerMutant() throws Exception {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ fieldClass.i = 10;
+ }
3.3 Third-Party Framework Dependency (TPFD)
Description: Tests could be through third-party libraries such as Mockito. Developers often mock methods rather than running them; e.g., they use Mockito to define what a specific method should return for a given input. After test execution, mocks can be queried to see what methods were called and how many times. Not resetting the state of Mockito can create an implicit dependency between tests that use Mockito.
Implementation details: TPFD mutation operator will first leverage Soot to go through all staticInvoke of a given test method if mockito.verify() is called to verify how many times a mock list adds elements, then this test can be a potential victim. To create dependencies between tests, TPFD will create a polluter and a cleaner. The polluter will add another two elements into the mock mutant, since the victim always asserts mockito.times equals the previous value, if the polluter runs before the victim, then the victim can fail the assertion; The cleaner will reset the mock list, so that if it is run before the victim, the victim can pass.
Example: TPFD mutants from commons-fileupload: org.apache.commons.fileupload2.ProgressListenerTest
@Test
public void CroissantMutant_OD_TPFD_VictimtestMockitoMockitoMutationOperator() {
mockkMutant.add("one");
mockkMutant.add("two");
((List)Mockito.verify(mockkMutant, Mockito.times(2))).add(Mockito.anyString());
}
+ @Test
+ public void TPFD_polluterMutant_testMockito_MockitoMutationOperator() {
+ mockkMutant.add("one");
+ mockkMutant.add("two");
+ List var1 = mockkMutant;
+ VerificationMode var2 = Mockito.times(2);
+ }
+ @Test
+ public void TPFD_fixedNum_cleanerMutant_MockitoMutationOperator() {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ Mockito.reset(new List[]{mockkMutant});
+ }
3.4 Cached Status Dependency (CSD)
Description: Tests may use a shared cache, and such dependencies can lead to flakiness if a polluter modifies the cache used by a victim. Croissant leverages the Caffeine library to create cache objects.
Implementation details: CSD mutation operator will first use Soot to go through the units of a given test method, if there exists "com.github.benmanes.caffeine.cache.Cache" of all interfaceInvokeExpr, at the same time, Cache calls put() method to change its status, then there can be a potential dependency between tests due to shared Cache. CSD operator will create dependencies by generating polluter and cleaner: 1) the polluter will put new elements into Cache and 2) the cleaner will clear Cache, in the brittle, an assertion will be added to check the initial state of the new elements in Cache is null. Thus the brittle can fail if it runs after the polluter, but pass if cleaner runs before it.
Example: CSD mutants from commons-codec: org.apache.commons.codec.binary.StringUtilsTest
@Test
public void CroissantMutant_OD_CSD_victimCaffeineCDMO() {
Cache cache;
+ Assertions.assertNull(cache.getIfPresent("a"));
}
+ @Test
+ public void CSD_polluterMutant_cacheTest_CaffeineCDMO() {
+ cache.put("a", "b");
+ }
+ @Test
+ public void CSD_fixedNum_cleanerMutant_CaffeineCDMO() {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ cache.invalidateAll();
+ }
3.5 Database Status Dependency (DSD)
Description: Relying on other tests to populate a shared database or failing to reset the state of the database in tearDown results in DSD flaky tests. We provide two types of DSD mutants, 1) one is implemented with @BeforeAll and @AfterAll annotations, which is more popular when developers write tests; 2) the other one is not with lifecycle annotations. Since iDFlakies does not support @BeforeAll and @AfterAll annotations in Junit5, we applied 2) with evaluation experiments.
Implementation details: DSD mutation operator will first use Soot to do static analysis, if there exists operations of database, such as "java.sql.Connection" and "java.sql.PreparedStatement", then this test can be a potential candidate to inject DSD flakiness. To create test dependency, DSD operator will add a new assertion into brittle test to search for a certain item in the database, and then create a stateStter to insert this item into database, finally create a stateUnsetter to remove the item.
Example 1: DSD - 1 mutants from commons-graph: org.apache.commons.graph.connectivity.FindConnectedComponetTestCase
+ @BeforeAll
+ public static void setUpSQL() throws ClassNotFoundException, SQLException{
+ getConnection();
+ }
+ @AfterAll
+ public static void tearDownSQL()throws ClassNotFoundException, SQLException{
+ new File("testdb.db").delete();
+ }
@Test
public void CroissantMutant_OD_DSD_brittle() throws ClassNotFoundException,SQLException {
Connection connection = con;
Assertions.assertTrue(connection.prepareStatement("SELECT * FROM user WHERE fName='FirstName*' AND
lName='LastName*' ").executeQuery().next());
}
public static void getConnection() throws ClassNotFoundException, SQLException {
Class.forName("org.sqlite.JDBC");
con = DriverManager.getConnection("jdbc:sqlite:testdb.db");
initialize();
}
+ @Test
+ public void DSD_stateSetterMutant() throws ClassNotFoundException,SQLException{
+ PreparedStatement stmt = con.prepareStatement("INSERT INTO user(id, fName, lName) values(?, ?, ?)");
+ stmt.setString(3, "LastName*");
+ stmt.setString(2, "FirstName*");
+ stmt.execute();
+ }
+ @Test
+ public void DSD_fixedNum_stateUnsetterMutant() throws ClassNotFoundException,SQLException{
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ Connection connection = con;
+ connection.prepareStatement("DELETE FROM user WHERE fName='FirstName*' AND lName='LastName*' ").execute();
+ }
Example 2: DSD - 2 mutants from commons-graph: org.apache.commons.graph.connectivity.FindConnectedComponetTestCase
@Test
public void CroissantMutant_OD_DSD_brittle() throws ClassNotFoundException,SQLException {
if (con == null) {
getConnection();
}
Connection var1 = con;
assertTrue(var1.prepareStatement("SELECT * FROM user WHERE fName='firstName' AND lName='lastName'
").executeQuery().next());
}
public static void getConnection() throws ClassNotFoundException, SQLException {
Class.forName("org.sqlite.JDBC");
if (dbName == null){
Timestamp timestamp = new Timestamp(System.currentTimeMillis());
dbName = timestamp.toString();
}
con = DriverManager.getConnection("jdbc:sqlite:" + dbName);
initialize();
}
+ @Test
+ public void DSD_stateSetterMutant() throws ClassNotFoundException,SQLException{
+ if (con == null) {
+ getConnection();
+ }
+ PreparedStatement var1 = con.prepareStatement(\"INSERT INTO user(id, fName, lName) values(?, ?, ?)");
+ var1.setString(3, "firstName*");
+ var1.setString(2, "lastName*");
+ var1.execute();
+ }
+ @Test
+ public void DSD_fixedNum_stateUnsetterMutant() throws ClassNotFoundException,SQLException {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ if (con == null) {
+ getConnection();
+ }
+ Connection var3 = con;
+ var3.prepareStatement("DELETE FROM user WHERE fName='firstName*' AND lName='lastName*' ").execute();
+ }
3.6 File Permission Dependency (FPD)
Description: For FPD, the shared state between tests is a file. With such a dependency, the polluter may modify the permission of a file the victim later attempts to access. Consequently, access to the file may fail. We also provide two types of FPD mutants, similar to DSD mutants, 1) one is implemented with @BeforeAll annotation, which is more popular when developers write tests; 2) the other one is not with lifecycle annotations. Since iDFlakies does not support @BeforeAll annotation in Junit5, we applied 2) with evaluation experiments.
Implementation details: FPD mutation operator will first use Soot to go through the units of a given test method, if there exists file read or write operations, such as "<java.io.BufferedWriter: void <init>(java.io.Writer)>", then this test can be a potential candidate to inject FPD mutation operator. To create file permission dependency between tests, FPD operator will add statements in the victim to write to a certain file, and then create a polluter to disable write permission of the file, finally create a cleaner to reset the file writable.
Example 1: FPD - 1 mutants from commons-text: org.apache.commons.text.lookup.ConstantStringLookupTest
+ @BeforeAll
+ public static void setUpWriter()throws java.io.IOException {
+ File file = new File("testWriter.txt");
+ file.setWritable(true);
+ }
@Test
public void CroissantMutant_OD_FPD_victim() throws java.io.IOException{
File file = new File("testWriter.txt");
BufferedWriter bf = new BufferedWriter(new FileWriter(file));
bf.write("hi");
bf.close();
}
+ @Test
+ public void FPD_polluterMutant()throws java.io.IOException {
+ File file = new File("testWriter.txt");
+ if (file.canWrite()){
+ file.setWritable(false);
+ }
+ }
+ @Test
+ public void FPD_fixedNum_cleanerMutant() {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ File file = new File("testWriter.txt");
+ file.setWritable(true);
+ }
Example 2: FPD - 2 mutants from commons-text: org.apache.commons.text.lookup.ConstantStringLookupTest
static Timestamp writerTimestamp = new Timestamp(System.currentTimeMillis());
static String writerFileName = writerTimestamp.toString();
@Test
public void CroissantMutant_OD_FPD_victim() throws java.io.IOException{
File file = new File(writerFileName);
FileWriter fw = new FileWriter(file);
fw.write("victim\n");
fw.close();
}
+ @Test
+ public void FPD_polluterMutant() throws java.io.IOException {
+ File file = new File(writerFileName);
+ file.setWritable(true);
+ FileWriter fw = new FileWriter(file);
+ fw.write("pollute\n");
+ fw.close();
+ file.setWritable(false);
+ }
+ @Test
+ public void FPD_fixedNum_cleanerMutant() throws Exception {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ File var3;
+ var3 = new File(writerFileName);
+ var3.setWritable(true);
+ }
3.7 Resource Availability (RA)
Description: A unit test that assumes the existence of a certain resource, e.g., a file, can become a brittle test. It can fail if run in isolation but pass if run after another test that creates the required resource. We also provide two types of FPD mutants, similar to DSD mutants, 1) one is implemented with @BeforeAll annotation, which is more popular when developers write tests; 2) the other one is not with lifecycle annotations. Since iDFlakies does not support @BeforeAll annotation in Junit5, we applied 2) with evaluation experiments.
Implementation details: RA mutation operator will first use Soot to go through the units of a given test method, if there exists a file object, then RA operator will further create a stateSetter and a stateUnsetter. The stateSetter will create a certain file, and the stateUnsetter will delete the file. RA operator will also add assertions in brittle to check if the file exists or not. In this case, test dependencies of file resources are created. If stateSetter runs before the brittle, then the brittle pass; while the stateUnsetter runs before the brittle, then it can fail.
Example 1: RA - 1 mutants from unix4j: org.unix4j.codegen.OptionSetDefinitionDataLoaderTest
+ @BeforeAll
+ public static void setUpFileStatus() {
+ File file= new File("test.txt");
+ file.delete();
+ }
@Test
public void CroissantMutant_OD_RA_brittle() {
assertTrue((new File("test.txt")).exists());
}
+ @Test
+ public void RA_stateSetterMutant() throws java.io.IOException {
+ (new File("test.txt")).createNewFile();
+ }
+ @Test
+ public void RA_fixedNum_stateUnsetterMutant() {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ File file = new File("test.txt");
+ file.delete();
+ }
Example 2: RA - 2 mutants from unix4j: org.unix4j.codegen.OptionSetDefinitionDataLoaderTest
static String fileName;
@Test
public void CroissantMutant_OD_RA_brittle() {
assertNotNull(fileName);
assertTrue((new File(fileName)).exists());
}
+ @Test
+ public void RA_stateSetterMutant() throws java.io.IOException {
+ if (fileName!=null){
+ (new File(fileName)).createNewFile();
+ }
+ else {
+ Timestamp timestamp = new Timestamp(System.currentTimeMillis());
+ fileName = timestamp.toString();
+ (new File(fileName)).createNewFile();
+ }
+ }
+ @Test
+ public void RA_fixedNum_stateUnsetterMutant() throws Exception {
+ int index = fixedNum;
+ if (index >= getCleanerCount()) {return;}
+ if (fileName != null) {
+ (new File(fileName)).delete();
+ }
+ }