RegMiner: Towards Constructing Ultra-Large Regression Dataset from Code Evolution History

(anonymous website for review)

1. Regression Visualization and Annotation tool

1.Regression List: We list the currently collected regressions.

2. Change Investigation for RFC/RIC: Given a regression, we list the regression fixing commit (RFC) in the right and the regression inducing commit (RIC) in the left. In either RFC or RIC, we show the diff information (i.e., what lines of code are changed) with a diff view.

3. Runtime Information Investigation: For either RFC or RIC, user can click the "Run" button to run the test case in the revision and see the failure information.

4. Diff Annotation (for labelling failure-inducing and fixing change): User can annotate each ground-truth failure-inducing diff (or, the fix diff) with three options, i.e., confirmed, rejected, and undecided. This function is intended for crowdsourcing.

2. Regression research paper collected from 1999-2021:

3. Detailed method matching solution:

Given two methods m_1 and m_2, we calculate their signature and body similarity as follows:

  • Signature similarity (return type, method name, and parameters):

    • Return type (sim_r): if m_1.return_type == m_2.return_type: sim_r = 1; otherwise sim_r = 0

    • Method name (sim_name):

      • We tokenize method name by camel convention or underscore, to have two token sequences nseq_1 and nseq_2, then computer the similarity by sim_name = 2*LCS(seq1, seq2)/(|seq_1| + |seq_2|)

    • Parameters (sim_param):

      • We can two sets of parameter types, i.e., Set_p1 and Set_p2, we calculate their Jaccard coefficient as the similarity, i.e., sim_param = Intersection(Set_p1 , Set_p2)/Union(Set_p1, Set_p2).

    • Finally, we have sim_sig = (sim_r + sim_name + sim_param)/3

  • Body similarity:

    • We tokenize m_1 and m_2 into two token sequences seq_1 and seq_2, we compute the similarity by 2*LCS(seq1, seq2)/(|seq_1| + |seq_2|)

4. AST rewriting rules:

See details in this page.

5. Ground-truth regression dataset

Download the details in this link.

6. Collected regressions in open-world experiment

Download the details in this link.

7. Qualitative study for open-world experiment

See details in this page.

8. Diversity analysis in open-world experiment

The table shows that the regressions collected by RegMiner are potentially more diverse than Defects4j regarding the number of libraries, the involved library topics, and the types of exceptions. Comparing to Defects4j, regressions in RegMiner use 7.3X libraries, covering 3.5X topics, and raises 1.2X more types of exceptions. (š» represents the entropy).

The detailed document can be downloaded in this link.

The Venn Diagram comparing the library number of RegMiner and Defects4j. The blue areas represent RegMiner and the green areas represent Defects4j.

The Venn Diagram comparing the topic number of RegMiner and Defects4j. The blue areas represent RegMiner and the green areas represent Defects4j.

The following figures show the word clouds of the two projects' topics. The left one is Defect4j, and the right one is RegMiner.