Embedded Files
RQ2: How do the bug fixes change when we add mutations into the buggy code? This investigates the LLM-based repairs when they do not have enough context or the variable names are changed.
Results of bug fixes after mutating the buggy programs: Table 3. (Supplement results not presented in paper)
Here are the specific bugs for Defects4J under each setting:
GPT-3.5-Mask:
Plausible: 'Chart-11', 'Chart-13', 'Chart-3', 'Chart-9', 'Closure-122', 'Closure-126', 'Closure-18', 'Closure-31', 'Closure-62', 'Closure-70', 'Closure-73', 'Closure-92', 'Lang-14', 'Lang-21', 'Lang-22', 'Lang-33', 'Lang-45', 'Lang-58', 'Lang-59', 'Math-30', 'Math-50', 'Math-57', 'Math-80', 'Math-82', 'Math-94', 'Mockito-29', 'Time-15'
Correct: 'Lang-21', 'Lang-59', 'Closure-126', 'Chart-11', 'Closure-31', 'Math-94', 'Chart-9', 'Closure-62', 'Time-15', 'Lang-33', 'Math-82', 'Mockito-29', 'Closure-18', 'Lang-14'
Type-1: 'Closure-62', 'Math-94', 'Chart-11', 'Closure-126', 'Closure-31', 'Math-82', 'Lang-21', 'Closure-18'
Type-2: 'Lang-59', 'Time-15'
Type-3: 'Chart-9', 'Lang-33', 'Mockito-29'
Type-4: 'Lang-14'
GPT-3.5-Omit:
Plausible: 'Chart-11', 'Chart-13', 'Chart-9', 'Closure-11', 'Closure-125', 'Closure-18', 'Closure-31', 'Closure-38', 'Closure-73', 'Closure-92', 'Lang-14', 'Lang-21', 'Lang-22', 'Lang-33', 'Lang-59', 'Math-30', 'Math-50', 'Math-53', 'Math-82', 'Math-85', 'Math-94', 'Mockito-29'
Correct: 'Lang-14', 'Mockito-29', 'Closure-31', 'Closure-73', 'Math-94', 'Chart-11', 'Math-82', 'Lang-21', 'Lang-33', 'Chart-9', 'Math-85'
Type-1: 'Math-85', 'Mockito-29', 'Math-94', 'Chart-9', 'Closure-73', 'Chart-11', 'Lang-21', 'Math-82'
Type-2: 'Closure-31'
Type-3: 'Lang-33'
Type-4: 'Lang-14'
GPT-3.5-Rename:
Plausible: 'Chart-1', 'Chart-11', 'Chart-20', 'Chart-24', 'Closure-11', 'Closure-126', 'Closure-57', 'Closure-62', 'Closure-92', 'Lang-21', 'Lang-24', 'Lang-43', 'Lang-45', 'Lang-59', 'Lang-61', 'Math-41', 'Math-57', 'Math-80', 'Math-82', 'Math-85', 'Math-94', 'Time-15', 'Closure-70', 'Lang-33', 'Math-59', 'Mockito-29', 'Lang-16', 'Math-32'
Correct: 'Closure-57', 'Math-85', 'Math-80', 'Chart-20', 'Math-41', 'Chart-11', 'Math-94', 'Chart-1', 'Lang-33', 'Closure-11', 'Closure-70', 'Lang-43', 'Closure-92', 'Lang-61', 'Chart-24', 'Math-59', 'Lang-21', 'Time-15', 'Lang-24', 'Lang-59', 'Lang-16', 'Math-82', 'Math-57'
Type-1: 'Chart-1', 'Chart-20', 'Lang-24'
Type-2: 'Lang-29'
Type-3: -
Type-4: -
GPT-3.5-Swap:
Plausible: 'Chart-1', 'Chart-11', 'Chart-20', 'Chart-24', 'Closure-57', 'Closure-62', 'Lang-21', 'Lang-29', 'Lang-43', 'Lang-45', 'Lang-59', 'Lang-61', 'Math-41', 'Math-57', 'Math-80', 'Math-82', 'Math-85', 'Math-94', 'Time-15', 'Closure-70', 'Lang-33', 'Math-59', 'Math-32'
Correct: 'Lang-33', 'Lang-43', 'Lang-59', 'Math-94', 'Chart-1', 'Closure-57', 'Closure-70', 'Chart-11', 'Lang-61', 'Math-57', 'Math-59', 'Time-15', 'Lang-29', 'Lang-21', 'Chart-24', 'Math-80', 'Closure-62', 'Math-41', 'Math-85', 'Chart-20'
Type-1: 'Chart-20'
Type-2: 'Closure-126', 'Closure-129', 'Lang-24', 'Closure-97'
Type-3: -
Type-4: -
CodeLlama-Mask:
Plausible: 'Math-96', 'Chart-13', 'Math-80', 'Math-85', 'Chart-1', 'Lang-33', 'Lang-29', 'Math-56', 'Mockito-29', 'Time-19', 'Math-30', 'Lang-59', 'Math-82', 'Closure-62', 'Closure-92', 'Closure-11', 'Math-94', 'Lang-21', 'Closure-18', 'Closure-126', 'Closure-31', 'Chart-11', 'Closure-65', 'Closure-73', 'Lang-58', 'Math-50', 'Math-11'
Correct: 'Lang-59', 'Math-80', 'Math-94', 'Closure-73', 'Chart-1', 'Closure-31', 'Lang-33', 'Math-85', 'Closure-126', 'Chart-13', 'Math-96', 'Closure-18', 'Time-19', 'Closure-11', 'Math-30', 'Closure-65', 'Lang-21', 'Chart-11', 'Closure-92', 'Math-82', 'Closure-62', 'Math-11', 'Lang-29'
Type-1: 'Closure-126', 'Closure-18', 'Chart-11', 'Chart-13', 'Lang-59', 'Closure-62', 'Math-30', 'Lang-21', 'Math-85', 'Math-94', 'Math-82', 'Closure-73', 'Closure-92', 'Chart-1', 'Lang-29', 'Math-80', 'Time-19', 'Closure-65'
Type-2: 'Closure-11'
Type-3: 'Lang-33'
Type-4: 'Closure-31', 'Math-11', 'Math-96'
CodeLlama-Omit:
Plausible: 'Lang-58', 'Math-85', 'Lang-59', 'Math-11', 'Chart-1', 'Math-30', 'Time-19', 'Closure-18', 'Lang-29', 'Closure-31', 'Lang-39', 'Closure-62', 'Math-50', 'Math-96', 'Mockito-29', 'Math-94', 'Closure-126', 'Closure-92', 'Lang-33', 'Chart-11', 'Lang-21', 'Lang-22', 'Closure-70', 'Chart-13', 'Lang-38', 'Time-15', 'Math-82', 'Closure-11'
Correct: 'Closure-62', 'Math-30', 'Closure-70', 'Math-96', 'Chart-11', 'Lang-59', 'Lang-21', 'Closure-18', 'Math-94', 'Chart-1', 'Closure-126', 'Lang-29', 'Lang-33', 'Mockito-29', 'Math-85', 'Math-82', 'Math-11'
Type-1: 'Closure-70', 'Chart-1', 'Lang-29', 'Mockito-29', 'Math-94', 'Math-30', 'Closure-18', 'Lang-33', 'Math-82', 'Closure-62', 'Math-85', 'Lang-21', 'Chart-11', 'Lang-59'
Type-2: 'Closure-126'
Type-3: -
Type-4: 'Math-11', 'Math-96'
CodeLlama-Rename:
Plausible: 'Math-50', 'Chart-13', 'Lang-29', 'Math-96', 'Math-94', 'Closure-126', 'Lang-57', 'Math-82', 'Closure-10', 'Closure-123', 'Lang-59', 'Closure-11', 'Chart-1', 'Math-85', 'Lang-21', 'Math-57', 'Lang-24', 'Math-70', 'Closure-73', 'Closure-62', 'Math-41', 'Chart-20', 'Chart-11', 'Math-30', 'Closure-57', 'Math-59', 'Chart-8'
Correct: 'Chart-1', 'Lang-29', 'Math-30', 'Closure-57', 'Closure-73', 'Closure-123', 'Lang-24', 'Math-85', 'Math-59', 'Chart-8', 'Math-57', 'Math-41', 'Lang-21', 'Chart-13', 'Closure-11', 'Closure-62', 'Lang-59', 'Math-70', 'Math-94', 'Chart-11', 'Math-96', 'Math-82', 'Chart-20'
Type-1: 'Chart-1'
Type-2: 'Math-11'
Type-3: -
Type-4: -
CodeLlama-Swap:
Plausible: 'Math-59', 'Closure-126', 'Math-11', 'Chart-1', 'Lang-24', 'Math-82', 'Math-57', 'Chart-20', 'Math-70', 'Closure-11', 'Lang-45', 'Lang-21', 'Lang-29', 'Lang-57', 'Math-96', 'Closure-62', 'Closure-10', 'Closure-125', 'Math-30', 'Closure-123', 'Math-50', 'Chart-11', 'Chart-8', 'Closure-73', 'Math-85', 'Math-94', 'Lang-59', 'Chart-13'
Correct: 'Math-59', 'Chart-11', 'Math-94', 'Lang-59', 'Lang-21', 'Math-85', 'Math-70', 'Closure-123', 'Lang-29', 'Chart-13', 'Math-30', 'Closure-73', 'Closure-62', 'Math-11', 'Chart-1', 'Closure-11', 'Lang-24', 'Chart-8', 'Math-82', 'Math-57', 'Chart-20', 'Lang-45'
Type-1: 'Lang-24'
Type-2: -
Type-3: -
Type-4: -
Memorized bugs in GPT-3.5:
'Math-85', 'Chart-1', 'Lang-59', 'Chart-11', 'Mockito-29', 'Closure-73', 'Closure-18', 'Closure-62', 'Chart-20', 'Lang-21', 'Lang-24', 'Math-94', 'Chart-9', 'Closure-126', 'Closure-31', 'Math-82', 'Time-15'
Memorized bugs in Codellama:
'Lang-33', 'Math-85', 'Chart-1', 'Chart-11', 'Closure-73', 'Closure-70', 'Closure-11', 'Chart-13', 'Math-94', 'Math-82', 'Closure-65', 'Lang-29', 'Closure-62', 'Math-30', 'Closure-126', 'Closure-18', 'Lang-59', 'Lang-24', 'Lang-21'
The overlap (14 for GPT-3.5 and 16 for CodeLlama) of potential memorized cases (in Defects4J) identified by RQ1 and RQ2 is as following:
Here are the specific bugs for GitBug-Java under each setting:
GPT-3.5-Mask:
Plausible: TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563, crawler-commons-crawler-commons-2c2cb3bf7a95, fishercoder1534-Leetcode-2110c6b023b7, traccar-traccar-1b8993293646, traccar-traccar-6f59f756a7d3, traccar-traccar-d244b4bc4999, traccar-traccar-fdbd269b9b99, traccar-traccar-3331593759a2, traccar-traccar-a9c311855a49
Correct: TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563, crawler-commons-crawler-commons-2c2cb3bf7a95, fishercoder1534-Leetcode-2110c6b023b7, traccar-traccar-1b8993293646, traccar-traccar-d244b4bc4999
Type-1: crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999
Type-2: -
Type-3: -
Type-4: -
GPT-3.5-Omit:
Plausible: TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-6f59f756a7d3, traccar-traccar-7ce4fb9a628f, traccar-traccar-d244b4bc4999, traccar-traccar-fdbd269b9b99, traccar-traccar-3331593759a2
Correct: TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999, traccar-traccar-fdbd269b9b99, traccar-traccar-7ce4fb9a628f
Type-1: crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999
Type-2: -
Type-3: -
Type-4: -
GPT-3.5-Rename:
Plausible: TheAlgorithms-Java-e5c7a08874a6, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-5a1a8d9192ee, traccar-traccar-6f59f756a7d3, traccar-traccar-7ce4fb9a628f, traccar-traccar-d244b4bc4999, traccar-traccar-fdbd269b9b99, TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563, fishercoder1534-Leetcode-2110c6b023b7, traccar-traccar-1b8993293646, traccar-traccar-3331593759a2
Correct: TheAlgorithms-Java-e5c7a08874a6, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-5a1a8d9192ee, traccar-traccar-6f59f756a7d3, traccar-traccar-7ce4fb9a628f, traccar-traccar-d244b4bc4999, traccar-traccar-fdbd269b9b99, TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563, fishercoder1534-Leetcode-2110c6b023b7, traccar-traccar-1b8993293646
Type-1: -
Type-2: -
Type-3: -
Type-4: -
GPT-3.5-Swap:
Plausible: TheAlgorithms-Java-e5c7a08874a6, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-1b8993293646, traccar-traccar-6f59f756a7d3, traccar-traccar-d244b4bc4999, traccar-traccar-5a1a8d9192ee, traccar-traccar-7ce4fb9a628f, traccar-traccar-fdbd269b9b99, TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563, fishercoder1534-Leetcode-2110c6b023b7, traccar-traccar-3331593759a2
Correct: TheAlgorithms-Java-e5c7a08874a6, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-1b8993293646, traccar-traccar-6f59f756a7d3, traccar-traccar-d244b4bc4999, traccar-traccar-5a1a8d9192ee, traccar-traccar-7ce4fb9a628f, traccar-traccar-fdbd269b9b99, TheAlgorithms-Java-96c1a96647c9, TheAlgorithms-Java-a3a2d845d563
Type-1: -
Type-2: -
Type-3: -
Type-4: -
CodeLlama-Mask:
Plausible: TheAlgorithms-Java-a3a2d845d563, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999, traccar-traccar-3331593759a2, traccar-traccar-fdbd269b9b99, traccar-traccar-a9c311855a49, TheAlgorithms-Java-e5c7a08874a6, mthmulders-mcs-eff905bef8d8, traccar-traccar-6f59f756a7d3, traccar-traccar-7ce4fb9a628f, traccar-traccar-8de9a36abef8
Correct: crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999, traccar-traccar-3331593759a2, traccar-traccar-fdbd269b9b99, traccar-traccar-a9c311855a49, crawler-commons-crawler-commons-2c2cb3bf7a95, mthmulders-mcs-eff905bef8d8
Type-1: crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999
Type-2: -
Type-3: -
Type-4: -
CodeLlama-Omit:
Plausible: crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999, traccar-traccar-6f59f756a7d3, traccar-traccar-7ce4fb9a628f, traccar-traccar-8de9a36abef8, traccar-traccar-a9c311855a49, mthmulders-mcs-eff905bef8d8, traccar-traccar-fdbd269b9b99, traccar-traccar-3331593759a2
Correct: crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999, traccar-traccar-7ce4fb9a628f, traccar-traccar-a9c311855a49, traccar-traccar-3331593759a2, traccar-traccar-a9c311855a49, mthmulders-mcs-eff905bef8d8
Type-1: crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-d244b4bc4999
Type-2: -
Type-3: -
Type-4: -
CodeLlama-Rename:
Plausible: mthmulders-mcs-eff905bef8d8, traccar-traccar-d244b4bc4999, traccar-traccar-7ce4fb9a628f, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-8de9a36abef8, traccar-traccar-fdbd269b9b99, traccar-traccar-6f59f756a7d3, traccar-traccar-3331593759a2, traccar-traccar-a9c311855a49
Correct: mthmulders-mcs-eff905bef8d8, traccar-traccar-d244b4bc4999, traccar-traccar-7ce4fb9a628f, crawler-commons-crawler-commons-2c2cb3bf7a95, traccar-traccar-8de9a36abef8, traccar-traccar-fdbd269b9b99, traccar-traccar-6f59f756a7d3, traccar-traccar-3331593759a2, traccar-traccar-a9c311855a49
Type-1: -
Type-2: -
Type-3: -
Type-4: -
CodeLlama-Swap:
Plausible: crawler-commons-crawler-commons-2c2cb3bf7a95, TheAlgorithms-Java-e5c7a08874a6, traccar-traccar-fdbd269b9b99, mthmulders-mcs-eff905bef8d8, traccar-traccar-8de9a36abef8, traccar-traccar-6f59f756a7d3, traccar-traccar-a9c311855a49, traccar-traccar-7ce4fb9a628f, traccar-traccar-3331593759a2, traccar-traccar-d244b4bc4999
Correct: crawler-commons-crawler-commons-2c2cb3bf7a95, TheAlgorithms-Java-e5c7a08874a6, traccar-traccar-fdbd269b9b99, traccar-traccar-8de9a36abef8, traccar-traccar-6f59f756a7d3, traccar-traccar-a9c311855a49, traccar-traccar-7ce4fb9a628f, traccar-traccar-3331593759a2, traccar-traccar-d244b4bc4999
Type-1: -
Type-2: -
Type-3: -
Type-4: -
Page updated
Google Sites
Report abuse