DATASET
Task 1: You need to register to get the password to access the Seed Data
Task 2: Download the Conala dataset
Registration: The Tab - Track Registration Form
TASK 1
Seed Dataset: A dataset of code and comment pairs, along with tools for comment analysis, are provided to participants. A set of 9048 comments (from Github) with comment text, surrounding code snippets, and a label that specifies whether the comment is useful or not. Sample data is shown below.
Ground Truth Generation Process for the Seed Dataset:For every comment, a label (Useful or Not Useful) has been generated by a team of 14 annotators. Every comment has been annotated by 2 annotators with a kappa (κ) value of 0.734 (Cohen’s metric). The annotation process has been supervised through weekly meetings and brainstorming sessions and peer review. Out of the total 16,000 comments, 2,285 comments were annotated by every individual annotator. A total of 156 man-hours were required to complete the annotation process.
Generated Dataset: A dataset of code and comment pairs extracted from github with labels - (useful, not useful) generated using any LLM models like GPT4 or Bard. This is to be generated by the participants.
TASK 2
A dataset of 40000 code and intent pairs from the CoNaLa: The Code/Natural Language Challenge dataset ({https://conala-corpus.github.io/})
An example code comment snippet
{
"question_id": 36875258,
"intent": "copying one file's contents to another in python",
"rewritten_intent": "copy the content of file 'file.txt' to file 'file2.txt'",
"snippet": "shutil.copy('file.txt', 'file2.txt')",
}