Task1: Generative AI based Software Metadata Classification

A binary code comment quality classification model needs to be augmented with generated code and comment pairs that can improve the accuracy of the model.

Input: a) 9048 pairs of code and comments written in C, labeled as either Useful or Not Useful.

b) Code and Comment Pairs, written in C with generated labels of useful / not useful using any Large Language Model Architecture

Output: Classification model with and without the new set of code comment pairs and generated labels

• We will provide a parent LLM and a RAG (Retrieval-Augmented Generation) architecture specifically designed to generate metadata for C source code. The RAG system will include question-answer pairs and comments focused on understanding and interpreting C code snippets.

• Participants will be required to scrape C code and associated comments from codebases and use this data to build a small language model from scratch. This small model can then be fine-tuned using responses generated by the larger parent LLM.

• The generated comments for a set of 50 C programs will be evaluated based on their semantic similarity to human-written reference comments for the same code.