Abstract
Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse application domains. However, their potential in solving intricate problems within competitive environments, such as Kaggle competitions, remains relatively unexplored. This paper presents the first investigation into leveraging LLMs to autonomously build deep learning solutions specifically for Kaggle competitions. Our proposed method, CompeteRAG, employs a Retrieval-Augmented Generation (RAG) framework, utilizing top-voted Jupyter notebooks as exemplary templates to construct targeted solutions. If initial code-generation attempts result in incomplete or erroneous outputs, CompeteRAG dynamically analyzes these errors and iteratively refines the generated code. We rigorously evaluate CompeteRAG's efficacy using two state-of-the-art (SoTA) LLMs specialized in code generation: GPT-04-mini-high and DeepSeek-R1. Additionally, we introduce a comprehensive dataset comprising highly-rated Jupyter notebooks that build DNN solutions for Kaggle competitions. Our study provides critical insights into the strengths and inherent limitations of current LLM methodologies when confronting complex structured-data tasks common in competitive environments. Experimental results underscore CompeteRAG's robust capability in generating effective deep neural network (DNN) solutions within Kaggle's competitive landscape. Additionally, we introduce a comprehensive dataset comprising highly-rated Jupyter notebooks that build DNN solutions for Kaggle competitions. This dataset serves as a valuable resource for enhancing future code-generation techniques. Our study provides critical insights into the strengths and inherent limitations of current LLM methodologies when confronting complex structured-data tasks common in competitive environments.