Course Information

Class Time: Thursday 2:10-4:00 pm,  Location: 963 EXT SCHERMERHORN HALL
Instructor: Baishakhi Ray, E-mail: rayb@cs.columbia.edu, Office: CEPSR 604,  Office Hour: By Appointment
Head TA: Bowen Yang, E-mail: by2365@columbia.edu, Office Hour: By Appointment

Grading

• Participation: 5%

• Paper Presentations & Critiques:  35%

• Course Project: 60%

Schedule

Introduction

Slides

Link

Guest Lecture Slides (Robin, Microsoft Research)

Topic 1. Code Large Language Models 

Papers for Presentation, Critique, & Discussion

Popular Generative Model For Code

1. Code Llama https://arxiv.org/abs/2308.12950

2. StarCoder https://arxiv.org/abs/2305.06161

3. DeepSeek Coder https://arxiv.org/pdf/2401.14196

Alternative Generative Models For Code

4. CodeSage https://arxiv.org/pdf/2402.01935

Supplemental for Paper 1

1. Llama https://arxiv.org/abs/2302.13971

2. Llama 2 https://arxiv.org/abs/2307.09288

3. CodeFuse https://arxiv.org/abs/2310.06266

4. Casual Masking https://arxiv.org/abs/2201.07520

Supplemental for Paper 2

1. SantaCoder https://arxiv.org/abs/2301.03988

2. The Stack https://arxiv.org/pdf/2211.15533

3. 8K token context length https://arxiv.org/html/2402.17463v2

4. Fill-in-the-middle https://arxiv.org/pdf/2207.14255

5. Multi-Query-Attention https://arxiv.org/pdf/1911.02150

6. BERT https://arxiv.org/abs/1810.04805

Supplemental for Paper 3

1. DeepSeek Coder Repo https://deepseekcoder.github.io/

2. CodeFuse https://arxiv.org/abs/2310.06266

3. Fill-in-the-middle https://arxiv.org/pdf/2207.14255

4. CodeGen https://arxiv.org/abs/2203.13474

5. BPE https://arxiv.org/abs/1508.07909

6. RoPE https://arxiv.org/abs/2104.09864

Supplemental for Paper 4

1. SimCSE https://aclanthology.org/2021.emnlp-main.552/

2. UniXCoder https://aclanthology.org/2022.acl-long.499/

3. DOBF https://openreview.net/forum?id=3ez9BSHTNT

General Supplemental

1. Large Language Models Meet NL2Code https://aclanthology.org/2023.acl-long.411/

2. A Survey on Language Models for Code https://arxiv.org/abs/2311.07989

3. Deep Learning for Source Code Modeling and Generation https://arxiv.org/abs/2002.05442

4. CodeT5+ (Encoder-Decoder Models) https://arxiv.org/abs/2305.07922

5. CodeFusion (Diffusion Models) https://www.microsoft.com/en-us/research/publication/codefusion-a-pre-trained-diffusion-model-for-code-generation/

6. DALL-E 2 https://arxiv.org/abs/2204.06125

Topic 2. Evaluation of Code Models

Papers for Presentation, Critique, & Discussion

1. LiveCodeBench  https://arxiv.org/abs/2403.07974

2. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?  https://arxiv.org/abs/2310.06770

Supplemental for Paper 1

1. LiveCodeBenchRepo https://livecodebench.github.io/

2. HumanEval/Codex (Accuracy) https://arxiv.org/abs/2107.03374

3. ReCode: Robustness Evaluation of Code Generation Models (Trustworthiness) https://arxiv.org/abs/2212.10264

4. MBPP https://arxiv.org/abs/2108.07732

Supplemental for Paper 2

1. DevBench: A Comprehensive Benchmark for Software Development https://arxiv.org/abs/2403.08604 

2. DevEval: Evaluating Code Generation in Practical Software Projects https://arxiv.org/abs/2401.06401 

3. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion https://arxiv.org/abs/2310.11248 

4. Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT https://arxiv.org/abs/2304.10778

General Supplemental

1. ReCode: Robustness Evaluation of Code Generation Models (Trustworthiness) https://arxiv.org/abs/2212.10264

2. CodeXGLUE https://arxiv.org/abs/2102.04664 

3. XLCoST https://arxiv.org/abs/2206.08474

4. APPS https://arxiv.org/abs/2105.09938

5. CodeContest/AlphaCode https://arxiv.org/abs/2203.07814

6. DS-1000 https://arxiv.org/abs/2211.11501

7. xCodeEval https://arxiv.org/abs/2303.03004

8. BigCode Eval Harness https://github.com/bigcode-project/bigcode-evaluation-harness

9. BigCodeBench https://huggingface.co/blog/leaderboard-bigcodebench

10. LMSYS Coding https://lmarena.ai/?leaderboard

Topic 3. Agents

Papers for Presentation, Critique, & Discussion

1. MASAI https://arxiv.org/pdf/2406.11638 

2. AutoCodeRover https://arxiv.org/abs/2404.05427 

General Supplemental

1. TBD

Topic 4. Improving Code Generation

Papers for Presentation, Critique, & Discussion

1. CODEDPO  https://arxiv.org/pdf/2410.05605

2. LintSeq https://arxiv.org/pdf/2410.02749

3. GAD https://arxiv.org/pdf/2405.21047

4. PIE https://arxiv.org/pdf/2407.03157

General Supplements

1. CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context https://arxiv.org/abs/2212.10007

2. RepoFusion: Training Code Models to Understand Your Repository https://arxiv.org/abs/2306.10998

3. Guiding Language Models of Code with Global Context using Monitors https://arxiv.org/abs/2306.10763

4. CodePlan: Repository-level Coding using LLMs and Planning https://arxiv.org/abs/2309.12499

5. A^3-CodGen: A Repository-Level Code Generation Framework for Code Reuse with Local-Aware, Global-Aware, and Third-Party-Library-Aware https://arxiv.org/abs/2312.05772

6. REPOFUSE: Repository-Level Code Completion with Fused Dual Context https://arxiv.org/abs/2402.14323

7. Generation-Augmented Retrieval for Open-domain Question Answering https://arxiv.org/abs/2009.08553

8. Query2doc: Query Expansion with Large Language Models https://arxiv.org/abs/2303.07678 

Topic 5. Interpretability of Code Models

Papers for Presentation, Critique, & Discussion

1. Explainable AI  https://www.mdpi.com/1099-4300/23/1/18

2. Explainable AI  https://www.mdpi.com/1099-4300/23/1/18

Note: Explainable AI is a heavy paper, both groups of people should focus on the same paper.

General Supplemental

1. Rethinking Interpretability in the Era of Large Language Models https://arxiv.org/abs/2402.01761

2. Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges https://arxiv.org/abs/2103.11251

3. Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach https://arxiv.org/abs/2310.06680

4. Benchmarking Causal Study to Interpret Large Language Models for Source Code https://arxiv.org/abs/2308.12415 

5. Towards Causal Deep Learning for Vulnerability Detection https://arxiv.org/abs/2310.07958 

Topic 6. Applications

Papers for Presentation, Critique, & Discussion

1. SafeCoder https://arxiv.org/pdf/2402.09497

2. Coeditor https://www.cs.utexas.edu/~isil/coeditor.pdf

Topic 7. Neurosymbolic AI

Papers for Presentation, Critique, & Discussion

1. COSMOS https://arxiv.org/pdf/2310.12690

2. LLM on Program Invariants https://drive.google.com/file/d/1t8Veh-JX7xCRtcHcHPmFtnfM38zXK31D/view