Course Information

Class Time: Friday 2:10-4:00 pm,  Location: MUD 337
Instructor: Baishakhi Ray, E-mail: rayb@cs.columbia.edu, Office: CEPSR 604,  Office Hour: By Appointment
Head TA: Marcus Min, E-mail: jm5025@columbia.edu, Office Hour: Friday 4:10-6:00 pm, CEPSR 6LE1

Grading

• Participation: 2.5%

• Paper Discussion Summaries: 2.5%

• Paper Presentations & Critiques:  35%

• Course Project: 60%

Schedule

Introduction (1/19, 1/26)

Slides

Link

Topic 1. Search-Based Program Synthesis (2/2, 2/9)

Guest Lecture Slides (Mark Santolucito, Barnard)

Link

Papers for Presentation, Critique, & Discussion

1. Sketching Stencils https://dl.acm.org/doi/10.1145/1250734.1250754

2. Program Synthesis from Polymorphic Refinement Types https://dl.acm.org/doi/10.1145/2908080.2908093

3. Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications https://dl.acm.org/doi/10.1145/3183713.3196891

Supplemental for Paper 1

1. Combinatorial Sketching for Finite Programs https://dl.acm.org/doi/abs/10.1145/1168857.1168907 

2. Programming by Sketching for Bit-Streaming Programs https://dl.acm.org/doi/10.1145/1065010.1065045

3. The Sketching Approach to Program Synthesis https://people.csail.mit.edu/asolar/papers/Solar-Lezama09.pdf

4. Program Synthesis by Sketching https://people.csail.mit.edu/asolar/papers/thesis.pdf

Supplemental for Paper 2

1. Refinement Types: A Tutorial https://arxiv.org/abs/2010.07763 

2. Program Synthesis by Type-Guided Abstraction Refinement https://dl.acm.org/doi/abs/10.1145/3371080

Supplemental for Paper 3

1. Leveraging Parallel Data Processing Frameworks with Verified Lifting https://arxiv.org/abs/1611.07623

2. Optimizing Data-Intensive Applications Automatically By Leveraging Parallel Data Processing Frameworks https://dl.acm.org/doi/10.1145/3035918.3056440

General Supplemental

1. Program Synthesis https://www.nowpublishers.com/article/Details/PGL-010

2. Syntax-Guided Synthesis https://www.cis.upenn.edu/~alur/SyGuS13.pdf

3. FlashFill https://dl.acm.org/doi/10.1145/1926385.1926423

4. FlashFill++ https://dl.acm.org/doi/abs/10.1145/3571226

Topic 2. Code Large Language Models (2/16, 2/23)

Guest Lecture Slides (Saikat Chakraborty, Microsoft Research)

Papers for Presentation, Critique, & Discussion

1. AlphaCode https://arxiv.org/abs/2203.07814

2. CodeGen2 https://arxiv.org/abs/2305.02309

3. Code Llama https://arxiv.org/abs/2308.12950

Supplemental for Paper 1

1. AlphaCode 2 https://storage.googleapis.com/deepmind-media/AlphaCode2/AlphaCode2_Tech_Report.pdf

2. AlphaCodium https://arxiv.org/abs/2401.08500 

Supplemental for Paper 2

1. CodeGen https://arxiv.org/abs/2203.13474

2. CodeGen2.5 https://blog.salesforceairesearch.com/codegen25/

3. InCoder https://arxiv.org/abs/2204.05999

4. SantaCoder https://arxiv.org/abs/2301.03988

Supplemental for Paper 3

1. Llama https://arxiv.org/abs/2302.13971

2. Llama 2 https://arxiv.org/abs/2307.09288

3. StarCoder https://arxiv.org/abs/2305.06161

4. DeepSeek Coder https://deepseekcoder.github.io/

5. CodeFuse https://arxiv.org/abs/2310.06266

General Supplemental

1. Large Language Models Meet NL2Code https://aclanthology.org/2023.acl-long.411/

2. A Survey on Language Models for Code https://arxiv.org/abs/2311.07989

3. Deep Learning for Source Code Modeling and Generation https://arxiv.org/abs/2002.05442

Topic 3. Alternative Code Generation Models

Papers for Presentation, Critique, & Discussion

1. UniXcoder (Unified LM) https://arxiv.org/abs/2203.03850

2. CodeT5+ (Encoder-Decoder Models) https://arxiv.org/abs/2305.07922

3. CodeFusion (Diffusion Models) https://www.microsoft.com/en-us/research/publication/codefusion-a-pre-trained-diffusion-model-for-code-generation/

Supplemental for Paper 1

1. Unified LM https://arxiv.org/abs/1905.03197

2. UniLMv2 https://arxiv.org/abs/2002.12804 

3. CugLM https://arxiv.org/abs/2012.14631

4. CuBERT https://arxiv.org/abs/2001.00059

5. CodeBERT https://arxiv.org/abs/2002.08155

Supplemental for Paper 2

1. CodeT5 https://arxiv.org/abs/2109.00859

2. PLBART https://arxiv.org/abs/2103.06333

3. SPT-Code https://arxiv.org/abs/2201.01549

4. ALBEF https://arxiv.org/abs/2107.07651

5. VLMo https://arxiv.org/abs/2111.02358

6. CoCa https://arxiv.org/abs/2205.01917

Supplemental for Paper 3

1. DDPM https://arxiv.org/abs/2006.11239

2. DALL-E 2 https://arxiv.org/abs/2204.06125

3. A Survey of Diffusion Models in Natural Language Processing https://arxiv.org/abs/2305.14671

4. Diffusion-LM https://arxiv.org/abs/2205.14217

5. DiffuSeq https://arxiv.org/abs/2210.08933

6. DiffusER https://openreview.net/forum?id=nG9RF9z1yy3

Topic 4. Evaluation of Code Models

Papers for Presentation, Critique, & Discussion

1. HumanEval/Codex (Accuracy) https://arxiv.org/abs/2107.03374

2. ReCode: Robustness Evaluation of Code Generation Models (Trustworthiness) https://arxiv.org/abs/2212.10264

3. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Practicality) https://arxiv.org/abs/2310.06770

Supplemental for Paper 1

1. Multi-lingual Evaluation of Code Generation Models https://arxiv.org/abs/2210.14868

2. HumanEvalPlus https://arxiv.org/abs/2305.01210

3. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code https://arxiv.org/abs/2302.05527

4. CodeScore: Evaluating Code Generation by Learning Code Execution https://arxiv.org/abs/2301.09043

5. A Static Evaluation of Code Completion by Large Language Models https://arxiv.org/abs/2306.03203 

Supplemental for Paper 2

1. Can ChatGPT Replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation https://arxiv.org/abs/2308.10335

2. Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain https://arxiv.org/abs/2310.14053

3. Do Large Code Models Understand Programming Concepts? A Black-box Approach https://arxiv.org/abs/2402.05980 

4. Toward Trustworthy Neural Program Synthesis https://arxiv.org/abs/2210.00848

5. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts https://arxiv.org/abs/2306.04528

Supplemental for Paper 3

1. DevBench: A Comprehensive Benchmark for Software Development https://arxiv.org/abs/2403.08604 

2. DevEval: Evaluating Code Generation in Practical Software Projects https://arxiv.org/abs/2401.06401 

3. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion https://arxiv.org/abs/2310.11248 

4. Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT https://arxiv.org/abs/2304.10778

General Supplemental

1. CodeXGLUE https://arxiv.org/abs/2102.04664 

2. XLCoST https://arxiv.org/abs/2206.08474

3. MBPP https://arxiv.org/abs/2108.07732

4. APPS https://arxiv.org/abs/2105.09938

5. CodeContest/AlphaCode https://arxiv.org/abs/2203.07814

6. DS-1000 https://arxiv.org/abs/2211.11501

7. xCodeEval https://arxiv.org/abs/2303.03004

Topic 5. Interpretability of Code Models

Papers for Presentation, Critique, & Discussion

1. What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code https://arxiv.org/abs/2202.06840

2. Evidence of Meaning in Language Models Trained on Programs https://arxiv.org/abs/2305.11169

3. Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach https://arxiv.org/abs/2310.06680

Supplemental for Paper 1

1. Explainable AI for Pre-Trained Code Models: What Do They Learn? When They Do Not Work? https://arxiv.org/abs/2211.12821

2. Naturalness of Attention: Revisiting Attention in Code Language Models https://arxiv.org/abs/2311.13508

Supplemental for Paper 2

1. Traces of Memorisation in Large Language Models for Code https://arxiv.org/abs/2312.11658

2. A Structural Probe for Finding Syntax in Word Representations https://aclanthology.org/N19-1419/

3. Designing and Interpreting Probes with Control Tasks https://arxiv.org/abs/1909.03368

Supplemental for Paper 3

1. Benchmarking Causal Study to Interpret Large Language Models for Source Code https://arxiv.org/abs/2308.12415 

2. Towards Causal Deep Learning for Vulnerability Detection https://arxiv.org/abs/2310.07958 

General Supplemental

1. Rethinking Interpretability in the Era of Large Language Models https://arxiv.org/abs/2402.01761

2. Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges https://arxiv.org/abs/2103.11251

Topic 6. Improving Code Generation I: Self-Refinement, Instruction Tuning, & Reinforcement Learning

Papers for Presentation, Critique, & Discussion

1. Teaching Large Language Models to Self-Debug (Self-Refinement) https://arxiv.org/abs/2304.05128 

2. Magicoder (Instruction Tuning) https://arxiv.org/abs/2312.02120

3. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback (Reinforcement Learning) https://arxiv.org/abs/2402.01391 

Supplemental for Paper 1

1. Is Self-Repair a Silver Bullet for Code Generation? https://arxiv.org/abs/2306.09896

2. CYCLE: Learning to Self-Refine Code Generation https://arxiv.org/abs/2403.18746 

3. LeTI: Learning to Generate from Textual Interactions https://arxiv.org/abs/2305.10314

4. Self-Refine: Iterative Refinement with Self-Feedback https://arxiv.org/abs/2303.17651

5. Large Language Models Cannot Self-Correct Reasoning Yet https://arxiv.org/abs/2310.01798

Supplemental for Paper 2

1. WizardCoder https://arxiv.org/abs/2306.08568

2. Improving Code Style for Accurate Code Generation https://openreview.net/forum?id=maRYffiUpI

3. WizardLM https://arxiv.org/abs/2304.12244

4. Let's Verify Step by Step https://arxiv.org/abs/2305.20050 

Supplemental for Paper 3

1. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning https://arxiv.org/abs/2207.01780

2. Execution-based Code Generation using Deep Reinforcement Learning https://arxiv.org/abs/2301.13816

3. RLTF: Reinforcement Learning from Unit Test Feedback https://arxiv.org/abs/2307.04349

4. Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis https://arxiv.org/abs/2310.13669 

Topic 7. Improving Code Generation II: Multimodal Learning

Papers for Presentation, Critique, & Discussion

1. GraphCodeBERT (Data Flow) https://arxiv.org/abs/2009.08366

2. CODE-MVP (Souce Code + AST + CFG + Transformation) https://arxiv.org/abs/2205.02029

3. TRACED (Execution Trace) https://arxiv.org/abs/2306.07487

4. CodeExecutor (Execution Trace) https://arxiv.org/abs/2305.05383

5. RepoCoder (Repository) https://arxiv.org/abs/2303.12570 

Supplemental for Papers 1 & 2

1. TreeBERT (AST) https://arxiv.org/abs/2105.12485

2. SynCoBERT (AST) https://arxiv.org/abs/2108.04556

3. StructCoder (AST + Data Flow) https://arxiv.org/abs/2206.05239 

4. CugLM (Type) https://arxiv.org/abs/2012.14631

5. DOBF (Transformation) https://arxiv.org/abs/2102.07492 

6. Deep Learning Type Inference https://dl.acm.org/doi/10.1145/3236024.3236051 

Supplemental for Papers 3 & 4

1. CRUXEval: https://arxiv.org/abs/2401.03065

2. CodeMind: https://arxiv.org/abs/2402.09664 

Supplemental for Paper 5

1. CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context https://arxiv.org/abs/2212.10007

2. RepoFusion: Training Code Models to Understand Your Repository https://arxiv.org/abs/2306.10998

3. Guiding Language Models of Code with Global Context using Monitors https://arxiv.org/abs/2306.10763

4. CodePlan: Repository-level Coding using LLMs and Planning https://arxiv.org/abs/2309.12499

5. A^3-CodGen: A Repository-Level Code Generation Framework for Code Reuse with Local-Aware, Global-Aware, and Third-Party-Library-Aware https://arxiv.org/abs/2312.05772

6. REPOFUSE: Repository-Level Code Completion with Fused Dual Context https://arxiv.org/abs/2402.14323

7. Generation-Augmented Retrieval for Open-domain Question Answering https://arxiv.org/abs/2009.08553

8. Query2doc: Query Expansion with Large Language Models https://arxiv.org/abs/2303.07678 

Topic 8. Neurosymbolic Synthesis

Papers for Presentation, Critique, & Discussion

1. DreamCoder https://dl.acm.org/doi/10.1145/3453483.3454080

2. Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions https://arxiv.org/abs/2212.10561

3. Prompting Is Programming: A Query Language for Large Language Models https://arxiv.org/abs/2212.06094 

Supplemental for Paper 1

1. DeepCoder https://arxiv.org/abs/1611.01989

2. RobustFill https://arxiv.org/abs/1703.07469

3. LambdaBeam https://arxiv.org/abs/2306.02049

Supplemental for Paper 2

1. Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs https://arxiv.org/abs/2210.12283

2. Communicating Natural Programs to Humans and Machines https://arxiv.org/abs/2106.07824 

Supplemental for Paper 3

1. LangChain https://python.langchain.com/docs/get_started/introduction

2. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models https://arxiv.org/abs/2109.05093

3. Synchromesh: Reliable Code Generation from Pre-trained Language Models https://arxiv.org/abs/2201.11227

4. OpenPrompt: An Open-source Framework for Prompt-learning https://arxiv.org/abs/2111.01998

5. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts https://arxiv.org/abs/2202.01279 

Topic 9. Neurosymbolic AI

Papers for Presentation, Critique, & Discussion

1. Chain of Code https://arxiv.org/abs/2312.04474

2. Scallop: A Language for Neurosymbolic Programming https://arxiv.org/abs/2304.04812

3. WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment https://arxiv.org/abs/2402.12275 

Supplemental for Paper 1

1. Binding Language Models in Symbolic Languages https://arxiv.org/abs/2210.02875

2. Program of Thought https://arxiv.org/abs/2211.12588

3. Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning https://arxiv.org/abs/2309.10814

4. ViperGPT: Visual Inference via Python Execution for Reasoning https://arxiv.org/abs/2303.08128

5. Toolformer: Language Models Can Teach Themselves to Use Tools https://arxiv.org/abs/2302.04761 

Supplemental for Paper 2

1. Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning https://proceedings.neurips.cc/paper/2021/hash/d367eef13f90793bd8121e2f675f0dc2-Abstract.html

2. DeepProbLog: Neural Probabilistic Logic Programming https://arxiv.org/abs/1805.10872

3. SATNet: Bridging Deep Learning and Logical Reasoning using a Differentiable Satisfiability Solver https://arxiv.org/abs/1905.12149 

Supplemental for Paper 3

1. World Models https://arxiv.org/abs/1803.10122

2. Neurosymbolic Grounding for Compositional World Models https://arxiv.org/abs/2310.12690

3. Language Models Meet World Models: Embodied Experiences Enhance Language Models https://arxiv.org/abs/2305.10626

4. ReAct: Synergizing Reasoning and Acting in Language Models https://arxiv.org/abs/2210.03629

5. Imitation-Projected Programmatic Reinforcement Learning https://arxiv.org/abs/1907.05431

6. Code as Policies: Language Model Programs for Embodied Control https://arxiv.org/abs/2209.07753

7. Bootstrapping Cognitive Agents with a Large Language Model https://arxiv.org/abs/2403.00810