Course Description

Welcome to CS2950-K Special Topics in Computational Linguistics! This semester we will look at research at the intersection of natural language processing (NLP) and computer vision (CV). Over the past few years, the tremendous progress in NLP and CV has enabled researchers to tackle challenging research problems that require both visual perception and language understanding, from generating rich descriptions of images and videos, to teaching robots to follow human instructions. In this course, we introduce the students to the landscape of vision & language research, and offer a quick recap of recent deep learning methods for visual modeling and language modeling. We then go over several important topics on vision and language, such as image captioning, visual question answering, visual language navigation, and multimodal learning. The course is organized as a combination of paper reading, student presentation, and invited guest lectures.

Instructors: Eugene Charniak and Chen Sun

Class time: Monday, Wednesday and Friday, 2:00-2:50pm, online.

Office hours: By appointment, please email the instructors.

Course schedule: Schedule

Learning goals: Students who complete this course will:

• Be familiar with state-of-the-art CV and NLP technologies, their common building blocks, and why we want to jointly model language and vision.

• Develop skills for critically reading research papers, identifying their high-level insights and limitations.

• Build in-depth knowledge in one or more areas of active research directions for vision and language.

Textbook: None. Papers and background reading materials are available online.

Piazza: http://piazza.com/brown/fall2020/csci2950k

Recommended background: Brown Banner has the prerequisite as CS146 (Computational Linguistics) or permission from the instructor. Permission will be pretty much automatic for students with a background in Deep Learning (e.g., CS1470) and background in either Computer Vision (e.g. CS1430)) or NLP.

Grading policy: Tentative break down of final grades as follows:

• 20% Pre-class paper discussion (posted 24-hr before each class)

• 20% Paper presentation (paper signup)

• 30% Coding assignments (3 in total)

• 30% Final project & presentation

Students will be given 5 "late day" credits for assignments handed in after the official due date. These credits will be given expressly for unavoidable lateness such as that for medical or professional reasons. You will not be asked to justify their use, but be forewarned that even with a dean's note no extra extensions will be given short of truly unusual circumstances.

General course policies: see this page.