COS 597F: Embodied Language Understanding

Prerequisites: COS 484/584 or equivalent

Class times: MW 11:00 - 12:20pm

Location: FC 009

Instructor: Karthik Narasimhan

Office hours: by appointment

Presentation signup form: https://forms.gle/5pw3ejaRgGTY7u2H9 (please sign up by 10pm on Thursday, Sept 2)

Idea bank form: https://forms.gle/YK3v23bUPCJmmWUG7

Anonymous feedback form: https://forms.gle/hRmses2vKwa1yZsN7

Recent advances in machine learning have ushered in exciting developments in natural language processing (NLP) but comprehending language requires an understanding of contextual signals beyond just reading text, such as visual stimuli, interactive signals from an agent's environment, or even the mental state of other agents. This seminar studies the latest research in the area of embodied language understanding, to develop artificial agents that can understand and generate natural language in the context of extra-linguistic signals. The course revolves around papers assigned for reading, and students complete a project.

Goals:

This course is intended for you to learn and explore some of the cutting edge research in embodied language understanding and prepare you to perform further research in these directions. The course will help you:

  1. Learn more about the different sub-areas of research that are relevant and useful for building systems for embodied language understanding.

  2. Improve oral and written scientific presentation skills as well as providing constructive feedback

  3. Practice different parts of the research cycle including reading and understanding prior literature, brainstorming ideas, choosing a research problem, model and algorithm design, and empirical evaluation.

Course structure and grading breakdown:

There are four main components to the course (with corresponding grading weights):

  1. Class participation (30%): Each class will cover ~1-2 papers, which will be posted online on Perusall. You are expected to read these papers before class, add in comments (at least five high quality ones) on Perusall (for class participation). Please come to the class prepared with several points that will substantially contribute to the group discussion. Your participation grade will be determined based on attendance and more importantly, substantial contributions to paper discussions both on Perusall and in class.

  2. Presentation (20%): You will present (about once/twice during the semester, but may be more depending on enrollment and whether you have a partner) on the ~1-2 papers assigned for a particular day. We will do scheduling and signups at the beginning of the semester. The goal is to educate the others in the class about the topic, so do think about how to best cover the material, do a good job with slides, and be prepared for lots of questions. You will receive feedback on your presentation from 2-3 classmates.

  • Papers have already been chosen for each topic. You are still welcome to suggest relevant papers that you like to present (and coordinate with the instructor). It is your job to decide what to cover in the lecture and how to divide the work with your partner.

  • You are also required to meet with the instructor before the lecture (Monday 4:30-5pm for Wednesday lectures and Thursday 4:30-5pm for Monday lectures). Please send your draft slides via email before the meeting and we will go over your slides during the meeting.

  1. Lecture feedback (5%): You will get the chance to provide written feedback to the presenter(s) on their lecture (around 2-3 lectures overall), around 1 page in length, commenting on the content, delivery, clarity, completeness, etc. No need for complete sentences, bullet points are fine, but feedback should be thorough and constructive. These notes should be emailed to the presenter(s) and the instructor within a day of the lecture. We’ll sign up for these as we do the lecture scheduling.

  2. Course project (45%): The final component of the course involves a research project on a topic of your choosing. You can choose to either do one individually or in a team of two (in rare cases, larger teams may be allowed with prior permission of the instructor). You will be required to submit a written paper at the end of the course. Project guidelines can be found here.

Useful background reading:

  1. Ian Goodfellow and Yoshua Bengio and Aaron Courville, Deep Learning

  2. Jacob Eisenstein. Natural Language Processing

  3. Richard Sutton and Andrew Barto, Reinforcement Learning: An introduction

Schedule:

COS 597F schedule (with assignments)