About

The class will cover representation learning / induction methods for natural language processing problems. Various kinds of representations will be considered: from discrete and structured representations (e.g., hidden hierarchical structure of text) to real-valued vectors (as in deep learning). The main focus will be on problems from natural language processing but most of the methods we will consider will have applications in other domains (e.g., bioinformatics, vision, information retrieval, etc). From attenders we expect some interest in natural language processing and machine learning (incl. deep learning).

Examples of problems we will consider are question answering, machine translation, relation extraction, syntactic and semantic parsing, morphological analysis, learning to play text-based games, etc. Unlike NLP 1 class which covered mostly classic NLP techniques, in this course we will study modern methods. Some of these methods result in state-of-the-art performance on the tasks (as neural machine translation models) but some of them are more experimental and promising in a longer run (e.g., memory networks). The goal is to give you a perspective on modern research in statistical NLP.

Though the title contains the term "unsupervised", we will treat it in a rather general way: for example, any representation learning approach is in some sense unsupervised as features are not pre-specified by a model designer but rather induced during the learning process. So it is easier to say what this class is not about: it is not about supervised learning of (log-)linear models (largely focus of the NLP1 class many of you took). There will be two parts to the class:

  • Bayesian and non-parametric Bayesian modeling
  • Representation learning / deep learning methods

Besides lectures, you will also present papers yourself (chosen by you), as well as work on some programming project.

Most lectures will consist of two parts: a proper lecture for 45-60 minutes (incl. discussion) as well as a paper presentation and discussion for the remaining 30-45 minutes. We will try to make sure that the two parts are on related topics and complement each other.

Pre-requisites for this class: NLP1 and ML1 (or equivalent classes).

We will use Blackboard for announcements. Please make sure they reach you.

Instructors

Ivan Titov (coordinator) and guest lecturers

Assistants

Joost Bastings, Ehsan Khoddam, Sara Veldhoen

Time and Location

Please use Datanose for up-to-date information on timing and locations

Office Hours

Please send us an e-mail to arrange a meeting (preferably meetings are on Wednesday afternoon).

Course Requirements

Requirements for the course are:

  • Present a paper to the class (30 minute presentation + 15 minutes discussion) (in pairs)
  • Write a very short review (1-2 paragraphs) for at least one paper including in reading for each class (you can skip once; no need to tell us)
  • Implement 2 assignments and write papers for each of them (6 pages for the first one and 10 pages for the second one) (in groups of 3)
  • Serve as an opponent for one student talk (pairs of opponents for each student presentations)
  • Read papers before the talks and participate in discussion

Class Presentation

  • Present the chosen paper in an accessible way
  • Present sufficient background, do not expect the audience to know much about Machine Learning or Natural Language Processing, except for the material already covered in the class, NLP1 and ML1
  • Have a critical view on the paper: discuss shortcomings, possible future work, etc
  • To give a good presentation in many cases you will need to read one or two additional papers (e.g., those referenced in the paper)
  • The language for talks and discussions will be English
  • You are welcome to receive feedback to your slides, but in this case send Ivan your slides (preferably in PDF) 7 days before the talk by 6 pm. You will get feedback from us 3 days before the seminar. The first 2 presenters can send him slides 2 days before the talk and will receive feedback within a day. If needed we can meet to discuss the comments.

Opponent role

  • There will be 2 opponents for each student presentation
  • Read the paper carefully
  • Be ready to ask at least 3 questions
  • Preferably aim for critical comments which provide insights to other attenders and stimulate discussion
  • If there are many questions you can limit yourself to 2 questions

Reviews

  • A very short critical essay reviewing one of the papers recommended for each class
    • One (or at most two) paragraphs presenting the essence of the paper.
    • Another paragraph discussion both positive sides of the paper (what you like) and shortcomings (what you do not like).
  • You need to submit one review for each class
  • The review should be submitted via blackboard, the deadline will be before the class.
  • You can skip once (no need to tell us)
  • No copy-paste from the paper. It should be all your words.
  • Submit via Blackboard

Project reports

Project reports for each assignment should

  • describe and motivate the method,
  • describe the experimental set-up and the dataset(s) used,
  • present and analyze results
  • if possible, connect to other methods discussed in class (or familiar to you in other contexts)
  • compare with results reported in the original papers
  • discuss shortcomings and potential extensions

Example structure:

  • Introduction (incl. motivation)
  • Method
  • Empirical evaluation
    • Task and data
    • Models selection and parameters
    • Results and discussion
    • Qualitative evaluation (e.g., some visualization of the model)
  • Related work (when possible)
  • Conclusions

The report for the first assignment on Bayesian methods should be short (7-8 pages). The report for larger project on representation learning / deep learning should be 11-14 pages.

Grading criteria for the project:

  • Quality of the implementation and technical correctness
  • Clarity of the paper
  • Paper organization.
  • Technical correctness.
  • Style (written in research style without inappropriate speculations, correct citations, etc).
  • Quality of analysis and critical discussion

Format: Submitted in PDF via Blackboard

Grading Policy

Marks will be assigned as follows:

  • Assignment 1: 30%
  • Assignment 2: 40%
  • Paper presentation: 20%
  • Opponent role and other discussion in class: 10%

Reviews are a requirement but graded as PASS / FAIL: you should submit 10 reviews. If a review is not accepted we will notify you. In this situation, we may ask you to prepare an extra review instead.

Deadlines

  • Reviews - every lecture, before the lecture
  • Presentation / opponent registration - first come - first served
  • Assignment 1: April 17th (23:59)
  • Assignment 2: May 17th (23:59)

The deadlines are strict and no extensions will be granted