CS 613: Natural Language Processing

IIT Gandhinagar

Autumn 2024

Instructor: Mayank Singh (email: singh.mayank@iitgn.ac.in)

Office Hours: Friday 10 AM-11 AM (For any other day, email me for an appointment)

Class Schedule: Wednesday and Friday, 3:30-4:50 PM.

Location: AB 7/208

Communication Google group: cs613-2024.pvtgroup@iitgn.ac.in

Teaching Assistants

1. Himanshu Beniwal (himanshubeniwal@iitgn.ac.in)

2. Indrayudh Mandal (24210041@iitgn.ac.in)

3. Mithlesh Singla (24210063@iitgn.ac.in)

4. Alay Patel (alay.patel@iitgn.ac.in)

Prerequisite (Optional)

• Basic Probability & Statistics (ES 331/ MA 202) or equivalent

• Basic understanding of Python programming (ES 102/ ES 112) or equivalent

Course Contents

1. Text processing: Tokenization, Stemming, Spell Correction, etc.

2. Language Modelling: N-grams, smoothing

3. Morphology, Parts of Speech Tagging

4. Syntax: PCFGs, Dependency Parsing

5. Distributional Semantics, Topic Models

6. Lexical Semantics, Word Sense Disambiguation

7. Information Extraction: Relation Extraction, Event Extraction

8. Applications: Text Classification, Sentiment Analysis, Opinion Mining, Summarization

9. Deep Learning for NLP, Representation Learning

Lecture Slides and Additional Materials

• Intro to NLP [Slides]

• Basic Data Preprocessing [slides], [script]

• Distributional Semantics [slides] [scripts: word-to-doc-binary, word-to-word_binary, word-to-word-non-binary-count, word-to-word-non-binary-tf-idf]

• Continuous representations [Slides, Word2Vec paper, Parameter Learning, Wevi, Glove Paper]

• Language Modelling [Basics of LM Slides, Smoothing Slides, LM using NLTK, Section 3.8 for relation between entropy, cross-entropy, and perplexity, Good Turing Estimate]

• Neural Language Model [Slides]

• Seq2Seq approach, Attention Mechanism and Transformers [Slides]

• Contextual Word Embeddings, BERT [Slides]

• Calculating Parameters [Doc] [GPT-2 parameter calculation], Let's build GPT: from scratch [Video Link]

• Tokenization [Slides] [Colab: BPE, Wordpiece]

• Computational Morphology [Slides]

• Sequence Labelling (POS Tagging, limitations, HMM) [Slides]

• Lexical Semantics (Types, wordnet, and similarity metrics)  [Slides, NLTK's Wordnet Tutorial]

• Word Sense Disambiguation [Slides]

• Text Classification [Slides]

• Text Summarization [Slides]

• Metrics and benchmarks [Slides]

Assignments (All deadlines are 11:59 PM IST)

• Assignment 1

• Assignment 2

Paper Presentations I

• Team 1 - SQuAD: 100,000+ Questions for Machine Comprehension of Text - Video Link 

• Team 3 - Language Models are Unsupervised Multitask Learners  - Video Link 

• Team 4 - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - Video Link 

• Team 5 - GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding  - Video Link 

• Team 6 - GLUECoS: An Evaluation Benchmark for Code-Switched NLP  - Video Link 

• Team 8 -The Stack: 3 TB of permissively licensed source code  - Video Link 

• Team 9 - TruthfulQA: Measuring How Models Mimic Human Falsehoods  - Video Link 

• Team 10 - Toxicity in CHATGPT: Analyzing Persona-assigned Language Models  - Video Link 

• Team 11 - RoBERTa: A Robustly Optimized BERT Pretraining Approach  - Video Link

• Team 12 - T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer  - Video Link 

• Team 13 - ALBERT: A Lite BERT for Self-supervised Learning of Language Representations  - Video Link 

• Team 14 - XLNet: Generalized Autoregressive Pretraining for Language Understanding  - Video Link 

• Team 15 - GPT-3: Language Models are Few-Shot Learners  - Video Link 

• Team 16 - XLM-R: Robust Cross-lingual Representation Learning at Scale  - Video Link 

• Team 17 - mBART: Multilingual Denoising Pre-training for Neural Machine Translation  - Video Link 

Grading Policy & Schedule

• Three Assignments (30%)
  Three assignments (each carrying 10 marks). 

• Three Surprise quizzes (30%)
  Three surprise quizzes of 10% marks each. These quizzes will assess your grasp of the content covered in the class. 

• Exam (20%) [A Sample Paper]
  One exam during the examination I. 

• Paper Presentations (20%)

     Two paper presentations (each carrying 10 marks). 

Books

• [DJ] Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition (1st ed.). Prentice Hall PTR, Upper Saddle River, NJ, USA. (Main Textbook)

• [CH] Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA, USA.

• [SEE] Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python (1st ed.). O'Reilly Media, Inc.

• [IYA] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. The MIT Press.