Course Objectives
Learn the theories and techniques behind Web search engines.
Get hands on project experience by developing real-world applications, such as intelligent tools for improving search accuracy from user feedback, email spam detection, or scientific literature organization and mining.
Learn tools and techniques to do cutting-edge research in the area of information retrieval or text mining.
Open the door to the amazing job opportunities in Search Technology and E-commerce companies such as Google, Microsoft, Yahoo! and Amazon.
Learning Outcomes
LO1: Learn to write code for text indexing and retrieval
LO2: Learn to evaluate information retrieval systems
LO3: Learn to analyze textual and semi-structured data sets
LO4: Learn about text similarity measure
LO5: Understanding about search engine
Module 1: Overview of text retrieval systems
The nature of unstructured and semi-structured text. Inverted index and Boolean queries.
Text encoding: tokenization, stemming, stop words, phrases, index optimization, Index compression: lexicon compression and postings lists compression, Gap encoding, gamma codes, Zipf’s Law, Index construction, Postings size estimation, dynamic indexing, positional indexes, n-gram indexes, real-world issues.
Module 2: Retrieval models, implementation, and evaluation
Boolean, vector space, TF-IDF, Okapi, probabilistic, language modeling, latent semantic indexing, Vector space scoring, The cosine measure, Efficiency considerations, Document length normalization, Relevance feedback and query expansion, Rocchio.
Evaluating search engines, User happiness, precision, recall, F-measure. Creating test collections: kappa measure, interjudge agreement.
Module 3: Text classification and clustering
Introduction to text classification, Naive Bayes models, Spam filtering, Vector space classification using hyperplanes, centroids, k Nearest Neighbors, Support vector machine classifiers, Kernel functions, Boosting.
Clustering versus classification, Partitioning methods, k-means clustering, Mixture of gaussian model, Hierarchical agglomerative clustering, Clustering terms using documents.
Module 4: Web Information Retrieval
Hypertext, web crawling, search engines, ranking, link analysis, PageRank, HITS.
XML retrieval, semantic web.
Module 5: IR applications and Advanced Topics
Summarization, Topic detection and tracking, Personalization, Question answering, Cross language informtion retrieval.
Information extraction, Question answering, Opinion summarization, Social Network. Recommender Systems.
Lecture 1: Overview; Foundations of Information Retrieval; Source: CS54701 Information Retrieval by Clifton
Lecture 2: Indexing and Querying; Boolean Retrieval; Source: Chapter 1 & 2, Manning, Raghavan and Schütze
Lecture 3: Text Encoding; Source: Chapter 2, Manning, Raghavan and Schütze
Lecture 4: Dictionaries and Tolerant Retrieval; Source: Chapter 3, Manning, Raghavan and Schütze
Lecture 5: Index Construction; Source: Chapter 4, Manning, Raghavan and Schütze
Lecture 6: Index Compression; Source: Chapter 5, Manning, Raghavan and Schütze
Lecture 7: Scoring, Term weighting; Vector Space Model; Source: Chapter 6, Manning, Raghavan and Schütze
Lecture 8: Computing Scores; Source: Chapter 7, Manning, Raghavan and Schütze
Lecture 9: Evaluation in Information Retrieval; Source: Chapter 8, Manning, Raghavan and Schütze
Lecture 10: Relevance Feedback and Query Expansion; Source: Chapter 9, Manning, Raghavan and Schütze
Lecture 11: XML Retrieval; Source: Chapter 10, Manning, Raghavan and Schütze
Lecture 12: Probabilistic Information Retrieval; Source: Chapter 11, Manning, Raghavan and Schütze
Lecture 13: Language Models; Source: Chapter 12, Manning, Raghavan and Schütze
Lecture 14: Text Classification; Source: Chapter 13, Manning, Raghavan and Schütze
Lecture 15: Distributed Representations; Source: Chapter 14, Manning, Raghavan and Schütze
Lecture 16: Learning Ranking; Source: Chapter 15, Manning, Raghavan and Schütze
Lecture 17: Link Analysis; Source: Chapter ?, Manning, Raghavan and Schütze
Lecture 18: Crawling; Source: Chapter ?, Manning, Raghavan and Schütze
Lecture 19: Web QA; Source: Chapter ?, Manning, Raghavan and Schütze
Lecture 20: Personalization; Source: Chapter ?, Manning, Raghavan and Schütze
Self-Study 1: Web Crawling
Self-Study 2: Indexing and Searching using pySolr
Self-Study 3: Boolean Retrieval using Python
Self-Study 4: Vector Space Retrieval using Python
Self-Study 5: Probabilistic Retrieval using Python
Self-Study 6: Classification in Retrieval using Python
Unit 1: Foundations of Information Retrieval
Introduction to Information Retrieval, Information Retrieval vs Database Systems, Applications of IR, Components of an IR System, Document Collections and Corpora, Text Processing Pipeline, Tokenization, Stopword Removal, Stemming, Lemmatization, Inverted Index, Query Processing, Web Search Basics.
Unit 2: Data Compression and Indexing
Need for Compression in IR, Dictionary Compression, Blocked Storage, Front Coding, Posting List Compression, Variable Byte Encoding, Gamma Coding, Delta Coding, Skip Pointers, Efficient Index Construction, Dynamic Indexing, Distributed Indexing, Index Maintenance.
Unit 3: Implementation and Evaluation
IR System Architecture, Web Crawling, Document Acquisition, Building an Inverted Index, Query Execution, Ranked Retrieval, Relevance Feedback, Evaluation Methodology, Precision, Recall, F1-Score, Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), NDCG, TREC Collections and Benchmarks, Lucene, Elasticsearch, Whoosh, PyTerrier.
Unit 4: Retrieval Models
Boolean Retrieval Model, Boolean Queries, Vector Space Model (VSM), Term Frequency (TF), Inverse Document Frequency (IDF), TF-IDF Weighting, Cosine Similarity, Probabilistic Retrieval Model, Binary Independence Model, BM25, Language Models for IR, Query Likelihood Model, Jelinek-Mercer Smoothing, Dirichlet Smoothing, Latent Semantic Analysis (LSA), Topic Models.
Lecture 1: Overview; Foundations of Information Retrieval; Source: Chapter 1 & 2, Manning, Raghavan and Schütze
Lecture 2: Indexing and Querying;; Dictionaries and Tolerant Retrieval; Source: Chapter 3, Manning, Raghavan and Schütze
Lecture 3: Text Encoding; Index Construction Source: Chapter 4 & 5, Manning, Raghavan and Schütze
Lecture 4: Dictionaries and Tolerant Retrieval; Scoring, Term weighting; Source: Chapter 3, Manning, Raghavan and Schütze
Lecture 5: Boolean Retrieval; Vector Space Model; Source: Chapter 4, Manning, Raghavan and Schütze
Lecture 6: Source: Chapter 5, Manning, Raghavan and Schütze
Lecture 7: Source: Chapter 6, Manning, Raghavan and Schütze
Self-Study 1: Web Crawling
Self-Study 2: Indexing and Searching using pySolr
Self-Study 3: Boolean Retrieval using Python
Self-Study 4: Vector Space Retrieval using Python
Self-Study 5: Probabilistic Retrieval using Python
Self-Study 6: Classification in Retrieval using Python
Introduction to Information Retrieval, by C. Manning, P. Raghavan, and H. Schütze (Cambridge University Press, 2008).
Search Engines: Information Retrieval in Practice. Croft, W. Bruce; Metzler, Donald; Strohman, Trevor. Addison Wesley (2008)
Information Retrieval: Implementing and Evaluating Search Engines, Stefan Buettcher, Charles L. A. Clarke, Gordon V. Cormack. MIT Press. (2010)
Modern Information Retrieval, Ricardo Baeza-Yates and Berthier Ribeiro-Neto, Addison-Wesley, (1999)
Quiz 24%
Assignments 20%
Project 40% (Implementation 10%, Knowledge 10%, Analysis 20%)
Attendance 16%