DA6xx Information Retrieval

• Course Objectives

  • Learn the theories and techniques behind Web search engines.

  • Get hands on project experience by developing real-world applications, such as intelligent tools for improving search accuracy from user feedback, email spam detection, or scientific literature organization and mining.

  • Learn tools and techniques to do cutting-edge research in the area of information retrieval or text mining.

  • Open the door to the amazing job opportunities in Search Technology and E-commerce companies such as Google, Microsoft, Yahoo! and Amazon.

• Learning Outcomes

  • LO1: Learn to write code for text indexing and retrieval

  • LO2: Learn to evaluate information retrieval systems

  • LO3: Learn to analyze textual and semi-structured data sets

  • LO4: Learn about text similarity measure

  • LO5: Understanding about search engine

• Module 1: Overview of text retrieval systems

  • The nature of unstructured and semi-structured text. Inverted index and Boolean queries.

  • Text encoding: tokenization, stemming, stop words, phrases, index optimization, Index compression: lexicon compression and postings lists compression, Gap encoding, gamma codes, Zipf’s Law, Index construction, Postings size estimation, dynamic indexing, positional indexes, n-gram indexes, real-world issues.

• Module 2: Retrieval models, implementation, and evaluation

  • Boolean, vector space, TF-IDF, Okapi, probabilistic, language modeling, latent semantic indexing, Vector space scoring, The cosine measure, Efficiency considerations, Document length normalization, Relevance feedback and query expansion, Rocchio.

  • Evaluating search engines, User happiness, precision, recall, F-measure. Creating test collections: kappa measure, interjudge agreement.

• Module 3: Text classification and clustering

  • Introduction to text classification, Naive Bayes models, Spam filtering, Vector space classification using hyperplanes, centroids, k Nearest Neighbors, Support vector machine classifiers, Kernel functions, Boosting.

  • Clustering versus classification, Partitioning methods, k-means clustering, Mixture of gaussian model, Hierarchical agglomerative clustering, Clustering terms using documents.

• Module 4: Web Information Retrieval

  • Hypertext, web crawling, search engines, ranking, link analysis, PageRank, HITS.

  • XML retrieval, semantic web.

• Module 5: IR applications and Advanced Topics

  • Summarization, Topic detection and tracking, Personalization, Question answering, Cross language informtion retrieval.

  • Information extraction, Question answering, Opinion summarization, Social Network. Recommender Systems.

• Lecture 0: Course Overview;

• Lecture 1: Overview; Foundations of Information Retrieval; Source: CS54701 Information Retrieval by Clifton

• Lecture 2: Indexing and Querying; Boolean Retrieval; Source: Chapter 1 & 2, Manning, Raghavan and Schütze

• Lecture 3: Text Encoding; Source: Chapter 2, Manning, Raghavan and Schütze

• Lecture 4: Dictionaries and Tolerant Retrieval; Source: Chapter 3, Manning, Raghavan and Schütze

• Lecture 5: Index Construction; Source: Chapter 4, Manning, Raghavan and Schütze

• Lecture 6: Index Compression; Source: Chapter 5, Manning, Raghavan and Schütze

• Lecture 7: Scoring, Term weighting; Vector Space Model; Source: Chapter 6, Manning, Raghavan and Schütze

• Lecture 8: Computing Scores; Source: Chapter 7, Manning, Raghavan and Schütze

• Lecture 9: Evaluation in Information Retrieval; Source: Chapter 8, Manning, Raghavan and Schütze

• Lecture 10: Relevance Feedback and Query Expansion; Source: Chapter 9, Manning, Raghavan and Schütze

• Lecture 11: XML Retrieval; Source: Chapter 10, Manning, Raghavan and Schütze

• Lecture 12: Probabilistic Information Retrieval; Source: Chapter 11, Manning, Raghavan and Schütze

• Lecture 13: Language Models; Source: Chapter 12, Manning, Raghavan and Schütze

• Lecture 14: Link Analysis; Source: Chapter ?, Manning, Raghavan and Schütze

Introduction to Information Retrieval, by C. Manning, P. Raghavan, and H. Schütze (Cambridge University Press, 2008).

Search Engines: Information Retrieval in Practice. Croft, W. Bruce; Metzler, Donald; Strohman, Trevor. Addison Wesley (2008)

Information Retrieval: Implementing and Evaluating Search Engines, Stefan Buettcher, Charles L. A. Clarke, Gordon V. Cormack. MIT Press. (2010)

Modern Information Retrieval, Ricardo Baeza-Yates and Berthier Ribeiro-Neto, Addison-Wesley, (1999)

• 12 Quizzes 36%

• Midsem 30%

• Project 20%

• Viva 14%