Instructor: Manoj Gupta, Mayank Singh (office: AB-4/304, email: singh.mayank@iitgn.ac.in)
Location: Jasubhai Auditorium
Class Timings: Monday 10:00-11:30
Lecture Slides: The link to lecture slides will be updated before each class
QA sessions: Email me to book an appointment to discuss any doubts, clarifications, and concepts.
Algorithms for processing data: Understanding time complexity Asymptotics (big-O notation and examples). Storing sequential data: arrays and linked lists, data structures for storing tabular data. Organizing data and finding items : Searching (linear/binary search), Sorting: sorting almost sorted data (bubble sort, insertion sort), fast sorting in the worst-case (heapsort), using randomization to sort faster (quicksort), sorting without random-access ex. linked-lists (mergesort)
Data Analysis: Use of spreadsheets for data analysis; data manipulation specific Python libraries e.g. Pandas; introduction to SQL. Answering queries over tabular data: map-filter-reduce, joins and unions. Using vectorized operations for efficiency. Techniques: Data structuring (tidy data) and cleaning (dealing with multiple labels for the same data, missing data). Summarizing data (averages, variance, moments), functions of tables (loading, cleaning, normalizing). Introduction to data visualization using matplotlib; plotting various statistics; categorical distributions, numerical distributions, overlaid graphs.
Deriving Conclusions: Anomaly detection techniques in data: tests e.g. GRIM test, Benford’s law; Statistical traps: Correlation vs causation, Base-rate fallacy / prosecutor's fallacy. Simpson's paradox. Data censoring. Will Rogers effect, lead-time bias, and length time bias. Means versus medians. Importance of higher moments. Misleading visualizations.
Making Predictions: Linear regression; basics of classification (train, test, validation); creating features; naive Bayes classification, logistic regression and its interpretation; introduction to clustering (k-means). Confidence in predictions: prediction intervals, confidence intervals.
Introduction to Pandas [Colab Notebook]
Introduction to SQL [Slides]
Introduction to Matplotlib [Slides]
Introduction to Machine Learning [Colab Notebook]