Indian Institute of Technology Gandhinagar Lecture Hall - Shed 5 (202) Lecture Hours - H Slot 1205-1300 Hours on Monday, Tuesday and Friday
Office - Shed 5 (216) Teaching Assistant - Rajendra Nagar Office Hours - Any time, Any where you spot me
shanmuga@iitgn.ac.in DescriptionThe world we live has three dimensions (3D). Human visual system has evolved to perceive all these dimensions. However, the images we capture using conventional cameras are just the 2D projections of the 3D world. In 3D Computer Vision course, we shall explore various techniques for recovering the missing third dimension (depth information) from 2D images using primarily variational methods and projective geometry concepts. The course contents would enable the student to reconstruct the 3D real world scene from 2D images by various methods. The applications of this course range from cultural heritage to medical imaging, from robot navigation to 3D modeling. The assignments and projects associated with the course to be completed using OpenCV, Meshlab, Frankencamera (nVIDIA Tegra 3 and Nokia N900) kits would enable students to develop state-of-the-art 3D computer vision applications. This course is offered as an elective for BTech, MTech, and PhD students of IIT Gandhinagar. This course is also prescribed for minor degree in Computer Science.
Course Contents
Review of linear algebra, calculus of variations, signals and systems; Camera and image formation – optics; Feature detectors – edge and corner detection; Feature descriptors – SIFT, SURF, feature matching; Shape from X – Reflectance map, BRDF, shape from shading, photometric stereo, depth from defocus, depth from focus, RGB-D images; Single view geometry – finite projective cameras, camera parameters, point correspondences, estimation of camera matrix, direct linear transformation (DLT); Two view geometry – homography, epipolar geometry, estimation of fundamental matrix, image rectification, stereo correspondence, shape from stereo; Three view geometry – trifocal tensors; Motion – optical flow field, Estimation of dense and accurate optical flow field; Multi view geometry – structure from motion, triangulation, factorization, bundle adjustment; Internet vision – mining community photo collections (Flickr, Facebook, etc.). Textbooks - Horn, B. K. P. (1986). Robot Vision. The MIT Press.
- Hartley R. and Zisserman A. (2004). Multiple View Geometry in Computer Vision, 2nd Edition, Cambridge University Press.
- Szeliski, R. (2010). Computer Vision: Algorithms and Applications. Springer-Verlag New York Inc. Available Online.
- Nixon, M. S., & Aguado, A. S. (2012). Feature Extraction & Image Processing for Computer Vision. Third Edition. Academic Press.
Davies, E. R. (2012). Computer and Machine Vision: Theory, Algorithms, Practicalities. 4th Edition. Academic Press. Forsyth, D. A., & Ponce, J. (2015). Computer Vision: A Modern Approach. Second Edition. Prentice Hall of India. - Klette, R. (2014). Concise Computer Vision: An Introduction Into Theory and Algorithms. Springer Publishing Company, Incorporated.
- Marr, D. (2010). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. The MIT Press.
- Sonka, M., Hlavac, V., & Boyle, R. (2014). Image Processing, Analysis, and Machine Vision. 4th Edition. Cengage Learning.
- Trucco, E. and Verri, A. (1998). Introductory Techniques for 3D Computer Vision, Prentice- Hall.
- Prince, S. J. (2012). Computer Vision: Models, Learning, and Inference. Cambridge University Press. Available Online
- Ikeuchi, K. (2014). Computer Vision: A Reference Guide. Springer Publishing Company, Incorporated.
- Fisher, R. B., Breckon, T. P., Dawson-Howe, K., Fitzgibbon, A., Robertson, C., Trucco, E., & Williams, C. K. (2013). Dictionary of computer vision and image processing. John Wiley & Sons.
The book by Marr provides a viewpoint based on visual neuroscience concepts. The last 5 books can be used as reference for certain topics. Apart from these books, some topics would be taught from selected research papers.
Lecture notes you make in the classroom will provide pointers to look into topics in different books listed above. The topics taught in a lecture may have evolved from multiple books and research papers. Reading books would certainly aid lectures but can never replace the lectures.
Suggested Readings- Trefethen, L. N., and Bau III, D., Numerical Linear Algebra, SIAM, 1997.
- Watkins, D. S., Fundamentals of Matrix Computations, 3rd Edition, John Wiley & Sons, 2010.
- Courant, R., Robbins, H., and Stewart, I., What is Mathematics?: An Elementary Approach to Ideas and Methods, 2nd Edition, Oxford University Press, 1996.
- Gelfand, I. M., and Fomin, S. V., Calculus of Variations, Dover Publications, 2000.
- Lathi, B. P., Signal Processing and Linear Systems, Oxford University Press, 2000.
These suggested readings supplement the textbooks and reference books to understand various mathematical concepts in depth.
Grading
- Mid-semester exam - 30%
- End-semester exam - 40%
- Projects and programming assignments - 30%
Pre-requisitesExposure to Signals and Systems course at the UG level is required. Programming experience in C/C++/Python is desired for successful completion of the assignments. Lecture Schedule and Reading Material
Date | Topics Covered
| Reading Material
| 30 December 2014 | Introduction, Course Policies, Grading Scheme, Computer Vision and its Applications | Szeliski Chapter 1 | 2 January 2015 | Computer Vision - Low Level, Mid Level and High Level, Scope of the Course, Mathematical Preliminaries Required, Digital Camera and its Modules | Szeliski Chapter 2 | 5 January 2015 | Point Operators, Examples, Histogram Equalization | Szeliski Chapter 3 | 6 January 2015 | Thresholding, Otsu's Threshold Method | Szeliski Chapter 3, Otsu's Threshold | 9 January 2015 | Linear Filtering, 2D FIR Filters, Group Operators, Separable Filters, Box, Tent | Szeliski Chapter 3 | 12 January 2015 | Central Limit Theorem, Gaussian Filter, Averaging as a LPF, HPF - Sobel and Prewitt, Laplacian Operator | Szeliski Chapter 3 | 13 January 2015 | SVD and Matrix Properties | Trefethen and Bau | 16 January 2015 | Solving Linear System of Equations using SVD, Laplacian of Gaussian (LoG), Difference of Gaussian (DoG) Filters, Relation | Trefethen and Bau, Szeliski Chapter 3, Appendix, Marr-Hildreth Paper
| 20 January 2015 | LoG and DoG Filters as Edge Detectors and Blob Detectors, Canny Edge Detector | Szeliski Chapter 4, Canny's Paper, OpenCV | 23 January 2015 | Interest Points, Correspondence - SSD, NCC, Corner Detection, Harris Corner Detector | Szeliski Chapter 4, Harris-Stephens Paper, OpenCV | 27 January 2015 | Integral Image and Application, Image Pyramids, Gaussian Scale Space | Szeliski Chapter 3, Paper by Witkin, Gaussian Scale Space Paper | 30 January 2015
| Scale Invariant Feature Transform (SIFT)
| Szeliski Chapter 4, Lowe's Paper
| 02 February 2015
| Feature Detection and Description using SIFT
| SIFT Official Website
| 03 February 2015
| Feature Detection and Description using Speeded-Up Robust Features (SURF)
| SURF Official Website
| 06 February 2015
| Surface Normal, BRDF, Solid Angle, Radiance and Irradiance, Image Formation
| Horn Chapter 9
| 09 Februrary 2015
| Thin Lens Equation, Depth of Field, Angle of View, 2D Projective Plane, Lines and Points
| Hartley and Zisserman Chapter 2
| 10 February 2015
| Points and Lines at Infinity, Conics, Projective Transformation
| Hartley and Zisserman Chapter 2 | 13 February 2015 | 2D Transformations - Isometries, Similarities, Affinities, Projectivities, Degrees of Freedom, Points at Infinity over Transformations | Hartley and Zisserman Chapter 2 | 16 February 2015 | Solution to Over-determined Homogeneous and Non-homogeneous Linear Systems, Constrained Minimal Norm Solution using SVD, Estimation of 2D Projective Transformation using Direct Linear Transformation (DLT) | Hartley and Zisserman Chapter 4 | 20 February 2015 | DLT and Normalization, Robust Estimation, Outlier Rejection, RANSAC | Hartley and Zisserman Chapter 4 | 9 March 2015 | RANSAC for Robust Homography Estimation, Perspective Projection, Image Formation for PinHole Camera, Finite and Infinite Cameras, Intrinsic Camera Parameters | Hartley and Zisserman Chapters 4, 6 | 10 March 2015 | Global and Camera Coordinates, Extrinsic Parameters, General and Finite Projective Cameras, Camera Matrix Anatomy | Hartley and Zisserman Chapter 6 | 13 March 2015
| Estimation of Camera Centre, Principal Plane, Principal Point, Principal Axis from Camera Matrix, Significance of Rows and Columns of Camera Matrix
| Hartley and Zisserman Chapters 6,7 | 16 March 2015
| Normalized Estimation of Camera Matrix, Projections due to Cameras at Infinity, Affine Camera Matrix
| Hartley and Zisserman Chapters 6,7 | 17 March 2015
| Differences between Affine and Non-Affine Cameras, Normalized Estimation of Affine Camera Matrix, Homography Arising from Finite Projective Cameras, Projection of a Plane
| Hartley and Zisserman Chapters 7,8
| 20 March 2015 | Homography due to Rotation and Focal Length Change, Application to PTZ Cameras, Two Views - Baseline, Epipolar Geometry, Epipoles | Hartley and Zisserman Chapter 8, 9 | 23 March 2015 | Computer Vision using OpenCV Python | | 24 March 2015 | Computer Vision using OpenCV Python | | 27 March 2015 | Derivation of Fundamental Matrix, Properties Perceptual Computing - Guest Lecture by Dr. Achintya Bhowmik, Intel | Hartley and Zisserman Chapter 9 | 30 March 2015 | Estimation of Fundamental Matrix, 7-point and Normalized 8-point Algorithms, Essential Matrix | Hartley and Zisserman Chapter 11 | 31 March 2015 | Robust Estimation of Fundamental Matrix, Mapping of Points on Epipolar Line, Disparity and Depth | Hartley and Zisserman Chapter 9, 11 Horn Chapter 12 | 3 April 2015
| Perceptual Computing - Guest Lecture by Dr. Achintya Bhowmik, Intel | | 6 April 2015
| Image Rectification, Homography Estimation, Depth from Stereo
| Hartley and Zisserman Chapter 11 | 7 April 2015
| Projective Ambiguity, Linear Triangulation, Structure Computation from Stereo Pair
| Hartley and Zisserman Chapter 12
| 10 April 2015 | Structure from Motion (SfM), Affine Factorization, Projective Factorization | Hartley and Zisserman Chapter 18, Tomasi-Kanade Factorization
| 13 April 2015
| Factorization, Shape from X, Optical Flow, Calculus of Variations
| Hartley and Zisserman Chapter 18, Horn Appendix
| 17 April 2015 | Variational Problems, Euler's Equation, Boundary Conditions, Brightness Constancy Constraint for Optical Flow | Horn Appendix, Horn-Schunck Optical Flow | 20 April 2015 | Optical Flow Estimation - Horn-Schunck, Lucas-Kanade | Horn-Schunck Optical Flow, Lucas-Kanade Paper | 21 April 2015 | Project Presentations | | 24 April 2015 | Project Presentations | |
Discussions/Queries/Doubts
Use Piazza
|
|