EE645 3D Computer Vision - 2014-15

Instructor: Shanmuganathan Raman

Indian Institute of Technology Gandhinagar

Lecture Hall - Shed 5 (202)

Lecture Hours - H Slot 1205-1300 Hours on Monday, Tuesday and Friday

Office - Shed 5 (216)

Teaching Assistant - Rajendra Nagar

Office Hours - Any time, Any where you spot me

shanmuga@iitgn.ac.in

Description

The world we live has three dimensions (3D). Human visual system has evolved to perceive all these dimensions. However, the images we capture using conventional cameras are just the 2D projections of the 3D world. In 3D Computer Vision course, we shall explore various techniques for recovering the missing third dimension (depth information) from 2D images using primarily variational methods and projective geometry concepts.

The course contents would enable the student to reconstruct the 3D real world scene from 2D images by various methods. The applications of this course range from cultural heritage to medical imaging, from robot navigation to 3D modeling. The assignments and projects associated with the course to be completed using OpenCV, Meshlab, Frankencamera (nVIDIA Tegra 3 and Nokia N900) kits would enable students to develop state-of-the-art 3D computer vision applications.

This course is offered as an elective for BTech, MTech, and PhD students of IIT Gandhinagar. This course is also prescribed for minor degree in Computer Science.

Course Contents

Review of linear algebra, calculus of variations, signals and systems; Camera and image formation – optics; Feature detectors – edge and corner detection; Feature descriptors – SIFT, SURF, feature matching; Shape from X – Reflectance map,  BRDF, shape from shading, photometric stereo, depth from defocus, depth from focus, RGB-D images; Single view geometry – finite projective cameras, camera parameters, point correspondences, estimation of camera matrix, direct linear transformation (DLT); Two view geometry – homography, epipolar geometry, estimation of fundamental matrix, image rectification, stereo correspondence, shape from stereo; Three view geometry – trifocal tensors; Motion – optical flow field, Estimation of dense and accurate optical flow field; Multi view geometry – structure from motion, triangulation, factorization, bundle adjustment; Internet vision – mining community photo collections (Flickr, Facebook, etc.).

Textbooks 

  1. Horn, B. K. P. (1986). Robot Vision. The MIT Press.
  2. Hartley R. and Zisserman A. (2004). Multiple View Geometry in Computer Vision, 2nd Edition, Cambridge University Press.
  3. Szeliski, R. (2010). Computer Vision: Algorithms and Applications. Springer-Verlag New York Inc. Available Online.
  4. Nixon, M. S., & Aguado, A. S. (2012). Feature Extraction & Image Processing for Computer Vision. Third Edition. Academic Press.
  5. Davies, E. R. (2012). Computer and Machine Vision: Theory, Algorithms, Practicalities. 4th Edition. Academic Press.
  6. Forsyth, D. A., & Ponce, J. (2015). Computer Vision: A Modern Approach. Second Edition. Prentice Hall of India.
  7. Klette, R. (2014). Concise Computer Vision: An Introduction Into Theory and Algorithms. Springer Publishing Company, Incorporated.

References

  1. Marr, D. (2010). Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. The MIT Press.
  2. Sonka, M., Hlavac, V., & Boyle, R. (2014). Image Processing, Analysis, and Machine Vision. 4th Edition. Cengage Learning.
  3. Trucco, E. and Verri, A. (1998). Introductory Techniques for 3D Computer Vision, Prentice- Hall.
  4. Prince, S. J. (2012). Computer Vision: Models, Learning, and Inference. Cambridge University Press. Available Online
  5. Ikeuchi, K. (2014). Computer Vision: A Reference Guide. Springer Publishing Company, Incorporated.
  6. Fisher, R. B., Breckon, T. P., Dawson-Howe, K., Fitzgibbon, A., Robertson, C., Trucco, E., & Williams, C. K. (2013). Dictionary of computer vision and image processing. John Wiley & Sons.

The book by Marr provides a viewpoint based on visual neuroscience concepts. The last 5 books can be used as reference for certain topics. Apart from these books, some topics would be taught from selected research papers.

Lecture notes you make in the classroom will provide pointers to look into topics in different books listed above. The topics taught in a lecture may have evolved from multiple books and research papers. Reading books would certainly aid lectures but can never replace the lectures.

Suggested Readings

  1. Trefethen, L. N., and Bau III, D., Numerical Linear Algebra, SIAM, 1997.
  2. Watkins, D. S., Fundamentals of Matrix Computations, 3rd Edition, John Wiley & Sons, 2010.
  3. Courant, R., Robbins, H., and Stewart, I., What is Mathematics?: An Elementary Approach to Ideas and Methods, 2nd Edition, Oxford University Press, 1996.
  4. Gelfand, I. M., and Fomin, S. V., Calculus of Variations, Dover Publications, 2000.
  5. Lathi, B. P., Signal Processing and Linear Systems, Oxford University Press, 2000.

These suggested readings supplement the textbooks and reference books to understand various mathematical concepts in depth.

Grading

  • Mid-semester exam - 30%
  • End-semester exam - 40%
  • Projects and programming assignments - 30%

Pre-requisites

Exposure to Signals and Systems course at the UG level is required. Programming experience in C/C++/Python is desired for successful completion of the assignments.

Lecture Schedule and Reading Material

 Date    Topics Covered
 Reading Material
30 December 2014Introduction, Course Policies, Grading Scheme, Computer Vision and its ApplicationsSzeliski Chapter 1
2 January 2015Computer Vision - Low Level, Mid Level and High Level, Scope of the Course, Mathematical Preliminaries Required, Digital Camera and its Modules Szeliski Chapter 2
 5 January 2015     Point Operators, Examples, Histogram Equalization Szeliski Chapter 3
 6 January 2015 Thresholding, Otsu's Threshold Method     Szeliski Chapter 3, Otsu's Threshold    
 9 January 2015 Linear Filtering, 2D FIR Filters, Group Operators, Separable Filters, Box, Tent Szeliski Chapter 3
12 January 2015 Central Limit Theorem, Gaussian Filter, Averaging as a LPF,  HPF - Sobel and Prewitt, Laplacian OperatorSzeliski Chapter 3 
 13 January 2015 SVD and Matrix Properties Trefethen and Bau
16 January 2015  Solving Linear System of Equations using SVD, Laplacian of Gaussian (LoG), Difference of Gaussian (DoG) Filters, Relation Trefethen and Bau, Szeliski Chapter 3, Appendix, Marr-Hildreth Paper
20 January 2015 LoG and DoG Filters as Edge Detectors and Blob Detectors, Canny Edge Detector Szeliski Chapter 4, Canny's PaperOpenCV 
23 January 2015 Interest Points, Correspondence - SSD, NCC, Corner Detection, Harris Corner Detector Szeliski Chapter 4, Harris-Stephens Paper, OpenCV 
 27 January 2015Integral Image and Application, Image Pyramids, Gaussian Scale Space  Szeliski Chapter 3, Paper by Witkin, Gaussian Scale Space Paper
30 January 2015
Scale Invariant Feature Transform (SIFT)
Szeliski Chapter 4, Lowe's Paper
 02 February 2015
Feature Detection and Description using SIFT
 SIFT Official Website
03 February 2015
 Feature Detection and Description using Speeded-Up Robust Features (SURF)
 SURF Official Website
06 February 2015
Surface Normal, BRDF, Solid Angle, Radiance and Irradiance, Image Formation
 Horn Chapter 9
 09 Februrary 2015
 Thin Lens Equation, Depth of Field, Angle of View, 2D Projective Plane, Lines and Points
 Hartley and Zisserman Chapter 2
 10 February 2015
 Points and Lines at Infinity, Conics, Projective Transformation
  Hartley and Zisserman Chapter 2
 13 February 20152D Transformations - Isometries, Similarities, Affinities, Projectivities, Degrees of Freedom, Points at Infinity over Transformations Hartley and Zisserman Chapter 2
 16 February 2015Solution to Over-determined Homogeneous and Non-homogeneous Linear Systems, Constrained Minimal Norm Solution using SVD, Estimation of 2D Projective Transformation using Direct Linear Transformation (DLT)Hartley and Zisserman Chapter 4 
20 February 2015 DLT and Normalization, Robust Estimation, Outlier Rejection, RANSACHartley and Zisserman Chapter 4 
9 March 2015 RANSAC for Robust Homography Estimation, Perspective Projection, Image Formation for PinHole Camera, Finite and Infinite Cameras, Intrinsic Camera Parameters  Hartley and Zisserman Chapters 4, 6 
 10 March 2015 Global and Camera Coordinates, Extrinsic Parameters, General and Finite Projective Cameras, Camera Matrix Anatomy Hartley and Zisserman Chapter 6
 13 March 2015
 Estimation of Camera Centre, Principal Plane, Principal Point, Principal Axis from Camera Matrix, Significance of Rows and Columns of Camera Matrix
 Hartley and Zisserman Chapters 6,7
 16 March 2015
Normalized Estimation of Camera Matrix, Projections due to Cameras at Infinity, Affine Camera Matrix
 Hartley and Zisserman Chapters 6,7
 17 March 2015
Differences between Affine and Non-Affine Cameras, Normalized Estimation of Affine Camera Matrix, Homography Arising from Finite Projective Cameras, Projection of a Plane
 Hartley and Zisserman Chapters 7,8
20 March 2015 Homography due to Rotation and Focal Length Change, Application to PTZ Cameras, Two Views - Baseline, Epipolar Geometry, Epipoles  Hartley and Zisserman Chapter 8, 9
 23 March 2015Computer  Vision using OpenCV Python 
24 March 2015 Computer  Vision using OpenCV Python  
27 March 2015 Derivation of Fundamental Matrix, Properties
Perceptual Computing - Guest Lecture by Dr. Achintya Bhowmik, Intel
Hartley and Zisserman Chapter 9 
 30 March 2015Estimation of Fundamental Matrix, 7-point and Normalized 8-point Algorithms, Essential Matrix Hartley and Zisserman Chapter 11
 31 March 2015Robust Estimation of Fundamental Matrix, Mapping of Points on Epipolar Line, Disparity and Depth Hartley and Zisserman Chapter 9, 11
Horn Chapter 12
3 April 2015
 Perceptual Computing - Guest Lecture by Dr. Achintya Bhowmik, Intel 
6 April 2015
 Image Rectification, Homography Estimation, Depth from Stereo
Hartley and Zisserman Chapter 11
7 April 2015
 Projective Ambiguity, Linear Triangulation, Structure Computation from Stereo Pair
 Hartley and Zisserman Chapter 12
10 April 2015 Structure from Motion (SfM), Affine Factorization, Projective Factorization  Hartley and Zisserman Chapter 18, Tomasi-Kanade Factorization
 13 April 2015
 Factorization, Shape from X, Optical Flow, Calculus of Variations
 Hartley and Zisserman Chapter 18, Horn Appendix
17 April 2015 Variational Problems, Euler's Equation, Boundary Conditions, Brightness Constancy Constraint for Optical Flow Horn Appendix, Horn-Schunck Optical Flow
20 April 2015  Optical Flow Estimation - Horn-Schunck, Lucas-Kanade  Horn-Schunck Optical Flow, Lucas-Kanade Paper
21 April 2015  Project Presentations 
 24 April 2015Project Presentations  

Discussions/Queries/Doubts

Use Piazza