Teaching

Week 2 (2020-04-28) Introduction to Event-based Cameras

• How to Acquire visual information? (Slides)

• Frame- vs Event-based cameras - the curious case of the spinning dot (Slides)

• Bio-inspired motivation - the visual pathways (Slides)

• Sampling in time vs in range (Slides)

• High Dynamic Range (HDR) (Slides)

• Event-based camera companies and devices (Slides)

  • Reading: Section 2.5 of the Survey paper (TPAMI 2020)

• Event-based cameras: Advantages, Disadvantages and Challenges (Slides)

  • Reading: Sections 2.2 and 2.3 of the Survey paper (TPAMI 2020)

• EE Times article: On the rise of event cameras, by T. Delbruck. (On the history of event-based cameras and their current stage of development)

• Exercise 2

  • Instructions (in the Jupyter notebook): Event readout and visualization (representations).

  • Linear voting (smooth histogram computation)

Week 3 (2020-05-05) Camera designs, Event Representations and Event Processing

• Three main event camera designs (Slides)

  • Paper to read: Posch et al., Retinomorphic Event-Based Vision Sensors, Proc. IEEE, 2014.

  • Reading: Section 2.1 of the Survey paper (TPAMI 2020)

• Color event cameras (Slides)

• Event-based data representation (Slides)

  • Reading: Section 3.1 of the Survey paper (TPAMI 2020)

• Methods of event processing (Slides)

  • Reading: Sections 3.2 and 3.3 of the Survey paper (TPAMI 2020)

Week 4 (2020-05-12) Image Reconstruction from Events

• Event Generation Models (Slides)

  • Reading: Section 2.4 of the Survey paper (TPAMI 2020)

• Image Reconstruction from Events

  • Direct Integration (Slides)

  • History, timeline (Slides)

  • Methods (3 videos): 2011-2014, 2016-2017, 2018-2020 (Slides)

  • Discussion (Slides)

  • Reading: Section 4.6 of the Survey paper (TPAMI 2020)

  • Case Study BMVC 2014: image reconstruction and tracking of rotational motion (Slides)

• Exercise 3

  • Problem Statement: Event Visualizer in ROS

    • Here is an introductory video about ROS (there are many online)

    • ROS node handles

  • Repository

  • Video of running example

Week 6 (2020-05-26) Feature Detection and Tracking

Feature Detection and Tracking with Event Cameras (Slides. For animations, please take a look at the YouTube videos linked at the bottom of the slides).

• Tracking blobs of events. Ingredients of the per-event processing paradigm

• Tracking user-defined shapes with point sets

• Feature Detection and Tracking with a DAVIS

• Tracking motion-compensated event features

• Data Association and EKLT tracker

• Comparison of five feature trackers

• Tracking Discussion

• Reading: Section 4.1 of the Survey paper (TPAMI 2020)

• Exercise 4

  • Introduction: Event integration (direct and leaky)

  • Problem Statement - Event integration

  • Video: event integrator (ordinary or "direct" integrator)

  • Video: event integrator ("leaky" integrator)

  • Repository

  • Reference python code

  • Solution

Week 7 (2020-06-02) Optical Flow

Event-based Optical Flow (Slides)

• Introduction to Optical Flow

• Optical Flow: Frame-based vs Event-based

• Event-based Optical Flow: model-based methods

• Event-based Optical Flow: data-driven methods and comparison

• Reading: Section 4.2 of the Survey paper (TPAMI 2020)

Week 8 (2020-06-09) Monocular 3D Reconstruction

This week we change the lecture format. We read in advance a paper and discuss it in class. This will give a closer feeling to research (as in a reading club or seminar).

• Paper to discuss: Rebecq et al., EMVS: Event-based Multi-View Stereo, BMVC 2016. YouTube (BMVC'16 Best Industry Paper Award)
  - Longer version IJCV 2018 (for more details)
  - Please start early; you will not read and understand the paper in two hours.
  - Code

• Questions to guide your reading understanding of EMVS

• Kahoot quiz

• Reading: Section 4.3 of the Survey paper (TPAMI 2020)

• Exercise 5

  • Problem statement: Reconstructing Convolved Images

  • Repository

  • Video result on simulation_3planes sequence (IJRR 2017 dataset)

  • Video result on slider_depth sequence (IJRR 2017 dataset)

  • Video result on bicycle sequence (ACCV 2018 paper)

  • Solution

Week 11 (2020-06-30) Contrast Maximization Framework

• Papers to read in advance:

  • Contrast Maximization Framework (CVPR 2018), YouTube, Poster
    Section 2.1 is, from a global level (using the events in the whole image plane rather than in a local neighborhood), what we are implementing in Exercise 6

  • Motion Segmentation using Contrast Maximization (ICCV 2019), YouTube
    This paper extends the contrast maximization framework to solve the problem of motion segmentation (Section 4.7 of the Survey paper (TPAMI 2020))

  • Open the pdf files with Adobe Acrobat Reader to see the animations in the Figures.

• Kahoot quiz

• Discussed the solution of exercise 6 and difficulties found while completing the exercise.

• Preview of Exercise 7: creating a panoramic image (of reconstructed brightness) using an event camera.

Week 12 (2020-07-07) Case Study on Image Reconstruction from Events

• For Exercise 7, please review the following material from week 4:

  • Video from BMVC 2014: http://www.bmva.org/bmvc/2014/papers/paper066/ (BMVC'14 Best Industry Paper Award)

  • Image Reconstruction - Case Study BMVC 2014 (Slides)

  • Additionally, there are more details in Chapters 4 and 5 of Kim's PhD thesis

• Exercise 7

  • Problem statement: Panoramic image reconstruction with a rotating event camera

  • Repository

  • Video of the dvs_mosaic program in action

  • Sample video (from Matlab code)

  • Solution