Research

My research concerns the development of intelligent machines - to “teach” computers to (1) understand the behaviour and intent of humans, and (2) to correctly interpret (“Perceive” or “See”) objects and scenes depicted in colour/depth images or videos. I work in the areas of Computer Vision (RGB and 3D - KINECT), Machine Learning, Discrete Optimization, Behavioural Game Theory and Human-Computer Interaction. My current research interests include 3D Reconstruction and Rendering, Probabilistic Programming, Interpretable and Verifiable Knowledge Representations from Deep Models. In terms of applications, I am interested in Conversation agents for Task completion, Machine learning systems for Healthcare and 3D rendering and interaction for augmented and virtual reality.

Some Recent Projects

• Neural Program Synthesis 
• Deep Interpretable Models for Visual and Conversational Data
• Probabilistic Programming for Perception
• Generative Models for Adaptive Crowdsourcing and Aggregation
• Non-greedy Training of Decision/Regression Forests
• Scene Understanding (MobileFusion, Semantic Paint, Segmentation, Detection, Geometry)
• Human Pose Estimation for Kinect and RGB cameras
• KinectFusion: Real time 3D Reconstruction using the Kinect and Image and Video Editing
• Behavioural Game Theory via Research Games

Recent Academic and Industrial Duties (All Academic Duties)

• Principal and Team Lead, DeepMind (Present)
• Director of Strategic AI team (Cognition) in Microsoft Research.
• Machine learning advisor to Rick Rashid, Chief Research Officer of Microsoft.
• Editorial board member, PAMI, IJCV, CVIU
• Area Chair/ Senior PC, ICCV, AAAI, ICML, CVPR

Recent News

• May 2018, 2 papers on (Neural entailment reasoning and RL for Program Synthesis) appear in ICLR
• Dec 2017, 3 papers (disentangled representation learning, neural de-animation, program synthesis with control flow) appear in NIPS
• Sep 2017, 3 papers accepted to ICCV
• June 2017, 5 papers accepted to appear in ICML 2017
• May 2017, Moved to DeepMind
• March 2017, Neural De-rendering paper accepted at CVPR 2017 
• January 2017, 2 Papers accepted in ICLR 2017
• September 2016, 3 papers accepted in NIPS 2016
• August 2016, Paper on continuous relaxation based approaches for efficient inference in Dense CRFs will appear in ECCV 2016.
• June 2016, Our paper describing the role of time spent by crowdworkers can play while aggregating responses in crowdsourcing systems will appear in the Journal of Artificial Intelligence (JAIR).
• May 2016, New work on Hand pose estimation will appear in SIGGRAPH 2016.
• May 2016, Our paper on real time full body reconstruction (Fusion 4D) will appear in SIGGRAPH 2016.
• March 2016, We have developed a super efficient algorithm (Global Patch Collider) for correspondece estimation that will appear in CVPR 2016.
• March 2016, Paper on Layered Scene Decomposition via the Occlusion-CRF will be presented at CVPR 2016.
• February 2016, Our new Visual stortelling dataset and its accompanying paper will be released at NAACL 2016.
• February 2016, Cloze Test - our proposal for evaluating the ability to cature narrative structure - to be released in our NAACL 2016 paper.
• January 2016, Moving to the Microsoft headquaters in Redmond to start a new AI team.
• December 2015, Paper on Non-greedy training of decision trees appears in NIPS 2015.
• December 2015, Paper on interpretable autoencoders - Deep Convolutional Inverse Graphics Network is presented at NIPS 2015.
• September 2015, Paper on Hierarchical Sampling Optimization for Estimating Human Hand Pose will appear at ICCV 2015.
• August 2015, MobileFusion has received a lot of press converage (see articles in (WIRED, Mashable, and even the Register)
• August 2015, Our paper Mobile Fusion that describes a method for generating 3D reconstruction of objects using a standard smartphone has been accepted to appear in ISMAR 2015.
• July 2015, Our paper on Maximum Flows by Incremental Breadth-First Search (with Andrew Goldberg, Sagi Hed, Robert Tarjan, Renato Werneck and Haim Kaplan) will appear at the European Symposium on Algorithms (ESA)
• July 2015, Interactive Scene Understanding (Semantic Paint) is covered widely in the Press (BBC, Engadget, and even the Daily Mail).
• July 2015, Our paper on learning to decipher heaps for software verificiation wins the Best Paper Award at the Constructive ML workshop at ICML 2015.
• June 2015, Our paper on "Interactive Scene Understanding" - a collaboration with University of Oxford and Stanford will be presented at SIGGRAPH 2015. See video here.
• June 2015, The "Picture Probablitic Programming" paper receives the Best Paper Honorable Mention Award at CVPR 2015.
• May 2015, Paper on learning Perturbation Models with Multidimensional Parametric Min-cuts has been accepted to appear in UAI 2015.
• May 2015, Paper on Information Gathering in Networks via Active Exploration has been accepted to appear in IJCAI 2015.
• May 2015, Video of the Hand Pose Estimation work is now available here.
• April 2015, Picture, a new programming language for vision problems, has been accepted to appear in CVPR 2015 and has received a lot of press coverage (MIT News, The Register, Scientific Computing, iProgrammer).
• April 2015, Paper on Real-time Hand Pose Estimation is appearing in CHI 2015.
• March 2015, Our paper on fast hashing has been accepted to appear in CVPR 2015.
• February 2015, Paper on Crowdsourcing Language Understanding in the Wild is accepted to appear in WWW 2015. See blog posts here and here.
• January 2015, Paper on Consensus Message Passing is accepted to appear in AISTATS 2015
• December 2014, Joined the Advisory board for the NEMOG (New Economic Models for Digital Games) project.
• November 2014, Paper on Learning with Multiple Annotation-specific Loss Functions is accepted to appear in EMMCVPR 2015.
• October 2014, Paper and Video describing our work on automatic layout of virtual objects for augmented reality released at ISMAR 2014.
• October 2014, Paper on Just-in-Time Inference is accepted to appear in NIPS 2014.
• September 2014, Paper on Real-time Face Reconstruction from a Single Depth Image is accepted to appear in 3DV 2014.
• August 2014, Our paper on Exploration of Group Viewing Patterns Paper is the Runner-up for the Best Paper Award at TVX 2014.
• July 2014, 3 papers (contour completion, learning with perceptual loss functions, Non-parametric Higher-order MRFs) are accepted to appear in ECCV 2014.
• June 2014, Our demo on FilterForest for Image Denoising is the Runner-up for the Best Demo Award at CVPR 2014.
• May 2014, Paper on automatic layout of virtual objects for augmented reality is accepted to appear at ISMAR 2014.
• April 2014, Paper on community priors for crowdsourcing is the Runner-up for the Best Paper Award at WWW 2014.
• March 2014, Paper on Depth from IR illumination fall-off is accepted to appear in SIGGRAPH 2014.
• February 2014, Our edited volume Tractability: Practical Approaches to Hard Problems is released by Cambridge University Press.
• February 2014, 3 Papers (Learning Portfolios for camera relocatlization, Personalized gesture recognition, Filter Forests for Image Labelling) are accepted to appear in CVPR (2 for oral presentation).
• January 2014, Paper on encoraging diversity in multiple-output prediction is accepted for oral presentation at AISTATS 2014.
• January 2014, Paper on community priors for crowdsourcing is accepted to appear in WWW 2014.
• December 2013, Our paper on the effect of principles on power of agents in strategic games is accepted to appear in AAMAS 2014.
• December 2013, Paper on User Behaviour Adaptation Under Interface Change accepted to appear in IUI 2014.
• November 2013, Our Infer.Net based model for crowdsourcing Won the CrowdScale challenge at HCOMP 2013.
• October 2013, Gave talk at the Human Behaviour Understanding workshop at ACM multimedia.
• September 2013, Paper on Decision DAGs accepted to appear in NIPS 2013.
• September 2013, Invited to join the editorial board of CVIU.
• August 2013, Paper on Sementic labelling of Voxel Spaces accepted to appear in ICCV 2013.
• July 2013, Paper on text detection and recognition accept to appear at ICDAR 2013.
• June 2013, Tutorial on Solving real world problems with RGBD sensors at CVPR 2013.
• June 2013, Gave a talk at the London School of Economics (LSE) on the value of Big Data.
• April 2013, Papers on online algorithms for diverse recommendations and computation of coalition structures in coalition games are accepted to appear in AAAI 2013.
• March 2013, Four papers (two for oral presentation) accepted to appear in CVPR 2013.
• February 2013, Invited to join the editorial board of IJCV.
• January 2013, Paper on faster training of structural SVMs is accepted to appear in AISTATS 2012 ( video ).
• October 2012, PAMI paper describing the pose estimation system for KINECT has been accepted.
• September 2012, Two papers (1) structured output prediction with multiple choices (2) context driven random forests are accepted to appear in NIPS 2012.
• August 2012, The accepted ECCV and DAGM papers are now available online.
• July 2012, Jamie Shotton and I have written a new book chapter on our recent work on Human Pose Estimation for the KINECT.
• June 2012, Five papers (two for oral presentation) accepted to appear in the European Conference on Computer Vision (ECCV).
• May 2012, The MSR KINECT gesture dataset collected for our CHI 2012 paper is now available.
• April 2012, 3 Papers on Behavioural Game Theory and Personality-Online Behaviour patterns are accepted to appear in ACM WebSci 2012 .
• March 2012, Papers accepted to CHI 2012, AISTATS 2012, Eurographics 2012 and PAMI/IJCV are now available online.
• January 2012, Carsten Rother and I are teaching the Advanced Computer Vision course in the Engineering department of the University of Cambridge.
• October 2011, Our paper on simulataneous localization and 3D mapping wins the Best Paper Award at ISMAR 2011.
• October 2011, Interviews on the role of Behavioural game theory appear in Forbes and The Economic Times.
• September 2011, Our first Facebook app (MSR Project Waterloo) for conducting behavioural game theory experiments is now online!
• September 2011, The KinectFusion system has been made public. (Project Page) (video) (See UIST and ISMAR papers for detail)
• June 2011, Our book on Advances in Markov Random Fields for Vision and Image Processing has been published by MIT press.
• June 2011, 4 papers on adpative algorithms for MAP inference appear in AISTATS 2011, ICML 2011 and CVPR 2011.
• December 2011, Our paper on evaluation and learning of interactive Segmentation systems wins the Best Paper Award at ICVGIP 2010.
• October 2010, Our paper on inference with co-occurence potential wins the Best Paper Award at ECCV 2010.

Books and Edited Volumes

• Andrew Blake, Pushmeet Kohli, Carsten Rother Advances in Markov Random Fields for Vision and Image Processing. MIT Press 2011.
• Lucas Bordeaux, Youssef Hamadi, Pushmeet Kohli Tractability: Practical Approaches to Hard Problems . Cambridge University Press 2014.
• Pushmeet Kohli. Minimizing Dynamic and Higher Order Energy Functions using Graph Cuts . Phd Dissertation, 2007. [pdf] 
  (Winner, Sullivan Doctoral Thesis Award) 
  (Runner up, BCS Distinguished Dissertations Award)