Pushmeet Kohli

pushmeet [@] google.com

Publications Research Videos Press Coverage 
Biography | Awards | CV 


My research concerns the development of intelligent machines - to “teach” computers to (1) understand the behaviour and intent of humans, and (2) to correctly interpret (“Perceive” or “See”) objects and scenes depicted in colour/depth images or videos. I work in the areas of Computer Vision (RGB and 3D - KINECT), Machine Learning, Discrete Optimization, Behavioural Game Theory and Human-Computer Interaction. My current research interests include 3D Reconstruction and Rendering, Probabilistic Programming, Interpretable and Verifiable Knowledge Representations from Deep Models. In terms of applications, I am interested in Conversation agents for Task completion, Machine learning systems for Healthcare and 3D rendering and interaction for augmented and virtual reality.

(May 2017) After 10 years at Microsoft research, I have moved to DeepMind.  
(Feb 2016)
 I am starting a new AI team in Microsoft Research called Cognition. We are Hiring! 
(May 2015) I have moved to Microsoft Redmond where I will be acting as the Machine Learning advisor to the Chief Research Officer, Rick Rashid.

Some Recent Projects

Recent Academic and Industrial Duties (All Academic Duties)

  • Principal and Team Lead, DeepMind (Present)
  • Director of Strategic AI team (Cognition) in Microsoft Research.
  • Machine learning advisor to Rick Rashid, Chief Research Officer of Microsoft.
  • Editorial board member, PAMI, IJCV, CVIU
  • Area Chair/ Senior PC, ICCV, AAAI, ICML, CVPR

Recent News

  • May 2018, 2 papers on (Neural entailment reasoning and RL for Program Synthesis) appear in ICLR
  • Dec 2017, 3 papers (disentangled representation learning, neural de-animation, program synthesis with control flow) appear in NIPS
  • Sep 2017, 3 papers accepted to ICCV
  • June 2017, 5 papers accepted to appear in ICML 2017
  • May 2017, Moved to DeepMind
  • March 2017, Neural De-rendering paper accepted at CVPR 2017 
  • January 2017, 2 Papers accepted in ICLR 2017
  • September 2016, 3 papers accepted in NIPS 2016
  • August 2016, Paper on continuous relaxation based approaches for efficient inference in Dense CRFs will appear in ECCV 2016.
  • June 2016, Our paper describing the role of time spent by crowdworkers can play while aggregating responses in crowdsourcing systems will appear in the Journal of Artificial Intelligence (JAIR).
  • May 2016, New work on Hand pose estimation will appear in SIGGRAPH 2016.
  • May 2016, Our paper on real time full body reconstruction (Fusion 4D) will appear in SIGGRAPH 2016.
  • March 2016, We have developed a super efficient algorithm (Global Patch Collider) for correspondece estimation that will appear in CVPR 2016.
  • March 2016, Paper on Layered Scene Decomposition via the Occlusion-CRF will be presented at CVPR 2016.
  • February 2016, Our new Visual stortelling dataset and its accompanying paper will be released at NAACL 2016.
  • February 2016, Cloze Test - our proposal for evaluating the ability to cature narrative structure - to be released in our NAACL 2016 paper.
  • January 2016, Moving to the Microsoft headquaters in Redmond to start a new AI team.
  • December 2015, Paper on Non-greedy training of decision trees appears in NIPS 2015.
  • December 2015, Paper on interpretable autoencoders - Deep Convolutional Inverse Graphics Network is presented at NIPS 2015.
  • September 2015, Paper on Hierarchical Sampling Optimization for Estimating Human Hand Pose will appear at ICCV 2015.
  • August 2015, MobileFusion has received a lot of press converage (see articles in (WIREDMashable, and even the Register)
  • August 2015, Our paper Mobile Fusion that describes a method for generating 3D reconstruction of objects using a standard smartphone has been accepted to appear in ISMAR 2015.
  • July 2015, Our paper on Maximum Flows by Incremental Breadth-First Search (with Andrew Goldberg, Sagi Hed, Robert Tarjan, Renato Werneck and Haim Kaplan) will appear at the European Symposium on Algorithms (ESA)
  • July 2015, Interactive Scene Understanding (Semantic Paint) is covered widely in the Press (BBCEngadget, and even the Daily Mail).
  • July 2015, Our paper on learning to decipher heaps for software verificiation wins the Best Paper Award at the Constructive ML workshop at ICML 2015.
  • June 2015, Our paper on "Interactive Scene Understanding" - a collaboration with University of Oxford and Stanford will be presented at SIGGRAPH 2015. See video here.
  • June 2015, The "Picture Probablitic Programming" paper receives the Best Paper Honorable Mention Award at CVPR 2015.
  • May 2015, Paper on learning Perturbation Models with Multidimensional Parametric Min-cuts has been accepted to appear in UAI 2015.
  • May 2015, Paper on Information Gathering in Networks via Active Exploration has been accepted to appear in IJCAI 2015.
  • May 2015, Video of the Hand Pose Estimation work is now available here.
  • April 2015, Picture, a new programming language for vision problems, has been accepted to appear in CVPR 2015 and has received a lot of press coverage (MIT NewsThe RegisterScientific ComputingiProgrammer).
  • April 2015, Paper on Real-time Hand Pose Estimation is appearing in CHI 2015.
  • March 2015, Our paper on fast hashing has been accepted to appear in CVPR 2015.
  • February 2015, Paper on Crowdsourcing Language Understanding in the Wild is accepted to appear in WWW 2015. See blog posts here and here.
  • January 2015, Paper on Consensus Message Passing is accepted to appear in AISTATS 2015
  • December 2014, Joined the Advisory board for the NEMOG (New Economic Models for Digital Games) project.
  • November 2014, Paper on Learning with Multiple Annotation-specific Loss Functions is accepted to appear in EMMCVPR 2015.
  • October 2014, Paper and Video describing our work on automatic layout of virtual objects for augmented reality released at ISMAR 2014.
  • October 2014, Paper on Just-in-Time Inference is accepted to appear in NIPS 2014.
  • September 2014, Paper on Real-time Face Reconstruction from a Single Depth Image is accepted to appear in 3DV 2014.
  • August 2014, Our paper on Exploration of Group Viewing Patterns Paper is the Runner-up for the Best Paper Award at TVX 2014.
  • July 2014, 3 papers (contour completion, learning with perceptual loss functions, Non-parametric Higher-order MRFs) are accepted to appear in ECCV 2014.
  • June 2014, Our demo on FilterForest for Image Denoising is the Runner-up for the Best Demo Award at CVPR 2014.
  • May 2014, Paper on automatic layout of virtual objects for augmented reality is accepted to appear at ISMAR 2014.
  • April 2014, Paper on community priors for crowdsourcing is the Runner-up for the Best Paper Award at WWW 2014.
  • March 2014, Paper on Depth from IR illumination fall-off is accepted to appear in SIGGRAPH 2014.
  • February 2014, Our edited volume Tractability: Practical Approaches to Hard Problems is released by Cambridge University Press.
  • February 2014, 3 Papers (Learning Portfolios for camera relocatlization, Personalized gesture recognition, Filter Forests for Image Labelling) are accepted to appear in CVPR (2 for oral presentation).
  • January 2014, Paper on encoraging diversity in multiple-output prediction is accepted for oral presentation at AISTATS 2014.
  • January 2014, Paper on community priors for crowdsourcing is accepted to appear in WWW 2014.
  • December 2013, Our paper on the effect of principles on power of agents in strategic games is accepted to appear in AAMAS 2014.
  • December 2013, Paper on User Behaviour Adaptation Under Interface Change accepted to appear in IUI 2014.
  • November 2013, Our Infer.Net based model for crowdsourcing Won the CrowdScale challenge at HCOMP 2013.
  • October 2013, Gave talk at the Human Behaviour Understanding workshop at ACM multimedia.
  • September 2013, Paper on Decision DAGs accepted to appear in NIPS 2013.
  • September 2013, Invited to join the editorial board of CVIU.
  • August 2013, Paper on Sementic labelling of Voxel Spaces accepted to appear in ICCV 2013.
  • July 2013, Paper on text detection and recognition accept to appear at ICDAR 2013.
  • June 2013, Tutorial on Solving real world problems with RGBD sensors at CVPR 2013.
  • June 2013, Gave a talk at the London School of Economics (LSE) on the value of Big Data.
  • April 2013, Papers on online algorithms for diverse recommendations and computation of coalition structures in coalition games are accepted to appear in AAAI 2013.
  • March 2013, Four papers (two for oral presentation) accepted to appear in CVPR 2013.
  • February 2013, Invited to join the editorial board of IJCV.
  • January 2013, Paper on faster training of structural SVMs is accepted to appear in AISTATS 2012 ( video ).
  • October 2012, PAMI paper describing the pose estimation system for KINECT has been accepted.
  • September 2012, Two papers (1) structured output prediction with multiple choices (2) context driven random forests are accepted to appear in NIPS 2012.
  • August 2012, The accepted ECCV and DAGM papers are now available online.
  • July 2012, Jamie Shotton and I have written a new book chapter on our recent work on Human Pose Estimation for the KINECT.
  • June 2012, Five papers (two for oral presentation) accepted to appear in the European Conference on Computer Vision (ECCV).
  • May 2012, The MSR KINECT gesture dataset collected for our CHI 2012 paper is now available.
  • April 2012, 3 Papers on Behavioural Game Theory and Personality-Online Behaviour patterns are accepted to appear in ACM WebSci 2012 .
  • March 2012, Papers accepted to CHI 2012, AISTATS 2012, Eurographics 2012 and PAMI/IJCV are now available online.
  • January 2012, Carsten Rother and I are teaching the Advanced Computer Vision course in the Engineering department of the University of Cambridge.
  • October 2011, Our paper on simulataneous localization and 3D mapping wins the Best Paper Award at ISMAR 2011.
  • October 2011, Interviews on the role of Behavioural game theory appear in Forbes and The Economic Times.
  • September 2011, Our first Facebook app (MSR Project Waterloo) for conducting behavioural game theory experiments is now online!
  • September 2011, The KinectFusion system has been made public. (Project Page) (video) (See UIST and ISMAR papers for detail)
  • June 2011, Our book on Advances in Markov Random Fields for Vision and Image Processing has been published by MIT press.
  • June 2011, 4 papers on adpative algorithms for MAP inference appear in AISTATS 2011, ICML 2011 and CVPR 2011.
  • December 2011, Our paper on evaluation and learning of interactive Segmentation systems wins the Best Paper Award at ICVGIP 2010.
  • October 2010, Our paper on inference with co-occurence potential wins the Best Paper Award at ECCV 2010.

Books and Edited Volumes