Razvan Pascanu

I'm currently a research scientist at DeepMind.

I grew up in Romania and studied computer science and electrical engineering for my undergrads in Germany. I got my MSc from Jacobs University, Bremen in 2009. I hold a PhD from University of Montreal (2014), which I did under the supervision of prof. Yoshua Bengio. My PhD thesis can be found here.

I was involved in developing Theano and help write some of the deep learning tutorials for Theano. I've published several papers on topics surrounding the topics of deep learning and deep reinforcement learning (see my scholar page). I'm one of the organizers of EEML (www.eeml.eu).

Specifically, my research interests include topics as:

    • Optimization & Learning -- During my PhD I was fascinated by second order methods and natural gradient, on which I spend some time. I am interesting in understanding how we can optimize more efficiently at scale deep models. How we can make learning more data-efficient either in supervised scenarios or RL.
    • Memory & RNNs -- I'm particularly interested in how memory is formed in recurrent models, and what kind of structures can help us utilize memory better (topic I've been working on since my Masters). This ranges from altering the optimizer or adding regularizers to impose different properties to structuring the hidden state of the recurrent model. Several of my published work center around understanding and exploring alternative recurrent model formulations.
    • Learning with multiple tasks: Continual Learning, Transfer Learning, Multi-task learning, Curriculum Learning, Meta-learning -- With the explicit goal of improving data efficiency, I have been working on multiple problems formulated around training with multiple tasks. From Continual Learning, when said tasks are explored sequentially, to Transfer Learning, Multi-task learning, Curriculum Learning or Meta-Learning. I've published a few different algorithms that are trying to address these problems from different angles, as well as organized workshop on the topic.
    • Graph Neural Networks -- Adding meaningful structure to neural networks is definitely an important future direction that I believe we need to understand. I have looked at the impact of graph structured neural networks or how to apply neural models to graph structured data, including a recent survey of the field.
    • Theory for deep networks (representation/learning) -- I'm interested in understanding how neural networks work. I've been looking in this direction both from a representational angle (what family of functions can they represent for a fixed sized model) as well as from a learning perspective (the structure of the loss surface of these models) and published works on both topics.
    • Reinforcement Learning, Generative Models -- While my work does not focus directly on these topics, I have been a few works in this space and is a topic of interest to me that comes in and out of focus.

News

I've been involved in organizing:

Invited Talks & Lectures

    • Talk at Center of Theoretical Neuroscience, Columbia University, NY, US, 8th March 2019
    • Talk at Center for Data Science, NYU Data Science, NY, US, 7th March 2019
    • Lecture at TMLW (Timisoara Machine Learning Workshop), Timisoara, Romania, 23 February 2019
    • Talk at TFML (Theoretical Foundation of Machine Learning) Krakow, Poland 13th February 2019.
    • Talk at Institute of Science and Technology (IST) Austria, Vienna, Austira Feb 7th 2018
    • Lecture at TMLSS (Transylvanian Machine Learning Summer School), Cluj-Napoca, Romania, 16th-22nd July 2018
    • Talk at Bucharest Deep Learning MeetUp, Bucharest, Romania, 25th june, 2017
    • Talk at Cluj Innovation Days, Cluj-Napoca, Romania, March, 2017
    • Lecture at Computational and Cognitive Neuroscience Summer School, Shanghai, China, 9th-27th July 2016
    • Lecture at IK (Interdisciplinary Kollege), Gunee/Mohnsee, Germany 4th -11th March, 2016
    • Lecturer for Deep Learning Summer School organized by DENIS, Helsinki, Finland