Joseph Ellis
Ph.D. Candidate / NSF GFRP Fellow / CEO @ Vidrovr
Department of Electrical Engineering
Columbia University
CESPR 707


About Me
I am currently a Ph.D. student at Columbia University in the Digital Video and Multimedia Lab (DVMM), conducting research in computer vision, multimedia, and machine learning under the supervision of Professor Shih-Fu Chang.  I was an NSF IGERT Fellow for the IGERT:From Data to Solutions program, which is directed by Professor Chang and Professor Julia Hirschberg.  I am pretty sure it's also related to the new Institute for Data Sciences at Columbia University.  Starting in Fall of 2014 I am supported now by the NSF Graduate Research Fellowship program, for my thesis direction of structured information extraction in diverse and unconstrained video content. I received a Bachelors of Art and a Bachelors of Science Summa Cum Laude from the University of San Diego in Mathematics and Electrical Engineering in 2012.  I received my M.S. from Columbia University in Electrical Engineering in February of 2014. Before coming to Columbia I worked as a Communications Engineer at MITRE, and interned at Qualcomm.  While at Columbia I have interned at IBM Research under the supervision of Ching-Yung Lin, and at Google in NYC for the Relatedness Matrix Team. For the past 3 years my colleagues and I have been developing News Rover, which has been awarded 1st prize at ACM MM Grand Challenge, 2nd place in the NYC Media Labs Demo Day, and 1st place at the Greater New York Multimedia and Vision Meeting. Check out the live system here.  I am starting a company leveraging the video search and indexing technologies that I have been developing while at Columbia called Vidrovr.

My current research interests lie in using multimodal information from multimedia documents for information extraction, in particular focusing on the interplay between text and visual content.  I am particularly interested in automatically developing visual event schemas given a large unstructured set of image and document pairs.  Given the recent success of deep-learning and large scale supervised methods for computer vision, I believe that the next frontier is to try to understand visual content from weakly supervised data, such as image and caption pairs.  Aside from my main research goals, I am also interested in probabilistic machine learning models and frameworks for multimodal data fusion.


"An investment in knowledge pays the best interest" -- Benjamin Franklin

News
  • My colleague and I have founded a new company, Vidrovr.
  • Our team was asked to write a blog post about our experiences so far in the Verizon Open Innovation Challenge, read it here.
  • I received an Open Innovation grant from Verizon to explore the possibility of linking visual social media content to videos automatically.
  • We were accepted into the first cohort for the NYC Media Lab start-up incubator known as The Combine.
  • My paper "Objective Variables for Probabilistic Revenue Maximization in Second Price Auction with Reserve" has been accepted at WWW2016.
  • My recent paper "Objective Variables for Probabilistic Revenue Maximization in Second-Price Auction with Reserve" has been released on arXiv.
  • I accepted a summer internship position at Google in NYC working with the Relatedness Matrix Team from June - September 2015.
  • My paper "Why We Watch the News: A Dataset For Exploring Sentiment in Broadcast Video News" was accepted at ACM ICMI.
  • My paper "Predicting Evoked Emotions in Video" was accepted at the IEEE International Symposium on Multimedia.
  • News Rover was presented at the NYC Media Lab Annual Summit, and won 2nd prize at the NYC demo showcase.
  • I spent the summer at IBM Research in Yorktown Heights, NY in the Network Science and Big Data Analytics Group.
  • I have accepted the National Science Foundation Graduate Research Fellowship award and my tenure will begin in Fall 2014.
  • I gave a rapid-fire lecture on web development with Flask and MongoDB for Crucial Python [video].
  • I was offered the Department of Defense's National Defense Science and Engineering Graduate Research Fellowship for 2014-2017
  • I was offered the National Science Foundation Graduate Research Fellowship for 2014-2017.
  • News Rover was featured at the OT Media International Workshop, in Paris France in December 2013.
  • News Rover was featured in AlleyWatch (read article) and MadisonandMountainView (read article).
  • News Rover was presented at the New York Tech Meetup in October, and the video of the presentation can be seen here (I am the presenter).
  • We released a YouTube video showcasing the News Rover system, which can be viewed here.
  • My paper "Structured Exploration of Who, What, When, and Where in Heterogeneous Multimedia News Sources" was awarded 1st place at ACM MM 2013 in the Grand Challenge Session.
  • My (Co-Author) paper "Structured Exploration of Who,What,When, and Where in Heterogeneous Multimedia News Sources" has been accepted for ACM MM 2013, in the Grand Challenge Session.
  • I will be attending ECI 2013 at the University of Buenos Aires on July 22-27, 2013. 
  • My (Co-Author) paper "News Rover: Exploring Topical Structures and Serendipity in Heterogeneous Multimedia News" has been accepted for ACM MM 2013, in the Technical Demo Session.
  • I (Co-Author) was recently awarded the Best Technical Demo at the 3rd Annual Greater New York Workshop on Multimedia and Vision.
  • I passed the Doctoral Qualifying Exam at Columbia University in December 2012.