Learning to Look, Looking to Learn