Deep Learning the Deep Sky