Towards Efficient and Portable Robot Learning for Real-World Settings