Training reinforcement learning agents to perform complex tasks in real world environments is a difficult process, requiring heavy engineering. In fact, we can formulate the interaction between the human engineer and RL agent under training as a decision-making process that the human agent performs, and consequently automate the RL training by learning a decision making policy. In this talk we will cover several examples that illustrate the process, learning intrinsic rewards, RL loss functions, neural network architecture search, curriculum for continual learning, and even learning the accelerator parameters. We show that across different applications, learning to learn methods improve RL agents generalization and performance.