Keeping Humans in the Loop