Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?