Data Collection – Language Instructions
Data Collection Pipeline for Language-Conditioned Imitation (Real-World Experiments)
Overview
As mentioned in the main text, we use ChatGPT (https://chat.openai.com/chat) dated January 9th to generate diverse language instructions for the following five behaviors grounded in the environment below:
Closing the drawer (by pushing it closed)
Throwing away the bag of chips (green jalapeno chips, to the trash can in the upper left)
Throwing away the coffee pods (on plate, to the trash can)
Moving the coffee mug to the purple/lilac-colored plate
Moving the coffee mug to the yellow-colored plate
Prompt
We use the following prompt:
I'm trying to train a robot assistant that can follow diverse language instructions. One task requires moving an empty chip bag (a green bag of those jalapeno chips) to the garbage. Can you generate 25 natural-sounding instructions (e.g., "throw away the chips")?
The bold, orange text is specific to each behavior/meta-task:
Closing the drawer
"One task requires pushing an open drawer (black, in a shelf) closed."
e.g., "close the drawer"
Throwing away the chips
"One task requires moving an empty chip bag (a green bag of those jalapeno chips) to the garbage"
e.g., "throw away the chips"
Throwing away the coffee pods
"One task requires moving a plate of used coffee pods (the disposable plastic kind) to the garbage"
e.g., "throw out the used coffee pods"
Moving the coffee mug to the purple/lilac plate
"One task requires moving a coffee cup (a cyan mug) to a purple/lilac-colored plate"
e.g., "put the coffee cup on the purple plate"
Moving the coffee mug to the yellow plate
"One task requires moving a coffee cup (a cyan mug) to a yellow-colored plate"
e.g., "put the coffee cup on the yellow plate"
The first 20 generated instructions are used for training, while the remaining 5 are held out for evaluation. Note that all instructions are unique.
Closing the Drawer
Training Instructions
1. "close the drawer"
2. "shut the drawer"
3. "close the black drawer"
4. "push the drawer closed"
5. "push the black drawer closed"
6. "push the black drawer closed all the way"
7. "shut the black drawer"
8. "shut the black drawer tightly"
9. "close the drawer tightly"
10. "push the drawer closed securely"
11. "close the drawer securely"
12. "shut the lower drawer"
13. "close the lower drawer"
14. "push the lower drawer closed"
15. "push the lower drawer closed gently"
16. "close the black drawer by pushing it"
17. "shut the black drawer by pushing it"
18. "push the lower black drawer closed"
19. "shut the drawer all the way"
20. "push the drawer closed all the way"
Heldout Instructions
1. "close the drawer all the way"
2. "push the drawer closed gently"
3. "shut the lower drawer all the way"
4. "shut the lower drawer tightly"
5. "push the black drawer closed fully"
Throwing Away the Chips
Training Instructions
1. "throw away the chips"
2. "put the bag of chips in the trash"
3. "throw the jalapeno chips into the trash"
4. "move the bag of chips to the garbage"
5. "dispose of the empty chips bag"
6. "toss the chips into the black trash can"
7. "put the jalapeno chips in the garbage"
8. "put the empty chips bag in the trash bin"
9. "toss the chips into the garbage can"
10. "throw the chips away"
11. "throw the bag of chips away"
12. "toss the chips in the trash"
13. "put the chips bag in the garbage"
14. "move the chips bag to the trash"
15. "throw the empty jalapeno chips bag into the garbage"
16. "dump the chips bag in the trash"
17. "dispose of the chips"
18. "move the jalapeno chips to the trash"
19. "throw the green bag of chips in the trash"
20. "toss the green chips bag in the garbage"
Heldout Instructions
1. "throw the chips in the garbage"
2. "throw away the chips in the trash"
3. "move the bag of chips to the trash"
4. "move the chips to the trash"
5. "put the empty bag of chips in the trash"
Throwing Away the Coffee Pods
Training Instructions
1. "throw out the used coffee pods"
2. "put the coffee pods in the garbage"
3. "take the coffee pods to the trash"
4. "throw away the coffee pods"
5. "throw the used coffee pods into the garbage"
6. "dispose of the used coffee pods"
7. "throw away the spent coffee pods"
8. "move the coffee pods to the trash"
9. "move the coffee pods to the garbage can"
10. "throw the spent coffee pods in the trash bin"
11. "throw out the used pods"
12. "put the coffee pods in the trash can"
13. "dispose of the coffee pods"
14. "throw out the used coffee pods by placing them in the trash"
15. "dispose of the coffee pods by tossing them in the trash"
16. "toss the coffee pods in the trash"
17. "throw out the used coffee pods"
18. "move the used coffee pods to the trash can"
19. "place the coffee pods in the garbage"
20. "discard the used coffee pods"
Heldout Instructions
1. "throw away the used coffee pods"
2. "throw the used coffee pods away"
3. "throw the coffee pods in the trash"
4. "discard the coffee pods"
5. "move the used coffee pods to the garbage can"
Moving the Coffee Mug to the Purple Plate
Training Instructions
1. "put the coffee cup on the purple plate"
2. "put the mug on the purple plate"
3. "take the coffee mug to the purple plate"
4. "move the coffee to the lilac plate"
5. "place the coffee cup on the lilac plate"
6. "set the coffee mug on the purple plate"
7. "transfer the coffee cup to the lilac plate"
8. "put the coffee cup onto the purple plate"
9. "place the coffee on the lilac plate"
10. "move the coffee mug onto the purple plate"
11. "put the mug on the lilac plate"
12. "move the coffee mug onto the purple plate"
13. "transfer the coffee to the purple-ish plate"
14. "transfer the mug to the purple-ish plate"
15. "transfer the coffee to the lilac plate"
16. "transfer the coffee cup onto the lilac plate"
17. "set the mug on the purple plate"
18. "set the coffee mug on the lilac plate"
19. "set the coffee on top of the purple plate"
20. "put the coffee cup on top of the purple plate"
Heldout Instructions
1. "move the mug to the purple plate"
2. "place the coffee cup on the purple plate"
3. "put the coffee on the purple-ish plate"
4. "take the coffee cup to the lilac colored plate"
5. "set the coffee cup on the purple plate"
Moving the Coffee Mug to the Yellow Plate
Training Instructions
1. "put the coffee cup on the yellow plate"
2. "place the coffee cup onto the yellow plate"
3. "move the coffee cup over to the yellow plate"
4. "move the coffee mug to the yellow plate"
5. "set the coffee cup on top of the yellow plate"
6. "put the coffee mug down on the yellow plate"
7. "transfer the coffee cup to the yellow colored plate"
8. "move the coffee to the yellow colored plate"
9. "move the coffee cup to the yellow colored plate"
10. "transport the coffee mug to the yellow plate"
11. "set the coffee mug on the yellow plate"
12. "place the coffee mug on the yellow colored plate"
13. "position the coffee cup onto the yellow plate"
14. "transfer the coffee cup to the yellow plate"
15. "set the coffee on top of the yellow colored plate"
16. "place the coffee mug gently on top of the yellow plate"
17. "take the coffee cup and set it on the yellow plate"
18. "take the coffee mug and place it on the yellow plate"
19. "take the coffee mug and transport it to the yellow plate"
20. "set the coffee cup down on the yellow plate"
Heldout Instructions
1. "put the coffee mug on top of the yellow plate"
2. "place the coffee cup on the yellow plate"
3. "transfer the coffee cup to the yellow colored plate"
4. "transport the cup of coffee to the yellow plate"
5. "transfer the coffee mug to the yellow plate"