HandMeThat
Human-Robot Communication in Physical and Social Environments
Demonstrations
(partially-observable setting for simplicity)
Welcome to the world! In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
The human agent has taken a list of actions towards a goal which includes:
Human moves to the countertop.
Human picks up the vegetable_oil 0 at the countertop.
Human moves to the cabinet.
Human opens the cabinet.
Human puts the vegetable_oil 0 into the cabinet.
Human moves to the countertop.
Human picks up the vegetable_oil 1 at the countertop.
Human moves to the cabinet.
Human puts the vegetable_oil 1 into the cabinet.
Human moves to the refrigerator.
Human opens the refrigerator.
Human picks up the vegetable_oil 2 at the refrigerator.
Human moves to the cabinet.
Human puts the vegetable_oil 2 into the cabinet.
Human moves to the refrigerator.
Human picks up the sushi 2 at the refrigerator.
Human moves to the cabinet.
Human puts the sushi 2 into the cabinet.
Human moves to the microwave.
Human opens the microwave.
Human picks up the sushi 0 at the microwave.
Human moves to the cabinet.
Human puts the sushi 0 into the cabinet.
Human moves to the microwave.
Human picks up the sushi 1 at the microwave.
Human moves to the cabinet.
Human puts the sushi 1 into the cabinet.
Human stops and says 'Please bring me the cooked one in the microwave.' Now you are standing on the floor.
You are at the floor. You see floor.
On floor you can see green highchair 0 green closed dusty briefcase 1 bucket 0 red small closed package 1 small ashcan 0 large ashcan 1 large ashcan 2 xmas_tree 0 xmas_tree 1 dusty xmas_tree 2 gym_shoe 0.
Now it is your turn to help human to achieve the goal!
> move to microwave
You move from the floor to the microwave.
> look
You are at the microwave.
You see open toggled-off dusty microwave.
In microwave you can see cooked candy 1.
> pick up candy 1
You pick up the candy (candy 1) from the microwave.
> move to human
You move from the microwave to the human.
> give candy 1 to human
You give the candy (candy 1) to human.
You reach the goal!
moves: 5, score: 95
Welcome to the world! In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
The human agent has taken a list of actions towards a goal which includes: Human moves to the pool.
Human picks up the rag 1 at the pool.
Human moves to the sink.
Human puts the rag 1 into the sink.
Human stops and says 'Can you move the piece_of_cloth in the pool to the sink.' Now you are standing on the floor.
You are at the floor. You see floor.
On floor you can see blue seat 0 blue seat 1 red seat 2 green highchair 0 red dusty highchair 1 green chair 0 blue basket 1 green small closed dusty package 2 small dusty ashcan 0 small dusty ashcan 1 large ashcan 2 dusty xmas_tree 0 dusty xmas_tree 1 xmas_tree 2.
Now it is your turn to help human to achieve the goal!
> move to pool
You move from the floor to the pool.
> look
You are at the pool. You see stained pool.
In pool you can see toothbrush 0 dishtowel 0 vacuum 0 toothpaste 0 dusty tile 0 tile 1 dusty plywood 0 plywood 1 plywood 2.
> pick up dishtowel 0
You pick up the dishtowel (dishtowel 0) from the pool.
> move to sink
You move from the pool to the sink.
> put dishtowel 0 into sink
You put the dishtowel (dishtowel 0) into the sink.
You reach the goal!
moves: 5, score: 95
Welcome to the world! In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
The human agent has taken a list of actions towards a goal which includes: Human moves to the table.
Human picks up the highlighter 0 at the table.
Human puts the highlighter 0 into the xmas_stocking 0.
Human moves to the cabinet.
Human opens the cabinet.
Human picks up the cube 0 at the cabinet.
Human moves to the shelf.
Human puts the cube 0 into the xmas_stocking 1.
Human moves to the countertop.
Human picks up the candy 0 at the countertop.
Human moves to the shelf.
Human puts the candy 0 into the xmas_stocking 1.
Human moves to the table.
Human picks up the candy 2 from the tray 0 at the table.
Human puts the candy 2 into the xmas_stocking 0.
Human moves to the shelf.
Human stops and says 'Put it over there.' Now you are standing on the floor.
You are at the floor. You see stained floor.
On floor you can see green dusty highchair 0 red closed dusty backpack 0 blue closed dusty sack 0 bucket 1 red basket 1 small ashcan 0 large ashcan 1 small ashcan 2 xmas_tree 0 xmas_tree 1 dusty xmas_tree 2 shoe 1 dusty sock 0.
Now it is your turn to help human to achieve the goal!
> question: can you say it clearly
I mean the candy on the countertop.
> move to countertop
You move from the floor to the countertop.
> look
You are at the countertop. You see stained countertop.
On countertop you can see blue closed jar 0 small bowl 2 large tray 2 apple 2 raspberry 0 raspberry 1 strawberry 0 lemon 0 lemon 2 orange 1 grape 0 grape 1 banana 0 banana 1 olive 1 date 2 tomato 0 juice 0 tea_bag 0 sauce 1 cake 2 candy 1 salad 1 soup 1 marker 2 highlighter 2 pen 0 pen 1 spoon 0 fork 1 lamp 0.
On tray 2 you can see cookie 0 sushi 2.
> pick up candy 1
You pick up the candy (candy 1) from the countertop.
> move to shelf
You move from the countertop to the shelf.
> look
You are at the shelf. You see shelf.
On shelf you can see red small closed box 2 green large closed dusty package 0 dusty xmas_stocking 1 xmas_stocking 2 hammer 0 headset 0 highlighter 1.
In xmas_stocking 1 you can see candy 0 cube 0.
> put candy 1 into xmas_stocking 2
You put the candy (candy 1) into the xmas_stocking (xmas_stocking 2) at the shelf.
> question: where do you want to place it
Put it at the xmas_stocking (xmas_stocking 0).
> pick up candy 1 from xmas_stocking 2
You pick up the candy (candy 1) from the xmas_stocking (xmas_stocking 2) at the shelf.
> move to table
You move from the shelf to the table.
> put candy 1 into xmas_stocking 0
You put the candy (candy 1) into the xmas_stocking (xmas_stocking 0) at the table.
You reach the goal!
moves: 11, score: 51
Welcome to the world! In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
The human agent has taken a list of actions towards a goal which includes: Human moves to the cabinet.
Human opens the cabinet.
Human picks up the kettle 0 at the cabinet.
Human moves to the sink.
Human puts the kettle 0 into the sink.
Human moves to the cabinet.
Human picks up the cup 0 at the cabinet.
Human moves to the sink.
Human puts the cup 0 into the sink.
Human stops and says 'Move the large receptacle in the dishwasher to the sink.' Now you are standing on the floor.
You are at the floor. You see dusty floor.
On floor you can see green dusty seat 0 blue seat 1 blue closed sack 0 bucket 0 bucket 1 small dusty ashcan 0 small dusty ashcan 1 small dusty ashcan 2 xmas_tree 0 dusty xmas_tree 1 dusty xmas_tree 2 sandal 0.
In bucket 1 you can see soap 0.
Now it is your turn to help human to achieve the goal!
> move to dishwasher
You move from the floor to the dishwasher.
> open dishwasher
You open the dishwasher.
> look
You are at the dishwasher. You see open toggled-off dusty dishwasher.
In dishwasher you can see small bowl 0 large bowl 1 small dish 0 pan 1 knife 2 spoon 1 spoon 2.
> pick up bowl 1
You pick up the large bowl (bowl 1) from the dishwasher.
> move to sink
You move from the dishwasher to the sink.
> put bowl 1 into sink
You put the large bowl (bowl 1) into the sink.
You reach the goal!
moves: 6, score: 94
Welcome to the world! In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
The human agent has taken a list of actions towards a goal which includes: Human moves to the refrigerator.
Human opens the refrigerator.
Human picks up the knife 1 at the refrigerator.
Human moves to the table.
Human slices up the onion 1 with the knife 1 at the table.
Human moves to the countertop.
Human slices up the onion 0 with the knife 1 at the countertop.
Human puts the knife 1 on the countertop.
Human picks up the onion 0 at the countertop.
Human moves to the table.
Human puts the onion 0 into the mug 0.
Human picks up the onion 1 at the table.
Human puts the onion 1 into the mug 1.
Human picks up the mug 0 at the table.
Human moves to the refrigerator.
Human puts the mug 0 into the refrigerator.
Human picks up the chestnut 0 at the refrigerator.
Human moves to the countertop.
Human puts the chestnut 0 on the countertop.
Human moves to the refrigerator.
Human picks up the chestnut 2 at the refrigerator.
Human moves to the countertop.
Human puts the chestnut 2 on the countertop.
Human picks up the knife 1 at the countertop.
Human slices up the chestnut 0 with the knife 1 at the countertop.
Human slices up the chestnut 2 with the knife 1 at the countertop.
Human slices up the chestnut 1 with the knife 1 at the countertop.
Human puts the knife 1 on the countertop.
Human picks up the chestnut 1 at the countertop.
Human moves to the refrigerator.
Human puts the chestnut 1 into the mug 0.
Human moves to the countertop.
Human picks up the chestnut 0 at the countertop.
Human moves to the refrigerator.
Human puts the chestnut 0 into the mug 0.
Human moves to the countertop.
Human picks up the chestnut 2 at the countertop.
Human moves to the refrigerator.
Human puts the chestnut 2 into the mug 0.
Human moves to the countertop.
Human picks up the knife 1 at the countertop.
Human moves to the table.
Human slices up the orange 2 with the knife 1 at the table.
Human slices up the orange 1 with the knife 1 at the table.
Human slices up the orange 0 with the knife 1 at the table.
Human puts the knife 1 on the table.
Human picks up the orange 0 at the table.
Human moves to the refrigerator.
Human puts the orange 0 into the mug 0.
Human stops and says 'Can you hand me the unfrozen sliced food on the table.' Now you are standing on the floor.
You are at the floor. You see dusty stained floor.
On floor you can see red highchair 0 bucket 0 bucket 1 red large closed box 1 green large closed box 2 green large closed dusty package 1 blue small closed dusty package 2 small ashcan 0 large dusty ashcan 1 small dusty ashcan 2 xmas_tree 0 dusty xmas_tree 1 xmas_tree 2.
In bucket 0 you can see dishtowel 1.
Now it is your turn to help human to achieve the goal!
> move to table
You move from the floor to the table.
> look
You are at the table. You see stained table.
On table you can see red closed kettle 0 green closed bottle 1 red closed jar 0 large dusty mug 1 small plate 1 small plate 2 saucepan 0 large tray 2 blue basket 1 xmas_stocking 0 dusty xmas_stocking 1 raspberry 0 raspberry 1 raspberry 2 lemon 2 peach 2 date 1 sliced orange 1 sliced orange 2 banana 0 apple 0 apple 1 grape 1 water 1 chicken 0 catsup 0 catsup 1 sauce 1 cookie 1 cake 1 cake 2 chip 0 chip 1 oatmeal 0 oatmeal 1 salad 0 saw 0 toggled-off printer 0 calculator 1 earphone 1 alarm 1 highlighter 0 pencil 0 fork 1 knife 1 lamp 0.
In mug 1 you can see sliced onion 1.
> pick up orange 2
You pick up the orange (orange 2) from the table.
> move to human
You move from the table to the human.
> give orange 2 to human
You give the orange (orange 2) to human.
You reach the goal!
moves: 5, score: 95
Welcome to the world! In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
The human agent has taken a list of actions towards a goal which includes: Human moves to the bathtub.
Human picks up the soap 0 at the bathtub.
Human moves to the countertop.
Human opens the briefcase 0 at the countertop.
Human puts the soap 0 into the briefcase 0.
Human moves to the pool.
Human picks up the toothbrush 0 at the pool.
Human moves to the countertop.
Human puts the toothbrush 0 into the briefcase 0.
Human moves to the sofa.
Human stops and says 'Can you hand me that.' Now you are standing on the floor.
You are at the floor. You see floor.
On floor you can see green highchair 0 red dusty highchair 1 blue dusty highchair 2 green chair 0 red chair 1 blue chair 2 red closed sack 1 dusty bucket 0 green small closed dusty box 0 large ashcan 0 large ashcan 1 small ashcan 2 xmas_stocking 2 dusty xmas_tree 0 xmas_tree 1 xmas_tree 2 gym_shoe 1 shoe 2.
In bucket 0 you can see rag 0 hand_towel 1 scrub_brush 0 vacuum 0.
Now it is your turn to help human to achieve the goal!
> move to sofa
You move from the floor to the sofa.
> look
You are at the sofa.
You see dusty sofa.
On sofa you can see blue closed duffel_bag 0 blue small closed dusty package 0 xmas_stocking 0 dusty bracelet 1 wreath 1 necklace 0 dusty painting 0 dusty painting 1 dusty hat 0.
> question: which type do you mean
I mean the apparel.
> move to cabinet
You move from the sofa to the cabinet.
> open cabinet
You open the cabinet.
> look
You are at the cabinet. You see open dusty stained cabinet.
In cabinet you can see small mug 1 blue large closed dusty box 1 beer 1 juice 0 juice 2 sauce 0 catsup 0 catsup 1 saw 1 toggled-off modem 0 toggled-off modem 1 toggled-off facsimile 1 headset 2 calculator 1 earphone 1 earphone 2 toothbrush 1 broom 0 spoon 1 knife 1 fork 1 bow 1 wreath 0 hardback 0 dusty newspaper 1 gym_shoe 0 dusty gym_shoe 2 shoe 0 shoe 1 hat 1 dusty underwear 0 apparel 0 sweater 0 cube 2.
> pick up apparel 0
You pick up the apparel from the cabinet.
> move to human
You move from the cabinet to the human.
> give apparel 0 to human
You give the apparel to human.
You reach the goal!
moves: 9, score: 82
Welcome to the world! In the room there is the human a countertop a sofa a bed a stove a table a shelf a toilet a cabinet a bathtub a microwave an oven a dishwasher a refrigerator a sink a pool.
The human agent has taken a list of actions towards a goal which includes: Human moves to the cabinet.
Human opens the cabinet.
Human picks up the pop 1 at the cabinet.
Human moves to the floor.
Human puts the pop 1 into the ashcan 1.
Human stops and says 'Can you move the uncooked one to the large ashcan (ashcan 1).' Now you are standing on the floor.
You are at the floor. You see floor.
On floor you can see blue highchair 0 red highchair 1 blue highchair 2 red chair 0 red dusty chair 1 blue dusty chair 2 blue closed sack 0 bucket 0 blue basket 0 blue large closed dusty package 1 small ashcan 0 large ashcan 1 small ashcan 2 xmas_stocking 0 dusty xmas_stocking 1 xmas_tree 0 xmas_tree 1 dusty xmas_tree 2 gym_shoe 0 dusty sock 0 sandal 0.
In ashcan 1 you can see pop 1.
Now it is your turn to help human to achieve the goal!
> move to cabinet
You move from the floor to the cabinet.
> look
You are at the cabinet. You see open dusty cabinet.
In cabinet you can see large dusty mug 2 saucepan 0 casserole 0 large tray 1 small tray 2 green basket 1 green large closed box 1 water 0 milk 1 milk 2 hammer 1 toggled-off facsimile 0 headset 2 marker 0 fork 2 dusty jewelery 2 dusty bracelet 1 document 1 dusty gym_shoe 1 sock 1 sweater 0 cube 0.
On tray 2 you can see salad 1.
> move to table
You move from the refrigerator to the table.
> look
You are at the table. You see dusty stained table.
On table you can see red closed caldron 0 green closed caldron 1 blue closed caldron 2 large dusty mug 0 large plate 0 pan 0 red closed backpack 0 dusty bucket 1 dusty bucket 2 lemon 0 lemon 1 peach 0 apple 0 apple 1 water 1 egg 0 egg 1 chicken 0 beef 0 sugar 1 catsup 1 sauce 0 cake 1 bread 0 hammer 0 toggled-off facsimile 1 spoon 0 candle 0 candle 1.
In bucket 1 you can see toothpaste 1 toothpaste 2.
> move to countertop
You move from the cabinet to the countertop.
> look
You are at the countertop. You see countertop.
On countertop you can see large cup 0 large dish 0 blue closed dusty duffel_bag 1 small dusty tray 0 red small closed dusty box 2 red large closed dusty package 2 xmas_stocking 2 chestnut 0 chestnut 1 chestnut 2 date 0 date 2 peach 1 melon 0 apple 2 pop 0 fish 0 pork 1 pork 2 vegetable_oil 1 catsup 0 cracker 0 cake 0 cookie 0 carving_knife 0 scraper 0 scraper 1 toggled-off scanner 1 headset 1 marker 1 knife 1 lamp 0.
On tray 0 you can see bread 1 hamburger 2 salad 0.
> pick up pop 0
You pick up the pop (pop 0) from the countertop.
> move to floor
You move from the countertop to the floor.
> put pop 0 into ashcan 1
You put the pop (pop 0) into the large ashcan (ashcan 1) at the floor.
You reach the goal!
moves: 9, score: 91