Addendum + More Evaluations

Compound and Complex Sentences

We wish to evaluate if compound sentences work, i.e., do these map equivalently:

"Go to left of north" ?= "Go to north east"

Training Phrases

========= TRAIN SET =========

Agent, navigate to the west edge

Agent, pathfind to the west edge

Agent, find your way to the west edge

Agent, move to the west edge

Agent, your goal is the west edge

Agent, make your way to the west edge

Agent, head towards the west edge

Agent, travel to the west edge

Agent, reach the west edge

Agent, proceed to the west edge

Agent, the west edge is your target

Agent, navigate to the east edge

Agent, pathfind to the east edge

Agent, find your way to the east edge

Agent, move to the east edge

Agent, your goal is the east edge

Agent, make your way to the east edge

Agent, head towards the east edge

Agent, travel to the east edge

Agent, reach the east edge

Agent, proceed to the east edge

Agent, the east edge is your target

Agent, navigate to the south edge

Agent, pathfind to the south edge

Agent, find your way to the south edge

Agent, move to the south edge

Agent, your goal is the south edge

Agent, make your way to the south edge

Agent, head towards the south edge

Agent, travel to the south edge

Agent, reach the south edge

Agent, proceed to the south edge

Agent, the south edge is your target

Agent, navigate to the north edge

Agent, pathfind to the north edge

Agent, find your way to the north edge

Agent, move to the north edge

Agent, your goal is the north edge

Agent, make your way to the north edge

Agent, head towards the north edge

Agent, travel to the north edge

Agent, reach the north edge

Agent, proceed to the north edge

Agent, the north edge is your target

Agent, navigate to the left edge

Agent, pathfind to the left edge

Agent, find your way to the left edge

Agent, move to the left edge

Agent, your goal is the left edge

Agent, make your way to the left edge

Agent, head towards the left edge

Agent, travel to the left edge

Agent, reach the left edge

Agent, proceed to the left edge

Agent, the left edge is your target

Agent, navigate to the right edge

Agent, pathfind to the right edge

Agent, find your way to the right edge

Agent, move to the right edge

Agent, your goal is the right edge

Agent, make your way to the right edge

Agent, head towards the right edge

Agent, travel to the right edge

Agent, reach the right edge

Agent, proceed to the right edge

Agent, the right edge is your target

Agent, navigate to the bottom edge

Agent, pathfind to the bottom edge

Agent, find your way to the bottom edge

Agent, move to the bottom edge

Agent, your goal is the bottom edge

Agent, make your way to the bottom edge

Agent, head towards the bottom edge

Agent, travel to the bottom edge

Agent, reach the bottom edge

Agent, proceed to the bottom edge

Agent, the bottom edge is your target

Agent, navigate to the top edge

Agent, pathfind to the top edge

Agent, find your way to the top edge

Agent, move to the top edge

Agent, your goal is the top edge

Agent, make your way to the top edge

Agent, head towards the top edge

Agent, travel to the top edge

Agent, reach the top edge

Agent, proceed to the top edge

Agent, the top edge is your target

Agent, navigate to the lower edge

Agent, pathfind to the lower edge

Agent, find your way to the lower edge

Agent, move to the lower edge

Agent, your goal is the lower edge

Agent, make your way to the lower edge

Agent, head towards the lower edge

Agent, travel to the lower edge

Agent, reach the lower edge

Agent, proceed to the lower edge

Agent, the lower edge is your target

Agent, navigate to the upper edge

Agent, pathfind to the upper edge

Agent, find your way to the upper edge

Agent, move to the upper edge

Agent, your goal is the upper edge

Agent, make your way to the upper edge

Agent, head towards the upper edge

Agent, travel to the upper edge

Agent, reach the upper edge

Agent, proceed to the upper edge

Agent, the upper edge is your target

Agent, navigate to the south west

Agent, pathfind to the south west

Agent, find your way to the south west

Agent, move to the south west

Agent, your goal is the south west

Agent, make your way to the south west

Agent, head towards the south west

Agent, travel to the south west

Agent, reach the south west

Agent, proceed to the south west

Agent, the south west is your target

Agent, navigate to the west south

Agent, pathfind to the west south

Agent, find your way to the west south

Agent, move to the west south

Agent, your goal is the west south

Agent, make your way to the west south

Agent, head towards the west south

Agent, travel to the west south

Agent, reach the west south

Agent, proceed to the west south

Agent, the west south is your target

Agent, navigate to the south east

Agent, pathfind to the south east

Agent, find your way to the south east

Agent, move to the south east

Agent, your goal is the south east

Agent, make your way to the south east

Agent, head towards the south east

Agent, travel to the south east

Agent, reach the south east

Agent, proceed to the south east

Agent, the south east is your target

Agent, navigate to the east south

Agent, pathfind to the east south

Agent, find your way to the east south

Agent, move to the east south

Agent, your goal is the east south

Agent, make your way to the east south

Agent, head towards the east south

Agent, travel to the east south

Agent, reach the east south

Agent, proceed to the east south

Agent, the east south is your target

Agent, navigate to the north west

Agent, pathfind to the north west

Agent, find your way to the north west

Agent, move to the north west

Agent, your goal is the north west

Agent, make your way to the north west

Agent, head towards the north west

Agent, travel to the north west

Agent, reach the north west

Agent, proceed to the north west

Agent, the north west is your target

Agent, navigate to the west north

Agent, pathfind to the west north

Agent, find your way to the west north

Agent, move to the west north

Agent, your goal is the west north

Agent, make your way to the west north

Agent, head towards the west north

Agent, travel to the west north

Agent, reach the west north

Agent, proceed to the west north

Agent, the west north is your target

Agent, navigate to the bottom left

Agent, pathfind to the bottom left

Agent, find your way to the bottom left

Agent, move to the bottom left

Agent, your goal is the bottom left

Agent, make your way to the bottom left

Agent, head towards the bottom left

Agent, travel to the bottom left

Agent, reach the bottom left

Agent, proceed to the bottom left

Agent, the bottom left is your target

Agent, navigate to the left bottom

Agent, pathfind to the left bottom

Agent, find your way to the left bottom

Agent, move to the left bottom

Agent, your goal is the left bottom

Agent, make your way to the left bottom

Agent, head towards the left bottom

Agent, travel to the left bottom

Agent, reach the left bottom

Agent, proceed to the left bottom

Agent, the left bottom is your target

Agent, navigate to the bottom right

Agent, pathfind to the bottom right

Agent, find your way to the bottom right

Agent, move to the bottom right

Agent, your goal is the bottom right

Agent, make your way to the bottom right

Agent, head towards the bottom right

Agent, travel to the bottom right

Agent, reach the bottom right

Agent, proceed to the bottom right

Agent, the bottom right is your target

Agent, navigate to the right bottom

Agent, pathfind to the right bottom

Agent, find your way to the right bottom

Agent, move to the right bottom

Agent, your goal is the right bottom

Agent, make your way to the right bottom

Agent, head towards the right bottom

Agent, travel to the right bottom

Agent, reach the right bottom

Agent, proceed to the right bottom

Agent, the right bottom is your target

Agent, navigate to the top left

Agent, pathfind to the top left

Agent, find your way to the top left

Agent, move to the top left

Agent, your goal is the top left

Agent, make your way to the top left

Agent, head towards the top left

Agent, travel to the top left

Agent, reach the top left

Agent, proceed to the top left

Agent, the top left is your target

Agent, navigate to the left top

Agent, pathfind to the left top

Agent, find your way to the left top

Agent, move to the left top

Agent, your goal is the left top

Agent, make your way to the left top

Agent, head towards the left top

Agent, travel to the left top

Agent, reach the left top

Agent, proceed to the left top

Agent, the left top is your target

Agent, navigate to the top right

Agent, pathfind to the top right

Agent, find your way to the top right

Agent, move to the top right

Agent, your goal is the top right

Agent, make your way to the top right

Agent, head towards the top right

Agent, travel to the top right

Agent, reach the top right

Agent, proceed to the top right

Agent, the top right is your target

Agent, navigate to the right top

Agent, pathfind to the right top

Agent, find your way to the right top

Agent, move to the right top

Agent, your goal is the right top

Agent, make your way to the right top

Agent, head towards the right top

Agent, travel to the right top

Agent, reach the right top

Agent, proceed to the right top

Agent, the right top is your target

Agent, navigate to the lower left

Agent, pathfind to the lower left

Agent, find your way to the lower left

Agent, move to the lower left

Agent, your goal is the lower left

Agent, make your way to the lower left

Agent, head towards the lower left

Agent, travel to the lower left

Agent, reach the lower left

Agent, proceed to the lower left

Agent, the lower left is your target

Agent, navigate to the left lower corner

Agent, pathfind to the left lower corner

Agent, find your way to the left lower corner

Agent, move to the left lower corner

Agent, your goal is the left lower corner

Agent, make your way to the left lower corner

Agent, head towards the left lower corner

Agent, travel to the left lower corner

Agent, reach the left lower corner

Agent, proceed to the left lower corner

Agent, the left lower corner is your target

Agent, navigate to the lower right

Agent, pathfind to the lower right

Agent, find your way to the lower right

Agent, move to the lower right

Agent, your goal is the lower right

Agent, make your way to the lower right

Agent, head towards the lower right

Agent, travel to the lower right

Agent, reach the lower right

Agent, proceed to the lower right

Agent, the lower right is your target

Agent, navigate to the right lower corner

Agent, pathfind to the right lower corner

Agent, find your way to the right lower corner

Agent, move to the right lower corner

Agent, your goal is the right lower corner

Agent, make your way to the right lower corner

Agent, head towards the right lower corner

Agent, travel to the right lower corner

Agent, reach the right lower corner

Agent, proceed to the right lower corner

Agent, the right lower corner is your target

Agent, navigate to the upper left

Agent, pathfind to the upper left

Agent, find your way to the upper left

Agent, move to the upper left

Agent, your goal is the upper left

Agent, make your way to the upper left

Agent, head towards the upper left

Agent, travel to the upper left

Agent, reach the upper left

Agent, proceed to the upper left

Agent, the upper left is your target

Agent, navigate to the left upper corner

Agent, pathfind to the left upper corner

Agent, find your way to the left upper corner

Agent, move to the left upper corner

Agent, your goal is the left upper corner

Agent, make your way to the left upper corner

Agent, head towards the left upper corner

Agent, travel to the left upper corner

Agent, reach the left upper corner

Agent, proceed to the left upper corner

Agent, the left upper corner is your target

Agent, navigate to the upper right

Agent, pathfind to the upper right

Agent, find your way to the upper right

Agent, move to the upper right

Agent, your goal is the upper right

Agent, make your way to the upper right

Agent, head towards the upper right

Agent, travel to the upper right

Agent, reach the upper right

Agent, proceed to the upper right

Agent, the upper right is your target

Agent, navigate to the right upper corner

Agent, pathfind to the right upper corner

Agent, find your way to the right upper corner

Agent, move to the right upper corner

Agent, your goal is the right upper corner

Agent, make your way to the right upper corner

Agent, head towards the right upper corner

Agent, travel to the right upper corner

Agent, reach the right upper corner

Agent, proceed to the right upper corner

Agent, the right upper corner is your target

Agent, navigate to the south left

Agent, pathfind to the south left

Agent, find your way to the south left

Agent, move to the south left

Agent, your goal is the south left

Agent, make your way to the south left

Agent, head towards the south left

Agent, travel to the south left

Agent, reach the south left

Agent, proceed to the south left

Agent, the south left is your target

Agent, navigate to the west lower

Agent, pathfind to the west lower

Agent, find your way to the west lower

Agent, move to the west lower

Agent, your goal is the west lower

Agent, make your way to the west lower

Agent, head towards the west lower

Agent, travel to the west lower

Agent, reach the west lower

Agent, proceed to the west lower

Agent, the west lower is your target

Agent, navigate to the south right

Agent, pathfind to the south right

Agent, find your way to the south right

Agent, move to the south right

Agent, your goal is the south right

Agent, make your way to the south right

Agent, head towards the south right

Agent, travel to the south right

Agent, reach the south right

Agent, proceed to the south right

Agent, the south right is your target

Agent, navigate to the east lower

Agent, pathfind to the east lower

Agent, find your way to the east lower

Agent, move to the east lower

Agent, your goal is the east lower

Agent, make your way to the east lower

Agent, head towards the east lower

Agent, travel to the east lower

Agent, reach the east lower

Agent, proceed to the east lower

Agent, the east lower is your target

Agent, navigate to the north left

Agent, pathfind to the north left

Agent, find your way to the north left

Agent, move to the north left

Agent, your goal is the north left

Agent, make your way to the north left

Agent, head towards the north left

Agent, travel to the north left

Agent, reach the north left

Agent, proceed to the north left

Agent, the north left is your target

Agent, navigate to the west upper

Agent, pathfind to the west upper

Agent, find your way to the west upper

Agent, move to the west upper

Agent, your goal is the west upper

Agent, make your way to the west upper

Agent, head towards the west upper

Agent, travel to the west upper

Agent, reach the west upper

Agent, proceed to the west upper

Agent, the west upper is your target

Agent, navigate to the north right

Agent, pathfind to the north right

Agent, find your way to the north right

Agent, move to the north right

Agent, your goal is the north right

Agent, make your way to the north right

Agent, head towards the north right

Agent, travel to the north right

Agent, reach the north right

Agent, proceed to the north right

Agent, the north right is your target

Agent, navigate to the east upper

Agent, pathfind to the east upper

Agent, find your way to the east upper

Agent, move to the east upper

Agent, your goal is the east upper

Agent, make your way to the east upper

Agent, head towards the east upper

Agent, travel to the east upper

Agent, reach the east upper

Agent, proceed to the east upper

Agent, the east upper is your target

If this works for completely unseen phrases, we can be certain that our policies can also generalize to the LLM embedding space. We also show that it is possible to use a new training set that contains phrases that aren't easy to template or bin.

Test Phrases

Agent, go down

Agent, go up

Agent, go down and left

Agent, go down and right

Agent, go up and left

Agent, go up and right

LLM understanding this space

If our regression test converges to a small spatial loss, we can argue that these language commands will be interpreted correctly (mapped to correct coordinates) through an LLM encoding.

We find that this indeed the case !

Neither the 'destinations' nor 'action expressions' (and consequently, their combinations) are seen during training.

For unseen tasks (with compound expressions now), we can condition our offline MARL policies with correct goals.

MARL policies operating in this space

Evaluations on the test set.

We find that the robots head towards the correct locations (as expected from the mapping above). The policies are coarsely trained, and for 1/6th the epochs compared to our main results, to quickly ascertain feasibility.

rebuttal_test_rebuttal_test.mp4

Page updated

Google Sites

Report abuse