Embedded Files
Results on Montezuma's Revenge
Results on Montezuma's Revenge
mz-0-clean.mp4Reward 13500 (Seed 0)
mz-1-clean.mp4Reward 10500 (Seed 1)
mz-2-clean.mp4Reward 11100 (Seed 2)
mz-3-clean.mp4Reward 12600 (Seed 3)
Absplore (ours): 11020 (mean)
Absplore (ours): 11020 (mean)
Note: Absplore uses RAM information not used by Burda et al., '18
Note: Absplore uses RAM information not used by Burda et al., '18
Prior state-of-the-art: 8152 (mean)
Prior state-of-the-art: 8152 (mean)
(Burda et al. '18)
(Burda et al. '18)
Average human performance: 4753
Average human performance: 4753
(Pohlen et al., '18)
(Pohlen et al., '18)
Results on Pitfall!
Results on Pitfall!
pitfall-0-clean.mp4Reward 9000 (Seed 0)
pitfall-1-clean.mp4Reward 15000 (Seed 1)
pitfall-2-clean.mp4Reward 9750 (Seed 2)
pitfall-3-clean.mp4Reward 9790 (Seed 3)
Pitfall35k.mp4Reward 35000 (20B frames of training)
Absplore (ours): 9959.6 (mean)
Absplore (ours): 9959.6 (mean)
Prior state-of-the-art: 80.52 (mean)
Prior state-of-the-art: 80.52 (mean)
(Keramati et al. '18)
(Keramati et al. '18)
Note: Keramati et al., '18 requires prior knowledge much stronger than that used by Absplore
Note: Keramati et al., '18 requires prior knowledge much stronger than that used by Absplore
Average human performance: 6464
Average human performance: 6464
(Pohlen et al., '18)
(Pohlen et al., '18)
Results on Private Eye
Results on Private Eye
private_eye-0-clean.mp4Reward 31454 (Seed 0)
private_eye-1-clean.mp4Reward 32372 (Seed 1)
private_eye-2-clean.mp4Reward 37400 (Seed 2)
private_eye-3-clean.mp4Reward 33400 (Seed 3)
Absplore (ours): 35636.1 (mean)
Absplore (ours): 35636.1 (mean)
Note: Absplore uses RAM information not used by Ostrovski et al., '17
Note: Absplore uses RAM information not used by Ostrovski et al., '17
Prior state-of-the-art: 15806.5 (mean)
Prior state-of-the-art: 15806.5 (mean)
(Ostrovski et al. '17)
(Ostrovski et al. '17)
Average human performance: 69571
Average human performance: 69571
(Pohlen et al., '18)
(Pohlen et al., '18)
Transferring to New Reward Functions
Transferring to New Reward Functions
After learning the abstract MDP on one task, Absplore can use its learned dynamics model to quickly transfer to new reward functions, achieving ~3x higher reward in 1000x fewer samples than SmartHash (Tang et al., 2017) trained from scratch on the following reward functions:
After learning the abstract MDP on one task, Absplore can use its learned dynamics model to quickly transfer to new reward functions, achieving ~3x higher reward in 1000x fewer samples than SmartHash (Tang et al., 2017) trained from scratch on the following reward functions:
Kill spider: The agent receives 1000 reward for killing the spider in room 13. To kill the spider, the agent must pick up the sword in room 6, and not use the sword on any other monsters (i.e., the skulls in rooms 1, 2, 3, 5, 18 or the spiders in rooms 4, 21).
Kill spider: The agent receives 1000 reward for killing the spider in room 13. To kill the spider, the agent must pick up the sword in room 6, and not use the sword on any other monsters (i.e., the skulls in rooms 1, 2, 3, 5, 18 or the spiders in rooms 4, 21).
Get key: The agent receives 1000 reward for picking up the key in room 14. It receives -100 reward for picking up any other objects (i.e., other keys, gems, or swords), or opening any doors.
Get key: The agent receives 1000 reward for picking up the key in room 14. It receives -100 reward for picking up any other objects (i.e., other keys, gems, or swords), or opening any doors.
Enter room 8: The agent receives 1000 reward for entering room 8, and no other rewards.
Enter room 8: The agent receives 1000 reward for entering room 8, and no other rewards.
Example paths that solve the new reward functions; Kill spider (yellow), Get key (red), Enter room 8 (blue),
kill_spider.mp4Absplore: Transfer to Kill spider
get_key.mp4Absplore: Transfer to Get key
room8.mp4Absplore: Transfer to Enter room 8
Under the Hood
Under the Hood
The abstract MDP consists of abstract states (numbered grid cells) and learned abstract actions (navigate between abstract states). The green squares represent abstract states in the known set.
The abstract MDP consists of abstract states (numbered grid cells) and learned abstract actions (navigate between abstract states). The green squares represent abstract states in the known set.
The agent is currently at the abstract state with the yellow square.
The agent is currently at the abstract state with the yellow square.
The manager makes a plan (red squares), which the worker follows by navigating between abstract states.
The manager makes a plan (red squares), which the worker follows by navigating between abstract states.
mz-0.mp4pitfall-0.mp4private_eye-0.mp4Google Sites
Report abuse