Abstract State/Action Defination

We perform an abstraction on actions and merge a sequence of actions into one action. Consider the example in Figure shown below to find and start the conversation with Mike, the player needs to perform a sequence of actions such as walking on the map, looking at a person and confirming that it is Mike.

We use an abstract action find to represent this process. In fact, find is implemented as the original APIs of games, encapsulating all the steps of finding an NPC of the game automatically. Using such abstract actions, we avoid making the MDP unnecessarily large and complex.

Due to the huge space of games, We define the state as a vector <v1, v2, . . . , vn>, where each variable vi in the vector is defined based on the domain knowledge of the expert so that only relevant attributes are included, e.g., items of the player and the scenes and positions. We applied a bucket partition to discretize the continuous values into different buckets. Note that, during the abstraction, only the states that have the same actions are grouped into one abstract state such that we can replay the game with the MDP. The original state and selected abstract state collected from the game is shown below.