Single-agent

Now it's time to actually run an experiment using AI Platform:

Note that running the below command will create numerous machines on the Google Cloud Platform. In this configuration it will incur a cost of about 170 USD in GCP money per one day of training (as of March 2020). Details about the machines assigned to particular jobs can be found here: https://console.cloud.google.com/ai-platform/jobs.

gcp/train_football_checkpoints.sh - starts training with single agent setup on SMM observations.

The above script first builds a docker image that contains code for both the learner (master) and the workers. Briefly, workers execute multiple environment instances and send observations along with rewards to the learner, which performs inference and sends back the actions. The below chart shows an overview of the architecture. More details on the algorithm can be found in the SEED paper.

Multi-agent

The code in SEED repository is by default not suitable for multi-agent setup. If you wish to use a multi-agent setup you need to modify the SEED repo.

Example modifications: https://github.com/Zuuja/seed_rl/tree/multiagent

Below list highlights code that you need to look at to get multi-agent setup running.

• Environment creation. You need to pass number_of_players_agent_controls argument to gym.make called here: seed_rl/football/env.py
• The network. If you wish to modify the network used by default, it is defined here: seed_rl/football/networks.py and used here: seed_rl/football/vtrace_main.py#L39
• Packet-bit observation wrapper. It is responsible for compressing data before sending them over the network. It's good to be aware that it expects specific format of data and adds additional padding to observations: seed_rl/football/observation.py

To use the default network you need to provide it with exactly one observation and one scalar reward. GFootball environment outputs separate observation and reward for each of the controlled players. You can join them by creating wrappers like these:

class SampleMultiAgentRewardWrapper(gym.RewardWrapper):

  def __init__(self, env):

    super(SampleMultiAgentRewardWrapper, self).__init__(env)

  def reward(self, reward):

    return numpy.max(reward)

# Beware that this wrapper is probably not the best (information

# about players and ball is duplicated)

class SampleMultiAgentObservationWrapper(gym.ObservationWrapper):

  def __init__(self, env):

    super(SampleMultiAgentObservationWrapper, self).__init__(env)

    observation_shape = env.observation_space.shape[1:-1] +\

                        (env.observation_space.shape[0] *\

                         env.observation_space.shape[-1], )

    self.observation_space = gym.spaces.Box(

      low=0,

      high=255,

      shape=observation_shape,

      dtype=numpy.uint8)

  def observation(self, observation):

    return numpy.concatenate(observation, axis=-1)

You need to apply the wrappers when the environment is created (in seed_rl/football/env.py).

flags.DEFINE_integer('controlled_agents', 1, 'Number of controlled left agents')

...

def create_environment(_):

 """Returns a gym Football environment."""

 logging.info('Creating environment: %s', FLAGS.game)

 assert FLAGS.num_action_repeats == 1, 'Only action repeat of 1 is supported.'

 channel_dimensions = {

     'default': (96, 72),

     'medium': (120, 90),

     'large': (144, 108),

 }[FLAGS.smm_size]

 env = gym.make(

     'gfootball:GFootball-%s-SMM-v0' % FLAGS.game,

     stacked=True,

     rewards=FLAGS.reward_experiment,

     channel_dimensions=channel_dimensions,

     number_of_left_players_agent_controls= FLAGS.controlled_agents)

 # Beware that football network expects one scalar reward

 # and observation of shape [X,Y,L]

 env = SampleMultiAgentRewardWrapper(env)

 env = SampleMultiAgentObservationWrapper(env)

  # Beware that PackedBitsObservation expects that observation

 # consist of 255 and 0

 return observation.PackedBitsObservation(env)

Above we added the flag for the number of controlled players. You need to add it to the starting script (for example gcp/train_football_3vs1_with_keeper.sh). To control 3 players you need to add:

    - parameterName: game

      type: CATEGORICAL

      categoricalValues:

      - academy_3_vs_1_with_keeper

    - parameterName: controlled_agents

      type: INTEGER

      minValue: 3

      maxValue: 3

You can run an experiment in the same manner as it was run in the single-agent setup i.e. via gcp/train_football_3vs1_with_keeper.sh script created above.

The described modifications are present here: https://github.com/Zuuja/seed_rl/tree/multiagent

Screenshot from tensorboard after 10,5 hours of training on 3vs1 with keeper scenario:

Changing state representation

GFootball environment can provide plenty of information about the current game state e.g. position, velocity, tiredness factor and current action of each player, position, velocity, and rotation of the ball and more. The environment provides three different representations: super mini-map (SMM), simple115 (floats), and rendered game frames (pixels). Each way of encoding the data has its pros and cons e.g. one can take a little space whilst another can provide faster learning rate.

By default SEED uses the SMM representation where observations consist of several 72 by 96 planes of byte data. Each plane focuses on different aspects of the game e.g. position of right/left team or position of the ball.

Another way of representing the game state is simple115 a.k.a. floats. In this representation the whole game state is encoded in 115 floats. More detailed information about the possible game representations can be found in the GFootball documentation: gfootball/doc/observation.md.

To change the representation you need to modify the same locations as when changing to multi-agent setup - network definition and creation, env creation, and packed bits observation wrapper.

For example, if you want to adopt the float representation instead of mini-map, you can use other network (for example seed_rl/agents/vtrace/networks.py). Since floats don’t need compression you should remove the PackedBitsObservation wrapper (at seed_rl/football/env.py#L49):

return observation.PackedBitsObservation(env)

You also need to change the gym make command to change the observation format (seed_rl/football/env.py#L45).

For example to use simple115 observation (the floats observation) change this line:

'gfootball:GFootball-%s-SMM-v0' % FLAGS.game,

to:

   'gfootball:GFootball-%s-simple115-v0' % FLAGS.game,

The example of needed modifications: https://github.com/Zuuja/seed_rl/tree/floats

Screenshot from tensorboard after 9 hours of training on 3vs1 with keeper scenario: