Action Space: There are 7 actions [0,1,2,3,4,5,6,7]

State Space: The board is represented by 6x7 matrix where each element is either empty, taken by player 1 or player 2. There are more than 4 trillion legal states

Reward: -1 for loss, 1 for win

NN Architecture: MLP with 2 hidden layers

Observation Representation:

• Using the board state is not the best for training as it does not capture which players' turn it is.

• Observation:

  • 3 * 6 * 7 tensor

  • First channel is 6*7 matrix of 0 or 1, if first player has that spot

  • Second channel is 6*7 matrix of 0 or 1, if second player has that spot

  • Third channel is 6*7 matrix of -1 or 1, -1 for second player, 1 for first player

Training Details:

• Exploration

  • Some experiments were run where the agent took random moves with a certain probability

• Demonstration

  • Some experiments were run where the agent took expert moves with a certain probability. This allowed for faster convergence.

• Evaluation Metrics

  • Optimal play does not necessarily equate to 100% success. An optimal player going first should always win, but an optimal player going second will lose sometimes. I used metrics such as game length and draw rate to see if the agent was learning defensive tactics