Welcome to the course!

We'll update this page with information about the course as we go.

Expectations

You can edit this document to tell us about your expectations for the course. 

• Only put your comments below the title / subtitle

• Please do not delete other people's comments or the title of the document

• If you see a common theme appearing, make a heading 

• Move your comments so that they appear below an appropriate heading

• If your comments disappear, check the version history (but copy them back, do not just restore a previous version)

Feedback

You can give feedback about this course while the course is happening. Use this link to provide feedback. You will need a google id and you can update your answer as we go. 

So far (as of Tuesday night) we have some feedback (3 responses) that was positive about OpenAI Gym but negative about the intro and other feedback that was positive about the whole course! Some like more hands-on programming, others seem to prefer more theory. Will try to have more of both! Please keep the feedback coming.

Vote for your demo

Use this link to vote

Slide materials

Slides are available for reference at https://github.coventry.ac.uk/aa3172/rl-course 

The intro to RL slides are at https://github.coventry.ac.uk/aa3172/presentations/tree/main/2023-02-rl 

Photos

Photos from the event are in a onedrive folder.

Notes

Monte Carlo code

On Tuesday night, we left with our monte carlo routine having a bug. Hopefully you found the immediate problem. However, I also asked GPT-4 for an opinion about the code. I thought the answer was insightful (apart from telling us we should be using gym and not gymnasium, so here it is:

Yes, there are a few bugs and potential issues in this code.

1. The first error is in the import statement. `gym` is the correct library for reinforcement learning environments, not `gymnasium`. 

    Change `import gymnasium as gym` to `import gym`.

2. In the line `Returns = [] * N_S`, you are trying to create a list of lists. But it doesn't create a list of empty lists, it creates an empty list instead. For creating a list of lists you should use list comprehension.

    Change `Returns = [] * N_S` to `Returns = [[] for _ in range(N_S)]`.

3. In this line: `observation, reward, terminated, truncated, info = env.step(int(a))`, the standard `gym` environment's `step` function only returns four values: `observation, reward, done, info`. So trying to unpack these into five variables will lead to an error.

    Change this line to `observation, reward, done, info = env.step(int(a))`, and replace all instances of `terminated` and `truncated` with `done` in the code.

4. In the line `G = GAMMA * G + reward[i]`, you're trying to access the i-th reward with `reward[i]`, but `reward` is a float number, not a list. You should replace `reward[i]` with `rewards[i]`.

5. There is a potential off-by-one error in the for loop. The `range(i)` does not include `i` itself, yet the last step in the episode should be included in the return calculation. 

    Change `for j in range(i):` to `for j in range(i+1):`.

6. Lastly, there's no implementation of a policy. The function `first_visit_mc` takes `policy` as an argument, and it is assumed to be a function or an array that given an observation (or state) provides an action. If it's not a function, and instead an array or list, the dimensions and value ranges of this object should be validated.