The paper review assignment template is available at this Overleaf link.
Each week will typically involve 6 paper-related presentations (see the weekly syllabus). You are required to choose 2 of them to write a paper review on. Please pick either 2 of the Monday papers OR 2 of the Wednesday papers. We have 3 each for Monday and Wednesday, so just pick 2 out of 3 to write a review on. There are two special weeks where we are only meeting one day of the week due to a holiday (Labor Day and Thanskgiving). In the first case, we are combining the September 06 papers with those from August 30 for reviews, and will not have paper reviews from August 28 papers. In the second case, pick 2 of 3 papers on the sole day we are meeting that week (November 20). We will provide additional details to students in class.
Please format your paper review according to the following template (see above for the Overleaf link):
1 page summarizing the first paper (5 points).
1 page summarizing the second paper (5 points).
1 page summarizing the connection between the papers (3 points).
For the first two pages, please answer the following questions (again, see the template):
What task is the paper trying to address / solve?
What are the inputs to the learned component?
What is the output of the learned component?
What training signal is used? For example, if RL, what is reward? If not, where does the ground-truth for training come from? What is the loss? If this is a paper which might not train a policy (e.g., it's about datasets) then discuss what policy might be able to make use of this dataset (and how it would be trained). If it's a benchmarking paper, discuss how a policy might be trained on the benchmark.
What insights did they use to enable their method to successfully learn the policy? If this is a benchmarking-style paper that does not demonstrate a successfully trained policy, then write about insights related to the challenges in training such a policy and why the benchmarks fail.
If there are multiple learned components, you can describe inputs/outputs to all of them (if you think they are roughly equal in importance) or to just the most important part (if there are multiple learned parts but one of them is clearly more important and core to the paper's contribution). If it's a dataset-based paper feel free to explain how the dataset forms the inputs to such policies.
We ask these questions to (1) make sure you think carefully about different components of each paper and (2) to avoid allowing a rewrite of the abstract and/or introduction as the paper summary.
The last page is deliberately more open-ended and some things you can write about include:
Comparing and contrasting.
How ideas from one paper can be applied to solve the problem in the other.
A new method incorporating ideas from both paper.
How you might address limitations of both papers.
The grading for a single paper review submission (which, again, involves 2 papers) is out of 13 points:
First page: 5 points (one per question).
Second page: 5 points (one per question).
Third page: 3 points.
For each of the questions on the first and second pages, the following fraction of points will be awarded:
0%: answer is missing.
50%: answer contains little information or mostly misses the point.
100%: answer demonstrates a clear understanding of the paper(s).
For the third page about the open-ended question, it will be graded on a 0-3 scale based on the quality of the answer.
Your overall "paper review" grade for the course is computed as follows:
There are 11 paper review assignments for the semester. There are 15 weeks in the semester. We do not have paper reviews for the first week and the last two weeks, and for the second and third weeks, we are combining the paper review period. Hence that leaves 11 paper reviews ("once a week") for the semester.
Late paper reviews will not be accepted. If the deadline is very close and the review is only partially done, just submit what you have, otherwise Gradescope won't be able to accept it.
Each paper review is graded on the scale above and then we add their sum to get the overall number for the course. Thus your paper reviewing grade is based on 11 reviews (or "review credits"), or 11 x 13 = 143 points so you can get a maximum of 143/143 points.
For extra credit, pick the other day of the week you are not doing paper reviews for, and submit a paper review for that, graded according to the same scale. We will then add on the points for those reviews as part of the normal reviewing grade, so in a single week, it would be possible to get a maximum of 26 points.