Submitting to and Reviewing for AAMAS-2021

Right here. While the occasional update to the timeline might be unavoidable, we will try to always keep the information on this page up to date. Note that anything that qualifies as a deadline always comes into force at the end of the day specified, anywhere on Earth (UTC-12).

Important dates for authors:

• Abstract submission: by Friday, 2 October 2020

• Paper submission: by Friday, 9 October 2020

• Rebuttal period: Monday-Wednesday, 23-25 November 2020 (72 hours)

• Author notification: Friday, 18 December 2020

• Camera-ready paper submission (full papers): by Monday, 8 February 2021 (changed)

• Camera-ready paper submission (extended abstracts): by Monday, 15 February 2021 (changed)

• Registration of papers and at least one presenting author per paper: by Monday, 1 March 2021

• Video upload (full papers): by Monday, 22 March 2021

• Poster upload (extended abstracts): by Wednesday, 31 March 2021

Important dates for regular PC members:

• Bidding for papers to review: Monday-Sunday, 5-11 October 2020

• Review assignment released: Sunday, 18 October 2020

• Check you can review the papers assigned to you: by Wednesday, 21 October 2020

• Complete at least 50% of your reviews: by Tuesday, 10 November 2020

• Complete your remaining reviews: by Tuesday, 17 November 2020

• Discussion of papers: Friday-Thursday, 27 November to 3 December 2020

Important dates for senior PC members:

• Bidding for papers to handle: Monday-Sunday, 5-11 October 2020

• Review assignment released: Sunday, 18 October 2020

• Check you can handle the papers assigned to you: by Wednesday, 21 October 2020

• Approve reviews submitted by first deadline: by Friday, 13 November 2020

• Approve remaining reviews: by Friday, 20 November 2020

• Discussion of papers: Friday-Thursday, 27 November to 3 December 2020

• Complete your meta-reviews: by Tuesday, 8 December 2020

Important dates for area chairs:

• Help with desk-rejections as needed: Monday-Wednesday, 12-14 October 2020

• Support the reviewing process as needed: throughout

• Discussion of papers: Friday-Thursday, 27 November to 3 December 2020

• Complete your paper recommendations: by Friday, 11 December 2020

• Area chair meetings: Sunday-Tuesday, 13-15 December 2020

Here it is: https://easychair.org/conferences/?conf=aamas2021

Most email you receive from us will be sent through EasyChair from the address aamas2021@easychair.org (or noreply@easychair.org). Please make sure that such email does not end up in your spam folder (or, if that is impossible, please regularly check your spam folder).

The most likely explanation is that you have more than one EasyChair account, associated with different email addresses, or that the email address on which you received your PC invitation is not the address associated with your existing EasyChair account. See the EasyChair FAQ for more information.

Yes, this year the reviewing schedules of AAAI and AAMAS do not permit authors to resubmit a paper to AAMAS that was submitted to but not accepted to AAAI. Want to understand all the nitty-gritty? Then please read on.

First, some background: In recent years the schedules of AAAI, AAMAS, and IJCAI were carefully aligned so as to give authors of papers rejected from AAAI the opportunity to submit to AAMAS, and those with papers rejected from AAMAS the opportunity to submit to IJCAI. Many people valued this opportunity, as it increased their chances of getting a paper accepted at a top-level conference in a timely manner and as it also provided them with the opportunity to directly respond to reviews. At the same time it is worth pointing out that many people in the community also were somewhat concerned about the time between notifications for one conference and the submission deadline for the next being extremely short (often just a few days). This made it very hard for authors to adequately deal with the feedback received and arguably incentivised them to adopt what many would consider an unethical approach to publishing, namely to simply ignore reviews and just submit one's paper to as many conferences as possible, hoping to get lucky eventually. This has been stressful for authors, frustrating for reviewers, and probably not particularly good for science. But this actually is not the reason why we changed the schedule this year.

Rather, the reason is the change in the AAAI reviewing schedule this year, which in turn is a consequence of the delays in the NeurIPS and EMNLP reviewing schedules caused by the COVID-19 crisis.

AAAI notifications will be sent out on 1 December 2020. Moving the AAMAS submission deadline to December would have been impossible, as there would not have been enough time left to also accommodate the reviewing period for the workshop programme before the conference in early May.

We considered but rejected the idea of having the AAMAS deadline shortly after 13 October 2020, when the first-round rejections of AAAI will be sent out (for papers that receive “two [out of two] reviews that are both sufficiently negative”). Given our experience with reviewing for major AI conferences, we consider it highly unlikely that a paper that received two clearly negative reviews (and no other review) for one such conference can be revised in a matter of days so as to have a realistic chance of getting accepted at the next. So very few, if any, authors would have benefited from a later deadline. At the same time, shortening the reviewing period would have made life more difficult for several hundred core members of our community, besides lowering the overall quality of the AAMAS reviewing process. Finally, authors who might want to revise papers not accepted to AAMAS for submission to IJCAI would have had less time to do so.

We also considered and rejected the idea of introducing a two-phase submission system, similar to the one for AAAI, to allow papers rejected from AAAI with reasonably positive reviews to be considered during a second reviewing phase. This would have pushed the AAMAS notifications to mid February, thereby ruling out resubmission to IJCAI while also making it very hard to organise a good workshop programme.

Yes, the reviewing schedules of the two conferences are aligned so as to give you several weeks to revise your paper in case it does not get accepted to AAMAS. The conditions under which you may submit a paper to IJCAI not accepted to AAMAS are determined by IJCAI, so please carefully study their Call for Papers before submitting.

For AAMAS we review full papers of up to 8 pages (references not included). Your paper should start with a short abstract of roughly 100-300 words. A good abstract will indicate the broader research area you are contributing to, outline the research question you are addressing or the problem you are trying to solve, sketch the results obtained, and mention the methods used. It should allow a potential reviewer to assess whether they want to read the full paper and would be qualified to evaluate it.

You are required to submit a copy of this abstract (in plain text) one week before the main paper submission deadline. That’s the abstract submission deadline. We need this information to be able to assign reviewers to papers so as to be able to start the actual reviewing process shortly after the main paper deadline in a timely manner. For this reason, you should not make significant changes to the abstract of your paper after the abstract submission deadline. Minor reformulations are fine.

We recommend that you try to essentially finish your paper by the abstract submission deadline and use the days between the two deadlines for a few final rounds of polishing and proof-reading.

Many of the papers submitted to AAMAS will be relevant to more than one area. Please choose the area you deem most relevant. This choice of area will determine which area chairs will be handling your paper. But independently from the area you submit to, every regular and senior PC member (except those you have a conflict of interest with) will be able to bid to review/handle your paper.

The rationale for this approach is the following. First, you have a guarantee that someone familiar with the conventions of your own field will be ultimately responsible for the handling of your paper. That’s the reason for having area chairs. Second, when assigning the people who will engage with your paper in depth (the regular and senior PC members), we can choose from the largest possible pool of experts, thereby maximising chances that there is a good match between papers and evaluators.

Please note that, while we reserve the right to assign a paper to a different area in case we believe that doing so will improve the quality of the reviewing process, we do not expect to make much use of those powers.

Coordination, Organisations, Institutions, and Norms

• Coordination and Control

• Organisations and Institutions

• Normative Systems

• Policy, Regulation, Sanctions, and Legislation

• Trust and Reputation

• Fairness, Accountability, and Transparency

• Privacy

• Socio-Technical Systems

• Self-Organising Systems

• Social Networks

Markets, Auctions, and Non-Cooperative Game Theory

• Auctions and Mechanism Design

• Bargaining and Negotiation

• Non-Cooperative Games: Theory and Analysis

• Non-Cooperative Games: Computation

• Practical Applications of Game Theory

• Behavioural Game Theory

Social Choice and Cooperative Game Theory

• Voting and Preference Aggregation

• Social Choice on Social Networks

• Judgment Aggregation

• Fair Allocation

• Matching

• Coalition Formation

• Cooperative Games: Theory and Analysis

• Cooperative Games: Computation

Knowledge Representation, Reasoning, and Planning

• Agent Models and Theories

• Logics for Agent Reasoning

• Reasoning about Epistemic and Social Features

• Formal Specification and Verification

• Argumentation Theory

• Agent Communication Languages

• Distributed Problem Solving

• Multiagent Planning and Scheduling

• Teamwork and Team Formation

• Uncertainty

Learning and Adaptation

• Multiagent Learning

• Adaptive Multiagent Systems

• Adversarial Learning

• Evolutionary Algorithms

• Multi-Task Learning

• Learning Agent Capabilities

• Reinforcement Learning

• Supervised Learning

• Unsupervised Learning

• Deep Learning

Modelling and Simulation of Societies

• Agent-Based Modelling and Simulation: Methodology

• Agent-Based Modelling and Simulation: Analysis

• Agent-Based Modelling and Simulation: Applications

• Emergent Behaviour

• Complex Systems

• Simulation Techniques, Tools, and Platforms

• Agent-Based Computational Epidemiology

Humans and AI / Human-Agent Interaction

• Agent-Based Analysis of Human Interaction

• Agents for Improving Human Cooperation

• Human-Agent Competition and Collaboration

• Multi-User/Multi-Agent Interaction

• Explainability and Trust in Human-Agent Systems

• Ethical Considerations for Human-Agent Systems

• Intelligent Virtual Agents

• Social Agent Models and Architectures

Engineering Multiagent Systems

• Programming Multiagent Systems

• Agent-Oriented Software Engineering

• Formal Methods and Declarative Technologies

• Interoperability and Standardisation Efforts

• Tools and Testbeds for Engineering Multiagent Systems

• Field Reports for Engineered Multiagent Systems

Robotics

• Human-Robot Interaction and Collaboration

• Multi-Robot Systems

• Swarm and Collective Behaviour

• Robot Planning and Control

• Mapping, Localisation, and Exploration

• Machine Learning for Robotics

• Knowledge Representation for Robotics

• Experimental and Fielded Robotic Systems

Innovative Applications

• Emerging Applications

• Deployed Applications

• Integration of Agent-Based and Other Technologies

• Challenges of Applying Agent-Based Technologies

• User Studies

• AI for Social Good

Probably yes. The list of topics is an attempt to categorise most of the submissions we expect to receive for AAMAS, but of course no such list can ever be exhaustive. If, after having read a number of papers published in the proceedings of AAMAS in recent years, you feel that your paper is a good fit for the conference, then please submit.

If you do not find a topic in the list that is an obvious fit for your paper, the best strategy is to pick one or more topics that you think will attract the attention of the kind of people you would like to review your paper.

Topics come from a predefined list, while keywords you pick yourself.

Each area is associated with a number of topics. At submission time you will be asked to pick one or more of these topics. You need to pick at least one topic from the area you are submitting to, but you are free to pick topics from other areas as well.

In addition, you will also be asked to specify three or more keywords (or short key phrases). Unlike topics, these do not come from a predefined list but are determined by you.

Topics and keywords serve similar purposes. First, we use them to help regular and senior PC members identify the papers most relevant to them. Second, we use them to help us group together related papers when designing the final programme of the conference.

Here: https://allea.org/code-of-conduct/

Members of the PC will 'bid' for papers they wish to review. We want to make this bidding data available to the scientific community in anonymised form. This kind of data is useful for researchers working in fields such as computational social choice, preference handling, and recommender systems. We are going to make the data available through PrefLib.org, where you will also find the data collected during a number of previous editions of AAMAS.

We hope that you will agree that this is a useful service to the community. Every principal author (at submission time) and every PC member (sometime after review assignment) nevertheless has the option to opt out of this project and not have their part of the data be included in the dataset released. You can opt out for any reason, no questions asked, and opting out will not have any adversary consequences for you. We are going to compile and release the data only after paper notifications have been sent out, and we (the programme chairs) are going to take care of anonymising the data ourselves, so nobody else will get to see the original data.

If you have any questions or concerns regarding this matter, please do not hesitate to contact us (the programme chairs).

Yes, you can update your paper (and the supplementary material, if applicable) as many times as you like before the paper submission deadline. Similarly, you can update the abstract of your paper as many times as you like before the abstract submission deadline.

No, we ask you not to do this, as handling such conflicts of interest would be particularly difficult.

The 10 areas of interest all overlap somewhat with each other, so we are confident that you will manage to find a reasonably good match for your paper elsewhere. As regular and senior PC members are not tied to specific areas, there will be no problem ensuring that your paper gets reviewed by people with the right kind of expertise.

This seems like a fair question. So, yes: here's a screenshot. Note that we added an extra field after the rebuttal phase.

Double-blind reviewing means that reviewers should not be aware of the identity of the authors of the papers they review (on top of authors not being aware of the identity of their reviewers).

The idea is that double-blind reviewing will make it less likely that the work of well-known authors (or authors from well-known institutions) will be treated more favourably than everyone else’s submissions. More generally, double-blind reviewing is intended to reduce the impact of the various unconscious biases most reviewers will be subject to. This not only makes the system fairer but it also improves the quality of the research we publish as a community.

Of course, the system is not perfect and it is important that everybody is aware of its imperfections. Reviewers may still, in some cases, guess whose paper they are reviewing—even if they are professional about it and try to resist the temptation to do so. And, of course, double-blind reviewing is not a panacea circumventing all and every kind of bias a reviewer might harbour (e.g., against people who aren’t native speakers of English). Reviewers will be instructed to ensure that such potential bias should be avoided as much as possible.

Yes, you can cite yourself. To decide whether doing so would be appropriate, the following thought experiment might help. Imagine another researcher were to write the paper you are working on. Then, would an expert in your field say that this other researcher should cite your prior work? If yes (and only then), you should cite it as well.

Of course, to guarantee anonymity, you must use the third person rather than the first person when referring to your prior work (thus, “X et al. [42] showed …” rather than “We showed … [42]”). If your paper gets accepted, then you may want to reformulate this again (it can sound a bit weird if you are referring to yourself in the third person in a published paper!).

Provided the earlier work has not been published (or is due to be published) in an archival forum, this is ok.

If the paper you are submitting is a variant of a paper previously presented at an informal workshop, then do not cite that earlier paper. Doing so would make it impossible to maintain anonymity. The same applies in case a variant of your paper is available as a technical report or preprint, or in case your paper is based on your thesis.

If your paper gets accepted, then you can cite those earlier versions of your work in the camera-ready version of your paper, and it is up to you to decide what would be appropriate in that case.

If the earlier version of your paper is publicly available at the time of submission, then you must enter that information in the submission form. This information will be visible only to us (the programme chairs). In case one of your reviewers happens to be aware of that earlier work and worries that the anonymously submitted paper may be someone else's attempt to plagiarise your earlier work, we will be in a position to clear up that misunderstanding.

Indeed, the Call for Papers states that you should not “actively promote your paper to a global audience during the reviewing period or the four weeks preceding the abstract submission deadline”. This policy is inspired by the so-called “anonymity period” common, for instance, for NLP conferences (see, e.g., here). It is supposed to address the following dilemma.

On the one hand, you may want to present your work at informal workshops, seminars, and the like, before it becomes sufficiently mature for submission to a major archival conference, such as AAMAS. This is an important part of the process of producing good science and necessary for getting feedback from your peers. So this should be possible. Some of you may also be concerned about getting ‘scooped’ and want to time-stamp your results, for instance, by uploading your work to arXiv. This also is a valid concern. On the other hand, sharing your work with the world—particularly when you do so during the reviewing period or shortly before it—makes it much harder to implement a true double-blind reviewing policy. And that policy is important to be able to ensure the integrity of peer review.

Please try to interpret the policy stated in the Call for Papers in good faith when applying it to your own specific circumstances. The following examples should help:

• Presenting your work to a local and/or small audience is completely ok, at all times. Such activities are crucial for the research process, and having one of your reviewers in the audience should be unlikely.

• Uploading a version of your paper to arXiv (or updating it) during the reviewing period or the four weeks preceding the abstract submission deadline is not ok. Such a move would trigger a notification email being sent to lots of people working in your field, quite possibly including some of your reviewers.

• Uploading (or updating) a version of your paper to arXiv more than four weeks before the abstract submission deadline is fine. You might want to do this if you are worried about getting scooped. You might of course also do this if your submission to AAMAS has grown out of a working paper you have been busy with for a prolonged period of time.

• Putting a version of your paper on your own website during the reviewing period or the four weeks preceding the abstract submission deadline is not in direct violation of our policy. But we think it is much better if you don’t do this. You don’t lose much by waiting, and you help maintain the integrity of peer review in doing so.

• Putting a paper not (yet) accepted to AAMAS-2021 online (anywhere and at any time) using the official AAMAS-2021 style (with the AAMAS-2021 footer on the first page) is never ok. That this is not ok should be obvious: doing so essentially amounts to claiming that your paper has been validated by the AAMAS programme committee, when in fact it hasn't.

Yes, you can submit such supplementary material together with your paper. Put everything in a single ZIP file and upload it separately from your paper (which must be uploaded as an unzipped PDF). Make sure that all of it is fully anonymised.

First of all, provided you do not go beyond 8 pages (plus references) you of course are free to call one of the sections of your paper an 'appendix'.

Then, yes, uploading a separate document as supplementary material is possible in principle. There are no formal restrictions on length or style. Make sure everything is fully anonymised. If you are submitting supplementary material including more than one file (say, a document and some code), then put everything in a single ZIP container before uploading it.

You must make sure that any supplementary material you submit really is supplementary in nature: any information that is essential for understanding or evaluating your paper must be included in the paper itself. For example, it is not acceptable to relegate most of the proofs of your theoretical results to the supplementary material. If the core of your contribution is, say, a theorem that requires a 10-page proof, then AAMAS is not the right venue for your paper. In such a case, you should submit to a journal instead.

When working on a paper for a conference, it is common that your first complete draft is way too long, given the page limit imposed by the conference organisers. These limits are there, first and foremost, to protect the time of the reviewers (and in the old days to save paper), but in our experience, being forced to revise (and revise again and again) a paper to make it fit can also greatly improve the quality of your writing. Explanations have to be formulated more carefully, easier proofs have to be found, tables with less exciting experimental results have to be omitted or replaced by more insightful prose, and so forth. So please do not interpret the availability of the supplementary-material option as a license to circumvent this crucial stage in the research process and simply cut off the last few pages just before the deadline and declare them 'supplementary material'. This almost certainly will result in a poorly presented submission that will have little chance of getting accepted.

We recommend that you work under the assumption that your reviewers will not actually look at the supplementary material you submit.

No, it does not mean that. Including a proof sketch in a full paper can be perfectly reasonable. If you are able to provide enough information so that an expert would be able to fill in the missing details and verify the correctness of your claimed result, then this is absolutely fine. Indeed, sometimes leaving out certain details can greatly improve readability without really affecting reproducibility of the results.

For example, for an NP-hardness proof it will sometimes make sense to just indicate the reduction used but to leave it to the reader to verify the correctness of that reduction. Another situation in which a proof sketch is perfectly acceptable is one where you are using the same technique several times in a row to obtain similar results. If you illustrate your approach properly in the context of the first such proof, then it will often be perfectly acceptable to reduce the remaining proofs to very short sketches that only hint at the main difference relative to the first proof.

We expect that the vast majority of authors of theoretical papers will not submit any supplementary material and that a fair number of those papers will include a couple of proof sketches.

No. This would violate the integrity of the reviewing process. Crucially, we do not have any means of ensuring that the material will remain unaltered throughout the reviewing period. Also, it is conceivable that an author might try to track who is accessing the material, thereby violating the anonymity of reviewers.

The basic principle at work here is that the readers of your published paper should have access to the same information as the reviewers of your submission.

So you should make your (de-anonymised and suitably revised) supplementary material openly available in archival form at the time of publication of your paper. For code or data you may wish to use a service such as Zenodo. For a technical appendix with additional proof details or similar, you may wish to use an archival preprint server such as arXiv. Try to find a place for your supplementary material where you can be reasonably sure that it will still be available, say, 50 years from today.

We recommend that you include a reference to the supplementary material in the camera-ready version of your paper (listed as an item in your bibliography). It should be possible for people to cite your supplementary material independently from citing your paper.

Be brief. Be polite. Try to answer all concrete questions raised in the reviews. Try to make it very clear which point you are responding to (this should be clear also for, say, an area chair or a senior PC member who did not read your paper in any depth).

Avoid the temptation to argue with your reviewers about matters that are subjective in nature (such as the importance of a result). You are unlikely to convince them with just a short paragraph of text. But do try to correct factual misunderstandings of your contribution.

If you can indicate a couple of very concrete changes you plan to make to your paper to address the concerns of reviewers, then do. But avoid making vague or far-reaching promises. Keep in mind that we need to evaluate the paper you submitted, not the paper you promise to write in the future.

Of course, make sure you do not compromise the anonymity of your submission. Never include a link to additional supplementary material. Naturally, including a link to publicly available information is fine (say, to a paper disproving a factually wrong claim by a reviewer). In case this is one of your own publications, do not reveal your identity .

This is the reviewer's answer to the question of whether they believe it would be possible to communicate the main message of your paper in a two-page extended abstract. It is not a statement regarding the scientific quality of your paper. If you indicated in the submission form that you do not wish your paper to be considered for acceptance as an extended abstract, then you will not receive an invitation to do so, independently of how your reviewers have answered this question.

Remember that you can find a copy of the entire review form on this very page.

If a reviewer raises doubts about whether your work is sufficiently original for publication relative to some other paper in the literature and that other paper is actually one of your own papers, then it can be tricky to respond to this without breaking anonymity.

If the other paper is a formally published paper reporting different (but related) results, then you should respond to it in the same way as you would if the other paper were somebody else's work. Once your prior work has become part of the literature, your new work needs to distinguish itself from that prior work in the same way as it has to distinguish itself from prior work by others.

If the other paper is an earlier version of your submitted paper that you presented at a workshop (without formal proceedings) or that you made available as a preprint (at least four weeks before the abstract submission deadline), then you hopefully did the right thing and notified us of this fact in the submission form. In this case, please send us (the programme chairs) an email to notify us of the misunderstanding. We will then be able to clear up the misunderstanding in a manner that preserves anonymity as best as possible. The same applies in case the overlap in contribution noted by a reviewer concerns work in a Master’s or PhD thesis of yours.

We really hope you won't, and we have put mechanisms in place we believe will make it very unlikely that you will (such as carefully selecting PC members, providing clear instructions for less experienced reviewers on this very page, giving them sufficient time to write their reviews, and asking senior PC members to read all reviews before you receive them). But no system is entirely foolproof and most of us will receive a couple of unprofessional reviews at some point in their career.

These are typical signs for unprofessional reviews:

• The review includes rude or otherwise inappropriate language.

• The review is very short (just a few lines of text).

• The review does not focus on the content of your paper but immediately drifts off into only vaguely related territory.

Please note that the following cases do not qualify as unprofessional reviews:

• The reviewer does not seem to grasp parts of your paper that you consider very basic.

• The reviewers asks a question already answered in the paper or laments the omission of information that in fact is there already.

• The reviewer makes a suggestion for a technical improvement that you already know to be wrong or unfruitful.

• The review has grammatical mistakes or looks messy.

• The reviewer disagrees with you about the significance of one of your findings.

• The reviewer shares a negative opinion about the timeliness or importance of the entire research area you are contributing to. They might feel they are giving good advice, so please take it in good faith. We won't reject papers on such grounds.

Please also note that (of course!) different reviewers will sometimes disagree amongst themselves about the merits of your paper. This also is not a sign of lack of professionalism.

So, what do you do if you receive a review that really is unprofessional? In most cases, the best thing to do will be to politely address the issue in your rebuttal. The other reviewers and the senior PC member assigned to your submission will get to see all reviews and your rebuttal. If you highlight a specific problem with one of the reviews, then they will get to discuss that point and, if appropriate, disregard the issue when making the final decision on your paper. The general advice for writing rebuttals applies also here: make it very easy for people to understand what you are referring to.

If you feel the situation is too delicate to address in your rebuttal, then please send us (the programme chairs) an email. Before you do this, discuss the situation with all your coauthors (if any) and recall that in such a situation it is always a good idea to take time to reflect and wait for 24 hours before sending off your email. Please keep your email short and explain your concern in a way that allows us to assess the situation without reading your paper (but only the offending review). We will do our best to respond in a timely manner, but please do not expect an immediate reply.

Yes, of course, you can (and should!) put your paper on your homepage as soon as you have produced a carefully proof-read revision that takes the feedback received from your reviewers into account. You can also post that version to a preprint server such as arXiv.

However, if you put a version of you paper online that includes the official AAMAS copyright statement and/or reference format block, then it must be exactly the version of the paper that has been accepted at AAMAS (i.e., the 8-page version for full papers and the 2-page version for extended abstracts).

In particular, if your paper has been accepted as an extended abstract, you still are welcome to put the full version of the paper online as well. But if you do, it is important that you reformat the full paper using a neutral style and that you remove the AAMAS-2021 copyright statement and reference format block. The reason is that it should be clear to everyone that the full version of the extended abstract is not a paper that has been accepted by the AAMAS programme committee. Only the extended abstract has.

Similarly, if you want to put an extended version of your full paper online (e.g., a version that includes an additional appendix), then this is also fine, but once again, you should remove the official AAMAS copyright statement and reference format block, because the paper you are putting online is not the paper that will appear in the proceedings (which is what you would be claiming if you were to forget to remove those official markers).

If you want to use the AAMAS LaTeX style without the copyright block, simply use the 'nonacm' option: \documentclass[sigconf,nonacm]{aamas}

Yes. Papers that have been accepted as extended abstracts will be published in the official proceedings of AAMAS-2021 alongside the full papers. The only difference is that they are shorter (2 pages plus references) and that they must include the subtitle "Extended Abstract" (which should be mentioned whenever someone is citing the paper).

You may also find relevant the fact that presenting the core idea of your work in an extended abstract does not bar you from publishing a full paper on the same topic at a later point. AAMAS explicitly permits this (see this year’s Call for Papers) and many other conferences (though possibly not all of them) will treat such cases similarly.

You should have received two emails about this on 15 January 2021, one from us (the programme chairs) via EasyChair and one from Sheridan Communications. One of these emails is sufficient to recover all the information you need. If you cannot retrieve either email, please contact us.

Yes, possibly. Please read on.

PC members should be experienced members of our community with relevant technical expertise. The level of experience and expertise required is something that most people attain around the time they complete their PhD, or possibly a little later than that. This is why holding a PhD (or an equivalent qualification) is a necessary (though obviously not a sufficient) condition for joining the PC. Of course, some individuals would in principle be ready to take on this role already somewhat earlier. But as programme chairs we simply have no good means of judging when an exception to the rules might be called for—so we won't make any exceptions.

We believe that anyone writing a review for a conference should first have been the recipient of a significant number of reviews for their own papers submitted to comparable venues. Furthermore, anyone joining the PC of a conference should first have written several one-off reviews for that or comparable conferences as an auxiliary reviewer. That is, there is a crucial difference between getting asked to review a single paper (by someone who specifically thought of you as a good reviewer for that particular paper) and joining the PC (which involves reviewing a whole bunch of papers, not all of which will be a perfect fit to your own area of research).

So, if you are a PhD student and you want to get involved, then ask your supervisor or other more senior colleagues in your immediate environment who serve on the PC whether they might need your help with reviewing one of the papers assigned to them. They remain ultimately responsible for the review and they should give you some feedback on your work. This is an important part of getting trained as a researcher. All such auxiliary reviewers will get acknowledged in the proceedings.

No, of course not.

AAMAS has a rule saying that, as much as possible, nobody should serve as senior PC member (or area chair) for more than two years in a row. This is both about fairness (younger people should have the chance to grow into these senior roles) and quality (experienced people should keep writing reviews). Of course, there are a few instances where this rule was not followed to the letter in the past (mostly due to imperfect bookkeeping), but the intention is to stick to it as much as possible.

It's also helpful to keep in mind that we need around 10 times as many regular PC members as senior PC members, so everyone should expect that in most years their contribution to AAMAS will take the form of a regular PC membership.

Indeed, 23 out of our 604 regular PC members received a submission to review they did not bid for. Sorry about that! This was simply impossible to avoid, given the bids we received. Those 23 "no"-matches represent just over 1% of all matches between submissions and regular PC members. The good news is that almost 93% of all matches are based on "yes"-bids (and the remaining 6% on "maybe"-bids). For the unavoidable "no"-matches we tried to optimise for relevance in terms of topics declared.

For the matching of submissions to SPC members the statistics are not quite as good, just as one would expect given the much smaller number of SPC members (80% "yes", 12% "maybe", 9% "no"). Still, we are confident that every SPC member will be able to competently oversee the reviewing process for the submissions assigned to them.

Thanks to Ronald de Haan for helping with computing the review assignment (of course, without ever getting to see the name of a single author or PC member).

Indeed, this year we ask each regular PC member to deliver the reviews for at least 50% of the papers assigned to them one week ahead of the final reviewing deadline. The main reason for this deviation from what you might be used to is that we want to make the work of our senior PC members a bit more manageable.

While the vast majority of AAMAS reviewers manage to deliver high-quality reviews on time, there are always a few exceptions that require extra attention from senior PC members and, in some cases, force them to solicit additional emergency reviews from other members of the community. The challenge is that, as a senior PC member, you do not know in advance which reviews will turn out to be the problematic ones that take up most of your time.

Given that typically around half of all reviews arrive on the day of the deadline—and many in the final hours—this core part of the work of a senior PC member (and the emergency reviewers they approach) traditionally happens over the course of a single sleepless weekend just after the final review deadline. This clearly is suboptimal. By requiring half of the reviews to arrive one week earlier, we not only allow the senior PC members to spread their work a little, but we also will find out one week earlier which reviewers might need some extra support.

Finally, we of course also think that it is healthy for a regular PC member to not cram all their reviewing assignments into those last couple of days just before the final deadline. We all know this to be true for other people; we just sometimes need a bit of a nudge to apply this golden rule also to ourselves. The staggered review deadline is such a nudge.

First of all: welcome, great that you are getting involved! The fact that you have been included in the PC means we believe that you are qualified for the job. In our experience, newcomers often are particularly thorough and put a lot of effort into their reviews (that’s great, try to keep that for as long as you can manage!). But they sometimes are also particularly harsh in their assessment (please try to resist that temptation!). One of us has written some notes about reviewing a while ago, which you may find helpful.

You are expected to treat all the information regarding submissions you get access to as a PC member as confidential information. This includes the papers assigned to your for review, but also the (anonymised) list of all submissions you get to see during the bidding phase.

Specifically, you should never tell an author that you reviewed their paper, independently of whether the paper was accepted or rejected.

You also cannot make use of the information you have obtained as a PC member to advance your own research. Of course, once a paper becomes publicly available (e.g., if it has been accepted and gets published in the AAMAS proceedings, or if it eventually appears elsewhere), you are free to make full use of the results in that paper.

First of all, please try to adhere to the principles of double-blind peer reviewing as much as possible. Specifically, we ask you to not actively try and de-anonymise the submissions assigned to you (by googling for the paper titles and so forth). Yes, we know that this can be difficult, but this level of self-discipline certainly is expected.

Of course, despite your and the authors’ best efforts to ensure the integrity of peer review, sometimes you simply do know or are easily able to guess who wrote a given paper. In those cases, please try to ignore this information as much as possible when forming your judgment about the quality of the paper.

If you find out who wrote one of the papers you are reviewing/handling and this happens to be someone you should have declared a conflict of interest for, then please get in touch with us (the programme chairs) right away.

While we hope that you will review most of the papers you receive yourself, you also have the option of outsourcing some of these reviews to trusted colleagues in your direct vicinity (so as to not accidentally violate any conflicts of interest). Importantly, if you choose to do this, you still remain responsible for the timely submission and quality of those reviews. You might want to outsource a review if you happen to know someone with specific expertise relevant to a given paper. You might also want to do this in your role as the supervisor of an advanced PhD student who has published at AAMAS or comparable venues before and who now needs to receive training in writing reviews (before they get to fulfil this important duty on their own, after graduation). Involving a PhD student in this manner is fine, as long you remain actively involved yourself and can vouch for the quality of the review submitted.

All such auxiliary reviewers will get acknowledged in the proceedings.

This is about the relevance of the research question addressed and the methods used to the general theme of the conference (autonomous agents and multiagent systems, broadly conceived, and taking into account the diverse ways in which these terms are used throughout the research community).

Try to distinguish this aspect from whether you believe those methods have been used correctly (this is about 'soundness') or whether the results obtained will have impact (this is something you might address when assessing 'significance'). If several papers addressing similar questions and using similar methods have been published at AAMAS before, then the paper is relevant to the conference by definition. In all other cases, please try to be open-minded!

This is about the clarity of exposition. Thus, when evaluating a paper in view of its clarity, you may want to consider questions such as these: Is it clear what the problem is the authors are addressing? Is it clear why they consider this a worthwhile direction of research? And maybe most obviously: Is it clear how they have solved the problem?

That is, are you able to understand the paper—with reasonable effort on your part and to the extent to which you feel that someone with your specific technical background should be able to understand a paper in this specific domain? We would expect that at least the problem statement ('what?') and its motivation ('why?') should be accessible to the majority of attendees of the conference. Of course, parts of the paper may only be fully accessible to an expert in its specific area of specialisation, and that's completely ok. But also those most technical parts of the paper should not be unnecessarily difficult or diffuse.

This criterion covers the scientific importance of the work under review and its likely impact. Importantly, 'impact' is not restricted to impact on society at large by means of deployed applications, but also covers a paper’s impact on the (possibly entirely theoretical) research done by others in the field. So when assessing the significance of a paper, you may consider questions such as whether it is likely to be widely read and whether the approach taken is likely to influence others in their own work. While positive results are often those with the greatest impact, the community relies on negative results seeing the light of day as well, so please try not to be overly biased against such negative results when assessing their significance.

This is about the originality of the ideas found in the paper under review, so this covers the novelty of the research questions considered and the innovativeness of the approach taken. Note that it is perfectly possible to recognise the originality of an idea while doubting its significance or having concerns about the technical soundness of its implementation.

This is, first and foremost, about the technical correctness of the results presented in the paper. But this criterion also covers matters such as the general quality of mathematical writing and the adequacy of experimental methods.

If you are unable to certify correctness because the authors did not provide sufficient information for you to do so, then you should evaluate the paper as performing poorly in terms of this criterion (and also in terms of the 'reproducibility' criterion). If you are unable to certify correctness because you lack some relevant expertise or because you simply do not have enough time to check all the details, then you should remain open-minded as to the soundness of the paper and give little (positive or negative) weight to this criterion when assessing the paper's overall quality (assuming, of course, you did not uncover any major flaws in the parts you were able to check).

Research reported in an AAMAS paper should be reproducible, at least in principle. This means that the paper should provide sufficient detail to allow another team of researchers working in the same area to obtain the same results and thereby confirm their correctness. For experimental results, this means that sufficient information needs to be provided about, for instance, the data used and the parameters of the experimental design chosen. For a theoretical result, this means that sufficient information needs to be provided to allow readers to check the technical correctness of the arguments made.

This is about understanding and appropriate referencing of the state of the art.

If you feel that important references are missing, please be specific and provide full bibliographic details in your review (names of all authors, paper title, conference/journal name, year of publication). As a rule of thumb, try to avoid suggesting that some of your own prior work should have been cited (it's difficult to be objective about such matters, and for clear-cut omissions probably one of the other reviewers will spot the problem, so you don't have to mention the issue).

Note that it is also possible for authors to include too many references. If you feel that some of the references really are not relevant, if you feel that some part of the literature (maybe by a particular group of researchers) is over-represented in the list of references, or if you feel that some inclusions in the list of references should have been motivated better, please do not hesitate to say so in your review.

This is about the basic presentational quality of the paper and covers matters such as grammar, orthography, formatting, the usefulness of figures, and the correctness of the bibliographic information in the list of references. While authors should put significant efforts into polishing the presentation of their work, please be mindful that not everyone can be expected to deliver a manuscript composed in perfect English.

In past editions of AAMAS, reviewers usually were asked to provide numerical scores not only for their overall evaluation of a paper but also for each of several individual reviewing criteria. We decided not to include this feature this year for several reasons.

One reason is that, in our experience, giving people the opportunity to express their evaluation in terms of numerical scores comes with the rather unfortunate temptation to try and compute weighted averages of those scores. But it of course is not the case that the appropriate overall score for a paper can be computed as a weighted average of the individual scores (such as, say, 0.1 * relevance + 0.15 * significance + 0.25 * soundness + …). One mathematical paper with a severe technical mistake might score perfectly on all criteria other than soundness (and still must be rejected), while for another paper, that is more conceptual in nature, it might be hard to even specify what 'soundness' should mean exactly (so that criterion certainly should not be decisive for that particular paper).

Another reason for not asking for numerical scores for individual reviewing criteria is that it is difficult to give a precise semantics to the various scores. This is difficult enough for the overall evaluation (where, arguably, it is unavoidable to use some form of score), but harder still for individual criteria.

A third reason is that, in our experience, having the option to express one's views in terms of a number often leads to people not also expressing that view verbally. But the latter is the much more useful kind of information: we would like you to comment, in words, on all the reviewing criteria that are relevant to the paper you are assessing.

While every paper accepted for publication in the AAMAS proceedings should meet the same high standards, there are important differences in how to apply those standards when reviewing different kinds of papers. For example, the expectations we have for a paper on game theory will be very different from those for a paper on virtual agents.

When reviewing a given paper please take into account the specific requirements for papers in the area the paper has been submitted to. You most likely will be familiar with those requirements, but if you are unsure, please ask the area chair ("associate chair" in EasyChair jargon) assigned to the paper for advice.

Keep in mind that not every paper needs to come with theoretical results in the form of theorems, not every paper needs to present a new algorithm, not every paper needs to discuss an implementation, not every paper needs to contain experimental results, and not every paper needs to demonstrate its immediate relevance to a concrete application.

Specifically, for papers reporting on deployed or emerging applications using agent-based technology, we do not expect authors to necessarily present new theoretical results or new algorithms. Instead, authors might focus on the challenges associated with deploying agent-based technology in the real world, on how users experience a particular application, or on how agent-based technology can be integrated with other technologies. For deployed applications, we would expect that applications have been deployed over a reasonable amount of time, so that evidence of impact is available and lessons learned for the autonomous agents and multiagent systems community about what worked and what did not can be included in the discussion. For emerging applications, it is important that the applications discussed really are novel applications, which have not been tackled with agent-based approaches before.

No. Please try to judge each paper on its own merits, independently from the other papers you happen to be reviewing for this one edition of this one conference. Your batch of papers is unlikely to be representative: as a regular PC member you will get to see fewer than 1% of all submissions, while as a senior PC member you will get to see fewer than 2%.

First, it is simply the polite thing to do: to acknowledge that you have read the response of the authors and considered it.

In most cases, you will agree with some of the response and disagree with other parts. Similarly, you probably will understand some of the response and find other parts unclear. Sometimes, certain remarks will have made you change your mind on some aspect of the paper, while for other aspects your opinion remains unchanged, even in the face of a seemingly good response. Finally, in some cases you may find a specific part of a response rather helpful but ultimately irrelevant to your overall assessment of the paper, while in other cases a response may really have changed your opinion in important ways. Please let the authors know about all of this! This kind of feedback can be very helpful.

Note that you will find an additional field in the review form specifically for this. In addition, feel free to also edit the rest of your review (e.g., to fix typos, correct misunderstandings resolved thanks to the author rebuttal or the feedback from other reviewers, and so forth).

Of course, this can happen, but you should realise that this can easily look unfair to the authors. This applies in particular to points that could have easily been discovered during the initial reviewing phase and that maybe could have been resolved by means of a simple clarification question that the authors could have answered during the rebuttal phase. So if a new point of criticism arises during the PC discussion, please be extra careful to handle the matter correctly.

Naturally, any new point of criticism emerging during the PC discussion should be included in one of the reviews (possibly the metareview). Please make it clear whether the new point you are adding ended up playing a crucial role in arriving at your final recommendation or whether you are simply providing additional information that you hope will be useful to the authors.

No, not always. It is perfectly possible for a group of PC members to agree about whether to recommend acceptance or rejection without all of you reporting the same score.

Indeed, if a paper ends up getting accepted but your own review is more critical than the others, then it is helpful for the authors to have clarity on this matter and see it reflected in your score. Similarly, if a paper ends up getting rejected but your own review is more positive than the others, then it is nice for the authors to see that too.

Of course, sometimes it will be appropriate to update your score. If you do, then you must also change the text of your review. An unmotivated change of score is never acceptable.

All papers are submitted as full papers of up to 8 pages (plus references), but some of these submissions will only be accepted as extended abstracts of up to 2 pages (plus references).

This might be a good option for a submission that is not yet sufficiently mature for publication as a full paper but that includes exciting and promising ideas that are worth sharing with the community.

A necessary condition for accepting a submission as an extended abstract is that its main message can be communicated in the space of 2 pages (plus references). This is a matter that is orthogonal to scientific quality: some strong papers can be condensed in this way, while some cannot; likewise, some weak papers can be, while some cannot. Recall that this is the question we asked you to answer in the review form.

Note that authors had to declare, at submission time, whether or not they would accept an invitation to publish their work as an extended abstract. We have made the conscious decision to not share this information with you, so as to not introduce a bias against accepting submissions as full papers for which authors declared their readiness to publish them in the form of an extended abstract. This means that some papers for which you recommend acceptance as an extended abstract will end up getting rejected simply because the authors chose to opt out of this possibility.

The metareview typically consists of a single paragraph of text. Its core purpose is to explain the decision taken by the programme committee to the authors.

On top of this, you may also use the metareview to communicate additional information to the authors, such as relevant points arising during the discussion that none of the reviewers included in their regular reviews, or some of your own feedback regarding the paper. (The latter—particularly if you have a lot of things to say—you can also do by submitting a regular review yourself.) The remainder of this answer only touches on the aforementioned core purpose of the metareview.

If the decision to be taken is obvious given the reviews, then a single sentence stating this fact is sufficient. For example, if two reviewers recommended to accept the paper and justified their recommendation with high-quality reviews, and the third reviewer did not manage to fully make up their mind but also did not put forward any strong arguments against acceptance, then the paper should get accepted.

But for around half of all submissions, the decision to be take will not be entirely obvious and some explanation will be called for. For example, if one reviewer was arguing for rejection and you, with the support of the other reviewers, decided to overrule this objection, you should say something about this in your review (recalling the main arguments in favour of the paper, then acknowledging the criticism but explaining why it was given less weight).

If you feel that one of the reviews did not fully live up to the standards you would have liked to see, it is ok to signal to the authors that the final decision was largely based on the other reviews. If you overlooked something that's really a little inappropriate in the initial reviews, it is ok to apologise for that oversight to the authors and, again, to assure them about the process that has led to the final decision. Of course, all of this needs to be done with due consideration for the reviewers concerned.

If you recommend that a paper be accepted as an extended abstract, then it is great if you can suggest what part of the work the authors might want to focus on in that extended abstract.

Never mention numerical scores in your metareview. Scores are just an imperfect and overly simplistic reflection of the content of a review. Instead, justify your decision in terms of the content of the reviews.

Note that there is no need to include phrases such as "I recommend to accept this paper" in your review. First, that sounds like a message directed at us (the programme chairs), not the authors, while it is the authors you should be addressing. Second, by the time the authors will get to your metareview, they already know what the acceptance decision is.

For clear-cut or otherwise well-argued decisions, it is extremely unlikely that we would see any need to overrule your decision. But, of course, there will be borderline cases where you only very slightly prefer one decision over another, and we eventually will go for that other decision due to considerations of a more "global" nature (say, comparisons with other borderline papers that are not in your batch). In such a case, it is best if you phrase your metarview as if you knew that your suggested decision will get implemented, but if you also provide some extra text for us in the confidential comments that we can use in case we need to edit your metareview to justify a different final decision.

Just as the content of the anonymous submissions you are reviewing is confidential, so is the content of the PC discussion you are witnessing and contributing to. It goes without saying that it would be a clear betrayal of the trust put in you if you were to reveal to an outsider (such as an author) what a given reviewer said about a given paper.

In principle, we rely on everyone who accepts our invitation to join the PC to ensure they are able to dedicate the time required to their duties and meet all relevant deadlines. If you think you might not be able to do this, then we prefer it if you decline our invitation. But of course, we understand very well that unforeseen problems can occur. In such a case, if you are a regular PC member, please inform the SPC members of all the papers for which you may miss a review deadline as soon as possible. If you are an SPC member, please inform the ACs of the areas for which you are handling papers. If things are really bad and you can only manage to write a single email, please make sure you send it to us (the programme chairs).

Please understand that it is not always possible to arrive at a perfect paper assignment. PC members for AAMAS are experienced representatives of our research community and should be able to provide useful feedback also on papers outside their own immediate area of research. Indeed, a carefully crafted outside assessment can be particularly useful and may help make a paper more accessible and more broadly relevant. But in extreme cases, if you let us (the programme chairs) know within 72 hours of the paper assignment having been sent to you, we will try to find somebody else to review the paper.

Please send us (the programme chairs) an email immediately, mentioning the submission number and title of the paper, and briefly outlining the nature of the CoI. Please carefully check all the papers assigned to you within 72 hours of the assignment having been sent to you, as it will be very difficult for us to reassign a paper while still ensuring a high-quality reviewing process if we only find out later than that.

Please send us (the programme chairs) an email, mentioning the paper submission number and title, and briefly outline your concern.

If the paper has appeared in an archival venue (a journal or a conference/workshop with formally published proceedings), then this would be a clear violation of the terms stated in the Call for Papers. Please send us (the programme chairs) an email, mentioning the submission number, paper title, and details of the other publication (a link to the official publication would be most helpful).

Please note that presentations at conferences/workshops without formally published proceedings are acceptable (even if the paper is, for example, available from the conference/workshop website). Formal publications of extended abstracts of (about) two pages (such as extended abstracts at AAMAS-2020) are also acceptable, as are manuscripts uploaded to arXiv or other preprint servers.

Note that the Call for Papers states that authors should not “actively promote [their] paper to a global audience during the reviewing period or the four weeks preceding the abstract submission deadline”. This stipulation is inspired by the so-called “anonymity period” common, for instance, for NLP conferences (see, e.g., here). We included this clause in the Call for Papers mostly to alert authors to the issue, before it becomes a more serious problem in this community, and not because we expect to invoke it to desk-reject any significant number of papers. Still, please do let us know if you happen to notice violations of this rule that seem particularly egregious.

Please send us (the programme chairs) an email, mentioning the submission number, paper title, and whatever information you have about the other submission.

Including a link to a website with supplementary material, even if fully anonymised, constitutes a clear violation of the terms started in the Call for Papers. Please send us (the programme chairs) an email, mentioning the submission number and the paper title.

Of course, including a link to publicly available information is perfectly fine.

The authors of a small number of papers misunderstood the instructions regarding supplementary material and included an appendix at the end of their paper. Instead, they should have submitted this supplementary material as a separate document.

Provided the appendix starts after the references, so the first 8+1 pages look like a correct submission, we decided to not impose any sanctions in such cases. Please treat this kind of supplementary material just as you would have done if it had been submitted correctly—which means, do feel free to simply ignore it.