Information for PC Members

Here it is: https://easychair.org/conferences/?conf=ecai24

Most email you receive from us will be sent through EasyChair from ecai24@easychair.org. Please make sure that such email does not end up in your spam folder (or, if that is impossible, please regularly check your spam folder).

The most likely explanation is that you have more than one EasyChair account, associated with different email addresses, or that the email address on which you received your PC invitation is not the address associated with your existing EasyChair account. See the EasyChair FAQ for more information.

You can reach the two PC chairs by sending email to pc-chairs@ecai2024.eu or ecai24@easychair.org. This will reach both Ulle Endriss and Francisco Melo (and nobody else).

You can find the email address of the SPC member and the AC assigned to a paper on the page with the list of submissions assigned to you (first item in the "Reviews" menu).

Using the the "Add comment" feature on the page with the reviews for a given paper, you can reach everyone assigned to that paper. Your comment, labelled with your name, will be added to the bottom of the page and everyone will get notified by email (unless they removed the paper from their watchlist). Regular PC members and subreviewers will see this only after they have submitted their own review. For the final PC discussion, this is the best way of communicating.

As an SPC member or an AC, you can use the "Email to reviewers" feature on the page with the reviews for a given paper to reach anyone assigned to the paper. But keep in mind that this email will be sent from a generic email address. So you must sign any such message with your name and you must include instructions for the recipient for how to reach you if they want to reply. (EasyChair is actively working on improving this, so this might change soon.)

Please note that minor updates to the schedule are still possible. We will make sure to always have this list reflect the latest version. 

• Selecting topics of expertise in EasyChair: as soon as possible

• Bidding for papers to review: Thursday-Monday, 25-29 April 2024

• Review assignment released: Monday, 6 May 2024

• Check and confirm you can review the papers assigned to you: by Thursday, 9 May 2024

• Complete at least 50% of your reviews: by Thursday, 30 May 2024

• Complete your remaining reviews: by Thursday, 6 June 2024

• Possibly help with emergency reviews: Friday-Friday, 7-14 June 2024

• Participate in discussion of papers: Thursday-Tuesday, 20-25 June 2024

All deadlines are understood to be at the end of the day specified, anywhere on Earth (UTC-12). 

We aim at assigning each regular PC member no more than 4 papers to review. Whether this will be possible depends on the number of submissions and how well the community responds to our calls for joining the programme committee. (Update: We have almost met this target. The maximum number of papers per PC member is 5.)

In case you are a member of the Emergency Reviewer Pool, we aim at assigning you no more than 2 papers to review. You will receive papers to review when an emergency arises (likely in the first half of June).

Please note that minor updates to the schedule are still possible. We will make sure to always have this list reflect the latest version. 

• Selecting topics of expertise in EasyChair: as soon as possible

• Help with recruiting PC members: when prompted (during December-March)

• Bidding for papers to handle: Thursday-Monday, 25-29 April 2024

• Review assignment released: Monday, 6 May 2024

• Check and confirm you can handle the papers assigned to you: by Thursday, 9 May 2024

• Approve reviews submitted by the 50% deadline: by Monday, 3 June 2024 (so: within 4 days)

• Approve remaining reviews: by Monday, 10 June 2024 (so: within 4 days)

• Possibly help with emergency reviews: Friday-Friday, 7-14 June 2024

• Lead discussion of papers: Thursday-Tuesday, 20-25 June 2024

• Complete your metareviews: by Wednesday, 26 June 2024

All deadlines are understood to be at the end of the day specified, anywhere on Earth (UTC-12). 

We aim at assigning each SPC member no more than 12 papers to handle. Whether this will be possible depends on the number of submissions and how well the community responds to our calls for joining the programme committee. (Update: We have met this target.)

Please note that minor updates to the schedule are still possible. We will make sure to always have this list reflect the latest version. Importantly though, as an AC your main task is to help us when things go a little wrong, so we cannot provide a fully worked-out schedule in advance. 

• Selecting topics of expertise in EasyChair: as soon as possible

• Help with recruiting PC and SPC members: when prompted (during December-March)

• Bidding for papers to handle: Thursday-Monday, 25-29 April 2024

• Possibly help with desk rejections: Friday-Friday, 26-31 April 2024

• Review assignment released: Monday, 6 May 2024

• Check and confirm you can monitor the papers assigned to you: by Thursday, 9 May 2024

• Support SPC members to monitor quality of reviews as needed: Friday-Monday, 1-10 June 2024

• Possibly help with handling ethical concerns raised by reviewers: when prompted 

• Possibly help with emergency reviews: Friday-Friday, 7-14 June 2024

• Monitor and support discussion of papers: Thursday-Tuesday, 20-25 June 2024

• Approve or rewrite metareviews: by Friday, 28 June 2024 (so: within 2 days)

All deadlines are understood to be at the end of the day specified, anywhere on Earth (UTC-12). 

We aim at assigning each AC no more than 50 papers to monitor. Whether this will be possible depends on the number of submissions and how well the community responds to our calls for joining the programme committee. (Update: We have met this target.)

No, of course not.

We find it important that also the most experienced members of our community serve in the role of regular PC member every now and then. This is both about fairness (younger people should have the chance to grow into the more senior roles) and quality (experienced researchers should keep writing reviews). 

It's also helpful to keep in mind that we need around 10 times as many regular PC members as SPC members, so everyone should expect that in most years their contribution to ECAI and similar conferences will take the form of a regular PC membership.

Yes. And thanks for asking!

To volunteer as a PC member please complete the for at https://bit.ly/join-ecai-pc (now closed).

Please note that every PC member must (1) be in possession of a PhD (by April 2024), (2) have expertise in the field of AI (as evidenced by having published at relevant conferences), and (3) have some reviewing experience (e.g., as an auxiliary reviewer).

Yes, possibly. Please read on.

PC members should be experienced members of our community with relevant technical expertise. The level of experience and expertise required is something that most people attain around the time they complete their PhD, or possibly a little later than that. This is why holding a PhD (or an equivalent qualification) is a necessary (though obviously not a sufficient) condition for joining the PC. Of course, some individuals would in principle be ready to take on this role already somewhat earlier. But as PC chairs we simply have no good means of judging when an exception to the rules might be called for—so we won't make any exceptions.

We believe that anyone writing a review for a conference should first have been the recipient of a significant number of reviews for their own papers submitted to comparable venues. Furthermore, anyone joining the PC of a conference should first have written several one-off reviews for that or comparable conferences as an additional reviewer. That is, there is a crucial difference between getting asked to review a single paper (by someone who specifically thought of you as a good reviewer for that particular paper) and joining the PC (which involves reviewing a whole bunch of papers, not all of which will be a perfect fit to your own area of research).

So, if you are a PhD student and you want to get involved, then ask your supervisor or other more senior colleagues in your immediate environment who serves on the PC whether they might need your help with reviewing one of the papers assigned to them. They remain ultimately responsible for the review and they should give you some feedback on your work. This is an important part of getting trained as a researcher. All such additional reviewers will get acknowledged in the proceedings.

How? To select your topics, on the EasyChair page for ECAI, go to the 'Conference' menu and select 'My topics'. You can also have a look at the list of topics right here.

Why? We ned you to select topics for two reasons:

• First, this will allow us to estimate for which areas we still need to recruit more PC members. That's why it is important that you select your topics right away.

• Second, during paper bidding, you will see the submissions ordered in terms of overlap with your selected topics, making this step much easier for you. 

How many? To be able to generate a reasonably accurate paper ordering for you during bidding, we require that you select at least 3 topics but believe that the quality of the ranking will be better in almost all cases if you select more topics than that. SPC members and area chairs definitely should select significantly more topics.

Making small changes to your topic selection every now and then is completely fine.

Go to the EasyChair site, log in, select ECAI-2024, and then click on "Bidding" (in the top left corner). You will see the papers ordered by relevance, given the topics of expertise you have specified.

Pick "yes" if you would be happy to be assigned to the paper. Bid "maybe" if you are less excited about the paper but feel that you would be able to deal with it. Of course, others will also bid for some of these papers, so for us to have a chance of computing a good assignment, you need to bid for many more papers than you will receive in the end. Bidding for more papers means getting better papers, not getting more papers. Note that you can change your list of topics to see the paper in a different order.

Please also use this opportunity to indicate conflicts of interest for those (anonymous) papers where you happen to be aware of such a conflict.

Computing a good review assignment is very difficult, and in most cases it is simply impossible to assign papers only to people who bid for them. Indeed, a small number of papers usually do not receive any bids at all, and these papers still need to get reviewed. Also keep in mind that any paper you bid for might also receive bids from many others. This is why we recommended that everyone should bid for significantly more papers than they will end up receiving.

Of course, getting a paper outside your core area of expertise need not always be bad. Sometimes a considerate review by a non-expert providing an outside perspective can be very helpful. If applicable, please try to provide such a review.

In any case, the quality of our (initial) review assignment is actually pretty good:

• PC: For regular PC members, 90% of paper assignments are based on "yes" bids and 8% on "maybe" bids. So for only 2% of the cases where a PC member received a paper that PC member did not bid for that paper. Furthermore, not a single PC member received more than one paper they did not bid for. We had recommended that PC members bid for at least 30 papers. The median size of bids received was 23 (amongst PC members who did bid at all).

• SPC: For SPC members, finding a good review assignment tends to be more difficult, because the pool of people is much smaller (and thus less diverse). Luckily, getting a couple of papers outside of your core area of expertise is much less of a problem at this level. In the end, 73% of paper assignments to SPC members were based on "yes" bids, 14% on "maybe" bids, and 13% on "no" bids. We had recommended that SPC members bid for at least 50 papers. The median size of bids submitted was 46.

• AC: For area chairs, due to the recommended bid size being very large, we gave people the option to generate their bids for them, on the basis of their declared topics of expertise. Around one third of area chairs took us up this offer. It therefore would not be meaningful to report detailed statistics as we did above, but the overall quality of the AC assignment seems to be similar as for the SPC assignment (or possibly a bit better).

Finally, here is some additional information for the social choice theorists amongst our esteemed readership: All three assignments are optimal in the utilitarian sense, for some (arbitrary but not unreasonable) way of scoring "yes" and "maybe" matches, and subject to certain constraints on the number of papers per person. The PC assignment is also optimal in an egalitarian sense (as it does not give anyone more than one paper they did not bid for). For SPC and AC assignments, it is less clear what a suitable egalitarian objective might be, but we generally tried to trade size against quality of assignment (so if you got too many papers you did not ask for, we at least tried to keep your assignment fairly small).

We use the "reviewer expertise" feature of EasyChair as a way for you to communicate to us that you are aware of your reviewing assignment, that you checked it for possible conflicts of interest, and that you will be able to complete your work on time (even if you may only have "low" expertise for some of the papers assigned to you).

We do this to avoid situations where a PC member only finds out on the day of the deadline that they in fact are unable to handle a given paper. In other words: if there are problems with the review assignment, as there undoubtedly will be, it's better to know early, so there is enough time to find other people to do the work instead.

Yes, within reason.

While we hope that you will write most of your reviews yourself, you also have the option of outsourcing some of them to trusted colleagues in your direct vicinity (so as to not accidentally violate any conflicts of interest). Importantly, if you choose to do this, you still remain responsible for the timely submission and quality of those reviews.  You also still need to contribute to the PC discussion or facilitate the participation of your colleague.

You might want to outsource a review if you happen to know someone with specific expertise relevant to a given paper. You might also want to do this in your role as the supervisor of an advanced PhD student who has published at ECAI or comparable venues before and who now needs to receive training in writing reviews (before they get to fulfil this important duty on their own, after graduation). Involving a PhD student in this manner is completely fine, as long you remain actively involved yourself and can vouch for the quality of the review submitted.

All such additional reviewers will get acknowledged in the proceedings.

We ask each regular PC member to deliver the reviews for at least 50% of the papers assigned to them one week ahead of the final reviewing deadline (e.g., if you were assigned 5 papers you should deliver 3 reviews by the first deadline). The reason for this deviation from what you might be used to is that we want to make the work of our SPC members a bit more manageable.

While the vast majority of ECAI reviewers manage to deliver high-quality reviews on time, there are always a few exceptions that require extra attention from SPC members and, in some cases, force them to solicit additional emergency reviews. The challenge is that, as an SPC member, you do not know in advance which papers will turn out to be the problematic ones that take up most of your time.

Given that typically around half of all reviews arrive on the day of the deadline—and many in the final hours—this core part of the work of an SPC member (and the emergency reviewers they approach) traditionally happens over the course of a single sleepless weekend just after the final review deadline. This clearly is not very healthy. By asking for half of the reviews to arrive one week earlier, we not only allow the SPC members to spread their work a little, but we also will find out one week earlier which reviewers might need some extra support.

First of all: welcome, great that you are getting involved! The fact that you have been included in the PC means we believe that you are qualified for the job. In our experience, newcomers often are particularly thorough and put a lot of effort into their reviews (that's great, try to keep that for as long as you can manage!). But they sometimes are also particularly harsh in their assessment (please try to resist that temptation!).

In a nutshell, we expect every review to seriously engage with the paper under consideration and to provide detailed feedback to its authors. This of course is easier if you are an expert on the specific topic of the paper yourself, but even if it is a bit further removed from your main area of interest you should be able to provide useful input. Naturally, the tone of every review must be professional and polite.

One of us has put together some general advice about reviewing a while ago, which you may find helpful.

As a discipline-wide conference, ECAI is serving many different subcommunities within AI. The perception of the conference may differ a little across these subcommunities.

Independently of your own subjective impressions of ECAI to date, for the purposes of your work as a PC member of this edition of ECAI, we ask you to treat the conference as a top conference. For instance, if you are more familiar with the standards expected at IJCAI and AAAI, please apply the same standards also here. Similarly, if there is a subfield-specific conference (such as NeurIPS, ACL, or KR) that you would consider a clear example of a top conference, please apply those same standards also to ECAI papers.

Importantly, none of this means that there is a need to fabricate an artificially low acceptance rate. We want to accept (and we have space for) all submissions that are of excellent quality.

No. Please try to judge each paper on its own merits, independently from the other papers you happen to be reviewing for this one edition of this one conference. Your batch of papers is unlikely to be representative: as an SPC member you will get to see around 0.5% of all submissions, while as a regular PC member you will get to see even fewer than that.

Please try to provide a holistic assessment of the paper under review that is appropriate in view of the specific research area it is contributing to. The exact set of criteria that are relevant to a given paper might vary a little from area to area, but the list below should give a good indication in most cases. Importantly though, there is no need to specifically organise your review by criteria (organisng the points in a review by importance usually is more helpful).

Relevance. This is about the relevance of the research question addressed and the methods used to the general theme of the conference (which is AI in the broadest sense).

Try to distinguish this aspect from whether you believe those methods have been used correctly (this is about "soundness") or whether the results obtained will have impact (this is something you might address when assessing "significance"). If several papers addressing similar questions and using similar methods have been published at ECAI (or IJCAI or AAAI) before, then the paper is relevant to the conference by definition. In all other cases, please try to be open-minded!

Clarity. This is about the clarity of exposition. Is it clear what the problem is the authors are addressing? Is it clear why they consider this a worthwhile direction of research? Is it clear how they have solved the problem? 

That is, are you able to understand the paper—with reasonable effort on your part and to an extent to which you feel that someone with your specific technical background should be able to understand a paper in this specific domain? We would expect that at least the problem statement ("what?") and its motivation ("why?") should be accessible to the majority of attendees of the conference. Of course, parts of the paper may only be fully accessible to an expert in its specific area of specialisation, and that's completely ok. But also those most technical parts of the paper should not be unnecessarily difficult or diffuse.

Significance. This criterion covers the scientific importance of the work under review and its likely impact. Importantly, "impact" is not restricted to impact on society at large by means of deployed applications, but also covers a paper's impact on the (possibly entirely theoretical) research done by others in the field. So when assessing the significance of a paper, you may consider questions such as whether it is likely to be widely read and whether the approach taken is likely to influence others in their own work.

While positive results are often those with the greatest practical impact, the community relies on negative results seeing the light of day as well, so please try not to be overly biased against such negative results when assessing their significance.

Originality. This is about the originality of the ideas found in the paper. So this covers the novelty of the research questions considered and the innovativeness of the approach taken. Note that it is perfectly possible to recognise the originality of an idea while doubting its significance or having concerns about the technical soundness of its implementation.

Soundness. This is, first and foremost, about the technical correctness of the results presented in the paper. But this criterion also covers matters such as the general quality of mathematical writing and the adequacy of experimental methods. 

If you are unable to certify correctness because the authors did not provide sufficient information for you to do so, then you should evaluate the paper as performing poorly in terms of this criterion (and also in terms of the reproducibility criterion). If you are unable to certify correctness because you lack some relevant expertise or because you simply do not have enough time to check all the details, then you should remain open-minded as to the soundness of the paper and give little (positive or negative) weight to this criterion when assessing the paper's overall quality (assuming, of course, you did not uncover any major flaws in the parts you were able to check).

Reproducibility. Research reported at ECAI should be reproducible, at least in principle. This means that the paper should provide sufficient detail to allow another team of researchers working in the same area to obtain the same results and thereby confirm their correctness. For experimental results, this means that sufficient information needs to be provided about, for instance, the data used and the parameters of the experimental design chosen. For a theoretical result, this means that sufficient information needs to be provided to allow readers to check the technical correctness of the arguments made.

Scholarship. This is about understanding and appropriate referencing of the state of the art. 

If you feel that important references are missing, please be specific and provide full bibliographic details in your review (names of all authors, paper title, conference/journal name, year of publication). As a rule of thumb, try to avoid suggesting that some of your own prior work should have been cited (it's difficult to be objective about such matters, and for clear-cut omissions probably one of the other reviewers will spot the problem, so you don't have to mention the issue).

Note that it is also possible for authors to include too many references. If you feel that some of the references really are not relevant, if you feel that some part of the literature (maybe by a particular group of researchers) is over-represented in the list of references, or if you feel that some inclusions in the list of references should have been motivated better, please do not hesitate to say so in your review.

Presentation. This is about the basic presentational quality of the paper and covers matters such as grammar, orthography, formatting, the usefulness of figures, and the correctness of the bibliographic information in the list of references. While authors should put significant efforts into polishing the presentation of their work, please be mindful that not everyone can be expected to deliver a manuscript composed in perfect English. 

We opted for a very simple review form, with just a single numerical score for the paper as a whole. We did not include numerical scores for individual review criteria for multiple reasons.

First, it would be difficult to give a precise semantics to the various scores. This is difficult enough for the overall evaluation (where, arguably, it is unavoidable to use some form of score), but harder still for individual criteria.

Second, having the option to express one's views in terms of a number often leads to people not also expressing that view verbally. But the latter is the much more useful kind of information: we would like you to comment, in words, on all the reviewing criteria that are relevant to the paper you are assessing.

Third, including numerical scores might tempt people to try and compute weighted averages of those scores. But it of course is not the case that the appropriate overall score for a paper can be computed as a weighted average of the individual scores (such as, say, 0.1 * relevance + 0.15 * significance + 0.25 * soundness + …). One mathematical paper with a severe technical mistake might score perfectly on all criteria other than soundness (and still must be rejected), while for another paper, that is more conceptual in nature, it might be hard to even specify what "soundness" should mean exactly (so that criterion certainly should not be decisive for that particular paper).

The purpose of the PC discussion for a given paper is to come to a decision on whether it should get accepted.

Please put all the papers assigned to you on your EasyChair watchlist (so you will receive email updates) and regularly check the discussion page for each paper.

The SPC member assigned, all PC members assigned (whether or not they wrote reviews themselves or delegated to subreviewers), and all subreviewers assigned (if any) should actively participate in the discussion. The area chair assigned should get involved where needed (e.g., when the SPC member does not seem to be active). It is customary that the SPC member starts off the discussion, but as they have to handle up to 12 papers, they might not manage to do this for all their papers on the first day. So it is also completely fine for someone else to start. 

PC members are ultimately responsible for the reviews produced by their subreviewers as well as for their subreviewers' contributions to the discussion.

The purpose of the PC discussion is not to align scores. In fact, please avoid focusing the discussion only on scores. Statements such as "I'm willing to increase my score" are not helpful. Instead, you should be talking about the content of the paper and your arguments for and against accepting it. 

The final decision also is not a simple function of the scores reported. If a review is not substantive, it can be perfectly fine to ignore it and to base the decision only on the other reviews.

Sometimes an individual reviewer will have an opinion that differs from the majority view, but they might still be able to accept that it is reasonable to implement the majority decision. In such a case, there is no need for the minority reviewer to align their score with the others. It is perfectly fine to accept a paper with an outlier "reject score", or to reject a paper with an outlier "accept score". It is helpful for the authors to see that there were diverging views on the quality of their work.

Once you have come to a consensus on what decision to recommend, the SPC member should write a metareview and communicate this consensus decision to us. If it there is no real consensus, the SPC member will still need to take a decision on what to recommend to us, and they should record the disagreement in the confidential comments of the metareview, so area chair and PC chairs can easily check and verify that the final decision is reasonable.

First, in case the authors submitted a rebuttal, it is simply the polite thing to do: to acknowledge that you have read the response of the authors and considered it. 

In most cases, you will agree with some of the response and disagree with other parts. Similarly, you probably will understand some of the response and find other parts unclear. Sometimes, certain remarks will have made you change your mind on some aspect of the paper, while for other aspects your opinion remains unchanged, even in the face of a seemingly good response. Finally, in some cases you may find a specific part of a response rather helpful but ultimately irrelevant to your overall assessment of the paper, while in other cases a response may really have changed your opinion in important ways. Please let the authors know about all of this! This kind of feedback can be very helpful.

Also, we often see interesting discussions happening amongst the reviewers. Please make sure that the essence of any such discussion is passed on to the authors, by including it in one of the reviews (or the metareview).

Note that you will find an additional field in the review form specifically for acknowledging that you have taken note of the rebuttal and providing feedback on it. In addition, feel free to also edit the rest of your review (e.g., to fix typos, correct misunderstandings resolved thanks to the author rebuttal or the feedback from other reviewers, and so forth).

Important: never change your score without also changing the text of your review.

Of course, this can happen, but you should realise that this can easily look unfair to the authors. This applies in particular to points that could have easily been discovered during the initial reviewing phase and that maybe could have been resolved by means of a simple clarification question that the authors could have answered during the rebuttal phase. So if a new point of criticism arises during the PC discussion, please be extra careful to handle the matter correctly.

Naturally, any new point of criticism emerging during the PC discussion should be included in one of the reviews (possibly the metareview). Please make it clear whether the new point you are adding ended up playing a crucial role in arriving at your final recommendation or whether you are simply providing additional information that you hope will be useful to the authors.

No, not always. It is perfectly possible for a group of PC members to agree about whether to recommend acceptance or rejection without all of you reporting the same score.

Indeed, if a paper ends up getting accepted but your own review is more critical than the others, then it is helpful for the authors to have clarity on this matter and see it reflected in your score. Similarly, if a paper ends up getting rejected but your own review is more positive than the others, then it is nice for the authors to see that too.

Of course, sometimes it will be appropriate to update your score. If you do, then you must also change the text of your review. An unmotivated change of score is never acceptable.

The metareview typically consists of a single paragraph of text. Its core purpose is to explain the decision taken by the programme committee to the authors.

On top of this, you may also use the metareview to communicate additional information to the authors, such as relevant points arising during the discussion that none of the reviewers included in their regular reviews, or some of your own feedback regarding the paper. (The latter—particularly if you have a lot of things to say—you can also do by submitting a regular review yourself.) The remainder of this answer only touches on the aforementioned core purpose of the metareview.

If the decision to be taken is obvious given the reviews, then a single sentence stating this fact is sufficient. For example, if two reviewers recommended to accept the paper and justified their recommendation with high-quality reviews, and the third reviewer did not manage to fully make up their mind but also did not put forward any strong arguments against acceptance, then the paper should get accepted.

But for around half of all submissions, the decision to be take will not be entirely obvious and some explanation will be called for. For example, if one reviewer was arguing for rejection and you, with the support of the other reviewers, decided to overrule this objection, you should say something about this in your metareview (recalling the main arguments in favour of the paper, then acknowledging the criticism but explaining why it was given less weight).

If you feel that one of the reviews did not fully live up to the standards you would have liked to see, it is ok to signal to the authors that the final decision was largely based on the other reviews. If you overlooked something that's really a little inappropriate in the initial reviews, it is ok to apologise for that oversight to the authors and, again, to assure them about the process that has led to the final decision. Of course, all of this needs to be done with due consideration for the reviewers concerned.

Never mention numerical scores in your metareview. Scores are just an imperfect and overly simplistic reflection of the content of a review. Instead, justify your decision in terms of the content of the reviews.

Note that there is no need to include phrases such as "I recommend to accept this paper" in your review. First, that sounds like a message directed at us (the PC chairs), not the authors, while it is the authors you should be addressing. Second, by the time the authors will get to your metareview, they already know what the acceptance decision is. 

For clear-cut or otherwise well-argued decisions, it is extremely unlikely that we would see any need to overrule your decision. But, of course, there will be borderline cases where you only very slightly prefer one decision over another, and we eventually will go for that other decision due to considerations of a more "global" nature (say, comparisons with other borderline papers that are not in your batch). In such a case, it is best if you phrase your metarview as if you knew that your suggested decision will get implemented, but if you also provide some extra text for us in the confidential comments that we can use in case we need to edit your metareview to justify a different final decision.

Just as the content of the anonymous submissions you are reviewing is confidential, so is the content of the PC discussion you are witnessing and contributing to. It goes without saying that it would be a clear betrayal of the trust put in you if you were to reveal to an outsider (such as an author) what a given reviewer said about a given paper.

We rely on everyone who accepts our invitation to join the PC to ensure they are able to dedicate the time required to their duties and meet all relevant deadlines. If you think you might not be able to do this, then we prefer it if you decline our invitation. But, of course, unforeseen problems can occur. In such a case, if you are a regular PC member, please inform the SPC members of all the papers for which you may miss a deadline as soon as possible. If you are an SPC member, please inform the ACs assigned to the papers you are handling. If things are really bad and you can only manage to write a single email, please make sure you send it to us (the PC chairs).

It is not always possible to arrive at a perfect paper assignment. But PC members are experienced representatives of our research community and should be able to provide useful feedback also on papers outside their own immediate area of research. Indeed, a carefully crafted outside assessment can be particularly useful and may help make a paper more accessible and more broadly relevant. 

But in extreme cases, if you let us (the PC chairs) know by sending us an email right after the problematic paper has been assigned to you, we will try to find somebody else to review the paper. The purpose of the "reviewer expertise" declaration on EasyChair is to have a clear point in time where every PC member actively checks that they will be able to handle the papers assigned to them. Problems discovered only a couple of days before the deadline are extremely difficult to resolve, so please make sure this does not happen.

If a paper is not anonymous, if it is too long, or if it clearly does not conform to the formatting guidelines, it should be rejected without review ("desk-rejected"). We will try to catch such cases before the review assignment, but we surely will miss a few.

If you notice such a violation of the conference policies in a paper assigned to you, please let us know (by email) right away. While it is always regrettable when a paper needs to get desk-rejected, if it does happen, it is better for everyone if it happens sooner rather than later: the authors will find out earlier (and can move on) and you and your fellow reviewers will have (a lot) less work. 

Should you happen to know that a paper has been submitted to another venue with overlapping reviewing periods, please also let us know. This would be a violation of the concurrent submission policy.

If you have other concerns of an ethical nature, please also get in touch.

The double-blind reviewing system is not perfect and you sometimes will be able to guess who wrote a given paper. That's ok. But please do not actively try to de-anonymise submissions (by googling for the paper titles and so forth).

Posting papers on arXiv close to or during the reviewing period is discouraged but not forbidden. So this is not a reason to desk-reject. Similarly, it is fine if authors have presented their work at a non-archival workshop in the past.

If you accidentally find out who wrote one of the papers you are reviewing/handling and this happens to be someone you should have declared a conflict of interest for, then please get in touch with us (the PC chairs) right away.

If one of the papers you are reviewing raises ethical concerns that you think might not be adequately addressed in the (optional) Ethics Statement at the end of the paper, please discuss this point with the SPC member and/or area chair responsible for that paper, and please do so well before the reviewing deadline. You might also want to include a question to the authors into your review, asking for clarification on a specific ethical matter.

If your discussion concludes that there is cause for serious concern, please bring the matter to our (the PC chairs') attention, so we can decide what to do.

Here is a (nonexhaustive) list of issues that might raise concerns of an ethical nature:

1. The research done has clear potential for unethical use, and these dangers are not adequately discussed in the paper. Examples include anything from "killer robots" to the intrusion of privacy.

2. The research process itself raises ethical concerns. An example would be the inadequate compensation of experimental subjects drawn from an underprivileged group.

3. The research uses data that has been collected (not necessarily by the authors of the paper under review) without first obtaining informed consent from the individuals concerned.

4. The paper (or the supplementary material) presents data that includes personal information that can be linked to specific individuals.

5. Either the research process or parts of the text exhibit bias or aggravate existing stereotypes regarding specific groups of people.

6. Parts of the text might cause (reasonably and unnecessarily) offence to some readers. For example, this might happen when describing data collected regarding a socially sensitive topic. In some cases it will still be appropriate to show the offensive data, but in those cases the paper should include an appropriate warning to readers.

Naturally, it is impossible to provide rules that cover every possible scenario and every possible cause for making an exception. Please use your own judgement. 

In general, it is most important that authors actively reflect on these issues. We don't want to shut down research in difficult areas, but we want to stimulate and incentivise people to think about possible dangers very carefully.

Finally, acts of scientific dishonesty, such as plagiarism or the falsification of data, are also of ethical concern and should be brought to the attention of SPC members and area chairs.

Yes. You are very welcome to, and many of your colleagues on the PC will submit a paper. This can easily be arranged in a way that does not give rise to conflicts of interest.

The only people not permitted to submit a paper are the editors of the proceedings, which in particular includes the PC chairs.

You should not be involved in reviewing someone's paper if you have a personal interest in that paper getting accepted. This might be the case if that person is a close friend, a departmental colleague, or a (current or past) adviser or advisee. This is what is known as a conflict of interest (CoI). It is also possible to have a CoI in the negative sense, where you have a personal interest in getting someone's paper rejected, e.g., if they are a direct competitor of yours.

It is important that you do your best to help us identify any such CoI, so we can avoid assigning you papers for which you have a conflict. You will be able to do this during paper bidding.

Of course, sometimes a CoI becomes apparent only after the review assignment has been made. In such a case, please alert us to the issue as soon as you become aware of it.

No. To be able to provide an impartial assessment, as a reviewer you are not supposed to be aware of the identity of the authors of the papers assigned to you.

All papers must have been properly anonymised by the authors. If you notice a paper that clearly violates the anonymity policy, please inform us right away.

Of course, the system is not perfect and it is important that everybody is aware of its imperfections. Reviewers may still, in some cases, guess whose paper they are reviewing, even if they are professional about it and try to resist temptation to do so. And, of course, double-blind reviewing is not a panacea circumventing all and every kind of bias a reviewer might harbour (e.g., against people who aren't native speakers of English). 

Yes. We will use double-blind but not triple-blind reviewing. That is, the reviewers cannot see the authors' names and the authors cannot see the reviewers' names, but the reviewers can see each others' names.

We think that it is important that reviewers are aware of one another's identity for several reasons.

First, in our experience this creates are more pleasant and fruitful atmosphere, particularly during the PC discussion.

Second, it allows individual members of our community, particularly younger members, to build a reputation as a good reviewer.

Third, it makes it easier to prevent unethical behaviour (such as reviewers pushing their own work when suggesting related work to authors).

We are aware of the fact that there also are downsides to our policy. Most importantly, some (younger) reviewers may feel shy about openly expressing their views on a paper when they see a more senior colleague expressing a different opinion. We ask senior colleagues to be mindful of this issue, and we ask everyone to engage in discussion in an open and respectful manner.

Many reviewers will have been in the situation where they felt that the authors of a paper they were reviewing really should have been citing one of the reviewer's own papers. This happens all the time. But it is important to acknowledge that it is very difficult to be truly objective about such matters.

It also is important to acknowledge that as a reviewer you have a lot of power. You must not abuse that power by forcing the authors to cite you.

We therefore strongly recommend that you avoid mentioning your own papers in your reviews. If indeed mentioning your paper is a must, then surely one of the other reviewers will bring up the issue anyway. So you don't have to.

If, despite this recommendation, you choose to mention your own work in your review, then you must (a) list that work with the full names of all authors (which anyway is something you always should do when mentioning a paper in a review) and you must (b) explicitly highlight that you have referred to your own work in your review by leaving a comment in the confidential comments, so the SPC member in charge can easily check that the matter is being handled correctly.

Using tools such as ChatGPT to polish the language of your reviews can be acceptable, but anything going beyond that would be a clear-cut violation of basic standards of scientific integrity. So you must not use such tools to automatically generate any part of your review.

Note that even using a generative AI tool to just polish the language of your review can raise ethical concerns. By entering a draft of your review into the system you are revealing confidential information about the unpublished work of the authors to the company offering the tool. So you are breaking the confidentiality agreement you have committed to by having agreed to review for the conference. This is not a serious concern if you use the tool to reformulate a couple of isolated sentences, but it can become an issue if you enter a full review. Please use your own judgment to decide. In any case, simply not using such tools is definitely the preferred route.

The information regarding submissions you get access to as a PC member is confidential. This includes the papers assigned to your for review, but also the (anonymised) list of all submissions you get to see during the bidding phase.

Specifically, you should never tell an author that you reviewed their paper, independently of whether the paper was accepted or rejected.

You also cannot make use of the information you have obtained as a PC member to advance your own research. Of course, once a paper becomes publicly available (e.g., if it has been accepted and gets published in the ECAI proceedings, or if it eventually appears elsewhere), you are free to make full use of the results in that paper.