Information for PC Members

Thank you very much for contributing your time to help make ECAI a great event! We will do our best to make your experience as PC member (be it as regular PC member, SPC member, or area chair) as pleasant and productive as possible, and we specifically want to support those of you who are serving in this important role for the first time. Below we have collected answers to some of the questions you might have after having read the Call for Papers. Please do not hesitate to let us know if there's more we can do to help.

Basic Logistics

Where can I find the EasyChair site for (the main track of) ECAI-2024?

I missed an important email. What should I do?

Most email you receive from us will be sent through EasyChair from ecai24@easychair.org. Please make sure that such email does not end up in your spam folder (or, if that is impossible, please regularly check your spam folder).

I'm a PC member, but EasyChair seems to not recognise me as such. What should I do?

The most likely explanation is that you have more than one EasyChair account, associated with different email addresses, or that the email address on which you received your PC invitation is not the address associated with your existing EasyChair account. See the EasyChair FAQ for more information.

Timeline and Duties

What are the important dates and main duties for regular PC members?

Please note that minor updates to the schedule are still possible. We will make sure to always have this list reflect the latest version. 

All deadlines are understood to be at the end of the day specified, anywhere on Earth (UTC-12). 

We aim at assigning each regular PC member no more than 4 papers to review. Whether this will be possible depends on the number of submissions and how well the community responds to our calls for joining the programme committee. (Update: We have almost met this target. The maximum number of papers per PC member is 5.)

In case you are a member of the Emergency Reviewer Pool, we aim at assigning you no more than 2 papers to review. You will receive papers to review when an emergency arises (likely in the first half of June).

What are the important dates and main duties for SPC members?

Please note that minor updates to the schedule are still possible. We will make sure to always have this list reflect the latest version. 

All deadlines are understood to be at the end of the day specified, anywhere on Earth (UTC-12). 

We aim at assigning each SPC member no more than 12 papers to handle. Whether this will be possible depends on the number of submissions and how well the community responds to our calls for joining the programme committee. (Update: We have met this target.)

What are the important dates and main duties for area chairs?

Please note that minor updates to the schedule are still possible. We will make sure to always have this list reflect the latest version. Importantly though, as an AC your main task is to help us when things go a little wrong, so we cannot provide a fully worked-out schedule in advance. 

All deadlines are understood to be at the end of the day specified, anywhere on Earth (UTC-12). 

We aim at assigning each AC no more than 50 papers to monitor. Whether this will be possible depends on the number of submissions and how well the community responds to our calls for joining the programme committee. (Update: We have met this target.)

I previously served as SPC member or AC but now got a regular PC member invitation. Is that a demotion?

No, of course not.

We find it important that also the most experienced members of our community serve in the role of regular PC member every now and then. This is both about fairness (younger people should have the chance to grow into the more senior roles) and quality (experienced researchers should keep writing reviews). 

It's also helpful to keep in mind that we need around 10 times as many regular PC members as SPC members, so everyone should expect that in most years their contribution to ECAI and similar conferences will take the form of a regular PC membership.

Volunteering

Can I volunteer to join the programme committee?

Yes. And thanks for asking!

To volunteer as a PC member please complete the for at https://bit.ly/join-ecai-pc (now closed).

Please note that every PC member must (1) be in possession of a PhD (by April 2024), (2) have expertise in the field of AI (as evidenced by having published at relevant conferences), and (3) have some reviewing experience (e.g., as an auxiliary reviewer).

I don't yet have a PhD. Can I still help somehow?

Yes, possibly. Please read on.

PC members should be experienced members of our community with relevant technical expertise. The level of experience and expertise required is something that most people attain around the time they complete their PhD, or possibly a little later than that. This is why holding a PhD (or an equivalent qualification) is a necessary (though obviously not a sufficient) condition for joining the PC. Of course, some individuals would in principle be ready to take on this role already somewhat earlier. But as PC chairs we simply have no good means of judging when an exception to the rules might be called for—so we won't make any exceptions.

We believe that anyone writing a review for a conference should first have been the recipient of a significant number of reviews for their own papers submitted to comparable venues. Furthermore, anyone joining the PC of a conference should first have written several one-off reviews for that or comparable conferences as an additional reviewer. That is, there is a crucial difference between getting asked to review a single paper (by someone who specifically thought of you as a good reviewer for that particular paper) and joining the PC (which involves reviewing a whole bunch of papers, not all of which will be a perfect fit to your own area of research).

So, if you are a PhD student and you want to get involved, then ask your supervisor or other more senior colleagues in your immediate environment who serves on the PC whether they might need your help with reviewing one of the papers assigned to them. They remain ultimately responsible for the review and they should give you some feedback on your work. This is an important part of getting trained as a researcher. All such additional reviewers will get acknowledged in the proceedings.

Reviewing

How do I select my topics? Why should I do this right away? How many should I pick?

How? To select your topics, on the EasyChair page for ECAI, go to the 'Conference' menu and select 'My topics'. You can also have a look at the list of topics right here.

Why? We ned you to select topics for two reasons:

How many? To be able to generate a reasonably accurate paper ordering for you during bidding, we require that you select at least 3 topics but believe that the quality of the ranking will be better in almost all cases if you select more topics than that. SPC members and area chairs definitely should select significantly more topics.

Making small changes to your topic selection every now and then is completely fine.

How do I enter my bid for the papers I would like to review?

Go to the EasyChair site, log in, select ECAI-2024, and then click on "Bidding" (in the top left corner). You will see the papers ordered by relevance, given the topics of expertise you have specified.

Pick "yes" if you would be happy to be assigned to the paper. Bid "maybe" if you are less excited about the paper but feel that you would be able to deal with it. Of course, others will also bid for some of these papers, so for us to have a chance of computing a good assignment, you need to bid for many more papers than you will receive in the end. Bidding for more papers means getting better papers, not getting more papers. Note that you can change your list of topics to see the paper in a different order.

Please also use this opportunity to indicate conflicts of interest for those (anonymous) papers where you happen to be aware of such a conflict.

I received a paper to review I did not bid for. Why is that? 

Computing a good review assignment is very difficult, and in most cases it is simply impossible to assign papers only to people who bid for them. Indeed, a small number of papers usually do not receive any bids at all, and these papers still need to get reviewed. Also keep in mind that any paper you bid for might also receive bids from many others. This is why we recommended that everyone should bid for significantly more papers than they will end up receiving.

Of course, getting a paper outside your core area of expertise need not always be bad. Sometimes a considerate review by a non-expert providing an outside perspective can be very helpful. If applicable, please try to provide such a review.

In any case, the quality of our review assignment is actually pretty good:

Finally, here is some additional information for the social choice theorists amongst our esteemed readership: All three assignments are optimal in the utilitarian sense, for some (arbitrary but not unreasonable) way of scoring "yes" and "maybe" matches, and subject to certain constraints on the number of papers per person. The PC assignment is also optimal in an egalitarian sense (as it does not give anyone more than one paper they did not bid for). For SPC and AC assignments, it is less clear what a suitable egalitarian objective might be, but we generally tried to trade size against quality of assignment (so if you got too many papers you did not ask for, we at least tried to keep your assignment fairly small).

Why do I need to indicate my "reviewer expertise" for each paper assigned to me?

We use the "reviewer expertise" feature of EasyChair as a way for you to communicate to us that you are aware of your reviewing assignment, that you checked it for possible conflicts of interest, and that you will be able to complete your work on time (even if you may only have "low" expertise for some of the papers assigned to you).

We do this to avoid situations where a PC member only finds out on the day of the deadline that they in fact are unable to handle a given paper. In other words: if there are problems with the review assignment, as there undoubtedly will be, it's better to know early, so there is enough time to find other people to do the work instead.

What should I do if a paper I'm reviewing violates one of the policies of the Call for Papers?

If a paper is not anonymous, if it is too long, or if it clearly does not conform to the formatting guidelines, it should be rejected without review ("desk-rejected"). We will try to catch such cases before the review assignment, but we surely will miss a few.

If you notice such a violation of the conference policies in a paper assigned to you, please let us know (by email) right away. While it is always regrettable when a paper needs to get desk-rejected, if it does happen, it is better for everyone if it happens sooner rather than later: the authors will find out earlier (and can move on) and you and your fellow reviewers will have (a lot) less work. 

Should you happen to know that a paper has been submitted to another venue with overlapping reviewing periods, please also let us know. This would be a violation of the concurrent submission policy.

If you have other concerns of an ethical nature, please also get in touch.

I think I know who wrote the paper I'm reviewing. I might have seen it on arXiv. What should I do?

The double-blind reviewing system is not perfect and you sometimes will be able to guess who wrote a given paper. That's ok. But please do not actively try to de-anonymise submissions (by googling for the paper titles and so forth).

Posting papers on arXiv close to or during the reviewing period is discouraged but not forbidden. So this is not a reason to desk-reject. Similarly, it is fine if authors have presented their work at a non-archival workshop in the past.

If you accidentally find out who wrote one of the papers you are reviewing/handling and this happens to be someone you should have declared a conflict of interest for, then please get in touch with us (the PC chairs) right away.

As a PC member, can I delegate a review to someone else (a "subreviewer")?

Yes, within reason.

While we hope that you will write most of your reviews yourself, you also have the option of outsourcing some of them to trusted colleagues in your direct vicinity (so as to not accidentally violate any conflicts of interest). Importantly, if you choose to do this, you still remain responsible for the timely submission and quality of those reviews.  You also still need to contribute to the PC discussion or facilitate the participation of your colleague.

You might want to outsource a review if you happen to know someone with specific expertise relevant to a given paper. You might also want to do this in your role as the supervisor of an advanced PhD student who has published at ECAI or comparable venues before and who now needs to receive training in writing reviews (before they get to fulfil this important duty on their own, after graduation). Involving a PhD student in this manner is completely fine, as long you remain actively involved yourself and can vouch for the quality of the review submitted.

All such additional reviewers will get acknowledged in the proceedings.

Why are there two different review deadlines?

We ask each regular PC member to deliver the reviews for at least 50% of the papers assigned to them one week ahead of the final reviewing deadline (e.g., if you were assigned 5 papers you should deliver 3 reviews by the first deadline). The reason for this deviation from what you might be used to is that we want to make the work of our SPC members a bit more manageable.

While the vast majority of ECAI reviewers manage to deliver high-quality reviews on time, there are always a few exceptions that require extra attention from SPC members and, in some cases, force them to solicit additional emergency reviews. The challenge is that, as an SPC member, you do not know in advance which papers will turn out to be the problematic ones that take up most of your time.

Given that typically around half of all reviews arrive on the day of the deadline—and many in the final hours—this core part of the work of an SPC member (and the emergency reviewers they approach) traditionally happens over the course of a single sleepless weekend just after the final review deadline. This clearly is not very healthy. By asking for half of the reviews to arrive one week earlier, we not only allow the SPC members to spread their work a little, but we also will find out one week earlier which reviewers might need some extra support.

I do not yet have a lot of experience with reviewing. How can I make sure I do a good job?

First of all: welcome, great that you are getting involved! The fact that you have been included in the PC means we believe that you are qualified for the job. In our experience, newcomers often are particularly thorough and put a lot of effort into their reviews (that's great, try to keep that for as long as you can manage!). But they sometimes are also particularly harsh in their assessment (please try to resist that temptation!).

In a nutshell, we expect every review to seriously engage with the paper under consideration and to provide detailed feedback to its authors. This of course is easier if you are an expert on the specific topic of the paper yourself, but even if it is a bit further removed from your main area of interest you should be able to provide useful input. Naturally, the tone of every review must be professional and polite.

One of us has put together some general advice about reviewing a while ago, which you may find helpful.

What are the paper acceptance standards at ECAI?

As a discipline-wide conference, ECAI is serving many different subcommunities within AI. The perception of the conference may differ a little across these subcommunities.

Independently of your own subjective impressions of ECAI to date, for the purposes of your work as a PC member of this edition of ECAI, we ask you to treat the conference as a top conference. For instance, if you are more familiar with the standards expected at IJCAI and AAAI, please apply the same standards also here. Similarly, if there is a subfield-specific conference (such as NeurIPS, ACL, or KR) that you would consider a clear example of a top conference, please apply those same standards also to ECAI papers.

Importantly, none of this means that there is a need to fabricate an artificially low acceptance rate. We want to accept (and we have space for) all submissions that are of excellent quality.

Am I supposed to recommend acceptance (rejection) for the best (worst) paper in my batch? 

No. Please try to judge each paper on its own merits, independently from the other papers you happen to be reviewing for this one edition of this one conference. Your batch of papers is unlikely to be representative: as an SPC member you will get to see around 0.5% of all submissions, while as a regular PC member you will get to see even fewer than that.

What are those reviewing criteria mentioned in the review form all about?

Please try to provide a holistic assessment of the paper under review that is appropriate in view of the specific research area it is contributing to. The exact set of criteria that are relevant to a given paper might vary a little from area to area, but the list below should give a good indication in most cases. Importantly though, there is no need to specifically organise your review by criteria (organisng the points in a review by importance usually is more helpful).

Relevance. This is about the relevance of the research question addressed and the methods used to the general theme of the conference (which is AI in the broadest sense).

Try to distinguish this aspect from whether you believe those methods have been used correctly (this is about "soundness") or whether the results obtained will have impact (this is something you might address when assessing "significance"). If several papers addressing similar questions and using similar methods have been published at ECAI (or IJCAI or AAAI) before, then the paper is relevant to the conference by definition. In all other cases, please try to be open-minded!

Clarity. This is about the clarity of exposition. Is it clear what the problem is the authors are addressing? Is it clear why they consider this a worthwhile direction of research? Is it clear how they have solved the problem? 

That is, are you able to understand the paper—with reasonable effort on your part and to an extent to which you feel that someone with your specific technical background should be able to understand a paper in this specific domain? We would expect that at least the problem statement ("what?") and its motivation ("why?") should be accessible to the majority of attendees of the conference. Of course, parts of the paper may only be fully accessible to an expert in its specific area of specialisation, and that's completely ok. But also those most technical parts of the paper should not be unnecessarily difficult or diffuse.

Significance. This criterion covers the scientific importance of the work under review and its likely impact. Importantly, "impact" is not restricted to impact on society at large by means of deployed applications, but also covers a paper's impact on the (possibly entirely theoretical) research done by others in the field. So when assessing the significance of a paper, you may consider questions such as whether it is likely to be widely read and whether the approach taken is likely to influence others in their own work.

While positive results are often those with the greatest practical impact, the community relies on negative results seeing the light of day as well, so please try not to be overly biased against such negative results when assessing their significance.

Originality. This is about the originality of the ideas found in the paper. So this covers the novelty of the research questions considered and the innovativeness of the approach taken. Note that it is perfectly possible to recognise the originality of an idea while doubting its significance or having concerns about the technical soundness of its implementation.

Soundness. This is, first and foremost, about the technical correctness of the results presented in the paper. But this criterion also covers matters such as the general quality of mathematical writing and the adequacy of experimental methods. 

If you are unable to certify correctness because the authors did not provide sufficient information for you to do so, then you should evaluate the paper as performing poorly in terms of this criterion (and also in terms of the reproducibility criterion). If you are unable to certify correctness because you lack some relevant expertise or because you simply do not have enough time to check all the details, then you should remain open-minded as to the soundness of the paper and give little (positive or negative) weight to this criterion when assessing the paper's overall quality (assuming, of course, you did not uncover any major flaws in the parts you were able to check).

Reproducibility. Research reported at ECAI should be reproducible, at least in principle. This means that the paper should provide sufficient detail to allow another team of researchers working in the same area to obtain the same results and thereby confirm their correctness. For experimental results, this means that sufficient information needs to be provided about, for instance, the data used and the parameters of the experimental design chosen. For a theoretical result, this means that sufficient information needs to be provided to allow readers to check the technical correctness of the arguments made.

Scholarship. This is about understanding and appropriate referencing of the state of the art. 

If you feel that important references are missing, please be specific and provide full bibliographic details in your review (names of all authors, paper title, conference/journal name, year of publication). As a rule of thumb, try to avoid suggesting that some of your own prior work should have been cited (it's difficult to be objective about such matters, and for clear-cut omissions probably one of the other reviewers will spot the problem, so you don't have to mention the issue).

Note that it is also possible for authors to include too many references. If you feel that some of the references really are not relevant, if you feel that some part of the literature (maybe by a particular group of researchers) is over-represented in the list of references, or if you feel that some inclusions in the list of references should have been motivated better, please do not hesitate to say so in your review.

Presentation. This is about the basic presentational quality of the paper and covers matters such as grammar, orthography, formatting, the usefulness of figures, and the correctness of the bibliographic information in the list of references. While authors should put significant efforts into polishing the presentation of their work, please be mindful that not everyone can be expected to deliver a manuscript composed in perfect English. 

Why are there no numerical scores for the individual review criteria in the review form?

We opted for a very simple review form, with just a single numerical score for the paper as a whole. We did not include numerical scores for individual review criteria for multiple reasons.

First, it would be difficult to give a precise semantics to the various scores. This is difficult enough for the overall evaluation (where, arguably, it is unavoidable to use some form of score), but harder still for individual criteria.

Second, having the option to express one's views in terms of a number often leads to people not also expressing that view verbally. But the latter is the much more useful kind of information: we would like you to comment, in words, on all the reviewing criteria that are relevant to the paper you are assessing.

Third, including numerical scores might tempt people to try and compute weighted averages of those scores. But it of course is not the case that the appropriate overall score for a paper can be computed as a weighted average of the individual scores (such as, say, 0.1 * relevance + 0.15 * significance + 0.25 * soundness + …). One mathematical paper with a severe technical mistake might score perfectly on all criteria other than soundness (and still must be rejected), while for another paper, that is more conceptual in nature, it might be hard to even specify what "soundness" should mean exactly (so that criterion certainly should not be decisive for that particular paper).

Ethics

As a PC member, SPC member, or AC, am I allowed to submit a paper?

Yes. You are very welcome to, and many of your colleagues on the PC will submit a paper. This can easily be arranged in a way that does not give rise to conflicts of interest.

The only people not permitted to submit a paper are the editors of the proceedings, which in particular includes the PC chairs.

What's a conflict of interest?

You should not be involved in reviewing someone's paper if you have a personal interest in that paper getting accepted. This might be the case if that person is a close friend, a departmental colleague, or a (current or past) adviser or advisee. This is what is known as a conflict of interest (CoI). It is also possible to have a CoI in the negative sense, where you have a personal interest in getting someone's paper rejected, e.g., if they are a direct competitor of yours.

It is important that you do your best to help us identify any such CoI, so we can avoid assigning you papers for which you have a conflict. You will be able to do this during paper bidding.

Of course, sometimes a CoI becomes apparent only after the review assignment has been made. In such a case, please alert us to the issue as soon as you become aware of it.

Will I be able to see the names of the authors of the papers assigned to me?

No. To be able to provide an impartial assessment, as a reviewer you are not supposed to be aware of the identity of the authors of the papers assigned to you.

All papers must have been properly anonymised by the authors. If you notice a paper that clearly violates the anonymity policy, please inform us right away.

Of course, the system is not perfect and it is important that everybody is aware of its imperfections. Reviewers may still, in some cases, guess whose paper they are reviewing, even if they are professional about it and try to resist temptation to do so. And, of course, double-blind reviewing is not a panacea circumventing all and every kind of bias a reviewer might harbour (e.g., against people who aren't native speakers of English). 

Will I be able to see the names of the other reviewers?

Yes. We will use double-blind but not triple-blind reviewing. That is, the reviewers cannot see the authors' names and the authors cannot see the reviewers' names, but the reviewers can see each others' names.

We think that it is important that reviewers are aware of one another's identity for several reasons.

First, in our experience this creates are more pleasant and fruitful atmosphere, particularly during the PC discussion.

Second, it allows individual members of our community, particularly younger members, to build a reputation as a good reviewer.

Third, it makes it easier to prevent unethical behaviour (such as reviewers pushing their own work when suggesting related work to authors).

We are aware of the fact that there also are downsides to our policy. Most importantly, some (younger) reviewers may feel shy about openly expressing their views on a paper when they see a more senior colleague expressing a different opinion. We ask senior colleagues to be mindful of this issue, and we ask everyone to engage in discussion in an open and respectful manner.

Why should I not mention my own work in my review?

Many reviewers will have been in the situation where they felt that the authors of a paper they were reviewing really should have been citing one of the reviewer's own papers. This happens all the time. But it is important to acknowledge that it is very difficult to be truly objective about such matters.

It also is important to acknowledge that as a reviewer you have a lot of power. You must not abuse that power by forcing the authors to cite you.

We therefore strongly recommend that you avoid mentioning your own papers in your reviews. If indeed mentioning your paper is a must, then surely one of the other reviewers will bring up the issue anyway. So you don't have to.

If, despite this recommendation, you choose to mention your own work in your review, then you must (a) list that work with the full names of all authors (which anyway is something you always should do when mentioning a paper in a review) and you must (b) explicitly highlight that you have referred to your own work in your review by leaving a comment in the confidential comments, so the SPC member in charge can easily check that the matter is being handled correctly.

Can I use ChatGPT to help me write my reviews?

Using tools such as ChatGPT to polish the language of your reviews can be acceptable, but anything going beyond that would be a clear-cut violation of basic standards of scientific integrity. So you must not use such tools to automatically generate any part of your review.

Note that even using a generative AI tool to just polish the language of your review can raise ethical concerns. By entering a draft of your review into the system you are revealing confidential information about the unpublished work of the authors to the company offering the tool. So you are breaking the confidentiality agreement you have committed to by having agreed to review for the conference. This is not a serious concern if you use the tool to reformulate a couple of isolated sentences, but it can become an issue if you enter a full review. Please use your own judgment to decide. In any case, simply not using such tools is definitely the preferred route.

I heard that the information on submissions I get to see is confidential. What does this mean?

The information regarding submissions you get access to as a PC member is confidential. This includes the papers assigned to your for review, but also the (anonymised) list of all submissions you get to see during the bidding phase.

Specifically, you should never tell an author that you reviewed their paper, independently of whether the paper was accepted or rejected.

You also cannot make use of the information you have obtained as a PC member to advance your own research. Of course, once a paper becomes publicly available (e.g., if it has been accepted and gets published in the ECAI proceedings, or if it eventually appears elsewhere), you are free to make full use of the results in that paper.