Reviewer Instructions

Dear AIIDE program committee member,

This year, we’re experimenting with a different review form. We’re asking you to organize your evaluation in terms of:

Strengths

  • The things that make you want to see the paper published

Fixable weaknesses

  • Problems that could be easily addressed by the authors for the final version

Unfixable weaknesses

  • Problems that couldn’t be fixed in time for the final version
  • Problems that make it impossible to evaluate the work in the first place

Your job as a reviewer is to select the papers you believe will have the greatest impact on your work and the work of others. Of course, as many of you are aware, peer review often selects not for the most interesting papers, but rather those with which one can find the least fault. The new review form is an attempt to make it easier for you to support the papers you think matter the most.

We ask you to begin by writing text evaluating the strengths, fixable weaknesses, and unfixable weaknesses of the paper, and only then to give each one a score. The scoring system is laid out in the review form, but I would call your attention to the scoring system for the strengths section:

1: No strengths whatsoever

2-4: Contribution is insufficient to warrant inclusion in the conference

5-7: Contribution is either limited or of interest to a limited community

8-9: I want to talk to the author(s) at the coffee break

10: Everyone will want to talk to the author(s) about it at the coffee break

Your goal is to maximize coffee break conversation subject to the constraint that we not accept papers with truly serious flaws. I would personally rather publish a paper that made you think differently about a problem you cared about, even if it had a few flaws, than one that unassailably proved something you don’t care about and could have predicted without reading the paper.

Thank you again for agreeing to serve,

Ian Horswill

Program Chair

AIIDE 2013 Review form

Please use this form when writing your review and paste text portions into ConfMaster

Criteria Scoring (1-10 rating buttons)

Please rate the strengths of the submission, irrespective of any weaknesses (Strengths):

· 1=none

· 2-4=the submission makes a contribution but is insufficient to warrant inclusion in the conference

· 5-7=the submission makes a contribution, although that contribution is either limited or of interest to a limited community

· 8-9=very interesting submission; I would want to talk to the author(s) about it at the coffee break

· 10=everyone will want to talk to the author(s) about it at the coffee break

Please rate the seriousness of its fixable weaknesses (Fixable):

· 1=none

· 2-4=minor issues

· 5-7=would require significant work to put in condition acceptable for inclusion in the conference

· 8-10=theoretically fixable but contains too many issues to accept

Please rate the seriousness of its unfixable weaknesses (Unfixable):

· 1=none

· 2-5=there are unfixable problems, but they aren’t fatal by themselves

· 6-10=there are serious unfixable problems that make the submission inappropriate for inclusion in the conference

Please rate your confidence in this review (Confidence):

· 1=the submission is completely outside my expertise

· 2-4= the submission is outside my area and/or was very difficult to understand, but I can make an educated guess

· 5-7=there were things I didn’t know or didn’t check, but I’m reasonably confident

· 8-9=to the best of my knowledge, I understood and checked everything and know the relevant literature

· 10=Absolutely confident

Comments to Authors

Narrative evaluation:

Brief description or summary of the submission (1-4 sentences)

What can researchers and/or practitioners learn from this submission? Who would be most interested in it?

What are its primary strengths?

What fixable weaknesses does it have, if any? These are problems that could still reasonably be corrected for the final version.

What unfixable weaknesses does it have, if any? These are problems that are intrinsic to the work, that could not be corrected in time for the final version, or that make it difficult even to review the submission.

Please provide any other comments you have for the authors, especially constructive suggestions as to how it could be improved.