AEs 

Ex Ordo is not set up to handle multiple levels of review (i.e. (1) Reviewers 🡪 (2) AE 🡪 (3) Conference Co-Chairs), so the work around is that Ex Ordo set up “Review Groups”, corresponding to a number of submissions assigned to each AE/Chair. They made that process easy and handled assigning AEs and Working Paper chairs to submissions based on their Content/Method codes, so it was fine. Of course, some AEs/Chairs ended up handling submissions that were not in their wheelhouse, but that’s unavoidable with such a diverse set of submissions. The key is to ensure well-qualified reviewers (below). 

To get them into the system, all AEs and chairs were invited as ‘reviewers’ at which point they provided their content/methods codes. This is important because otherwise AEs/Chairs could be asked to handle their own submissions or submissions from scholars/students at their own school (if you opt for the “Prevent Nepotism” option in the system, that is. We did). The Ex Ordo system does a nice job ensuring no conflicts. Once the AE/Chairs are registered, send their names to Ex Ordo and they create the Review Groups. Once that is done, the AEs and Chairs are removed as Reviewers. Here is the email for that (sent in March, after we’d invited reviewers): 

Hello ACR AE’s: 

Hope you are well! Please, we need you to register with Ex Ordo, our ACR conference software. Please accept the invitation you’ll receive from Ex Ordo (hopefully today) and register.  

During that process, it’ll ask you to indicate your Research and Methods codes – please put in as many codes as you are comfortable with. This is how we will be allocating submissions and reviewers to you. How many codes you enter has nothing to do with the number of reviews/submissions you’ll be responsible for. It’s just used to improve fit. 

There is only one tricky detail: When registering, please ONLY USE the email address you receive the invitation at. You can check this by clicking on the “TO:…” line in the invitation. During the registration process, if you want to change this email to another one, you can (once in the system).  

Other than registering, we don’t need you do anything right now. Questions etc…? Send my way! We will be arranging training down the road (though it is a very intuitive system)… 

AEs handled Competitive Papers. Their jobs were to keep reviewers on track, assure high quality reviews, provide feedback to authors and make recommendations to conference co-chairs about what to accept/reject. We also instructed them not to return any unprofessional comments made by reviewers to authors and to flag those reviewers to us. We have all received snarky and petty reviews and we were not going to have that at our conference. We also played around with the idea of allowing AEs make final decisions, but in the end we made all acceptance/rejection decisions ourselves. Most AEs were amazing but there were a few we had challenges with. For example, it appeared they were not reading reviewer comments or incorporating that information into their recommendations to us. We had to recheck their work and we ended up disagreeing with their recommendations a lot. There were also many submissions that were genuinely challenging and we probably made better decisions after all three co-chairs read the reviews, AE feedback and papers and then made decisions. Ultimately every aspect of the conference was our responsibility, so we opted to keep final decision making to ourselves. For really good or bad submissions, deciding what to accept/reject was easy and we nearly always agreed with the AEs. The ones with mixed reviews were the difficult and time-consuming decisions. 

Notes:

• On the day we were doing our AE training (Ex Ordo does a training session for AEs/Chairs), a couple AEs bowed out. We had to scramble to replace them. One replacement came from our personal network – well qualified and able to step up to handle submissions in a particular area – and one came from our Program Committee, someone we had thought about asking to be AE Point is, these things will happen.  

• AE’s and Working Paper chairs asked about what would be an ideal accept/reject ratio. We told them to focus on quality. If they accepted 100% or 0%, it didn’t matter as long as the ultimate decision was based on quality. That said, if we were really on the fence with a submission, I think we usually gave it the benefit of the doubt. 

• We did not require our AEs to provide comments to us regarding their recommendations. Many did, but in hindsight it’s better to tell AEs comments are needed, maybe not for all submissions but at least for the ambiguous ones. We rarely overruled AEs on submissions that uniformly scored high or low (depending on what numbers you pay attention to and how you weight them). However, the submissions with very mixed reviews or that were in the middle, we really valued the AEs comments explaining their decision to us. Absent such comments, it was much more work. 

• All AEs, Chairs (and reviewers) should be asked not to make determinative statements about ‘this paper will be accepted’ etc… in the feedback to authors. Final dispositions are made by the conference co-chairs and these instructions are designed to avoid the awkwardness of over-ruling AEs etc…

• We asked AEs to send personal notes of thanks to reviewers who had done an excellent job. The response to that effort was quite positive.