Thoughts on the academic review process

I was invited by Prof. Yael Steinhardt to give a PhD seminar at Tel-Aviv University’s Coller School of Management on the academic review process and how scholars can effectively interact with a review team as they submit their research. I have been fortunate to be able to publish over 40 academic articles and at the same time be on the “other side” - serving as a reviewer and Associate Editor for multiple journals in my field (marketing). In preparation for the seminar I put together some insights that may be useful for the academics here.

Bottom line: The publication process is very long and can be frustrating (although it will significantly improve your research work). Being aware of the review team mindset and expectations can be useful. Also, it is very helpful to talk about the challenges and frustrations, learn from, and support each other.

Stage 1: Honeymoon

Academic scholars have the prevelidge to work on projects they are passionate about, that they believe will have an impact on the world, as well as the opportunity to collaborate with co-authors they like to work with. A research project often involves curiosity, creativity, a lot of learning and open-mindedness, and in spite of setbacks (e.g., the experiment did not work as planned) overall this is an exciting period for scholars. It ends up with a final outcome – a research article. Scholars often view their work in a very positive light; we are not objective, often fall in love with our work and sometimes refer to it as “our baby”.

Reviewers’ expectations: Reviewers (in top journals) will have expectations for your work on several dimensions:

(a) Wow effect/novelty.

(b) Meaningful contribution above and beyond prior work (write an explicit paragraph of contribution).

(c) Holistic approach: main effect + underlying process + moderator(s).

(d) Rich empirical package (online/lab, students/Mturk+Prolific, field data/field experiment/big data).

Selecting reviewers: Can one influence the selection of reviewers? Typically the Editor selects them as follows: one from the editorial review board, one based on your article’s references, one based on your request. So overall, it is hard – if not impossible – to influence this process.

Stage 2: Frustration

Submitting a paper to a journal is a great achievement. Further – submitting a paper to a top journal – hopefully your first target – suggests you have put a lot of efforts, developed a novel idea and put together a strong empirical package. But you need to be realistic about what’s coming. Acceptance rates at the top journals are lower than 10%. You are competing with excellent scholars from around the globe. Some has more resources, great training, and a better network. It is an extremely tough competition.

As you work on your project, you may have presented your work to colleagues at a research seminar, a PhD workshop, or a conference, and you may have even asked for a friendly review. These are excellent ways to get feedback and improve your work. But this pre-review feedback is typically not at the same depth and quality level you will receive from anonymous reviewers of a good academic journal. They will not prepare you to the reality check you will get from the review team. And given we “fall in love” with our research work, the reality check from the review team is hard. It is really hard.

Stage 3: Growth mindset and addressing the review

You should expect for your work to be heavily criticized (typically in a constructive way) and the paper – more likely than not – will be rejected, but you can use this opportunity to improve your work further. This will enable you to make sure your next attempt in submitting will result in a much stronger work.

After you receive the Decision Letter – whether it is more or less positive – take a few days to cool off before you dive into it. If this is a Reject letter (meaning you can’t re-submit your paper to the same journal), identify the key comments, and make a conscious decision which one of the comments you would need to address. Don’t take the paper as is and submit it to a different journal; journals and editors communicate with each other and sometimes the same reviewer can review for two different journals so it will look really bad if you got useful feedback and totally ignored it.

If this is a Revise & Resubmit (or a new trendy option: Reject & Resubmit) – copy the Decision Letter and turn it into Response Letter that will guide your revision. It will first serve as a revision plan and it will allow you to see that you are not missing any of the review team’s comments.

Any Revise & Resubmit is likely at first to be a Major Revision (or Major and Risky revision).

I find it very important and useful to develop a detailed response for each reviewer. (you can ignore the formal request for a limited number of response pages…).

Reviewers might ask a lot. If you really want to publish in the specific journal you need to address what they ask for and then do some more.

You should not ignore a reviewer’s comment but you do not have to do everything they ask. Nevertheless – explain/justify why you believe your approach makes sense. Additional citations or robustness checks will go a long way making your point.

At times different reviewers may pull in different directions. The most important person then becomes the Editor or Associate Editor (in case the review team has an AE). Follow their advice.

If you are not sure about your revision plan or specific significant or conflicting comments – you can email the Editor for advice. This is very much acceptable.

Some additional thoughts:

-Being a reviewer is important: it keeps you up to date on research in your field, it is your contribution and service to the discipline, it allows you to shape the field, and it can bring recognition and reputation.

-Start as an ad-hoc reviewer (typically in conferences) and focus on your research area; you will get many opportunities to review – pick only the best journals (for you).

-Be proactive: email the Editor of a journal important for your career and ask to become a reviewer, and later (once you were able to publish there and reviewed for a while) ask to join their Editorial Review Board.

-Don’t handle the frustration alone. Talk about it with your colleagues, mentors, advisors, co-authors. They are all in the same boat with you. There are even social media groups that share stories and offer support.

-Remind yourself: everyone is constantly being rejected. Even top-tier scholars from the best schools constantly get rejected.