Post date: Dec 30, 2008 6:36:22 PM
John Langford had an interesting post (as usual) on his Hunch.net blog about Adversarial Research. In the comment to his post, someone anonymously posted the following:
I believe you have the ethical duty *reversed*. The point of reviewing is to identify the papers most likely to carry our field forward. It is our duty as reviewers, as PCs, and as editors to accept the work we view as the highest value, and to discourage work we feel is ‘philosophically incorrect’. If you don’t do this, why bother reviewing? This role is even more important in today’s environment where any work can be made easily available on the web– the only roles for reviewers is to identify the best material and to improve and correct the submissions they read by their comments. Identifying the best fundamentally involves our own philosophical viewpoints– we are trying to remove the nonsense.
This has inspired me to write down some of my thoughts on the subject. I welcome your comments, unfortunately in the form of e-mail because Google Sites doesn't allow anonymous comments.
What is the role of a reviewer? I think the answer differs from the viewpoint of the author or the program committee.
Here are a few angles I've heard on the topic from the perspective of the authors.
(Most of the commentary is my personal, and humble viewpoint as of this posting)
The "test" approach
Reviewers are there to test whether my work is good enough to be published.
If your work passes the test of peer review, then it's good enough to publish. This is weird for many reasons, not the least of which is that these same people acknowledge that reviews are very stochastic. This mindset is dangerous because it can lead to advocacy of a shotgun approach to paper submissions, which overwhelms the system, and disenfranchises reviewers by diluting the quality of submissions.
The "constructive" approach
Reviewers are excellent resources to help me develop my ideas and improve my work.
Submit a partially developed idea with partially convincing results and minimal citations from the literature. If it's good enough, it'll get in to the conference. If not, the reviewers will make valuable suggestions on how to improve it, and also will point out useful related work from the literature. The reviewers might also check if the proof is right and point out typos.
The "check and balance" approach
Reviewers are necessary red-tape to protect the community from other authors that don't take submission as seriously as I do.
In a perfect world, I think this would be the ideal role of reviewers from the perspective of the authors. The author should submit work that they champion and will stand behind. The author must believe that their work should be published, that their results are reproducible and sufficiently general. Inaccuracies or other issues found by the reviewers will be a surprise to this author, and a lesson that the author didn't quite do his/her due diligence: something to correct for next time. This is an unrealistic ideal, but I think it's something to strive toward.
Implications
Both "test" and "constructive" authors think that the reviewer owes them something, that their responsibility (at last partly) is to help the author.
The "check and balance" author believes that the reviewer owes him nothing, except a cogent argument in the case of rejection. This author may appreciate feedback and suggestions, but considers them to be strictly a bonus.
The anonymous post on John Langford's blog was from the perspective of the reviewer, not the author. That post advocated political reviewing based on selecting the best papers that are most likely to move the community forward. This won't necessarily bother "test" or "constructive" authors, but will seriously offend "check and balance" authors.
The "check and balance" author believes that the reviewer should be rejecting papers that have not proved their point or made their case. These authors would probably agree with the Wikipedia definition of peer review (because I certainly do):
Excerpt of definition of Peer Review, from Wikipedia (emphasis mine):
Peer review (also known as refereeing) is the process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field. Peer review requires a community of experts in a given (and often narrowly defined) field, who are qualified and able to perform impartial review.
By allowing and even advocating reviewers to "identify the papers most likely to carry our field forward", being impartial is thrown out the window. It does explain the following other statement from wikipedia.
Excerpt of definition of Peer Review, from Wikipedia (emphasis mine):
Impartial review, especially of work in less narrowly defined or inter-disciplinary fields, may be difficult to accomplish; and the significance (good or bad) of an idea may never be widely appreciated among its contemporaries.
I don't really have anything universal to say in this post, I just wanted to tie down a few thoughts and arguments that I've had with others in the last few years.
It's my sincere opinion that authors should strive for "check and balance" expectations, which probably implies submitting fewer papers. If reviewers try to be impartial and also helpful, perhaps the whole system can work a little bit better, and we can even stop the numbers game. (That's a plug for a very interesting paper about how counting publications slows scientific progress overall... if you don't have an ACM account the full text is posted here).
Disclaimer: I do research in Canada. I'm told that we tend to be more collaborative and less competitive because our funding model supports that more than certain other countries.