Note: In companion with this, I have a blog listing wrong published mathematical results (Link).
Here are some of my personal thoughts on a better publication model, for both mathematics and science. This reflects also some discussions I have with friends. These thoughts will be ever changed and improved. If you feel interested in further discussion about this, please contact me. These may also be useful for other referee systems, such as in grant applications, awards competitions or job applications.
In the real life, in particular in democratic countries, there are rules and regulations people follow, to protect minority and disadvantage people, customers, and so on, and most people are kind and helpful. But it seems that in publishing in science (and many other aspects in science which I do not discuss here), not many rules are for authors, and bad behaviours and misconducts are abundant. Why is this possible? Can we do something to improve?
Ultimately, I think that there should be a better way to evaluate research, other than publications or peer reviews. While waiting for that to come, here is what I have to offer to improving the current system.
Summary: Here is a summary of the below essay: I advocate for an open, fair and democratic model in science publishing. By this I mean:
- Scientific journals, besides common regulations for all journals, should be subject to additional special regulations (from the governments). This is because they are, in most of the cases, the sole indication committees use when evaluate one researcher.
- Authors should be able to get support from universities and national research funds for open access publishing.
- The authors are automatically given a chance to respond to referees' criticism. Just as in life, even if someone is a criminal, he/she still has the right to self defend.
- The authors are automatically given the chance to receive full referees' reports.
- The authors have the right to know the identities of referees, if not right away, then after a certain amount of time, say 5 years. (Thinking carefully, I see that 5 years are too long. 1 year or 1 and a half year is a reasonable time.) This is similar to the fact that democratic countries allow people to get access of secret government documents after a certain amount of time.
In case of wrong published papers in the journal, the public (not just the author) has the right to know the identities of referees after a certain amount of time.
This rule can also be applied to editors.
- There should be a policy to prevent a particular line of thought to become monopoliser, like in the real life what governments do with companies. (There have been, even in the current age, many lines of thoughts which were declared to be very promising and would lead to enormous breakthroughs or benefits, which at the end or currently could/cannot deliver those promises even after some decades. In the process, however, researchers in those particular lines of thoughts have been favoured in terms of jobs, publications and promotions, many cases unfairly.)
- The papers submitted should be only evaluated on the contents of the results it obtains only, not by what methods it derived at the results. This is to follow the wisdom of Galilei "All truths are easy to understand once they are discovered. The point is to discover them".
- There should be specific, measurable methods to compare the paper with the level of the journal. Not just a vague comment that it is not up to the level, since it is well-known that even in top journals many published papers are wrong and cannot be repaired by the methods in the published papers. I propose to compare a paper with an average published paper in the journal before.
- Other vague comments without supported evidence, together with personal criticism and undue comparison, should not be allowed from referees' reports.
- There should be again an environment of public debate in science, as we used to have before, in particular in Europe.
- To avoid conflict of interest, the authors of previously published papers, whose results are found to be wrong, should not be the referees of new papers having correct proofs of those results. At best, these authors of wrong papers can only serve as consultants and are informed that their papers are wrong. There should be a formal procedure to deal with this. (Think about the analogy in real life, and you will see how strange this is. A person B is accused by a person C, and then the police invites the person B to judge whether the accusation by the person C is correct. )
- There should be strict deadlines for referees to submit reports. If they feel that they do not have time, then from the beginning they should refuse to do the work.
- Wrong published papers should be acknowledged in the journal.
- Being an editor/referee is a privilege, so those people should have strict ethics. Editors/referees with good behaviour should be acknowledged, and those with bad behaviour should be banned from serving for the journal.
- Writing a referee's report should follow the same rules as writing a research paper: the object of researching here is the submitted paper. Arguments should be followed by evidence. What is the point of needing a referee if he/she writes whatever imaginary things he/she can have?
These may be also helpful for other referee systems, such as in grant applications, awards competitions and job applications. These should be as open as possible, ideally like what we see in America's got talents.
Main essay:
There have been campaigns about having better publication models, mostly concerned about whether the price is high or if the journal is free to the public and/or the authors. I think it is fine if authors are required not too much an amount for accepted papers (usually they can get some funding for that purpose) so that the public can have free access.
Note: An argument usually used to require public access to journal papers is that the public should get free knowledge. However, this is not, if we scrutinise very closely, a real problem at all in this age. Usually, authors will upload their papers to personal webpages or online archives. So the public can get free knowledge easily. Even in case there is no free online version, the interested public can just simply email the author, and I think that in most of the case the authors will be happy to send a copy to that interested public. So there is or not public access to journals is no problem at all, concerning knowledge.
Note: I received a comment that research fundings are difficult nowadays for researchers, in particular in pure science to get. Here is my answer: This is true, as I witness myself. However, that is about regular research funds, which are quite a big amount. For fundings for publication in open option, the amount is much much smaller (I estimate on average about 1000 USD for a paper). Hence, it is much easier to get. For example, there is a movement among European universities towards open access publishing, and authors can apply for publication funding from their universities. I think national research funds will also be happy to give funds for this, because on average a researcher can have maximum 2-3 published papers a year, just a tiny money compared with regular research funds.
Note: I received a comment that the average number of published papers a year depends on the field, the above number of 2-3 is for mathematics, but there are fields where people can publish 10 papers/year on average. I have two options for this. In fields where people publish many papers a year, maybe the price for each paper is less? One other option is that in such fields, I think people will easily get more funding than in mathematics, and hence they have more to pay for publishing costs.
From the publisher viewpoint, it is also good, because they have to pay salaries to at least some staffs. There may be also other costs and taxes. Who could do volunteer work without getting bread and butter?
Note: You may ask: "Why should journal publishers get money"? Well, their job is to maintain or improve the quality (more precisely, the reputation) of their journals, so authors publishing in the journal get special treatments and favours when it comes to promotion, grants and so on. It is true nowadays that everyone can become a publisher on themselves, but to get to the level where promotion or grant committees recommend (usually not publicly, but in practice) as a good indication is not that easy.
(While it is usually claimed that editors, referees and authors do all work for free, on a closer inspection it is not so. Editors and referees can use their contribution to support their promotion and grant applications, and authors can use their published papers for the same purpose. The benefit they get from promotions and grants is much bigger than if the journals would pay them salaries. Moreover, editors and referees can decide who gets published in their journals, which is very important for all subfields and researchers, in particular new fields and junior researchers, and if they are unfair about this, it will be a disaster.) Of course, it is better if authors are also not required to pay any money, but as the previous sentence shows, this is not very practical. Some new journals have been advocated to this. However, a disadvantage of these new journals is that the old journals have better reputations, and hence these journals are not attractive to a junior researcher from a view to their research career. One way to get around this, which is not very practical at least in a near future, is that all top researchers from all all top institutions will from now on only publish their papers in free journals, and will serve as editors or referees of only such journals. Even if this is achieved, I think a much more fundamental problem, to be discussed below, is the referee process. Having free journals, with all top researchers always publish in them, but not improving the referee process, is just a tiny resolution to the problem of improving publishing quality.
There have been a growing outcry of authors, not only in mathematics but all over fields of science, about unfairness in publishing in journals. This goes all the way to the referee system. Note that a referee system was not always needed (Einstein was offended when knowing that one of his papers was refereed). The referee system was a good force, for example in Einstein case one error was spotted. However, it more and more becomes skewed and unfair, in that it likes to preserve and promote established fields and ideas, while more to reserve about new fields and ideas. (It happens that if an author works in a hot topic or has good connections, then a paper he/she writes of an average quality easily gets published in top or very good journals, while otherwise it will be extremely difficult. This is in relation to what is called "paradigm" in Kuhn's book, see a relevant citation from his book below. This can be explained by for example if that a field is well established, it is often the case that researchers in that field can hold important positions in research institutes, journals and grant committees, and there is a bigger pool of referees to choose, and many more of these referees will be friendly to that author. Think, if your close friend is referee of your paper, would he be more to reject or accept your paper?) This is not the way we see in the business world, where if your new ideas are useful then you can get money, no matter who you are and what you do.
Note: I received a comment that maybe editors and referees can get some small payments from journals. This may be helpful in that now people must have more responsibility to the referee work. On the other hand, even without pay, I think referee work is one responsibility of researchers who are working at research institutes. Their salary from the institutes consist of one part to develop science, and referee work is part of that. If they don't do that, who will do? We cannot ask the general public to do so, since they are not paid for the work and they do not directly get benefit from having papers published in journals.
I agree with many opinions that the problem with the current referee system is that: 1) it is not transparent (you do not know who write on you, is that a top professor in the field who is competent to do so, or just a graduate student who really know only vaguely or nothing at all about what your paper is about); 2) it gives the referees kind of an absolute power (they can write some very vague comments, such as the ideas used in the paper are not new, and so on, or on the other extreme give flying words for a not-so-good work); and 3) it does not give authors a chance to respond to referees' comments.
Having identified the problems, I propose the following steps which can be done to make a more transparent and better referee system:
- Scientific journals, besides the usual regulations for all journals, should be subject to additional special regulations. This is because they have a very special role: In most of the cases, they are the sole indication the committees use to evaluate a researcher. (Another indication is patents, but we do not expect all researchers to have patents.) While it is not acknowledged publicly, in practice all committees (whether at institutions, at national or international agencies) use journal publications for judgement. This is understandable from the practical viewpoint: If a reviewer is not familiar to the research of an applicant, then the safest instrument he/she uses to argue is through the journals. (To this point, I am very much disappointed. If a reviewer is not familiar with the research and is not willing to spend time to go deep into the application, then he/she should refuse to do the job. I believe that committees can always find a couple of experts out there in the world who understands enough the application to provide a precise evaluation. But this is going to far into another subject, so I prefer not discuss more on this here.) Governments should have some guidelines in the laws about the activities of scientific journals and how the public agencies use the information from these journals when forming their decisions.
Note: If we look at usual journals, such as New York Times, we can see clearly the differences between popular journals and scientific journals. The popular journals have the aim to give news to the public, and they also have the aim to influence the public's opinion, but officially they cannot influence the public agencies' opinion. Imagine if there is any agency which, based solely on an article on New York Times and all other popular journals praising a person A, to appoint the person A to an important position? However, if we replace "all popular journals" by "all scientific journals" (or "all scientific journals in a specialised field"), then this is a real scenario.
- For arguments about the methods used are not new: To this, my favourite response is Galilei's quote "All truths are easy to understand once they are discovered. The point is to discover them". One other is what I heard from my PhD adviser "All mathematics comes from Euclid". I think a referee should think in the following lines: "If I, or my students, were the author of this paper, should I submit it to this journal"? I think the main point in a discovery is the content of the discovery, not the methods. To take the analogy to the business world, if you can make some new apps, no matter how you did it, then you get money. Or if you like philosophy, here is a famous quote from the Buddha: "I am a finger pointing to the moon. Don't look at me; look at the moon."
I think the fact that you can prove/discover new results mean that you have new ideas/methods. A new point of view (even of very old objects/methods) should also be counted as a new idea/method. It happens quite often in mathematics and science that a new point of view is all that we need. Einstein's relativity theories are a favourite example (All the theory, that is the mathematics, was known before. His success was in his new point of view about light and gravity.). So from historical lessons, subduing other lines of thinking in favour of a special line of thinking, however seemingly promising and supported that line of thinking, will not do good to science and to the world. Taking real life example, in many countries there are laws to restrict a company to become monopoliser in an area. Should that practice be applied to science? (There have been, even in the current age, many lines of thoughts which were declared to be very promising and would lead to enormous breakthroughs or benefits, which at the end or currently could/cannot deliver those promises even after some decades. In the process, however, researchers in those particular lines of thoughts have been favoured in terms of jobs, publications and promotions, many cases unfairly.)
Actually, there were cases where papers of not-new-methods of not-new results are published in top journals. My favourite example is that of Selberg's paper on an elementary proof of the prime theorem. (Of course, here I just mean that the argument that something is not new to reject a paper is unfair, and do not mean anything about the quality.)
Note: One comment suggested the book "The structure of scientific Revolutions" by Thomas S. Kuhn link. This is very relevant to the point here that there are often oppositions between people of old practices and those of new emerging practices. It is also relevant to the whole essay here. Highly recommended! Here is a quote from the book (on page 36):
"Mop-ping-up operations are what engage most scientists through their careers. They constitute what I am here calling normal science. Closely examined, whether historically or in the contemporary laboratory, that enterprise seems an attempt to force nature into the preformed and relatively inflexible box that the paradigm supplies. No part of the aim of normal science is to call forth new sorts of phenomena; indeed those that will not fit the box are often not seen at all. Nor do scientists normally aim to invent new theories, and they are often intolerant of those invented by others. Instead, normal-scientific research is directed to the articulation of those phenomena and theories that the paradigm already supplies."
- Arguments such as: "This paper is not up to the level of this journal", without specific supporting evidence, should not appear in a referee's report. It is well-known that even in top journals, a not small percent of published papers is not correct and cannot be revised by the methods in these published papers. So talking about level is quite vague and pointless, if no explicit arguments are given for the paper in consideration. Authors have the right to ask: "Could you please give me some specific criteria against it to measure the level of your journals". I think a good indication is to see what question the paper answers, what is the importance of that question in the subfield and in the bigger field, how much advance is the paper compared with other papers working in the same direction, where these papers are published, and how the paper is compared with an average paper in the journal. This is at least practical and sound: if a person sees that several recent papers of the same quality as or less than his paper was published in the journal before, he should be correct to think that his paper is also of the level of the journal.
Relating to the point of vague reports, there are cases where papers are rejected because of not very being well-written. (Of course, being well-written is important, but mathematics and science are not literature, so asking for a perfect writing is a wrong attitude.)
Writing a referee's report should follow the same rules as writing a research paper: the object of researching here is the submitted paper. Arguments should be followed by evidence. What is the point of needing a referee if he/she writes whatever imaginary things he/she can have?
- Authors should be automatically given a chance to respond to referees' criticism, as we see in life. Even a criminal has the right to self defend. Why are authors declined of this basic civil right? Blindly trusting referees will not do good to science, as Einstein said "A foolish faith in authority is the worst enemy of truth". Some editors told me that it's up to the editors whether to give an author a chance to reply, but this practice is bias and unfair. Many journals even have the practice of not accepting any response from the author.
I propose the following formal procedure that the journal can implement:
The editor in charge will send the criticism of the referees to the author. The author is then given a deadline to answer these questions. When the answers come, the editor in charge will study it together with the referees. This process may be repeated until all parties agree. Then the whole process is documented in an official form from the journal, with the signature of the editor in chief.
Note: I received a comment that in case a referee is obviously no-reasonable obstructing the progress, with unsupported comments/claims, then the editor should or the author has the right to ask for a new referee.
- Authors should automatically have the right to obtain full referees' reports. There are cases where authors received only parts of referees' reports. A common argument for this is that some parts of the reports are impolite, and hence the editors must do some rephrasing. That may be true, but a full report will be useful for the author to know the whole report to know the full evaluation of the referees. To the problem of impoliteness in the reports, there are some solutions. Either the journal can put in their policy some guidelines about how the referees write the reports, or if the editors feel some impolite points are present they can point to the referees and ask them whether they like to send a revised version before sending to the authors.
- Referees, when they are willing to review a paper, should also be willing to take responsibility on comments they write. They should be able to, when asked by people why giving those reports, give concrete reasons why. Not just because they are anonymous that they have the right to write whatever absurd things they like.
It is the best if the referee process is open, that is authors know who referees are. If this is not possible, then maybe a policy of allowing authors to know the identities of referees after a certain amount of time, say 3 years. 3 years are not a small amount in a researcher's life. (Thinking carefully, I see that 3 years are too long. 1 year or 1 and a half year is a reasonable time.) This is similar to the practice of democratic countries, such as USA, allowing the public to get access of secret government documents after a certain amount of time.
Also, in case of papers with wrong results/methods published in a journal, the public (and not only the authors) should have the right to know the identities of the referees after a certain amount of time. My argument for this is that having a wrong paper published takes the other researchers in the public off a chance to publish a correct and excellent paper, so the public rightly has the right.
The above policy can also be made to editors.
The world, in particular Europe, used to have public debates about science. One famous example is the debate between Tartaglia and Ferrari on cubic equations. (Unfortunately, Tartaglia lost the debate, but the point here is that the debate is public.) Could we have that environment again in science and referee work?
- There are cases where authors of papers previously published, and later the proofs are found wrong, are invited to be referees. In this case, it would be a strong motivation for these referees to reject papers with correct proofs of stronger results, in particular in the case the wrong proofs are specifically mentioned in the new paper. I do not know how often this is, but my feeling is that it happens quite often. In my opinion, this is a case of "conflicts of interest", and should never be allowed. Think about the analogy in real life, and you will see how strange this is. A person B is accused by a person C, and then the police invites the person B to judge whether the accusation by the person C is correct. (I heard this anecdote from a mathematician: "Conference organisers usually invite their friends". This may also apply to referee work?)
Another case is when the new papers have new methods/ideas/point of view, which can solve cases not able to be done by previous methods. Authors of the old methods, who are invited to be referees, may (while not always the case and I think, not very often) try to delay the publication of the new papers. In the old days, this method was very effective, and even ideas from submitted papers may be stolen. Nowadays, with the advance of online archives such as arXiv, this is not so in terms of spreading new knowledge and claiming authority. However, it still affects very adversely career of a junior researcher and the funding of a senior researcher.
- Also, there is the issue of how much time a referee takes for his/her work. I think journals should impose a strict deadline on this, for example at most 6 months for papers not longer than 30 pages. (In my experience, this is a right amount of time. Compute: then on average the referee works 5 pages a month. That he/she can do on a beautiful Sunday morning, or whatever 2-3 hours duration suitable him/her.) Of course, a referee may say that he/she has no time, but if that is the case the referee should refuse to do the work at beginning. Some journals already have this in practice, but many journals don't. In the real life, when you work for a company or do some services, there will be always deadlines. So why not when you do referee work? (An argument is that referee work is free, but as I wrote in the beginning of this essay, it is not actually so. Doing a referee work gives you more benefits than the amount of time you spent on it. If a referee feels that he/she works for free and that is not good, then simply he/she just refuses to do the work. I believe there will be many other suitable researchers who are happy to fill that referee work. Also, the question is that how much time that a referee really spends on a paper, and how careful the referee is? Understanding this will give an understanding of why, even in mathematics, where people play with strict rules and logics, there are too many wrong published papers.)
- There are also arguments such as: "This paper is too narrow a field". Frankly speaking, any paper will be in a certain narrow field. An average scientist may know the names of Fields or Nobel's awardees, but usually do not know or have interest about the work of these. (Here is what I myself witnessed when I was still a PhD student: One mathematician, after hearing a talk presenting the work of a Fields medalist, asked: Is this all that got him Fields medal?) So what about other papers published in that journal? Except if the journal actually has a policy that it publishes only papers on certain fields, I think this argument alone should not be a reason for rejecting a paper.
- There are also referees' reports which instead of writing on papers, criticise the authors on a personal level or make undue comparisons. By undue comparisons, I mean the referee irrelevantly compares the author's work with a specific author(s). This is unfair, because if a comparison is to be made, it should be made with an average published paper in that journal. Otherwise, if the referee compares any submitted paper with one of the best papers published in that journal, the chance is that the new paper won't seem good enough, and then recommend rejection. (I use seem here, because who knows about the future.) On the other hand, a referee can always compare any paper to a paper in the journal with wrong results/methods (there are always such papers in any journals, either scientific or popular, news, I am sure), and then conclude that because the journal published wrong papers before, it should publish any paper. I think this kind of comments should never be allowed. In the real life, such insulting behaviour may be sued. This is not to mention that how much the referees be confident about the future of a certain paper or author, when making comparisons? Would they not be on the same tracks of referees who refused to publish Harry Porter?
- Related to the above issue, there is also the responsibility of journals/authors concerning published papers with wrong methods/results. These wrong papers should be acknowledged in the journal, so that future researchers won't mistakenly use the results before having found another way to correct the results. This is also to make sure that these wrong papers don't get underserving citations just because of their authors' status. Science is not a monarchy government, with kings and knights and nobles get priority, right?
I propose the following formal procedure that journals can implement regarding this:
In the best case, the authors will identify the errors in his/her paper and send either an acknowledgment or a correction of it. If he/she cannot correct the error then the journal acknowledge that the paper was wrong. Otherwise, move to step 3 below.
In other cases, where another researcher identify the errors, he/she can send a complain to the journal. He/she can choose to be anonymous to the author, but in that case, the author will have the right to know the identity of this complainer after the same amount of time mentioned above regarding referees.
The journal then ask the editor(s) together with the referees who accepted the paper before, together with a new editor and new referee to deal with the case. The complain will be sent to the author, and he/she is given a deadline to answer the complain. Then the answer of the author will be studied by this new team together with the complainer. The process may be repeated several times until all parties agree with the final conclusion. The whole process will be then documented in an official form from the journal, with the signature of the editor in chief.
- Being an editor/referee is a privilege, so those people should have strict ethics. Those with good behaviour should be acknowledged and those with bad behaviour should be banned from serving on the journal.