How can we have all scientists publish their works on equal grounds? How can we make sure that the good ideas and results get published fairly instead of falling through the cracks? How can we prevent the major journals publish fake/bad results because of limited or insufficient reviews?
How can we prevent an elite circle publish their works easily while bullying others or new comers outside the circle? How can we prevent one biased referee/editor killing the publication of one brilliant paper? How can we use every critical eye in the scientific community to vet a manuscript before it gets published?
Here my goal is to imagine an ideal publication system for science, at least basic science. We can start with physics. But it may apply to other disciplines as well.
Successful arXiv
“Started in August 1991 by Paul Ginsparg, arXiv.org is internationally acknowledged as a pioneering digital archive and open access distribution service for research articles. The e-print repository, which moved to the Cornell University in 2001, has transformed the scholarly communication infrastructure of multiple fields of physics and plays an increasingly prominent role in mathematics, computer science, quantitative biology, quantitative finance, and statistics.” — Cornell University
ArXiv (arxiv.org, first started as a physics preprint archive in 1991), after so many years, has turned out to be the best invention for scientific publication. First and the foremost, the quality of arXiv papers is superb. Personally, I found a lot of gems in arXiv that could not get published in traditional journals. The service of arXiv (as compared to other journals) is probably the most important for the physics community, at least for physicists in the fields of high energy physics and cosmology. Secondly, many traditional journals require subscription services to have access to their published articles while arXiv can provide a free copy, although not polished by professional editors (they are doing less and less polishing work anyway), for anyone who don’t have or can’t afford such services. Open access as part of the open standards is critical for the advancement of sciences.
Even as brilliant as the arXiv system, it is still prone to faults when the human factor is involved. For example, arXiv uses volunteer moderators to screen or re-categorize submitted manuscripts. Ideally, you’d like the moderators to exercise the minimum in the review process to ensure the publication of all reasonable scientific submissions. In particular, a submission should not be rejected solely due to the moderator’s personal bias or a very wild idea in the work. Such kind of over-screening, unfortunately, indeed occurred, for example, as reported in the Nature News article “ArXiv rejections lead to spat over screening process“.
Personally, I also witnessed the over-exercise of arXiv moderation. I posted a manuscript titled “Dark energy and spontaneous mirror symmetry breaking” in arXiv. I cross-listed it in the cosmology (astro-ph.CO) category but got removed by the moderator. I felt that dark energy is one of the main topics in cosmology and I’d like to present my work to the cosmology community. So I appealed but got denied (probably by the same moderator). Mmh, I thought they must have very strict cross-list rules. I was curious enough to try to cross-list the same paper in the general relativity (gr-qc) category and it succeeded. To my own taste, my paper definite fits better in astro-ph.CO than in gr-qc. The community in astro-ph.CO would probably have a better chance of getting interested in my work. It seems that the moderators of gr-qc are less restrictive that those of astro-ph.CO.
Nevertheless, necessary moderation is one of the reasons why arXiv is so successful. Together with the brilliant endorsement system (an author has to be endorsed by another established arXiv author before submitting), arXiv has ensured surprisingly rare crackpot articles in the server. Too many crackpot papers will definitely drive the real scientists away from it.
There are other great features that have contributed to the success of arXiv. For example, publication in arXiv takes almost no time as compared to the typical waiting time of months to years in traditional journals. This is essential for timely disseminating scientific results in today’s fast-paced world. Another nice feature is that arXiv give authors a grace period of typically one or two days before the submission goes online. This is good for both necessary moderation and possible change of mind by the author right after the submission.
Improving arXiv
However, arXiv, as it is now, still has shortcomings and is not a complete service. One of the most glaring issues is lack of a commenting/rating system. Human’s knowledge is exploding these days. More and more papers and ideas are coming out of more and more sophisticated human minds. How can each of us seek what we want without reading them all from such an overwhelming information explosion? A commenting/rating system is the best to save us all our limited time.
Such a review/rating system is provided in regular journals, which sort of make up what arXiv is lacking. But it is far from ideal. Each journal has its own review system and essentially they do not share information between them. For example, if one article is rejected by one journal and then submitted to another one, the 2nd one will start the review process all over again. This is a dramatic waste of resources. For such closed systems, a bad article can always find a journal to get published while a good one may never get published at all.
As an open and largest e-print server, arXiv can provide a much better review/rating system avoiding those drawbacks in regular journals. Meanwhile keeping the existing endorsement / moderation system (maybe with some improvements), arXiv could be the best ever publication source for scientific research replacing (possibly by overlaying) most if not all other relevant journals.
Another key principle is to maximize academic freedom and minimize regulation while keeping good quality of the arXiv service, what arXiv has already done amazingly. The further improvement will be towards a self-servicing and self-regulating community, which could be achievable when adding the review/rating system as discussed below.
Members in the arXiv community could have four different roles: author, reviewer, endorser, and moderator. The improvements on arXiv can start from how these roles can be played . All members can post their first article as an author with no strings attached. But any further submissions will need credits (e.g., 10 point per article) that can be earned by doing work as a reviewer, endorser, and/or moderator. The more credits you earn, the more articles you can submit (of course, you don’t have to). This will ensure a healthy community if everybody participates in some service. All members in good standing should be able to review / rate articles other than their own. On the other hand, roles of endorser and moderator may require more experience (e.g., >30 credit points, at least 3 months as a member and/or posted at least 3 articles, etc.) and the two roles could even be combined into one.
One important privilege for members is that they should have options to make themselves public or anonymous for all their activities or any individual activity such as reviewing and moderating. Once a member’s credit is high enough, she/he will be given a chance to volunteer her/his service in various committees like the appeal committee, the advisory board, etc.
On the other hand, the authors can decide if they want their papers to be commented or not under three different cases. They can allow no comments at all on their papers. They can also allow comments from identified members only. In the last case, they can permit comments from both identified and anonymous members. Obviously they have to be aware that restricting comments could limit their chances for publication in overlay journals (discussed later). As far as the rating system (either a thumb-up/down or 1-5 star system) is concerned, one may argue that rating a paper should be allowed under any circumstance. But a similar author-granting policy for the rating system could be implemented as well.
The above two-way freedom should make most members if not all happy. There are also added bonuses. Compared to traditional journals and existing arXiv moderation, the above model can provide a much larger pool for review and moderation. More eyes on a paper can make it better vetted before publication on an overlay or any other journal. On one hand, it can make the creditable work published in a better (error-free) form. On the other hand, it can reduce the chances of wrong results or even fraudulent papers that get published, which are occurring too frequently nowadays in some of the most esteemed traditional journals.
It seems to be harder and harder for the traditional journals to get timely review reports from quality referees. The problem is that there is little incentive for a good referee to do more review work for those journals, which is more and more like a burdensome service. On the contrary, in the improved arXiv system, reviewers are motivated to do their part of the review work in order to publish more of their own works. In addition, excellent contributors could be involved in various committees of their research categories or play a bigger role in the community. Last but not least, their valuable comments could be appropriately acknowledged should they choose not being anonymous. If their contribution is so significant, they could even be included as authors in the revised paper. This is not possible for a blind review process adopted in traditional journals.
More experienced members participating in moderation can alleviate the burden and more importantly avoid over-regulation or incorrect rejections due to personal bias of few individual moderators as in the current situation. The moderators need to judge if a submission should be accepted, frozen, or rejected. “Accepted” means the submission is at least reasonable relevant serious scientific work even though a moderator may not agree with the results. “Rejected” means for works that are either irrelevant to any covered topics or inappropriate to general public (e.g., abusive, violent, etc). “Frozen” means anything in between, most likely belongs to the crackpot articles.
The moderation strategy would be using a decision threshold and following the order of accepted->frozen->rejected to be conservative. For example, if net N (e.g., N=2) favorable votes received, the submission will be accepted (then visible to members and public and ready for comments). Otherwise, if net N frozen votes received, it will be frozen in a separate database (e.g., prefixed with “unfiltered”), not visible to public and not allowed for comments, but could be seen by members or at least by moderators. At last, if net N rejection votes received, it will be moved to a trash database that could be accessed only by relevant appeal committee members. If a good submission is mistakenly not accepted, it can be voted back by other moderators. If a bad one leaks into the main database, it can be removed by moderation votes as well within a time limit (e.g., three days).
The success of arXiv relies on little if any refereeing. If given the opportunity, human beings tend to over-regulate. So the moderators should be educated to reject the submission only when it is completely inappropriate or irrelevant, and to freeze it when it is clearly scientifically inappropriate. As far as the cross-listing is concerned, it should ideally be up to the authors. ArXiv can set a limit, e.g., at most 3 categories for each submission to prevent abuse. But the moderators should refrain from over-regulation. Only the clearest wrong cross-listings should be removed. The credit system could be used to prevent moderators from abusing their regulating right.
More detailed rules regarding viewing, submission, comment/rating, moderation, and credit could be established after a period of trying the new model. Here we can imagine some of them in a simple picture. General public can view the “accepted” database only and can not make comments. Members, i.e., logged-in users can comment/review and/or rate “accepted” submissions. The comment itself could be rated as well. A good comment could get one credit point while an excellent one can get more (e.g., 10) points. A useless comment will be removed and may get zero or even -1 point. An abusive comment will get 10 points deducted from the commenter’s credit. Members should be advised to comment professionally like they are doing a review report for a regular journal article. Additional constraint for the rating system could be that members can rate as many papers as 10 times their credit points. It could be another incentive for members to do more and quality review work. Members with low credit score will not be able to publish, comment, or rate any more.
A member’s first paper is free. But the next one can be submitted only when the member has earned at least 10 credit points. The idea is that you have to serve more if you want to publish more. 10 credit points per paper seems to be reasonable. To prevent over-comment, a constraint could be set for each paper, for example, comment function for a given paper could be closed if its number of comments exceed 100, or the paper is more than one year old. All credit-earning comments should follow official review/reply standards as in the journal refereeing process. A link from each paper to discussion boards / forums that may be hosted separately could be introduced and direct any further discussions and unofficial comments elsewhere. Inappropriate comments should be deleted while the lowly-rated (i.e., not very useful) ones could automatically be moved to the forums.
Experienced or senior members may also earn credit with their moderating work (e.g., one point for checking every 10 papers). These members could be defined, for example, by a group with at least three accepted papers, more than 50 credit points, and > 1 year membership. Members with even higher credentials can form the appeal committee within their research category.
With the new review/rating and credit system, arXiv could become the most reliable and complete source/service for scientific research papers. Naturally it will lead to the formation of arXiv-based overlay journals as discussed below.
arXiv-based overlay journals
The arXiv-based overlay journals have actually started forming. Many journals now ask the arXiv number when one submits his/her work. But something is still missing in the current arXiv system for a working overlay mechanism.
The current financial model of arXiv is becoming more sustainable. In particular, arXiv started to collect membership fees from 200+ member institutions since about 2012, which take up about half of the revenue for the operation expenses. By introducing the overlay journals based on arXiv, it will open up another source of revenue and may make the business model of arXiv really sustainable in the future.
The above-discussed review/credit system will provide the backbone for the overlay journal system. Any publication organizations can start overlay journals on arXiv. They just need to pay arXiv certain amount of access fees based on the quantity and quality (e.g., by rating) of the papers they adopted. Then they can enjoy the benefits of the existing review system making the publication process smoother and faster. Meanwhile, knowledgeable editors from these journals could make the arXiv review system better.
A typical publication procedure in an overlay journal could be like the following. First, the authors of a posted article decide to publish it in this overlay journal. Their action could be just a simple click on their arXiv user interface and the considered journal will be notified. The editor of the journal can then check if the article has received any comments. The editor can deem any number of comments as valuable (which could receive up to 10 credit points each) and ask the authors to address the issues if raised. It may take several rounds of comments/responses between reviewers and authors but eventually the editor will decide if the article will be published in this journal. If there are no comments yet for the paper or more reviews are desirable, the editor can solicit more comments from other experts. These solicited comments will automatically receive the highest credit points (e.g., 10). Similar to the other regular comments, these comments can be identified or anonymous as the reviewers wish.
Another important difference than the traditional review process is that all review comments and responses are visible to all members and all members can participate. Personal bias could be corrected by more eyes from the community. A thoroughly vetted paper would make the publication quality that much better, which the overlay journals should definitely appreciate. If one journal decided not to publish the paper and another one wanted to pick it up, they don’t have to restart the review process.
The overlay journals can be categorized in different disciplines like physics, math, chemistry, etc. A journal could also focus on a sub-field like cosmology in physics. It could be like a high-impact journal for short letters (e.g., Physical Review Letters) or a journal for long and comprehensive reviews (e.g., Reviews of Modern Physics). As a matter of fact, the Physical Review series from the American Physical Society could be the first arXiv-based overlay journals.
Besides the review system, journals and editors could also utilize the rating system or at least use it for gauging the impact of the considered articles. For example, the top rated (e.g., 5 star) papers may be published more likely in the top tier journals. Accordingly, the four-star papers could be published in the 2nd tier journals of different fields. The lowest-rated papers may not be published in any journals.
Such an overlay system is not much of a burden for arXiv once the review/credit improvements are implemented. Yet it can provide significant and stable income for arXiv’s continuing success. It is a win-win situation for both arXiv and publishing organizations and even better for scientific research. It is also expandable. More research fields / disciplines and their researchers may join in arXiv. More and more publishing organizations may move more of their journals to be arXiv-based. It may include papers written in different languages and overlay journals from different countries.
More details could be implemented. For example, each article should be posted on arXiv for a period of time (e.g., one month) before being considered by an overlay journal. The star ranking could be two-fold: short term rating (e.g., one month) and long-term rating (e.g., a year). The review/rating system could be closed upon publication of the paper. Or a post-publication rating system is separate.
Outlook
I wish such an arXiv-based system would be implemented in the near future. It is time but unwillingness to change may still take some time to overcome, in particular, within the elite circles of real talking powers. However, it may just take one willing figure to realize such a dream.
See a past survey article for arXiv-based commenting resources here: https://content.iospress.com/articles/information-services-and-use/isu856
Other disciplines have started to copy the arXiv model and establish new preprint servers in other fields. See the October article from the APS news.
Over-regulation at arXiv caused discontents. An alternative e-print server viXra.org began in July 2009 to publish what arXiv is not willing.
No comments:
Post a Comment