Challenging conventional wisdom and the conventional peer-review system—a recent experience

by Demetris Koutsoyiannis & Zbigniew W. Kundzewicz

Having published some hundreds of papers over decades and having been co-editors of Hydrological Sciences Journal (HSJ) for a long time (ZWK for 18 years, DK for 12 years), we have been interested in the peer-review system. Trying to diagnose its pathologies and suggest remedies, we have written several editorials published in HSJ (Kundzewicz and Koutsoyiannis, 2005, 2006; Koutsoyiannis and Kundzewicz, 2007, 2009). We also collaborated with editors of other hydrological journals and compiled joint editorials, published in several journals simultaneously (Blöschl et al., 2014; Koutsoyiannis et al., 2016; Quinn et al., 2018).

There are three different peer review systems, from the transparency viewpoint:

  1. double-anonymous (double-blind) system, where the identity of the reviewers is not disclosed to the authors and the identity of the authors is not disclosed to the reviewers;
  2. single-anonymous (single-blind) system, where the identity of the reviewers is not disclosed to the authors, while the identity of the authors is disclosed to the reviewers;
  3. eponymous, fully transparent, system, where the identity of the reviewers is disclosed to the authors and the identity of the authors is disclosed to the reviewers;

System (1) is rather a delusion or even a joke, because very often the identity of the authors can be easily guessed, based on references. System (3) is the most ethical and responsible—yet very rare. Hence, far more common (if not virtually exclusive) is the system (2).

Anonymity has been so prevalent in scientific transactions that many colleagues only know the Greek word anonymous (ἀνώνυμος) and not its antonym, eponymous (ἐπώνυμος).

Overall, we have supported the need for a change in culture in the peer-review process toward enhanced transparency.

Recently, we had an interesting experience, not as editors but as authors of a paper challenging conventional wisdom (Koutsoyiannis and Kundzewicz, 2020; graphical abstract in Fig. 1). Trusting that this case deserves being disseminated, we describe this experience below as a didactic case, whose lessons could point to the direction where—in our opinion—progress lies.


Fig. 1. Graphical abstract of Koutsoyiannis and Kundzewicz (2020): “Atmospheric temperature and CO₂: Hen-or-egg causality?”

Initially, we submitted our paper to Science of the Total Environment (STOTEN). It was swiftly rejected based on two negative anonymous reviews. This was no surprise to us: it is well known that anonymous reviewers tend to kill papers presenting new ideas. We challenged the editor’s decision as we felt that we could rebut every one of the critical review comments. As usual, we received a negative reply by the editor: “Unfortunately, we can no longer consider the paper for publication in STOTEN”. We made our challenge quite strong, by rendering the manuscript public in ResearchGate along with the reviews. We asked the STOTEN editor “to let the reviewers know that our paper and their reviews are uploaded in ResearchGate and that we will be happy to post our rebuttal there if the reviewers agree to post replies to our rebuttal”.

We are pretty sure that the editor and the reviewers visited our posts (the manusript and the reviews) on ResearchGate. (Some of the sections of the final published paper contain replies to these latter comments, as we have specified on ReserachGate). This platform allows posting comments—eponymously of course. But no comments were posted (or received in private communication). Again, no surprise. These reviewers could easily kill the paper in an anonymous mode—but they would have difficulties in justifying their criticism if they were eponymous.

Indeed, eponymity is more demanding. Eponymous reviewers should articulate their claims and should be ready to receive criticism on them by the authors. In other words, eponymity needs more effort and courage. But it is cooperative, democratic, equitable, ethical, productive and responsible.

In our view, in an era where the quest for transparency has become extremely important, it is time for a radical change in scientific ethics. Thus, when we are tempted to submit an anonymous review, a good question to ask ourselves is this: If I cannot be an eponymous reviewer, is it accurate to be called a reviewer? (And if yes, who is actually that reviewer? Myself or my anonymous, perhaps frightened, clone?)

Some modern scholars use Clarivate’s PUBLONS to secure recognition for their anonymous reviews. We suggest that they ask themselves another question: What is the ethical and aesthetic value of seeking credit for anonymous transactions?

After the unpleasant experience with STOTEN, we decided not to submit our paper to another journal employing blind review. We were attracted by the novel, eponymous, review system of the emerging, open-access, journal Sci published by the Multidisciplinary Digital Publishing Institute (MDPI) group (Rittman and Vazquez, 2019; Jacob et al., 2019). In this innovative, community-driven, so-called post-publication peer-review system, a decision for (pre-)publication was made by an editor and then reviewers were invited (or volunteered, as everything is public and transparent). After one or two review rounds, a final decision was made and, if this was positive, the paper would be included in a journal issue.

We decided to submit our paper to Sci for two reasons. First, to avoid another likely rejection after another blind review, with the reason of rejection being au fond the fact that we challenge conventional wisdom. And second, to support the innovative progressive peer-review system of Sci. We were aware, of course, that Sci, as a new journal, is not included in the Clarivate’s Web of Science and Science Citation Index. We did not use this as a criterion for our choice.

Our experience with Sci was positive. The public and eponymous reviews were  constructive and recognized the usefulness and importance of our paper. In the second round, the reviewers fully accepted the way we handled their comments and our reasoning about it. In fact, we considered even the negative (anonymous) review comments of the rejected submission to STOTEN and we strengthened our paper against them. Eventually, our paper was accepted and published. The timing of the several stages of the process is shown in Figs 2. and 3.

Unfortunately, the acceptance of our paper coincided in time with a major change of Sci’s peer review pattern to the conventional (single-blind) system (Vazquez et al. 2020).  The change is also visually reflected in the journal’s web platform: compare Fig. 2 (before) and Fig. 3 (after).

In our case, the rules of the game changed during the game. We communicated our complaints to the editors and the publisher, explaining that we would not submit our paper in that journal if it was run with the conventional peer-review system. We also expressed our disappointment that a big step of progress was followed by a big step of regression. We understood the publisher’s reasoning that there exists a clear conflict—if Sci continued with this system, Clarivate Analytics would not include it in its indices. As we wrote, for us this is not important, but we understand that for most colleagues—particularly the youngsters—it is. Indeed, bibliometric indices spanned over recognized sources from the Clarivate’s Web of Science, and citation counts in particular, are the currency in which scientists are evaluated nowadays. Given that Clarivate has invested a lot in anonymity, through its PUBLONS system, it is explainable (yet saddening) that it discourages an innovative progressive system in favour of the traditional, anonymous, transactions.


Fig. 2. The Sci’s site with our paper in its original (post-publication peer review) setting with full recognition of reviewers’ contributions.


Fig. 3. The Sci’s site in its changed setting, with the final version of our paper; yet the eponymous reviewers’ contributions are accessible by hitting “Review Reports”.

Nonetheless, this Sci’s step back was, fortunately, partly counterbalanced by a subsequent arrangement by the journal not to fully abolish eponymous open review. Thus, it runs now with two alternative options. As explained in the instructions for authors, “Authors are given the option for open peer review or single-blind peer review.

We welcome this arrangement and, thanks to it, we will continue to support Sci for its first option.

 

References

Blöschl, G. Bardossy, A., Koutsoyiannis, D. Kundzewicz, Z.W. Littlewood, I.G., Montanari, A., and Savenije, H.H.G., 2014. Joint Editorial—On the future of journal publications in hydrology, Hydrological Sciences Journal, 59 (5), 955–958, doi: 10.1080/02626667.2014.908041.

Jacob, C., Rittman, M., Vazquez, F., and Abdin, A.Y., 2019. Evolution of Sci‘s Community-Driven Post-Publication Peer-Review. Sci , 1, 16, doi: 10.3390/sci1010016.v1.

Koutsoyiannis, D., Blöschl, G., Bardossy, A., Cudennec, C., Hughes, D., Montanari, A., Neuweiler, I., and Savenije, H.H.G., 2016. Joint Editorial: Fostering innovation and improving impact assessment for journal publications in hydrology, Hydrological Sciences Journal, 61 (7), 1170–1173, doi: 10.1080/02626667.2016.1162953.

Koutsoyiannis, D., and Kundzewicz, Z.W., 2007. Editorial—Quantifying the impact of hydrological studies, Hydrological Sciences Journal, 52 (1), 3–17, 2007, doi: 10.1623/hysj.52.1.3.

Koutsoyiannis, D., and Kundzewicz, Z.W., 2009. Editorial—Recycling paper vs recycling papers, Hydrological Sciences Journal, 54 (1), 3–4, doi: 10.1623/hysj.54.1.3.

Koutsoyiannis, D., and Kundzewicz, Z.W., 2020. Atmospheric temperature and CO₂: Hen-or-egg causality?, Sci, 2 (4), 83, doi: 10.3390/sci2040083.

Kundzewicz, Z.W., and Koutsoyiannis, D., 2005. Editorial—The peer-review system: prospects and challenges, Hydrological Sciences Journal, 50 (4), 577–590, doi: 10.1623/hysj.2005.50.4.577.

Kundzewicz, Z.W., and Koutsoyiannis, D., 2006. Pathologies, improvements and optimism, Hydrological Sciences Journal, 51 (2), 357–363, doi: 10.1623/hysj.51.2.357.

Quinn, N., Blöschl, G., Bardossy, A., Castellarin, A., Clark, M., Cudennec, C., Koutsoyiannis, D., Lall, U., Lichner, L., Parajka, J., Peters-Lidard, C.D., Sander, G. Savenije, H.H.G., Smettem, K., Vereecken, H., Viglione, A., Willems, P., Wood, A., Woods, R., Xu, C.-Y., and Zehe, E., 2018. Invigorating hydrological research through journal publications, Hydrological Sciences Journal, 63 (8), 1113–1117, doi: 10.1080/02626667.2018.1496632.

Rittman, M. and Vazquez, F., 2019. Sci—An Open Access Journal with Post-Publication Peer Review. Sci, 1, 1, doi: 10.3390/sci1010001.v1

Vazquez, F.; Lin, S.-K.; Jacob, C., 2020. Changing Sci from Post-Publication Peer-Review to Single-Blind Peer-Review. Sci, 2, 82, doi: 10.3390/sci2040082.

 

This entry was posted in - peer review. Bookmark the permalink.

11 Responses to Challenging conventional wisdom and the conventional peer-review system—a recent experience