At Language Science Press, we are experimenting with Open Review. While investigating how we could implement an Open Review system, we discovered that there are actually very different things which could be called Open Review, and even in a small team, opinions differ as to what is the real thing.

Axes

We have established the following dimensions:

The  following criteria do not apply to openness, but are nevertheless relevant:

Selection process

One meaning of “Open” is that everybody is invited to participate. That shade of meaning is for instance present in Open Source,  Wikipedia, or the concept of Citizen Science. One aim of movements with this interpretation of “Open” is a more democratic culture in allowing everybody to express their opinions regardless of privileges such as formal education, academic degree, or belonging to a group of peers.

For Open Review, this would entail that no preselection is made as to who is allowed to review. This can be compared to the forums attached to newspaper articles, where everybody can express their thoughts about the article. One obvious drawback is that the quality (and civility) of the comments cannot be guaranteed. The opposite would be that the editorial board chooses a number of reviewers for the Open Review. That review might then be open according to other criteria listed here, but not according to an open selection process.

Transparency

Another meaning of “Open” is that the process is transparent and everybody can trace the individual steps which led to the final outcome. This shade of meaning is for instance present in Open Government or Open Notebook Science. The aim is normally to prevent fraud, collusion, corruption, abuse of power and the like.

For Open Review, this would entail that the names of the reviewers as well as the reviews are revealed so that the general public can scrutinize whether established best practices were observed. The opposite would be anonymous reviews whose authors’ names are not disclosed.

Note, however, that the principle of Transparency conflicts with Openness in the selection process. If anybody shall be able to comment, it is practically impossible to require previous registration with an ID card or the like.

Online

A third aspect of Openness is whether the reviews are “born online”, normally in a browser using a web application. In that case, the individual comments will have their own URIs/DOIs and a history might be available. The possibility to uniquely identify the comments allow to reply to a comment, to support or oppose a proposed change and so forth.

Issue tracking in Open Source software development works like this for instance. The opposite would be a “classical review” written offline, maybe even with a pen. Such a review can still meet other criteria of openness (e.g. transparency), but it would not be an online open peer review.

Scope

The fourth axis is the scope of the commentary. Reviews of books or articles very often consist of two parts: one general commentary about the merit and presentation of the research and a list of more local remarks concerning grammars, style, spelling, missing references and the like. The aim of the open review process can be to either model the former, the latter, or both. The decision will have consequences on some other axes. If the list of local changes is aimed at, for instance, online reviewing is a natural choice.

Time

The fifth axis concerns the timing of the Open Review and is tightly connected to the next axis, Relevance. The open phase can precede the publication. Comments during this phase can be incorporated in the final version. The opposite is post-publication commentary, which is arguably not a very good example of a “review”, but rather a reply or  a rebuttal.

Note that neither option makes the review more or less open, but it will have consequences on the implementation, and it might have consequences on the acceptance.

Relevance

The final axis we consider is relevance: do the comments make any difference about whether the paper/book gets accepted? This is related to Scope: global appreciations tend to have an influence on the acceptance, whereas local remarks about grammar or style generally do not. If the comments have no influence on the acceptance, a separate process of quality assurance should be established.

Implementations

As mentioned above, we are still in an exploratory phase. For the time being we have implemented two different approaches. Both are found in the series “Textbooks in Language Sciences“. They are remarkably different:

#1 Preselected transparent offline global prepublication pre-acceptance

Schäfer, Roland (2015). Einführung in die grammatische Beschreibung des Deutschen.

This work has been reviewed in the classical way. The reviewers were chosen by the editorial board. One reviewer, Eric Fuß, has accepted to have his review published. The review was written offline, concerns the whole book, and the individual remarks cannot be referenced individually via an URI or a DOI. This review has influenced the decision to accept this book (there was anonther review, which is not disclosed, which was also favorable).

#2 Open anonymous online local prepublication post-acceptance

Müller, Stefan (2015). Grammatical Theory.

This work is an expanded translation of the German original. As such, the quality of the book had already been established during the first edition. For this book, we use the online service hypothes.is, which allows anybody to mark passages of the text and comment on them. This open commentary phase will last for 2 months, and the suggested changes will be considered for the final version. While we do not have any empirical data yet, we expect that most comments will be rather local as the chosen technology favours this kind of interaction.