Jacob

The referee answers certain questions to help the editor to help the editor to decide what should be published. Weaknesses in the reviewing system stem from inadequate instruction of referees as to what they should be doing.

About fifteen year ago I wrote an article, ‘Rules for referees’, that had as its goal the establishment of efficient practices in editorial refereeing. At that time I was afraid that basically sound idea – the manuscript review system – might come to grief as result of being used improperly. To my knowledge, that article has had little effect, the review system is still ailing, and editors still discuss the problem. Therefore, I am going to try again to come to the rescue.

My first effort was written when I was what I call an ‘author’s editor’ – that is, I polished manuscripts for authors before the manuscripts were sent to the journals. So I was the authors’ agent and probably wrote from that point of view. Now I am the managing editor of journal, the other side of the coin. I see between fifteen hundred and two thousand referees reports each year but, except in rare cases, I do not participate in the accept-or-reject decision. Thus, I have had the change to watch the system work without actually being participant. I am not sure how to classify my point of view at this writing, but I am still in favour of the system and quite critical of the way I see it being used.

A good place to start is with some definitions of goals and terms. The goal of the manuscript review system, at least as it is applied to journals, is to protect the literature and the reader from identifiable error. To this end, the concept of ‘peer review’ is not appropriate. Whenever possible, the reviewers (or referees) should be quite expert in the topic of the paper – that is, better than peers. As I tried to establish in 1965, the reviewers have the primary task of answering certain questions. On the basis of these answers, the journal editor then makes the decision to publish or to reject or to offer publication if the paper is modified. I suggested, ‘The referee’s task is to read the manuscript and answer the questions. He should neither look for something to criticize to prove his diligence and capability as a referee not overlook or condone ommisions or errors to prove his graciousness’. In other words, the reviewer is testifying, as an expert witness, to matters of fact.

One group of questions makes use of the reviewer’s specialized knowledge. The following are important:

1. Does the main part of the paper present new facts, observation, or ideas?

If the reviewer reports that it does not, he must supply references to prove that this material has been published previously. In other words, the answer to this question depends not on what is known to the reviewer but rather on what is generally known or available in the literature.

2. Has the published literature been treated fairly?

This broad question should be taken to include just recognition of priorities, accurate citation of the views of others, and a fair if not complete survey of the truly relevant prior publication.

3. Can the data reported be obtained with the methods used?

In other words, are the measurements or observations believable? Special knowledge or experience is very important here because even the most cautious reader can be misled in interpreting results of unfamiliar methods. The reviewer is being asked, ‘Will this method of measurement give a valid answer in these circumstances?

There is another group of questions that require less special knowledge, and I suggest that the reviewer may answer or not, as the individual case seems to require. For example:

1. Can the observations be explained by one or more interpretations other than the interpretation offered by the author? An affirmative answer here is not necessarily a criticism of the paper, but the editor should know that such additional interpretations exist. To what extent the author should be require to discuss them is a complex matter that requires astute editorial judgement.
2. Do the observations support the conclusion presented, anequivocally, strongly, reasonably, or barely? Are there deficiencies in the chain of evidence? If there are, how significant are they? The existence of a deficiency per se need not require of the paper. Again, editorial judgement as to relative importances must be exercised.

Reviewers tend to let their attention wander in to other areas. The reports I see range from one sentence to several pages of single-space typescript. Variation of this great an extent suggests that there is some misunderstanding of the task. For example, I see extensive corrections of grammatical and stylistic matter; this is a waste of the reviewer’s time (we make these corrections when we prepare the paper for the printer) and of the author’s time (because of the advice frequently is incorrect). I see extensive attention paid to minor details and all sorts of ‘ifs’ and ‘buts’. No paper is going to be perfect or be the final word on the subject, so why carry on at great length about trivia?

There are some topics about which the reviewer’s comments do not carry the weight of expertise, and perhaps carry no weight at all. This is a vague area. It includes criticism of the subject matter, of the scope or goal of the paper, and of the direction of the approach. I look at these comments in a negative way because they interfere with the author’s freedom to select the problem he wants to consider. The author should not be required to perform additional experiments or studies to validate conclusions invented by the reviewer.

A serious form of reviewer’s trespass occurs in the matter of decisions-making, and this occurs because editors permit it to occur. An important basic feature of the whole system is that an ‘editor’ (editor, associate editor, editorial board member, etc.) make the decision to accept or reject. In other word, the reviewer report that the paper is fact or fancy. The editor decides on the significance, to his readers, of the fact or the stimulation value of the fancy. The decision-making must lie within the office of the journal. This responsibility cannot be farmed out or delegated. An editor who will not assume this responsibility has no business being an editor.

Thus, fifteen years later I still hold to the ideas that the system is good, that there are important questions that reviewers must answer, and that there are decisions and topics that are closed to reviewers. However there is a worrisome cloud in this picture. In the laboratory, when an analytical method is applied several times to samples of the same substance and gives significantly different results each time, one is forced to doubt the dependability of the method of the analyst. If the requirement of reproducibility were to be manuscripts reviews system, we would very soon search for a new method.

My journal requires at least two reviews per paper for one class of papers, and I would guess that there it as least some lack of agreement in more than 50 per cent of the cases and seriously lack of agreement in at least 25 per cent . This certainly does not speak well for the validity of the method. In the absence of a specific study of this phenomenon, I suggest that this variation result – at least in large part – from inadequate instruction of the reviewers in regard to what they should be doing.

Many journals provide their reviewers with report forms that carry specific questions. Some instruct the reviewers in their task. My journal uses a printed form with questions, but we provide space for ‘additional comment’, and that is where the reproducibility disappears.

By the time a scholar archives the recognition of an invitation to review, I think it is too late to attempt instruction. Perhaps what is needed is a new college (and thus the holder of multiple experiences with the course). I suggest that this course be called R3 – reading, writing, and reviewing.

Whenever manuscript reviewing is discussed, the topic of anonymity soon comes up. And it comes up in two forms: anonymity of the author and anonymity of the reviewer. I have serious doubts about the value of achieving anonymity of the author, at least in the ‘leading edge’ of laboratory science, where my journal operates. Reviewers who would be unduly influenced by the author’s name have other faults, too, and these reviewers should be culled from the list. In specialized fields, it is impossible to achieve anonymity of the author because there are too many internal clues (for example, the name appearing most frequently in the reference list). Thus, the value to be gained from author anonymity is now worth the effort, provided that the other faults of the manuscript review system (dishonesty, haste, vindictiveness, procrastination) can be corrected.

What about anonymity for the reviewer? On this question I continue to take the firm stand I took in 1965: yes and no. in other words, it all depends. In the normal operation of the system, the reviewer should remain anonymous. However, if anonymity is to be preserved, the editor must weed from the reviewer’s report those statements that are personal attacks or otherwise inappropriate to the matter at hand. (I did indeed see such types of comments when I was an author’s editor. It is interesting to note that the Proceedings of the National Academy of Science, where the reviewer’s reports are read by the nation’s leading scientist, I have never good could be accomplished by a conference with the author and will initiate that. Another circumstance for abandoning anonymity is when resolved and the editor believes that the paper is important. In such case, I proposed that the paper be printed with the contradictory review appended, of course with the consent of the author and the reviewer. This is a common practice in some journals except that the contradictory review is called an ‘editorial’.

I wish to make absolutely clear that the anonymity of a reviewer must never be violated without that reviewer’s permission. Likewise, the author mus be protected from unfair attack by a reviewer hiding behind the shield of anonymity.

What have I learned in the fifteen year since I wrote ‘Rules for Referees’? Mainly that my experience supports what I wrote then in relative naivety. At this time I would strengthen my plea that editors take their position seriously and guard their decision-making rights. Reviewers advise; editors decide. Editors do not have apologize to reviewers or to authors; they do have to be courteous. The decision to print or to reject is based on factors beyond those considered by the reviewer and therefore is not considered here.

Author : Bernard K. Forscher

Scholarly Publishing 1980, University of Toronto Press

Posted by Jacob's on Thursday, May 31, 2007 at 6:56 PM | Permalink