I am an associate editor for Math Zeitschrift. I thought that here could be a good place to record a few useful comments that I often pass on to reviewers. It is my intention for future referee requests to include a link to this post. I have tried to keep to issues specifically related to Math Zeitschrift rather than issues common to all reviews, although much of this advice could be applied more broadly. In particular, I have avoided discussing problems like “how much of the proof am I supposed to check” because that is not really an answerable question (beyond noting that is is the authors who are ultimately responsible for the validity of their arguments).
This post should also be useful for authors as well, although I will have a few more specific comments from them in the next post. The first thing a reviewer needs to understand is:
What is the level of the journal? This is probably the most important question to answer. The usual way that journals position themselves is to say that they are looking for submissions “similar to the level of Duke” which ranges from accurate to ambitious at best but is not necessarily useful for the reviewer unless they take it at face value (which they shouldn’t). Another way to proceed is to discuss the acceptance rate. At Math. Zeit, I believe the acceptance rates are in the range of 20% or so. Again, this is not so useful unless you have access to all the papers which are submitted — which you don’t. Slightly more informative is the following experimental observation: early on in my tenure as associate editor, approximately 60% or so of papers had favourable “quick opinion” requests. So, most likely, the standards are higher than you think. This reflects the unfortunate fact that more decent papers are submitted than can be accepted. Perhaps the most useful measure of quality, however, is to give a list of papers I have managed which have been accepted.
This is by no means a complete list, but I believe it is a complete list from some contiguous period of time (so no cherry picking going on, slipping in Jack Thorne papers just to look good):
https://arxiv.org/abs/1709.05759
https://arxiv.org/abs/1507.03238
https://arxiv.org/abs/1111.3654
https://arxiv.org/abs/1606.02341
https://arxiv.org/abs/1601.03131
https://arxiv.org/abs/1512.00586
https://arxiv.org/abs/1303.4630
https://arxiv.org/abs/1509.00320
You may form your own judgement from this. More importantly, as the quality keeps increasing, it might be useful to compare the paper I send to you with the papers above in areas most familiar to you and ask: “is this paper better? is it at least as good?“
What are you looking for from a quick opinion? At this point, I have decided not to convey any information from rejections in quick opinions to the authors. The reason is that enough authors respond with complaints that it is too much trouble, especially since (negative) quick opinions are mostly opinions concerning relative quality, not in depth reviews with useful suggestions. (Of course if the quick opinion comes with a useful suggestion I will pass it on.) So mostly I am looking for two things: is the paper at the level of Math Zeit, and (if so) do you have suggestions for possible referees for this paper? On the other hand, if it’s not up to snuff, could you give me a brief summary of why you came to that opinion (this is for my own benefit). Importantly, I am looking to reject most papers quickly at this stage. Mathematics has a problem with slow reviewing times, but it is especially painful to get your paper rejected (without any real useful feedback) after some especially long period of time. My dream is that all the papers that I reject will be rejected within a month. The journal process will always lead to good papers being overlooked and rejected; it’s hard for me to solve that problem but at least I can reject them quickly! I should say that I am pleased with how quickly I manage to reject most papers that are ultimately rejected (but by no means am I 100% compliant with my goal, unfortunately).
What if I don’t respond to your email asking for a quick opinion? Then you are added to the list of names destined for a sulfurous end, or, if you are lucky, merely a accidental prod from a poisoned umbrella next time we are at the same conference.
What if I write back but decline to offer an opinion, either with some excuse (busy, unqualified, uninterested, mortal enemies with one of the authors, or no excuse at all)? That’s perfectly acceptable.
What if I’m actually the advisor/mentor of one of the authors submitting the paper and don’t respond because I’m annoyed you didn’t check that before hand? Oops, sorry! An acceptable excuse but I would still prefer an email.
How long should a quick opinion take? Anywhere from a day to a month depending on exactly what you write back or where you are (I have one reviewer who always seems to get my emails while at the beach, for example.) That said, I have reviewers who always seem to write back several pages of detailed notes within 24 hours.
Should I summarize the paper? I mostly think that is not really necessary for any review. It is useful to explain what is novel or interesting about the paper, although the author should really have made an effort to do that as well.
Should I send my referee report as a pdf or a text file? I think I would prefer that, if it is a pdf file, the pdf file not contain the recommendation of the reviewer to accept the paper or not. Especially in cases in which the recommendation is not definitive, it puts me in an awkward position to reject the paper if the recommendation is generally (but not strongly) positive. So optimally send everything as a text file or send the summary as a text file and the detailed review (sans recommendation) in the pdf.
What happens if I always say no to quick opinion/referee requests? The immediate answer is that I will probably stop asking you to referee any more, which may sound like a positive to you. The down side is that the more people in your area do the same thing, the harder and harder it gets to find referees to do the job. At some point, the effect is going to be that I am just going to reject papers in that area unless they are obviously extremely strong, since it’s honestly a huge annoyance to email 10 people to get a single response, even when I am just asking for a quick opinion which could honestly take under 10 minutes. Some areas are definitely worse than others. Honestly, every time I get a paper related to the equivariant Tamagawa number conjecture I die a little, since it’s so hard to find anyone to review those papers.
How long should a review take? It would be nice if you could do it within about 3 months. At the point where I am finding people to review the paper, I am generally hopeful that the paper will be accepted. Under such circumstances, I am usually inclined to allow 6-9 months for the paper to be reviewed. I often ask referees to take a quick look early on to get a sense that they will be inclined to give a positive recommendation on the paper as well. Whether you think 6-9 months is way too long or relatively short probably depends on your own experiences. From my perspective, however, it’s such a difficult job getting people to agree to review that when I ask someone for a report in December and they say “I would be happy to, but I won’t get a chance to get to look at this paper in detail until the summer” I am almost always going to happily agree. Of course, some papers are harder to referee than others. Also, it may (and does) happen that, even after a positive quick opinion and a positive early indication from the referee the paper is rejected. The usual reason that this happens is that it is only when the reviewer looks at the paper in detail do they decide it is not as interesting as they first thought. The only way to avoid this is to only get referees who agree to write reports within 3 months. But if it takes me 6 months to find such a person, then it doesn’t really solve the problem.
What annoys you most about referee reports? Beggars can’t be choosers! But it does annoy me to wait 9 months for a review and get back a one sentence response “it is fine.” It’s honestly just embarrassing to me to have to send back such a report to the author after so long — can’t you at least find a few spelling mistakes/typos? I have received such reports on a number of occasions and then basically had to review the paper myself.
What pleases you most about referee reports? When referees time and time again selflessly put effort into helping the author make genuine improvements to their paper.
As editor, are you judge, jury, and executioner? The closer a paper is to my own expertise, the more likely it is that I have my own initial impressions of the paper. I don’t believe that my job is to be purely a functionary, and that I add value by having my own opinions. I have, on a few rare occasions, even solicited people to submit papers (and I would like to do so more frequently, although often by the time papers are on the arXiv they have already been submitted). In the end, I do believe that the process is impartial, although of course railway trains are impartial too, but if you lay down the lines for them that’s the way they go.
I’ve written a few referee reports where I do what I’m supposed to do, I write about the paper, but can’t bring myself to make a definite recommendation re acceptance or not. It’s almost always because I can’t gauge the level the journal is aiming for because the request comes with a comment by the editor that they are getting many more submissions they used to so they need to be more selective and it’s unclear to me how to interpret that. I figure it’s ultimately the editor’s decision and I’ve hopefully provided them with useful information. What do you think of that? And is it an automatic reject? It’s not meant to be.
As an author, I like it when the report is just “it is fine” and the paper is accepted (doesn’t happen very often). If it’s taken nine months, maybe I say to myself “really, that long for this?” but, on balance, I’m still happy.
I think that the only context in which a reviewer should be expected to make a definitive recommendation (outside obvious cases in either direction) is when they are given precise guidelines as to what the level of the journal is (or if they know them already). That’s part of the reason for this post — to try to make it clearer exactly what the level of Math Zeit is. As a referee, I often give my thoughts on how I would judge the paper had instead been submitted to journals X, Y, and Z that I know better, e.g. immediate reject from X, plausible for Y but maybe a stretch, clear winner for Z, etc, and then hope that is useful information.
How often do you get submissions that are so obviously bad that you don’t even ask for a quick opinion before rejection?
I’m not sure that has ever happened to me at Math Zeit, although it happened frequently when I was an associate editor at the Journal of Number Theory. I think there *have* been papers which I rejected without soliciting an opinion, but only because they were in my immediate area and so I could referee them myself rather that they were obviously bad. And even this doesn’t happen frequently (I don’t remember the last time). Possibly Math Zeit is sufficiently “botique” that it doesn’t attract super low quality papers.
Do you ever accept papers without soliciting an opinion if they are in your immediate area?
Of course not. Added: To be honest, I think I might even be tougher on papers in my immediate area. Of course even if I think a paper looks great, I want it to be refereed!
Your “similar to the level of Duke” comment reminded of all the times I had to cringe when receiving a quick opinion or a referee request from a certain uber-hopeful editor of a journal above Math Z. but below, say, Compositio claiming that the level is “just below Duke”. Some editors are simply divorced from reality when it comes to their own journals… It always seemed that such unrealistic guidelines are harmful for everyone: for the authors, who compare with other papers in the journal and not imaginary standards; for the referees, who hopefully ignore the guideline; and for the journal itself.
Pingback: Mathematische Zeitschrift (Part II: for authors) | Persiflage
Suppose I’ve already refereed (and mostly rejected) the paper for another journal, say of similar stature. Should I tell the ediotr this? In the past, I have decline to referee (and said why), but give the editor abbreviated comments of my prior ordeal.
Good question. In cases when I think the case for rejection is pretty clear, I would have no problem rejecting the paper a second (or third) time if it was sent to a similar journal, and I’ve often done exactly as you say (sending along previous comments). On the other hand, there have been occasions when I gave a mixed review and was ambivalent about whether I thought the paper should be accepted, and felt that it might be better if the authors got a “second chance” with a different referee. In this case, I’m usually just honest with the editor and and say something like “I had mixed feelings about this paper when I gave a quick opinion for X; since my opinion might not reflect how everyone thinks about this paper, I recommend you also ask other people as well.” You can massage this message in a number of directions of course, and I think it all comes down to how close to the level of the journal you think it is but also how strongly you trust your own judgement of the paper.
Have you ever had problems with the referee wanting specific questions answered (on a critical issue, say to verify correctness/importance), and the editor largely being the bottleneck to expediting such an exchange?
E.g., the linked Math Overflow thread suggests (“Every time I referee a paper…”) this is not too uncommon (and of course they propose a tech-nerd solution of an anonymous phpBB system, etc.).
Personally, I’ve never had it as an author (30+ papers), and maybe only once as a referee (when I improved an error term, and the author some time later wanted more details as to my sketch of how). So I’d have to agree with the responder (Patrick I-Z) who says nothing needs to be changed, and the current system mostly works. Maybe it’s an issue of how lax/strict author/referees are in varying mathematical fields?
https://mathoverflow.net/questions/50947/on-referee-author-communications
As an editor, my approach is as follows: the author only has a finite number of times to “satisfy” the reviewer. I know it is annoying as an author to be asked to clarify something you think is already clear, but the fear of getting the paper rejected is usually enough incentive to entice an author to make the effort.
Only once as a referee did I email the author directly using some fake account in order to sort out some details. But the situation was a little different — the result was a particularly interesting one and had been submitted to a top journal, and I was sufficiently invested in improving the exposition of the paper (and checking all the details) to make the effort to do this.
When reading this thread, the one main question which remains for me is whether there is any more that can be said about direct comparisons between the level Math Zeit aims for and the level of other well-known journals, perhaps particularly other general journals (versus subject-specific), since Math Z belongs to the general-journal category. What I mean by this is not any kind of rigid ranking, so much as some direct indication of the overall tier of Math Z relative to some well-known journals. Basically, comments of the flavor of “Duke is just shy of of JAMS/Annals, comparable to JEMS, significantly more competitive than Compositio..”
I found the percentages of acceptance and favorable quick opinions mentioned helpful to know no doubt, but alas, perhaps through my missing it, or the lack of availability of comparable info about many other journals online, I couldn’t quite infer the kinds of direct comparisons I mention.
I fully realize this post may have declined to offer such information for various reasons, but still, it is tough not to be curious. (I also realize the post is from a reasonable while ago.)
Some well known general journals that it would be interesting to compare to Math Z would include the Transactions of the AMS, International Mathematics Research Notices, Math Annalen, Selecta, Compositio…
I don’t include any journal above or below these in level, as my impression is that Math Z isn’t shooting to be outside that range, but if this is incorrect, I’d be curious.