Introduction
Efficienctly drafting useful reviews and properly writing rebuttals can be incredibly difficult and needs quite a bit of practice. On top there may be time and space constraints making things more difficult — reviewing 5+ papers in a few weeks can be challenging on top of day-to-day work and writing a 1-page rebuttal in a week while coordinating collaborations is clearly stressful. This makes it even more surprising that many PhD students never properly learn how to do this. However, there are a few best practices I picked up over the years that I want to share.
Before getting started, a few words on whether to do reviews in the first place. Throughout a typical PhD it takes a while until you are able to review for most of the top-tier conferences. Often they require that you published a paper before. Here, it makes sense to accept most reviewing requests, including lower tier conferences or journals. This is to get some practice. Also, organizers will remember you and ask you to review for the same or other conferences in the future. Towards the end of the PhD, your reviewing capabilities usually become quite desirable as you have experience but your burden in terms of academic service is still low. This means you will get more and more requests to review. At this point it makes sense to become more strategic. First, it makes sense not to review for too many overlapping conferences. Quality of reviews will suffer and you will feel less productive with your actual research. Then, you should manage expectations. Accepting reviewing duties for a journal is OK, but if the timing is bad, check how flexible the timeslines are. Having reviewed for way too many venues the last few years, I would say that reviewing for 3-5 venues a year should be the maximum. Nowadays I actually feel that submissions get more and more noisy and I personally reduced it even more — but this is the topic of a separate article.
Writing reviews
Reviewing essentially involves three key steps that need to be tackled within certain time constraints and sometimes are forced into specific formats or processes:
- Reading a paper and taking notes
- Writing the review
- Making a recommendation
The first step sounds easy but also offers the most opportunity to streamline the whole process — in order to write a good review and make it as efficient as possible. Of course, you want to read the full paper carefully. For me, this is still prerequisite for a good review — no LLM or tool can replace this fully. However, you want to avoid having to re-read the paper over and over again. For this, I recommend annotating at least two things while reading: questions and high-level structure. Questions will pop up naturally and I usually write down all of them — even if I think some of them might be answered later. They can be questions on notation, structure, theory, experiments, or really anything. Sometimes I also just make a note that something is confusing even though I do not understand why exactly. Regarding structure, I usually mark the main components of a paper: list of contributions, description of main method, key theoretical results, experimental setup and main experimental results. Ideally, of course, a paper should be structured in a way for you to pick these things up easily. Unfortunately, many papers are not structured well or structured in a way that is not intuitive to you. Immediately after reading the paper, I also note down a 1-sentence summary and an initial impression. The former just tests whether I understood the gist of the paper; the latter is usually pretty short and just says that I like the method, or that the experiments are convincing or that this is very close to an already published paper. It’s a gut feeling in the end.
This gut feeling after the first read also is a good indicator how much time to invest in the remaining review. Essentially, I learned that reviews for papers that are clear accepts or clear rejects can be rather short as there are only a few key reasons for that decision. Instead, papers you are unsure about or that could be an accept after some changes, might require a more thorough review in order to come to a confident decision.
Writing the review is essentially about condensing the notes into a proper review and structuring it in a way that is useful for the authors and follows the venues format. Personally, possibly biased by venues such as CVPR, I tend to structure all my reviews as follows:
Summary: ... Strengths: [bullet points] ... Weakness and questions: [bullet points] ... Conclusion and rating (if applicable): ...
Some venues may require to split weaknesses and questions, some others only have a single long-text field for everything, some prefer to have a separate summary. This template covers all of them.
The summary serves the main purpose of checking whether you understood the paper well enough to condense the contributions to 2-3 sentences. This should not include opinions yet and just summarize the paper as presented by the authors. If you feel this is difficult and you need to double check the abstract and copy&paste parts of the abstract, you have not understood the paper. You might need to re-read it.
Strengths should be at least a few bullet points on what is good about the paper. Here it makes sense to clarify the objective of reviewing. For most venues, the objective of reviewing is to figure out whether the paper is supposed to be published in that venue. Most venues usually define their criteria somewhere. does this very explicitly; for many other venues it is more implicit in the reviewing guidelines and — in my opinion — not all communities do a good job in teaching this to their reviewers. So here is my condensed version for most top-tier AI conferences: you are supposed to decide whether a paper reports new knowledge that some readers will find interesting. This is admittedly very broad. Often, it is translated as reporting novel and significant results. Personally, however, I prefer to take the view of a potential reader and ask myself whether I or my research would benefit if this paper would be published. This is a bit more general because for a paper to be beneficial for the problem I work on, it does not necessarily require new or significantly better results. It could also include a new view on existing results or do a more extensive ablation of existing experiments that saves me time. If the paper does provide new knowledge, you should highlight this in the strengths section and mention why you feel this is an interesting and important finding.
Beyond this key question of whether a paper should be published, I also tend to give feedback on secondary aspects. This generally includes writing and experiments as well as method for methodological papers, theory/analysis for more theoretical papers. I tend to only highlight writing if I found something particularly nicely written. For example, this could be a really good related work section, or a well structured ablation. For novel methods, I tend to highlight nice properties, including better quanitative or qualitative results, simplicity, low runtime, etc. In terms of experiments, I tend to highlight good ablations, use of multiple datasets, inclusion of particularly interesting baselines, etc.
The weaknesses section, covers the same parts as above, assessing whether the paper reports new knowledge, writing, method/theory, and experiments. In most cases, I try to combine these weaknesses with questions or suggestions hinting in how these weaknesses could be addressed in my opinion. This is generally not your job as a reviewer — your responsibility is not to make the paper better —, but I learned that this way of thinking adds a more positive tone to listing a bunch of weaknesses. After all, every author likes to receive a kind review. Moreover, as a reviewer, it is more likely than not that you are simply wrong. Ideally, papers you get assigned are in your core area of expertise. However, it is very likely that the authors actually obtained more understanding about their work from writing the paper than you can obtain from reading the paper. By acknowleding that you might have missed something, this usually leads to a more constructive reviewer-author discussion.
Finally, the weaknesses section will also include questions on everything I did not understand. Often, questions can be related to poor writing or presentation, unclear equations or experiments. In these cases, this is worth pointing out because it helps the authors understand where you are coming from. You can also highlight mistakes in the text or equations. But I generally ignore these, assuming that these are honest mistakes and will be gone after another pass over the paper when submitting the camera ready.
The conclusion will be 2-3 sentences in which you justify your rating. I like to think about this as the 2-3 arguments you would give the AC alongside your recommendation. I tend to highlight the main strengths and weaknesses that led to my rating. This requires prioritizing. For example, any issues with writing will usually not impact my score and will therefore not be highlighted in the conclusion. At venues where authors can update their paper throughout the reviewing process, the conclusion should also indicate the main points that need to be addressed in order to increase your rating.
Discussion
The discussion phase depends mostly on the venue. Many conferences do not have an interactive discussion phase that allows some back and forth. Some journals allow this in the form of multiple iterations of review and rebuttal. For venues that allow an interactive discussion, I recommend checking the author’s rebuttal as soon as it is available and at least acknowledging their rebuttal. You can also follow-up with additional questions, but acknowledging their effort is a good start. Typically, I also include a short statement towards the end of the discussion whether I changed my opinion or not.
As an author, there is a trade-off: you can respond fairly early, giving you less time to polish your response and provide additional results but leaving more time for a discussion; or you can wait, share a more polished rebuttal but have less time left for a discussion. There is probably not a perfect approach but I generally prefer to split the rebuttal. Responding to most points as soon as possible while still having a more polished response (needing maybe 2-3 days) and then offering additional results or responses that you add later on. This allows to get a discussion started (depends strongly on the reviewers’ mood for a discussion) while still having a good rebuttal, potentially include additional results.
Responding to reviews
As with writing a review, the first step for preparing a rebuttal is actually reading and understanding the reviews carefully. Make sure to read the full review — not only the questions or weaknesses and make sure to also check the rating and any other information that the reviewer provided (some venues have additional multiple-choice questions). Each review should be assessed through the lens of the rating. Weaknesses can easily sound negative; but if the rating is positive, the weaknesses may sound less harsh.
Before thinking about a response, the most important rule to keep in mind is that the reviewer is almost always right. Of course this does not mean that the reviewers’ ratings are correct or the reasons given have to make sense. But it means that there is always some reason for the reviewer’s opinion. For example, if the reviewer leaves you the impression that they did not understand your paper, it might be due to your presentation and writing not being optimal. If they do not see the problem you tackle as important, maybe you have not given a good intuition or enough evidence why the problem is important. If the results are not seen as significant, maybe you are highlighting the wrong metrics or results on the wrong dataset. You see, the main reason of assuming that the reviewer is always right prompts you to search for the underlying reason of the reviewer’s opinions. Moreover, it makes you think about things that are under your control (your writing, your experiments, etc.).
Because the reviewer is always right, you should be nice. That does not mean you have to agree with the reviewer, but you should appreciate the reviewer’s effort and time. I tend to start rebuttals with thanking all reviewers and highlight some of the strengths highlighted by the reviewers. The latter has the intention to make all reviewers notice these strengths because not all reviewers actually read the other reviews. For example, this can read as
We thank all reviewers [and the editor, in case of a journal] for providing constructive feedback. All reviewers [agree that our paper tackles an important problem|mention that our method improves over state-of-the-art results|...].
After this first paragraph, I tend to follow-up with one or multiple paragraphs addressing common questions/weaknesses raised by all reviewers. Essentially, these are meant to be read by all reviewers and the AC/editor. Then, I include a separate section for each reviewer. I usually assume that each reviewer only reads the section corresponding to their review.
In terms of responding to individual comments, I like to distinguish three settings: a factual question, an opinion that I can agree with or an opinion I clearly disagree with. I tend to err on the side of agreement rather than disagreement. Here is how I go about these:
For a factual question, I generally try to make a factual responds as concise as possible while answering the full question. For example, if there is a question about how a specific aspect of your method works, you would try to explain this (in different words than used in the paper) as concisely as possible. If additional results or clarifications are asked for, include them. For these comments, you might not need any sugar coating or anything. Just start with the answer, do not be sorry for it, do not include any justification or anything.
If a reviewer shares an opinion, for example that some part of your paper is confusing, your experiments are not significant or your method is not novel, you have to decide to either agree or not agree. Generally, it is easier to agree in the spirit that the reviewer is likely right. For example, you can say “We agree with the reviwer that Section X can be tricky to follow.” Then you follow this up with an actual response. If you do not agree, you should also mention this; like “We disgree with reviewer X’s statement that YYY.” This is important to show the other reviewers’ as well as the AC/editor that there may be something wrong with the reviewer’s assessment. Again, you follow things up with a response. In both cases, the response should be supported by references or facts and potentially include an action item, i.e., what you will do with the paper to address the comment. If you are really constrained by space, you might not explicitly agree, disagree or mention action items. But they should always be included implicitly in the response.
Let us try some examples (adapted from real rebuttals):
- Factual queries:
Responding to a factual question:
Q: Claim X needs a statistical test. A: Correlation coefficients are X with p-values < 1e-3.
Q: Are you claiming that X already improves metric Y? Can you comment on why this observation is different from [A]? A: In line with [A], we only argue that X *can* improve Y, but does not necessarily result in better Z. In fact, Fig. X in [A] aligns with our results in Table Y. This is because ...
Often, these questions will also ask for additional results. In all cases, the response should just respond to the question, without any fuzz and be fairly concise and to the point (which requires correctly understanding the question).
-
Disagreeing with a reviewer’s opinion is difficult and probably requires the most effort to make a convincing argument. Note that the argument is not only for the reviewer, but also for all other reviewers as well as the editor/AC:
Q: Technical contribution and novelty are limited. Though this paper has some interesting findings, the designed metrics are straightforward and the ideas are borrowed from existing works. A: [Briefly restate core contribution:] We present an extensive empirical study relating X to Y. Thereby, we are the first to provide Z. While we build on top of previous work, we do believe that our work provides sufficient novelty [i.e., we disagree with the reviewer]. In contrast to [A], we ... Thereby, we also address important limitations of [B] ... Overall, we are convinced that our paper makes an important step on top of [A,B].
Q: This paper starts from a intriguing application of X. The challenges of Y are valid. However, the method of using Z are not novel. Very similar ideas and implementations can be found in the following references: [...] A: We appreciate that the reviewer recognizes the practical relevance of our work as well as the additional references. Based on the provided references [A, B, C], we suspect that there might be a slight misunderstanding that we want to resolve: ...
-
Agreeing with a reviewer is usually easier. Sometimes you might agree and then provide a different view on the question. And you should usually include some action items for the camera ready, or already have adapted the paper (if possible):
Q: this paper is recommended to reduce the length [...] A: We agree that the paper inludes a large body of experiments. We followed the reviewers suggestion and reduced the overall content of the paper by ...
It is important to realize that writing rebuttals is to some extent a numbers game. You will likely not be able to correctly understand all reviewers. This means some of your responses will be best guesses.
A final note is on prioritization. For most venues, you will be unable to address all comments in detail. You have to prioritize. In my opinion, a good rule of thumb is as follows: you want to put more weight on comments from negative reviewers and generally prioritize addressing concernsregarding the core of your paper — how your paper adds new knowledge. The latter may refer to novelty or significance of results. Essentially, this means you want to address any doubts regarding your method or experiments before addressing questions on individual equations, hyper-parameters or writing.
