Reflections on Reviews, Rebuttals and Respect

Having recently served as associate chair for both the CSCW 2011 and CHI 2011 Papers & Notes Committees, I've read a large number of papers, an even larger number of reviews, and a slightly smaller number of rebuttals. In participating in back-to-back committees, a few perspectives and practices that impact the process of scientific peer review have become clearer to me, and I wanted to share a few of those here. I believe all of these boil down to a matter of mutual respect among the participants, and wanted to delve more deeply into some resources that offer guidelines for respectful practices.

I want to start out with a brief review of The Four Agreements, by don Miguel Ruiz, as I believe they provide a strong foundation for how to best approach the review process, as well as other areas of life and work (and I'll include links to earlier elaborations on three of the four agreements):

1. Be Impeccable With Your Word: Speak with integrity. Say only what you mean. Avoid using the word to speak against yourself or to gossip about others. Use the power of your word in the direction of truth and love.
2. Don't Take Anything Personally: Nothing others do is because of you. What others say and do is a projection of their own reality, their own dream. When you are immune to the opinions and actions of others, you won't be the victim of needless suffering.
3. Don't Make Assumptions: Find the courage to ask questions and to express what you really want. Communicate with others as clearly as you can to avoid misunderstandings, sadness and drama. With just this one agreement, you can completely transform your life.
4. Always Do Your Best: Your best is going to change from moment to moment; it will be different when you are healthy as opposed to sick. Under any circumstance, simply do your best, and you will avoid self-judgment, self-abuse and regret.

I see examples of these agreements being violated throughout all aspects of the review process. Reviewers say hurtful things about authors, their work and/or their papers in their reviews and/or online discussions. Some reviewers appear personally offended that authors would have the audacity to submit a paper the reviewers judge to be unworthy. Many reviews reflect implicit or explicit assumptions the reviewers are making about the paper, the work described by the paper, and/or the authors who have written the paper. Some reviews are so short that I have a hard time believing that the reviewers are really doing their best in fully applying their skills and experience to help us make the best possible decision on a paper (but I acknowledge this is an assumption).

Another framework that I believe is helpful to apply in this context is nonviolent communication (NVC), which is predicated on the assumption that everything we do is an attempt to meet our human needs, that conflict sometimes arises through the miscommunication of those needs, and that further conflict can be avoided by refusing to use coercive or manipulative language that is likely to induce fear, guilt, shame, praise, blame, duty, obligation, punishment, or reward. The Wikipedia entry for nonviolent communication offers four steps (that are very similar to some earlier distinctions I'd written about between data, judgments, feelings and wants):

• making neutral observations (distinguished from interpretations/evaluations e.g. "I see that you are wearing a hat while standing in this building."),
• expressing feelings (emotions separate from reasons and interpretation e.g. "I am feeling puzzled"),
• expressing needs (deep motives e.g. "I have a need to learn about other people's motives for doing what they do") and
• making requests (clear, concrete, feasible and without an explicit or implicit demand e.g. "Please share with me, if you are willing, your reasons for wearing the hat in this building.")

Drawing on both of these sources for inspiration, ideally, a well-written review would have the following characteristics:

• Focus on the paper, vs. the underlying work or the authors. All comments address [only] what is written in the paper. They should not address the work described by the paper or the authors who have written the paper. In a blind review process, reviewers typically do not have first-hand knowledge of the work described in the paper beyond what is written; reviewers who do have first-hand knowledge should recuse themselves due to a conflict of interest (i.e., they were co-authors or collaborators on the work). Thus, any comments about the work (vs. what is written about the work) are based on assumptions.
• Follow the principles of non-violent communication (NVC). In particular, use "I" statements wherever possible, and void any direct references to the authors. For example, rather than saying "You don't say how you do X", an NVC phrasing might be something more like "It is not clear to me from the paper how X was done", or rather than saying "Why didn't you do X?", re-phrasing this as "I believe this or a future paper would be strengthened if it included X, or at least a compelling argument as to why X was not done".
• Be compassionate and generous. Assume that the authors were doing their best in composing the paper, and look for reasons to accept in addition to reasons to reject (the latter usually being more readily identified by people trained in critical thinking). I was particularly inspired by the use of generosity in the directives issued by the CHI 2011 Papers & Notes Chairs at the committee meeting. Perhaps it's the proximity to the holiday season, but I found the use of that term more resonant throughout the meetings than the more traditional (and technical) "reasons to accept" that are often promoted by chairpersons.
• Reverse the golden rule. The golden rule is "Do unto others as you would have them do unto you". A variation on this theme - which I first encountered in a book about positive psychology called How Full is Your Bucket? - is "Do unto others as they would have you do unto them." Particularly in a multi-disciplinary conference, different norms may be at work. I've had some strong disagreements with reviewers who are used to receiving terse and potentially offensive reviews, who implicitly apply the golden rule and figure if they can take it in reviews of their own papers, so should the authors whose papers they are reviewing. I always try to convince them to break the cycle of violent communication, with varying degrees of success. In a blind review process, of course, reviewers don't know the identities of the authors, and so can't really know how they would "have you do unto them". But I believe it is best to err on the side of nonviolence.

The rebuttal process also offers an opportunity for applying these practices. I won't go into as many details about the rebuttals, but I will say that if there was a category for "best rebuttal" (along the lines of "excellent reviews" and "best paper awards"), I saw two rebuttals among the papers we discussed that were outstanding exemplars of effective rebuttals. These had several factors in common:

• a heartfelt expression of gratitude for the constructive feedback provided by the reviewers (and the reviews for these submissions were excellent)
• the correct, gracious and effective identification of misinterpretations by reviewers, and a gentle articulation of the intended interpretation
• an honest acknowledgment of correctly identified errors or omissions by the reviewers, and an explicit statement of how these would be addressed in a revision (if accepted)

I also witnessed some angry rebuttals, some of which included disparaging remarks about the committee and/or the conference community, none of which had any positive influence on the ultimate decision made on those papers. I won't go into any further details, as I do not believe that would be constructive. However, I would encourage all authors to wait at least 24 hours after they recieve their reviews to even start composing their responses, as I believe this will lead to a more constructive engagement.

Due to the desire to respect confidentiality agreements, I won't disclose any specific reviews or rebuttals from the CSCW or CHI conferences as positive or negative examples, but I will conclude with a few rather extreme examples of negativity - which are so extreme they are humorous - in a blog post on Twisted Bacteria about peer review of scientific papers:

• This paper is desperate. Please reject it completely and then block the author’s email ID so they can’t use the online system in future.
• The biggest problem with this manuscript, which has nearly sucked the will to live out of me, is the terrible writing style.
• The writing and data presentation are so bad that I had to leave work and go home early and then spend time to wonder what life is about.
• The finding is not novel and the solution induces despair.

There are several more examples of violations of The Four Agreements and the principles of nonviolent communication available at Twisted Bacteria, and I'm grateful that the reviews I've seen (and written) in the CSCW and CHI communities do not reflect the extreme expressions found in this selection from the environmental microbiology community.

I hope that highlighting some of the more positive and constructive approaches one might take to peer reviewing (and rebutting) will promote a more mindful, respectful and effective process for all participants.

Comments

[Pedro Alves]

It's interesting that I've gone through the review process for CSCW2011 as an author of a paper (which was accepted btw) and completely agree with everything you said.

Even though I didn't receive reviews as extreme as those on Twisted Bacteria, I was still a bit infuriated with some remarks but I think I managed to write an objective non-emotional rebuttal, and I believe that was one the main reasons why my paper was accepted, in the end. Your suggestion on waiting at least 24h before responding is right on.

[Joe McCarthy]

Pedro: congratulations on your CSCW 2011 paper acceptance!

I'm glad that you were able to rise above the tone you perceived in the reviews you received. In reading your comment, the idea of [academic] bullying came to mind, and I suspect the unconscious cycle of bullying is at often work here: people who are beaten up often turn around and beat up someone else (or their dog). I hope you - and others who have received harsh reviews - will be able to call upon your own experience and consciously break the cycle when you write constructive, nonviolent reviews of others' papers.

[Jed Brubaker]

Perhaps I can offer a perspective from the flip side? I am a new-ish PhD student and I submitted papers to CSCW this year. Like Pedro, I am really happy that I will be going.

My reviews: Some were short, some were long, but I felt that all of them approximated the kind of approach that you are advocating. They were conscientiously written, seriously engaged with the work, and kindly pointed out more perilous moments. They were not all positive (why is it always reviewer #3?), but I felt that they were trying to help my work improve and help me grow as a researcher.

From the perspective of someone new to this community (and anticipating the legendary vitriol that surrounds reviews), the feedback I received, even the negative feedback, was so validating that I asked my advisor, "Are you sure we can't un-blind these reviews? I would really like to say thank you."

[Joe McCarthy]

Jed: thanks for sharing a positive perspective and a positive experience. I'm glad your reviews were constructive and helpful, and hope you will continue submitting to - and reviewing for - CSCW.

Your comment prompts me to offer a clarification on my original post, which may have appeared more negative than I'd intended.

I believe the vast majority of reviews in both the CSCW and CHI communities (which have considerable overlap) are constructive and helpful. And most of the rebuttals are courteous and focused on facts.

However, I always encounter a few reviews where the reviewers refer directly - and unnecessarily - to the authors in the 2nd person ("you" statements), which makes it hard for the authors to not take the feedback personally. And there are occasionally more extreme cases ... though not as extreme as seems to be the case in the field of environmental microbiology.

I wrote this post to promote greater awareness among those who may not be consistently using nonviolent communication in their reviewing or rebutting, as I believe that adopting relatively simple strategies can enhance the peer review process for all involved.

[Eric]

Good post, Joe.

This has broad applicability beyond conference paper reviews and rebuttal. These are good guidelines for reviewing corporate documents and presentations also and responding to questions about your work in general.

I think there may be more of a tendency towards this sort of snarkiness in academia. I experienced some of it myself at CSCW 2004. If someone provided some of the more extreme sort of comments in a corporate environment, they'd soon be unemployed.

[Joe McCarthy]

Eric: thanks for broadening the discussion to include the corporate environment. I suspect the anonymity of the blind review process may explain some of the harshness I see in some reviews. In my experience of corporate environments, reviews of work (or worker performance) are rarely anonymous. I will say, though, that I believe that corporate research environments would probably benefit from opening up more to peer review (vs. reviews by [org chart] superiors).

Your comment prompts me to wonder about whether non-anonymous criticism is harsher in academic vs. corporate environments, and if so, why. Critical thinking is a skill actively cultivated in the best universities and the best companies, but it does sometimes seem that the expression of criticism can be harsher in the former than the latter. Perhaps it has to do with the currencies in the two environments, i.e., the marketplace of ideas vs. the marketplace of products and services.

[Eric]

One can clearly can give direct and pointed feedback without being a jerk.

While the degree to which this skill is valued depends on the particular corporate culture, almost all corporations base at least a portion of compensation on group success. Your incentive is to help others succeed, so you'll give helpful feedback that doesn't alienate others.

The same interest doesn't necessarily exist in academia, especially if you are tenured.