Reviewing Standards

All research track chairs and their committee members agree to abide by the following rules, which they electronically agree to when signing up for the role.

1. PERSONALLY REVIEW PAPERS YOU ARE ASSIGNED

You will read and write a review for every paper you are assigned to review. You may solicit additional reviews from trusted colleagues and submit them or integrate them into your review, but such reviews may not replace your own review.

The due date for submitting reviews will be [DATE]. The bulk of the reviewing will be done in the period between [PERIOD]. You should expect to review [NUMBER] papers, although if we get an unexpectedly large number of papers we will have to ask you to review more.

2. PARTICIPATE IN A COLLABORATIVE DECISION MAKING PROCESS

After all individual reviews have been collected, we need to come to a conclusion on each submitted paper. Most will be easy to decide on, but for some we need the combined intelligence of all reviewers assigned to a paper in debating its merits. The debate will be carried out online and if necessary, in a telephone conference. The day of the telephone conference is [DATE]. Please pencil it into your calendar now. Please accept this invitation only if you are committed to contributing to this debate.

The emphasis on this debate reflects our definition of a program committee, which is the group of people who discuss the reviews and reach consensus on acceptance and rejection decisions of submitted papers.

3. ABIDE BY CONFLICT OF INTEREST RULES

Throughout the reviewing and decision process we will strictly abide by the following rules on conflicts of interest: You are considered to have a conflict of interest on a paper that has an author or co-author in any of the following categories:

  1. yourself,
  2. your past and current graduate students,
  3. your graduate advisors,
  4. members of your research group within the last 5 years,
  5. a co-author of a paper submitted for publication within the last 5 years,
  6. an employee of your immediate organization (academic department, research lab unit, etc.) within the last 5 years,
  7. someone with whom you have had a significant funding or financial relationship within the last 5 years, and
  8. a member of your family, a close personal friend, or anyone else whose papers you feel you cannot evaluate objectively.

You will not be allowed to see the reviews of papers on which you have a conflict. The research track chair will be able to see all reviews for all papers, but s/he will assign a temporary chair for discussion of papers on which they have a conflict).

4. PARTICIPATE IN THE PAPER BIDDING PROCESS

You will provide your areas of expertise and interest, read abstracts of all technical papers submitted to the conference and bid on the abstracts, indicating papers you would like to review and papers on which you have a conflict of interest.

We apologize in advance if you don’t get all the papers you bid for—we will use bids as a key criteria in assigning papers. But our primary goal will be to ensure that each paper is read by appropriate experts.

5. SUBMISSIONS BY PROGRAM COMMITTEE MEMBERS

The research track chair will not submit papers to the conference. Research track committee members, however, are welcome to submit papers.

We will handle research track committee member papers according to this rule of thumb: “Research track committee papers should be accepted only if they are of above average quality”.

In practice, this means two things:

  • Research track member papers are rejected automatically if there is strong objection from one of the reviewers,
  • we will determine what exactly means to be “above average” quality once we have an understanding of what is the average given the reviews for the submissions we will get.

We will also assign more reviewers to submissions of research track committee members.


Adapted from James Noble’s committee rules for ECOOP 2012 who adapted them from those who came before him.

3 thoughts on “Reviewing Standards

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.