Paper Review & Turnin
For a paper you are assigned to review (Canvas > Home > Paper Review > paper_assignment), you should complete this form. Please make a copy of this document, fill in the information, and and make sure that the final version is uploaded here. This folder is visible to other students. This simulates real reviewing where your reviews will be visible by other reviews, (senior) area chairs, and program chairs. After the deadline, I recommend looking at other reviews and checking your work.
Before you start writing the review, carefuly go over this tutorial for more details and useful tips. You will see that the tutorial suggest to NOT do the following:
- The authors could also do [extra experiment X]
- The authors should have done X instead
For the purpose of this class, feel free to make a note of additional experiments or a different framing in a special section titled “Additional comments for the class”.
Grading
- Summary (5 points)
- Strengths (5 points)
- Weaknesses (5 points)
- Comments (2 points)
- Limitations and Societal Impact (2 points)
- Ethical Concerns (1 point)
Some resources to help you write better reviews
- Vivek Srikumar’s tips
- Rogers and Augenstein. How to review for ACL Rolling Review.
- Rogers and Augenstein. What Can We Do to Improve Peer Review in NLP? EMNLP Findings 2020.
- EACL 2021 Tutorial “Reviewing NLP Research”
- NLP Highlights Podcast: On Writing Quality Peer Reviews, with Noah A. Smith
- Two example good reviews from NAACL 2018 presented in their reviewing form
- Discursive advice in ACL 2017 from leading lights in the field: Mirella Lapata, Marco Baroni, Yoav Artzi, Emily Bender, Joel Tetreault, Ani Nenkova, and Tim Baldwin
- Advice on Reviewing for EMNLP 2020
- If you can’t be kind in peer review, be neutral
- Mark Allman. 2008. Thoughts on reviewing. ACM SIGCOMM Computer Communication Review, 38(2):47–50.
- Graham Cormode. 2009. How not to review a paper: The tools and techniques of the adversarial reviewer. ACM SIGMOD Record, 37(4):100–104.
- Alan Jay Smith. 1990. The task of the referee. IEEE Computer, 23(4):65–71.