Calibrated Peer Review Assignment

This assignment is designed to give you exposure to a single topic in geology in somewhat more depth. I think this topic (The Great American Interchange) is a very interesting one, and I hope you will, too. The nature of this exercise will be new to you, and perhaps a little strange. Please read this document for a brief introduction, and instructions.

Introduction & Rationale

Calibrated Peer Review, or "CPR" is a way for you to learn about writing and a specific topic of interest. Without it, a small writing assignment would be impossible for me to grade effectively. This will allow you to delve more deeply into a small geologic topic, and spend some time working on your writing skills. I've chosen the topic because I think it's very interesting and other students have in the past as well.

Instructions

  1. Read this document.
  2. Log in to the CPR website.
  3. Take the CPR tour, and answer the CPR pre-test questions. You cannot progress further until you pass this pre-test. (This ensures that you have at least a rudimentary understanding of how CPR works). If you have trouble with this step, check out the guide.
  4. Ask questions (use the discussion board) if there are parts of the CPR process you still don't understand
  5. Read the background material. These are your primary source for writing your essay.
  6. Read the writing guidelines.
  7. Write a brief essay on our topic and submit it to the website.
  8. Wait for everyone else to submit their essays. You can revise your essay until the end of the text entry period.
  9. Review three calibration essays to determine your calibration score and your Reviewer Compentency Index (RCI). This entails both answers to specific questions about both the content and the style, as well as a final rating between 1 and 10.
  10. Review three essays written by classmates to determine their text scores and your reviewer score.
  11. Review your own essay to determine your self-assessment score.
  12. Once the reviewing period is over, log onto CPR to check your score. This score may be altered by changes I make. (I review some of the essays myself, and if when I rate an essay that rating replaces the weighted average, so it may change your reviewing score).

How CPR is scored

CPR is Calibrated Peer Review, and what this means is that your score is calculated on the basis of reviews by your peers. There are four sections, each scored separately. Here are some brief notes on how each score is calculated:

Calibration score (10%)

After Phase One (text entry) is complete, you will be asked to read and review three calibration essays. These are essays that I have reviewed already and for which I have entered correct answers to the specific review questions as well as a final rating. For each essay you have two tries to come within a certain range of the instructor's answers and rating. If you do this, you will get full credit for that calibration essay (1/3 of the calibration score). If not, you will get zero for that calibration. In addition, you will get a RCI (Reviewer Competency Index) rating from 0-6, indicating how good a reviewer you are. This rating is not directly linked to your calibration score.

Reviewer score (25%)

You will then review three essays written by your peers, just as with the calibration essays. Each paper is reviewed by three students in total. Your score as a reviewer is based on how well your numerical rating agrees with the ratings provided by your peers for each paper you review. You get full credit for being within a certain range of the weighted average score (see below), and half credit for being within a larger range for each score. Each paper is worth 1/3 of the points for the Reviewing section of your CPR score, so it is possible to get full credit on two papers and no credit on the third, which yould give you 67% of the possible points for reviewing.

Text score (50%)

Each paper is reviewed by three students. Your text rating is the average of your reviewers' ratings of your essay, weighted by their RCI. This means that the rating from a reviewer who did a good job on the calibration task counts more than the rating from one who did a poor job on the calibrations. The mean score is multiplied by 10 to get the percentage of the text points you receive as your score. So if there are 40 text points possible, and the weighted average of the ratings you received was 7.5, then you would get 75% of the possible points, or 30.00.

Self-assessment score (15%)

Your self-assessment score is similar to your reviewer score. It is calculated based on how closely you approach the weighted average rating of your own essay. It is not an opportunity for you to argue or justify your essay, but a learning opportunity. Just as with the reviewer score, you receive full credit for being within a certain range of the weighted average text rating, and half credit for being within a larger range.

How to rate the papers

The most subjective part of rating an essay in CPR is the final numerical rating. Unfortunately, this is one of the most important parts as well - it determines your score for reviewing and the score you receive for your text. You should think of a rating of one as being the worst essay you could imagine a classmate possibly turning in, and ten as the best essay possible for a classmate of yours. (As opposed to a one being the essay a second-grader might write and a ten being the essay an expert geologist might write). Part of the purpose for the calibration work is to give you an idea of how to apply these scores in practice.

Frequently Asked Questions

  1. Why are we doing a writing assignment in a science class?
  2. Why are we graded on spelling and grammar in a science class, instead of just content?
  3. Why can't we just have a regular writing assignment, and have you grade them all?
  4. Isn't it unfair to be graded on other people's opinion of our work?
  5. It seems like one of my reviewers didn't even read my paper!
  6. Can't people just put whatever they want for their reviews, without even thinking about them?
  7. Isn't it unfair to be graded on how well we can read an essay and answer questions about it?

David M. Hirsch
Modified on Mon, Jan 31, 2005 at 3:43 PM