Calibrated Peer Review Assignment
This assignment is designed to give you exposure to a single topic in geology in somewhat more depth. I think this topic (The Great American Interchange) is a very interesting one, and I hope you will, too. The nature of this exercise will be new to you, and perhaps a little strange. Please read this document for a brief introduction, and instructions.
Introduction & Rationale
Calibrated Peer Review, or "CPR" is a way for you to learn about writing and a specific topic of interest. Without it, a small writing assignment would be impossible for me to grade effectively. This will allow you to delve more deeply into a small geologic topic, and spend some time working on your writing skills. I've chosen the topic because I think it's very interesting and other students have in the past as well.
Instructions
- Read this document.
- Log in to the CPR website.
- Take the CPR tour, and answer the CPR pre-test questions. You cannot progress further until you pass this pre-test. (This ensures that you have at least a rudimentary understanding of how CPR works). If you have trouble with this step, check out the guide.
- Ask questions (use the discussion board) if there are parts of the CPR process you still don't understand
- Read the background material. These are your primary source for writing your essay.
- Read the writing guidelines.
- Write a brief essay on our topic and submit it to the website.
- I would advise you to compose your essay in a word processor, and then copy and paste it into the CPR window. This allows you to use spell-checking.
- The essay should be in HTML, which is basically just text. In HTML, paragraphs are not created by hitting [Return], because spaces and line breaks are ignored. You must type in one of these symbols: <p> or <br>. (<p> gives space between the paragraphs, <br> just works like a [Return]). However, please do not use "Save as HTML" from MS Word.
- Wait for everyone else to submit their essays. You can revise your essay until the end of the text entry period.
- Review three calibration essays to determine your calibration score and your Reviewer Compentency Index (RCI). This entails both answers to specific questions about both the content and the style, as well as a final rating between 1 and 10.
- Review three essays written by classmates to determine their text scores and your reviewer score.
- Review your own essay to determine your self-assessment score.
- Once the reviewing period is over, log onto CPR to check your score. This score may be altered by changes I make. (I review some of the essays myself, and if when I rate an essay that rating replaces the weighted average, so it may change your reviewing score).
How CPR is scored
CPR is Calibrated Peer Review, and what this means is that your score is calculated on the basis of reviews by your peers. There are four sections, each scored separately. Here are some brief notes on how each score is calculated:
Calibration score (10%)
After Phase One (text entry) is complete, you will be asked to read and review three calibration essays. These are essays that I have reviewed already and for which I have entered correct answers to the specific review questions as well as a final rating. For each essay you have two tries to come within a certain range of the instructor's answers and rating. If you do this, you will get full credit for that calibration essay (1/3 of the calibration score). If not, you will get zero for that calibration. In addition, you will get a RCI (Reviewer Competency Index) rating from 0-6, indicating how good a reviewer you are. This rating is not directly linked to your calibration score.Reviewer score (25%)
You will then review three essays written by your peers, just as with the calibration essays. Each paper is reviewed by three students in total. Your score as a reviewer is based on how well your numerical rating agrees with the ratings provided by your peers for each paper you review. You get full credit for being within a certain range of the weighted average score (see below), and half credit for being within a larger range for each score. Each paper is worth 1/3 of the points for the Reviewing section of your CPR score, so it is possible to get full credit on two papers and no credit on the third, which yould give you 67% of the possible points for reviewing.
Text score (50%)
Each paper is reviewed by three students. Your text rating is the average of your reviewers' ratings of your essay, weighted by their RCI. This means that the rating from a reviewer who did a good job on the calibration task counts more than the rating from one who did a poor job on the calibrations. The mean score is multiplied by 10 to get the percentage of the text points you receive as your score. So if there are 40 text points possible, and the weighted average of the ratings you received was 7.5, then you would get 75% of the possible points, or 30.00.Self-assessment score (15%)
Your self-assessment score is similar to your reviewer score. It is calculated based on how closely you approach the weighted average rating of your own essay. It is not an opportunity for you to argue or justify your essay, but a learning opportunity. Just as with the reviewer score, you receive full credit for being within a certain range of the weighted average text rating, and half credit for being within a larger range.
How to rate the papers
The most subjective part of rating an essay in CPR is the final numerical rating. Unfortunately, this is one of the most important parts as well - it determines your score for reviewing and the score you receive for your text. You should think of a rating of one as being the worst essay you could imagine a classmate possibly turning in, and ten as the best essay possible for a classmate of yours. (As opposed to a one being the essay a second-grader might write and a ten being the essay an expert geologist might write). Part of the purpose for the calibration work is to give you an idea of how to apply these scores in practice.
Frequently Asked Questions
- Why are we doing a writing assignment in a science class?
- Despite what you may have learned in other courses, writing is not only for the humanities and social sciences. All disciplines require writing, science no less than others, and more than some. Communication is key to all disciplines, and until telepathy comes along, writing is the best form we've got.
- Why are we graded on spelling and grammar in a science class, instead of just content?
- Effective communication requires clarity, and that is provided by the standards of spelling, grammar, and punctuation that you should have learned in your previous education. A poorly spelled document is an ineffective one. This is as true in scientific writing as it is in literature.
- Why can't we just have a regular writing assignment, and have you grade them all?
- If I had a phalanx of TAs to grade such writing, we would. Sadly, I do not. Although it may be news to you, every instructor has responsibilities other than the course you are in, including (but not limited to) other courses and research. The CPR system is a non-traditional, but tested way to correctly score student writing.
- If you believe your score is in error somehow, let me know and I will evaluate it myself. However, I may grade more harshly than your peers did.
- Isn't it unfair to be graded on other people's opinion of our work?
- Your peers naturally vary in how good they are at grading your work. This variation is taken into account by the system (see text scoring section, above). In this way, a fair score is achieved. In previous CPR assignments, the scores assigned have rarely been lower than those I would assign myself.
- It seems like one of my reviewers didn't even read my paper!
- If this is the case, you should bring it to my attention.
- Can't people just put whatever they want for their reviews, without even thinking about them?
- They can, but chances are good that they will get a low score for the reviewing section of CPR, because the other reviewers will have been careful about their reviews and will probably provide a different rating.
- Isn't it unfair to be graded on how well we can read an essay and answer questions about it?
- No. This type of critical thinking review is a higher-level task than merely writing an essay. Completing a careful review of an essay requires not only a thorough understanding of the material, but also a good understanding of what makes up good writing. This type of competency is part and parcel of good science writing. Furthermore, critical reading of science writing may serve most of you better than the writing itself: most of you will not become scientists, but all of you will read about science. It would be wise to learn to read carefully and critically.
David M. Hirsch
Modified on Mon, Jan 31, 2005 at 3:43 PM