CrowdGrader Grading in Detail

We assume here that you have chosen consensus-based grading.

Submission Grade

No rubric. If a rubric is not used, then the submission grade is simply the median of the grades provided by the individual reviewers. 

Rubric.  If a rubric is used, the submission grade is computed in two steps:

• • CrowdGrader computes the consensus grade for each rubric entry (such as, "Is the code well documented?") as the median of the grades provided by the individual reviewers.

  • The submission grade is computed as the sum of the consensus grades for the rubric entries.

Review Grade

No rubric.  For an assignment graded out of p total poiints, if the submission grade is c and the student has given a grade of g, then the student receives a review grade of p - abs(c - g).

Rubric.  

• • In a rubric entry worth p total points, if the consensus grade is c and the student has given a grade of g, the student receives p - abs(c - g) precision points.  

• The review grade is the sum of the precision points for all rubric entries. 

The review grade thus measure the agreement between the student grades and the consensus grades. 

Crowd-Grade

The Crowd-Grade received by each student is computed by merging submission and review grades according to the proportions specified in the assignment (a typical proportion is 80% submission grade and 20% review grade).

Example

All this is best explained by an example.  Let us consider a submission graded by students A, B, C, D, E, on the basis of a rubric with 4 entries, each worth 5 points.  The following spreadsheet indicates how the consensus grade, and the submission grade, are computed.  The spreadsheet also indicates how the precisions for the students on this assignment are computed. 

The following spreadsheet illustrates how overall review grades are computed, and how they are combined with the submission grade of each student (which would be computed in a manner similar to the above spreadsheet) to yield the overall crowd-grade for the student.  The crowd-grade can be used as overall grade for the student in the assignment.