Abstract:

Quasi-Arithmetic Scoring Theory (QuAST) grew out of a desire to justify peer assessment (PA) as a meaningful educational measurement technique. PA is used to evaluate group work and other collaborative learning activities. Students evaluate each other’s participation in joint (project) assignments. The evaluations are used to split an overall team score t into individual student scores to inform student grading. So far, formal treatment of PA methodology has been lacking. QuAST puts judgmental scales in edumetrics on a sound mathematical foundation. Such scales may be viewed as bounded scales, different from the scale types proposed by Stevens. PA is modelled by bivariate module theory of bounded scales.

A bounded scale is a mathematical structure called module with addition and scalar multiplication as its operations. Modules have the right structure to define educational metrics and statistics (scores, ratings, marks, grades, etc.). Addition and scalar multiplication of scores behave similarly as their arithmetic counterparts. The percentage scale [0,1] plays the role of standard bounded scale. Properties valid for the standard scale hold for all bounded scales.

Modules of bounded scales:
(1) A bounded scale is an interval [min,max] of scores s such that min<=s<= max
(2) Bounded scales are closed under addition and scalar multiplication of scores
(3) Operations obey the properties of associativity, commutativity, and distributivity of scalar multiplication
(4) Bounded scales have neutral elements wrt addition (zero) and scalar multiplication (unit). Inverses exist
(5) Scores may be multidimensional composites, ie consisting of subscores (items or criteria)
(6) The mean and variance of multiple scores or subscores are well-defined
(7) A bounded scale can be transformed to another bounded scale preserving its structure.

Bivariate modules for PA. To fully model PA QuAST needs to be extended with bivariate modules: a bounded scale of peer rating acts upon another bounded scale of student scores. The assessor sets a group score t as the default student score. Also, he specifies a scoring rule to define the action of peer ratings on student scores. There exist three distinct types of scoring rules.

Features:
(1) Both rating and scoring scales of a scoring rule may be freely chosen by the assessor
(2) The action of peer ratings is such that scores are constrained to a subscale around t so that unrealistic (too large) deviations from t can be avoided. For this, a tolerance parameter z>=1 is specified. The subrange takes the form: t^z<=s<=1–(1–t)^z. If z=1, PA is disabled (s=t). If z=2 (default) then s lies in a symmetric range around the group score: t ± t*(1–t)
(3) The action of peer ratings may be more or less pronounced, as specified by an impact parameter p. Default p is 1. A p of 0 implies that peer ratings have no influence
(4) The action of ratings on scores depends upon whether one assumes peer ratings and group score are related or not. If they are, e.g. because the group score is disclosed to the students, they are merged with t before being used to modify t; otherwise, un-modified peer ratings are used
(5) Unmodified peer ratings yield linear (p=1) or quasi-linear (p≠1) scoring rules. Modified peer ratings lead to weak or strong non-linear but still plausible models
(6) The scoring function may be forced to obey the Split-Join-Invariance principle: the quasi-arithmetic mean of calculated student scores equals the group score.

Keywords:

Edumetrics, percentage scale, group projects, peer assessment, scoring, rating, scoring functions, module theory, split-join-invariance, scoring models.