Abstract:

Human judgments are biased and noisy. Bias is a systematic error that invites a causal explanation. It leads to judgments that deviate in a predictable direction from the correct value. [1] Noise means a random scattering of judgments. [2] This paper addresses the serious problems of bias and noise in the assessment of scholarly articles as well as of students in schools and universities.

To identify bias, one must know the correct value that was missed. Noise, on the other hand, can be identified even if we cannot tell where the true value lies. [2] In the case of essays in subjects like philosophy or humanities in general, there is no objective method for determining the ‘correct’ evaluation. However, as long as there are more or less appropriate judgments, noise is an obvious source of error that should be eliminated.

In mathematical statistics, the mean square error (MSE) indicates how much judgments deviate from a real or imagined ideal value. The MSE decreases as bias or noise decreases. So, reducing noise is worthwhile regardless of whether bias is also avoidable. [2]

The generic term for noise within a system is system noise. [2] If within a department, the grade point average of professor A is constantly higher than that of professor B, this is called level noise. If both professors have a similar grade point average, but professor A evaluates certain traits more positively or weights them more heavily than professor B, this is called pattern noise. Pattern noise can be stable or depending on the occasion. If a professor always evaluates cases with identical properties in the same way, this is called stable pattern noise. If she evaluates the same kind of case differently on different occasions, this is called occasion noise.

Peer assessment may reduce level noise if different experts with an outside view evaluate the same text independently. It does, however, not reduce pattern noise, as long as experts judge intuitively, without using a case scale and strict protocols. [3] It is easier to implement the latter in teacher training. But often, institutions lack the time and money to evaluate the same text more than once by different teachers, tutors, or professors.

The problem of noise is easily overlooked at one-time evaluations. [2] That is why it is highly underestimated in educational assessments. However, the uniqueness of an assessment does not remedy its possible inaccuracy; it only makes it more difficult to detect the inaccuracy.

We need to collect empirical data that can be used to uncover the true amount of noise in our educational institutions. My paper shows how to conduct such a noise audit, the extent of noise that I identified among prospective ethics teachers at my home university, and what can be done to address this problem time efficiently through structured peer assessment in teacher training. [4]

References:
[1] D. Kahneman, Thinking, fast and slow. London: Penguin Books, 2012.
[2] D. Kahneman, O. Sibony, C. R. Sunstein, Noise. A Flaw in Human Judgment. London: William Collins, 2021, Introduction, chs. 3, 5, 6, 23, 25, 26.
[3] D. Kahneman, D. Lovallo, O. Sibony, “A Structured Approach to Strategic Decisions: Reducing Errors in Judgment Requires a Disciplined Process”, in MIT Sloan Management Review 60 (2019), pp. 67-73.
[4] F. Brosow, “TRAP-Mind-Theory. Philosophizing as an Educational Process”, in Journal of Didactics of Philosophy IV (1/2020), pp. 14-33. URL: www.philosophie.ch/jdph. Accessed 13 July 2021.

Keywords:

Bias, noise, peer assessment, teacher training, evaluation, judgment, grades, philosophy, ethics, humanities, education, noise audit, debiasing, noise reduction.