Abstract:

Online assessments are important at all levels of education. During the height of the Covid-19 pandemic lockdown, for example, practically all educators were teaching online and using online assessments. One of the many challenges Covid posed was academic dishonesty during online testing. Educators faced the need for a method to construct online assessments that were robust to cheating. However, the need to “pivot” quickly online added the requirement that assessment construction did not entail an unrealistic amount of time and effort.

One solution to minimizing cheating in online assessments is the use of question banks. Students are given questions randomly chosen from a question bank, i.e., a collection of possible questions, making it unlikely that two students will have the same questions in common. However, to what extent students cheat depends on many variables: the size of the question bank, the size of the class, the number of questions on the assignment, and “social factors” (such as how well the students know each other, the mechanism of how they cheat, and how dishonest they are). Not finding a published systematic study of this problem, we undertook a theoretical analysis to model how well assessments constructed from a question bank safeguard against cheating. This analysis aims to allow educators to “reverse engineer” the size of question banks to safeguard against cheating. Specifically, the question we answer in our analysis is the following: “How large does a question bank need to be to ensure a probability of p that two students (in a class of s students) have at most k questions in common on an assessment of n total questions?”

Our model supposes an assessment containing the same number of questions, randomly drawn (with replacement) for each student, from the same question bank. We assume the only mechanism by which students cheat is by sharing answers to questions they have in common. Under our model, the extent to which two students can cheat depends on the number of questions they have in common. We, therefore, define a metric for the upper bound on the extent to which students can cheat, called integrity, as the percentage of questions two students do not have in common. So, for example, on an assessment with 90% integrity, students can only cheat on 10% of the questions. However, due to the random nature of the draws, two students can have any number of questions in common. For this reason, we define a new metric, the reliability of an assessment, which is the likelihood that any two students randomly chosen from a class will meet the integrity criterion. Thus, an assessment with 90% integrity has 80% reliability if there is an 80% likelihood that no two students will have more than 10% of the questions in common.

In this presentation, we first derive the mathematical formulas for integrity and reliability for a class of two students. Next, we extend the model to a class of s students. Finally, we incorporate how social factors determine the extent to which a class cheats and how this affects the size of the question bank to ensure a certain level of integrity and reliability. Hopefully, this will give educators a better understanding of how to construct question banks for online assessments that are robust and limit the extent to which students can cheat.

Keywords:

Online assessment, question banks, cheating.