Abstract:

The increased awareness of artificial intelligence (AI) in the education sector has led to concern amongst educators around the quality of their assessment tasks (Sweeney, 2023) because of the ability of AI to generate responses to assignments that have high validity. Given that AI detection tools are likely to struggle to detect the learner use of AI tools in generating answers to assessments (Lee & Palmer, 2023), there is a need to enhance cultures of academic integrity and improve the design of assessments to make them more resilient to misuse of AI. This paper discusses the latter.

In this paper we discuss the design and testing of a rubric designed to evaluate the resilience of an assessment task in multiple disciplines based on current literature on AI and assessment e.g. (Chaudhry et al., 2023; Swiecki et al., 2022). The rubric identifies areas key to the design of a task which, whilst not yielding tasks which are completely AI proof, nonetheless may require greater learning engagement from students. The elements of the rubric to be discussed are the type, format and content of the assessment task, how it targets higher order thinking, how generative AI tools ‘answer’ the demands of the task and how anti-dishonesty tools may still play a role in that task.

We discuss the types of tasks that are typically more resilient to AI misuse (such as oral vivas), how group work may provide greater resilience than individual tasks and how aiming high for cognitive level of tasks and providing strong constructive alignment to learning outcomes and explicit scaffolding of authentic tasks may support a more robust assessment.

A pilot study of seven courses across multiple disciplines showed only one course achieved a ‘score’ of greater than 75%. Areas that performed consistently well in the rubric were the targeting of higher order thinking, the intent shown to support academic integrity and the types of tasks chosen to encourage and measure learning. The weakest area was in the content of the assessments, which measured aspects such as constructive alignment, the authenticity of the task, explicit scaffolding and re-use of learning from earlier tasks, and if the outcome was more related to a product than a process. In this area only one course obtained a ‘pass’ score.

This paper will encourage discussion around the results and the nature of the rubric with the goal of supporting colleagues in addressing the impact of AI in assessment.

References:
[1] Chaudhry, I. S., Sarwary, S. A. M., El Refae, G. A., & Chabchoub, H. (2023). Time to revisit existing student’s performance evaluation approach in higher education sector in a new era of ChatGPT—a case study. Cogent Education, 10(1), 2210461.
[2] Lee, D., & Palmer, E. (2023). How hard can it be? Testing the dependability of AI detection tools. THE Campus. https://www.timeshighereducation.com/campus/how-hard-can-it-be-testing-dependability-ai-detection-tools
[3] Sweeney, S. (2023). Academic dishonesty, essay mills, and artificial intelligence: rethinking assessment strategies. 9th International Conference on Higher Education Advances (HEAd’23),
[4] Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022). Assessment in the age of artificial intelligence. Computers and Education: Artificial Intelligence, 3, 100075.

Keywords:

AI, artificial intelligence, assessment, technology, education.