Abstract:

Today, educational institutions are facing new challenges related to the assessment of written papers or exams, because such texts (or parts of them) can be generated by artificial intelligence (AI) tools such as ChatGPT (Rudolph et al., 2023).

If the amount of submitted work is small, it can be reliably checked by the examiner. However, when it comes to a hundred assignments or even more, assessment may turn into a significant hurdle due to the lack of time resources.

In the past, various applications have been developed to score essays, such as the Automated Essay Scoring (AES) or the Computerized Adaptive Testing (CAT) systems. It must also be noted that there are such tools as the ChatGPT content detector which help identify whether a certain text was written by AI or not. However, while such tools can help detect AI-generated content, they still do so with varying degrees of success.

It is known that while ChatGPT can produce an imitation of what is being shared, it is unable to evaluate the relevance and accuracy of the information. ChatGPT cannot provide an opinion or write a novel text with depth. In addition, at this point of time, it is not always good at citing and providing references accurately (Rudolph, 2023; University of Agder, 2023). A recent study done by Aydın and Karaarslan (2022) showcases the high matching rates of the plagiarism tool and the paraphrase tests produced by ChatGPT, which clearly demonstrates that ChatGPT does not produce original texts after paraphrasing. All this may lead to the so-called “paraphrasing plagiarism”.

At the University of Agder, in the Spring semester 2023, the decision has been made at the university level that such AI-based systems as ChatGPT can be used by students in those examinations that allow using various aids such as books, articles, and online sources. If a student uses a text produced by, for example, ChatGPT, this must be cited just like any other source. In addition to the reference, students are asked to attach the whole text generated by the AI, together with the conversation and what the AI was asked to do. If a student fails to do so, it is regarded as a case of plagiarism.

In this paper, we will address the following questions:
(1) What challenges do AI-based systems such as ChatGPT pose for the assessment of students’ written texts?
(2) What potential do such systems have to improve or optimize the assessment of students’ written texts? and
(3) How can such systems be used as a tool to support teaching and learning?

First, we will examine the state-of-the-art landscape of such AI-based systems as ChatGPT in education. Second, we will test selected applications with free-text questions/problems. Third, we will discuss possible suggestions on how a larger set of written papers/exams can be assessed effectively bearing in mind they (at least partly) may have been generated with AI-based systems. This work is concerned with both summative and formative assessment.

References:
[1] Aydın, Ö., & Karaarslan, E. (2022). OpenAI ChatGPT generated literature review: Digital twin in healthcare. In Ö. Aydın (Ed.), Emerging Computer Technologies 2 (pp. 22-31). İzmir Akademi Dernegi.
[2] Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?. Journal of Applied Learning and Teaching, 6(1).
[3] University of Agder. (2023). 15 ting ChatGPT ikke klarer. Retrieved from https://www.uia.no/nyheter/15-ting-chatgpt-ikke-klarer

Keywords:

AI, ChatGPT, summative assessment, formative assessment, free-text.