Abstract:

The rapid emergence of Large Language Models (LLMs) has changed the landscape in Higher Education (HE) and in many other institutions worldwide. Assessment, already problematic in many universities, in terms of students’ satisfaction and understanding of it, has been heavily affected by the easy availability of LLMs. Various positions have been adopted, from banning the use of Artificial Intelligence (AI) in university assignments to adopting its use more flexibly. I have always ensured that my students develop the skills needed to become confident, critical and knowledgeable. As Prof Acar (Harvard University & King’s College London, KCL) recently proposed, the following five skills are needed to teach students to use AI effectively (the PAIR framework): Problem formulation, AI Exploration, Interaction and experimentation, critical thinking and willingness to Reflect. I have recently introduced the use of LLMs in my two undergraduate Neuroscience modules at KCL; the initial part of this work has been presented at INTED24 in Seville and here in Valencia I am now presenting the results. Methods: During the academic year 2023-24 I have introduced the use of a modified PAIR framework in both my 3rd Year undergraduate Neuroscience modules (Behavioural Science and Perspectives on Pain and Nervous system disorders) at KCL with 120 students each. I have introduced the PAIR framework and the proposed assignment with class workshops aimed at explaining the usefulness of developing transferable skills apt for an AI-driven world. The students were then asked to produce a 2,000-word essay (peer-reviewed before submission by two students) in their own words; then they had to ask the AI tool to write an essay with the same title, record prompts used, tools explored, chosen or discarded and then write a 500-word reflection on the comparison between the two essays. To ensure accessibility and an inclusive learning environment, the students have been asked to use tools available from KCL subscriptions thus avoiding using those which have a paywall. Thematic analysis has then been used to evaluate the students’ reflections. Results: All the 240 own students’ essays and the reflections have been double marked and the AI-produced ones have been read. High variability was observed in the proficiency with which students were able to obtain essays from their chosen AI tool; the reflective piece was by far the most interesting one to read, overall showing a high level of students’ enthusiasm and engagement with the task. Thematic analysis of the students’ reflections has resulted in the following themes to emerge: own students’ essays almost always were perceived as being of a better quality; AI lacks critical evaluation, perceived as superficial and providing generic facts; AI can mostly help with essay planning; AI sources are not always reliable; lack of ethical evaluation (e.g., avoiding stereotypes) and issues of sustainability were raised by quite a few students who complained that the use of LLMs, with its high energy consumption needed for its training leads to a very high carbon footprint. Conclusion: Embedding the use of LLMs in undergraduate students’ assessment seems to have elicited interesting discussions about AI suitability, ethics and sustainability; having given the students the opportunity to critical think about the use of AI this will provide them with essential transferable skills supporting them in their future careers once graduated.

Keywords:

Large Language Models, Generative Artificial Intelligence, Higher Education, Assessment for learning, PAIR framework.