Abstract:

Educators and institutions have had mixed reactions to the transformative potential of generative AI (GenAI), such as ChatGPT. Some seek to limit or ban its use, while others embrace it as another learning tool. The former fear that students will let GenAI do their learning for them. The latter want to harness GenAI's potential, recognizing that it has already become deeply embedded in educational environments; most post-secondary students already report using GenAI for coursework. Also, it is fast becoming integral to personal and workforce environments. So, denying students access does not seem a viable long-term solution. Thus, we must find ways to use GenAI to support rather than undermine student learning.

This issue is particularly critical in postsecondary education because it is the level at which students are expected to use their literacy skills ever more independently and flexibly, across an increasing variety of disciplinary contexts, to address increasingly complex reasoning and problem-solving tasks. Thus, a central aim of this project is to help college instructors to use GenAI to develop (rather than to undermine) student learning, critical reasoning, and applied skills.

To reach this aim, it is particularly important to transform the way that knowledge and skills are assessed in college courses. We believe that there will never be a transformative change in instruction absent a fundamental change in the nature of how we assess students. It is an old adage, but we need to build tests worth teaching to (Pellegrino, 2001, 2023). In the 21st century, this means expecting students to apply their knowledge and skills as necessary to address complex, ecologically valid problems.

This multi-institution, transformative research grant project aims to demonstrate the feasibility, validity, and viability of instructors building and implementing scenario-based assessments (SBAs) as both formative and summative instruments to support student learning. SBAs are one of a family of performance assessments that situate assessment of knowledge and skills in authentic practical and professional social scenarios/contexts, helping students to understand how course content is applied in the world outside the classroom.

We have been working with instructors across multiple college departments (psychology, English, interdisciplinary studies) to build SBA prototypes and implement them in their courses. To demonstrate the feasibility of SBAs, we have targeted large, lecture style courses, as those pose logistical constraints on instructors in tailoring and personalizing instruction and assessment feasibly. We have also targeted remote-delivery courses as those are most susceptible to over-reliance or cheating by using GenAI. As we enter the second year of the project, faculty have already designed and implemented SBAs in their courses, and plan to implement revised/extended versions again in Spring semester. We have also expanded our SBA development to new faculty in other departments.

We are employing a mixed methods, design-based research approach. We will report on results of pilots, as well as progress in designing a GenAI-powered SBA authoring system for faculty.

References:
[1] Pellegrino (2001). Rethinking and Redesigning Education Assessment. Preschool through Postsecondary. ERIC.
[2] Pellegrino (2023). Introduction: Arguments in support of innovative assessments. In Innovating Assessments to Measure and Support Complex Skills. OECD.

Keywords:

Assessment, Generative AI, Instruction.