Abstract:

In this session, we present development work and initial pilot trials of scenario-based assessments (SBAs), stemming from a recently funded transformative research project funded by the US Institute for Education Science. The purpose of this project is to create a generative artificial intelligence (gen-AI) enhanced authoring tool for SBAs. SBAs place knowledge and skills into a scenario or practical context so that test-takers can be both observers of and active participants in their own performance, building on a body of research in developing and validating SBAs. This creates the opportunity for a reflective, metacognitive, and self-regulatory loop that enables instructors to use SBAs for formative assessment purposes while maintaining the psychometric properties necessary to evaluate student success and monitor the quality of instruction. Because SBAs are difficult to develop, college instructors struggle to develop and deploy SBAs in their courses. Recent advances in gen-AI make it possible to scale up and democratize SBA development, enabling postsecondary instructors to design and administer localized, personalized, and discipline appropriate performance assessments that provide better feedback, higher levels of adaptivity, and richer diagnostic information. The project will support the widespread use of gen-AI enabled SBAs in college courses by putting SBA development into the hands of instructors rather than a team of psychometricians and assessment developers.

We have adopted a design-based research methodology. In the initial phase of the 3-year project, researcher-instructor led development teams have been building SBA exemplars (not using the authoring tool) to support development of authoring and support system modules. We are working with teams teaching courses in psychology, English writing composition, and interdisciplinary studies. The instructors will be implementing task sets starting in Fall, alongside their typical assessments. We have already begun collecting baseline data in which we evaluate student literacy and basic writing skills, as well as on previously designed SBAs. We will also be analyzing course syllabi and assessment regimen, observing classrooms, and interviewing instructors. Each course has a significant online, digital aspect; the interdisciplinary studies course in entirely digital. Because we are partnered with urban universities, the student body is diverse, for many, they are the first generation in their families to attend college.

The project plan calls for us to engage higher-education instructors by conducting training workshops at conferences and meetings or through synchronously online settings where we will:
(a) demo the SBA prototype and
(b) conduct hands on tutorials using the gen-AI SBA authoring system.

While we are in the early phase of the project, we will address these aims.

References:
[1] Deane, P., Song, Y., et al. (2019). Scenario-based assessment of written argumentation. Reading and Writing: An Interdisciplinary Journal..
[2] Sabatini, J. P., O’Reilly, T., Halderman, L., & Bruce, K. (2014). Broadening the scope of reading comprehension using scenario-based assessments: Preliminary findings and challenges. L’Année Psychologique, 114(04), Article 04.
[3] Sabatini, J., O’Reilly, T., Weeks, J., & Wang, Z. (2020). Engineering a Twenty-First Century Reading Comprehension Assessment System Utilizing Scenario-Based Assessment Techniques. International Journal of Testing, 20(1).

Keywords:

Technology, assessment, literacy.