Abstract:

New AI tools that can provide answers to a student's questions are easy to use and are becoming increasingly accessible. Their potential to transform the student experience has changed significantly with this increasing availability. With access to AI tools in Java programming courses, students can receive real-time, around-the-clock, AI-moderated support. The design goal of this study is define a protocol and baseline that addresses the concerns that students may not develop sufficient programming skills with the use of AI tools. The potential benefits of using AI for students are significant: access to AI tools increases the ability for students to work constructively on their own schedule; the ease of use of the AI tools may engage students who might otherwise hesitate in asking for support; the access to AI tools may enable a more diverse population of students to be successful in a CS major. Before these tools can be meaningfully introduced into a course, research is needed to study the impact of these AI tools on a student learning. This paper describes a study design which explores the student experience by analyzing the competency-building and cognitive experience of students using AI tools while performing a programming task. The design of this study contributes to the research community a methodology for understanding on the impact of the use of AI chatbots and generative AI tools, such as OpenAI ChatGPT and GitHub Copilot, as an alternative to traditional lab environments where students learn programming without AI tools. This late-breaking paper will present the design of a between subjects study that consists of a pre-task assessment quiz; and a programming task in one of three conditions: (1) with no external help, (2) with the help of an AI chatbot, or (3) with the help of a generative AI tool like GitHub Copilot; followed by a post-task assessment and an interview on their experience and perceptions of the tools.

Keywords:

AI, ChatGPT, CS education, Student Experience, Intersectionality.