Abstract:

Generative Artificial Intelligence (Gen-AI) has revolutionized education by offering personalized learning experiences and automated feedback. While prior research has explored Gen-AI's influence on education, limited studies have examined its effects on students' coding proficiency. This study investigates the impact of Gen-AI on the coding skills of novice students in the Advanced Object-Oriented Programming course. To ensure consistency in evaluation, Gradify, a Java-based auto-grading system, was developed to analyze student performance. It comprised a control group of 698 students who completed the course before Gen-AI integration and an experimental group of 203 students with access to Gen-AI tools. Remarkably, the results showed a decline in the performance of the experimental group across all topics except recursion. This suggests a potential misalignment between the feedback provided by Gen-AI and assignment objectives, particularly in areas requiring adherence to specific program structures such as class/method names and method signatures. This indicates that students may rely on Gen-AI responses without fully comprehending them, potentially diminishing engagement and independent problem-solving. Recursion was the only topic where performance improved, suggesting that Gen-AI effectively enhanced understanding in a topic where students typically struggle. Additionally, an analysis of submissions prone to errors revealed an increase in file name mismatches, implying an over-reliance on Gen-AI for routine tasks and reduced attention to submission details. These findings underscore the importance of strategically incorporating Gen-AI in programming education. While Gen-AI can be a powerful tool, this study demonstrates that its effectiveness is limited when merely used as a standalone solution. For this technology to be beneficial, students must first develop a strong understanding of problems and concepts. This foundation allows them to identify discrepancies between generated solutions and the program specifications in order to refine the generated solutions. A balanced approach, integrating Gen-AI-driven support with structured teaching strategies, will help maximize learning outcomes while minimizing dependency.

Keywords:

Generative AI, Advanced object-oriented programming, Programming education, Student performance.