Abstract:

With the growing popularity of generative AI (GenAI), particularly large language models (LLMs), it is unsurprising that students have started using them in programming courses. In our cohort of 600 first-year students, this came with a clear downside: the mean exam score dropped from 72% to 54%, while the average homework grade (done outside class) increased.

However, a total ban on LLMs feels unrealistic and probably counterproductive. To address this issue, we developed SocraticAI, a family of LLMs designed to produce guiding hints that nudge students to reflect, think, and debug, similar to how an instructor would. We use SocraticAI in Collaborative E-Quiz (CE-Quiz), an in-class programming assignment where students can ask GenAI for help, but only in the form of hints, not answers.

We describe several training strategies used to develop SocraticAI and improve the quality of its hints. These include the creation of synthetic datasets for supervised fine-tuning (SFT), training the model using reinforcement learning (RL) with GRPO applied both from scratch and on top of an SFT-initialized model, and an adversarial curriculum-learning approach, in which a stronger model is trained to generate increasingly challenging tasks and incorrect solutions that expose SocraticAI’s weaknesses, while SocraticAI is simultaneously optimized to produce higher-quality hints on these difficult cases.

This approach is supported by OPTMentor, a visual in-browser debugger that integrates LLM calls plus a jailbreak-prevention system to keep the interaction academically safe. We release open-source SocraticAI models for Python and C++, ranging from 1B-parameter models that run locally on a student’s laptop without a GPU to 7B-parameter models suitable for centralized deployment.

Empirically, the use of SocraticAI in our classroom yielded promising results. In a pilot run in Semester A 2026/27, students who had performed poorly in earlier assessments showed improvement on the final exam: most increased their percentage score by 10-30%, with an average gain of 11.9%. In a subsequent full run in Semester B 2026/27, the average course grade rose to over 70%, compared to 54% in Semester A 2026/27.

Acknowledgement:
This work was supported in part by the grant on “Socratic AI: AI-Enhanced T&L with Hints but Not Answers” (Project No. 9220151) and the grant on “DIVE for Team-Based Learning” (Project No. 6953002).

Keywords:

Generative AI, large language models, programming education, intelligent tutoring systems, hint generation, reinforcement learning, curriculum learning, educational technology, academic integrity.