Abstract:

Background:
Recent advances in artificial intelligence have introduced AI chatbots (e.g., ChatGPT) as potential intelligent tutoring systems in education. These tools can provide instant feedback, personalized guidance, and conversational support to students. However, questions remain about their effectiveness in secondary-level programming courses and how to best integrate them into teaching given concerns about reliability and cognitive impacts.

Objective:
This pilot study examines the use of an AI chatbot (ChatGPT) as an intelligent tutor for teaching programming (pseudocode, flowcharts, HTML, and C++) in a secondary school context. We aim to evaluate learning outcomes, cognitive load, and student attitudes, using synthetic data to simulate a class of 22 high school students.

Methods:
A mixed-method design was employed in a simulated Italian high school "Web Community Manager" track. Quantitatively, pre/post-test programming scores, NASA-TLX cognitive load ratings, an attitude survey, and chatbot interaction logs were analyzed. Qualitatively, chat transcripts were reviewed for student help-seeking patterns. Paired t-tests, descriptive statistics, Pearson correlations, and a regression analysis were used to examine learning gains and their relationship with cognitive load, attitudes, and chatbot usage metrics.

Results:
Students’ average programming test scores improved significantly from pre- to post-test (mean increase ~13 points, p < 0.001). Cognitive load measured by NASA-TLX was moderate (mean ~49/100), and students reported positive attitudes (mean ~4.0/5) toward the AI tutor. A moderate positive correlation was observed between the extent of chatbot use (number of messages) and learning gains, while perceived cognitive load showed a slight negative correlation with gains.
Regression analysis (controlling for prior knowledge) suggested that pre-test scores strongly predicted post-test performance, with a positive but non-significant contribution from the level of chatbot interaction. Qualitative log analysis indicated that students primarily used the chatbot for step-by-step hints and debugging help rather than direct answers.

Conclusion:
The synthetic pilot results suggest that AI chatbots like ChatGPT can function as supportive tutoring agents in secondary programming education, leading to knowledge gains without imposing excessive cognitive load. Students were receptive to the chatbot tutor, engaging actively with it. The findings point to the potential of AI tutors to scaffold learning in programming courses, though realworld trials are needed. Key implications include the importance of guiding students to use the chatbot effectively to maximize learning and minimize over-reliance. Future work will involve validating these results with actual classroom implementations, refining the integration strategy, and addressing challenges such as ensuring solution accuracy and maintaining student agency in problem-solving.

Keywords:

AI in education, intelligent tutoring systems, programming instruction, pseudocode and flowcharts, cognitive load, secondary school, mixed-methods.