Abstract:

The educational paradigm within the field of statistics at the undergraduate level frequently confronts the dual challenges of conceptual complexity and student engagement. Traditional pedagogical approaches often fall short in providing the individualized attention and adaptive learning pathways that are essential for personalised learning. In this study, two generative artificial intelligent (Gen-AI) tutors are designed to facilitate a stepwise learning process for statistical problem-solving, presents a novel opportunity to address these challenges. This research aims to systematically evaluate the efficacy of this innovative educational tool.

The focal point of this investigative research is the Gen-AI tutor's approach to decomposing complex statistical problems into a series of manageable, logically sequenced steps. This tutoring approach was adopted as it aligns with the Socratic questioning method where students are scaffolded with questions to achieve deep learning. It is also aligned with cognitive load theory, which posits that learning is maximized when information is presented in chunks that do not overwhelm the learner's working memory. By guiding students through a problem in step-by-step manner, the AI tutor seeks to ostensibly enhance the student understanding of the problem-solving process and to facilitate retention, as each step builds upon the previous one, cementing the foundation before adding complexity.

Two Gen-AI tutors are built using the large language model (LLM) model available on the Microsoft Azure and Google enterprise solution, which are GPT-4-8K and text-bison, respectively. The system or meta-prompts design to guide the Gen-AI to behave as intended are the same for both Gen-AI tutors. Both are grounded with the same curated materials from the faculty and are instructed to provide answers based on the curated materials first. The main difference is that the Google version includes very deliberate prompt flow as part of the Retrieval Augmented Generation (RAG) process to further guide the behaviour of the chatbot.

40 questions across 10 statistical topics were chosen as the test cases. Since we anticipate that weaker students would require more personalised support, we generated interactions that mimics the behaviour of these students for both Gen-AI tutors. We also include the vanilla ChatGPT responses for the 40 questions as a baseline.

The quality of the Gen-AI tutor will be evaluated by two human tutors of the course and 50 student volunteers. The evaluation rubric measures not only the accuracy of the answers from the gen-AI tutors, but also how well the gen-AI tutors can perform stepwise guidance in terms of clarity and helpfulness.

Our analysis would be of interest to educators interested to find out if Gen-AI tutors can perform stepwise guidance for statistics and if investment into developing RAG-based Gen-AI tutors is worth the effort.

Keywords:

Personalised learning, AI tutor, Generative AI, large language model, stepwise guidance.