Abstract:

This practice report presents the trial implementation of a generative AI chatbot teaching assistant (TA) in an introductory information literacy course with approximately 2,600 students. The course is delivered in an on‑demand format using web‑based texts, which traditionally limits opportunities for students to ask instructors or human TAs for clarification. To address this, we developed a chatbot that combines Retrieval‑Augmented Generation (RAG)—where course materials are searched and used to enrich the prompt—and the GPT‑4o language model to provide context‑sensitive answers. The server is built with PHP and SQL, and the local DifyCommunity environment is used to host the RAG‑enabled chatbot accessed via an API. Login identifiers are pseudonymized and freely chosen by students, ensuring privacy while allowing aggregated analysis of learning behaviour.

We conducted statistical analyses of navigation logs and chat logs collected during the semester to clarify when students accessed materials relative to scheduled class times, how different types of login identifiers were used, the distribution of question types and lengths, and the most frequent terms used in questions. We also classified chatbot responses to determine how often hints were provided versus direct answers when students requested correct solutions to assignments or short quizzes. Although the RAG‑based prompt extension effectively constrained responses to course content, some interactions revealed mismatches between students’ expectations and the chatbot’s hints, indicating a need for further refinement of prompt design and response control. Survey feedback showed that many users appreciated immediate hints and the friendly tone of the chatbot, suggesting that it can effectively support motivation in on‑demand learning. This implementation demonstrates the potential and challenges of using generative AI at scale in first‑year education and offers data‑driven insights for researchers in educational technology.

Keywords:

Generative AI, chatbot teaching assistant, retrieval‑augmented generation, learning analytics, on‑demand education.