Abstract:

Chatbots are software systems capable of interacting with users through natural language via text or voice. In educational contexts, these systems have increasingly been adopted to support teaching and learning by providing continuous and personalized assistance to students. Their ability to operate 24/7 is particularly valuable in asynchronous learning environments, such as online courses, where students can obtain support regardless of time constraints. However, the growing use of educational chatbots introduces challenges for pedagogical monitoring. Instructors often lack effective mechanisms to systematically analyze student queries and verify whether chatbot responses are accurate, relevant, and pedagogically appropriate. Without structured analysis, identifying recurring doubts, knowledge gaps, and chatbot failure points becomes a time-consuming and impractical task. To the best of our knowledge, few solutions focus on the automated analysis of chatbot interaction logs from a pedagogical perspective.

To address this issue, this paper introduces BotAnalyzer, a tool that transforms historical chatbot interaction logs into analytical dashboards containing key performance indicators. BotAnalyzer employs Natural Language Processing (NLP) techniques to perform statistical and semantic analyses of conversational data. Its architecture is designed to detect interaction patterns and identify communication failures between students and chatbots. A core feature of the system is the automatic detection of unanswered or inadequately addressed questions, which may indicate gaps in the chatbot's knowledge base, training data, or instructional coverage.

BotAnalyzer was evaluated through a proof-of-concept using real interaction data from 50 undergraduate students enrolled in a Software Testing course. The students interacted with TOB-STT, a chatbot designed to assist with questions related to software testing concepts, testing criteria, and test case design. The analysis identified the most recurrent student doubts and specific deficiencies in the chatbot's responses.

The results indicate that BotAnalyzer generates actionable insights for instructors by highlighting complex topics for students, recurrent interaction patterns, and communication breakdowns in chatbot responses. The system automatically detects cases in which the chatbot fails to provide a satisfactory answer or explicitly signals a lack of knowledge, enabling targeted improvements through prompt refinement or knowledge base updates. Additionally, BotAnalyzer provides engagement metrics and topic-based access seasonality, allowing instructors to identify peak usage periods and shifts in student interest over time. By identifying both the most and least frequent topics in student queries, the tool supports the detection of knowledge gaps and the monitoring of class progress. The results suggest that BotAnalyzer contributes to the field of Computers in Education by providing a scalable platform that transforms conversational logs into actionable insights that support pedagogical decision-making and the continuous improvement of educational chatbots.

Keywords:

Chatbot, Conversational Agent, Decision Support Systems.