Abstract:

A common problem in distance learning is the excessive demand placed on teachers and tutors by institutions, especially by students. Since students can study at different times, they often seek answers to their questions using various tools in the Learning Management Systems (LMSs) at different hours. Using technologies that apply Artificial Intelligence techniques can reduce the management and monitoring effort for teachers and tutors and enhance the teaching and learning process through immediate availability and the provision of management indicators.

The research that led to this article aimed to support the interaction between students and teachers/tutors by utilizing interactive technologies that employ Artificial Intelligence (AI), such as Chatbots. It involved an indirect analysis of interactions in a real environment to gather knowledge and generate accurate models. The methodological process included an indirect analysis of interactions in a real environment, the development and training of the conversational agent (Chatbot), and the validation of accuracy and acceptance by users. Data from the interaction history of several courses in different subjects at the Federal Institute of Paraíba, using the Moodle LMS, were mapped to analyze the most frequent types of student questions in tools such as direct messaging, discussion forums, and other asynchronous collaborative tools. The goal was to shape the conversational agent's knowledge base. Courses with distinct natures and profiles were combined to provide broader coverage and different questions and answers. User interactions were crucial in adding aspects of natural language within the employed context, guiding changes in conversational flows, especially when users did not respond well to answers or had difficulties following the intended flow. The conversational agent was developed using the Rasa NLU engine, chosen through a competitive analysis process that employed the Benchmark methodology, considering the application context.

To evaluate the accuracy of the conversational agent, the chatbot model generated from training underwent two well-defined stages. Firstly, model testing involved statistical analysis to assess accuracy and improvement. Secondly, functional validation included manual functional tests and an evaluation questionnaire based on the Technology Acceptance Model (TAM) with real users in a controlled environment. Regarding the accuracy of the Alpha chatbot, the results suggest that the model can be improved in terms of intentions, either through daily usage or by training with a larger dataset. Feeding the model with new data will refine intention discovery and correct association, enabling the chatbot to evolve with increased usage. On the other hand, the entities performed well, showing a low failure rate in classification (0.071). The validation of the chatbot with users using a functional testing plan and the TAM model yielded highly satisfactory results.

Keywords:

Conversational agent, chatbot, Moodle LMS.