Abstract:

With the rapid evolution of Natural Language Processing (NLP), Large Language Models (LLMs) like ChatGPT have emerged as powerful tools capable of transforming various sectors. Their vast knowledge base and dynamic interaction capabilities represent significant potential in improving education by operating as a personalized assistant. However, the possibility of generating incorrect, biased, or unhelpful answers are a key challenge to resolve when deploying LLMs in an education context.

Given these challenges, we designed and implemented a new chatbot architecture for education called ChatEd, which is retrieval-based and integrated with a large language model. Retrieval-based models, traditionally employed by specifically trained chatbots, select the most appropriate document from a predefined set of documents based on the user's input, which ensures validity and relevancy. Instead of returning only the reference document and answer, the raw document is sent to a LLM for generating the query response. This integration with the LLM introduces the benefits of human-like conversations, contextual understanding, and depth of conversation, while preserving the key benefits of traditional information retrieval by ensuring accuracy of responses and verified references to original source documents. The combination has better usability than traditional retrieval chatbots and improved accuracy and source verification compared to using only an LLM. By having responses include the sources used to generate an answer makes it easier for students to locate the information.

The key system features include:
- Enhanced Accuracy and Contextual Relevance: By combining the power of advanced NLP techniques with a rich knowledge base, the system delivers contextually accurate responses.
- Conversational Memory: The model is designed to remember and reference chat history, allowing for deeper and more meaningful interactions.
- Ease of Integration: A user-friendly interface allows seamless integration with existing Learning Management Systems (LMS) and leverages existing course materials to ensure accurate and educational context-specific answers.
- Streamlined and Easy Training: Unlike previous chatbot approaches, there is no training required on the Q&A data.

The primary contribution is introducing and evaluating a new chatbot framework for educational use. We show that the system has an extremely high level of question answering ability through leveraging both a retrieval based framework and ChatGPT. Furthermore, a distinctive feature of our model is its scalability and ease of use for instructors. Instead of relying on traditional and often labor-intensive methods of chatbot training, our model facilitates the direct input of existing course materials. This approach not only streamlines the setup process but also ensures that the chatbot's knowledge base is closely aligned with the course content.

A comprehensive series of evaluations is performed to assess the core framework of our proposed system, particularly its question-answering ability and context awareness. Evaluations are performed to measure if the question answers are helpful, relevant, accurate, and correct. Experiments show the ChatEd chatbot excels in all these criteria. ChatEd is compared with ChatGPT for answer quality and ability to support conversational interactions.

Keywords:

Large language model, chatbot, ChatGPT, information retrieval, educational personal assistant.