Abstract:

Advising and supporting students in their career choices and the associated further education paths is becoming increasingly important. Current studies show that there is a high drop-out rate in Bachelor‘s degree courses, which is due to the fact that the ideas and expectations of the students and the content offered in the course do not overlap sufficiently. While pupils widely started to use ChatGPT as a tool for education, it is not specifically a career-counselling system, lacking domain knowledge and country-specific data. Another drawback is that there is no guidance in the conversation and a conversation is driven by the users questions, not by the system itself.

To meet this challenge, we have developed an AI-based assistant to provide individual advice on career choices and identify suitable educational paths. The assistant is a chatbot based on a modern open-source large language model (LLM) hosted in a data-sovereign manner and adapted to the educational domain using prompt engineering. This allows for an open conversation that is more human-like. In a conversation, the goals, ideas, experiences and abilities of the students are determined and a user profile is created. This user profile is used to make several recommendations for an occupation. If the user is not satisfied, the conversation is deepened further for more details. As soon as an occupation matches the student‘s requirements, several possible educational paths are calculated. These can consist of apprenticeships and different bachelor and master courses leading to the selected occupation and adapted to the student‘s profile.

In order to evaluate the newly developed assistant, we designed and conducted an experiment with pupils to test the usability of the assistant in recommending careers and educational pathways. We created two additional assistants. First, a form-based assistant functions as the baseline of the comparison. The second assistant is an AI-based chatbot that is intent-based, i.e. a user utterance is mapped to an intent and an action is triggered. So the conversation is prescribed with a fixed set of utterances and the assistant is not able to have open conversations. One benefit is that the conversation is very structured and guided by the assistant.

An important aspect of such systems is whether they are trusted and accepted by the users. Therefore, we examine in the user test the effect of the assistants on trustworthiness and associated conditioning factors. These include perceived competence, autonomy and the anthropomorphization of the assistant. We investigate on the perceived ability of the assistants on explaining its utterances and recommendation.

With this paper we contribute to the development of a trustworthy educational path recommendation that can give personalized career suggestions. Therefore we compare three different assistants evaluating how they make recommendations that pupils feel they can trust.

Keywords:

Generative Artificial Intelligence, User Study, Conversational User Interface, Chatbot, Educational Pathway Recommendation, Large Language Model.