Abstract:

This study presents an at scale field evaluation of “Kilea”, a retrieval augmented (RAG) AI tutor that answers student questions with study booklet–grounded explanations and correct referencing. The broader problem we address is not only finding the right content, but helping diverse learners stay on track supporting self regulation and motivation at scale. Kilea personalizes what learners see along the path by constraining the AI to official materials, yet it does not adapt to the person.

The following research questions guided the study:
1) Does a study booklet–grounded RAG tutor (“Kilea”) produce measurable learning gains at scale in real world use?
2) Which engineering requirements emerge to move beyond content level help toward truly individualized learner support?

The analysis covers a university wide rollout (December 2024) and combines a quasi experimental difference in differences comparison of grades (users vs. non users, pre/post) with user feedback analytics. Acceptance is high: Net Promoter Score +39 from N=578 valid ratings across 186 modules (May–Aug 2025) and students value on demand answers with transparent source links. At the same time, the grade signal is minimal (≈ −0.03 additional improvement for the Kilea group), which shows that in relation to RQ1 content level personalization alone is not sufficient to shift outcomes at scale, especially when learners also use external tools.

Improvement requests cluster around pagination consistency, stronger factual handling, chat history, math rendering, a practice/exam mode, and stability.

In direct response to RQ2, the results point to the need for a data driven, privacy aware learner profile, represented in institutional systems and injected as context into the RAG pipeline, so individualized support can scale across courses and the AI can adapt not only to materials, but also to the person.

Kilea serves as a necessary first step, it proves demand and usability, but the evidence indicates that it is not enough to improve learning outcomes. The next iteration we identified is a type informed, anti procrastination tutor grounded in psychology, diagnostics, and ethics by design, with the goal of achieving measurable gains in self regulation, motivation, and academic performance, moving from content personalization to personalized learner support. Further studies are needed to evaluate the magnitude and conditions of these gains. The present findings motivate this shift and define the requirements for making it effective.

Keywords:

Technology, Education, AI, generative AI, Ed-Tech, Digital Transformation, Digitalisation, Chatbot, Learning Support, RAG, AI-Tutor, Personalized Learning, Learning Analytics, Higher Education.