Abstract:

LLM-generated feedback has the potential to deliver personalized and motivating guidance to large numbers of students without increasing instructor workload. Understanding who engages with such feedback, as well as when and how students engage, is critical for designing systems that realize this potential at scale. Using log data and performance assessments from the experimental arm of a largescale randomized controlled trial in a fully online undergraduate economics course (N = 28,824), we examined who engaged with LLM-generated feedback, when engagement occurred across the semester, and how interaction patterns related to academic performance. Among experimental-group students who completed at least one assessment, 39.9% viewed at least one LLM-generated feedback message (n = 3,945 of 9,875). Engagement was more common among students who were female, slightly older, and already more active in the learning management system (LMS). Among students who engaged, those in the highest-performing tertile generated substantially more interaction events than students in the medium- and low-performing tertiles. Engagement increased sharply in the weeks leading up to the midterm and final examinations, with students frequently revisiting previously delivered feedback when assessment stakes were highest. Despite equivalent message delivery and availability to feedback across students by design, engagement varied systematically by performance level. These patterns are descriptive and likely reflect self-selection, whereby higher-performing students were more inclined to engage with feedback opportunities. This suggests that simply providing access to feedback is not enough and highlights a need to better support uptake among lower-performing learners.

Keywords:

LLM-Generated Feedback, Online Learning, Feedback Interaction Patterns, Learning Analytics.