Abstract:

Short-reading comprehension questions are widely used in reading education to foster understanding of the text prompt's structure. These questions typically require students to read a specific passage(text prompt) and then articulate their understanding of its contents in a few sentences. Despite the widespread usage of short-reading comprehension questions, the lack of feedback for students' responses hinders their effectiveness in school education. Students often encounter difficulties in identifying and correcting their errors, as short answer responses may contain various error types. However, manually creating effective feedback imposes extensive labor on educators. This limitation necessitates the automatic generation of scaffolding feedback that helps students identify their errors by linking a student's response to the scoring rubric to promote further understanding of the text prompt's structure.

Natural Language Processing (NLP) has evolved significantly in recent years. However, the main focus of applications for short-reading comprehension questions has been on the automatic grading of responses, with very few studies exploring automatic feedback generation. To address these issues, we aim to construct a system that generates feedback for student's responses.

The contribution of this study is two-fold: first, it is the first system to generate feedback for short-answer reading comprehension questions. In this question format, the answer exists in the text prompt, and the appropriate answer can be derived by clarifying the structure of the text prompt. To automatically generate feedback, we propose a graph structure called the 'answer diagnosis graph,' which integrates the logical structure of the text prompt and the feedback templates. We then use this graph and NLP techniques to estimate the extent to which students understand the structure of the sentences. Finally, based on the estimation results, we generate feedback that facilitates understanding according to the student's responses.

Our second contribution was to conduct the first demonstration experiment based on this generated feedback. We conducted the experiment with the aim of ascertaining what impact our feedback design would have on actual students and what challenges these are. Japanese high school students(n=39) were asked to answer two questions that required them to answer in 70 to 80 words. In the experiment, the students were divided into two groups, taking into account the difference in academic ability as little as possible. One received a model answer, and the other was also provided with generated feedback. Then both groups re-answered the two questions, and we compared their score changes. A questionnaire was also used to ask the students to evaluate the feedback directly and to investigate their motivation.

The results showed no significant difference in the improvement of scores after re-answering in both groups. On the other hand, according to the questionnaire survey, system feedback facilitated the student’s discovery of points to be improved in their own responses and points to be focused on in the referenced parts of the text prompts, and it was also confirmed that the feedback was significantly effective in increasing the student's motivation. However, there is still room for improvement in promoting understanding of the text prompt’s structure.

Keywords:

Feedback generation, Natural Language Processing (NLP), Short Answer Questions, Logical structure.