Abstract:

This study explored the effectiveness of NLP (Natural Language Processing) supported scaffolding techniques on students’ reflection writing process. We compared two sections of an introductory computer programming course (N = 188) by using a quasi-experimental research approach. In Section 1, students did not receive any scaffolding while generating reflections, whereas in Section 2, students were scaffolded during the reflection writing process. Student reflections were collected using two versions of the CourseMIRROR mobile application (standard version in Section 1 and adaptive version in Section 2).

CourseMIRROR was designed and developed by our research team based on the Reflection Informed Learning and Instruction (RILI) model. The RILI model expands on the hypothesis that students’ meaningful reflection can improve their learning experiences. The RILI model suggests that when both students and instructors are engaged in identifying gaps and difficulties in the lecture and concepts, it helps instructors provide effective feedback, improving students’ knowledge and skills. Also, students can utilize various responses to improve their learning, such as seeking resources or help materials and interacting with peers and the instructional team to ask questions.

In this study, to introduce such a model in large classes and ensure students’ participation effectively, we used CourseMIRROR mobile app, which prompts students to reflect on their learning experiences after each lecture. The students are prompted to generate reflections based on two perspectives:
1) muddiest point (MP)
2) point of interest (PI).

Furthermore, the application uses Natural Language Processing (NLP) algorithms to summarize students’ reflections for each lecture. These summaries are made available to both students and teachers. It is noteworthy that the summary’s quality depends on how specific students were writing their reflection on each perspective of reflection writing, i.e., MP and PI. For the specificity, the NLP further converts each written reflection into an equivalent specificity score ranging from 1-4 points, where score 1 indicates shallow reflection specificity and score 4 indicates excellent reflection specificity.

Using Natural Language Processing (NLP) algorithm, the app calculated each reflection's specificity score. We conducted an independent sample t-test between the students’ reflection specificity scores in these two sections. The results indicated a significant mean difference between adaptive and standard versions. Students using the adaptive version wrote more specific reflections than students using the standard version of the app, suggesting that automated scaffolding helped students write more specific reflections, which may be helpful in their overall learning outcomes and engagement in an introductory computer programming course.

Keywords:

Reflections, NLP, mobile, scaffolding.