Abstract:

In order to facilitate student reflection on learning for engaging in learning self-directedly and responsibly, we have developed a student reflection support system that automatically classifies written reflections on learning by means of a machine learning model. In this paper, we described practices to evaluate the effectiveness of the developed system and the results of the evaluation.

To facilitate student reflection, we first need to grasp the situation of the reflection on learning from students’ written reflections. Second, we need to facilitate student reflection adaptively on the basis of the situation that has been grasped.

To satisfy these requirements, we focus on the automatic text classification of written reflections. We also utilize prompts that are provided adaptively according to the classification result.

For classifying written reflections, we came up with the concept of “phases”, with a focus on the function of metacognition in reflection. These phases consist of “recalling facts”, “reflection with cognitive awareness” and “reflection about precepts of learning”. We then created training data comprising 2,216 sentences of written reflections with a label that identifies these phases. After that, we constructed a machine learning model by supervised learning. Additionally, we defined prompts to elicit metacognition according to the phase in order to facilitate reflection on learning.

To implement the above approach, we developed a student reflection support system, which has three functions. Function 1 automatically classifies written reflections using the machine learning model. Function 2 visualizes the situation of each student’s reflection on learning on the basis of the classification results in order to help classroom teachers. Function 3 adaptively provides prompts to facilitate student reflection on learning according to the classification results in order to facilitate student reflection directly by the system itself.

We conducted two practices to evaluate the effectiveness of our system through ten classes in a lecture at the university. The first five times students used the system, it was enabled with functions 1 and 3 (practice 1), and in of the remaining five times, it was not enabled with functions 1 and 3 (practice 2).

After the practices, we conducted two evaluations. First, to determine whether the system was able to grasp the situation of student reflection on learning, we calculated the degree of coincidence (Kappa coefficient) between the classification results of the machine learning model and those of the teacher in charge of the lecture (evaluation 1). Second, to investigate the influence on reflection on learning with and without support by the system, we administered a questionnaire to students after each practice (evaluation 2).

The results of evaluation 1 showed that the Kappa coefficient was 0.86, which indicates a strong degree of agreement between the two. The results of evaluation 2 showed that students became more reflective about what they noticed and thought was important during classes when there was support from the system.

The results of both evaluations demonstrate that our system can grasp the individual situation of each student’s reflection on learning from their written reflections and can facilitate student reflection on learning.

In the near future, we will investigate whether our system is effective for long-term cross-curriculum learning.

Keywords:

Reflection on learning, written reflections, text classification, machine learning, prompt, e-learning systems.