Abstract:

As studying electrical engineering is particularly challenging in the first semesters, approaches are constantly being sought to identify and subsequently solve students' problems. This is the reason why our students wore eye trackers while solving two tasks from the basics of electrical engineering. The idea is that the recorded data can help to provide students with targeted support.

The recorded data was analyzed with regard to classical significant metrics concerning the correctness of the students' solutions. Significant metrics were identified. However, the interpretation of the eye-tracking data is not easy. The problem is that although there are generally valid statements on the interpretation of e.g. fixation durations that are related to attention (cf. Holmqvist et al., p. 379), fixation durations are also dependent on the context (cf. Holmqvist et al., p. 378). Consequently, an interpretation in the context of the fundamentals of electrical engineering is difficult. For this reason, a method from the field of machine learning is used to identify patterns within the metrics that indicate a correct or an incorrect solution.

In addition to the classic metrics, a further metric, the so-called dwell string, is used. This encodes the order in which the students looked at relevant areas of the tasks and thus depicts which parts of the task influenced their attention and in what order.

Both the classic significant metrics and the dwell strings are suitable for training machine learning models that can predict whether students will solve the task correctly or incorrectly. Since the dwell strings can only be available after the task has been completed, the paper presents an approach that only uses partial sequences (chunks) of the dwell strings. This makes it possible to predict the correctness of the solution based on the length of the required chunk. In addition, the significant metrics provide further information that can potentially be used in a machine learning model.

Consequently, the research questions arise:
RQ1: How many transitions (length of the chunk) are necessary to create a model that can predict the outcome of the student’s work with sufficient accuracy?
RQ2: Can other significant metrics improve the model’s performance, and if so, which ones?

The models used consist of a long short-term memory layer (LSTM), which is suitable for identifying patterns in the sequence of the dwell string. A second part, parallel to the LSTM layer, also an LSTM layer, is trained with the conventional metrics identified as significant.

Results:
We trained the models with dwell string lengths of 40, 50 and 60 in two different tasks, each with an additional significant metric. The results show that it depends on the task which chunk size is chosen. For the first task, a maximum test accuracy of 85% and for the second task, a maximum test accuracy of 91% could be achieved. For both tasks, the addition of the significant metric proved to be helpful for accuracy, both for training accuracy and for test accuracy. The models achieve a test accuracy that is sufficient to support students who are highly unlikely to produce a correct solution.

References:
[1] Holmqvist et al. Eye tracking: A comprehensive guide to methods and measures. 1. publ. in paperback. Oxford: Oxford University Press, 2015. ISBN: 9780198738596.

Keywords:

Eye-tracking, machine learning, electrical fundamentals.