Abstract:

Tests are the most commonly used method for measuring how much learners have understood. S-P tables are tabular arrays of the correct and incorrect responses to each question for each learner, and S-P table analysis is one technique for visualizing and analyzing test results. This type of analysis can be easily used by teachers without statistical knowledge, allowing them to find responses that do not match learners’ abilities, such as learners with low ability correctly answering a difficult question. The conventional S-P table analysis is based on information about correct and incorrect response, but cannot reveal the state of comprehension in detail. For example, when a learner answers a question correctly, we cannot determine whether the response is based on a solid understanding. Accordingly, here we propose the confidence-integrating S-P table, which is an S-P table incorporating information about the learner’s degree of confidence into the analysis of their state of comprehension. The aim of this study was understand the state of comprehension in ways that cannot be detected with conventional S-P table analysis, by using data that includes information about the degree of confidence. We assume that it is preferable for learners to “answer correctly with confidence” or “answer incorrectly without confidence,” and that responses such as “answering correctly without confidence” or “answering incorrectly with confidence” should ideally not happen. We define the latter two types of responses as “heterogeneous responses.” Learners or problems with numerous heterogeneous responses are likely to indicate an undesirable state of comprehension, and analysis based on confidence-integrating S-P tables can detect heterogeneous responses based on heterogeneous comprehension, by defining heterogeneous comprehension in the same way as the caution index used in conventional S-P table analysis. To evaluate the proposed confidence-integrating S-P table, we conducted an experiment with two steps. First, we had 20 participants take a quiz consisting of 20 questions and collected data about their degree of confidence, as well as their answers. From the data, we created a confidence-integrating S-P table. Second, for learners with a high degree of heterogeneous comprehension, we inferred their state of comprehension based on how the heterogeneous responses appeared, and compared the inferred state and the one determined from interviews with the learners. Similarly, for questions with a high level of heterogeneous comprehension we made inferences about the questions and examined them in detail. As a result, we were able to verify that our inferences were largely in line with our expectations. Accordingly, we believe that the proposed confidence-integrating S-P table can be used effectively in teaching situations.