INVESTIGATING TEST-TAKING BEHAVIOR IN SIMULATION-BASED ASSESSMENTS – VISUAL DATA EXPLORATION

Modern assessment methods include more and more complex test items such as simulations (e.g. OECD PIAAC, 2012) which aim at measuring 21st century skills, for instance solving problems in real life situations. Due to the dynamic nature of the measured construct, students use the assessment system and interact with the problem in various ways. In order to get detailed information about individual interactions with the online resources, students’ actions are automatically stored in log files.

During the assessment process a large amount of log data is accumulated with various characteristics (clicking, typed answers, and temporal data). For an efficient analysis of test-taking behavior in complex large data sets, it is necessary to get an insight into the actual process data. One very promising way to examine test-taking behavior is the visual data exploration. It aims at presenting data in visual form (Keim, 2002), and providing an opportunity to examine assessment activities. On the basis of the findings, we can choose the best methods to derive hypotheses and to further analyze process-related data.

The aim of this paper is to illustrate the benefits of using process-related data exploration. Our examination gives an insight into methods that are useful in depicting, identifying trends in test-taking behavior in simulations, and examining individual differences on the grounds of test-taking processes. For this reason we applied tools and visualization layouts, which support educational practitioners in making use of the data assembled in computerized test delivery.
The sample of the study was composed of 189 secondary school students (M=17.63 years,SD=.78). 57% of the sample was male. The computer-based ICT literacy test designed by Pfaff & Goldhammer (2011) was administered. It contained interactive items to measure skills required in the evaluation of online information with respect to credibility. The items used in this study simulated web search engines within various contexts. The data collection took place in 2010.

In this paper, three visualization layouts with two tools (ProM and Cytoscape) were applied. First, individual navigation paths and time spent on web pages were aggregated into a cumulated model to gain information about the test-taking processes of the whole sample. It was depicted as a directed graph. The advantage of this representation over other visual presentations (e.g. frequency tables) is that it enables the depiction of navigation actions according to page visit frequencies with navigation directions; and it presents activities in a comprehensive way on complex items with lots of sub pages, as well. In addition, items including various types of information resources (like pop-ups or html pages) can be differentiated by shapes of nodes. The second layout, a dot chart, allows us to investigate student-item interactions at individual level. It plots all link selection actions with time. On this chart, differently behaving subgroups of students is outlined. To take a closer look at examine single navigation behavior (third chart); directed graphs were used to depict time spent on pages and the order of link selection.
In sum, we demonstrated three graphical representation layouts. The benefit of these charts is that they give a quick overview on actual test-taking activities at various levels and the overall item functionality by representing students’ navigation activities with time information at the same time.