Abstract:

Empirical research poses numerous challenges for beginners. This is especially true for data analysis - a task that usually requires knowledge from two distinct areas: statistics and programming. To support prospective researchers with that task, we developed a web-based tool called eyenalyzer. It supports common activities in the data analysis phase of empirical studies in a way that is suitable for novices in both, statistics and programming.

The present article describes a controlled experiment investigating the impact of this tool with a total of 20 participants (all engineering students). All of them are given a set of common data analysis tasks, such as building a structured file from raw data, recoding or filtering data, calculating measures of descriptive statistics, visualising the data, or performing hypothesis tests for group differences as well as correlation analyses. Half of the participants complete the tasks using eyenalyzer, the other half can use anything except for eyenalyzer (i.e., they can use any software system, the internet, or even chatbots). For each task and participant, we record the processing time in minutes, the task score (i.e., as one of not completed, wrong, borderline, acceptable, correct), and the perceived difficulty (i.e., on a 7-point Likert scale). The results confirm that our tool is a valuable support for novice researchers: With eyenalyzer, the participants are faster, achieve higher scores, and perceive the tasks to be less difficult. All those differences are significant at alpha=0.05.

The article is structured as follows: We first motivate and introduce the tool; after that, we describe the study conduction in detail (e.g., the sample, the task set, and the measurements), present the study results, and elaborate on their implications.

Keywords:

Empirical research, eye tracking, data analysis, tool evaluation.