Abstract:

Multiple choice exams are widely used in educational institutions, and normally strongly linked with an e-learning preparation phase and sometimes more recently embedded into an e-assessment environment. Besides many advantages (and some disadvantages), MC exams offer the possibility to be easily analyzed by various statistical methods, ranging from basic descriptive statistics to sophisticated models, e.g. the Item Response Theory (IRT) with the Rasch model and its various extensions as the most prominent representatives.

However, most lecturers using multiple choice exams are either unfamiliar with these IRT models, or do not have the necessary statistical knowledge, time or inclination to apply them themselves to receive feedback about the inner workings of multiple choice tests.

We developed a tool to automatically analyze multiple choice exams and their embedded test items, using several statistical methods like the mentioned Rasch Model and its extensions for polytomous item response categories as well as more commonly known methods like descriptive statistics, hierarchical clustering, factor analysis or analysis of variance. The automatically generated report lists the respective key figures of the various analyses, together with detailed explanations to help non-statisticians to easily interpret the results.

We implemented this tool at our university’s exam server that administers and evaluates all multiple choice exams, where it is executed whenever new exam data are uploaded. Next to the exam’s results, lecturers receive a detailed statistical analysis report of the exam, which helps them to advance the overall quality of their exams.

A first feedback session has given very positive responses so far: lecturers appreciate the standardized summary of the exam’s key figures, as well as the easy means to monitor the quality of their multiple choice exams, and the hints how they could improve their test items, very much.

Keywords:

Multiple Choice Exams, Item Response Theory, Rasch Model, Automated Analysis, Assessment Evaluation.