ASSESS THE ASSESSMENT: AN AUTOMATED TOOL FOR ANALYZING MULTIPLE CHOICE EXAMS

Multiple choice exams are widely used at educational institutions: They are an efficient method to examine very large student groups, due to automated evaluations the results are available shortly after the exam and the evaluation is unbiased by factors like students’ formulation, handwriting, language skills or different demographic factors.
Another advantage offered by multiple choice exams is that they can be easily analyzed by various statistical methods. The Item Response Theory provides many methods and models for this purpose. The most famous is the Rasch Model, formulized by the Danish statistician Georg Rasch, and its various extensions. Unfortunately, few lecturers (outside the field of mathematics, statistics or psychology) are familiar with these models and/or have the statistical knowledge to apply them.
We developed a tool to automatically analyze multiple choice exams and the items within the exams, using several statistical methods like the mentioned Rasch Model and its extensions for polytomous items response categories (i.e. the Partial Credit Model), but also more commonly used methods like hierarchical clustering, multidimensional scaling, factor analysis or analysis of variance.
After the user has provided the raw data from the multiple choice exam, that is, how each student has performed at each item of the exam, our tool computes
• the difficulty parameter for each test item and its distribution,
• the person aptitude parameter for each student and its distribution,
• statistical tests for multidimensionality, followed by factor analysis, hierarchical clustering and multidimensional scaling to group the test items into various categories, and
• statistical tests for person homogeneity (analysis of variance, Anderson’s likelihood ratio test, graphical model checks)
We use the open source software R (extended by additional packages for the various statistical methods) for computing and writing the respective results into a file which can be compiled with LaTeX, giving a pdf-document with the final report on the multiple choice exam and its test items.
Together with the respective key figures the report gives detailed explanations, so that each user with some rather basic statistical knowledge can correctly interpret their meaning, thus enabling him to assess and improve the quality of his multiple choice exams and test items.