Abstract:

In this work, we develop a model to analyse exams based on multiple choice questions and present UCV-Test as the implementation of the model in a spreadsheet context.

The analysis of an exam with UCV-Test has a double perspective: the evaluation of the students and the evaluation of the exam and its questions. Both perspectives interact because the analysis of the psychometric properties of the exam, applied to a group of students, allows us to refine the parameters of the model. The refined parameters again change the students’ scores and the psychometric properties of the exam. After several iterations, the final scores of the students and the final evaluation of the quality of the questions and the exam, considered as a measurement instrument, converge in the refined model.

The value of the parameters of the model and the students' answers, together, determine the automatic weights that serve to assign a value for each question (its contribution to the scores of the students who answer the question correctly) based on three elements:
• The a priori importance of the question, which is a relative measure specified by the teacher.
• The a posteriori difficulty of the question, measured by its score (the higher the score for the question, the easier it is).
• The discriminating capacity of the question, measured by the correlation between a measure of the students' grade (the pre-score) and the contribution of the question to this pre-score.

In the automatic weighting system, we follow the principle: the easier questions weigh more, instead of the more intuitive principle: the most difficult questions weigh more. This is justified by thinking that it is worse to fail an easy question than a difficult one and, therefore, the first case deserves more penalty.

We have implemented the model in Microsoft Excel because it is software easy to use and widely available, which allows the analysis to do the analysis interactively. The analyst can modify the parameters of UCV-Test to correct any possible problems in the process in three ways:
• Changing the a priori importance of a question or even cancel it.
• Changing the penalty for failing or for leaving the question blank.
• Changing the distribution of the weights assigned to the a posteriori difficulty and the discriminating capacity in the calculation of the value of each question.

Each change in the parameters of the model triggers a chain of changes in the spreadsheet, immediately showing its effect on the score and the discriminating capacity of the questions, as well as on the students' score and the psychometric properties of the exam.

UCV-Test also gives us a measure of the internal consistency of the exam and, for each question, a measure of its impact on it. This allows us to detect if a question reduces the internal consistency of the exam and it is convenient to eliminate it or reduce its weight.

To gain confidence about the validity of the content of the exam (the test measures what we want to measure), we suggest complementing the multiple-choice questions with several short questions and verifying the agreement between the scores obtained with both types of questions.

UCV-Test is a flexible framework for the analysis of an exam, with intuitive tables, graphs and controls, which allows the analyst to automatically create and send detailed and personalized reports to each student.

UCV-Test has been widely tested in the School of Medicine of the Catholic University of Valencia.

Keywords:

UCV-Test, multiple choice questions, assessment.