Abstract:

Higher education institutions are always looking for more objective and transversal evaluation mechanisms. From panels of several professors where final marks are obtained as the average between all proposed partial marks, to exams put through different quality assurance mechanisms, the Quality Assurance Systems of every University proposes different strategies. However, one evaluation mechanism has been considered the most objective for the last fifty years: multiple-choice exams. Actually, they are extensively used in exams to become civil servant and similar administrative processes where equality is a legal requirement. But students typically are afraid of this evaluation mechanism and claim it is not a fair approach. Their informal observations indicate students have several problems to prove their real learning level because of the format of this exam and frustration is usual in courses implementing this mechanism.
On the other hand, multi-choice exams in engineering courses have a very special format. Most courses in engineering are applied science, so typical evaluation systems consist of problems to be solved through mathematical instruments. The final result from one of these engineering problems is a number answering the main question posed together with the context description. In that way, multi-choice examen in engineering courses follow a similar structure but students cannot provide a full description of their reasoning process and solving methodology. The exam just includes a collection of different numerical values as options.
Several professors also noted this approach influences negatively the students results, so for many years its use was negligible. But the COVID19 crisis has highly increased the popularity of multiple-choice exams, as they allow their automatization and randomization in a easier way than other exam formats. In this context, and before promoting an extensive use of multiple-choice exams, a formal scientific study about how the exam format influences the students’ results is required.
In order to scientifically analyze how the exam format affects the students’ results, a pilot experience was planned and carried out in Universidad Politécnica de Madrid during the 2022/23 academic year. One hundred and twenty-two (122) students enrolled in the Computer Networks course in the Computer Engineering degree. All of them attended to the same learning activities and followed the same teaching methodology. But, for the final evaluation, two different groups were organized. Both groups were similar in number of students and social composition (age and women percentage). One group (control group) did a short-answer type exam. In this exam, students could submit all their reasoning and solving process. Although only the short answer (number or final result) written in the official form was evaluated. But students were not aware of this fact. On the other hand, a second group (pilot group) received the exact same exam but prepared according to a multi-choice format. Problems were exactly identical, but students had to choose between four different options. And no additional material could be submitted.
Results confirmed students’ informal observations. Statistical tests proved a significant difference between academic results in the pilot and control groups, being results in the pilot groups worse than in the control group. Effects such as the Pygmalion effect may explain this difference.

Keywords:

Engineering education, pilot experiences, test-type exams, academic results, evaluation, computer engineering.