Abstract:

Objectives:
The objectives of this study were to assess the reliability of an uncued question format (UCQ-format), when compared to a short answer question (SAQ-format) in the course of Biopathology at the Faculty of Medicine of Porto University and compare the satisfaction of students after the introduction of a glossary based on an uncued question format.

Methods:
Two hundred and eighteen students (218) were tested using SAQ-format and 283 students were assessed using UCQ-format. We resorted to both the Classical Test Theory and the Item Response Theory to compare both types of questions.
To assess the level of satisfaction with the new glossary, we analyzed the answers of 227 students that fulfilled a questionnaire that evaluates 4 dimensions (Satisfaction with new glossary (G1); Satisfaction with new answer sheet (G2); Preference between the previous and new glossary (G3); Classification on the previous course Biopathology (G4)). The Spearman's Correlation Test was applied between dimensions.

Results:
Trough the Classical Test Theory, we determined that the differences between the Difficulty Index on the first test (0.674 +/- 0.291) and on the second test (0.638 +/- 0.303) were not significant (p=0.367). In addition, the differences found on the Discrimination Index of the first and second test (0.264+/-0.152 and 0,260+/-0.152, respectively) were not statistically different (p=0.832).
Using the Item Response Theory, the Discrimination Index (SAQ: 0.685 +/- 0.163 and UCQ: 0.639 +/- 0.303) and the Difficulty Index (SAQ: 0.698 +/- 0.312 and UCQ: 0.655 +/- 0.325 after) were not statistically different between both tests (p=0.3185 and p=0.06985, respectively).
Results indicate that both approaches were reliable, with Cronbach's Alpha of 0.9298 on the SAQ-format and 0.9760 on the UCQ-format.
The students analyzed slightly disagreed with the change in the model of evaluation (XGIII=3.84).
The analysis of the satisfaction questionnaires showed that students who preferred the UCQ-format also preferred the new answer sheet. In general, students were not satisfied with the answer sheet. Similarly, students who were not satisfied with the UCQ-format preferred the previous model of evaluation (ρGI/GIII=-0.664 and ρGII/GIII=-0.542). We found no correlation between the student's classification and their preference with the new model of evaluation (p=0.428)

Conclusions:
There is no effect on the changed of format in the quality of test, however the students were not very satisfied with the new format and this fact was un correlated with the student's classification. Even though this format did not decrease the quality parameters and was more efficient tool in terms of time and resources saved by the correctors and faculty in general.