Abstract:

The rising popularity of Multiple-Choice assessment comes with the need for accurate evaluation. However, previous studies provide conflicting and scarce evidence about the effect that the position of the response options has on the items’ psychometric properties, that is, on their difficulty and discrimination. Regarding the position of the key or correct answer (KA), different studies have found that when the option is placed at the first [1] or middle [2] positions, items become easier, while other studies have not found any effect of the position of the KA. As for the position of the distractors or incorrect answers, studies mainly focus on the most plausible or attractive distractors, finding that highly attractive options at the beginning make items more difficult [3].

In this study, response-option positional effects were assessed, using 3, 4 and 5 response-option items (ROI) from a sample of 1,989 items from university undergraduate tests. While no effects of the KA position on the items’ properties were found, the position of the distractors seems to have an effect on item difficulty. When the most attractive distractor was placed in the last position, items became more difficult; meanwhile, when the least attractive distractor was placed in the last position, items became easier. These two last results seem to be related to the ‘anchoring’ effect [2], which describes the tendency of the examiners to write the distractors orderly with decreasing plausibility. Thus, students would expect the most plausible options in the first positions, and the least plausible ones in the last positions. With respect to the number of options, 3 ROI appeared to be more robust to positional effects than 4 or 5 ROI.

These findings are relevant to Multiple-Choice assessment. Firstly, examiners should redistribute the position of all response options, and not just the KA, which is the usual practice. Secondly, 3 ROI seem more suitable when it comes to avoiding positional effects. Future studies should further investigate the effects of distractor-position on item difficulty and discrimination, which seem to be a novel and promising line of research.

References:
[1] B. Hagenmüller, “On the Impact of the Response Options’ Position on Item Difficulty in Multiple-Choice-Items”, in European Journal of Psychological Assessment, vol. 0, pp. 1-10, 2020.
[2] Y. Attali, and M. Bar-Hillel, “Guess Where: The Position of Correct Answers in Multiple-Choice Test Items as a Psychometric Variable”, in Journal of Educational Measurement, vol. 40, no. 2, pp. 109-128, 2003.
[3] J. Shin, O. Bulut, and M. J. Gierl, “The Effect of the Most-Attractive-Distractor Location on Multiple-Choice Item Difficulty”, in The Journal of Experimental Education, vol. 88, no. 4, pp. 643-659, 2019.

Keywords:

Multiple-choice items, distractor position, key answer position, option order.