About this paper

Appears in:
Pages: 5921-5924
Publication year: 2013
ISBN: 978-84-616-3822-2
ISSN: 2340-1117

Conference name: 5th International Conference on Education and New Learning Technologies
Dates: 1-3 July, 2013
Location: Barcelona, Spain

ASSESSING INTER-RATER AGREEMENT ABOUT ITEM-WRITING FLAWS IN MULTIPLE-CHOICE QUESTIONS OF CLINICAL ANATOMY

B. Guimarães, J. Pais, E. Coelho, A. Silva, A. Povo, I. Lourinho, M. Severo, M.A. Ferreira

Faculdade de Medicina da Universidade do Porto (PORTUGAL)
Multiple-choice questions (MCQs) are regularly used in exams in order to assess students in health science disciplines. Despite this fact, MCQ items often have item-writing flaws, and few educators have formal instruction in writing MCQs [1].

The major purpose of our study was to estimate the inter-rater agreement about item classification as either standard or flawed.

In order to achieve this goal, four judges (2 teacher/2 students), blinded to all item performance data, independently classified each one of 920 test items from 10 examinations as either standard or flawed. If flawed the exact type of item flaw or flaws present in the question stem and respective options was recorded.

In this study, the standard item was operationally defined as any item that did not violate one or more of the 31 principles noted in a review article [2] which summarized current educational measurement recommendations concerning item writing.
The Fleiss' Kappa was use to evaluate the inter-rater agreement between 4 judges previous the consensus process.

In respect to the agreement about item classification as either standard or flawed was fair (kappa=0.3).

Despite the agreement was substantial for the more prevalent principles, generally the results showed many disagreements among judges about item classification, previous the consensus process. In a future investigation it is important to assess if presence of flaw or flaws in the MCQ item have impact its quality, namely, if there are interferences with difficulty and discrimination indices of the MCQ item.

References:
[1] Tarrant, M. and Ware, J. (2008), Impact of item writing flaws in multiple choice questions on student achievement in high stakes nursing assessments, Medical Education, 42 (2), pp. 198-206.
[2] Haladyna, T. M., Downing, S. M., and Rodriguez, M. C. (2002), A review of multiple-choice item-writing guidelines for classroom assessment, Applied Measurement in Education, 15 (3), pp. 309-333.

Supported by IJUP Controlo de Qualidade em Educação Médica: prevalência, determinantes e recomendações na meta-avaliação de testes de escolha múltipla em Anatomia (Ref. PP_IJUP2011 67).
@InProceedings{GUIMARAES2013ASS,
author = {Guimar{\~{a}}es, B. and Pais, J. and Coelho, E. and Silva, A. and Povo, A. and Lourinho, I. and Severo, M. and Ferreira, M.A.},
title = {ASSESSING INTER-RATER AGREEMENT ABOUT ITEM-WRITING FLAWS IN MULTIPLE-CHOICE QUESTIONS OF CLINICAL ANATOMY},
series = {5th International Conference on Education and New Learning Technologies},
booktitle = {EDULEARN13 Proceedings},
isbn = {978-84-616-3822-2},
issn = {2340-1117},
publisher = {IATED},
location = {Barcelona, Spain},
month = {1-3 July, 2013},
year = {2013},
pages = {5921-5924}}
TY - CONF
AU - B. Guimarães AU - J. Pais AU - E. Coelho AU - A. Silva AU - A. Povo AU - I. Lourinho AU - M. Severo AU - M.A. Ferreira
TI - ASSESSING INTER-RATER AGREEMENT ABOUT ITEM-WRITING FLAWS IN MULTIPLE-CHOICE QUESTIONS OF CLINICAL ANATOMY
SN - 978-84-616-3822-2/2340-1117
PY - 2013
Y1 - 1-3 July, 2013
CI - Barcelona, Spain
JO - 5th International Conference on Education and New Learning Technologies
JA - EDULEARN13 Proceedings
SP - 5921
EP - 5924
ER -
B. Guimarães, J. Pais, E. Coelho, A. Silva, A. Povo, I. Lourinho, M. Severo, M.A. Ferreira (2013) ASSESSING INTER-RATER AGREEMENT ABOUT ITEM-WRITING FLAWS IN MULTIPLE-CHOICE QUESTIONS OF CLINICAL ANATOMY, EDULEARN13 Proceedings, pp. 5921-5924.
User:
Pass: