About this paper

Appears in:
Pages: 4522-4528
Publication year: 2018
ISBN: 978-84-697-9480-7
ISSN: 2340-1079
doi: 10.21125/inted.2018.0882

Conference name: 12th International Technology, Education and Development Conference
Dates: 5-7 March, 2018
Location: Valencia, Spain

DETECTING CONFUSING ITEMS IN A DIGITAL SKILLS CERTIFICATION SYSTEM THROUGH ASSESSMENT ANALYTICS

J. Bartolomé1, P. Garaizar2

1Tecnalia Research & Innovation (SPAIN)
2University of Deusto, Faculty of Engineering (SPAIN)
An increasing number of recruiting processes -both in public and private sectors- require assessing digital skills of job applicants. With this aim in mind, the Basque Government created the "IT Txartela" (IT Card), a public digital skills certification system. So far, more than 111.000 citizens certified up to 332.000 digital skills through IT Txartela. This online e-assessment service is comprised of several modules with different types of items (multiple-choice, matching, simulations, etc.). In some cases, applicants claim items as confusing or misleading. Considering this, some of these items have been improved or removed from the modules during the last years. However, this feedback about items is provided mostly by participants who did not pass the certification exams. With the aim of better understanding the clarity of the IT Txartela items, we gathered a fine-grained log of 6730 IT Txartela participants (470000 distinct user-question interactions). Then, we asked IT experts to perform an analysis of the clarity and difficulty of the items. Experts identified some items that were confusing and/or tagged with a level higher or lower than expected. At the same time, we used assessment analytics techniques to identify items that required more time and more visualizations than others or had extremely low success rates. Surprisingly, we found a whole set of items that should be revised and remained unnoticed after the experts' analysis. These results show that assessment analytics techniques can identify unforeseen problems in certifications systems and help to improve the items bank periodically.
@InProceedings{BARTOLOME2018DET,
author = {Bartolom{\'{e}}, J. and Garaizar, P.},
title = {DETECTING CONFUSING ITEMS IN A DIGITAL SKILLS CERTIFICATION SYSTEM THROUGH ASSESSMENT ANALYTICS},
series = {12th International Technology, Education and Development Conference},
booktitle = {INTED2018 Proceedings},
isbn = {978-84-697-9480-7},
issn = {2340-1079},
doi = {10.21125/inted.2018.0882},
url = {http://dx.doi.org/10.21125/inted.2018.0882},
publisher = {IATED},
location = {Valencia, Spain},
month = {5-7 March, 2018},
year = {2018},
pages = {4522-4528}}
TY - CONF
AU - J. Bartolomé AU - P. Garaizar
TI - DETECTING CONFUSING ITEMS IN A DIGITAL SKILLS CERTIFICATION SYSTEM THROUGH ASSESSMENT ANALYTICS
SN - 978-84-697-9480-7/2340-1079
DO - 10.21125/inted.2018.0882
PY - 2018
Y1 - 5-7 March, 2018
CI - Valencia, Spain
JO - 12th International Technology, Education and Development Conference
JA - INTED2018 Proceedings
SP - 4522
EP - 4528
ER -
J. Bartolomé, P. Garaizar (2018) DETECTING CONFUSING ITEMS IN A DIGITAL SKILLS CERTIFICATION SYSTEM THROUGH ASSESSMENT ANALYTICS, INTED2018 Proceedings, pp. 4522-4528.
User:
Pass: