Abstract:

Over the last decade, increased student numbers, limited resources accompanied with the quick evolution and penetration of new learning technologies, have led to amplified use of multiple choice questions (MCQ). In an ideal case, not only the process of question and item development is documented, but responses are also studied to ensure that the building blocks of tests are sound. The reliability and validity of test questions and items are important aspects when assessing whether a student understands content and if not, where and in what way he/she is struggling. This paper aims at identifying those curriculum parts of a Management Information Systems training program that students find challenging based on the analysis of their MCQ based test results. All in all, more than 150000 test fills were analysed collecting data from tests of general and specific Management Information Systems related courses. Beside the application of descriptive statistical methods, item level analysis has also been run to rank the responses in a multiple-choice question and to identify other factors (like the effect of item length) which enable examinees with to gain extra marks. As a conclusion, the paper describes how findings may be used to inform on future improvements concerning teaching approaches and methods.

Keywords:

Multiplce choice question, test validity, item validity, management information systems.