Abstract:

The total test scores are interpreted as indicators of the level of proficiency. Even if the test is built with guarantees of content validity, the validity of these interpretations is not guaranteed for all people tested. The process that has followed each assessed person to give their answers to the test questions is crucial to ensure the validity of individual interpretations. The guidelines for the reporting of test score, published in 2012 by the International Test Commission, recommend use statistical techniques to identify invalid scores. In tests with multiple-choice questions, the person-fit statistics are suitable for identifying atypical patterns of response that invalidate the interpretation of the total score.

The main aim is to quantify the atypical response patterns present at an English language proficiency test. Our interest is to provide a useful tool for formative assessment. The data are from an institutional assessment conducted by the Department of Education of the Catalan Government . The test was completed by 56.428 students of fourth course of Secondary Education throughout all schools in Catalonia. Analyses were carried out with PerFit R package. Modified Caution Index was calculated to quantify the presence of atypical response patterns.

The results show a high number of atypical response patterns. Our attention is addressed to students with central scores, because this target group has a higher potential improvement. We present different identified atypical patterns and discuss its relevance in evaluative assessments.

* This work was received financial support from the Dirección General de Investigación y Gestión del Plan Nacional de I+D+I, of the Spanish Ministry for Economy and Competitiveness, through the project EDU2013-41399-P

Keywords:

Pedagogical innovations, Evaluation of Student Learning, Educational Assessment, Aberrant response pattern, Learning Styles, Compulsory Education.