Abstract:

In recent years, educational assessment is undergoing a paradigm shift from the traditional summative assessment, which sorts students on a continuum according to their grade, to formative assessment, which focuses on identifying in greater detail the strengths and weaknesses of students in order to remedy potential shortcomings. This theoretical drift is being reinforced by a series of advances in the field of psychometrics which, together with technological improvements, now allow us to easily estimate models that, instead of being limited to the sum of correct scores, are adjusted to the different cognitive processes that underlie the responses to the items. Along these lines, some programs and web applications such as FoCo (https://psychometricmodelling.shinyapps.io/FoCo/) have recently been proposed in order to facilitate the application of these methodological advances to real practice.

The aim of this presentation is to illustrate how formative assessments can be implemented through a family of psychometric models known as cognitive diagnostic models (CDMs). To this end, during the 2021/2022 and 2022/2023 academic years, responses were collected from over three hundred Psychology undergraduate students to nearly one hundred items from the Psychometrics course. The items were administered as part of voluntary self-assessments that students could answer at the end of each topic of the course as a method of review. These data are analyzed from a classical perspective (e.g., difficulty, internal consistency) and from CDMs (e.g., fit, classification accuracy). From these results, the predictive ability of summative and formative information is compared with respect to performance on the course exams. The results indicate that it is possible to extract diagnostic information in these applied contexts. In particular, the information provided by the CDMs can be useful for students, as they serve as feedback to guide the learning process, and for teachers, who can have objective and immediate information about their students' competencies so that, if necessary, they can carry out effective reinforcement tasks. Ultimately, this project unifies the increasing access to technology in the classroom with modern psychometric models in order to improve the accuracy and effectiveness of educational assessments.

Keywords:

Assessment, Competencies, Psychometrics, Cognitive Diagnosis Models.