THE RELIABILITY AND VALIDITY OF RUBRICS FOR THE ASSESSMENT OF MANUAL SKILLS IN THE SUBJECT PHYSIOTHERAPY OF NEUROLOGICAL PROCESSES

Background:
Rubrics facilitate the communication between lecturers and students, while providing the latter with a method for self-evaluation of their own learning process1. Likewise, they serve as a guide because they show the indicators, criteria or evidence that informs students of what they have to do to achieve the expected competencies. However, in the design and use of rubrics, there should be a consensus between lecturers and students. In this regards, the reliability (degree of agreement between students) and the validity (degree of agreement between students and lecturers) of this evaluative strategy should be ensured.

Purpose:
To present the results of an educational innovation project which was performed to study the reliability and validity of rubrics for the assessment of manual skills in the subject Physiotherapy of Neurological processes of the University of Valencia.

Methods:
A series of rubrics (N: neurodevelopment approach; P: proprioceptive neuromuscular facilitation; and C: children assessment) were developed in the course Physiotherapy of Neurological processes of the University of Valencia to overcome the difficulties students had found to achieve manual skills in case of the complex techniques studied in this subject. After carrying out an activity in which self-assessment, peer-evaluation and lecturer assessment were performed, the fulfilled rubrics were used in this study. The level of convergence between the assessments made by lecturers and students (acting as peer-evaluator and self-assessors) was analysed from a holistic perspective (global score)2. The degree of agreement between lecturers and students was calculated by using two different estimation procedures (conservative and one lax) accepted and commonly used in this field. In both cases, it was determined through the kappa statistic.

Results:
One hundred fifty-five undergraduate Physiotherapy students (49% men, average age 22.5±4.1 years) and two lecturers participated in the activity. The level of agreement between lecturers and peers using the conservative estimation procedure was discreet (N: kappa = 0.40, p <.001); moderate (P: kappa = 0.45, p <.001) or substantial (C: kappa = 0.61, p <.001) depending on the rubric. On the other hand, the level of agreement between lecturers and peers using the lax procedure was moderate (N: kappa = 0.60, p <.001); substantial (P: kappa = 0.70, p <.001) or almost perfect (C: kappa = 0.88, p =.001). In general, the agreement was better between students when peer-evaluating and self-assessing than between lecturers and students.

Conclusions:
Analysing the convergent validity between lecturers and peers when rubrics are used is especially relevant. It involves considering the adequacy of the rubric for the evaluation of the students´ competences by lecturers and is an indicator of the degree in which the students have internalized the rubric criteria. In this regard, our rubrics, especially those related to neurodevelopment approach, should be modified in order to get a better agreement between lecturers and students.

References:
[1] Martínez-Figueira E, Tellado-González F, Raposo-Rivas M. (2013). La rúbrica como instrumento para la autoevaluación: un estudio piloto. Revista de Docencia Universitaria,11(2),373-390.
[2] García-Ros R. (2011). Análisis y validación de una rúbrica para evaluar habilidades de presentación oral en contextos universitarios. Electronic Journal of Research in Educational Psychology, 9(3),1043.