Abstract:

Rating written performances arouses controversial opinions about the measurement of learners’ papers and its quality. Test developers attempt to provide a number of procedures to eliminate subjective judgements of raters as much as possible and increase validity and reliability of this open-ended measurement.

Our research is based on the verification of the facets that are expected to enable language learners to get trustful data which will provide them with relevant feedback important for their further language enhancement. On the other hand, data can have the negative consequences on test-takers’ future lives in case of high-stakes language tests. Therefore, there is a tendency not to provide language learners with skewed data. The procedures used in the rater training workshop include a list of facets that are necessary to be respected such as a structured task, marking criteria and training based on the use of exemplar performances.

Our study presents a project in which university students in their pre-service-teacher-training programme participated in the rater training workshop. After being introduced to the rating procedures and explained a number of descriptors used in the analytic marking scale, student raters were asked to rate a set of performances at two reference levels: B1 and B2, required at the school-leaving examination in English in the country. After each rated paper, students’ ratings were collected and each level score was explained by one of the students who had to justify their ratings giving the reasons for their judgements. This process revealed different educational backgrounds and students’ experiences during their own secondary-school studies.

The students engaged in the process of rating were later requested to write a task that can be used for testing writing. These tasks were presented, analysed and commented on by their peers. After discussions, all the problematic areas they had been exposed to either while rating secondary-school students’ written performances or writing their own tasks were reported, which resulted in a number of recommendations that will be presented in this paper.

Keywords:

Written performances, measurement, objectivity, training raters.