Abstract:

The aim of this presentation is to describe the main technical characteristics of the Testing-Quest (TQ) methodology, and the previous experience that has led to its present format. Developed by the Department of Social Psychology and Quantitative Psychology at the University of Barcelona (UB), TQ is a service for professionals involved in psychological and educational appraisal who are interested in verifying the quality of their evaluative instruments (e.g. tests, exams). Assessments, level tests, and conventional/online questionnaires are all frequent in this type of environment, and it is accepted by default that they possess good evaluative qualities. Nevertheless, practical experience in auditing tests demonstrates that many instruments present biases and design errors (e.g. defective items, unreliable answer keys, response patterns) which negatively affect the result validity and lead to uncertainty amongst the examinees. In this respect, the new technologies have not led to an improvement in the quality of the instruments but rather to a simple increase in their use. In order to provide quality control, TQ, an online application and work methodology, has been created to carry out a ‘test of the test’ as an instrument of evaluation. TQ is a derivate of previous psychometric developments and currently, through the Bosch y Gimpera Foundation, forms part of the catalogue of UB licensed technologies.

It is composed of four auditory approaches: qualitative, quantitative, preventive, and reactive. The qualitative one focuses on the revision of the test’s content and structure and how it is applied. To do so, the suitability of the items with respect to the principles of good practice in test design is verified. In addition, the existence of possible patterns and signals in the layout and editing of the content is checked. With the quantitative audit, the metric qualities of the items are analyzed through the response matrices of the examinees. From these data the global functioning of the test is evaluated with each question and, if this is the case, each possible answer. TQ also assesses the coherence of the responses through information regarding omissions, and atypical response and error patterns that strongly coincide amongst the examinees. The preventive audit is carried out before the results are published giving the person in charge of the evaluation the chance to reconsider possible errors. The reactive one is generally employed when there is a suspicion of unfair or biased results once the test has been applied and its results published.

Keywords:

Assessment quality, Exams audit, Test validity, Biased assessment, Item analysis, Testing-Quest procedure.