DIGITAL LIBRARY
EVALUATING A TEST’S CONCEPTUAL PROPERTIES AND UTILISATION POTENTIAL: LINKING RESULTS FROM VALIDATION AND APPLIED RESEARCH AS A BASIS FOR IMPLEMENTATION
University of the Witwatersrand, Johannesburg (SOUTH AFRICA)
About this paper:
Appears in: EDULEARN11 Proceedings
Publication year: 2011
Pages: 6723-6732
ISBN: 978-84-615-0441-1
ISSN: 2340-1117
Conference name: 3rd International Conference on Education and New Learning Technologies
Dates: 4-6 July, 2011
Location: Barcelona, Spain
Abstract:
This paper first focuses on the validation model we have used over a ten year period to evaluate the conceptual properties of a test which can be used for screening purposes to identify children with learning difficulties. The instrument uses error profiles to identify children requiring special treatment programmes, based on the preponderance of particular types of phonic errors.

Our validation model has been a mixed one linking traditional psychometric evaluations with case studies of the instrument in use. The initial stages in the validation process involved content and construct validations conducted longitudinally with the instrument, as well as two large-scale evaluations which compared the types of phonic errors made by learners in different grades at primary school. These produced evidence of distinctive patterns of error made by children in mainstream and remedial classrooms, indicating that the profiling system could be used for predictive and discriminatory purposes.

Given findings indicating the internal and external validity of the instrument, a number of evaluations have also been conducted which focus on the utilization potential of the instrument. These studies have focused on the pragmatics of using the test for screening purposes as well as for planning and monitoring instruction in classrooms and clinics. This evidence has been produced through seventeen case studies of the instrument in use, which have then been integrated through meta-evaluation based on aggregative case survey analysis. Results from other classroom based studies indicate that the instrument is able to be used for screening purposes in classrooms with a high degree of accuracy. Information from the instrument can also be used by teachers for planning classroom programmes targeting the types of phonic errors made by children.

The second part of this paper links these two forms of evidence, and describes the implementation model we are currently adopting for making the instrument available for use by others in our country. This involves an initial stage in which training and certification in practical use of the instrument is provided, followed by internet-based support in the form of accurate computer-based scoring of test protocols. The aim is to provide detailed diagnostic information on the types of phonic errors made by children in the classroom, or by children requiring remedial instruction.

Our implementation model is based on consistent findings of both validation and applied research with the instrument over a ten year period. This indicates that the type of information yielded by the instrument can be used as for screening purposes to identify children with learning difficulties, and is also useful for planning instruction which targets the specific types of errors made by individual children or groups of children in the classroom. Our studies also indicate that the error profiles from the test can be used by teachers for planning instructional programmes, and have additional clinical uses in planning remedial programmes for children with learning problems.

We thus see potential for wider use of the instrument for monitoring and evaluation purposes in classrooms, schools, and for comparative purposes across schools. Given the focus of the millennium goals on achieving universal literacy, there would be value in using the instrument in this way both in our own country, as well as more broadly internationally.
Keywords:
Phonic error analysis, classroom usage, validation model, utilization potential, implementation model.