Abstract:

Numeracy skills has been widely recognized during the last two decades as essential for person’s educational achievements and ability to live a successful life in modern society. Thus, successful early interventions for all children who has trouble with numeracy skills is important. To plan and implement such interventions teachers and school administration need valid and reliable diagnostic instruments to identify what students know and what skills they have at the end or start of learning period.

Development and application of automated scoring technologies and feedback systems is seen as solution for teachers to track student progress. However to use such tools for diagnostic purposes the design of back-bone of system, i.e., valid and reliable instrument to measure what students already know and understand is crucial.

The aim of this paper is to describe the development and validation of diagnostic instrument for assessing numeracy skills in 7th grade based on three dimensional numeracy assessment framework. This study is a part of broader study where we are intended to develop IT-based support system designed to obtain data on students’ literacy and numeracy skills important for teachers and school administrations to plan data-based interventions and implement support for students.

To achieve this, initially, a three-dimensional assessment framework were prepared by experts based on literature. Assessment framework distinguished disciplinary content knowledge (e.g., numbers, probability etc.), the application including data acquiring, analysis, interpretation and mathmatical reasoning as well as complexity of performance (e.g. SOLO taxonomy, i.e., the Structure of Observed Learning Outcomes and traditional marking), dimensions of learning. Second, the assessment framework were used to construct a 33 – item numeracy skills assessment test. The test included multiple-choice as well as open-ended response items. Third, the test was administered to 320 7th grade students in four secondary schools in Latvia. Finally, the students’ answers on all items were evaluated by 8 teachers using traditional marking scheme as well as by 2 experts using SOLO taxonomy.

To validate the test, it’s validity and reliability were examined employing, the range of traditional statistical techniques including item analysis, calculation of Cronbach’s alpha and computation of raw score commonly used to develop assessment instruments. To facilitate decision making on refining and revision of test the Rasch analysis were made.

Data analysis revealed that diagnostic instrument is suitable to assess 7th grade students’ numeracy skills and provide information for teachers to adjust instructions and target weaknesses of students. However, minor revision of test items was necessary to ensure that all criteria for all three dimensions in assessment framework is measured at least by one item as well as to increase discrimination between low and high performing students. The analysis of data set were SOLO taxonomy as marking scheme for open-ended items were used showed higher discrimination between low and high performing students than traditional marking. Finally we can conclude that the three dimensional assessment framework for numeracy can be used for development of similar diagnostic instruments to measure numeracy skills in other grades in the future.

Keywords:

Numeracy skills, diagnostic assessment, assessment validation.