Abstract:

The main aim of the PISA (Programme for International Student Assessment) study is to measure students’ skills essential for full participation in the society of knowledge. In order to test their literacy in different areas such as: reading, mathematics and science, experts from countries participating in the study, design assessment frameworks and questions (items) that represent these frameworks. These items are then carefully piloted in all countries before a final test is constructed.

The studies in education need a large number of items assessing students’ skills in different areas. Laboratory of Thinking – Diagnosis of Science Education in Poland has been carried out by the Science Section, Educational Research Institute, Warsaw, Poland. Its aim was to examine the level of skills and scientific knowledge of the ISCED2 level graduates in the four subsequent years. Assessment tools for the study were prepared during workshops organized by the Institute of Philosophy and Sociology, Polish Academy of Sciences and Educational Research Institute. Both type of items – the cognitive ones (in biology, chemistry, physics and geography) as well as contextual ones (for the students’ questionnaire) were prepared according to the procedures used in the PISA study. As in PISA, the items preparing process involved multiple reviews by academic specialists and a field trials performed on random samples of student, as well as cognitive interviews (verbal probing technique).

Interviews using verbal probe technique performed on selected students in order to track the possible reasons for items inconsistencies and students misconceptions. Students interviewed for each study were ISCED2 seniors (normally aged 15–16) were familiar with all of the content covered by the items. They came from different schools, and were selected by their teachers to fit in one of the three categories: gifted but not diligent, gifted and diligent, and not gifted but diligent. In that sort of study it is essential that the students should be communicative, and this is also why the selection had to be made by their teachers, who knew them best.

Each interview lasted for ca. 1.5 h and was recorded. To begin with, the student solved a test of several close-ended questions in science subjects, including the four described in the present paper. To complete this task, the student had as much time as he or she needed and additionally could write down a commentary to each of the items. Then he or she was asked questions about the school subjects they preferred and the reason behind it. For each of the items general and item-specific questions were asked.

In the publication, we will describe the results of the cognitive interviews on the exemplary tasks measuring the ability to recognize groups of compounds in chemistry and problems encountered by students during solving those items. We will describe also how the verbal probing technique has made it possible to detect specific misconceptions in students’ thinking and how the tasks measuring this skill have been improved.

Keywords:

PISA, science items, problem solving, verbal probing technique, cognitive laboratory assessment.