Abstract:

The three most known international large-scale (student) assessments (ILSA) are the International Association for the Evaluation of Educational Achievement (IEA) Trends in International Mathematics and Science Study (TIMSS) (Mullis & Martin, 2017) and Progress in International Reading Literacy Study (PIRLS) (Mullis & Martin, 2015) and the Organization for Economic Co-operation and Development (OECD) Program for International Student Assessment (PISA) (OECD, 2019), which measure the different ages of primary and secondary school students in science, reading, mathematics and other fields. One of the aims of the assessments is to inform the stakeholders of the participating countries about the effectiveness of their education systems and to help them find good practices.

Each of the assessments was originally a paper-and-pencil-based (PB) test. The three international assessments are now computer-based (CB) to varying degrees. In 2016, PIRLS offered ePIRLS as an option, in addition to the traditional measurement, in the same framework as PIRLS assessment, but measuring digital reading skills (Mullis et al., 2017). In 2021, digitalPIRLS was conducted in a digital format, with the opportunity of paper-based administration (PIRLS | IEA). TIMSS 2019, after previous research on item equivalence (Fishbein et al., 2018), was implemented in computer-based administration, leaving the participating countries free to choose the paper-and-pencil form.

The largest change compared to the paper-and-pencil form is in the case of PISA. In 2015, after a study related to the field trial assessment (Robitzsch et al., 2020), the tests were administered computer-based, and in 2018, the linear test structure was replaced by multistage adaptive testing in the main measurement area (reading literacy) (Yamamoto et al., 2019). Multistage adaptive testing means that the next part of the assessment is selected based on the student's previous answers, adjusted in difficulty to the student's estimated ability point.

The aim of our research is to explore and summarize the international scientific results related to the computer-based administration of PISA, TIMSS and PIRLS assessments. We present an overview of the systematic review process of the literature in the international scientific space, which included the publications available in EBSCO, ERIC, JSOTRE, Science Direct, Web of Science and ProQuest databases and the technical reports of PIRLS, PISA and TIMSS. Finally, we summarize the findings on the difference between paper-and-pencil and computer-based administration as known as mode effect or item equivalence.

Keywords:

Mode effect, paper-and-pencil test, computer-based test, PISA, systematic literature review.