Abstract:

The strategic framework for education 2030 (ET2030) highlights EU countries cooperation and learning from each other to improve the quality of education. The Programme for International Student Assessment (PISA) as an international large-scale survey contributes to the implementation of this strategy as it provides data on 15-year-old students‘ literacy in mathematics, science, and reading for cross-country comparison. PISA gathers also information about student motivational, family, and institutional factors that can explain differences in student performance at individual, school, and country level. However, comparison across countries or different groups in the same country is meaningful only if there is sufficient evidence about the invariance of measurement instruments. Previous studies have analysed the question of measurement invariance for PISA cognitive constructs (e.g., Khorramdel et al., 2020), socio-economic background (e.g. Erylimaz, Rivera-Gutierrez & Sandoval-Hernandez, 2020), perception of teaching behaviour (e.g. Sherer et al., 2016), ICT engagement (Ma, 2020). Invariance testing of motivational constructs has received less attention in research. PISA motivational constructs are latent variables that are evaluated using multiple questions and a student provides self-reported answers. Therefore, there is space to interpret the statements differently depending on the group (for example, country, culture, or similar) and a bias in the assessments might be observed. The purpose of the present study was to analyse the measurement invariance of motivational constructs (competitiveness, work mastery, and fair of failure) across EU countries using PISA 2018 data. Measurement invariance was tested employing a Multi-Group Confirmatory Factor Analysis (MG-CFA) framework as it is one of the best-known procedures (Awisati, Le Donne & Paccagnella, 2019).

The sample included 183824 15-years-old students from 26 EU countries in PISA 2018. We ran the single-group CFA using robust weighted least squares and theta parameterization. We obtained an acceptable model fit for the overall sample: chi-squared =4031.03, df=24, p<.001; CFI=.98; TLI=.97, RMSEA=.03 and SRMR=.028. The standardized item loadings of the measurement model for the overall sample ranged from .67 to .96. Analysing EU countries independently, the goodness-of-fit measures had following ranges CFI (.965 - .986), TLI (.948 - .979), RMSEA (.035 - .097), and SRMR (.024 - .047). All standardized item loadings for the individual EU countries were above .62. For the overall sample and for each EU country, we obtained that all observed items are valid indicators of motivational constructs. Using MG-CFA, we estimated configural, metric and scalar models for measurement invariance of motivational constructs across EU countries. Although the chi-squared difference test between the configural and metric model was significant (delta chi-squared=2394.5, df=150, p<.001), other model fit indices indicated that the more constrained metric model did not have a poorer fit than the less constrained configural model. Therefore, we were able to confirm metric invariance which indicates the factor loading equivalence of items. Differences between countries could be interpreted as real differences between students rather than possibly determined by the country’s cultural environment. However, we did not detect scalar invariance.

Keywords:

Measurement invariance, PISA, motivation, EU countries