Abstract:

Artificial intelligence (AI) literacy is increasingly recognized as a key competence in higher education, as AI systems become embedded in academic, professional, and social practices. A growing number of frameworks define AI literacy as a multidimensional construct, encompassing not only technical knowledge and ethical awareness but also critical understanding, interpretation, and responsible judgment when interacting with AI systems. However, far less attention has been given to how these competencies are operationalized and assessed in empirical research.

This paper presents a critical narrative synthesis and conceptual-measurement mapping of existing AI literacy assessment tools developed for higher education. Drawing on an updated dataset of validated instruments and conceptual frameworks, we categorize tools according to their format (objective tests, self-report scales, and framework-based instruments), their stated purposes, and the constructs they claim to measure. We then compare these operationalized constructs with the broader theoretical dimensions of AI literacy described in the literature.

The analysis reveals a consistent pattern: most assessment tools primarily capture factual knowledge, self-perceived competence, or general attitudes toward AI, while key theoretically emphasized dimensions—such as the ability to interpret AI-generated outputs, evaluate uncertainty and limitations, and exercise informed and reflective judgment—remain largely unmeasured. This misalignment suggests that current assessment practices may reduce AI literacy to familiarity with concepts or tools, rather than capturing the deeper epistemic and critical competencies that contemporary frameworks advocate.

The paper concludes by discussing implications for higher education research and practice, and by outlining directions for the development of a new generation of AI literacy assessment instruments that more adequately reflect the complexity of the construct.

Keywords:

AI literacy, AI literacy assessment, higher education, AI in education, educational measurement, competence assessment.